Compare commits

...

202 Commits

Author SHA1 Message Date
overtrue debd664016 ci: model server as a composition root 2026-07-29 10:38:01 +08:00
Zhengchao An 957080bea5 fix(swift): persist container and account metadata writes (#5398)
Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.

Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.

Persisting these writes required hardening the paths that now produce
durable state:

- Account metadata writes validate account ownership. This metadata
  holds the account's TempURL signing key, so an unauthenticated write
  for someone else's account would have become a durable, cluster-wide
  takeover of that account's pre-signed URLs. Reads stay open because
  TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
  metadata loader's "no metadata on disk" default would be persisted,
  creating an orphan metadata file and caching a fabricated default as
  authoritative.
- Container and account metadata are size- and count-limited, reusing
  the Swift limits object metadata already enforces; these tags land in
  the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
  unreadable, instead of merging onto an empty set and wiping the
  container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
  now carry real disk and quorum detail.

The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
2026-07-29 02:31:11 +00:00
cxymds c1538cf1c3 fix(multipart): serialize complete and abort (#5356)
* fix(multipart): serialize complete and abort

* test(multipart): order abort-first finalization

* fix(multipart): enforce quorum staging cleanup

* fix(multipart): remove stale mutable binding
2026-07-29 01:49:19 +00:00
houseme 5af56cbb02 test(ci): stabilize lifecycle timeout coverage (#5404)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 01:37:29 +00:00
houseme f99956eade test(lifecycle): classify mixed rollout harness results (#5384)
* test(lifecycle): classify mixed rollout harness results

Classify Docker #1508 evidence as strict, baseline, blocked, or failed so tiered-storage baseline runs cannot be mistaken for strict mixed-version rollout closure evidence.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: classify Docker manual transition preemption

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 01:21:23 +00:00
houseme 775279b6fd fix(notify): defer disabled bootstrap storage refresh (#5373)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 17:39:22 +00:00
cxymds eb755e2b97 fix(auth): compare sensitive tokens in constant time (#5371)
* fix(auth): compare sensitive tokens in constant time

* test(auth): scope constant-time source guard

* test(auth): ignore test source in comparison guard

* chore(deps): use constant-time s3s authentication
2026-07-28 17:56:22 +08:00
cxymds 7f146fc5de feat(tiering): model provider version capabilities (#5372) 2026-07-28 17:42:08 +08:00
cxymds 5426237a49 fix(filemeta): preserve canonical version order (#5353) 2026-07-28 17:38:59 +08:00
cxymds d7afa4e38e fix(heal): report partial rename failures (#5355)
* fix(heal): report partial rename failures

* fix(log-analyzer): track partial heal rename failures
2026-07-28 17:37:02 +08:00
cxymds a3f5a8eaf9 fix(ecstore): fence object tagging updates (#5360)
* fix(ecstore): fence object tagging updates

* test(ecstore): stabilize tagging lock regressions

* test(ecstore): restore setup type on current-thread runtime
2026-07-28 17:36:00 +08:00
cxymds 547c678eed fix(filemeta): reject positive size without parts (#5354)
* fix(filemeta): reject positive size without parts

* test(ecstore): keep optimized read fixture valid

* test(ecstore): keep listing fixtures valid
2026-07-28 17:35:53 +08:00
cxymds df945b275a fix(lifecycle): recover saturated scanner tasks (#5369) 2026-07-28 17:30:52 +08:00
cxymds 2e78a49c95 fix(lifecycle): reject invalid modification times (#5370) 2026-07-28 17:23:21 +08:00
cxymds 7ad0e726db fix(ecstore): use set read quorum for version reads (#5365)
* fix(ecstore): use set read quorum for version reads

* fix(ecstore): convert optimized read batch errors
2026-07-28 17:23:15 +08:00
cxymds 45c3386b68 fix(bitrot): reject trailing shard data (#5358)
* fix(bitrot): reject trailing shard data

* test(ecstore): align bitrot disk fixture type
2026-07-28 17:23:08 +08:00
cxymds 7af92b4f54 fix(ecstore): handle pre-epoch disk health clocks (#5383) 2026-07-28 17:22:26 +08:00
cxymds daa627ee0e fix(auth): enforce presigned URL expiry limit (#5368)
* fix(auth): enforce presigned URL expiry limit

* ci(deny): allow presigned expiry s3s fork

* chore(deps): update presigned expiry validation
2026-07-28 17:11:30 +08:00
cxymds 5c3d3a8220 fix(ecstore): handle mutex poisoning explicitly (#5379)
* fix(ecstore): handle mutex poisoning explicitly

* test(ecstore): satisfy poison assertion lint
2026-07-28 17:09:09 +08:00
cxymds 6bc5fc77b5 fix(notify): emit noop event for missing deletes (#5381) 2026-07-28 17:05:10 +08:00
cxymds e822fc1552 fix(swift): enforce cumulative container quotas (#5378) 2026-07-28 17:05:04 +08:00
cxymds 2f6115e058 fix(swift): enforce TempURL IP restrictions (#5377) 2026-07-28 17:04:58 +08:00
cxymds 882e1a71b1 fix(tiering): abort failed multipart restores (#5367)
* fix(tiering): abort failed multipart restores

* fix(tiering): compile restore abort regressions

* test(tiering): assert restored multipart metadata

* test(tiering): parse completed restore status
2026-07-28 17:04:52 +08:00
cxymds b432f31c2c fix(multipart): preserve retried parts on quorum failure (#5363)
* fix(multipart): preserve retried parts on quorum failure

* style(multipart): format transaction rollback

* fix(proto): regenerate multipart transaction RPCs

* fix(multipart): import rollback marker constant

* fix(multipart): export transaction action

* fix: import multipart transaction test requests
2026-07-28 17:04:46 +08:00
cxymds 6e0640444e fix(multipart): list uploads across all sets (#5362) 2026-07-28 17:04:40 +08:00
cxymds 4fb9b0dc7f fix(authz): fail closed on policy load errors (#5359)
* fix(authz): fail closed on policy load errors

* test(authz): cover policy failure precedence

* test: align policy failure expectations

* test: align upload part copy fail-closed expectation
2026-07-28 17:04:35 +08:00
唐小鸭 2216f00cfd fix(kms): unify persisted SSE data key envelopes (#5343)
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation

* feat: enhance local SSE DEK handling with JSON envelope format and versioning
2026-07-28 17:02:18 +08:00
cxymds fd2a87d47e fix(s3): reject empty multipart completions (#5352)
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-28 08:35:50 +00:00
houseme 1e10d752b9 test(e2e): stabilize inline full gate checks (#5351)
Align inline fast path cluster tests with the current tier mutation and manual transition contracts while keeping msgpack compat assertions focused on observed traffic.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 04:31:19 +00:00
houseme 362f6026ac chore(build): pin Rust toolchain configs to 1.97.1 (#5350)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 03:53:32 +00:00
GatewayJ 5cedab09ab fix(sts): return Query API-compatible responses (#5282)
* fix(sts): return Query API-compatible responses

* fix(sts): resolve review and CI failures

* fix(sts): harden query response conversion

* test(e2e): stabilize transition conflict coverage

* fix(ilm): retry vanished transition admission CAS

* test(e2e): narrow transition overlap coverage

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-28 10:39:17 +08:00
houseme d385cea7c6 test(lifecycle): add manual transition rollout harnesses (#5346)
* test(lifecycle): add manual transition rollout harnesses

Add a dedicated #1508 ignored E2E harness for non-empty manual transition rollout readback and a reusable Docker mixed-version harness entrypoint.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(lifecycle): satisfy rollout harness clippy

Use unwrap_or_default for the optional transition_failed report field while preserving the zero-default status readback behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 00:04:47 +00:00
Henry Guo d5df66ac4f fix(scanner): require authoritative usage snapshots (#5333) 2026-07-28 07:29:57 +08:00
Zhengchao An 5a768d3a44 perf(s3): resolve bucket versioning once per DeleteObjects request (#5340)
DeleteObjects rebuilt ObjectOptions inside its per-key phase-1 loop, and
del_opts re-fetched the bucket versioning configuration on every call: a
1000-key request paid 1001 identical metadata-sys lookups, each taking the
global bucket-metadata read lock and cloning a VersioningConfiguration.

Split del_opts into the existing async entry point plus a synchronous
del_opts_with_versioning that takes an already-resolved configuration, and
reuse the request-level version_cfg the handler had already fetched.

This also removes a snapshot inconsistency. The skip-stat decision
(can_skip_delete_objects_pre_stat) and the post-delete accounting already
derived from the request-level snapshot while del_opts re-read the config
per key, so a PutBucketVersioning committing mid-request could give a key
opts.versioned=true while the batch delete still ran with versioned=false:
the object was deleted outright with replication state attached to a delete
marker that was never created, and the advisory object-lock pre-check was
skipped on a false premise. All three now derive from one snapshot.

GET, HEAD and DeleteObject additionally establish bucket existence before
building ObjectOptions, so a request naming a nonexistent bucket no longer
performs bucket-metadata work first. GET keeps its cheap request-shape
validations (key, range, partNumber) ahead of the existence check so
InvalidArgument still wins over NoSuchBucket for malformed requests.

Bucket validation moves to validate_bucket_exists, which takes an explicit
store and shares the existing 5s-TTL cache. GET, PUT, HEAD and DeleteObject
now resolve the store through the request-bound server context
(backlog#1052 S6) instead of the process-global handle: get_validated_store
resolves the first-published global AppContext, so in an embedded
multi-instance process a request to the second server operated on the first
server's store. Remaining global-handle call sites in bucket_usecase and
select_object are left for a follow-up.
2026-07-28 06:33:17 +08:00
houseme 6d3ce90c0f fix(lifecycle): harden manual transition rollout checks (#5344)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 17:29:06 +00:00
houseme 0e42a3d1d9 fix(audit): load env-only targets at startup (#5342)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 16:41:38 +00:00
abdullahnah92 c3aac2279f fix(site-replication): keep reverse direction after config broadcast (#5292)
* fix(site-replication): keep reverse direction after config broadcast

`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.

Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.

Four defects kept that invisible or unrecoverable:

- `update_all_targets` discarded target-client build errors silently, and
  `replicate_object` logged the resulting missing-target drop at debug
  while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
  destination named a remote site, so two sites holding identical
  configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
  encrypted with a previous root secret decode as "no such account".
  Startup now reconciles the account, reseeding from the secret every
  site-replication bucket target already stores, and refuses the
  delete-then-create sequence when the parent cannot back an account.
  Bucket rules are reconciled too, so an already-broken site heals on
  upgrade.
- A joined site never verified it could reach the initiator, whose
  endpoint is derived from the Host header of the admin request that
  created the topology. The join now probes each peer and reports through
  `initial_sync_error_message`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): report unreachable targets and reconcile on a timer

Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.

Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.

The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(site-replication): invert reconcile dependency to satisfy layers

`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.

Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.

Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.

The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): harden reconcile per review feedback

Addresses the automated review on #5292.

Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.

Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.

Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.

Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.

Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-27 22:39:23 +08:00
houseme 300beff970 fix: classify manual transition tier failures (#5335)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 12:49:08 +00:00
houseme 0a2370c024 docs: clarify outbound allowlist scope (#5336) 2026-07-27 19:57:22 +08:00
Heracles 0c9d721910 docs: document RUSTFS_OUTBOUND_ALLOW_ORIGINS outbound policy (#5334) 2026-07-27 19:50:47 +08:00
houseme f6c227628f test(scripts): capture manual transition failure samples (#5331)
Add a reusable evidence collector for manual transition tier-failure attribution samples and wire the runbook tests to cover its dry-run and argument guards.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 18:49:48 +08:00
houseme 0cbfa1ac90 test(internode): pin JSON fallback compatibility (#5330)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 09:49:24 +00:00
houseme ab5aa54035 fix(ecstore): replay manual transition task journal (#5329)
Recover accepted manual-transition tasks from the durable task journal when worker-result markers are missing after a restart.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 17:36:40 +08:00
houseme 464bf45e15 chore(scripts): add manual transition follow-up monitor (#5328)
* test(scripts): cover manual transition runbooks

Add a script-test entry for the manual transition rollout and soak runbook generators. The test covers ratio fail-fast behavior, generated status/metrics/journal checks, and runnable dry-run artifacts.

Verification:

- ./scripts/test_manual_transition_runbooks.sh

- make script-tests

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(scripts): assert nightly stress snapshot hooks

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(scripts): add transition CI follow-up monitor

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 16:36:41 +08:00
houseme bc2d8e0c60 test(scripts): cover manual transition runbooks (#5327)
Add a script-test entry for the manual transition rollout and soak runbook generators. The test covers ratio fail-fast behavior, generated status/metrics/journal checks, and runnable dry-run artifacts.

Verification:

- ./scripts/test_manual_transition_runbooks.sh

- make script-tests

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 07:56:58 +00:00
GatewayJ 24cf2cdb78 fix(iam): align OIDC parent IDs with MinIO (#5290)
* fix(iam): align OIDC parent IDs with MinIO

* test(iam): cover OIDC STS binding policy lookup

* test(admin): use runtime facade for AppContext setup

* test(ilm): wait for lifecycle backfill before manual failure

* test(ilm): make overlapping admission check concurrent
2026-07-27 07:16:22 +00:00
houseme e076e8cc6e feat(scripts): add runbook entrypoints for #1508 #1510 (#5326)
* feat(ecstore): attribute manual transition worker failures

Add manual transition worker failure reason tracking and persistence recovery compatibility for checksum validation.

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(ilm): add manual transition diagnostics scripts

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(ecstore): format manual transition attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): avoid copying failure reasons via clone

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(manual-transition): improve matrix verification scripts

- Add strict mixed rollout phase ratio validation.
- Add strict read/write ratio validation for soak mix.
- Inline generated admin-check command blocks into mixed-rollout run script.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(scripts): add runbook entrypoints for #1508 #1510

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 06:47:28 +00:00
houseme 9d84056d7b feat(ecstore): attribute manual transition worker failures (#5324)
* feat(ecstore): attribute manual transition worker failures

Add manual transition worker failure reason tracking and persistence recovery compatibility for checksum validation.

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(ilm): add manual transition diagnostics scripts

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(ecstore): format manual transition attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): avoid copying failure reasons via clone

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 06:06:12 +00:00
houseme f566b382a0 fix(bench): formalize paired ABBA corrected artifacts (#5322)
Add a reusable offline rebuild path for pinned paired ABBA benchmark evidence so corrected summaries can be regenerated from raw warp logs after parser fixes.

Record ACK-contract comparability explicitly in the rebuilt product comparison output, preventing relaxed RustFS ACK results from being treated as direct strict MinIO comparisons.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 04:06:51 +00:00
dependabot[bot] a909f2f27b chore(deps): bump the dependencies group with 3 updates (#5321)
Bumps the dependencies group with 3 updates: [jiff](https://github.com/BurntSushi/jiff), [serial_test](https://github.com/palfrey/serial_test) and [hotpath](https://github.com/pawurb/hotpath-rs).


Updates `jiff` from 0.2.34 to 0.2.35
- [Release notes](https://github.com/BurntSushi/jiff/releases)
- [Changelog](https://github.com/BurntSushi/jiff/blob/master/CHANGELOG.md)
- [Commits](https://github.com/BurntSushi/jiff/compare/jiff-static-0.2.34...jiff-static-0.2.35)

Updates `serial_test` from 3.5.0 to 4.0.1
- [Release notes](https://github.com/palfrey/serial_test/releases)
- [Commits](https://github.com/palfrey/serial_test/compare/v3.5.0...v4.0.1)

Updates `hotpath` from 0.21.5 to 0.22.0
- [Release notes](https://github.com/pawurb/hotpath-rs/releases)
- [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.21.5...v0.22.0)

---
updated-dependencies:
- dependency-name: jiff
  dependency-version: 0.2.35
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serial_test
  dependency-version: 4.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-27 02:42:07 +00:00
houseme 5af997ce5f test(ilm): cover manual transition e2e stress (#5320)
* test(e2e): cover distributed manual transition admission

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: cover manual transition restart recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover queue pressure status readback

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 18:46:15 +00:00
houseme 4bd8cc1369 fix(bench): parse final warp report metrics (#5308)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:49:00 +00:00
houseme 3500f2e5ee test(e2e): cover distributed manual transition admission (#5319)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:42:29 +00:00
houseme e0e2eb30a9 test(ilm): cover pending worker budget status (#5317)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:38:28 +00:00
houseme 467fb0a15c test(ilm): cover restart cancel readback (#5316)
* test(ilm): cover restart cancel readback

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(ilm): format restart cancel e2e

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:36:58 +00:00
houseme 0902538ceb test(ilm): cover manual transition restart recovery (#5318)
test: cover manual transition restart recovery

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:33:59 +00:00
houseme fec09968b9 fix(ilm): preserve manual transition duration budget (#5315)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:32:33 +00:00
houseme 4b09239ceb test(ilm): cover parallel bucket admission (#5314)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:30:48 +00:00
houseme cc3c39da5c test(ilm): cover queue pressure job readback (#5313)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:16:53 +00:00
houseme 07f6d5cda6 test(admin): cover manual transition capabilities e2e (#5311)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:14:14 +00:00
houseme e79337bb5c test(ilm): cover queue-full terminal job records (#5310)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:11:47 +00:00
houseme 683ff52a7b feat(internode): track msgpack decode traffic (#5309)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:03:58 +00:00
Zhengchao An 63e57378d6 fix(ecstore): never cache fabricated bucket metadata as authoritative (#5307)
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.

Production changes:
- get_config caches only metadata actually read from disk; misses are
  recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
  set()) so repeated lookups for metadata-less names cost no extra
  namespace-lock + erasure-set fanout (reachable pre-auth via CORS
  preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
  entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
  object-lock gate from the set's own instance context (backlog#1052)
  instead of the ambient facade, so a second in-process instance (or, in
  tests, another test's transient DistErasure window) cannot reroute
  locking onto an empty dist locker list or answer with the wrong
  instance's bucket state.

Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
  registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
  other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
  erasure; tier-free non-serial test modules use them, guard-based
  SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
  the delete gate resolution source, and the ns-lock resolution source.

Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
  report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
  repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
  AGENTS.md; all findings fixed or rebutted with evidence
2026-07-27 00:53:48 +08:00
Zhengchao An b2a376c2d2 Merge commit from fork
* fix(admin): bound IAM import archive expansion

MAX_IAM_IMPORT_SIZE caps the compressed upload at 10 MB, but every member of the
archive was then read with read_to_end into an unbounded Vec. Deflate ratios well
above 100:1 are easy to construct, so a small authorized upload could expand
without limit across the seven members ImportIam reads.

Add a shared expansion budget (MAX_IAM_IMPORT_EXPANDED_SIZE, 10x the compressed
cap) drawn down by every member, and route all seven reads through one helper
that reads a byte past the remaining budget to detect overrun. Sharing the budget
bounds the archive as a whole rather than letting each member spend the full
limit independently.

Covers R03-CAN-024 through R03-CAN-030 plus R04-CAN-077 (backlog #1471) — one
fix rather than seven, since all seven call sites were byte-identical.

* fix(kms): confine local key paths and refuse silent key replacement

Local KMS key identifiers arrive from request input — the `name` tag on CreateKey,
the `keyId` body field or query parameter on DeleteKey — and were joined onto
`key_dir` with no validation. An identifier such as `../../tmp/evil` escaped the
configured directory, making key creation a constrained arbitrary-file write and
`DeleteKey` with `force_immediate` a cross-directory delete.

Validate in `master_key_path` and make it fallible, so every filesystem path in
this backend inherits the guard: decode_stored_key, load_master_key,
save_master_key, create_key and delete_key all derive their paths there. The rule
is containment rather than a character allowlist, so identifiers already in use
keep resolving; only separators, NUL, absolute paths and non-single-component
forms are refused. Note `.` and `..` are contained rather than refused — the
`.key` suffix turns them into the ordinary filenames `..key` and `...key`.

Separately, `LocalKmsBackend::create_key` had no existence check, while the
sibling `KmsClient::create_key` has always had one. Since `save_master_key`
renames over its destination, creating a key under an existing name silently
replaced its material and destroyed the ability to decrypt everything wrapped
under it — and the backend path is the one the admin API uses. It now returns
KeyAlreadyExists, matching StaticKmsBackend.

Covers R03-CAN-072, R03-CAN-073 and R07-CAN-103 (backlog #1475). R03-CAN-073
needed no separate change: delete_key routes both its load and its remove_file
through master_key_path.

* fix(swift): bound SLO manifest reads to the 2 MiB manifest limit

The three Swift SLO handlers that load a stored manifest (handle_slo_get,
handle_slo_get_manifest, handle_slo_delete) read the `<object>.slo-manifest`
object to EOF with AsyncReadExt::read_to_end. That key is predictable and
writable through the ordinary object PUT path, so a tenant can replace the
manifest with an arbitrarily large object and then make the server allocate
its full size on every SLO GET, multipart-manifest=get, or
multipart-manifest=delete request - a memory amplification bounded only by
the stored object size (CWE-400 / CWE-770). The 2 MiB manifest limit that
handle_slo_put enforces was not applied on the read side.

Introduce MAX_SLO_MANIFEST_SIZE (the existing 2 MiB PUT limit, now a named
constant) and a shared read_manifest_bytes helper that reads through a
`take(limit + 1)` and rejects anything larger, so an oversized manifest is
refused instead of being buffered first. All three call sites go through the
helper. handle_slo_put now checks the size before parsing the JSON.

Regression tests: test_read_manifest_bytes_rejects_oversized_manifest and
test_read_manifest_bytes_stops_reading_oversized_manifest (which asserts the
reader is not consumed past the limit), plus a boundary test that a manifest
at exactly 2 MiB is still accepted.

* fix(protocols): authorize every object in FTPS/WebDAV recursive deletes

The FTPS and WebDAV gateways authorized only the container before a
recursive delete and then destroyed everything inside it without a
further check:

- FTPS RMD (and DELE on a bucket path ending in '/') cleared
  s3:DeleteBucket, then delete_bucket_recursively listed the bucket and
  deleted every object.
- WebDAV DELETE on a bucket did the same via its own
  delete_bucket_recursively.
- WebDAV DELETE on a directory cleared s3:DeleteObject for the directory
  marker key ("dir/") only, then listed that prefix and deleted every
  child under it.

A principal holding s3:DeleteBucket (or s3:DeleteObject on a single
marker key) could therefore erase objects it had no s3:DeleteObject
permission for, and the operation reported success.

Deletion stays recursive - that is the expected behaviour for these
protocols - but each object now clears s3:DeleteObject on its own key
before it is removed, and the enumeration clears s3:ListBucket. A denial
aborts the whole operation with access denied rather than being skipped,
so the caller can never be told the delete succeeded while objects were
left behind or removed without authorization.

The test double gained shared-state cloning, delete_object/delete_bucket
call logs, and list/delete queue helpers so the regression tests can
observe that nothing is deleted once a deny lands.

* fix(server,ecstore): bound TLS handshakes and remote volume RPC waits

Three call sites let an unauthenticated client or a misbehaving peer hold
server resources with no deadline.

TLS listener (R03-CAN-035): process_connection awaited
`acceptor.accept(socket)` with no bound. A client that opens a TCP
connection and never finishes the handshake parks a Tokio task and a socket
forever, and the connection cap (RUSTFS_API_MAX_CONNECTIONS) is unlimited by
default, so nothing else sheds it. The handshake now runs under
accept_tls_with_deadline(), reusing the existing HTTP/1 header-read budget —
the established slow-client bound for the pre-request phase — and the
expiry is recorded through the same log/metric path as a handshake error,
under a new TIMEOUT failure kind.

Remote disk RPCs (R03-CAN-049, R03-CAN-050): list_volumes and delete_volume
passed Duration::ZERO, which execute_with_timeout treats as "no deadline",
so a peer that accepts the request and never answers stalls the coordinator
(and, for delete_volume, the bucket-deletion workflow). Both now pass
get_max_timeout_duration(), matching every sibling method in the file.

Regression tests: a silent TLS peer must be shed by the handshake deadline;
list_volumes/delete_volume against a peer that completes the TCP connect and
then goes silent must fail with DiskError::Timeout instead of hanging.

* fix(security): stop leaking signed headers and bound OIDC/KMS credentials

Three independent hygiene fixes found by the security review.

R03-CAN-018 (crates/signer): try_get_canonical_headers and get_signed_headers
logged the complete header map at DEBUG before signing. Runtime callers pass
session credentials and SSE-C key material through these headers, so anyone
able to raise the log level (or read DEBUG logs) recovered
X-Amz-Security-Token and SSE-C keys verbatim. The statements were debugging
leftovers with no operational value and are deleted rather than redacted.

R03-CAN-014 (crates/iam): the OIDC HTTP adapter buffered provider responses
with an unbounded Response::bytes(), so a configured, compromised or
attacker-pointed IdP endpoint could stream an arbitrarily large or endless
body into memory (the ValidateOidcConfig admin handler lets a ServerInfo
caller choose the endpoint). Responses are now read incrementally and fail
closed past MAX_OIDC_RESPONSE_SIZE, and the already SSRF-hardened client
builder gains request and connect timeouts so a stalled provider cannot pin
the calling task indefinitely.

R07-CAN-105 (helm): the Vault KMS token was serialized into the chart
ConfigMap, exposing it to every subject allowed to get ConfigMaps in the
namespace. It now renders into a dedicated Secret that the Deployment and
StatefulSet consume via envFrom; the Secret is separate from the main
credentials Secret so it also works when secret.existingSecret is set.

Regression tests:
- rustfs-signer: signing_never_logs_signed_header_material
- rustfs-iam: oidc_response_body_past_the_limit_is_rejected,
  oidc_response_body_at_the_limit_is_accepted
- scripts/test_helm_templates.sh: KMS token must never render in plaintext

* fix(webdav): enforce body limit, request timeout and connection cap

The configured WebDAV maximum body size was enforced from Content-Length, so a
chunked request declared no length and bypassed it entirely. The configured
request timeout was never applied to the connection at all, and the accept loop
spawned a task per connection with no bound, so an unauthenticated client could
hold resources indefinitely and in unbounded number.

Enforce the limit on bytes actually read rather than the declared length, apply
the configured timeout to the request, and bound accepted connections with a new
RUSTFS_WEBDAV_MAX_CONNECTIONS (default 1024) surfaced in the config report.

Covers R03-CAN-051, R03-CAN-052, R03-CAN-067, R04-CAN-089, R05-CAN-094 and
R05-CAN-097 (backlog #1471, #1474).

* fix(security): stop STS credentials from crossing the parent trust boundary

Two related credential-boundary holes let a short-lived STS credential act
with the full, unrestricted authority of the long-term user it was minted
from.

AddUser (R03-CAN-021, CWE-269/863): should_check_deny_only relaxes the admin
policy check to deny-only when a Console/STS session targets the IAM user it
represents. Nothing then stopped that session from calling AddUser with its
own parent's access key, so the handler wrote an attacker-chosen secret key
and status over the parent's stored Credentials via create_user ->
save_user_identity. A session that expires in minutes became permanent
control of the account. AddUser now rejects any temp or service-account
requester whose resolved parent equals the target access key, resolving the
parent the same way should_check_deny_only does (parent_user field, else the
JWT `parent` claim, since some stores persist the parent only in the token).

FTPS/SFTP/WebDAV password auth (R04-CAN-086, CWE-287/862): these protocols
looked the access key up with check_key, which falls back to the STS account
cache, and then compared only the stored secret. An STS access key plus
secret therefore authenticated with no session token presented and no
session-policy claims applied - the holder got the parent's full permissions.
Password authentication now rejects temporary credentials before the secret
comparison. The discriminator is is_temp() && !is_service_account(), the same
one IamCache::update_user_with_claims uses to route an identity into the STS
cache, so service accounts - which resolve policy from stored IAM state
rather than a client-presented token - keep working over these protocols.

Regression tests cover both predicates and pin the guards to their call
sites so neither can be dropped without a test failure.
2026-07-27 00:22:50 +08:00
houseme 887868e7cd test(admin): expose manual transition job capabilities (#5306)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 00:17:40 +08:00
houseme 0ea7f17fd7 test(ilm): cover transition budget partial records (#5305)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 00:15:58 +08:00
houseme 5532510e42 test(ilm): cover terminal job status readback (#5304)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 14:56:31 +00:00
houseme fc6dfa891a test(ilm): cover heartbeat backpressure status (#5303)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 14:41:18 +00:00
houseme 994678cfcf test(ilm): cover unknown checkpoint counters (#5299)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 14:33:21 +00:00
houseme cee5009c57 test(ilm): cover manual job status cancel matrix (#5302)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 14:07:02 +00:00
houseme bb130e2655 test(ilm): cover async partial continuation resume (#5301)
* test(ilm): cover async partial continuation resume

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(ilm): apply rustfmt to continuation e2e

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 21:59:47 +08:00
houseme 0711a6f4fe test(ilm): cover manual transition journal replay matrix (#5300)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:41:32 +00:00
houseme 0a25d25e68 test(ilm): cover cancelled continuation resume (#5294)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:40:36 +00:00
houseme 009dd93788 test(ilm): cover restart unknown readback (#5298)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:15:25 +00:00
houseme 7cacd1f558 feat(ilm): persist manual transition task journals (#5296)
* feat(ilm): persist manual transition task journal

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): reconcile manual task journals on recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 12:38:41 +00:00
houseme c3242f83ba test(tier): cover reporter committed prepare replay (#5297)
Add a deterministic four-hot AddTier regression for the reporter path where one peer has already durably committed prepare replay and the remaining peers still need commit fanout before local publish.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 12:37:14 +00:00
houseme e0bd18bd50 test(ilm): cover manual transition endpoint contract (#5295)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 12:36:53 +00:00
houseme 7f0c42e2bb test(perf): reject zero-byte paired warp sizes (#5289)
* test(perf): tolerate empty paired runner node args

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(perf): reject zero-byte paired warp sizes

Keep the #1432 pinned paired ABBA default matrix on positive object sizes because warp refuses zero-byte object generation. Add a dry-run guard so zero-byte compatibility stays a separate functional probe instead of failing mid-benchmark.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 20:31:00 +08:00
houseme 09a991e735 test(perf): cover GET fallback metric probes (#5291)
* test(perf): cover GET fallback metric probes

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(perf): prefer format fallback labels

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): remove duplicate manual transition import

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 11:44:53 +00:00
houseme 14c249c266 test(ilm): cover durable checkpoint persistence (#5288)
* test(ilm): cover durable checkpoint persistence

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: format checkpoint persistence imports

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 11:32:23 +00:00
houseme 5acc52b7f9 test(ilm): cover admission scope parallelism (#5293)
* test(ilm): cover cross-bucket admission scope

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): remove duplicate transition progress import

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 11:30:08 +00:00
houseme 26663e0d5d test(ilm): cover durable progress checkpoint renewal (#5287)
Add regression coverage for durable manual transition progress checkpoints so page progress persists the report, queue snapshot, and renewed scope admission lease together.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 10:34:08 +00:00
houseme caeaa4bf34 test(ilm): cover durable transition progress recovery (#5286)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 10:06:24 +00:00
houseme 05f52beea3 test(ilm): cover cancelled worker counter persistence (#5285)
* test(ilm): cover combined manual transition failures

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): route transition matrix imports

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): satisfy transition matrix lint

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover cancelled worker counter persistence

Add a focused manual transition job matrix regression for cancel-requested terminal records that already contain mixed worker completion and failure counters. The test verifies cancelled state reduction preserves worker counters and survives encode/decode.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 10:01:03 +00:00
houseme 25cf7922f5 test(perf): fix GET stage metrics capture (#5284)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:14:24 +08:00
houseme e6706bb94a test(ilm): cover combined manual transition failures (#5279)
* test(ilm): cover combined manual transition failures

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): route transition matrix imports

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): satisfy transition matrix lint

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:14:01 +08:00
houseme a609b92b3c fix(ilm): retry transient scope admission races (#5280)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:13:53 +08:00
Zhengchao An 4e353fb3c8 refactor(storage): resolve staged dead code in store module (#5283)
refactor(storage): resolve staged dead code in store module (#730)

Drop the store module's blanket #![allow(dead_code)] and resolve every
unit it was masking, instead of keeping them staged:

- write_persistent_key_only_index: the keys-only writer wrapper only had
  test callers; the in-process rebuild flow already exists and uses the
  _with_metadata variant (prepare -> rebuild, triggered lazily from the
  opt-in listing path). Tests call _with_metadata directly now.
- ListIndexLifecycle::recover_after_restart / mark_corrupt: production
  derives index health per request from the persisted artifacts (index
  file + namespace mutation journal), and restart recovery already lives
  in load_persistent_key_only_index's journal restore. The persisted-
  lifecycle design these transitions served was superseded, and Corrupt
  had no detector, so the never-constructed Corrupt state/reason
  variants go with them.
- record_list_objects_index_opt_in_fallback (+ helpers): superseded by
  the inline per-site fallback recording in the opt-in listing path; the
  recorder recomputed health with a hardcoded None provider, so it could
  only ever report a disabled-based reason.
- Provider-contract traits (Generation/Page/PageIterator/KeyLookup):
  provider dispatch went the ListObjectsIndexProviderKind enum route;
  only ListMetadataIndexHealth is live (dyn in source-mode selection)
  and stays, minus its unused is_healthy default.
- Delete the unused list_quorum_from_env/RUSTFS_API_LIST_QUORUM pair,
  fold should_resume_local_decommission and
  should_auto_start_rebalance_after_recovered_meta into their surviving
  primitives, drop get_disk_via_endpoint, and cfg(test) the remaining
  test-only conveniences.
2026-07-26 08:54:12 +00:00
houseme 058f81c61f fix(ilm): deduplicate manual transition worker results (#5278)
* fix(ilm): fail closed on unsafe manual job recovery

Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(ilm): format manual recovery imports

Apply rustfmt import ordering after merging main into the manual recovery branch.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): deduplicate manual transition worker results

Persist per-task manual transition worker result markers before applying job counters so duplicate worker completion reports are no-ops. Reconcile persisted markers during drained-queue lease renewal and recovery to restore marker-before-record crash windows. Fail closed when persisted worker result markers are corrupt.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 08:28:33 +00:00
houseme b4e3c7117e fix(ilm): fail closed on unsafe manual recovery (#5276)
fix(ilm): fail closed on unsafe manual job recovery

Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 07:36:58 +00:00
houseme 23f4683f20 test(ilm): cover queue terminal partial reports (#5277)
Cover manual transition terminal job records for queue closed and queue send timeout outcomes so backpressure summaries remain partial with the expected counters and queue snapshots.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 07:21:37 +00:00
houseme a539b33583 test(perf): tolerate empty paired runner node args (#5275)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 07:06:27 +00:00
houseme 4e63ee962f test(perf): preflight GET smoke buckets (#5274)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 07:01:35 +00:00
houseme 12b7f22fca test(ilm): cover stale transition admission reclaim (#5273)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 06:55:36 +00:00
houseme a047bbfcfb test(ilm): cover manual transition worker failure summary (#5272)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 06:45:06 +00:00
houseme 556f8ed62f chore(deps): update flake.lock (#5271)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/20535e4' (2026-07-18)
  → 'github:NixOS/nixpkgs/335f073' (2026-07-24)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/afacd68' (2026-07-19)
  → 'github:oxalica/rust-overlay/fe21243' (2026-07-26)
2026-07-26 06:40:33 +00:00
houseme 02ad75e552 test(tier): cover committed prepare replay fanout (#5270)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:41:37 +08:00
houseme 21c85481b8 test(ilm): cover unknown manual transition worker loss (#5269)
test(ilm): cover unknown state after lost worker result

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:41:26 +08:00
houseme d5409845e2 test(perf): add exact 1MiB ABBA benchmark drivers (#5260)
* test(perf): add exact 1mib get abba harness

Add a backlog#1434 exact-1MiB GET attribution wrapper around the codec streaming smoke runner. The harness fixes the legacy versus codec-legacy ABBA profile order, the outer ReaderStream buffer axis, path-proof artifacts, and GET stage-metrics capture so reviewers can reproduce isolated-host attribution without drifting the experiment matrix.

Verification:

- scripts/test_get_1mib_abba_stage_metrics.sh

- bash -n scripts/run_get_1mib_abba_stage_metrics.sh scripts/test_get_1mib_abba_stage_metrics.sh

- git diff --check

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(perf): add exact 1MiB handoff ABBA driver

Add a backlog#1434 benchmark orchestrator for isolated-host exact-1MiB handoff evidence. The runner records A/B/B/A and C/D/D/C legs, server provenance, expected reader path labels, inner and outer buffer capacity labels, and stage metrics capture arguments while delegating workload execution to the existing enhanced batch benchmark driver.

Verification:

- scripts/test_exact_1mib_handoff_abba.sh

- bash -n scripts/run_exact_1mib_handoff_abba.sh scripts/test_exact_1mib_handoff_abba.sh

- make script-tests

- git diff --check

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 13:41:00 +08:00
houseme 7667b7aaf8 feat(admin): expose ILM expiry status (#5268)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 05:32:28 +00:00
houseme 29b4a98e74 test(ilm): cover same-scope async transition conflict (#5267)
Add a black-box e2e for concurrent manual transition durable jobs with the same bucket and prefix. The test asserts that exactly one request is accepted, the other returns 409 with the active job status and cancel endpoint, and the accepted job reaches a completed terminal status after queued transitions settle.

Verification:

- cargo fmt --all --check

- git diff --check

- CARGO_TARGET_DIR=/private/tmp/rustfs-target-1481-same-scope cargo test -p e2e_test --lib reliant::tiering::test_manual_transition_async_same_scope_conflict_reports_active_job -- --exact --nocapture

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 04:51:51 +00:00
houseme fa7499ce1c fix: treat blank console address as disabled (#5266)
Keep startup validation and console server config consistent for whitespace-only console listener addresses.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 04:35:42 +00:00
houseme 1397a4e7ca feat(ilm): expose lifecycle expiry scanner counters (#5262)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 12:11:03 +08:00
houseme afaea2a3ec test(ilm): cover noncurrent version expiry e2e (#5264)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 12:10:34 +08:00
houseme 78dd2d40d3 test(ilm): cover deterministic manual transition cancel (#5263)
Add a focused ecstore regression that exercises active manual transition cancellation through the existing cancel_check hook after scan progress has been made. The test verifies the cancelled report, dry-run counters, and opaque resume cursor without relying on wall-clock timing.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 04:03:19 +00:00
houseme 342f9f94bc test(ilm): cover manual transition queue pressure status (#5265)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 04:02:44 +00:00
houseme d887e7e31d test(internode): pin msgpack rollout gates (#5261)
Extend the internode gRPC benchmark driver with P2 request-only, canary, after, and rollback phases. The request-only phase proves that requesting msgpack-only without fleet confirmation keeps compatibility JSON fields, while canary and rollback materialize the operator states needed for mixed-version convergence without claiming real fleet soak evidence.

Verification:

- scripts/test_internode_grpc_ab_bench.sh

- bash -n scripts/run_internode_grpc_ab_bench.sh scripts/test_internode_grpc_ab_bench.sh

- git diff --check

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 03:58:47 +00:00
houseme 0eb9dd5bdc test(perf): add pinned paired ABBA benchmark (#5259)
test(perf): add pinned paired abba bench

Add a backlog#1432 paired benchmark orchestrator that runs MinIO/RustFS in an A1/B1/B2/A2 schedule while requiring immutable server identity, source revision or release, and ACK contract labels. The runner delegates actual workload execution to run_object_batch_bench_enhanced.sh so node metrics, resource captures, and provenance stay in the shared artifact format.

Verification:

- scripts/test_pinned_paired_abba_bench.sh

- bash -n scripts/run_pinned_paired_abba_bench.sh scripts/test_pinned_paired_abba_bench.sh

- git diff --check

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 03:51:35 +00:00
cxymds 50c4dcca4f fix(helm): coordinate distributed pod startup (#5258) 2026-07-26 03:37:01 +00:00
houseme bdf3f0484d test(ilm): cover manual job status after restart (#5256)
* test(ilm): cover manual job status after restart

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): retry job status recovery after restart

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 03:29:47 +00:00
houseme e897b2d7bb fix: reject overlapping console listener address (#5257)
Fail startup when the configured console listener resolves to the same listener port covered by the S3 address. This keeps SO_REUSEPORT from hiding an intra-process S3/console endpoint collision.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 03:27:13 +00:00
houseme 92f72c3912 fix(ilm): normalize cancelled manual job reports (#5255)
Ensure cancelled durable manual transition jobs expose a cancelled report both when worker results drain after a cancel request and when legacy persisted records are decoded.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 03:18:53 +00:00
houseme 6fa2d06731 fix(ilm): honor explicit object lock metadata (#5253)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 02:57:37 +00:00
houseme 18ff36c22d test(ilm): cover manual transition job auth matrix (#5254)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 02:55:38 +00:00
harry han 03af8e472b fix(lifecycle): stop tier free-version recovery walk-timeout loop (#5194)
The background tier free-version recovery walk pinned a hardcoded 60s
total wall-clock timeout that overrides every operator knob, so any
bucket whose healthy full walk exceeds 60s fails forever; the failed
run's duration was also subtracted from the next 60s tick, restarting
the walk immediately and pinning CPU and disk I/O.

- Drop the total wall-clock budget on the recovery walk
  (walkdir_timeout: Duration::ZERO) and inherit the operator-tunable
  drive stall budget (RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS) for
  per-call progress, so hung disks still fail fast.
- Back off failed runs from completion time: 60s doubling to a 600s
  cap, reset on success; never subtract the failed run's duration.
- Add RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED (default true) to opt
  out of the recovery worker on deployments with no remote tiers;
  invalid values warn and fail open.

Fixes #5130

Co-authored-by: claude <claude@ehdtn.com>
2026-07-26 02:54:33 +00:00
houseme 42fc840630 test(e2e): cover manual transition active cancel (#5252)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 02:51:24 +00:00
houseme 3fe935982f test(e2e): assert msgpack decode-error controls (#5251)
Extend the inline fast-path mixed-msgpack E2E collector to capture msgpack/json decode-error counters by direction, message, and codec. Assert those counters remain unchanged for the multipart, part-number, SSE, compression, and transition fallback-control scenarios.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 02:07:39 +00:00
houseme 6d8e196f36 feat(ilm): trace lifecycle expiry execution (#5250)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 01:58:48 +00:00
Zhengchao An 5c99ca1328 fix(ecstore): fail fast on missing RPC secret in distributed startup (#5248)
When endpoints include remote nodes and no internode RPC secret is
resolvable, every remote format read fails client-side with "No valid
auth token" and startup dies ~2 minutes later with the misleading
"store init failed to load formats after 10 retries: erasure read
quorum" error (issues #4939, #5153).

Add a preflight to ECStore init that resolves the RPC secret once
before any disk opens: if any endpoint is non-local and resolution
fails, abort immediately with the operator-facing remediation message.
The preflight also emits the "RPC auth secret resolution failed" log
line so the log-analyzer rpc-secret-resolution rule keeps firing for
this scenario. Single-node (all-local) topologies never invoke the
resolver.
2026-07-26 00:28:38 +00:00
Zhengchao An 44d2c3bd34 test(kms): rename secret-named fixture bindings to satisfy logging guardrails (#5247)
scripts/check_logging_guardrails.sh flags any lowercase secret-named
identifier interpolated into a format string. The static-secret-file test
added in #5245 interpolates two fixture bindings (file_secret, env_secret)
into format! when constructing the secret file and env var, tripping the
heuristic and breaking make pre-commit on every branch based on main.

Rename the bindings to file_key_b64 / env_key_b64 so they no longer match
the heuristic. The fixtures are dummy base64 key material written to a
temp file and env var, not log output, and the test coverage is unchanged.
The guard script itself is untouched.
2026-07-25 23:28:26 +00:00
Zhengchao An 7f19e9a465 Merge commit from fork
docs/testing/security-regressions.md requires every fixed advisory to map to a
named, greppable regression test. Rename the tests added with the fixes to carry
their advisory id so `rg -i ghsa` finds them, and add the four rows to the
advisory -> test map.

Adds the FTPS MKD regression test that was missing. It primes the dummy backend
with a successful create_bucket, so the assertion distinguishes "denied at the
authorization boundary" from "backend refused" — without the queued success an
unconfigured create_bucket fails on its own and the test would pass even with
the authorization check removed. Verified it fails when the check is reverted.

DummyBackend gains a Debug impl (FtpsDriver's trait bounds require it) and a
queue_create_bucket_ok helper.

Records in the CI-execution map why ghsa_g3vq_* runs in the default pass despite
sitting behind the ftps feature: the rustfs crate defaults to ["ftps", "webdav"]
and cargo unifies features across the workspace build, so the test executes
there even though `cargo test -p rustfs-protocols` alone would skip it.
2026-07-26 07:10:50 +08:00
Zhengchao An 92f83bfe15 Merge commit from fork
* fix(policy): quantify negated string conditions per value

ForAllValues:/ForAnyValue: negated string operators computed the positive
quantified match and then negated the aggregate. That yields NOT(all match)
and NOT(any match), which is the semantics of the *other* quantifier, so
ForAllValues:StringNotEquals and ForAnyValue:StringNotEquals were exactly
transposed. The same applied to StringNotEqualsIgnoreCase, StringNotLike,
ArnNotEquals and ArnNotLike.

Introduce an explicit Quantifier and push negation into the per-value
predicate for the qualified forms, so ForAllValues requires every request
value to satisfy the operator and ForAnyValue requires at least one.
Unqualified operators keep negating the aggregate, preserving AWS
single-valued-key semantics. Absent keys now follow AWS: ForAllValues is
vacuously satisfied, ForAnyValue is not.

A request value set that is fully contained in or fully disjoint from the
policy set cannot distinguish the two quantifiers, which is why the existing
cases missed this; the new tests use partially overlapping sets.

* fix(auth): keep request headers out of server-derived condition keys

get_condition_values folded every remaining request header into the policy
condition map. HeaderMap lowercases header names, and the server-derived keys
userid, username, principaltype, versionid and signatureversion are lowercase
too, so a header of the same name collided with them. The collision branch used
extend(), and non-quantified string operators match if ANY value in the vector
matches, so sending `userid: admin` was enough to satisfy a condition on
aws:userid. The jwt:/ldap: claim keys were reachable the same way whenever the
credential carried no such claim.

groups was worse than an append: the header loop ran before the cred.groups
block, which is gated on !args.contains_key("groups"), so a `groups:` header
both injected a value and suppressed the credential's real group list.

Resolve claims and group membership before merging headers, then skip any header
naming a key the server already derived or a well-known identity/context key.
The reserved set comes from KeyName so it tracks the key registry; s3:x-amz-*
keys stay mergeable because they mirror request headers by design.

* fix(access): gate the ListBucketVersions fallback on public-access checks

An anonymous request for ListObjectVersions that the bucket policy does not
grant directly falls back to re-checking the grant as s3:ListBucket. That
fallback returned Ok(()) straight away, skipping the two gates the direct grant
passes through: deny_anonymous_table_data_plane_if_needed and the
RestrictPublicBuckets check on the bucket's public-access block.

So a bucket whose policy allows anonymous s3:ListBucket kept serving anonymous
version listings after an operator enabled RestrictPublicBuckets, even though
the equivalent GetObject was correctly denied.

Fold the fallback into policy_allowed so both routes reach the same gates.

* fix(ftps): authorize MKD against the CreateBucket boundary

FTPS MKD creates a bucket but ran no authorization check, so any principal that
could open an FTPS session could create buckets regardless of policy. Every
other operation in this driver authorizes first — LIST, RETR, STOR, DELE and
RMD all call authorize_operation — and the WebDAV gateway checks
S3Action::CreateBucket on the equivalent path.

Add the matching check so MKD clears the same boundary as an S3 CreateBucket.
2026-07-26 06:14:01 +08:00
Zhengchao An 21787a4742 test(kms): cover static secret file config (#5245) 2026-07-25 21:47:07 +00:00
Zhengchao An d874831cec chore(security): add guardrails against secret values in error strings (#5244)
* fix(kms): do not include secret value in static KMS format error

The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().

* chore(security): add guardrails against secret values in error strings

Follow-up to the PR #5222 review finding fixed in PR #5243: a config value
that fails secret-format parsing is typically the raw secret itself, and
error messages are log content — they propagate via ? and are printed by
startup/error logging far from the construction site.

Three layers so this class of leak is caught earlier next time:

- security-advisory-lessons: state explicitly that error/panic messages are
  log content; parse-failure hints must name the env var and expected
  format, never echo the input. Add a matching review prompt.
- adversarial-validation: add a security-reviewer probe that greps
  error-construction sites on secret-bearing config paths for interpolation
  of the raw value, including the duplicated-parse trap where the leak hid.
- check_logging_guardrails.sh: mechanical backstop — flag inline format
  interpolation of secret-named identifiers (secret/password/passwd/
  private_key) in checked files; add crates/kms/src/config.rs (the twin
  parse site) to the checked list. Verified the check passes on the fixed
  tree and catches the original leak at rustfs/src/init.rs:399.
2026-07-25 21:44:23 +00:00
Zhengchao An 6da69180d8 docs: streamline AGENTS.md per prompting best practices (#5246)
docs: add autonomy and approval boundaries to AGENTS.md
2026-07-25 21:34:45 +00:00
Zhengchao An 258b7d6f06 fix(kms): do not include secret value in static KMS format error (#5243)
The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().
2026-07-25 21:24:34 +00:00
Zhengchao An 05886a2c3c docs: fix dead TLS configuration doc links (#5242)
docs.rustfs.com dropped the .html suffix from page URLs, so
https://docs.rustfs.com/integration/tls-configured.html now returns 404.
Point both READMEs at the live canonical URL instead.

Supersedes #5154, which replaced the links with Wayback Machine
snapshots even though the docs site is alive and the page still exists.
2026-07-26 04:56:43 +08:00
唐小鸭 99e1f5fbd2 feat(kms): add static single-key backend (#5222) 2026-07-26 04:12:56 +08:00
Zhengchao An c984bc7251 fix(s3): return Cache-Control on GET object (#5241) 2026-07-26 04:12:11 +08:00
唐小鸭 233865d172 feat(kms): introduce KMS unavailability error and enhance data key handling (#5184) 2026-07-26 04:04:35 +08:00
Zhengchao An c5eb1c69ba test(rpc): cross-process replay/tamper acceptance for internode RPC signatures (#5236) 2026-07-26 04:04:16 +08:00
houseme 77b5e1b64c fix(ilm): report manual transition tier failures (#5238)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 18:02:37 +00:00
houseme 23a5db4012 fix(ilm): preserve lifecycle version groups (#5239)
* fix(ilm): evaluate complete version groups

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): log replication expiry blocks

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): diagnose incomplete noncurrent chains

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover purge-pending version groups

Add regression coverage for lifecycle-only version listing so purge-pending versions stay present for ILM evaluation while the public ListObjectVersions projection remains filtered.

Refs rustfs/backlog#1500

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): log lifecycle evaluation failures

Emit structured lifecycle_evaluation_failed events when version group loading or evaluation fails during immediate and existing-object expiry scans.

Refs rustfs/backlog#1503

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover incomplete noncurrent chains

Pin the fail-closed behavior for noncurrent expiration when successor_mod_time is missing and reuse a stable structured event name for that skip path.

Refs rustfs/backlog#1502

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover one-day noncurrent expiry boundary

Add a runtime regression test proving NoncurrentVersionExpiration Days=1 stays inactive before expected_expiry_time and deletes exactly at the computed due boundary.

Refs rustfs/backlog#1504

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): keep transition checks after incomplete expiry

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 17:42:02 +00:00
houseme 516f7fecc1 fix(tier): converge config after peer recovery (#5240)
Retry peer tier config reloads after committed mutations so recovered nodes converge without requiring a second admin change.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 17:25:37 +00:00
houseme 1e95e6d311 fix(ilm): harden durable transition admission (#5235)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 15:46:55 +00:00
houseme 3cbe3d6b94 fix(tier): reconcile committed mutation replay (#5230)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 15:20:54 +00:00
Zhengchao An 61d4e04d65 feat(rpc): bind canonical body digest into internode mutating disk RPC signatures (#5234)
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures

Binds a domain-separated, length-prefixed canonical request-body digest into the
v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path
attacker on the default-plaintext internode channel can no longer tamper with a
mutation payload (or strip the msgpack `_bin` field to force the JSON fallback
decode) without invalidating the signature.

Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions,
WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile,
RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the
msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by
default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so
rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now
configurable and overflow fails closed with a metric.

Refs https://github.com/rustfs/backlog/issues/1327

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(rpc): satisfy architecture-migration compat-marker guard

Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and
stop backticking env-var/metric names in the cleanup-register entry so the
guard's id extractor only sees the task-id.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:56:27 +08:00
houseme 6974963e20 feat(ilm): add durable manual transition job store (#5229)
* feat(ilm): add manual transition job route contract

Refs #1479

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): add durable manual transition job store

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): harden durable transition job cancellation

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:54:14 +00:00
houseme 7ab0955f8b test(e2e): stabilize tier and storage class checks (#5228)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:21:52 +00:00
houseme 2cf5fd6bfc fix(getobject): avoid duplicate downstream-close metrics (#5231)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:13:47 +00:00
houseme de2c337fae test(e2e): cover async transition limit partial (#5233)
Cover async manual transition jobs that finish as partial because maxObjects truncates the scan, including terminal cancel idempotence and unchanged report counters.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:05:47 +00:00
houseme 5988606e68 test(internode): extend fallback and transition coverage (#5232)
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 13:26:23 +00:00
Hiroaki KAWAI e73ed4f2a1 fix(lock): jitter distributed lock retries (#5113)
* fix(lock): jitter distributed lock retries

* fix(lock): keep retry jitter within bounds

* fix(lock): qualify test size_of usage

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 20:26:37 +08:00
houseme ffde6c43ee feat(ilm): add manual transition job route contract (#5221)
Refs #1479

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 12:22:46 +00:00
cxymds f52dde87d1 fix(object): make version undo preconditions atomic (#5225)
* fix(object): make version undo preconditions atomic

* fix(object): fence metadata-only undo copies

* fix(object): order versions by commit time
2026-07-25 19:26:50 +08:00
houseme 8e83087ba4 fix(tier): handle mutation intent CAS races (#5227)
* test(scripts): expand internode grpc ab env coverage

* fix(tier): handle mutation intent CAS races

Make tier mutation intent advancement retry idempotently when an If-Match CAS loses to a matching committed peer update.

Expand four-node inline fallback E2E coverage for mixed msgpack controls, transition readiness evidence, and per-node environment overrides.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 11:10:13 +00:00
Henry Guo a63b79004c fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative

* fix(scanner): close distributed refresh races

* fix(config): align scanner reload integration

* fix(admin): scope config test helpers

* fix(scanner): harden distributed usage convergence

* fix(scanner): preserve rolling activity compatibility

* fix(admin): expose non-secret optional config values

* fix(scanner): acknowledge distributed dirty usage

* fix(ecstore): make bucket mutations cancellation safe

* fix(scanner): preserve pending dirty acknowledgements

* test(obs): account for superseded scanner metric

* fix(api): reject excess detached bucket mutations

* test: close scanner convergence coverage gaps

* fix(scanner): make path tracking cleanup one-shot

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 18:45:16 +08:00
GatewayJ 0364523dad fix(s3select): enforce query and resource limits (#5028)
* fix(s3select): enforce query and resource limits

* fix(s3select): close query resource limit gaps

* fix(s3select): preserve timeout and stream invariants

* fix(s3select): enforce staged query limits

* fix(s3select): preserve policy error compatibility

* fix(s3select): bound error source traversal
2026-07-25 18:44:53 +08:00
houseme 2dc4d0b651 refactor(getobject): scope downstream close tagging (#5224)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 09:36:19 +00:00
houseme 2ee111ad8b feat(ilm): add durable manual transition jobs (#5223)
Refs #1479

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 09:12:44 +00:00
houseme 9ddb30139d fix(getobject): distinguish downstream output closure (#5220)
fix(getobject): preserve internal broken pipe failures

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 08:22:20 +00:00
Zhengchao An ffd1b94e1f feat(rpc): add server-side internode v2 signature verification (fail-open, method-path binding groundwork) (#5160)
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env

Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before.

Refs https://github.com/rustfs/backlog/issues/1327

* ci: fix internode auth test lint failures

* perf(ecstore): cache internode sig strict env

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 08:17:49 +00:00
houseme ce41adfa9b test(ilm): cover manual transition e2e gaps (#5219)
* test(ilm): cover manual transition queue pressure

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(protos): serialize msgpack-only env tests

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 07:00:24 +00:00
cxymds 59361ed786 feat(site-replication): make repair operations durable (#5217)
* feat(site-replication): make repair operations durable

* fix(site-replication): serialize durable repair execution
2026-07-25 10:35:37 +08:00
houseme dfb0a20048 test(rpc): add corrupt-json decode coverage (#5216)
* test(rpc): add corrupt-json decode coverage

* test(ilm): assert manual transition status-less contract
2026-07-25 01:03:59 +00:00
cxymds 7320d7fab2 fix(replication): make resync starts atomic (#5215) 2026-07-25 08:58:06 +08:00
Zhengchao An 28d19db9fc fix(console): handle apple touch icon paths (#5214) 2026-07-25 08:06:48 +08:00
dependabot[bot] e257573962 chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 in /fuzz in the cargo group across 1 directory (#5210)
chore(deps): bump quinn-proto

Bumps the cargo group with 1 update in the /fuzz directory: [quinn-proto](https://github.com/quinn-rs/quinn).


Updates `quinn-proto` from 0.11.14 to 0.11.16
- [Release notes](https://github.com/quinn-rs/quinn/releases)
- [Commits](https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.14...quinn-proto-0.11.16)

---
updated-dependencies:
- dependency-name: quinn-proto
  dependency-version: 0.11.16
  dependency-type: indirect
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-25 08:05:02 +08:00
cxymds 45b675c641 fix(replication): report authoritative backlog metrics (#5209) 2026-07-25 01:32:50 +08:00
cxymds 05caec0bd5 feat(replication): structure active check results (#5208) 2026-07-25 01:28:03 +08:00
cxymds eaa17e0441 feat(admin): add encrypted diagnostic archives (#5207) 2026-07-25 01:19:47 +08:00
cxymds 2b6cc0ee08 fix(admin): redact and seal config history (#5206) 2026-07-24 23:54:24 +08:00
cxymds 938f7296f9 fix(admin): report truthful diagnostic capabilities (#5205) 2026-07-24 23:33:45 +08:00
cxymds 866ac5073d fix(kms): validate configured backend cleanup (#5204)
test(kms): validate configured backend cleanup
2026-07-24 23:24:40 +08:00
houseme 187a060919 test(internode): pin msgpack gate rollback coverage (#5203)
Add test coverage for the msgpack-only request/fleet confirmation truth table and exact request JSON policy manifest.

Pin request and response rollback behavior so removing either gate restores JSON compatibility while both gates enter msgpack-only for eligible fields.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 15:16:23 +00:00
houseme cda443bd81 feat(ilm): reject overlapping manual transition runs (#5202)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 14:56:42 +00:00
houseme 44b1916103 fix(getobject): enrich stream error context (#5201)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 14:47:56 +00:00
Henry Guo 9d1b10144f perf(ecstore): avoid redundant leaf object scans (#5176)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-24 22:25:03 +08:00
cxymds d7f30fe0a2 fix(replication): persist sanitized resync errors (#5200) 2026-07-24 22:15:51 +08:00
houseme 20c4ea864a fix(ecstore): fix wide-prefix listing stalls and decode downstream logging (#5198)
* fix(ecstore): handle list stall and downstream decode logs

* docs(ecstore): keep wide-directory stall context for list_dir
2026-07-24 14:05:27 +00:00
houseme d1c2c42c90 fix(internode): align msgpack benchmark gates (#5197)
Update the internode gRPC benchmark P2 env output to require both the msgpack-only request flag and the fleet-confirmed gate before measuring a true msgpack-only phase.

Document the dual-gate rollback semantics and add a dry-run script test so the benchmark driver cannot silently regress to the single-flag flow.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 13:32:44 +00:00
cxymds fc43b149c5 fix(admin): restore complete config history snapshots (#5196) 2026-07-24 21:28:12 +08:00
cxymds fa235e9018 fix(s3): list multipart uploads by bucket prefix (#5195)
* fix(s3): list multipart uploads by bucket prefix

* fix(s3): preserve exact-key crash reclamation
2026-07-24 21:28:06 +08:00
houseme dd46de0945 fix(heal): report no-parity bitrot as unrecoverable (#5192)
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 20:20:40 +08:00
houseme bf6f0e5e81 test(internode): pin msgpack compatibility send sites (#5193)
test(internode): pin msgpack compat send sites

Add a checked test manifest for internode dual-encoded request and response payloads so node.proto _bin fields must be explicitly classified as msgpack-only eligible or always dual-write.

The manifest also pins the current JSON encoder call sites for request and response send paths, making future send-site drift visible before rollout.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 12:07:11 +00:00
houseme beb807ae2b perf(protos): cache internode msgpack-only flag (#5191)
Cache the internode msgpack-only env gate after the first read so metadata RPC send paths avoid repeated environment parsing. Keep a reset hook for tests that intentionally change the env in-process.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:54:14 +00:00
houseme d8426dc459 feat(ilm): gate durable manual transition mode (#5190)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:53:19 +00:00
cxymds f46ea6e14f fix(s3): enforce SSE-C copy key validation (#5185) 2026-07-24 19:11:36 +08:00
houseme fa26b6730d test(ilm): deny manual transition for non-admin (#5188)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:10:31 +00:00
houseme 7876319811 test(ilm): cover manual transition queue pressure (#5189)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:10:04 +00:00
houseme ea417b6a32 feat(ilm): trace manual transition runs (#5187)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 10:58:04 +00:00
houseme 4963412265 fix(internode): guard msgpack-only JSON fallback (#5180)
Keep internode JSON compatibility fields unless operators explicitly confirm fleet-wide msgpack-only readiness. This prevents a single legacy rollout flag from emptying JSON fields in mixed-version clusters where older peers may still read the legacy JSON payload.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 17:57:51 +08:00
houseme 8ac618e6c2 feat(ilm): count already transitioned manual runs (#5177)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 17:52:56 +08:00
GatewayJ 1c88aa43c1 fix(iam): virtualize OIDC service account parents (#5152)
* fix(iam): preserve OIDC service account policy boundary

* fix(iam): virtualize OIDC service account parents

* fix(iam): reject malformed OIDC policy boundaries

* test(iam): isolate federated policy regression

* fix(iam): keep OIDC replication envelope off claims
2026-07-24 17:41:52 +08:00
houseme a5a73610b6 fix(ecstore): self-verify no-parity bitrot writes (#5179)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 17:36:30 +08:00
cxymds 9eaf5fc8e3 fix(s3): complete CopyObject checksum support (#5178) 2026-07-24 16:57:11 +08:00
houseme 3132637294 feat(ilm): bound manual transition duration (#5174)
* feat(ilm): bound manual transition duration

Add maxDurationSeconds handling for manual transition runs so operators can bound long scoped scans without changing the existing enqueue_only job contract.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(ilm): document manual transition run limits

Document the enqueue_only manual transition contract, secret-handling guidance, and best-effort duration budget while pinning the new duration flag in the e2e response model.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 07:54:52 +00:00
cxymds 358caa23cb fix(storage): expose truthful storage class capabilities (#5172) 2026-07-24 15:28:03 +08:00
houseme 6765aca3f9 feat(ilm): add manual transition run endpoint (#5171)
* feat(ilm): report manual transition backfill outcomes

Add scoped lifecycle transition backfill reporting for backlog #1478 and expose enqueue outcomes needed by #1479 without changing the existing scanner/compensation bool API.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(admin): add manual transition run endpoint

Add a bounded POST /rustfs/admin/v3/ilm/transition/run API for backlog #1477 and cover the route, policy, query parsing, and partial status contract needed by #1481. Console operations from #1480 are intentionally left for a later client integration.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): allow manual transition resume markers

Accept additive marker and versionMarker parameters on the bounded manual transition run API so clients can continue from a partial report without changing the existing default scan behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): harden manual transition partial reports

Preserve the null-version cursor contract, stop manual scans on enqueue pressure without skipping the failed object, and keep raw resume markers out of admin JSON responses.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): add manual transition e2e coverage

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ilm): keep transition enqueue hot path direct

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 15:13:22 +08:00
houseme 4133fbe0fc fix(storage): cover inline reader fallback controls (#5169)
* test(ecstore): cover inline fast path boundaries

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(storage): cover inline reader fallback controls

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(tier): keep commit fanout concurrent

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 06:08:17 +00:00
cxymds 6f6d8a4d3e fix(s3): persist multipart upload storage class (#5170)
Fixes rustfs/backlog#1464.
2026-07-24 13:47:25 +08:00
cxymds cb344a3c77 fix(s3): honor CopyObject replacement metadata (#5168)
Fixes rustfs/backlog#1463.
2026-07-24 12:33:33 +08:00
Zhengchao An 36e97aba26 fix(security): reject federated STS credentials that collide with the root principal (#5161)
A federated OIDC / AssumeRoleWithWebIdentity login derives parent_user from the external identity via session_identity() (username -> email -> sub -> oidc-user-unknown). At IAM request time a temporary credential whose parent_user equals the root access key is treated as owner (see rustfs/src/admin/auth.rs and the rustfs_iam owner resolution used by the policy engine and service-account paths). A federated identity whose display name happens to equal the root access key would therefore be silently granted full owner access on a string match alone.

Reject such issuance up front: immediately after parent_user is derived, and before credential generation, set_temp_user, and the site-replication hook, deny the binding when parent_user equals the root access key. Both federated flows funnel through the single DefaultFederatedSessionBinding::bind -> issue_credentials chokepoint, so this covers browser OIDC callback and AssumeRoleWithWebIdentity alike.

- rustfs/src/admin/service/federated_identity.rs: issue_credentials resolves the root access key via current_action_credentials() and rejects with FederatedSessionBindingError::InvalidRequest on collision. Added the pure helper parent_user_is_reserved(parent_user, root_access_key) so the collision decision is unit-testable without process globals; the comparison is exact and case-sensitive, mirroring the request-time owner comparison.

This is the federated-principal / root isolation pre-work (batch 0A, the immediate issuance-time block) from the OIDC review in rustfs/backlog#1437. Layer 2 (a signature-protected federated credential origin that forces is_owner=false at request time regardless of the parent_user string) is a separate follow-up. Already-issued colliding credentials continue on their existing TTL and should be revoked proactively. Legitimate federated identities are unaffected; only an exact collision with the configured root access key is denied.
2026-07-24 11:32:05 +08:00
cxymds a2fc6e15df fix(s3): honor CopyObject tagging directives (#5165)
Closes rustfs/backlog#1462.
2026-07-24 10:30:43 +08:00
houseme 0269c47bc6 fix(runtime): update codec and debug stack headroom (#5164)
* fix(runtime): raise debug worker stack headroom

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(deps): update codec to 8.0.2

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 09:27:49 +08:00
Zhengchao An d26adc29ca fix(security): pin OIDC discovery/token client to the outbound egress policy (#5159) 2026-07-23 17:13:33 +00:00
houseme 5131ba8271 chore(deps): refresh workspace dependencies (#5157)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-23 15:59:37 +00:00
houseme 14f31b797a feat(bench): capture node attribution metadata (#5158)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-23 15:56:48 +00:00
houseme 9f61bad94f chore(deps): update erasure codec and russh (#5155)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-23 18:13:33 +08:00
Zhengchao An 8e214104f3 perf(ecstore): add guarded encrypted multipart range seeks (#5145)
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend

Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.

* fix(ecstore): guard encrypted multipart range seeks

* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix

Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
2026-07-23 08:33:54 +00:00
Zhengchao An 6bab9e421b fix(iam): route issuer-relative JWKS through config URL (#5150) 2026-07-23 13:46:51 +08:00
Zhengchao An d9e0a25174 fix(oidc): support separate discovery issuer (#5149) 2026-07-23 02:44:32 +00:00
383 changed files with 81357 additions and 6646 deletions
@@ -84,6 +84,9 @@ Null report example: "Rewrote the diff as an in-place edit (no smaller equivalen
- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
- Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
- Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
- Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
- Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
- Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
- Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
+8 -2
View File
@@ -10,10 +10,16 @@ never weaken a check to get green.
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
upward imports. Known legacy violations live in
Enforces `composition (server, startup/init) → interface (admin,
storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
files are composition roots, while imports of their exported HTTP contracts
are classified as interface dependencies. Known legacy violations live in
`scripts/layer-dependency-baseline.txt`.
Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
Inline `#[cfg(test)]` imports remain checked under their source file's layer;
move architecture-crossing test scaffolding into a dedicated test module.
- **New violation**: restructure your change so the dependency points
downward (move the shared type/function to the lower layer).
- **You legitimately removed a baseline entry**: run
@@ -99,6 +99,8 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
@@ -139,6 +141,7 @@ Use these prompts while reviewing a diff:
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
+5
View File
@@ -26,6 +26,11 @@ script-tests: ## Run shell script tests
@echo "Running script tests..."
./scripts/test_build_rustfs_options.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_exact_1mib_handoff_abba.sh
./scripts/test_pinned_paired_abba_bench.sh
./scripts/test_manual_transition_runbooks.sh
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+33 -17
View File
@@ -1,17 +1,14 @@
# nextest configuration for RustFS.
#
# Serialize two known load-sensitive / global-state-sharing ecstore test groups
# so the full parallel nextest suite stops producing spurious failures
# (backlog #937). These tests pass in isolation but flake under the loaded
# parallel run for two distinct reasons:
# Serialize the ecstore tests that share the process-wide disk registry or
# exercise a multi-disk commit handoff across nextest process boundaries.
#
# * store::bucket::tests::bucket_delete_* share process/global state (disk
# registry, lock client) and race make_bucket into InsufficientWriteQuorum
# when run concurrently with other ecstore tests.
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
# asserts a lock-acquire correctness property whose serialized cross-disk
# commits exceed the (already max'd, 60s) acquire deadline only when the
# suite saturates disk I/O.
# uses the shared multipart fixture and a deterministic uploadId-lock
# handoff, so it must not overlap another process mutating that fixture.
#
# serial_test's #[serial] attribute does NOT serialize these across runs:
# nextest executes each test in its own process, where the in-process
@@ -39,6 +36,7 @@ ecstore-serial-flaky = { max-threads = 1 }
# servers never run at once. ci-7's nightly picks these up via the e2e suite;
# they are deliberately NOT in the fast PR `e2e-smoke` filter.
e2e-reliability = { max-threads = 1 }
e2e-inline-boundaries = { max-threads = 1 }
# --- default profile (local): serialize the flaky groups, never retry --------
[[profile.default.overrides]]
@@ -54,6 +52,12 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the durable manual-transition checkpoint test across nextest's
# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
@@ -61,6 +65,10 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
test-group = 'e2e-reliability'
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -89,13 +97,6 @@ path = "junit.xml"
# profile's own overrides list, not the default profile's).
# ===========================================================================
# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
# under saturated disk I/O in the full parallel suite.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
retries = 2
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
# make_bucket into InsufficientWriteQuorum via shared global state under load.
[[profile.ci.overrides]]
@@ -103,6 +104,11 @@ filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(
test-group = 'ecstore-serial-flaky'
retries = 2
# Keep the deterministic multipart handoff isolated across nextest processes.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
# on producer/consumer timing windows that stretch past the budget on loaded
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
@@ -125,6 +131,12 @@ test-group = 'e2e-reliability'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the durable manual-transition checkpoint test under the ci profile
# too. No retries: failures stay visible.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# ---------------------------------------------------------------------------
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
# ---------------------------------------------------------------------------
@@ -156,7 +168,7 @@ test-group = 'ecstore-serial-flaky'
# the nightly profile derives its set as "the replication module MINUS this
# allowlist", so any new replication test lands in nightly by default (never
# silently unrun) until it is explicitly blessed as fast here. Keep the two
# regexes byte-identical. Count invariant: 20 here + 27 nightly = 47 total
# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
# (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -192,7 +204,7 @@ test-group = 'ecstore-serial-flaky'
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
@@ -211,7 +223,7 @@ fail-fast = false
# and poll until source and target converge; two replicate over HTTPS, two
# pin active SSE failure contracts, and one guards event/history observers.
# The SSE-S3 contract remains ignored under backlog#1291.
# * 11 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
# * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
# servers and drives the cross-process site-replication control plane.
# * 1 `_real_three_node` site-replication test.
# * 1 `_real_single_node` service-account round-trip test.
@@ -314,3 +326,7 @@ path = "junit.xml"
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
test-group = 'e2e-reliability'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
+1 -1
View File
@@ -141,7 +141,7 @@ jobs:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 60
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
+35 -8
View File
@@ -172,7 +172,7 @@
],
},
{
"name": "Debug executable target/debug/rustfs with sse",
"name": "Debug executable target/debug/rustfs with sse kms",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
@@ -200,7 +200,7 @@
// 2. kms local backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "local",
// "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
// "RUSTFS_KMS_KEY_DIR": "/tmp/kms-key-dir",
// "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
@@ -212,13 +212,40 @@
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 4. kms vault transit backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "vault-transit",
// "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
// "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
// "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 5、kms static backend test key
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "vault-transit",
"RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
"RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
"RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
"RUSTFS_KMS_BACKEND": "static",
"RUSTFS_KMS_STATIC_SECRET_KEY": "rustfs-master-key:2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM="
},
"sourceLanguages": [
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with local sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_SSE_S3_MASTER_KEY": "xGb3aYSp825j2tPpg8JrUzghiXsIkfdOtmrsJ/iafiM=",
"RUST_LOG": "rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
},
"sourceLanguages": [
"rust"
+6
View File
@@ -21,6 +21,12 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
## Autonomy and Approval Boundaries
- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
## Communication and Language
- Respond in the same language used by the requester.
Generated
+125 -81
View File
@@ -56,9 +56,9 @@ dependencies = [
[[package]]
name = "aes"
version = "0.9.1"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1fc76eaeac4c9164506c466d4ffdd8ec9d0c5bf57ee97177c4d8eceb3a0e138"
checksum = "f8eb277bec05f56a0e0591f155a484cbd0f4f07ff2905051a48c72f004f7ed58"
dependencies = [
"cipher 0.5.2",
"cpubits",
@@ -73,7 +73,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fdf011db2e21ce0d575593d749db5554b47fed37aff429e4dc50bc91ac93a028"
dependencies = [
"aead",
"aes 0.9.1",
"aes 0.9.2",
"cipher 0.5.2",
"ctr",
"ghash",
@@ -850,9 +850,9 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "aws-config"
version = "1.10.0"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "701418aa459dac33e50a0f8e818e5662a16bc018a6ac7423659b70f3799d67a8"
checksum = "1b180a3c8b55960db3426d8964b8745e652466a1a49fe1a2eda828046d30b5e4"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -917,9 +917,9 @@ dependencies = [
[[package]]
name = "aws-runtime"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6b50a43f3ccdf331521c6d6c68b7cc9668b6e09d439ebda9569df5722324d76"
checksum = "c9007227e10b5fed2f3e0a2beff489211e2b5604c400b7a9d5d81ca9d64c24bb"
dependencies = [
"aws-credential-types",
"aws-sigv4",
@@ -945,9 +945,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.139.0"
version = "1.140.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a159b9721a6a41468f967d1029bece78f410b0beb0594498435deb6ff72bfe48"
checksum = "e9660cf991e512fbe6094f1041ff3d3282bc5aeeeb6184e8de437ec2de024a10"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -982,9 +982,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
version = "1.104.0"
version = "1.105.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b53416d16c278234845392e38d93bd4481d2f09daa0f005a2277f0aa91f59c22"
checksum = "6ffd0fbe7873cb548a7aa60f9573c268fff94155397fd4f14dc9f1ecaaab8516"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1008,9 +1008,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
version = "1.106.0"
version = "1.107.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc9b706c3305ed0285d5b1b696c747aa34950f830fb03e3e6c76890f99b9f188"
checksum = "175763eb222a46377df7aa257a3bca980ab3e96703fefc8f4d0b8da6ad2e254c"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1034,9 +1034,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
version = "1.109.0"
version = "1.110.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32d214cdfa5bbe17f117e76a7643fadf32a5234fb597322ef8b1fb4b2f17dbbd"
checksum = "dd8b14781dfbff48984017d57167b6ea0b6471c6920ec52b44a2677c7feb3c13"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1210,9 +1210,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime"
version = "1.12.0"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bea94a9ff8464016338c851e24b472d7131c388c88898a502e781815b2ee6045"
checksum = "07505b34e8f4b3591a4fa69e9792b52289b95488dbbc68c3c0075b7bedb245e1"
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
@@ -1236,9 +1236,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api"
version = "1.13.0"
version = "1.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22ed1ebe6e0a95ea84570225f5a8208dec4b8f77e61a9b0d6f51773fcb4612f0"
checksum = "3b98f2e1fd67ec06618f9c291e5e495a468e60519e44c9c1979cd0521f3affdb"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api-macros",
@@ -1427,6 +1427,12 @@ version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]]
name = "base64"
version = "0.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9"
[[package]]
name = "base64-simd"
version = "0.8.0"
@@ -1697,9 +1703,9 @@ dependencies = [
[[package]]
name = "camino"
version = "1.2.4"
version = "1.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5f2d30e4173c4026932d51d31d6b0613b1fd3014bf3f9f8943d4ba139c437ba0"
checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d"
dependencies = [
"serde_core",
]
@@ -1754,9 +1760,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.3.0"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c89588d05638b5b4594a3348a2d6c20277e43a7f5c5202b05cc56888475a47b8"
checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -3619,13 +3625,13 @@ dependencies = [
[[package]]
name = "displaydoc"
version = "0.2.6"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
@@ -3656,8 +3662,9 @@ dependencies = [
"async-trait",
"aws-config",
"aws-sdk-s3",
"aws-sdk-sts",
"aws-smithy-http-client",
"base64 0.22.1",
"base64 0.23.0",
"bytes",
"chrono",
"clap",
@@ -3670,6 +3677,8 @@ dependencies = [
"hyper-util",
"local-ip-address",
"md5",
"opentelemetry-proto",
"prost 0.14.4",
"rand 0.10.2",
"rcgen",
"reqwest",
@@ -3677,6 +3686,7 @@ dependencies = [
"russh",
"russh-sftp",
"rustfs-config",
"rustfs-credentials",
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-filemeta",
@@ -3788,9 +3798,9 @@ dependencies = [
[[package]]
name = "either"
version = "1.16.0"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
checksum = "9e5e8f6c15a24b9a3ee5efec809ccd006d3b30e8b3bb63c39af737c7f87daa1d"
[[package]]
name = "elliptic-curve"
@@ -3936,11 +3946,10 @@ dependencies = [
[[package]]
name = "event-listener"
version = "5.4.1"
version = "5.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab"
checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2"
dependencies = [
"concurrent-queue",
"parking",
"pin-project-lite",
]
@@ -4375,7 +4384,7 @@ dependencies = [
"hex",
"hmac 0.13.0",
"http 1.4.2",
"jsonwebtoken",
"jsonwebtoken 10.4.0",
"reqwest",
"rustc_version",
"rustls",
@@ -4901,9 +4910,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.21.5"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "902f40dea4993d99db66732fddd9143e92657f1d47a642395f92b339b1375659"
checksum = "66750a77f4f6b408a148be5102ef1f3ba7172def7ee92b1cfc75d9f7a3870453"
dependencies = [
"arc-swap",
"cfg-if",
@@ -4923,9 +4932,9 @@ dependencies = [
[[package]]
name = "hotpath-macros"
version = "0.21.5"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "191f6e3b11c1f817e5c3737158812bc8d3a383e9d9d89526703ea6dc1e9fe647"
checksum = "afe0e1900d2dbe2e2df8e9522b97ebd7a5598ba18478f57e247957022dffedbe"
dependencies = [
"proc-macro2",
"quote",
@@ -5401,9 +5410,9 @@ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jiff"
version = "0.2.34"
version = "0.2.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e184d09547b80eb7e20d141ba2fb1fbac843ca53f4cf1b31210adc4c1adc6e16"
checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc"
dependencies = [
"defmt",
"jiff-core",
@@ -5427,9 +5436,9 @@ dependencies = [
[[package]]
name = "jiff-static"
version = "0.2.34"
version = "0.2.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "323da076b7a6faf914dc677cb05a4b907742ff7375c8322c9e7f5061e5e0e9de"
checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204"
dependencies = [
"jiff-core",
"proc-macro2",
@@ -5527,6 +5536,22 @@ name = "jsonwebtoken"
version = "10.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eba32bfb4ffdeaca3e34431072faf01745c9b26d25504aa7a6cf5684334fc4fc"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
"getrandom 0.2.17",
"js-sys",
"serde",
"serde_json",
"signature 2.2.0",
"zeroize",
]
[[package]]
name = "jsonwebtoken"
version = "11.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "881733cbc631fc9e472e24447ce32a64bedf2da498d6d8570b08edc87de71f65"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -5720,9 +5745,9 @@ checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
[[package]]
name = "libflate"
version = "2.3.0"
version = "2.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd96e993e5f3368b0cb8497dae6c860c22af8ff18388c61c6c0b86c58d86b5df"
checksum = "a4da9b700e758e57152a1fd1c52cbdc5727c1aa6d8743dc1acda917398f1d76c"
dependencies = [
"adler32",
"crc32fast",
@@ -6084,9 +6109,9 @@ dependencies = [
[[package]]
name = "metrique"
version = "0.1.28"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aa466af30a9fe0b1db1dae097e0ce7ddac4b666b1d3a9f5c682e889272511dd8"
checksum = "d2e394c63e2d1a30aeb3b9392ecf3439d8475d2df810a8f4f6e66d6866754017"
dependencies = [
"itoa",
"jiff",
@@ -6114,9 +6139,9 @@ dependencies = [
[[package]]
name = "metrique-macro"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f5febfaf14fea234b60e0ad43c728db274033893a38d0fa1f87d9b7056d3d61"
checksum = "786df1fd0abebd0db685f7e9a353c78756d4b370fb98a52376c2015fa55f141f"
dependencies = [
"Inflector",
"darling 0.23.0",
@@ -6143,9 +6168,9 @@ checksum = "2faca4e4480069ff02b1763b3b79f5cec7e8628e24d9dc5b6073f53d2577a4d9"
[[package]]
name = "metrique-writer"
version = "0.1.24"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "124326a2ac4c4f61562fa4d071735a1c463f9ba0317d1564f75dc01313dc12d7"
checksum = "82cdde44d241dab7fc8b7a32e0eb5dae6cd28f8de80b59f9a1e9f2f0b05e485e"
dependencies = [
"ahash",
"crossbeam-queue",
@@ -6164,9 +6189,9 @@ dependencies = [
[[package]]
name = "metrique-writer-core"
version = "0.1.18"
version = "0.1.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55b5bbb6d88bde29f6ed74a574cbe49e51ea0c36cccc7e63eee2040db5675b15"
checksum = "e57379b7ee2272efaeaaa6de062503563e57333b24aadc7f2255b3d602899e8b"
dependencies = [
"derive-where",
"itertools 0.14.0",
@@ -7460,7 +7485,7 @@ version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63d440a804ec8d6fafbb6b84471e013286658d373248927692ab3366686220ca"
dependencies = [
"aes 0.9.1",
"aes 0.9.2",
"aes-gcm",
"cbc 0.2.1",
"der 0.8.1",
@@ -8652,11 +8677,11 @@ dependencies = [
[[package]]
name = "russh"
version = "0.62.3"
version = "0.62.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "059dd24c0fe20721639f7acad7b82cd51ec3dd3254ed8cf7a0b7df6c20eaff1c"
checksum = "b8b67b5a0d8068c89dcbe9d95df986af7a851d1f3c604525274c37468e60464f"
dependencies = [
"aes 0.9.1",
"aes 0.9.2",
"aws-lc-rs",
"bitflags 2.13.1",
"block-padding 0.4.2",
@@ -8838,7 +8863,7 @@ dependencies = [
"aws-config",
"aws-sdk-s3",
"axum",
"base64 0.22.1",
"base64 0.23.0",
"base64-simd",
"bytes",
"chacha20poly1305",
@@ -8879,6 +8904,7 @@ dependencies = [
"quick-xml",
"rand 0.10.2",
"rcgen",
"regex",
"reqwest",
"rmp-serde",
"rsa 0.10.0-rc.18",
@@ -8919,6 +8945,7 @@ dependencies = [
"rustfs-signer",
"rustfs-storage-api",
"rustfs-targets",
"rustfs-test-utils",
"rustfs-tls-runtime",
"rustfs-trusted-proxies",
"rustfs-utils",
@@ -9051,7 +9078,7 @@ dependencies = [
"argon2",
"base64-simd",
"chacha20poly1305",
"jsonwebtoken",
"jsonwebtoken 11.0.0",
"pbkdf2 0.13.0",
"rand 0.10.2",
"rsa 0.10.0-rc.18",
@@ -9089,7 +9116,7 @@ dependencies = [
"aws-smithy-http-client",
"aws-smithy-runtime-api",
"aws-smithy-types",
"base64 0.22.1",
"base64 0.23.0",
"base64-simd",
"byteorder",
"bytes",
@@ -9194,9 +9221,9 @@ dependencies = [
[[package]]
name = "rustfs-erasure-codec"
version = "8.0.0"
version = "8.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8c8af301cb0273d7fb3a1670d74aa9eff015b448f427359658a9ca0f2184b06"
checksum = "bb8ba5edcc507013a0a7bcf7424be94f0ef070b38eab8bcbc38206a8148bc70a"
dependencies = [
"cc",
"cfg_aliases",
@@ -9249,7 +9276,7 @@ name = "rustfs-heal"
version = "1.0.0-beta.11"
dependencies = [
"async-trait",
"base64 0.22.1",
"base64 0.23.0",
"futures",
"http 1.4.2",
"metrics",
@@ -9283,7 +9310,7 @@ dependencies = [
"base64-simd",
"futures",
"http 1.4.2",
"jsonwebtoken",
"jsonwebtoken 11.0.0",
"moka",
"openidconnect",
"pollster",
@@ -9420,7 +9447,7 @@ dependencies = [
"arc-swap",
"argon2",
"async-trait",
"base64 0.22.1",
"base64 0.23.0",
"chacha20poly1305",
"insta",
"jiff",
@@ -9432,6 +9459,7 @@ dependencies = [
"rustfs-utils",
"serde",
"serde_json",
"sha2 0.11.0",
"temp-env",
"tempfile",
"thiserror 2.0.19",
@@ -9448,6 +9476,8 @@ name = "rustfs-lifecycle"
version = "1.0.0-beta.11"
dependencies = [
"async-trait",
"metrics",
"metrics-util",
"proptest",
"rustfs-common",
"rustfs-config",
@@ -9471,6 +9501,7 @@ dependencies = [
"crossbeam-queue",
"futures",
"parking_lot",
"rand 0.10.2",
"rustfs-io-metrics",
"rustfs-utils",
"serde",
@@ -9645,7 +9676,7 @@ dependencies = [
"chrono",
"futures",
"ipnetwork",
"jsonwebtoken",
"jsonwebtoken 11.0.0",
"moka",
"pollster",
"proptest",
@@ -9673,7 +9704,7 @@ dependencies = [
"async-compression",
"async-trait",
"axum",
"base64 0.22.1",
"base64 0.23.0",
"bytes",
"dav-server",
"futures",
@@ -9684,6 +9715,7 @@ dependencies = [
"http-body-util",
"hyper",
"hyper-util",
"ipnetwork",
"libunftp",
"md5",
"percent-encoding",
@@ -9700,7 +9732,9 @@ dependencies = [
"rustfs-policy",
"rustfs-rio",
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-tls-runtime",
"rustfs-trusted-proxies",
"rustfs-utils",
"rustls",
"s3s",
@@ -9738,6 +9772,7 @@ dependencies = [
"rustfs-utils",
"serde",
"serde_json",
"temp-env",
"tokio",
"tonic",
"tonic-prost",
@@ -9770,7 +9805,7 @@ dependencies = [
"aes-gcm",
"arc-swap",
"axum",
"base64 0.22.1",
"base64 0.23.0",
"bytes",
"crc-fast",
"faster-hex",
@@ -9857,6 +9892,7 @@ dependencies = [
"rustfs-test-utils",
"s3s",
"serde_json",
"serial_test",
"tempfile",
"thiserror 2.0.19",
"tokio",
@@ -9887,14 +9923,18 @@ name = "rustfs-scanner"
version = "1.0.0-beta.11"
dependencies = [
"async-trait",
"bytes",
"chrono",
"futures",
"hex-simd",
"hmac 0.13.0",
"http 1.4.2",
"metrics",
"rand 0.10.2",
"rmp-serde",
"rustfs-common",
"rustfs-config",
"rustfs-credentials",
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-filemeta",
@@ -9904,7 +9944,9 @@ dependencies = [
"serde",
"serde_json",
"serial_test",
"sha2 0.11.0",
"temp-env",
"tempfile",
"thiserror 2.0.19",
"time",
"tokio",
@@ -9935,6 +9977,7 @@ dependencies = [
"thiserror 2.0.19",
"time",
"tracing",
"tracing-subscriber",
]
[[package]]
@@ -10229,9 +10272,9 @@ dependencies = [
[[package]]
name = "rustls-pki-types"
version = "1.15.0"
version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046"
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
"web-time",
"zeroize",
@@ -10312,7 +10355,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.14.1"
source = "git+https://github.com/s3s-project/s3s.git?rev=a5471625975f5014f7b28eee7e4d801f1b32f529#a5471625975f5014f7b28eee7e4d801f1b32f529"
source = "git+https://github.com/cxymds/s3s.git?rev=fe3941d91fa1c69956f209a9145995c9f0235bff#fe3941d91fa1c69956f209a9145995c9f0235bff"
dependencies = [
"arc-swap",
"arrayvec",
@@ -10357,6 +10400,7 @@ dependencies = [
"transform-stream",
"url",
"urlencoding",
"xxhash-rust",
"zeroize",
]
@@ -10417,9 +10461,9 @@ dependencies = [
[[package]]
name = "schemars"
version = "1.2.1"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc"
checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a"
dependencies = [
"dyn-clone",
"ref-cast",
@@ -10652,7 +10696,7 @@ dependencies = [
"indexmap 1.9.3",
"indexmap 2.14.0",
"schemars 0.9.0",
"schemars 1.2.1",
"schemars 1.2.2",
"serde_core",
"serde_json",
"serde_with_macros",
@@ -10683,9 +10727,9 @@ dependencies = [
[[package]]
name = "serial_test"
version = "3.5.0"
version = "4.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "699f4197115b8a7e7ff19c9a315a4bd6fffec26cc4626ef45ecaea389e081c6d"
checksum = "a6df5ed973ad8d834e09f824f9e9f449af6b9a3745f78dec7cc752770bd3bf11"
dependencies = [
"futures-executor",
"futures-util",
@@ -10697,13 +10741,13 @@ dependencies = [
[[package]]
name = "serial_test_derive"
version = "3.5.0"
version = "4.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94e153fc76e1c6a068703d6d29c508a0b15c061c4b7e43da59cc097bc342673c"
checksum = "a22144e767da4ddd8416dbf383700542ffd8a5dc493dfecedfe1fe3ad03c98ae"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
@@ -11084,7 +11128,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d801accda99469cde6d73da741422610fdf6508a72d9a69d1b55cb241c720597"
dependencies = [
"aead",
"aes 0.9.1",
"aes 0.9.2",
"aes-gcm",
"chacha20",
"cipher 0.5.2",
@@ -11832,9 +11876,9 @@ dependencies = [
[[package]]
name = "toml_parser"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526"
checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56"
dependencies = [
"winnow",
]
@@ -13029,7 +13073,7 @@ version = "8.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b"
dependencies = [
"aes 0.9.1",
"aes 0.9.2",
"bzip2",
"constant_time_eq",
"crc32fast",
+16 -13
View File
@@ -68,7 +68,7 @@ resolver = "3"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.96.0"
rust-version = "1.97.1"
version = "1.0.0-beta.11"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
@@ -196,13 +196,13 @@ blake2 = "=0.11.0-rc.6"
chacha20poly1305 = { version = "=0.11.0" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "10.4.0" }
jsonwebtoken = { version = "11.0.0" }
openidconnect = { default-features = false, version = "4.0" }
pbkdf2 = "0.13.0"
rsa = { version = "=0.10.0-rc.18" }
rustls = { default-features = false, version = "0.23.42" }
rustls-native-certs = "0.8"
rustls-pki-types = "1.15.0"
rustls-pki-types = "1.15.1"
sha1 = "0.11.0"
sha2 = "0.11.0"
subtle = "2.6"
@@ -211,7 +211,7 @@ zeroize = { version = "1.9.0" }
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.34" }
jiff = { version = "0.2.35" }
time = { version = "0.3.54" }
# Database
@@ -225,13 +225,14 @@ arc-swap = "1.9.2"
astral-tokio-tar = "0.6.4"
atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.10.0" }
aws-config = { version = "1.10.1" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-s3 = { default-features = false, version = "1.139.0" }
aws-sdk-s3 = { default-features = false, version = "1.140.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-smithy-http-client = { default-features = false, version = "1.2.0" }
aws-smithy-runtime-api = { version = "1.13.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-types = { version = "1.6.1" }
base64 = "0.22.1"
base64 = "0.23.0"
base64-simd = "0.8.0"
brotli = "8.0.4"
clap = { version = "4.6.4" }
@@ -243,6 +244,7 @@ crossbeam-channel = "0.5.16"
crossbeam-deque = "0.8.7"
crossbeam-utils = "0.8.22"
datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
#datafusion = { default-features = false, version = "54.1.0" }
derive_builder = "0.20.2"
enumset = "1.1.14"
faster-hex = "0.10.0"
@@ -278,7 +280,7 @@ pretty_assertions = "1.4.1"
rand = { version = "0.10.2" }
ratelimit = "0.10.1"
rayon = "1.12.0"
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
@@ -286,8 +288,8 @@ redis = { version = "1.4.1" }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "a5471625975f5014f7b28eee7e4d801f1b32f529" }
serial_test = "3.5.0"
s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
smallvec = { version = "1.15.2" }
@@ -323,6 +325,7 @@ dial9-tokio-telemetry = "0.3"
opentelemetry = { version = "0.32.0" }
opentelemetry-appender-tracing = { version = "0.32.0" }
opentelemetry-otlp = { version = "0.32.0" }
opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.32.1" }
opentelemetry-semantic-conventions = { version = "0.32.1" }
opentelemetry-stdout = { version = "0.32.0" }
@@ -333,7 +336,7 @@ libunftp = { version = "0.23.0" }
unftp-core = "0.1.0"
suppaftp = { version = "10.0.1" }
rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.62.3" }
russh = { version = "0.62.4" }
russh-sftp = "2.3.0"
# WebDAV
@@ -341,7 +344,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1.52"
hotpath = "0.21.5"
hotpath = "0.22.0"
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.97-trixie
FROM rust:1.97.1-trixie
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
+1 -1
View File
@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.97-trixie AS builder
FROM rust:1.97.1-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
+7 -1
View File
@@ -163,6 +163,7 @@ docker run -d --name rustfs -p 9000:9000 \
-e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
-e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
-e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
-e RUSTFS_OUTBOUND_ALLOW_ORIGINS=http://<host-ip>:3020 \
rustfs/rustfs:latest
```
@@ -171,6 +172,11 @@ Notes:
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
blocked unless their exact `scheme://host:port` origin is listed in
`RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
[Outbound Connection Policy](docs/operations/outbound-connection-policy.md).
**NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
@@ -262,7 +268,7 @@ rustfs --help
2. **Create a Bucket**: Use the console to create a new bucket for your objects.
3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
### OIDC Roles Claim (Microsoft Entra ID)
+1 -1
View File
@@ -214,7 +214,7 @@ rustfs --help
2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
3. **上传对象**: 您可以直接通过控制台上传文件,或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
**注意**: 如果您希望通过 `https` 访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
**注意**: 如果您希望通过 `https` 访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured)。
## 文档
+142 -17
View File
@@ -768,6 +768,7 @@ pub struct Metrics {
last_scan_cycle_replication_checks: AtomicU64,
last_scan_cycle_usage_saves: AtomicU64,
failed_scan_cycles: AtomicU64,
superseded_scan_cycles: AtomicU64,
partial_scan_cycles_unknown: AtomicU64,
partial_scan_cycles_runtime: AtomicU64,
partial_scan_cycles_objects: AtomicU64,
@@ -785,6 +786,9 @@ pub struct Metrics {
scanner_expiry_queue_missed: AtomicU64,
scanner_expiry_queued_total: AtomicU64,
scanner_expiry_missed_total: AtomicU64,
scanner_expiry_blocked_total: AtomicU64,
scanner_expiry_not_enqueued_total: AtomicU64,
scanner_expiry_delete_failed_total: AtomicU64,
scanner_transition_queue_capacity: AtomicU64,
scanner_transition_queued: AtomicU64,
scanner_transition_active: AtomicU64,
@@ -833,10 +837,12 @@ const SCAN_CYCLE_RESULT_UNKNOWN: u8 = 0;
const SCAN_CYCLE_RESULT_SUCCESS: u8 = 1;
const SCAN_CYCLE_RESULT_ERROR: u8 = 2;
const SCAN_CYCLE_RESULT_PARTIAL: u8 = 3;
const SCAN_CYCLE_RESULT_SUPERSEDED: u8 = 4;
const SCAN_CYCLE_RESULT_UNKNOWN_LABEL: &str = "unknown";
const SCAN_CYCLE_RESULT_SUCCESS_LABEL: &str = "success";
const SCAN_CYCLE_RESULT_ERROR_LABEL: &str = "error";
const SCAN_CYCLE_RESULT_PARTIAL_LABEL: &str = "partial";
const SCAN_CYCLE_RESULT_SUPERSEDED_LABEL: &str = "superseded";
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum ScanCyclePartialReason {
@@ -1048,6 +1054,12 @@ pub struct ScannerLifecycleExpirySnapshot {
pub queue_missed: u64,
pub scanner_queued: u64,
pub scanner_missed: u64,
#[serde(default)]
pub scanner_blocked: u64,
#[serde(default)]
pub scanner_not_enqueued: u64,
#[serde(default)]
pub delete_failed: u64,
}
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
@@ -1208,6 +1220,8 @@ pub struct ScannerMetricsReport {
pub last_cycle_usage_saves: u64,
pub failed_cycles: u64,
#[serde(default)]
pub superseded_cycles: u64,
#[serde(default)]
pub partial_cycles_unknown: u64,
#[serde(default)]
pub partial_cycles_runtime: u64,
@@ -1310,6 +1324,7 @@ fn scan_cycle_result_label(result: u8) -> &'static str {
SCAN_CYCLE_RESULT_SUCCESS => SCAN_CYCLE_RESULT_SUCCESS_LABEL,
SCAN_CYCLE_RESULT_ERROR => SCAN_CYCLE_RESULT_ERROR_LABEL,
SCAN_CYCLE_RESULT_PARTIAL => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
SCAN_CYCLE_RESULT_SUPERSEDED => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL,
_ => SCAN_CYCLE_RESULT_UNKNOWN_LABEL,
}
}
@@ -1633,6 +1648,11 @@ pub fn emit_scan_cycle_partial_with_source(
metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_PARTIAL_LABEL).increment(1);
}
pub fn emit_scan_cycle_superseded(duration: Duration) {
global_metrics().record_scan_cycle_superseded(duration);
metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL).increment(1);
}
pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
let result = if success { "success" } else { "error" };
metrics::counter!(
@@ -1726,6 +1746,7 @@ impl Metrics {
last_scan_cycle_replication_checks: AtomicU64::new(0),
last_scan_cycle_usage_saves: AtomicU64::new(0),
failed_scan_cycles: AtomicU64::new(0),
superseded_scan_cycles: AtomicU64::new(0),
partial_scan_cycles_unknown: AtomicU64::new(0),
partial_scan_cycles_runtime: AtomicU64::new(0),
partial_scan_cycles_objects: AtomicU64::new(0),
@@ -1743,6 +1764,9 @@ impl Metrics {
scanner_expiry_queue_missed: AtomicU64::new(0),
scanner_expiry_queued_total: AtomicU64::new(0),
scanner_expiry_missed_total: AtomicU64::new(0),
scanner_expiry_blocked_total: AtomicU64::new(0),
scanner_expiry_not_enqueued_total: AtomicU64::new(0),
scanner_expiry_delete_failed_total: AtomicU64::new(0),
scanner_transition_queue_capacity: AtomicU64::new(0),
scanner_transition_queued: AtomicU64::new(0),
scanner_transition_active: AtomicU64::new(0),
@@ -1973,9 +1997,18 @@ impl Metrics {
self.scanner_expiry_queued_total.fetch_add(count, Ordering::Relaxed);
} else {
self.scanner_expiry_missed_total.fetch_add(count, Ordering::Relaxed);
self.scanner_expiry_not_enqueued_total.fetch_add(count, Ordering::Relaxed);
}
}
pub fn record_scanner_expiry_blocked(&self, count: u64) {
self.scanner_expiry_blocked_total.fetch_add(count, Ordering::Relaxed);
}
pub fn record_scanner_expiry_delete_failed(&self, count: u64) {
self.scanner_expiry_delete_failed_total.fetch_add(count, Ordering::Relaxed);
}
pub fn record_scanner_transition_enqueue_result(&self, count: u64, queued: bool) {
self.record_scanner_ilm_enqueue_result(count, queued);
if queued {
@@ -2349,6 +2382,18 @@ impl Metrics {
.store(duration_millis_saturated(duration), Ordering::Relaxed);
}
pub fn record_scan_cycle_superseded(&self, duration: Duration) {
self.record_scanner_cycle_end_time();
self.superseded_scan_cycles.fetch_add(1, Ordering::Relaxed);
self.last_scan_cycle_result
.store(SCAN_CYCLE_RESULT_SUPERSEDED, Ordering::Relaxed);
self.last_scan_cycle_partial_reason
.store(ScanCyclePartialReason::Unknown as u8, Ordering::Relaxed);
self.last_scan_cycle_partial_source.store(0, Ordering::Relaxed);
self.last_scan_cycle_duration_millis
.store(duration_millis_saturated(duration), Ordering::Relaxed);
}
pub fn record_scan_cycle_partial(&self, duration: Duration, reason: ScanCyclePartialReason) {
self.record_scan_cycle_partial_with_source(duration, reason, None);
}
@@ -2802,6 +2847,7 @@ impl Metrics {
m.last_cycle_replication_repair =
self.scanner_replication_repair_work_counter_snapshots(&self.last_scan_cycle_replication_repair_work);
m.failed_cycles = self.failed_scan_cycles.load(Ordering::Relaxed);
m.superseded_cycles = self.superseded_scan_cycles.load(Ordering::Relaxed);
m.partial_cycles_unknown = self.partial_scan_cycles_unknown.load(Ordering::Relaxed);
m.partial_cycles_runtime = self.partial_scan_cycles_runtime.load(Ordering::Relaxed);
m.partial_cycles_objects = self.partial_scan_cycles_objects.load(Ordering::Relaxed);
@@ -2825,6 +2871,9 @@ impl Metrics {
queue_missed: self.scanner_expiry_queue_missed.load(Ordering::Relaxed),
scanner_queued: self.scanner_expiry_queued_total.load(Ordering::Relaxed),
scanner_missed: self.scanner_expiry_missed_total.load(Ordering::Relaxed),
scanner_blocked: self.scanner_expiry_blocked_total.load(Ordering::Relaxed),
scanner_not_enqueued: self.scanner_expiry_not_enqueued_total.load(Ordering::Relaxed),
delete_failed: self.scanner_expiry_delete_failed_total.load(Ordering::Relaxed),
};
m.lifecycle_transition = ScannerLifecycleTransitionSnapshot {
current_queue_capacity: self.scanner_transition_queue_capacity.load(Ordering::Relaxed),
@@ -2950,19 +2999,15 @@ pub type CloseDiskFn = Arc<dyn Fn() -> Pin<Box<dyn Future<Output = ()> + Send>>
/// Register a new disk in the global path tracker and return two callbacks:
/// one to update the current path and one to deregister the disk when done.
pub fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn, CloseDiskFn) {
pub async fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn, CloseDiskFn) {
let tracker = Arc::new(CurrentPathTracker::new(initial.to_string()));
let disk_name = disk.to_string();
let tracker_clone = Arc::clone(&tracker);
let disk_insert = disk_name.clone();
tokio::spawn(async move {
global_metrics()
.current_paths
.write()
.await
.insert(disk_insert, tracker_clone);
});
global_metrics()
.current_paths
.write()
.await
.insert(disk_name.clone(), Arc::clone(&tracker));
let update_fn: UpdateCurrentPathFn = {
let tracker = Arc::clone(&tracker);
@@ -2990,23 +3035,28 @@ pub fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn,
// CloseDiskGuard
// ---------------------------------------------------------------------------
pub struct CloseDiskGuard(CloseDiskFn);
pub struct CloseDiskGuard(Option<CloseDiskFn>);
impl CloseDiskGuard {
pub fn new(close_disk: CloseDiskFn) -> Self {
Self(close_disk)
Self(Some(close_disk))
}
pub async fn close(&self) {
self.0().await;
pub async fn close(&mut self) {
let Some(close_disk) = self.0.clone() else {
return;
};
close_disk().await;
self.0 = None;
}
}
impl Drop for CloseDiskGuard {
fn drop(&mut self) {
if let Ok(handle) = tokio::runtime::Handle::try_current() {
let close_fn = self.0.clone();
handle.spawn(async move { close_fn().await });
if let Some(close_disk) = self.0.take()
&& let Ok(handle) = tokio::runtime::Handle::try_current()
{
handle.spawn(close_disk());
}
// If there is no runtime we are in a test or shutdown path; skip cleanup.
}
@@ -3016,6 +3066,61 @@ impl Drop for CloseDiskGuard {
mod tests {
use super::*;
#[tokio::test]
async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
let closed_tx = Arc::new(std::sync::Mutex::new(Some(closed_tx)));
let close_disk: CloseDiskFn = {
let closed_tx = Arc::clone(&closed_tx);
Arc::new(move || {
let closed_tx = closed_tx.lock().expect("close callback lock").take();
Box::pin(async move {
if let Some(closed_tx) = closed_tx {
let _ = closed_tx.send(());
}
})
})
};
let guard = CloseDiskGuard::new(close_disk);
drop(guard);
tokio::time::timeout(std::time::Duration::from_secs(1), closed_rx)
.await
.expect("drop cleanup should run")
.expect("drop cleanup should signal");
}
#[tokio::test]
async fn close_disk_guard_runs_explicit_cleanup_once() {
let close_count = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let close_disk: CloseDiskFn = {
let close_count = Arc::clone(&close_count);
Arc::new(move || {
close_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
Box::pin(std::future::ready(()))
})
};
let mut guard = CloseDiskGuard::new(close_disk);
guard.close().await;
drop(guard);
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
assert_eq!(close_count.load(std::sync::atomic::Ordering::Relaxed), 1);
}
#[tokio::test]
async fn current_path_updater_registers_before_return() {
let disk = format!("test-disk-{}", uuid::Uuid::new_v4());
let (_update_path, close_disk) = current_path_updater(&disk, "bucket-a").await;
assert!(global_metrics().current_paths.read().await.contains_key(&disk));
close_disk().await;
assert!(!global_metrics().current_paths.read().await.contains_key(&disk));
}
#[tokio::test]
async fn report_counts_active_scan_paths() {
let metrics = Metrics::new();
@@ -3304,6 +3409,8 @@ mod tests {
});
metrics.record_scanner_expiry_enqueue_result(6, true);
metrics.record_scanner_expiry_enqueue_result(2, false);
metrics.record_scanner_expiry_blocked(4);
metrics.record_scanner_expiry_delete_failed(1);
let report = metrics.report().await;
@@ -3314,6 +3421,9 @@ mod tests {
assert_eq!(report.lifecycle_expiry.queue_missed, 3);
assert_eq!(report.lifecycle_expiry.scanner_queued, 6);
assert_eq!(report.lifecycle_expiry.scanner_missed, 2);
assert_eq!(report.lifecycle_expiry.scanner_blocked, 4);
assert_eq!(report.lifecycle_expiry.scanner_not_enqueued, 2);
assert_eq!(report.lifecycle_expiry.delete_failed, 1);
}
#[tokio::test]
@@ -3850,6 +3960,21 @@ mod tests {
assert_eq!(report.failed_cycles, 1);
}
#[tokio::test]
async fn report_tracks_superseded_cycle_without_failed_increment() {
let metrics = Metrics::new();
metrics.record_scan_cycle_superseded(Duration::from_millis(750));
let report = metrics.report().await;
assert_eq!(report.last_cycle_result, SCAN_CYCLE_RESULT_SUPERSEDED_LABEL);
assert_eq!(report.last_cycle_result_code, u64::from(SCAN_CYCLE_RESULT_SUPERSEDED));
assert_eq!(report.last_cycle_duration_seconds, 0.75);
assert_eq!(report.failed_cycles, 0);
assert_eq!(report.superseded_cycles, 1);
assert_eq!(report.partial_cycles, 0);
}
#[tokio::test]
async fn report_tracks_successful_scan_cycle_without_failed_increment() {
let metrics = Metrics::new();
@@ -28,6 +28,15 @@ pub const MAX_ADMIN_REQUEST_BODY_SIZE: usize = 1024 * 1024; // 1 MB
/// Rationale: ZIP archives with hundreds of IAM entities. 10MB allows ~10,000 small configs.
pub const MAX_IAM_IMPORT_SIZE: usize = 10 * 1024 * 1024; // 10 MB
/// Maximum total size the members of an IAM import ZIP may expand to (100 MB).
/// Used for: bounding decompression of `ImportIam` archive members.
/// Rationale: `MAX_IAM_IMPORT_SIZE` caps the *compressed* upload only. Deflate
/// reaches ratios far above 100:1, so without a separate budget a 10 MB archive
/// can expand without bound. 100 MB keeps a 10x headroom over the compressed cap
/// — ample for legitimate IAM exports, which are small JSON documents — while
/// keeping the worst case bounded.
pub const MAX_IAM_IMPORT_EXPANDED_SIZE: u64 = 100 * 1024 * 1024; // 100 MB
/// Maximum size for bucket metadata import operations (100 MB)
/// Used for: Bucket metadata import containing configurations for many buckets
/// Rationale: Large deployments may have thousands of buckets with various configs.
@@ -54,3 +63,12 @@ pub const MAX_HEAL_REQUEST_SIZE: usize = 1024 * 1024; // 1 MB
/// 10MB provides generous headroom for legitimate responses while preventing
/// memory exhaustion from malicious or misconfigured remote services.
pub const MAX_S3_CLIENT_RESPONSE_SIZE: usize = 10 * 1024 * 1024; // 10 MB
/// Maximum size for OIDC provider response bodies (1 MB)
/// Used for: discovery documents, JWKS documents and token endpoint responses
/// Rationale: a hostile or compromised identity provider must not be able to exhaust
/// memory through an arbitrarily large or endless response body.
/// - Discovery documents: typically < 10KB
/// - JWKS documents: typically < 50KB
/// - Token responses: typically < 10KB
pub const MAX_OIDC_RESPONSE_SIZE: usize = 1024 * 1024; // 1 MB
+90 -7
View File
@@ -97,19 +97,80 @@ pub const ENV_INTERNODE_RPC_MAX_MESSAGE_SIZE: &str = "RUSTFS_INTERNODE_RPC_MAX_M
pub const ENV_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: &str = "RUSTFS_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES";
pub const DEFAULT_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: usize = 8 * 1024 * 1024;
/// Stop dual-writing the JSON compatibility strings on internode metadata RPCs and send only the
/// Request stopping the JSON compatibility strings on internode metadata RPCs and sending only the
/// msgpack `_bin` payloads (grpc-optimization P2-1).
///
/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is a rollout lever, not a
/// wire-format change: it may only be enabled **after** the JSON-fallback counter
/// (`rustfs_system_network_internode_msgpack_json_fallback_total`) has read zero across a release
/// window fleet-wide, confirming every peer decodes `_bin` first. Single-env rollback. See
/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is only a request; RustFS
/// keeps JSON compatibility fields unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED`] is also
/// true after the release-window convergence and rollback gates pass. See
/// `docs/operations/internode-msgpack-json-convergence-runbook.md`.
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
// Compile-time invariant: dual-write by default so the base build is byte-for-byte legacy behavior.
/// Explicit fleet-wide confirmation gate for [`ENV_INTERNODE_RPC_MSGPACK_ONLY`].
///
/// This separate default-off guard prevents a single legacy flag from accidentally emptying JSON
/// fields in a mixed-version fleet where an older peer still reads the JSON field.
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
///
/// Defaults to `false` (fail-open): a request without any v2 auth headers keeps authenticating
/// through the legacy signature, so legacy-only peers survive rolling upgrades with byte-for-byte
/// the pre-gate acceptance behavior. This is a rollout lever, not a wire-format change: it may only
/// be enabled **after** the v1-fallback counter
/// (`rustfs_system_network_internode_signature_v1_fallback_total`) has read zero across a release
/// window fleet-wide, confirming every peer already sends v2 authentication on every internode gRPC
/// request. Single-env rollback. Requests that do carry v2 headers are unaffected by this switch:
/// they are always verified as v2 with no downgrade, strict or not.
pub const ENV_INTERNODE_RPC_SIGNATURE_STRICT: &str = "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT";
pub const DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT: bool = false;
// Compile-time invariant: fail-open by default so legacy-only peers keep authenticating during
// rolling upgrades until the fleet-wide v1-fallback counter reads zero.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT);
/// Require a signature-bound canonical body digest on every mutating internode disk RPC
/// (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete,
/// DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes), rejecting requests
/// that authenticate without one (<https://github.com/rustfs/backlog/issues/1327>).
///
/// Defaults to `false` (fail-open): a mutating request without a body digest keeps authenticating
/// through the method-bound v2 (or legacy) signature, so peers from releases that predate
/// body-digest signing survive rolling upgrades unchanged. Requests that do carry a digest are
/// always verified with no downgrade, strict or not — the digest value is part of the signed v2
/// scope, so an on-path attacker cannot strip it without invalidating the signature. This is a
/// rollout lever gated on the body-digest fallback counter
/// (`rustfs_system_network_internode_body_digest_fallback_total`) reading zero across a release
/// window fleet-wide. Single-env rollback. It is deliberately separate from
/// [`ENV_INTERNODE_RPC_SIGNATURE_STRICT`]: the two enforcement flips converge on different
/// counters and must not gate each other.
pub const ENV_INTERNODE_RPC_BODY_DIGEST_STRICT: &str = "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT";
pub const DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT: bool = false;
// Compile-time invariant: fail-open by default so digestless peers keep authenticating during
// rolling upgrades until the fleet-wide body-digest fallback counter reads zero.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of body-bound v2 signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope, so the steady
/// state holds roughly `mutating RPS x 601s` entries; the default sustains ~1,700 body-bound
/// mutating RPCs per second (about 120 MiB worst case, allocated only under sustained load).
/// Overflow fails closed — legitimate signed traffic is the only thing that can fill the cache
/// (replays are rejected before insertion, and an attacker cannot mint valid nonces without the
/// shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak mutation rate.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
/// Consecutive-failure threshold after which an internode peer is marked offline (grpc-optimization
/// P3 observability).
@@ -273,8 +334,30 @@ mod tests {
#[test]
fn internode_msgpack_only_env_name_is_stable() {
// The dual-write-by-default invariant is asserted at compile time next to the definition.
// The dual-write-by-default invariants are asserted at compile time next to the definitions.
assert_eq!(ENV_INTERNODE_RPC_MSGPACK_ONLY, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY");
assert_eq!(
ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
"RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
);
}
#[test]
fn internode_signature_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT");
}
#[test]
fn internode_body_digest_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT");
}
#[test]
fn internode_replay_cache_capacity_defaults_and_env_name() {
assert_eq!(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY");
assert_eq!(DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, 1_048_576);
}
#[test]
+5 -1
View File
@@ -14,6 +14,7 @@
// OIDC configuration field keys (used in KVS)
pub const OIDC_CONFIG_URL: &str = "config_url";
pub const OIDC_ISSUER: &str = "issuer";
pub const OIDC_CLIENT_ID: &str = "client_id";
pub const OIDC_CLIENT_SECRET: &str = "client_secret";
pub const OIDC_SCOPES: &str = "scopes";
@@ -33,6 +34,7 @@ pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
// Environment variable names for OIDC
pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
pub const ENV_IDENTITY_OPENID_CONFIG_URL: &str = "RUSTFS_IDENTITY_OPENID_CONFIG_URL";
pub const ENV_IDENTITY_OPENID_ISSUER: &str = "RUSTFS_IDENTITY_OPENID_ISSUER";
pub const ENV_IDENTITY_OPENID_CLIENT_ID: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_ID";
pub const ENV_IDENTITY_OPENID_CLIENT_SECRET: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_SECRET";
pub const ENV_IDENTITY_OPENID_SCOPES: &str = "RUSTFS_IDENTITY_OPENID_SCOPES";
@@ -50,9 +52,10 @@ pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USE
pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
/// List of all environment variable keys for an OIDC provider.
pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 18] = &[
ENV_IDENTITY_OPENID_ENABLE,
ENV_IDENTITY_OPENID_CONFIG_URL,
ENV_IDENTITY_OPENID_ISSUER,
ENV_IDENTITY_OPENID_CLIENT_ID,
ENV_IDENTITY_OPENID_CLIENT_SECRET,
ENV_IDENTITY_OPENID_SCOPES,
@@ -74,6 +77,7 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
pub const IDENTITY_OPENID_KEYS: &[&str] = &[
crate::ENABLE_KEY,
OIDC_CONFIG_URL,
OIDC_ISSUER,
OIDC_CLIENT_ID,
OIDC_CLIENT_SECRET,
OIDC_SCOPES,
+1
View File
@@ -57,6 +57,7 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
pub const ENV_WEBDAV_MAX_CONNECTIONS: &str = "RUSTFS_WEBDAV_MAX_CONNECTIONS";
/// Default SFTP server bind address.
pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
+2 -4
View File
@@ -220,12 +220,10 @@ pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_
pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
/// Default set scan concurrency budget.
/// `0` means no additional limit beyond deployment topology.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4;
/// Default disk scan concurrency budget.
/// `0` means no additional limit beyond available disks in the set.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
/// Default object interval for cooperative scanner yields.
pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
+4
View File
@@ -142,6 +142,10 @@ pub const DEFAULT_H2_KEEP_ALIVE_TIMEOUT: u64 = 10;
/// proxy's upstream idle-keepalive, or lower the proxy's keepalive below this
/// value. Environments that expose RustFS directly to untrusted slow clients and
/// want tighter slowloris protection can lower it via the env var below.
///
/// The same budget bounds the TLS handshake on the listener, so an unauthenticated
/// peer cannot park an accept task and its socket indefinitely by opening a
/// connection and then stalling the handshake.
pub const ENV_HTTP1_HEADER_READ_TIMEOUT: &str = "RUSTFS_HTTP1_HEADER_READ_TIMEOUT";
pub const DEFAULT_HTTP1_HEADER_READ_TIMEOUT: u64 = 75;
+283 -14
View File
@@ -32,6 +32,20 @@ use std::{
/// save forever and freeze admin usage stats; callers must bypass the skip instead.
pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 * 60);
/// Cluster-wide usage snapshot written by coordinated scanners.
///
/// `usage_snapshot_complete` is an additive JSON field: older readers ignore
/// it, while current readers treat snapshots from older writers as unknown.
/// Keeping the existing object name preserves rolling-upgrade and rollback
/// compatibility without allowing an ambiguous snapshot to become authoritative.
pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
/// Usage snapshot written by scanner implementations predating distributed
/// leadership fencing. It is read only when neither authoritative snapshot
/// copy exists.
// RUSTFS_COMPAT_TODO(scanner-usage-v2): keep .usage.json readable and removable during rolling upgrades from pre-v2 scanners. Remove after supported direct-upgrade sources all write .usage.v2.json.
pub const LEGACY_DATA_USAGE_OBJECT_NAME: &str = ".usage.json";
/// Returns true when `existing_last_update` is ahead of `now` by more than
/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
@@ -95,7 +109,7 @@ impl AllTierStats {
}
/// Bucket target usage info provides replication statistics
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct BucketTargetUsageInfo {
pub replication_pending_size: u64,
pub replication_failed_size: u64,
@@ -107,7 +121,7 @@ pub struct BucketTargetUsageInfo {
}
/// Bucket usage info provides bucket-level statistics
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct BucketUsageInfo {
pub size: u64,
// Following five fields suffixed with V1 are here for backward compatibility
@@ -133,7 +147,7 @@ pub struct BucketUsageInfo {
}
/// DataUsageInfo represents data usage stats of the underlying storage
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct DataUsageInfo {
/// Total capacity
pub total_capacity: u64,
@@ -145,6 +159,22 @@ pub struct DataUsageInfo {
/// LastUpdate is the timestamp of when the data usage info was last updated
pub last_update: Option<SystemTime>,
/// Monotonic scanner cycle that produced this complete snapshot.
///
/// Older snapshots omit this field and continue to use `last_update` for
/// compatibility. New scanner snapshots use the cycle to fence stale
/// leaders independently of wall-clock skew.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub scanner_cycle: Option<u64>,
/// Persisted scanner leadership epoch that produced this snapshot.
///
/// The epoch is claimed through the cycle-state CAS before scanning. It
/// orders snapshots from different leaders even when their wall clocks or
/// cycle counters coincide.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub scanner_epoch: Option<u64>,
/// Objects total count across all buckets
pub objects_total_count: u64,
/// Versions total count across all buckets
@@ -160,6 +190,12 @@ pub struct DataUsageInfo {
pub buckets_count: u64,
/// Buckets usage info provides following information across all buckets
pub buckets_usage: HashMap<String, BucketUsageInfo>,
/// Whether this snapshot covers the complete bucket namespace.
///
/// Legacy snapshots default to `false`. A complete snapshot contains an
/// explicit entry for every bucket, including confirmed-empty buckets.
#[serde(default)]
pub usage_snapshot_complete: bool,
/// Deprecated kept here for backward compatibility reasons
pub bucket_sizes: HashMap<String, u64>,
/// Per-disk snapshot information when available
@@ -168,7 +204,7 @@ pub struct DataUsageInfo {
}
/// Metadata describing the status of a disk-level data usage snapshot.
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct DiskUsageStatus {
pub disk_id: String,
pub pool_index: Option<usize>,
@@ -268,12 +304,30 @@ impl DataUsageHash {
pub type DataUsageHashMap = HashSet<String>;
/// Size histogram for object size distribution
#[derive(Clone, Debug, Serialize, Deserialize)]
const SIZE_HISTOGRAM_LEN: usize = 11;
#[derive(Clone, Debug, Serialize)]
pub struct SizeHistogram(Vec<u64>);
impl Default for SizeHistogram {
fn default() -> Self {
Self(vec![0; 11]) // DATA_USAGE_BUCKET_LEN = 11
Self(vec![0; SIZE_HISTOGRAM_LEN])
}
}
impl<'de> Deserialize<'de> for SizeHistogram {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let values = Vec::<u64>::deserialize(deserializer)?;
if values.len() != SIZE_HISTOGRAM_LEN {
return Err(serde::de::Error::invalid_length(
values.len(),
&"exactly 11 object-size histogram buckets",
));
}
Ok(Self(values))
}
}
@@ -343,7 +397,7 @@ impl SizeHistogram {
.zip(names.iter())
.filter(|((_, (start, end)), name)| name != &&"BETWEEN_1024B_AND_1_MB" && *start >= 1024 && *end < ONE_MIB)
.map(|((count, _), _)| *count)
.sum();
.fold(0, u64::saturating_add);
let mut res = HashMap::new();
for (count, name) in self.0.iter().zip(names.iter()) {
@@ -364,12 +418,30 @@ impl SizeHistogram {
}
/// Versions histogram for version count distribution
#[derive(Clone, Debug, Serialize, Deserialize)]
const VERSIONS_HISTOGRAM_LEN: usize = 7;
#[derive(Clone, Debug, Serialize)]
pub struct VersionsHistogram(Vec<u64>);
impl Default for VersionsHistogram {
fn default() -> Self {
Self(vec![0; 7]) // DATA_USAGE_VERSION_LEN = 7
Self(vec![0; VERSIONS_HISTOGRAM_LEN])
}
}
impl<'de> Deserialize<'de> for VersionsHistogram {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let values = Vec::<u64>::deserialize(deserializer)?;
if values.len() != VERSIONS_HISTOGRAM_LEN {
return Err(serde::de::Error::invalid_length(
values.len(),
&"exactly 7 object-version histogram buckets",
));
}
Ok(Self(values))
}
}
@@ -535,13 +607,74 @@ impl DataUsageEntry {
}
}
for (i, v) in other.obj_sizes.0.iter().enumerate() {
self.obj_sizes.0[i] += v;
}
self.obj_sizes.merge_from(&other.obj_sizes);
self.obj_versions.merge_from(&other.obj_versions);
}
for (i, v) in other.obj_versions.0.iter().enumerate() {
self.obj_versions.0[i] += v;
pub fn checked_merge(&mut self, other: &DataUsageEntry) -> bool {
let scalar_counts_fit = self.objects.checked_add(other.objects).is_some()
&& self.versions.checked_add(other.versions).is_some()
&& self.delete_markers.checked_add(other.delete_markers).is_some()
&& self.size.checked_add(other.size).is_some()
&& self.failed_objects.checked_add(other.failed_objects).is_some();
let histograms_fit = self.obj_sizes.0.len() == SIZE_HISTOGRAM_LEN
&& other.obj_sizes.0.len() == SIZE_HISTOGRAM_LEN
&& self.obj_versions.0.len() == VERSIONS_HISTOGRAM_LEN
&& other.obj_versions.0.len() == VERSIONS_HISTOGRAM_LEN
&& self
.obj_sizes
.0
.iter()
.zip(other.obj_sizes.0.iter())
.all(|(left, right)| left.checked_add(*right).is_some())
&& self
.obj_versions
.0
.iter()
.zip(other.obj_versions.0.iter())
.all(|(left, right)| left.checked_add(*right).is_some());
let replication_fits = match (&self.replication_stats, &other.replication_stats) {
(_, None) | (None, Some(_)) => true,
(Some(left), Some(right)) => {
left.replica_size.checked_add(right.replica_size).is_some()
&& left.replica_count.checked_add(right.replica_count).is_some()
&& right.targets.iter().all(|(target, right_stats)| {
left.targets.get(target).is_none_or(|left_stats| {
left_stats.pending_size.checked_add(right_stats.pending_size).is_some()
&& left_stats.replicated_size.checked_add(right_stats.replicated_size).is_some()
&& left_stats.failed_size.checked_add(right_stats.failed_size).is_some()
&& left_stats.failed_count.checked_add(right_stats.failed_count).is_some()
&& left_stats.pending_count.checked_add(right_stats.pending_count).is_some()
&& left_stats
.missed_threshold_size
.checked_add(right_stats.missed_threshold_size)
.is_some()
&& left_stats
.after_threshold_size
.checked_add(right_stats.after_threshold_size)
.is_some()
&& left_stats
.missed_threshold_count
.checked_add(right_stats.missed_threshold_count)
.is_some()
&& left_stats
.after_threshold_count
.checked_add(right_stats.after_threshold_count)
.is_some()
&& left_stats
.replicated_count
.checked_add(right_stats.replicated_count)
.is_some()
})
})
}
};
if !scalar_counts_fit || !histograms_fit || !replication_fits {
return false;
}
self.merge(other);
true
}
}
@@ -554,6 +687,12 @@ pub struct DataUsageCacheInfo {
pub skip_healing: bool,
#[serde(default)]
pub failed_objects: HashMap<String, u64>,
/// Whether this per-set cache was produced by a completed scanner pass.
///
/// Older cache writers omit this field and therefore deserialize as
/// incomplete instead of exposing partial set totals as confirmed zeros.
#[serde(default)]
pub snapshot_complete: bool,
}
/// Data usage cache
@@ -873,6 +1012,7 @@ impl DataUsageCache {
objects_total_size: flat.size as u64,
buckets_count: u64::try_from(buckets.len()).unwrap_or(u64::MAX),
buckets_usage,
usage_snapshot_complete: self.info.snapshot_complete,
..Default::default()
}
}
@@ -951,6 +1091,13 @@ impl DataUsageInfo {
Self::default()
}
/// Whether this snapshot authoritatively covers every reported bucket.
pub fn is_complete_bucket_usage_snapshot(&self) -> bool {
self.usage_snapshot_complete
&& self.last_update.is_some()
&& u64::try_from(self.buckets_usage.len()).ok() == Some(self.buckets_count)
}
/// Add object metadata to data usage statistics
pub fn add_object(&mut self, object_path: &str, meta_object: &rustfs_filemeta::MetaObject) {
// This method is kept for backward compatibility
@@ -1315,6 +1462,35 @@ pub struct CompressionTotalInfo {
mod tests {
use super::*;
#[derive(Deserialize)]
struct LegacyUsageReader {
buckets_count: u64,
}
#[test]
fn completeness_marker_is_additive_for_legacy_named_readers() {
let current = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH),
usage_snapshot_complete: true,
..Default::default()
};
let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
let legacy: LegacyUsageReader = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore additive fields");
assert_eq!(legacy.buckets_count, 0);
assert!(current.is_complete_bucket_usage_snapshot());
}
#[test]
fn completeness_marker_requires_a_snapshot_timestamp() {
let untimestamped = DataUsageInfo {
usage_snapshot_complete: true,
..Default::default()
};
assert!(!untimestamped.is_complete_bucket_usage_snapshot());
}
#[test]
fn test_usage_last_update_future_tolerance_boundary() {
let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
@@ -1395,6 +1571,17 @@ mod tests {
assert_eq!(map["BETWEEN_512_KB_AND_1_MB"], 1);
}
#[test]
fn test_size_histogram_compat_rollup_saturates_on_corrupt_counts() {
let mut hist = SizeHistogram::default();
hist.0[1] = u64::MAX;
hist.0[2] = 1;
let map = hist.to_map();
assert_eq!(map["BETWEEN_1024B_AND_1_MB"], u64::MAX);
}
#[test]
fn test_data_usage_cache_merge_adds_missing_child() {
let mut base = DataUsageCache::default();
@@ -1708,4 +1895,86 @@ mod tests {
assert!(cache.find("bucket/large/a").is_some());
assert!(cache.find("bucket/large/b").is_some());
}
#[test]
fn checked_merge_rejects_scalar_and_replication_overflow_without_mutation() {
let mut entry = DataUsageEntry {
objects: usize::MAX,
replication_stats: Some(ReplicationAllStats {
replica_size: 7,
..Default::default()
}),
..Default::default()
};
let other = DataUsageEntry {
objects: 1,
replication_stats: Some(ReplicationAllStats {
replica_size: u64::MAX,
..Default::default()
}),
..Default::default()
};
assert!(!entry.checked_merge(&other));
assert_eq!(entry.objects, usize::MAX);
assert_eq!(entry.replication_stats.as_ref().map(|stats| stats.replica_size), Some(7));
}
#[test]
fn checked_merge_accepts_valid_usage() {
let mut entry = DataUsageEntry {
objects: 2,
size: 20,
..Default::default()
};
let other = DataUsageEntry {
objects: 3,
size: 30,
..Default::default()
};
assert!(entry.checked_merge(&other));
assert_eq!(entry.objects, 5);
assert_eq!(entry.size, 50);
}
#[test]
fn histogram_deserialization_rejects_noncanonical_lengths() {
let invalid_sizes =
rmp_serde::to_vec(&vec![0_u64; SIZE_HISTOGRAM_LEN + 1]).expect("encode invalid object-size histogram fixture");
let invalid_versions =
rmp_serde::to_vec(&vec![0_u64; VERSIONS_HISTOGRAM_LEN - 1]).expect("encode invalid object-version histogram fixture");
assert!(rmp_serde::from_slice::<SizeHistogram>(&invalid_sizes).is_err());
assert!(rmp_serde::from_slice::<VersionsHistogram>(&invalid_versions).is_err());
}
#[test]
fn replication_target_deserialization_preserves_large_historical_maps() {
let mut stats = ReplicationAllStats::default();
for index in 0..=1024 {
stats.targets.insert(format!("target-{index}"), ReplicationStats::default());
}
let encoded = rmp_serde::to_vec_named(&stats).expect("large replication target fixture should encode");
let decoded = rmp_serde::from_slice::<ReplicationAllStats>(&encoded)
.expect("historical replication target maps must remain readable");
assert_eq!(decoded.targets.len(), stats.targets.len());
}
#[test]
fn checked_merge_rejects_noncanonical_histograms_without_mutation() {
let mut entry = DataUsageEntry {
objects: 2,
..Default::default()
};
let other = DataUsageEntry {
objects: 3,
obj_sizes: SizeHistogram(vec![0; SIZE_HISTOGRAM_LEN + 1]),
..Default::default()
};
assert!(!entry.checked_merge(&other));
assert_eq!(entry.objects, 2);
}
}
+4
View File
@@ -30,6 +30,7 @@ sftp = []
[dependencies]
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-credentials.workspace = true
rustfs-ecstore.workspace = true
rustfs-data-usage.workspace = true
rustfs-rio.workspace = true
@@ -49,6 +50,7 @@ rustfs-filemeta.workspace = true
bytes = { workspace = true, features = ["serde"] }
serial_test = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
aws-config = { workspace = true }
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
async-compression = { workspace = true, features = ["tokio", "bzip2", "xz"] }
@@ -69,6 +71,8 @@ base64 = { workspace = true }
rand = { workspace = true, features = ["serde"] }
chrono = { workspace = true, features = ["serde"] }
md5 = { workspace = true }
opentelemetry-proto = { workspace = true }
prost.workspace = true
sha2 = { workspace = true }
astral-tokio-tar = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
+163
View File
@@ -46,6 +46,45 @@ mod tests {
use std::time::{Duration, Instant};
const ADMIN_INFO_PATH: &str = "/rustfs/admin/v3/info";
const ADMIN_MANUAL_TRANSITION_BUCKET: &str = "auth-deny-manual-transition";
const ADMIN_MANUAL_TRANSITION_PATH: &str =
"/rustfs/admin/v3/ilm/transition/run?bucket=auth-deny-manual-transition&maxObjects=1&mode=async";
fn assert_no_raw_manual_transition_markers(body: &str, context: &str) {
assert!(
!body.contains("\"marker\"") && !body.contains("\"versionMarker\"") && !body.contains("\"version_marker\""),
"{context} must not expose raw manual transition resume markers, body: {body}"
);
}
async fn wait_for_terminal_manual_transition_job(
env: &RustFSTestEnvironment,
status_endpoint: &str,
) -> Result<String, Box<dyn Error + Send + Sync>> {
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let (status, body) =
signed_request(&env.url, http::Method::GET, status_endpoint, None, &env.access_key, &env.secret_key).await?;
assert_eq!(
status,
reqwest::StatusCode::OK,
"root credential must query manual transition job status, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition status response");
let value: serde_json::Value = serde_json::from_str(&body)?;
let job_status = value
.get("status")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition job status response must include status")?;
if matches!(job_status, "completed" | "partial" | "cancelled" | "failed" | "unknown") {
return Ok(body);
}
if Instant::now() >= deadline {
return Err(format!("manual transition job did not reach terminal status within 30s; last={body}").into());
}
tokio::time::sleep(Duration::from_millis(250)).await;
}
}
/// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
/// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
@@ -158,6 +197,130 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn non_admin_credential_denied_on_manual_transition_run() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let user_ak = "ilmtransitionlimited";
let user_sk = "ilmtransitionlimitedsecret";
create_limited_user(&env, user_ak, user_sk).await?;
env.create_s3_client()
.create_bucket()
.bucket(ADMIN_MANUAL_TRANSITION_BUCKET)
.send()
.await?;
let (root_status, root_body) = signed_request(
&env.url,
http::Method::POST,
ADMIN_MANUAL_TRANSITION_PATH,
None,
&env.access_key,
&env.secret_key,
)
.await?;
assert_eq!(
root_status,
reqwest::StatusCode::ACCEPTED,
"root credential must reach the manual transition handler, body: {root_body}"
);
assert!(
root_body.contains("\"mode\":\"durable_job\""),
"root response should be the durable manual transition JSON contract, body: {root_body}"
);
assert_no_raw_manual_transition_markers(&root_body, "manual transition run response");
let root_value: serde_json::Value = serde_json::from_str(&root_body)?;
let job_id = root_value
.get("job_id")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include job_id")?;
let status_endpoint = root_value
.get("status_endpoint")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include status_endpoint")?;
let cancel_endpoint = root_value
.get("cancel_endpoint")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include cancel_endpoint")?;
assert_eq!(
cancel_endpoint, status_endpoint,
"manual transition durable jobs currently use the same status/cancel endpoint"
);
assert!(
status_endpoint.ends_with(job_id),
"status endpoint must address the returned job id, job_id={job_id}, status_endpoint={status_endpoint}"
);
let terminal_body = wait_for_terminal_manual_transition_job(&env, status_endpoint).await?;
let terminal: serde_json::Value = serde_json::from_str(&terminal_body)?;
assert_eq!(terminal.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
assert_eq!(
terminal
.get("report")
.and_then(|report| report.get("bucket"))
.and_then(serde_json::Value::as_str),
Some(ADMIN_MANUAL_TRANSITION_BUCKET)
);
let (root_status, root_body) =
signed_request(&env.url, http::Method::DELETE, status_endpoint, None, &env.access_key, &env.secret_key).await?;
assert_eq!(
root_status,
reqwest::StatusCode::OK,
"root credential must cancel/query a terminal manual transition job idempotently, body: {root_body}"
);
assert_no_raw_manual_transition_markers(&root_body, "manual transition cancel response");
let root_cancel: serde_json::Value = serde_json::from_str(&root_body)?;
assert_eq!(root_cancel.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
assert!(
matches!(
root_cancel.get("status").and_then(serde_json::Value::as_str),
Some("completed" | "partial" | "failed" | "unknown")
),
"terminal cancel must not rewrite the job into cancelled state, body: {root_body}"
);
let (status, body) =
signed_request(&env.url, http::Method::POST, ADMIN_MANUAL_TRANSITION_PATH, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition run, body: {body}"
);
assert!(
body.contains("AccessDenied"),
"manual transition rejection must carry the AccessDenied S3 error code, body: {body}"
);
let (status, body) = signed_request(&env.url, http::Method::GET, status_endpoint, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition status, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition status rejection");
assert!(
body.contains("AccessDenied"),
"manual transition status rejection must carry the AccessDenied S3 error code, body: {body}"
);
let (status, body) = signed_request(&env.url, http::Method::DELETE, status_endpoint, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition cancel, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition cancel rejection");
assert!(
body.contains("AccessDenied"),
"manual transition cancel rejection must carry the AccessDenied S3 error code, body: {body}"
);
env.stop_server();
Ok(())
}
/// Rotating the root credentials (restart with new `--access-key` /
/// `--secret-key` on the same data directory) takes effect: the new
/// credential is accepted and the old one is rejected, on both the S3 data
@@ -170,3 +170,81 @@ async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
info!("Test passed: anonymous access allowed with RestrictPublicBuckets=false");
Ok(())
}
/// A policy granting anonymous `s3:ListBucket` also permits ListObjectVersions.
/// That grant must still be subject to RestrictPublicBuckets: the versions listing
/// reaches authorization through a fallback branch, and that branch has to apply the
/// same public-access gate as a direct grant.
#[tokio::test]
#[serial]
async fn ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting test: anonymous ListObjectVersions denied with RestrictPublicBuckets=true...");
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket_name = "anon-test-restrict-versions";
let admin_client = env.create_s3_client();
admin_client.create_bucket().bucket(bucket_name).send().await?;
let policy_json = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAnonymousListBucket",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:ListBucket"],
"Resource": [format!("arn:aws:s3:::{}", bucket_name)]
}
]
})
.to_string();
admin_client
.put_bucket_policy()
.bucket(bucket_name)
.policy(&policy_json)
.send()
.await?;
admin_client
.put_object()
.bucket(bucket_name)
.key("test.txt")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"hello anonymous"))
.send()
.await?;
// Without the public-access block the fallback grant is expected to work.
let versions_url = format!("{}/{}?versions=", env.url, bucket_name);
let resp = local_http_client().get(&versions_url).send().await?;
assert_eq!(
resp.status().as_u16(),
200,
"Anonymous ListObjectVersions should succeed via the s3:ListBucket grant"
);
admin_client
.put_public_access_block()
.bucket(bucket_name)
.public_access_block_configuration(
PublicAccessBlockConfiguration::builder()
.restrict_public_buckets(true)
.build(),
)
.send()
.await?;
let resp = local_http_client().get(&versions_url).send().await?;
assert_eq!(
resp.status().as_u16(),
403,
"Anonymous ListObjectVersions must be denied when RestrictPublicBuckets is true"
);
info!("Test passed: anonymous ListObjectVersions denied with RestrictPublicBuckets=true");
Ok(())
}
+45
View File
@@ -892,6 +892,7 @@ pub struct RustFSTestClusterEnvironment {
pub access_key: String,
pub secret_key: String,
pub extra_env: Vec<(String, String)>,
pub node_extra_env: Vec<Vec<(String, String)>>,
pub topology: ClusterTopology,
}
@@ -990,6 +991,7 @@ impl RustFSTestClusterEnvironment {
access_key: "rustfs-cluster-test-access".to_string(),
secret_key: "rustfs-cluster-test-secret".to_string(),
extra_env,
node_extra_env: vec![Vec::new(); topology.node_count],
topology,
})
}
@@ -1003,6 +1005,22 @@ impl RustFSTestClusterEnvironment {
self.extra_env.push((key.into(), value.into()));
}
/// Add an extra environment variable applied to a single cluster node.
pub fn set_node_env<K, V>(
&mut self,
node_idx: usize,
key: K,
value: V,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
where
K: Into<String>,
V: Into<String>,
{
self.ensure_node_index(node_idx)?;
self.node_extra_env[node_idx].push((key.into(), value.into()));
Ok(())
}
fn ensure_node_index(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
if node_idx >= self.nodes.len() {
return Err(format!("node_idx {node_idx} is invalid").into());
@@ -1089,6 +1107,9 @@ impl RustFSTestClusterEnvironment {
for (key, value) in &self.extra_env {
command.env(key, value);
}
for (key, value) in &self.node_extra_env[i] {
command.env(key, value);
}
let process = command.current_dir(&node.data_dir).spawn()?;
@@ -1130,6 +1151,9 @@ impl RustFSTestClusterEnvironment {
for (key, value) in &self.extra_env {
command.env(key, value);
}
for (key, value) in &self.node_extra_env[node_idx] {
command.env(key, value);
}
let process = command.current_dir(&node.data_dir).spawn()?;
node.process = Some(process);
@@ -1371,6 +1395,7 @@ mod tests {
access_key: DEFAULT_ACCESS_KEY.to_string(),
secret_key: DEFAULT_SECRET_KEY.to_string(),
extra_env: Vec::new(),
node_extra_env: vec![Vec::new(); topology.node_count],
topology,
}
}
@@ -1455,4 +1480,24 @@ mod tests {
assert!(ClusterTopology::single_pool_multidrive(4, 4).validate().is_ok());
assert!(ClusterTopology::single_pool_multidrive(1, 1).validate().is_ok());
}
#[test]
fn cluster_node_env_supports_per_node_overrides() {
let mut env = fake_cluster(ClusterTopology::single_pool(4));
env.set_node_env(2, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY", "true").unwrap();
assert_eq!(
env.node_extra_env[2].as_slice(),
[("RUSTFS_INTERNODE_RPC_MSGPACK_ONLY".to_string(), "true".to_string())]
);
}
#[test]
fn cluster_node_env_rejects_invalid_index() {
let mut env = fake_cluster(ClusterTopology::single_pool(4));
let err = env
.set_node_env(4, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY", "true")
.unwrap_err()
.to_string();
assert!(err.contains("invalid"), "unexpected error: {err}");
}
}
@@ -80,6 +80,25 @@ mod tests {
assert_eq!(head_resp.content_encoding(), Some("zstd"), "HEAD should return Content-Encoding: zstd");
assert_eq!(head_resp.content_type(), Some("text/plain"), "HEAD should return correct Content-Type");
client
.delete_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("DELETE object failed");
client
.delete_bucket()
.bucket(bucket)
.send()
.await
.expect("DELETE bucket failed");
client
.list_buckets()
.send()
.await
.expect("RustFS must remain available after deleting a bucket");
env.stop_server();
}
@@ -12,18 +12,24 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression test for Issue #4996: CopyObject must return the destination object's
//! checksum in `CopyObjectResult` and persist it so a later checksum-mode HEAD/GET
//! returns the same value. Covers both the requested-algorithm case (compute fresh)
//! and the no-algorithm case (preserve the source object's existing checksum).
//! CopyObject checksum compatibility tests. Covers all supported algorithms,
//! source-checksum preservation, explicit override, and fail-closed handling of
//! unsupported algorithms before destination mutation.
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, ChecksumAlgorithm, ChecksumMode, VersioningConfiguration};
use aws_sdk_s3::types::{
BucketVersioningStatus, ChecksumAlgorithm, ChecksumMode, ChecksumType, CompletedMultipartUpload, CompletedPart,
VersioningConfiguration,
};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
use serial_test::serial;
use sha2::{Digest, Sha256};
use tracing::info;
@@ -48,6 +54,401 @@ mod tests {
.expect("Failed to enable versioning");
}
fn create_s3_client_no_auto_checksum(env: &RustFSTestEnvironment) -> aws_sdk_s3::Client {
let credentials = Credentials::new(&env.access_key, &env.secret_key, None, None, "copy-checksum-e2e");
let config = aws_sdk_s3::Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(format!("http://{}", env.address))
.force_path_style(true)
.behavior_version_latest()
.request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
.http_client(SmithyHttpClientBuilder::new().build_http())
.build();
aws_sdk_s3::Client::from_conf(config)
}
fn algorithms() -> [(ChecksumAlgorithm, RioChecksumType); 10] {
[
(ChecksumAlgorithm::Crc32, RioChecksumType::CRC32),
(ChecksumAlgorithm::Crc32C, RioChecksumType::CRC32C),
(ChecksumAlgorithm::Crc64Nvme, RioChecksumType::CRC64_NVME),
(ChecksumAlgorithm::Sha1, RioChecksumType::SHA1),
(ChecksumAlgorithm::Sha256, RioChecksumType::SHA256),
(ChecksumAlgorithm::Md5, RioChecksumType::MD5),
(ChecksumAlgorithm::Sha512, RioChecksumType::SHA512),
(ChecksumAlgorithm::Xxhash3, RioChecksumType::XXHASH3),
(ChecksumAlgorithm::Xxhash64, RioChecksumType::XXHASH64),
(ChecksumAlgorithm::Xxhash128, RioChecksumType::XXHASH128),
]
}
fn result_checksums(result: &aws_sdk_s3::types::CopyObjectResult) -> [Option<&str>; 10] {
[
result.checksum_crc32(),
result.checksum_crc32_c(),
result.checksum_crc64_nvme(),
result.checksum_sha1(),
result.checksum_sha256(),
result.checksum_md5(),
result.checksum_sha512(),
result.checksum_xxhash3(),
result.checksum_xxhash64(),
result.checksum_xxhash128(),
]
}
fn head_checksums(output: &aws_sdk_s3::operation::head_object::HeadObjectOutput) -> [Option<&str>; 10] {
[
output.checksum_crc32(),
output.checksum_crc32_c(),
output.checksum_crc64_nvme(),
output.checksum_sha1(),
output.checksum_sha256(),
output.checksum_md5(),
output.checksum_sha512(),
output.checksum_xxhash3(),
output.checksum_xxhash64(),
output.checksum_xxhash128(),
]
}
#[tokio::test]
#[serial]
async fn test_copy_supports_all_checksum_algorithms() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = create_s3_client_no_auto_checksum(&env);
let src_bucket = "copy-all-checksums-src";
let dst_bucket = "copy-all-checksums-dst";
let src_key = "objects/source.bin";
let content = b"deterministic CopyObject payload for all ten checksum algorithms";
create_versioned_bucket(&client, src_bucket).await;
create_versioned_bucket(&client, dst_bucket).await;
client
.put_object()
.bucket(src_bucket)
.key(src_key)
.body(ByteStream::from_static(content))
.send()
.await
.expect("PUT source failed");
for (index, (sdk_algorithm, rio_algorithm)) in algorithms().into_iter().enumerate() {
let expected = Checksum::new_from_data(rio_algorithm, content)
.expect("supported checksum must be computable")
.encoded;
let dst_key = format!("objects/destination-{index}.bin");
let copy = client
.copy_object()
.bucket(dst_bucket)
.key(&dst_key)
.copy_source(format!("{src_bucket}/{src_key}"))
.checksum_algorithm(sdk_algorithm)
.send()
.await
.expect("CopyObject with supported checksum must succeed");
let result = copy.copy_object_result().expect("CopyObject result");
let checksums = result_checksums(result);
assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: response checksum");
assert_eq!(
checksums.iter().filter(|checksum| checksum.is_some()).count(),
1,
"{rio_algorithm}: only the requested checksum may be returned"
);
let head = client
.head_object()
.bucket(dst_bucket)
.key(&dst_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("HEAD destination failed");
let checksums = head_checksums(&head);
assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: persisted checksum");
assert_eq!(
checksums.iter().filter(|checksum| checksum.is_some()).count(),
1,
"{rio_algorithm}: destination must persist only the requested checksum"
);
let body = client
.get_object()
.bucket(dst_bucket)
.key(&dst_key)
.send()
.await
.expect("GET destination failed")
.body
.collect()
.await
.expect("collect destination body")
.into_bytes();
assert_eq!(body.as_ref(), content, "{rio_algorithm}: full copied body");
}
env.stop_server();
}
#[tokio::test]
#[serial]
async fn test_copy_without_algorithm_preserves_every_supported_source_checksum() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = create_s3_client_no_auto_checksum(&env);
let src_bucket = "copy-preserve-all-src";
let dst_bucket = "copy-preserve-all-dst";
let content = b"source checksum preservation payload for all ten algorithms";
create_versioned_bucket(&client, src_bucket).await;
create_versioned_bucket(&client, dst_bucket).await;
for (index, (_sdk_algorithm, rio_algorithm)) in algorithms().into_iter().enumerate() {
let expected = Checksum::new_from_data(rio_algorithm, content)
.expect("supported checksum must be computable")
.encoded;
let checksum_header = rio_algorithm.key().expect("supported checksum header");
let request_checksum = expected.clone();
let src_key = format!("objects/source-{index}.bin");
let dst_key = format!("objects/destination-{index}.bin");
client
.put_object()
.bucket(src_bucket)
.key(&src_key)
.body(ByteStream::from_static(content))
.customize()
.mutate_request(move |request| {
request.headers_mut().insert(checksum_header, request_checksum.clone());
})
.send()
.await
.expect("PUT checksummed source failed");
let copy = client
.copy_object()
.bucket(dst_bucket)
.key(&dst_key)
.copy_source(format!("{src_bucket}/{src_key}"))
.send()
.await
.expect("CopyObject without algorithm must succeed");
let result = copy.copy_object_result().expect("CopyObject result");
let checksums = result_checksums(result);
assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: preserved response checksum");
assert_eq!(checksums.iter().filter(|checksum| checksum.is_some()).count(), 1);
let head = client
.head_object()
.bucket(dst_bucket)
.key(&dst_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("HEAD destination failed");
let checksums = head_checksums(&head);
assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: preserved stored checksum");
assert_eq!(checksums.iter().filter(|checksum| checksum.is_some()).count(), 1);
}
env.stop_server();
}
#[tokio::test]
#[serial]
async fn test_copy_without_algorithm_preserves_composite_checksum_type() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = create_s3_client_no_auto_checksum(&env);
let bucket = "copy-preserve-composite";
let source_key = "objects/multipart-source.bin";
let destination_key = "objects/copied-multipart.bin";
let content = b"multipart source checksum must remain composite";
create_versioned_bucket(&client, bucket).await;
let created = client
.create_multipart_upload()
.bucket(bucket)
.key(source_key)
.checksum_algorithm(ChecksumAlgorithm::Sha256)
.send()
.await
.expect("CreateMultipartUpload failed");
let upload_id = created.upload_id().expect("multipart upload ID");
let checksum = Checksum::new_from_data(RioChecksumType::SHA256, content)
.expect("SHA256 checksum")
.encoded;
let uploaded = client
.upload_part()
.bucket(bucket)
.key(source_key)
.upload_id(upload_id)
.part_number(1)
.checksum_sha256(&checksum)
.body(ByteStream::from_static(content))
.send()
.await
.expect("UploadPart failed");
let completed_part = CompletedPart::builder()
.part_number(1)
.e_tag(uploaded.e_tag().expect("part ETag"))
.checksum_sha256(uploaded.checksum_sha256().expect("part checksum"))
.build();
client
.complete_multipart_upload()
.bucket(bucket)
.key(source_key)
.upload_id(upload_id)
.multipart_upload(CompletedMultipartUpload::builder().parts(completed_part).build())
.send()
.await
.expect("CompleteMultipartUpload failed");
let source_head = client
.head_object()
.bucket(bucket)
.key(source_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("HEAD multipart source failed");
let source_checksum = source_head.checksum_sha256().expect("multipart source checksum");
assert_eq!(source_head.checksum_type(), Some(&ChecksumType::Composite));
let copied = client
.copy_object()
.bucket(bucket)
.key(destination_key)
.copy_source(format!("{bucket}/{source_key}"))
.send()
.await
.expect("CopyObject without algorithm failed");
let result = copied.copy_object_result().expect("CopyObject result");
assert_eq!(result.checksum_sha256(), Some(source_checksum));
assert_eq!(result.checksum_type(), Some(&ChecksumType::Composite));
let destination_head = client
.head_object()
.bucket(bucket)
.key(destination_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("HEAD copied multipart object failed");
assert_eq!(destination_head.checksum_sha256(), Some(source_checksum));
assert_eq!(destination_head.checksum_type(), Some(&ChecksumType::Composite));
env.stop_server();
}
#[tokio::test]
#[serial]
async fn test_copy_rejects_unknown_algorithm_without_destination_mutation() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = env.create_s3_client();
let bucket = "copy-reject-unknown-checksum";
let src_key = "objects/source.bin";
let dst_key = "objects/destination.bin";
let source = b"source must never replace destination";
let destination = b"pre-existing destination must remain byte-for-byte unchanged";
let expected = Checksum::new_from_data(RioChecksumType::SHA256, destination)
.expect("SHA256 checksum")
.encoded;
create_versioned_bucket(&client, bucket).await;
client
.put_object()
.bucket(bucket)
.key(src_key)
.body(ByteStream::from_static(source))
.send()
.await
.expect("PUT source failed");
let original = client
.put_object()
.bucket(bucket)
.key(dst_key)
.metadata("state", "original")
.checksum_algorithm(ChecksumAlgorithm::Sha256)
.body(ByteStream::from_static(destination))
.send()
.await
.expect("PUT destination failed");
let original_version = original.version_id().expect("versioned PUT must return a version id");
let missing_source_error = client
.copy_object()
.bucket(bucket)
.key(dst_key)
.copy_source(format!("{bucket}/objects/missing-source.bin"))
.checksum_algorithm(ChecksumAlgorithm::from("BLAKE3"))
.send()
.await
.expect_err("checksum validation must precede source lookup");
assert_eq!(
missing_source_error.as_service_error().and_then(|value| value.code()),
Some("InvalidArgument")
);
assert_eq!(missing_source_error.raw_response().map(|response| response.status().as_u16()), Some(400));
let error = client
.copy_object()
.bucket(bucket)
.key(dst_key)
.copy_source(format!("{bucket}/{src_key}"))
.checksum_algorithm(ChecksumAlgorithm::from("BLAKE3"))
.send()
.await
.expect_err("unsupported checksum algorithm must fail");
assert_eq!(error.as_service_error().and_then(|value| value.code()), Some("InvalidArgument"));
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(400));
let head = client
.head_object()
.bucket(bucket)
.key(dst_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("HEAD unchanged destination");
assert_eq!(head.version_id(), Some(original_version));
assert_eq!(
head.metadata().and_then(|metadata| metadata.get("state").map(String::as_str)),
Some("original")
);
assert_eq!(head.checksum_sha256(), Some(expected.as_str()));
let body = client
.get_object()
.bucket(bucket)
.key(dst_key)
.send()
.await
.expect("GET unchanged destination")
.body
.collect()
.await
.expect("collect unchanged destination")
.into_bytes();
assert_eq!(body.as_ref(), destination);
env.stop_server();
}
/// Requested algorithm: a CopyObject asking for SHA256 must compute it over the copied
/// bytes, return it in `CopyObjectResult.ChecksumSHA256`, and persist it so a checksum-mode
/// HEAD on the destination returns the identical value.
@@ -17,14 +17,17 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::MetadataDirective;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::{ByteStream, DateTime, DateTimeFormat};
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, MetadataDirective, StorageClass, VersioningConfiguration,
};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_self_copy_replace_metadata_preserves_readable_object() {
async fn copy_object_standard_metadata_copy_replace_and_clear() {
init_logging();
info!("Issue #2789: self-copy metadata replacement must preserve object data");
@@ -35,6 +38,14 @@ mod tests {
let bucket = "self-copy-metadata-replace-test";
let key = "assets/chunk-2F3R7JUG.js";
let content = b"console.log('metadata replacement should keep object data readable');";
let source_expires = DateTime::from_secs(1_893_456_000);
let source_expires_http_date = source_expires
.fmt(DateTimeFormat::HttpDate)
.expect("Test timestamp should format as an HTTP date");
let replacement_expires = DateTime::from_secs(1_924_992_000);
let replacement_expires_http_date = replacement_expires
.fmt(DateTimeFormat::HttpDate)
.expect("Test timestamp should format as an HTTP date");
client
.create_bucket()
@@ -47,7 +58,14 @@ mod tests {
.put_object()
.bucket(bucket)
.key(key)
.cache_control("max-age=60")
.content_disposition("inline; filename=source.js")
.content_encoding("br")
.content_language("en-US")
.content_type("text/javascript; charset=utf-8")
.expires(source_expires)
.website_redirect_location("/source.html")
.storage_class(StorageClass::ReducedRedundancy)
.metadata("mtime", "1777992333")
.metadata("stale", "must-be-removed")
.body(ByteStream::from_static(content))
@@ -55,13 +73,120 @@ mod tests {
.await
.expect("PUT failed");
let copied_key = "assets/default-copy.js";
client
.copy_object()
.bucket(bucket)
.key(copied_key)
.copy_source(format!("{bucket}/{key}"))
.send()
.await
.expect("default CopyObject failed");
let copied_head = client
.head_object()
.bucket(bucket)
.key(copied_key)
.send()
.await
.expect("HEAD failed after default copy");
assert_eq!(copied_head.cache_control(), Some("max-age=60"));
assert_eq!(copied_head.content_disposition(), Some("inline; filename=source.js"));
assert_eq!(copied_head.content_encoding(), Some("br"));
assert_eq!(copied_head.content_language(), Some("en-US"));
assert_eq!(copied_head.content_type(), Some("text/javascript; charset=utf-8"));
assert_eq!(copied_head.expires_string(), Some(source_expires_http_date.as_str()));
assert_eq!(
copied_head.storage_class(),
None,
"CopyObject without a storage class should write STANDARD"
);
assert_eq!(
copied_head.website_redirect_location(),
Some("/source.html"),
"default CopyObject should preserve source metadata"
);
assert_eq!(
copied_head.metadata().and_then(|metadata| metadata.get("stale")),
Some(&"must-be-removed".to_string())
);
client
.copy_object()
.bucket(bucket)
.key("assets/explicit-copy.js")
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Copy)
.send()
.await
.expect("explicit COPY directive failed");
let explicit_copy_head = client
.head_object()
.bucket(bucket)
.key("assets/explicit-copy.js")
.send()
.await
.expect("HEAD failed after explicit COPY");
assert_eq!(explicit_copy_head.cache_control(), Some("max-age=60"));
assert_eq!(
explicit_copy_head.website_redirect_location(),
None,
"explicit COPY does not inherit website redirect metadata"
);
client
.copy_object()
.bucket(bucket)
.key("assets/explicit-copy-redirect.js")
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Copy)
.website_redirect_location("/explicit-copy.html")
.send()
.await
.expect("explicit COPY with redirect failed");
let explicit_redirect_head = client
.head_object()
.bucket(bucket)
.key("assets/explicit-copy-redirect.js")
.send()
.await
.expect("HEAD failed after explicit COPY with redirect");
assert_eq!(explicit_redirect_head.website_redirect_location(), Some("/explicit-copy.html"));
client
.copy_object()
.bucket(bucket)
.key("assets/explicit-storage-class.js")
.copy_source(format!("{bucket}/{key}"))
.storage_class(StorageClass::ReducedRedundancy)
.send()
.await
.expect("CopyObject with an explicit storage class failed");
let explicit_storage_class_head = client
.head_object()
.bucket(bucket)
.key("assets/explicit-storage-class.js")
.send()
.await
.expect("HEAD failed after explicit storage class copy");
assert_eq!(
explicit_storage_class_head.storage_class().map(StorageClass::as_str),
Some("REDUCED_REDUNDANCY")
);
client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Replace)
.content_type("text/javascript; charset=utf-8")
.cache_control("no-cache")
.content_disposition("attachment; filename=replaced.js")
.content_encoding("gzip")
.content_language("fr-FR")
.content_type("application/javascript")
.expires(replacement_expires)
.website_redirect_location("/replaced.html")
.metadata("mtime", "1777992348")
.send()
.await
@@ -85,6 +210,14 @@ mod tests {
None,
"HEAD should not return metadata omitted by REPLACE"
);
assert_eq!(head_resp.cache_control(), Some("no-cache"));
assert_eq!(head_resp.content_disposition(), Some("attachment; filename=replaced.js"));
assert_eq!(head_resp.content_encoding(), Some("gzip"));
assert_eq!(head_resp.content_language(), Some("fr-FR"));
assert_eq!(head_resp.content_type(), Some("application/javascript"));
assert_eq!(head_resp.expires_string(), Some(replacement_expires_http_date.as_str()));
assert_eq!(head_resp.website_redirect_location(), Some("/replaced.html"));
assert_eq!(head_resp.storage_class(), None, "REPLACE without a storage class should write STANDARD");
let get_resp = client
.get_object()
@@ -123,6 +256,13 @@ mod tests {
None,
"HEAD should not return metadata omitted by empty REPLACE"
);
assert_eq!(empty_head_resp.cache_control(), None);
assert_eq!(empty_head_resp.content_disposition(), None);
assert_eq!(empty_head_resp.content_encoding(), None);
assert_eq!(empty_head_resp.content_language(), None);
assert_eq!(empty_head_resp.content_type(), None);
assert_eq!(empty_head_resp.expires_string(), None);
assert_eq!(empty_head_resp.website_redirect_location(), None);
let empty_get_resp = client
.get_object()
@@ -141,4 +281,333 @@ mod tests {
env.stop_server();
}
#[tokio::test]
#[serial]
async fn copy_object_replace_accepts_each_standard_field_independently() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = env.create_s3_client();
let bucket = "copy-object-metadata-fields";
let source = "source.txt";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("Failed to create bucket");
client
.put_object()
.bucket(bucket)
.key(source)
.cache_control("source-cache")
.content_disposition("inline")
.content_encoding("br")
.content_language("en")
.content_type("text/source")
.expires(DateTime::from_secs(1_893_456_000))
.body(ByteStream::from_static(b"field-by-field"))
.send()
.await
.expect("PUT failed");
let replacement_expires = DateTime::from_secs(1_924_992_000);
let replacement_expires_http_date = replacement_expires
.fmt(DateTimeFormat::HttpDate)
.expect("Test timestamp should format as an HTTP date");
for field in [
"cache-control",
"content-disposition",
"content-encoding",
"content-language",
"content-type",
"expires",
"website-redirect",
] {
let destination = format!("{field}.txt");
let request = client
.copy_object()
.bucket(bucket)
.key(&destination)
.copy_source(format!("{bucket}/{source}"))
.metadata_directive(MetadataDirective::Replace);
let request = match field {
"cache-control" => request.cache_control("field-cache"),
"content-disposition" => request.content_disposition("attachment"),
"content-encoding" => request.content_encoding("gzip"),
"content-language" => request.content_language("de"),
"content-type" => request.content_type("text/field"),
"expires" => request.expires(replacement_expires),
"website-redirect" => request.website_redirect_location("/field.html"),
_ => unreachable!("field table contains only supported entries"),
};
request.send().await.expect("field-specific CopyObject failed");
let head = client
.head_object()
.bucket(bucket)
.key(&destination)
.send()
.await
.expect("HEAD failed");
assert_eq!(head.cache_control(), (field == "cache-control").then_some("field-cache"));
assert_eq!(head.content_disposition(), (field == "content-disposition").then_some("attachment"));
assert_eq!(head.content_encoding(), (field == "content-encoding").then_some("gzip"));
assert_eq!(head.content_language(), (field == "content-language").then_some("de"));
assert_eq!(head.content_type(), (field == "content-type").then_some("text/field"));
assert_eq!(
head.expires_string(),
(field == "expires").then_some(replacement_expires_http_date.as_str())
);
assert_eq!(head.website_redirect_location(), (field == "website-redirect").then_some("/field.html"));
}
client
.copy_object()
.bucket(bucket)
.key("user-metadata-collision.txt")
.copy_source(format!("{bucket}/{source}"))
.metadata_directive(MetadataDirective::Replace)
.metadata("content-type", "user-content-type")
.metadata("content-encoding", "user-content-encoding")
.send()
.await
.expect("CopyObject should preserve user metadata namespaces");
let collision_head = client
.head_object()
.bucket(bucket)
.key("user-metadata-collision.txt")
.send()
.await
.expect("HEAD failed for metadata collision case");
assert_eq!(collision_head.content_type(), None);
assert_eq!(collision_head.content_encoding(), None);
assert_eq!(
collision_head.metadata().and_then(|metadata| metadata.get("content-type")),
Some(&"user-content-type".to_string())
);
assert_eq!(
collision_head
.metadata()
.and_then(|metadata| metadata.get("content-encoding")),
Some(&"user-content-encoding".to_string())
);
env.stop_server();
}
#[tokio::test]
#[serial]
async fn copy_object_replace_handles_versioned_multipart_source() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
let client = env.create_s3_client();
let bucket = "copy-object-metadata-multipart";
let source = "source.bin";
let multipart_body = b"multipart historical source";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("Failed to create bucket");
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("Failed to enable versioning");
let upload = client
.create_multipart_upload()
.bucket(bucket)
.key(source)
.content_type("application/source")
.send()
.await
.expect("Failed to create multipart upload");
let upload_id = upload.upload_id().expect("Multipart upload should return an ID");
let part = client
.upload_part()
.bucket(bucket)
.key(source)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from_static(multipart_body))
.send()
.await
.expect("Failed to upload multipart part");
let completed = client
.complete_multipart_upload()
.bucket(bucket)
.key(source)
.upload_id(upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.e_tag(part.e_tag().expect("Uploaded part should return an ETag"))
.build(),
)
.build(),
)
.send()
.await
.expect("Failed to complete multipart upload");
let historical_version = completed
.version_id()
.expect("Versioned multipart upload should return a version ID")
.to_string();
client
.put_object()
.bucket(bucket)
.key(source)
.body(ByteStream::from_static(b"new current version"))
.send()
.await
.expect("Failed to write current version");
let copy = client
.copy_object()
.bucket(bucket)
.key("restored.bin")
.copy_source(format!("{bucket}/{source}?versionId={historical_version}"))
.metadata_directive(MetadataDirective::Replace)
.content_type("application/replaced")
.send()
.await
.expect("Failed to copy historical multipart version");
assert_eq!(copy.copy_source_version_id(), Some(historical_version.as_str()));
let restored = client
.get_object()
.bucket(bucket)
.key("restored.bin")
.send()
.await
.expect("Failed to read copied multipart source");
assert_eq!(restored.content_type(), Some("application/replaced"));
assert_eq!(
restored
.body
.collect()
.await
.expect("Failed to collect restored body")
.into_bytes()
.as_ref(),
multipart_body
);
env.stop_server();
}
#[tokio::test]
#[serial]
async fn invalid_replacement_metadata_does_not_mutate_destination() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_REJECT_ARCHIVE_CONTENT_ENCODING", "true")])
.await
.expect("Failed to start RustFS");
let client = env.create_s3_client();
let bucket = "copy-object-invalid-metadata";
let key = "destination.zip";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("Failed to create bucket");
client
.put_object()
.bucket(bucket)
.key(key)
.content_type("application/zip")
.metadata("state", "original")
.body(ByteStream::from_static(b"original destination"))
.send()
.await
.expect("Failed to write destination");
let error = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Replace)
.content_type("application/zip")
.content_encoding("gzip")
.send()
.await
.expect_err("Invalid replacement metadata should be rejected");
assert_eq!(error.as_service_error().and_then(|err| err.code()), Some("InvalidArgument"));
let invalid_directive = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.customize()
.mutate_request(|request| {
request.headers_mut().insert("x-amz-metadata-directive", "UNKNOWN");
})
.send()
.await
.expect_err("Unknown metadata directives should be rejected");
assert_eq!(
invalid_directive.as_service_error().and_then(|error| error.code()),
Some("InvalidArgument")
);
let ignored_replacement = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.content_type("application/ignored")
.send()
.await
.expect_err("Replacement fields without REPLACE should be rejected");
assert_eq!(
ignored_replacement.as_service_error().and_then(|error| error.code()),
Some("InvalidRequest")
);
let unchanged = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("Destination should remain readable");
assert_eq!(unchanged.content_type(), Some("application/zip"));
assert_eq!(
unchanged.metadata().and_then(|metadata| metadata.get("state")),
Some(&"original".to_string())
);
assert_eq!(
unchanged
.body
.collect()
.await
.expect("Failed to collect destination body")
.into_bytes()
.as_ref(),
b"original destination"
);
env.stop_server();
}
}
@@ -0,0 +1,468 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! CopyObject tagging directive regression tests.
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, MetadataDirective, TaggingDirective, VersioningConfiguration};
use serial_test::serial;
use std::collections::BTreeMap;
async fn object_tags(client: &Client, bucket: &str, key: &str) -> BTreeMap<String, String> {
client
.get_object_tagging()
.bucket(bucket)
.key(key)
.send()
.await
.expect("GetObjectTagging should succeed")
.tag_set()
.iter()
.map(|tag| (tag.key().to_string(), tag.value().to_string()))
.collect()
}
#[tokio::test]
#[serial]
async fn copy_object_applies_copy_replace_and_empty_tagging_directives() {
init_logging();
let mut env = RustFSTestEnvironment::new()
.await
.expect("test environment should initialize");
env.start_rustfs_server(vec![]).await.expect("RustFS should start");
let client = env.create_s3_client();
let bucket = "copy-object-tagging-directive";
let source = "source.txt";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("bucket creation should succeed");
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("versioning should be enabled");
let first_version = client
.put_object()
.bucket(bucket)
.key(source)
.tagging("project=rustfs&stage=first")
.body(ByteStream::from_static(b"first"))
.send()
.await
.expect("first source version should be written")
.version_id()
.expect("versioned PUT should return a version ID")
.to_string();
client
.put_object()
.bucket(bucket)
.key(source)
.tagging("project=rustfs&stage=current")
.body(ByteStream::from_static(b"current"))
.send()
.await
.expect("current source version should be written");
client
.copy_object()
.bucket(bucket)
.key("default-copy.txt")
.copy_source(format!("{bucket}/{source}"))
.send()
.await
.expect("default CopyObject should preserve current source tags");
assert_eq!(
object_tags(&client, bucket, "default-copy.txt").await,
BTreeMap::from([
("project".to_string(), "rustfs".to_string()),
("stage".to_string(), "current".to_string()),
])
);
client
.copy_object()
.bucket(bucket)
.key("explicit-copy.txt")
.copy_source(format!("{bucket}/{source}?versionId={first_version}"))
.tagging_directive(TaggingDirective::Copy)
.send()
.await
.expect("COPY should preserve the selected historical version's tags");
assert_eq!(
object_tags(&client, bucket, "explicit-copy.txt").await,
BTreeMap::from([
("project".to_string(), "rustfs".to_string()),
("stage".to_string(), "first".to_string()),
])
);
client
.copy_object()
.bucket(bucket)
.key("replace.txt")
.copy_source(format!("{bucket}/{source}"))
.tagging_directive(TaggingDirective::Replace)
.tagging("project=cli&label=copy%20test")
.send()
.await
.expect("REPLACE should atomically apply requested tags");
assert_eq!(
object_tags(&client, bucket, "replace.txt").await,
BTreeMap::from([
("label".to_string(), "copy test".to_string()),
("project".to_string(), "cli".to_string()),
])
);
let replace_head = client
.head_object()
.bucket(bucket)
.key("replace.txt")
.send()
.await
.expect("HEAD should succeed after tag replacement");
assert_eq!(replace_head.tag_count(), Some(2));
client
.copy_object()
.bucket(bucket)
.key("empty-replace.txt")
.copy_source(format!("{bucket}/{source}"))
.tagging_directive(TaggingDirective::Replace)
.send()
.await
.expect("REPLACE without Tagging should clear the destination tag set");
assert!(object_tags(&client, bucket, "empty-replace.txt").await.is_empty());
let empty_head = client
.head_object()
.bucket(bucket)
.key("empty-replace.txt")
.send()
.await
.expect("HEAD should succeed after empty tag replacement");
assert_eq!(empty_head.tag_count(), None);
client
.copy_object()
.bucket(bucket)
.key("metadata-replace-tag-copy.txt")
.copy_source(format!("{bucket}/{source}"))
.metadata_directive(MetadataDirective::Replace)
.metadata("updated", "true")
.send()
.await
.expect("metadata REPLACE must preserve tags under the default COPY directive");
assert_eq!(
object_tags(&client, bucket, "metadata-replace-tag-copy.txt").await,
BTreeMap::from([
("project".to_string(), "rustfs".to_string()),
("stage".to_string(), "current".to_string()),
])
);
client
.copy_object()
.bucket(bucket)
.key("combined-replace.txt")
.copy_source(format!("{bucket}/{source}"))
.metadata_directive(MetadataDirective::Replace)
.metadata("updated", "true")
.tagging_directive(TaggingDirective::Replace)
.tagging("project=combined")
.send()
.await
.expect("metadata and tagging REPLACE directives must be independent");
assert_eq!(
object_tags(&client, bucket, "combined-replace.txt").await,
BTreeMap::from([("project".to_string(), "combined".to_string())])
);
client
.copy_object()
.bucket(bucket)
.key(source)
.copy_source(format!("{bucket}/{source}"))
.tagging_directive(TaggingDirective::Replace)
.tagging("project=self-copy")
.send()
.await
.expect("self-copy with tag replacement should update tags atomically");
assert_eq!(
object_tags(&client, bucket, source).await,
BTreeMap::from([("project".to_string(), "self-copy".to_string())])
);
let self_copy_body = client
.get_object()
.bucket(bucket)
.key(source)
.send()
.await
.expect("self-copy destination should remain readable")
.body
.collect()
.await
.expect("self-copy body should be complete")
.into_bytes();
assert_eq!(self_copy_body.as_ref(), b"current", "tag-only self-copy must preserve the object body");
client
.put_object()
.bucket(bucket)
.key("malformed.txt")
.tagging("state=original")
.body(ByteStream::from_static(b"original destination"))
.send()
.await
.expect("preexisting malformed-test destination should be written");
let malformed = client
.copy_object()
.bucket(bucket)
.key("malformed.txt")
.copy_source(format!("{bucket}/{source}"))
.tagging_directive(TaggingDirective::Replace)
.tagging("project=rustfs%ZZ")
.send()
.await
.expect_err("malformed tags must fail CopyObject");
assert_eq!(malformed.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidTag"));
assert_eq!(
object_tags(&client, bucket, "malformed.txt").await,
BTreeMap::from([("state".to_string(), "original".to_string())])
);
let preserved_body = client
.get_object()
.bucket(bucket)
.key("malformed.txt")
.send()
.await
.expect("malformed tags must not replace an existing destination")
.body
.collect()
.await
.expect("preserved destination body should be readable")
.into_bytes();
assert_eq!(
preserved_body.as_ref(),
b"original destination",
"malformed tags must leave destination data unchanged"
);
let discarded = client
.copy_object()
.bucket(bucket)
.key("discarded.txt")
.copy_source(format!("{bucket}/{source}"))
.tagging("project=must-not-be-discarded")
.send()
.await
.expect_err("Tagging without REPLACE must fail instead of discarding requested tags");
assert_eq!(discarded.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidRequest"));
let invalid_directive = client
.copy_object()
.bucket(bucket)
.key("invalid-directive.txt")
.copy_source(format!("{bucket}/{source}"))
.tagging_directive(TaggingDirective::from("UNKNOWN"))
.send()
.await
.expect_err("an unknown TaggingDirective must fail");
assert_eq!(
invalid_directive.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidArgument")
);
env.stop_server();
}
#[tokio::test]
#[serial]
async fn copy_object_tag_replacement_honors_request_tag_policy_denial() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
let source_bucket = "copy-tags-policy-source";
let destination_bucket = "copy-tags-policy-destination";
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let admin = env.create_s3_client();
admin.create_bucket().bucket(source_bucket).send().await?;
admin.create_bucket().bucket(destination_bucket).send().await?;
admin
.put_object()
.bucket(source_bucket)
.key("source.txt")
.tagging("source=allowed")
.body(ByteStream::from_static(b"source"))
.send()
.await?;
admin
.put_object()
.bucket(source_bucket)
.key("conditioned.txt")
.body(ByteStream::from_static(b"conditioned source"))
.send()
.await?;
let source_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{source_bucket}/source.txt")]
},
{
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{source_bucket}/conditioned.txt")],
"Condition": {
"StringEquals": {
"s3:RequestObjectTag/classification": "public"
}
}
}
]
})
.to_string();
admin
.put_bucket_policy()
.bucket(source_bucket)
.policy(source_policy)
.send()
.await?;
let destination_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:PutObject"],
"Resource": [format!("arn:aws:s3:::{destination_bucket}/*")]
},
{
"Effect": "Deny",
"Principal": "*",
"Action": ["s3:PutObject"],
"Resource": [format!("arn:aws:s3:::{destination_bucket}/*")],
"Condition": {
"StringEquals": {
"s3:RequestObjectTag/classification": "restricted"
}
}
}
]
})
.to_string();
admin
.put_bucket_policy()
.bucket(destination_bucket)
.policy(destination_policy)
.send()
.await?;
let copy_source = format!("/{source_bucket}/source.txt");
let allowed = local_http_client()
.put(format!("{}/{destination_bucket}/allowed.txt", env.url))
.header("x-amz-copy-source", &copy_source)
.header("x-amz-tagging-directive", "REPLACE")
.header("x-amz-tagging", "classification=public")
.send()
.await?;
assert_eq!(
allowed.status(),
reqwest::StatusCode::OK,
"a tag set allowed by the request-tag policy should copy successfully"
);
assert_eq!(
object_tags(&admin, destination_bucket, "allowed.txt").await,
BTreeMap::from([("classification".to_string(), "public".to_string())])
);
let denied = local_http_client()
.put(format!("{}/{destination_bucket}/denied.txt", env.url))
.header("x-amz-copy-source", copy_source)
.header("x-amz-tagging-directive", "REPLACE")
.header("x-amz-tagging", "classification=restricted")
.send()
.await?;
assert_eq!(
denied.status(),
reqwest::StatusCode::FORBIDDEN,
"CopyObject must honor a request-tag policy Deny"
);
let source_condition_bypass = local_http_client()
.put(format!("{}/{destination_bucket}/source-condition.txt", env.url))
.header("x-amz-copy-source", format!("/{source_bucket}/conditioned.txt"))
.header("x-amz-tagging-directive", "REPLACE")
.header("x-amz-tagging", "classification=public")
.send()
.await?;
assert_eq!(
source_condition_bypass.status(),
reqwest::StatusCode::FORBIDDEN,
"destination request tags must not satisfy source GetObject policy conditions"
);
let missing_destination = admin
.head_object()
.bucket(destination_bucket)
.key("denied.txt")
.send()
.await
.expect_err("an access-denied copy must not create a destination object");
assert_eq!(
missing_destination.as_service_error().and_then(ProvideErrorMetadata::code),
Some("NotFound")
);
let missing_bypass_destination = admin
.head_object()
.bucket(destination_bucket)
.key("source-condition.txt")
.send()
.await
.expect_err("a source authorization denial must not create a destination object");
assert_eq!(
missing_bypass_destination
.as_service_error()
.and_then(ProvideErrorMetadata::code),
Some("NotFound")
);
env.stop_server();
Ok(())
}
}
@@ -45,10 +45,10 @@
//! * Parity reconstruction: one data disk is taken offline
//! (`take_disk_offline`) and the SAME object matrix is GET both ways while
//! the EC 2+2 set rebuilds each large object from the surviving shards. The
//! codec-streaming reader gate never inspects drive health, so the codec
//! fast path is exercised end-to-end through reconstruction; the test
//! asserts byte- and header-equality vs the legacy path AND that the codec
//! phase never fell back to a duplex pipe while reconstructing.
//! eager first/single-part setup may keep its conservative whole-request
//! fallback when shard placement makes codec streaming unsafe, so this phase
//! asserts byte- and header-equality vs the legacy path rather than requiring
//! zero duplex fallbacks under degraded drive health.
//! * Missing object: a GET for an absent key is compared across both phases
//! to prove the error semantics (HTTP status + S3 error code) are identical
//! — the codec env must not perturb the NoSuchKey negative path.
@@ -475,15 +475,14 @@ mod tests {
"ranged GET length diverged with codec streaming enabled"
);
// ---- Phase B degraded: the same reconstruction, now on the codec path ----
// Re-run the reconstruction A/B with the codec-streaming gates still
// open. The reader gate decision is independent of drive health (it
// never inspects disk state), so the codec fast path is exercised
// end-to-end while the EC set rebuilds each large object from the
// surviving shards — this is a real codec-vs-legacy reconstruction test,
// not legacy-vs-legacy. Snapshot the duplex count first (the range GET
// above already used the duplex path) so we can measure only the markers
// these degraded codec GETs add.
// ---- Phase B degraded: the same reconstruction, with codec gates open ----
// Re-run the reconstruction A/B with codec-streaming enabled. If eager
// first/single-part setup cannot prove the codec path is safe for the
// surviving shards, the implementation intentionally preserves the
// whole-request legacy fallback; later multipart parts can degrade in
// place. This phase verifies parity-reconstructed bytes and headers,
// while the healthy phase above remains the strict zero-duplex path
// confirmation.
let dup_codec_before_degraded = count_marker(&codec_log, DUPLEX_MARKER);
harness.take_disk_offline(0)?;
let mut codec_degraded: BTreeMap<String, GetView> = BTreeMap::new();
@@ -511,16 +510,11 @@ mod tests {
);
}
// Path confirmation under reconstruction: the codec fast path must have
// served the reconstructed large objects without ever falling back to
// the legacy duplex pipe. Without this, the equivalence above could be
// legacy-vs-legacy and prove nothing about codec reconstruction.
// Keep degraded duplex markers as diagnostic evidence only: eager setup
// may fall back before streaming when shard safety cannot be proven.
sleep(Duration::from_millis(300)).await;
let dup_codec_degraded = count_marker(&codec_log, DUPLEX_MARKER).saturating_sub(dup_codec_before_degraded);
assert_eq!(
dup_codec_degraded, 0,
"codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
);
info!(dup_codec_degraded, "codec phase degraded-read legacy duplex marker count");
info!(
objects = baseline.len(),
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,573 @@
#![cfg(test)]
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Cross-process replay / tamper acceptance for the internode NodeService v2 RPC
//! signature (<https://github.com/rustfs/backlog/issues/1327>).
//!
//! # Why this exists on top of the in-process tests
//!
//! `http_auth.rs` unit-tests the signature algebra by calling the verifier
//! directly. That proves the crypto, but it cannot prove that a *deployed*
//! server actually reaches it: the request has to survive the hybrid HTTP/gRPC
//! router, `check_auth`, tonic's own metadata handling, and finally the
//! per-handler body-digest gate. A handler that forgets its
//! `verify_disk_mutation_digest` call, or a router change that bypasses
//! `check_auth`, is invisible in-process and wide open in production. These
//! tests drive a real `rustfs` child process over a real TCP socket, so every
//! one of those layers is in the path.
//!
//! # Attacker model
//!
//! The adversary is on-path: it observed one legitimately signed request and
//! can resend, retarget, or edit those bytes — including individual headers.
//! It does **not** hold the RPC secret. The test process does hold the secret,
//! but uses it for exactly one purpose: minting the request that stands in for
//! the captured one. Every attack then only *reuses or edits* an already-minted
//! header set; no attack step ever re-signs. If any of these tests could pass
//! by re-signing, it would be testing nothing.
//!
//! # Isolating one variable at a time
//!
//! Each rejection is paired with an acceptance that differs in exactly one
//! respect, because a misconfigured harness (wrong audience, dead server,
//! ambient strict env) would otherwise make every "rejected" assertion pass
//! vacuously. Two pairings carry most of the weight:
//!
//! - Editing the body alone is caught by the *handler* (`PermissionDenied`);
//! editing the body **and** repairing the digest header to match is caught by
//! the *signature* (`Unauthenticated`). The second only fails closed if the
//! digest is genuinely inside the signed scope, so the pair pins both layers.
//! - Replaying a captured nonce is caught by the replay cache; swapping in a
//! fresh nonce is caught by the signature. Again, only the pair proves the
//! nonce is signed rather than merely cached.
//!
//! # Why `MakeVolume` against a non-existent disk
//!
//! Every covered handler checks the digest before touching storage, and
//! `MakeVolume` resolves its disk *after* that check. Aiming at a disk that
//! cannot exist gives three cleanly separable outcomes with zero side effects
//! on the server's real data:
//!
//! - `Err(Unauthenticated)` — rejected by `check_auth` (signature layer).
//! - `Err(PermissionDenied)` — rejected by the handler's body-digest gate.
//! - `Ok(success: false)` — **authentication passed**; the request reached
//! handler logic and only then failed on the bogus disk.
//!
//! # Coverage of the issue's acceptance matrix
//!
//! | Acceptance item | Test |
//! |---|---|
//! | replay a signature onto another method → reject | [`cross_method_signature_transplant_is_rejected`] |
//! | replay same method + body after nonce consumed → reject | [`nonce_replay_of_a_captured_mutation_is_rejected`] |
//! | nonce is signed, not just cached → reject a swapped nonce | [`swapping_in_a_fresh_nonce_is_rejected`] |
//! | tamper one byte of the body → reject | [`tampered_mutation_body_is_rejected`] |
//! | body digest is inside the signed scope → reject a repaired digest | [`rewriting_the_digest_to_match_a_tampered_body_is_rejected`] |
//! | wrong destination node identity → reject | [`signature_minted_for_another_node_is_rejected`] |
//! | mixed version: legacy-only still served, not blocked | [`legacy_only_signature_is_accepted_in_default_posture`] |
//! | strict flip closes the signature downgrade | [`signature_strict_rejects_legacy_only_downgrade`] |
//! | strict flip closes the body-digest downgrade, incl. v1 | [`body_digest_strict_rejects_digestless_mutation`] |
//!
//! Two acceptance items are deliberately left to the in-process tests. A stale
//! timestamp cannot be forged from outside — it is inside the HMAC — so
//! observing it would mean idling out the full freshness window. And the
//! `signature_v1_fallback_total` / `body_digest_fallback_total` counter deltas
//! that gate the strict flips are asserted directly in `http_auth.rs`; the
//! legacy test below proves only the *accepted* half of that behaviour.
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::storage_api::internode_rpc_signature::{
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers, node_service_time_out_client_no_auth,
};
use http::{HeaderMap, Method};
use rustfs_config::{
ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, ENV_INTERNODE_RPC_SIGNATURE_STRICT,
};
use rustfs_protos::canonical_make_volume_request_body;
use rustfs_protos::proto_gen::node_service::{MakeVolumeRequest, MakeVolumeResponse};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::error::Error;
use tonic::{Code, Request, Status};
use uuid::Uuid;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// Shared internode secret handed to both the child server and this process.
///
/// Must not be the default credential: `resolve_rpc_secret` fails closed on
/// defaults (GHSA-r5qv), so a default here would break every request rather
/// than test anything.
const TEST_RPC_SECRET: &str = "rustfs-internode-signature-e2e-secret";
/// A disk path the server cannot possibly have configured, so a request that
/// clears authentication stops harmlessly at `find_disk`.
const ABSENT_DISK: &str = "/nonexistent/rustfs-signature-e2e-disk";
/// Wire names of the two v2 headers these tests edit. They are `pub(crate)` in
/// ecstore, so they are repeated here rather than imported — [`overwrite_header`]
/// asserts the header it replaces was actually present, which turns a rename
/// into a loud failure instead of silently reducing an attack to a no-op.
const CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
const NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
/// gRPC service name carried in the signed scope, i.e. `TONIC_RPC_PREFIX`
/// without its leading `/`.
fn node_service_name() -> &'static str {
TONIC_RPC_PREFIX.trim_start_matches('/')
}
/// Make the RPC secret of this test process match the child server's.
///
/// The secret lands in a process-wide `OnceLock`, so the first writer wins for
/// the whole test binary. Every test here uses the same constant, and the
/// assertion turns a cross-test collision into an explicit failure instead of a
/// confusing wall of signature rejections.
fn align_rpc_secret_with_server() {
let _ = rustfs_credentials::set_global_rpc_secret(TEST_RPC_SECRET.to_string());
let effective = rustfs_credentials::try_get_rpc_token().expect("RPC secret must resolve in the test process");
assert_eq!(
effective, TEST_RPC_SECRET,
"another test in this binary already fixed a different process-wide RPC secret; \
the signature tests cannot mint requests the child server will accept"
);
}
/// Start a `rustfs` child process sharing [`TEST_RPC_SECRET`], with the rollout
/// posture pinned explicitly.
///
/// The child inherits the ambient environment, so the strict gates and the
/// replay-cache capacity are set here rather than assumed: a developer or CI
/// runner exporting `RUSTFS_INTERNODE_RPC_*` would otherwise silently flip the
/// posture and fail these tests for a non-security reason. `extra_env` is
/// applied last so the strict tests can still override.
///
/// Uses the no-cleanup spawn so a `pkill` pattern cannot reap servers belonging
/// to other tests running in the same binary.
async fn start_server(extra_env: &[(&str, &str)]) -> Result<RustFSTestEnvironment, Box<dyn Error + Send + Sync>> {
let mut env = RustFSTestEnvironment::new().await?;
let mut child_env = vec![
("RUSTFS_RPC_SECRET", TEST_RPC_SECRET),
(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "false"),
(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "false"),
(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "1048576"),
];
child_env.extend_from_slice(extra_env);
env.start_rustfs_server_without_cleanup_with_env(&child_env).await?;
Ok(env)
}
/// Stop the child and drop the cached gRPC channel for its address.
///
/// `node_service_time_out_client_no_auth` memoises channels in a process-global
/// map keyed by URL, and ports handed out by `find_available_port` can recur
/// within one test binary. Evicting here keeps a later test from inheriting a
/// channel aimed at this test's dead server.
async fn stop_server(mut env: RustFSTestEnvironment, url: &str) {
env.stop_server();
rustfs_protos::evict_failed_connection(url).await;
}
/// The audience the server binds into the v2 signature: its own node authority.
///
/// A single-node server started with `--address 127.0.0.1:PORT` over filesystem
/// endpoints has no URL peer set, so `init_local_peer` falls back to
/// `host:port` — exactly the address we dialed. The positive controls below
/// fail loudly if that ever stops holding.
fn audience_of(env: &RustFSTestEnvironment) -> String {
env.address.clone()
}
fn hex_sha256(bytes: &[u8]) -> String {
Sha256::digest(bytes).iter().fold(String::new(), |mut acc, byte| {
use std::fmt::Write as _;
let _ = write!(acc, "{byte:02x}");
acc
})
}
fn make_volume_request(volume: &str) -> MakeVolumeRequest {
MakeVolumeRequest {
disk: ABSENT_DISK.to_string(),
volume: volume.to_string(),
}
}
fn canonical_digest(request: &MakeVolumeRequest) -> String {
hex_sha256(&canonical_make_volume_request_body(request).expect("canonical body must encode"))
}
/// Mint a full v2 header set for `(audience, rpc_method, content_sha256)`.
///
/// This is the only place a signature is produced. Tests treat the returned map
/// as an opaque captured artifact.
fn mint_v2_headers(audience: &str, rpc_method: &str, content_sha256: Option<&str>) -> HeaderMap {
gen_tonic_signature_headers(audience, node_service_name(), rpc_method, content_sha256)
.expect("minting a v2 signature must succeed once the RPC secret is aligned")
}
/// Mint the pre-v2 header set: a signature over the fixed
/// `TONIC_RPC_PREFIX|GET|timestamp` constant, with no v2 headers at all. This is
/// both what an un-upgraded peer sends and what an attacker sends to force a
/// downgrade.
fn mint_legacy_only_headers() -> HeaderMap {
gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("minting a legacy signature must succeed")
}
/// Replace one header of a captured set, asserting it was there to begin with.
fn overwrite_header(headers: &mut HeaderMap, name: &'static str, value: &str) {
assert!(
headers.contains_key(name),
"minted headers must carry {name}; the wire contract changed and this attack would edit nothing"
);
headers.insert(name, value.parse().expect("header value must be valid"));
}
/// Send `request` to the server's NodeService with exactly `headers` attached
/// and nothing else — no interceptor adds or rewrites auth metadata, so the
/// bytes on the wire are the ones the test chose.
async fn call_make_volume(url: &str, request: MakeVolumeRequest, headers: HeaderMap) -> Result<MakeVolumeResponse, Status> {
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(request);
rpc_request.metadata_mut().as_mut().extend(headers);
client.make_volume(rpc_request).await.map(|response| response.into_inner())
}
/// Assert a call cleared authentication.
///
/// Receiving *any* `Ok` response is the load-bearing signal: both auth layers
/// reject with a `Status`, so an `Ok` means the request reached handler logic.
/// The failed disk lookup underneath is what keeps it side-effect free.
fn assert_authenticated(result: Result<MakeVolumeResponse, Status>, context: &str) {
match result {
Ok(response) => {
assert!(
!response.success,
"{context}: the absent disk {ABSENT_DISK} must not yield a successful volume creation"
);
assert!(
response.error.is_some(),
"{context}: expected the request to reach disk lookup and fail there, got no error"
);
}
Err(status) => panic!(
"{context}: the request must clear authentication, but was rejected with {:?}: {}",
status.code(),
status.message()
),
}
}
/// Assert a call was rejected, optionally pinning which check spoke.
///
/// `PermissionDenied` responses carry the reason on the wire, so the digest
/// tests pin it and cannot be satisfied by an unrelated digest-gate failure.
/// `Unauthenticated` is deliberately generic on the wire; those tests pin their
/// cause structurally instead, by differing from a passing request in exactly
/// one respect.
fn assert_rejected(result: Result<MakeVolumeResponse, Status>, expected: Code, expected_message: Option<&str>, context: &str) {
match result {
Ok(response) => panic!(
"{context}: the request must be rejected, but the server accepted it and ran the handler \
(success={}, error={:?})",
response.success, response.error
),
Err(status) => {
assert_eq!(
status.code(),
expected,
"{context}: expected {expected:?}, got {:?}: {}",
status.code(),
status.message()
);
if let Some(needle) = expected_message {
assert!(
status.message().contains(needle),
"{context}: expected the rejection to cite {needle:?}, got {:?}",
status.message()
);
}
}
}
}
/// Default posture (both strict gates off): the protections that hold without
/// any operator flip.
///
/// Grouped into one server start because each case is independent and spawning
/// a `rustfs` process per assertion would dominate the runtime.
#[tokio::test]
#[serial]
async fn internode_rpc_signature_default_posture_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
let env = start_server(&[]).await?;
let url = env.url.clone();
let audience = audience_of(&env);
signed_mutations_are_accepted(&url, &audience).await;
unsigned_request_is_rejected(&url).await;
cross_method_signature_transplant_is_rejected(&url, &audience).await;
nonce_replay_of_a_captured_mutation_is_rejected(&url, &audience).await;
swapping_in_a_fresh_nonce_is_rejected(&url, &audience).await;
tampered_mutation_body_is_rejected(&url, &audience).await;
rewriting_the_digest_to_match_a_tampered_body_is_rejected(&url, &audience).await;
signature_minted_for_another_node_is_rejected(&url).await;
legacy_only_signature_is_accepted_in_default_posture(&url).await;
stop_server(env, &url).await;
Ok(())
}
/// Baseline: correctly signed mutations are accepted, both with and without a
/// body digest.
///
/// These anchor every rejection below. The body-bound case proves the audience
/// the server verifies against really is the address we dialed. The digestless
/// case is the control the transplant test needs: without it, a regression that
/// rejected every `UNSIGNED-PAYLOAD` request would make the transplant
/// assertion pass for entirely the wrong reason. It also documents that the
/// default posture still serves digestless mutations.
async fn signed_mutations_are_accepted(url: &str, audience: &str) {
let bound = make_volume_request("signature-e2e-control-bound");
let bound_headers = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&bound)));
assert_authenticated(
call_make_volume(url, bound, bound_headers).await,
"a correctly signed body-bound mutation",
);
let digestless = make_volume_request("signature-e2e-control-digestless");
let digestless_headers = mint_v2_headers(audience, "MakeVolume", None);
assert_authenticated(
call_make_volume(url, digestless, digestless_headers).await,
"a correctly signed digestless mutation in the default posture",
);
}
/// A request with no auth metadata at all must never reach a handler.
async fn unsigned_request_is_rejected(url: &str) {
let result = call_make_volume(url, make_volume_request("signature-e2e-unsigned"), HeaderMap::new()).await;
assert_rejected(result, Code::Unauthenticated, None, "an entirely unsigned mutation");
}
/// GHSA-c667 class: a signature captured from one gRPC method must not be
/// replayable onto another.
///
/// Before method-path binding every NodeService call signed the same constant,
/// so a captured `Ping` — the cheapest, least privileged call on the service —
/// authenticated a `MakeVolume` just as well. The captured `Ping` signature is
/// transplanted verbatim; the server recomputes the scope with
/// `rpc_method = MakeVolume` and the HMAC no longer matches. It differs from the
/// accepted digestless control above only in the method it was minted for.
async fn cross_method_signature_transplant_is_rejected(url: &str, audience: &str) {
let captured_ping = mint_v2_headers(audience, "Ping", None);
let result = call_make_volume(url, make_volume_request("signature-e2e-transplant"), captured_ping).await;
assert_rejected(
result,
Code::Unauthenticated,
None,
"a Ping signature transplanted onto a MakeVolume mutation",
);
}
/// A body-bound mutation must be consumable exactly once.
///
/// The first send establishes that the captured artifact is genuinely valid —
/// without it, the second rejection could just mean the headers were malformed
/// all along. The replay reuses the identical `(signature, timestamp, nonce)`
/// well inside the freshness window, so only the server's replay cache can
/// stop it.
async fn nonce_replay_of_a_captured_mutation_is_rejected(url: &str, audience: &str) {
let request = make_volume_request("signature-e2e-replay");
let captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&request)));
let first = call_make_volume(url, request.clone(), captured.clone()).await;
assert_authenticated(first, "the captured mutation on its first delivery");
let replayed = call_make_volume(url, request, captured).await;
assert_rejected(
replayed,
Code::Unauthenticated,
None,
"the same captured mutation replayed after its nonce was consumed",
);
}
/// The nonce must be *signed*, not merely remembered.
///
/// A replay cache alone would be trivially defeated: swap in a fresh UUID and
/// the cache has never seen it. This request is byte-identical to one the server
/// would accept apart from that one header, so it can only be stopped by the
/// nonce being inside the signed scope.
async fn swapping_in_a_fresh_nonce_is_rejected(url: &str, audience: &str) {
let request = make_volume_request("signature-e2e-nonce-swap");
let mut captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&request)));
overwrite_header(&mut captured, NONCE_HEADER, &Uuid::new_v4().to_string());
let result = call_make_volume(url, request, captured).await;
assert_rejected(
result,
Code::Unauthenticated,
None,
"a captured mutation resent under a freshly minted nonce",
);
}
/// Editing the body of a captured request must invalidate it, in the default
/// posture, with no operator flip required.
///
/// The headers are left byte-identical — including the signed digest of the
/// original body — so `check_auth` still passes. Only the handler, recomputing
/// the canonical body from the fields it actually received, can catch this. It
/// is the test that fails if a handler ever loses its digest gate.
async fn tampered_mutation_body_is_rejected(url: &str, audience: &str) {
let signed = make_volume_request("signature-e2e-tamper-a");
let captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&signed)));
// Exactly one byte of the volume name differs from what the digest covers.
let tampered = make_volume_request("signature-e2e-tamper-b");
let result = call_make_volume(url, tampered, captured).await;
assert_rejected(
result,
Code::PermissionDenied,
Some("RPC content SHA-256 mismatch"),
"a mutation whose body was edited after signing",
);
}
/// The body digest must be *inside the signed scope*, not merely cross-checked
/// by the handler.
///
/// This is the same tampered body as above, except the attacker also repairs the
/// digest header so it matches what it sends — defeating the handler's
/// comparison. The only thing left standing is the signature, which covers the
/// digest header itself. Drop `content_sha256` from `update_signature_v2` and
/// this is the test that goes green when it should not.
async fn rewriting_the_digest_to_match_a_tampered_body_is_rejected(url: &str, audience: &str) {
let signed = make_volume_request("signature-e2e-scope-a");
let mut captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&signed)));
let tampered = make_volume_request("signature-e2e-scope-b");
overwrite_header(&mut captured, CONTENT_SHA256_HEADER, &canonical_digest(&tampered));
let result = call_make_volume(url, tampered, captured).await;
assert_rejected(
result,
Code::Unauthenticated,
None,
"a tampered mutation whose digest header was repaired to match",
);
}
/// A signature is bound to its destination node, so a request captured against
/// one node cannot be aimed at another.
///
/// `127.0.0.1:1` stands in for a different peer; the audience is inside the
/// HMAC, so the server's own authority no longer reproduces it.
async fn signature_minted_for_another_node_is_rejected(url: &str) {
let request = make_volume_request("signature-e2e-wrong-node");
let headers = mint_v2_headers("127.0.0.1:1", "MakeVolume", Some(&canonical_digest(&request)));
let result = call_make_volume(url, request, headers).await;
assert_rejected(result, Code::Unauthenticated, None, "a signature minted for a different node");
}
/// Rolling-upgrade compatibility: a peer that predates v2 must still be served
/// while the strict gates are off.
///
/// This is the case the issue insists must not fail closed during an upgrade.
/// It is also, honestly, the open downgrade window: an attacker can strip the
/// v2 headers and land here too. That window is what
/// [`signature_strict_rejects_legacy_only_downgrade`] closes.
async fn legacy_only_signature_is_accepted_in_default_posture(url: &str) {
let result = call_make_volume(url, make_volume_request("signature-e2e-legacy"), mint_legacy_only_headers()).await;
assert_authenticated(result, "a legacy-only signature in the default posture");
}
/// With `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT` on, the legacy downgrade lane is
/// closed: the exact request accepted in the default posture is now refused.
///
/// The paired v2 positive control rules out "strict simply breaks everything".
#[tokio::test]
#[serial]
async fn signature_strict_rejects_legacy_only_downgrade() -> TestResult {
init_logging();
align_rpc_secret_with_server();
let env = start_server(&[(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "true")]).await?;
let url = env.url.clone();
let audience = audience_of(&env);
let downgraded = call_make_volume(&url, make_volume_request("signature-e2e-strict-legacy"), mint_legacy_only_headers()).await;
assert_rejected(
downgraded,
Code::Unauthenticated,
None,
"a legacy-only signature once signature-strict is enabled",
);
let request = make_volume_request("signature-e2e-strict-v2");
let signed = mint_v2_headers(&audience, "MakeVolume", Some(&canonical_digest(&request)));
assert_authenticated(
call_make_volume(&url, request, signed).await,
"a v2-signed mutation under signature-strict",
);
stop_server(env, &url).await;
Ok(())
}
/// With `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT` on, any mutation that arrives
/// without a body digest is refused — including one that downgraded all the way
/// to the legacy signature.
///
/// This gate converges independently of the signature gate, so it is exercised
/// on its own server with signature-strict left off. Both rejected requests
/// clear `check_auth` on their own terms (one is properly v2-signed, the other
/// takes the still-open legacy lane), which is what pins the rejection to the
/// handler's digest gate; the cited message confirms which check spoke.
#[tokio::test]
#[serial]
async fn body_digest_strict_rejects_digestless_mutation() -> TestResult {
init_logging();
align_rpc_secret_with_server();
let env = start_server(&[(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "true")]).await?;
let url = env.url.clone();
let audience = audience_of(&env);
let digestless = mint_v2_headers(&audience, "MakeVolume", None);
assert_rejected(
call_make_volume(&url, make_volume_request("signature-e2e-digestless"), digestless).await,
Code::PermissionDenied,
Some("RPC mutation requires a body-bound v2 signature"),
"a v2-signed but digestless mutation once body-digest-strict is enabled",
);
assert_rejected(
call_make_volume(&url, make_volume_request("signature-e2e-digestless-legacy"), mint_legacy_only_headers()).await,
Code::PermissionDenied,
Some("RPC mutation requires a body-bound v2 signature"),
"a v1-downgraded mutation once body-digest-strict is enabled",
);
let request = make_volume_request("signature-e2e-digest-bound");
let bound = mint_v2_headers(&audience, "MakeVolume", Some(&canonical_digest(&request)));
assert_authenticated(
call_make_volume(&url, request, bound).await,
"a body-bound mutation under body-digest-strict",
);
stop_server(env, &url).await;
Ok(())
}
+88 -12
View File
@@ -29,6 +29,10 @@ use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::ServerSideEncryption;
use base64::{Engine, engine::general_purpose::STANDARD as BASE64};
use http::header::{CONTENT_TYPE, HOST};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde_json;
use std::process::{Child, Command};
use std::time::Duration;
@@ -67,6 +71,49 @@ pub fn sse_customer_key_md5_base64(key: &str) -> String {
BASE64.encode(md5::compute(key).0)
}
pub async fn kms_admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<&str>,
access_key: &str,
secret_key: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}{path_and_query}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("KMS admin URL missing authority")?.to_string();
let mut builder = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
builder = builder.header(CONTENT_TYPE, "application/json");
}
let content_len = match body {
Some(value) => i64::try_from(value.len())?,
None => 0,
};
let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
let mut request = local_http_client().request(method.clone(), &url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(value) = body {
request = request.body(value.to_owned());
}
let response = request.send().await?;
let status = response.status();
let response_body = response.text().await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed with {status}: {response_body}").into());
}
Ok(response_body)
}
// KMS-specific helper functions
/// Configure KMS backend via admin API
pub async fn configure_kms(
@@ -75,8 +122,19 @@ pub async fn configure_kms(
access_key: &str,
secret_key: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}/rustfs/admin/v3/kms/configure");
awscurl_post(&url, config_json, access_key, secret_key).await?;
let response = kms_admin_request(
base_url,
http::Method::POST,
"/rustfs/admin/v3/kms/configure",
Some(config_json),
access_key,
secret_key,
)
.await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
if response["success"] != true {
return Err(format!("KMS configuration failed: {}", response["message"].as_str().unwrap_or("unknown error")).into());
}
info!("KMS configured successfully");
Ok(())
}
@@ -87,8 +145,19 @@ pub async fn start_kms(
access_key: &str,
secret_key: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}/rustfs/admin/v3/kms/start");
awscurl_post(&url, "{}", access_key, secret_key).await?;
let response = kms_admin_request(
base_url,
http::Method::POST,
"/rustfs/admin/v3/kms/start",
Some("{}"),
access_key,
secret_key,
)
.await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
if response["success"] != true {
return Err(format!("KMS start failed: {}", response["message"].as_str().unwrap_or("unknown error")).into());
}
info!("KMS started successfully");
Ok(())
}
@@ -99,8 +168,8 @@ pub async fn get_kms_status(
access_key: &str,
secret_key: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}/rustfs/admin/v3/kms/status");
let status = awscurl_get(&url, access_key, secret_key).await?;
let status =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/status", None, access_key, secret_key).await?;
info!("KMS status retrieved: {}", status);
Ok(status)
}
@@ -508,7 +577,8 @@ impl VaultTestEnvironment {
},
"mount_path": VAULT_TRANSIT_PATH,
"default_key_id": VAULT_KEY_NAME,
"skip_tls_verify": true
"skip_tls_verify": true,
"allow_insecure_dev_defaults": true
})
.to_string();
@@ -657,14 +727,19 @@ pub async fn test_multipart_upload_with_config(
.build();
info!("🔗 Completing multipart upload");
let complete_output = s3_client
let mut complete_request = s3_client
.complete_multipart_upload()
.bucket(bucket)
.key(&config.object_key)
.upload_id(upload_id)
.multipart_upload(completed_multipart_upload)
.send()
.await?;
.multipart_upload(completed_multipart_upload);
if let EncryptionType::SSEC { .. } = &config.encryption_type {
complete_request = complete_request
.sse_customer_algorithm("AES256")
.sse_customer_key(sse_c_key_b64.as_ref().unwrap())
.sse_customer_key_md5(sse_c_key_md5.as_ref().unwrap());
}
let complete_output = complete_request.send().await?;
debug!("Multipart upload finalized with ETag {:?}", complete_output.e_tag());
@@ -796,7 +871,8 @@ impl LocalKMSTestEnvironment {
"backend_type": "Local",
"key_dir": self.kms_keys_dir,
"file_permissions": 0o600,
"default_key_id": default_key_id
"default_key_id": default_key_id,
"allow_insecure_dev_defaults": true
})
.to_string();
@@ -0,0 +1,399 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Configured-backend validation for the KMS admin and SSE-KMS round-trip
//! contract tracked by rustfs/backlog#1378.
use super::common::{
LocalKMSTestEnvironment, VAULT_KEY_NAME, VaultTestEnvironment, configure_kms, get_kms_status, kms_admin_request, start_kms,
};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, ServerSideEncryption, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use uuid::Uuid;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn assert_configured_status(
base_url: &str,
access_key: &str,
secret_key: &str,
expected_backend: &str,
expected_default_key: &str,
) -> TestResult {
let body = get_kms_status(base_url, access_key, secret_key).await?;
let status: serde_json::Value = serde_json::from_str(&body)?;
assert_eq!(status["backend_type"], expected_backend);
assert_eq!(status["backend_status"], "healthy");
assert_eq!(status["default_key_id"], expected_default_key);
Ok(())
}
async fn create_and_verify_key(
base_url: &str,
access_key: &str,
secret_key: &str,
) -> Result<String, Box<dyn Error + Send + Sync>> {
let create_body = serde_json::json!({
"key_usage": "EncryptDecrypt",
"description": "configured KMS round-trip e2e key",
"tags": {
"test": "backlog-1378"
}
})
.to_string();
let created = kms_admin_request(
base_url,
http::Method::POST,
"/rustfs/admin/v3/kms/keys",
Some(&create_body),
access_key,
secret_key,
)
.await?;
let created: serde_json::Value = serde_json::from_str(&created)?;
assert_eq!(created["success"], true);
let key_id = created["key_id"]
.as_str()
.ok_or("create KMS key response omitted key_id")?
.to_string();
let described = kms_admin_request(
base_url,
http::Method::GET,
&format!("/rustfs/admin/v3/kms/keys/{key_id}"),
None,
access_key,
secret_key,
)
.await?;
let described: serde_json::Value = serde_json::from_str(&described)?;
assert_eq!(described["success"], true);
assert_eq!(described["key_metadata"]["key_id"], key_id);
assert_eq!(described["key_metadata"]["key_state"], "Enabled");
let listed =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
let listed: serde_json::Value = serde_json::from_str(&listed)?;
let keys = listed["keys"].as_array().ok_or("list KMS keys response omitted keys")?;
assert!(keys.iter().any(|key| key["key_id"] == key_id), "created KMS key must appear in list");
Ok(key_id)
}
async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_key: &str, key_id: &str) -> TestResult {
let scheduled = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"pending_window_in_days": 7,
"force_immediate": false
})
.to_string(),
),
access_key,
secret_key,
)
.await?;
let scheduled: serde_json::Value = serde_json::from_str(&scheduled)?;
assert_eq!(scheduled["success"], true);
assert!(scheduled["deletion_date"].is_string());
let cancelled = kms_admin_request(
base_url,
http::Method::POST,
"/rustfs/admin/v3/kms/keys/cancel-deletion",
Some(&serde_json::json!({ "key_id": key_id }).to_string()),
access_key,
secret_key,
)
.await?;
let cancelled: serde_json::Value = serde_json::from_str(&cancelled)?;
assert_eq!(cancelled["success"], true);
assert_eq!(cancelled["key_metadata"]["key_state"], "Enabled");
let removed = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"force_immediate": true
})
.to_string(),
),
access_key,
secret_key,
)
.await?;
let removed: serde_json::Value = serde_json::from_str(&removed)?;
assert_eq!(removed["success"], true);
let listed =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
let listed: serde_json::Value = serde_json::from_str(&listed)?;
assert_eq!(listed["success"], true);
let keys = listed["keys"]
.as_array()
.ok_or("list KMS keys response omitted keys after deletion")?;
if let Some(key) = keys.iter().find(|key| key["key_id"] == key_id) {
assert_eq!(key["status"], "PendingDeletion", "a retained force-deleted key must be pending deletion");
let removed = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"force_immediate": true
})
.to_string(),
),
access_key,
secret_key,
)
.await?;
let removed: serde_json::Value = serde_json::from_str(&removed)?;
assert_eq!(removed["success"], true);
}
let listed =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
let listed: serde_json::Value = serde_json::from_str(&listed)?;
assert_eq!(listed["success"], true);
let keys = listed["keys"]
.as_array()
.ok_or("final list KMS keys response omitted keys after deletion")?;
assert!(
keys.iter().all(|key| key["key_id"] != key_id),
"force-deleted KMS key must no longer appear in list"
);
Ok(())
}
async fn assert_versioned_sse_kms_roundtrip_and_cleanup(
env: &crate::common::RustFSTestEnvironment,
key_id: &str,
bucket_prefix: &str,
) -> TestResult {
let client = env.create_s3_client();
let bucket = format!("{bucket_prefix}-{}", Uuid::new_v4().simple());
let object = format!("configured-kms-probe/{}/object", Uuid::new_v4().simple());
let first_body = b"configured KMS version one";
let second_body = b"configured KMS version two";
client.create_bucket().bucket(&bucket).send().await?;
client
.put_bucket_versioning()
.bucket(&bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(&bucket)
.key(&object)
.server_side_encryption(ServerSideEncryption::AwsKms)
.ssekms_key_id(key_id)
.body(ByteStream::from_static(first_body))
.send()
.await?;
assert_eq!(first.server_side_encryption(), Some(&ServerSideEncryption::AwsKms));
assert_eq!(first.ssekms_key_id(), Some(key_id));
let first_version = first.version_id().ok_or("first SSE-KMS PUT omitted version_id")?.to_string();
let second = client
.put_object()
.bucket(&bucket)
.key(&object)
.server_side_encryption(ServerSideEncryption::AwsKms)
.ssekms_key_id(key_id)
.body(ByteStream::from_static(second_body))
.send()
.await?;
assert_eq!(second.server_side_encryption(), Some(&ServerSideEncryption::AwsKms));
assert_eq!(second.ssekms_key_id(), Some(key_id));
let second_version = second
.version_id()
.ok_or("second SSE-KMS PUT omitted version_id")?
.to_string();
assert_ne!(first_version, second_version);
let storage_root = std::path::Path::new(&env.temp_dir);
super::encryption_metadata_test::assert_storage_encrypted(storage_root, &bucket, &object, first_body);
super::encryption_metadata_test::assert_storage_encrypted(storage_root, &bucket, &object, second_body);
for (version_id, expected) in [
(&first_version, first_body.as_slice()),
(&second_version, second_body.as_slice()),
] {
let response = client
.get_object()
.bucket(&bucket)
.key(&object)
.version_id(version_id)
.send()
.await?;
assert_eq!(response.server_side_encryption(), Some(&ServerSideEncryption::AwsKms));
assert_eq!(response.ssekms_key_id(), Some(key_id));
let actual = response.body.collect().await?.into_bytes();
assert_eq!(actual.len(), expected.len());
assert_eq!(actual.as_ref(), expected);
}
let marker = client.delete_object().bucket(&bucket).key(&object).send().await?;
assert_eq!(marker.delete_marker(), Some(true));
assert!(marker.version_id().is_some(), "versioned delete must create a delete marker");
let before_cleanup = client.list_object_versions().bucket(&bucket).prefix(&object).send().await?;
assert_eq!(
before_cleanup
.versions()
.iter()
.filter(|version| version.key() == Some(object.as_str()))
.count(),
2
);
assert_eq!(
before_cleanup
.delete_markers()
.iter()
.filter(|delete_marker| delete_marker.key() == Some(object.as_str()))
.count(),
1
);
client
.delete_object()
.bucket(&bucket)
.key(&object)
.customize()
.mutate_request(|request| {
request.headers_mut().insert("x-rustfs-force-delete", "true");
})
.send()
.await?;
let after_cleanup = client.list_object_versions().bucket(&bucket).prefix(&object).send().await?;
assert!(
after_cleanup
.versions()
.iter()
.all(|version| version.key() != Some(object.as_str())),
"force cleanup must remove every encrypted object version"
);
assert!(
after_cleanup
.delete_markers()
.iter()
.all(|delete_marker| delete_marker.key() != Some(object.as_str())),
"force cleanup must remove the delete marker"
);
let head_error = match client.head_object().bucket(&bucket).key(&object).send().await {
Ok(_) => return Err("force-cleaned probe object remained readable".into()),
Err(error) => error,
};
let service_error = head_error
.as_service_error()
.ok_or_else(|| format!("force-cleaned HEAD failed with a non-service error: {head_error}"))?;
assert!(
service_error.is_not_found(),
"force-cleaned HEAD returned the wrong service error: {service_error:?}"
);
client.delete_bucket().bucket(&bucket).send().await?;
Ok(())
}
#[tokio::test]
#[serial]
async fn test_configured_local_kms_admin_and_versioned_cleanup() -> TestResult {
let mut env = LocalKMSTestEnvironment::new().await?;
env.base_env.start_rustfs_server(Vec::new()).await?;
let start_error = match start_kms(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await {
Ok(()) => return Err("unconfigured KMS start unexpectedly succeeded".into()),
Err(error) => error,
};
assert!(
start_error.to_string().contains("no configuration provided"),
"unconfigured KMS start returned the wrong business error: {start_error}"
);
let insecure_config = serde_json::json!({
"backend_type": "Local",
"key_dir": env.kms_keys_dir,
"file_permissions": 0o600,
"default_key_id": "rustfs-e2e-test-default-key"
})
.to_string();
let configure_error =
match configure_kms(&env.base_env.url, &insecure_config, &env.base_env.access_key, &env.base_env.secret_key).await {
Ok(()) => return Err("insecure Local KMS configuration unexpectedly succeeded".into()),
Err(error) => error,
};
assert!(
configure_error.to_string().contains("requires a master key"),
"invalid Local KMS configuration returned the wrong business error: {configure_error}"
);
let default_key_id = env.configure_local_kms().await?;
start_kms(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await?;
assert_configured_status(
&env.base_env.url,
&env.base_env.access_key,
&env.base_env.secret_key,
"local",
&default_key_id,
)
.await?;
let key_id = create_and_verify_key(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await?;
assert_versioned_sse_kms_roundtrip_and_cleanup(&env.base_env, &key_id, "kms-local-configured").await?;
assert_key_deletion_lifecycle(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key, &key_id).await?;
Ok(())
}
#[tokio::test]
#[serial]
#[ignore = "requires a Vault binary"]
async fn test_configured_vault_kms_admin_and_versioned_cleanup() -> TestResult {
let mut env = VaultTestEnvironment::new().await?;
env.start_vault().await?;
env.setup_vault_transit().await?;
env.start_rustfs_for_vault().await?;
env.configure_vault_transit_kms().await?;
start_kms(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await?;
assert_configured_status(
&env.base_env.url,
&env.base_env.access_key,
&env.base_env.secret_key,
"vault-transit",
VAULT_KEY_NAME,
)
.await?;
let key_id = create_and_verify_key(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await?;
assert_ne!(key_id, VAULT_KEY_NAME, "key lifecycle test must create a distinct Vault key");
assert_versioned_sse_kms_roundtrip_and_cleanup(&env.base_env, &key_id, "kms-vault-configured").await?;
assert_key_deletion_lifecycle(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key, &key_id).await?;
Ok(())
}
@@ -42,7 +42,7 @@ fn assert_managed_encryption_metadata_hidden(metadata: Option<&HashMap<String, S
}
}
fn assert_storage_encrypted(storage_root: &std::path::Path, bucket: &str, key: &str, plaintext: &[u8]) {
pub(super) fn assert_storage_encrypted(storage_root: &std::path::Path, bucket: &str, key: &str, plaintext: &[u8]) {
let mut stack = VecDeque::from([storage_root.to_path_buf()]);
let mut scanned = 0;
let mut plaintext_path: Option<std::path::PathBuf> = None;
@@ -625,6 +625,9 @@ async fn test_multipart_upload_with_sse_c(
.bucket(bucket)
.key(object_key)
.upload_id(upload_id)
.sse_customer_algorithm("AES256")
.sse_customer_key(&key_b64)
.sse_customer_key_md5(&key_md5)
.multipart_upload(completed_multipart_upload)
.send()
.await?;
+3
View File
@@ -50,3 +50,6 @@ mod encryption_metadata_test;
#[cfg(test)]
mod copy_object_version_restore_sse_test;
#[cfg(test)]
mod configured_roundtrip_test;
@@ -566,14 +566,19 @@ async fn test_multipart_encryption_type(
.set_parts(Some(completed_parts))
.build();
let _complete_output = s3_client
let mut complete_request = s3_client
.complete_multipart_upload()
.bucket(bucket)
.key(object_key)
.upload_id(upload_id)
.multipart_upload(completed_multipart_upload)
.send()
.await?;
.multipart_upload(completed_multipart_upload);
if matches!(encryption_type, EncryptionType::SSEC) {
complete_request = complete_request
.sse_customer_algorithm("AES256")
.sse_customer_key(sse_c_key.as_ref().unwrap())
.sse_customer_key_md5(sse_c_md5.as_ref().unwrap());
}
let _complete_output = complete_request.send().await?;
// Download and verify
let mut get_request = s3_client.get_object().bucket(bucket).key(object_key);
+25
View File
@@ -79,6 +79,12 @@ mod bucket_policy_check_test;
#[cfg(test)]
mod security_boundary_test;
// Cross-process replay/tamper acceptance for the internode NodeService v2 RPC
// signature (backlog#1327): method-path transplant, nonce replay, body tampering
// and the two strict rollout flips, all against a real spawned server.
#[cfg(test)]
mod internode_rpc_signature_e2e_test;
// Opt-in per-client S3 API rate limiting (backlog#1191)
#[cfg(test)]
mod api_rate_limit_test;
@@ -95,6 +101,9 @@ mod admin_auth_test;
#[cfg(test)]
mod existing_object_tag_policy_test;
#[cfg(test)]
mod sts_query_compat_test;
// Regression tests for Issue #2036: anonymous access with PublicAccessBlock
#[cfg(test)]
mod anonymous_access_test;
@@ -167,6 +176,10 @@ mod cluster_concurrency_test;
#[cfg(test)]
mod cluster_multidrive_pool_test;
// backlog#1433: real 4-node EC boundary gate for inline storage and GET paths.
#[cfg(test)]
mod inline_fast_path_cluster_test;
// PutObject / MultipartUpload with checksum (Content-MD5, x-amz-checksum-*)
#[cfg(test)]
mod checksum_upload_test;
@@ -187,12 +200,24 @@ mod heal_erasure_disk_rebuild_test;
#[cfg(test)]
mod copy_object_metadata_test;
#[cfg(test)]
mod copy_object_tagging_test;
#[cfg(test)]
mod copy_object_version_restore_test;
#[cfg(test)]
mod copy_object_checksum_test;
#[cfg(test)]
mod ssec_copy_test;
#[cfg(test)]
mod multipart_storage_class_test;
#[cfg(test)]
mod storage_class_capability_test;
// S3 dummy-compat bucket API tests
#[cfg(test)]
mod bucket_logging_test;
+3 -3
View File
@@ -1267,7 +1267,7 @@ async fn test_anonymous_post_object_accepts_storage_class_exact_policy_match()
let bucket = "anon-post-storage-class";
let object_key = "post-storage-class-object.txt";
let expected_body = b"post-storage-class-body".to_vec();
let storage_class = "STANDARD_IA";
let storage_class = "REDUCED_REDUNDANCY";
let admin_client = env.create_s3_client();
admin_client.create_bucket().bucket(bucket).send().await?;
@@ -5138,7 +5138,7 @@ async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(),
.put_object()
.bucket(bucket)
.key(archive_key)
.storage_class(aws_sdk_s3::types::StorageClass::StandardIa)
.storage_class(aws_sdk_s3::types::StorageClass::ReducedRedundancy)
.body(ByteStream::from(tar_bytes))
.customize()
.mutate_request(move |req| {
@@ -5155,7 +5155,7 @@ async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(),
.send()
.await?;
assert_eq!(head.storage_class().map(|value| value.as_str()), Some("STANDARD_IA"));
assert_eq!(head.storage_class().map(|value| value.as_str()), Some("REDUCED_REDUNDANCY"));
Ok(())
}
@@ -0,0 +1,364 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! CreateMultipartUpload storage-class persistence regression tests.
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart, StorageClass};
const PART_SIZE: usize = 5 * 1024 * 1024;
async fn assert_completed_object(
client: &Client,
bucket: &str,
key: &str,
expected_storage_class: &str,
expected_body: &[u8],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let head = client.head_object().bucket(bucket).key(key).send().await?;
let expected_head_class = (expected_storage_class != "STANDARD").then_some(expected_storage_class);
assert_eq!(
head.storage_class().map(StorageClass::as_str),
expected_head_class,
"HeadObject should use S3's implicit STANDARD representation"
);
let listed = client.list_objects_v2().bucket(bucket).prefix(key).send().await?;
let object = listed
.contents()
.iter()
.find(|object| object.key() == Some(key))
.ok_or("completed multipart object missing from ListObjectsV2")?;
assert_eq!(
object.storage_class().map(|storage_class| storage_class.as_str()),
Some(expected_storage_class),
"ListObjectsV2 should report the completed object's storage class"
);
let body = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(body.as_ref(), expected_body, "completed multipart body should be byte-exact");
Ok(())
}
#[tokio::test]
async fn multipart_upload_preserves_standard_and_rrs_across_retry_and_resume()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "multipart-storage-class-retry";
env.create_test_bucket(bucket).await?;
for storage_class in [StorageClass::Standard, StorageClass::ReducedRedundancy] {
let class_name = storage_class.as_str();
let key = format!("retry-{class_name}.bin");
let create = client
.create_multipart_upload()
.bucket(bucket)
.key(&key)
.storage_class(storage_class.clone())
.content_type("application/octet-stream")
.metadata("content-type", "user-content-type")
.metadata("x-amz-storage-class", "user-storage-class")
.send()
.await?;
let upload_id = create.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
let original_part = vec![b'a'; PART_SIZE];
let first_attempt = client
.upload_part()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from(original_part))
.send()
.await?;
let resumed_client = env.create_s3_client();
let before_retry = resumed_client
.list_parts()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.send()
.await?;
assert_eq!(before_retry.storage_class().map(StorageClass::as_str), Some(class_name));
assert_eq!(before_retry.parts().len(), 1, "resume should find the previously uploaded part");
let retried_part = vec![b'b'; PART_SIZE];
let retry = resumed_client
.upload_part()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from(retried_part.clone()))
.send()
.await?;
assert_ne!(
first_attempt.e_tag(),
retry.e_tag(),
"retrying the same part number with different bytes should replace the part"
);
let tail = format!("-tail-{class_name}").into_bytes();
let second = resumed_client
.upload_part()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.part_number(2)
.body(ByteStream::from(tail.clone()))
.send()
.await?;
let after_retry = resumed_client
.list_parts()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.send()
.await?;
assert_eq!(after_retry.storage_class().map(StorageClass::as_str), Some(class_name));
assert_eq!(after_retry.parts().len(), 2);
assert_eq!(after_retry.parts()[0].e_tag(), retry.e_tag());
resumed_client
.complete_multipart_upload()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.set_e_tag(retry.e_tag().map(str::to_owned))
.build(),
)
.parts(
CompletedPart::builder()
.part_number(2)
.set_e_tag(second.e_tag().map(str::to_owned))
.build(),
)
.build(),
)
.send()
.await?;
let mut expected_body = retried_part;
expected_body.extend_from_slice(&tail);
assert_completed_object(&resumed_client, bucket, &key, class_name, &expected_body).await?;
let metadata_head = resumed_client.head_object().bucket(bucket).key(&key).send().await?;
assert_eq!(metadata_head.content_type(), Some("application/octet-stream"));
assert_eq!(
metadata_head.metadata().and_then(|metadata| metadata.get("content-type")),
Some(&"user-content-type".to_string())
);
assert_eq!(
metadata_head
.metadata()
.and_then(|metadata| metadata.get("x-amz-storage-class")),
Some(&"user-storage-class".to_string())
);
}
env.stop_server();
Ok(())
}
#[tokio::test]
async fn multipart_copy_preserves_standard_and_rrs() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "multipart-storage-class-copy";
let source_key = "source.bin";
let source_body = vec![b'c'; 1024 * 1024];
env.create_test_bucket(bucket).await?;
client
.put_object()
.bucket(bucket)
.key(source_key)
.body(ByteStream::from(source_body.clone()))
.send()
.await?;
for storage_class in [StorageClass::Standard, StorageClass::ReducedRedundancy] {
let class_name = storage_class.as_str();
let key = format!("copy-{class_name}.bin");
let create = client
.create_multipart_upload()
.bucket(bucket)
.key(&key)
.storage_class(storage_class.clone())
.send()
.await?;
let upload_id = create.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
let copied = client
.upload_part_copy()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.part_number(1)
.copy_source(format!("{bucket}/{source_key}"))
.send()
.await?;
let e_tag = copied
.copy_part_result()
.and_then(|result| result.e_tag())
.ok_or("UploadPartCopy returned no ETag")?;
let parts = client
.list_parts()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.send()
.await?;
assert_eq!(parts.storage_class().map(StorageClass::as_str), Some(class_name));
assert_eq!(parts.parts().len(), 1);
client
.complete_multipart_upload()
.bucket(bucket)
.key(&key)
.upload_id(upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(CompletedPart::builder().part_number(1).e_tag(e_tag).build())
.build(),
)
.send()
.await?;
assert_completed_object(&client, bucket, &key, class_name, &source_body).await?;
}
env.stop_server();
Ok(())
}
#[tokio::test]
async fn invalid_and_aborted_uploads_leave_no_session_or_object() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "multipart-storage-class-errors";
let invalid_key = "invalid.bin";
let aborted_key = "aborted.bin";
env.create_test_bucket(bucket).await?;
let invalid = client
.create_multipart_upload()
.bucket(bucket)
.key(invalid_key)
.storage_class(StorageClass::from("INVALID"))
.send()
.await
.expect_err("invalid storage class should be rejected");
assert_eq!(
invalid.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidStorageClass")
);
let after_invalid = client
.list_multipart_uploads()
.bucket(bucket)
.prefix(invalid_key)
.send()
.await?;
assert!(
after_invalid.uploads().is_empty(),
"validation failure must not create a multipart session"
);
let create = client
.create_multipart_upload()
.bucket(bucket)
.key(aborted_key)
.storage_class(StorageClass::ReducedRedundancy)
.send()
.await?;
let upload_id = create.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
client
.upload_part()
.bucket(bucket)
.key(aborted_key)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from_static(b"aborted multipart part"))
.send()
.await?;
let before_abort = client
.list_parts()
.bucket(bucket)
.key(aborted_key)
.upload_id(upload_id)
.send()
.await?;
assert_eq!(before_abort.storage_class().map(StorageClass::as_str), Some("REDUCED_REDUNDANCY"));
client
.abort_multipart_upload()
.bucket(bucket)
.key(aborted_key)
.upload_id(upload_id)
.send()
.await?;
let after_abort = client
.list_parts()
.bucket(bucket)
.key(aborted_key)
.upload_id(upload_id)
.send()
.await
.expect_err("aborted upload should not be resumable");
assert_eq!(after_abort.as_service_error().and_then(ProvideErrorMetadata::code), Some("NoSuchUpload"));
let remaining_uploads = client
.list_multipart_uploads()
.bucket(bucket)
.prefix(aborted_key)
.send()
.await?;
assert!(remaining_uploads.uploads().is_empty(), "abort should remove the multipart session");
let aborted_head = client
.head_object()
.bucket(bucket)
.key(aborted_key)
.send()
.await
.expect_err("aborted upload should not create an object");
assert_eq!(aborted_head.raw_response().map(|response| response.status().as_u16()), Some(404));
env.stop_server();
Ok(())
}
}
@@ -400,6 +400,20 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn prepare_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::PreparePartTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::PreparePartTransactionResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn settle_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::SettlePartTransactionResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn rename_file(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RenameFileRequest>,
+138 -3
View File
@@ -38,7 +38,7 @@ use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
LifecycleRuleFilter, VersioningConfiguration,
LifecycleRuleFilter, NoncurrentVersionExpiration, VersioningConfiguration,
};
use std::time::Duration as StdDuration;
use time::OffsetDateTime;
@@ -98,7 +98,10 @@ async fn put_object_with_backdated_mtime(
/// still succeeds. Any other error is surfaced.
async fn object_is_gone(client: &Client, bucket: &str, key: &str) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
match client.get_object().bucket(bucket).key(key).send().await {
Ok(_) => Ok(false),
Ok(output) => {
output.body.collect().await?;
Ok(false)
}
Err(e) => {
if let Some(service_error) = e.as_service_error() {
if service_error.is_no_such_key() {
@@ -132,6 +135,39 @@ async fn wait_for_object_expired(client: &Client, bucket: &str, key: &str, deadl
}
}
async fn version_is_absent_from_listing(
client: &Client,
bucket: &str,
key: &str,
version_id: &str,
) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
let versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
Ok(!versions.versions().iter().any(|v| v.version_id() == Some(version_id)))
}
async fn wait_for_version_expired(
client: &Client,
bucket: &str,
key: &str,
version_id: &str,
deadline: StdDuration,
) -> TestResult {
let start = std::time::Instant::now();
loop {
if version_is_absent_from_listing(client, bucket, key, version_id).await? {
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"object version {bucket}/{key}?versionId={version_id} was not expired by the lifecycle scanner within {}s",
deadline.as_secs()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// Build a prefix-scoped `Days`-based expiration rule.
fn expiration_rule(id: &str, prefix: &str, days: i32) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
let rule = LifecycleRule::builder()
@@ -143,6 +179,20 @@ fn expiration_rule(id: &str, prefix: &str, days: i32) -> Result<LifecycleRule, B
Ok(rule)
}
fn noncurrent_expiration_rule(
id: &str,
prefix: &str,
days: i32,
) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
let rule = LifecycleRule::builder()
.id(id)
.filter(LifecycleRuleFilter::builder().prefix(prefix).build())
.noncurrent_version_expiration(NoncurrentVersionExpiration::builder().noncurrent_days(days).build())
.status(ExpirationStatus::Enabled)
.build()?;
Ok(rule)
}
async fn put_expiration_config(client: &Client, bucket: &str, rule: LifecycleRule) -> TestResult {
let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
client
@@ -277,9 +327,94 @@ async fn test_lifecycle_versioned_current_version_expiry_creates_delete_marker()
Ok(())
}
/// `NoncurrentVersionExpiration NoncurrentDays=1` on a versioned bucket,
/// accelerated with `RUSTFS_ILM_DEBUG_DAY_SECS`. Proves the scanner purges the
/// noncurrent data version from `ListObjectVersions` while preserving the
/// latest version as the normal readable object and without creating a delete
/// marker.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_lifecycle_noncurrent_version_expiry_removes_only_old_version() -> TestResult {
let mut env = RustFSTestEnvironment::new().await?;
let mut extra_env = fast_lifecycle_env();
extra_env.push(("RUSTFS_ILM_DEBUG_DAY_SECS", "2"));
env.start_rustfs_server_with_env(vec![], &extra_env).await?;
let client = env.create_s3_client();
let bucket = "ilm3-noncurrent";
client.create_bucket().bucket(bucket).send().await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let key = "versioned/noncurrent.txt";
let first_put = client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"old payload"))
.send()
.await?;
let old_version_id = first_put
.version_id()
.map(str::to_string)
.expect("first versioned PUT returns a version id");
let second_put = client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"latest payload"))
.send()
.await?;
let latest_version_id = second_put
.version_id()
.map(str::to_string)
.expect("second versioned PUT returns a version id");
assert!(
!version_is_absent_from_listing(&client, bucket, key, &old_version_id).await?,
"old noncurrent version must be readable before lifecycle is installed"
);
put_expiration_config(&client, bucket, noncurrent_expiration_rule("expire-noncurrent", "versioned/", 1)?).await?;
wait_for_version_expired(&client, bucket, key, &old_version_id, StdDuration::from_secs(90)).await?;
let latest = client.get_object().bucket(bucket).key(key).send().await?;
assert_eq!(latest.version_id(), Some(latest_version_id.as_str()));
assert_eq!(latest.body.collect().await?.into_bytes().as_ref(), b"latest payload");
let versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
let data_versions = versions.versions();
assert!(
data_versions
.iter()
.any(|v| v.version_id() == Some(latest_version_id.as_str())),
"latest data version {latest_version_id} must remain, got: {data_versions:?}"
);
assert!(
!data_versions.iter().any(|v| v.version_id() == Some(old_version_id.as_str())),
"old noncurrent version {old_version_id} must be removed, got: {data_versions:?}"
);
assert!(
versions.delete_markers().is_empty(),
"noncurrent version expiry must not create delete markers, got: {:?}",
versions.delete_markers()
);
Ok(())
}
/// `Days=0` expiration is invalid per S3 semantics (`Days` must be a positive
/// integer >= 1). A `PutBucketLifecycleConfiguration` carrying a zero-day rule
/// must be rejected with `InvalidArgument` (HTTP 400) see crates/lifecycle
/// must be rejected with `InvalidArgument` (HTTP 400) - see crates/lifecycle
/// `validate()` and the PutBucketLifecycleConfiguration handler. This is the
/// self-managed counterpart of the localhost-only
/// `test_bucket_lifecycle_rejects_zero_days` unit test.
File diff suppressed because it is too large Load Diff
@@ -2113,11 +2113,31 @@ async fn build_replication_pair(
async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let (_source_env, _target_env, source_bucket) = build_replication_pair(true).await?;
let response = run_replication_check(&_source_env, &source_bucket).await?;
let (source_env, target_env, source_bucket) = build_replication_pair(true).await?;
let response = run_replication_check(&source_env, &source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
assert!(response.text().await?.is_empty());
let payload: serde_json::Value = response.json().await?;
assert_eq!(payload["Status"], "OK");
assert_eq!(payload["ActiveMutation"], true);
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
assert_eq!(payload["Targets"][0]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
let target_client = target_env.create_s3_client();
let versions = target_client
.list_object_versions()
.bucket("replication-check-dst")
.prefix(".rustfs.sys/replication-check/")
.send()
.await?;
assert!(
versions.versions().is_empty() && versions.delete_markers().is_empty(),
"successful check must remove every probe version and delete marker"
);
Ok(())
}
@@ -2159,9 +2179,19 @@ async fn test_replication_check_rejects_target_without_object_lock() -> Result<(
let status = response.status();
let body = response.text().await?;
assert_eq!(status, StatusCode::BAD_REQUEST);
assert!(body.contains("InvalidRequest"), "unexpected response: {body}");
assert!(body.to_ascii_lowercase().contains("object lock"), "unexpected response: {body}");
assert_eq!(status, StatusCode::OK);
let payload: serde_json::Value = serde_json::from_str(&body)?;
assert_eq!(payload["Status"], "FAILED");
assert_eq!(payload["Targets"][0]["Status"], "FAILED");
assert_eq!(payload["Targets"][0]["Phases"]["ObjectLock"]["Status"], "FAILED");
assert!(
payload["Targets"][0]["Phases"]["ObjectLock"]["Error"]
.as_str()
.unwrap_or_default()
.contains("object lock"),
"unexpected response: {body}"
);
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "SKIPPED");
Ok(())
}
@@ -4773,6 +4803,153 @@ async fn test_site_replication_replicates_object_with_bucket_versioning_real_dua
Ok(())
}
/// Re-applying a site's own replication config must not disable the peer's reverse direction.
///
/// `PutBucketReplication` broadcasts the config to every peer — the console's replication
/// Save button, `mc replicate import`, and a bucket-metadata import all go through it. The
/// receiver used to overwrite its rules with the sender's, whose destination ARN names the
/// receiver itself. No bucket target can satisfy that ARN, so every object written on the
/// receiver was dropped with only a debug line, while `replicate status` still reported
/// "1/1 Buckets in sync" because both configs were byte-identical.
#[tokio::test]
#[serial]
async fn test_site_replication_config_broadcast_keeps_reverse_direction_real_dual_node() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
source_env
.start_rustfs_server_with_env(vec![], LOOPBACK_REPLICATION_TARGET_ENV)
.await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env
.start_rustfs_server_without_cleanup_with_env(LOOPBACK_REPLICATION_TARGET_ENV)
.await?;
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
let bucket = "site-repl-config-broadcast";
let add_status = site_replication_add(
&source_env,
&[
PeerSite {
name: "broadcast-source".to_string(),
endpoint: source_env.url.clone(),
access_key: source_env.access_key.clone(),
secret_key: source_env.secret_key.clone(),
..Default::default()
},
PeerSite {
name: "broadcast-target".to_string(),
endpoint: target_env.url.clone(),
access_key: target_env.access_key.clone(),
secret_key: target_env.secret_key.clone(),
..Default::default()
},
],
)
.await?;
assert!(add_status.success, "unexpected site add result: {add_status:?}");
wait_for_site_replication_enabled(&source_env, 2).await?;
wait_for_site_replication_enabled(&target_env, 2).await?;
source_client.create_bucket().bucket(bucket).send().await?;
wait_for_bucket_on_target(&target_client, bucket).await?;
// Both directions work before the broadcast.
source_client
.put_object()
.bucket(bucket)
.key("from-source.txt")
.body(ByteStream::from_static(b"written on the initiating site"))
.send()
.await?;
assert_eq!(
wait_for_object_on_target(&target_client, bucket, "from-source.txt").await?,
b"written on the initiating site".to_vec(),
);
target_client
.put_object()
.bucket(bucket)
.key("from-target.txt")
.body(ByteStream::from_static(b"written on the joined site"))
.send()
.await?;
assert_eq!(
wait_for_object_on_target(&source_client, bucket, "from-target.txt").await?,
b"written on the joined site".to_vec(),
);
// Round-trip the source's own config through PutBucketReplication, exactly what the
// console does when an operator opens the bucket's replication page and saves it.
let source_config = source_client
.get_bucket_replication()
.bucket(bucket)
.send()
.await?
.replication_configuration
.ok_or("source bucket has no replication configuration")?;
source_client
.put_bucket_replication()
.bucket(bucket)
.replication_configuration(source_config)
.send()
.await?;
let target_config = wait_for_site_replication_rule(&target_client, bucket).await?;
let target_deployment_id = site_replication_info(&target_env)
.await?
.sites
.iter()
.find(|peer| peer.endpoint == target_env.url)
.map(|peer| peer.deployment_id.clone())
.ok_or("joined site missing from its own replication info")?;
for rule in &target_config.rules {
let destination = rule
.destination
.as_ref()
.map(|destination| destination.bucket.as_str())
.unwrap_or_default();
assert!(
!destination.contains(&target_deployment_id),
"joined site adopted a rule pointing at itself: {destination}"
);
}
target_client
.put_object()
.bucket(bucket)
.key("from-target-after-broadcast.txt")
.body(ByteStream::from_static(b"written after the config broadcast"))
.send()
.await?;
assert_eq!(
wait_for_object_on_target(&source_client, bucket, "from-target-after-broadcast.txt").await?,
b"written after the config broadcast".to_vec(),
"config broadcast made replication one-directional"
);
Ok(())
}
async fn wait_for_site_replication_rule(
client: &aws_sdk_s3::Client,
bucket: &str,
) -> Result<aws_sdk_s3::types::ReplicationConfiguration, Box<dyn Error + Send + Sync>> {
for _ in 0..40 {
if let Ok(response) = client.get_bucket_replication().bucket(bucket).send().await
&& let Some(config) = response.replication_configuration
&& !config.rules.is_empty()
{
return Ok(config);
}
sleep(Duration::from_millis(250)).await;
}
Err(format!("bucket {bucket} never reported a replication rule").into())
}
#[tokio::test]
#[serial]
async fn test_site_replication_active_active_converges_without_loops_real_dual_node() -> TestResult {
+471
View File
@@ -0,0 +1,471 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Black-box SSE-C CopyObject and multipart-copy regression coverage (backlog#1467).
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::config::interceptors::{BeforeDeserializationInterceptorContextRef, BeforeTransmitInterceptorContextRef};
use aws_sdk_s3::config::{ConfigBag, Credentials, Intercept, Region, RuntimeComponents};
use aws_sdk_s3::error::BoxError;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use base64::Engine;
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const SSE_CUSTOMER_ALGORITHM_HEADER: &str = "x-amz-server-side-encryption-customer-algorithm";
const SSE_CUSTOMER_KEY_MD5_HEADER: &str = "x-amz-server-side-encryption-customer-key-md5";
struct CustomerKey {
raw: String,
encoded: String,
md5: String,
}
struct InvalidSsec<'a> {
algorithm: Option<&'a str>,
key: Option<&'a str>,
md5: Option<&'a str>,
}
#[derive(Clone, Debug, Default)]
struct ResponseHeaderCapture {
headers: Arc<Mutex<HashMap<String, String>>>,
abort_attempts: Arc<AtomicUsize>,
}
impl ResponseHeaderCapture {
fn snapshot(&self) -> Result<HashMap<String, String>, BoxError> {
self.headers
.lock()
.map(|headers| headers.clone())
.map_err(|_| std::io::Error::other("response header capture mutex was poisoned").into())
}
fn abort_attempts(&self) -> usize {
self.abort_attempts.load(Ordering::SeqCst)
}
}
impl Intercept for ResponseHeaderCapture {
fn name(&self) -> &'static str {
"ssec-copy-response-header-capture"
}
fn read_before_deserialization(
&self,
context: &BeforeDeserializationInterceptorContextRef<'_>,
_runtime_components: &RuntimeComponents,
_cfg: &mut ConfigBag,
) -> Result<(), BoxError> {
let mut captured = self
.headers
.lock()
.map_err(|_| std::io::Error::other("response header capture mutex was poisoned"))?;
captured.clear();
for name in [SSE_CUSTOMER_ALGORITHM_HEADER, SSE_CUSTOMER_KEY_MD5_HEADER] {
if let Some(value) = context.response().headers().get(name) {
captured.insert(name.to_owned(), value.to_owned());
}
}
Ok(())
}
fn read_before_transmit(
&self,
context: &BeforeTransmitInterceptorContextRef<'_>,
_runtime_components: &RuntimeComponents,
_cfg: &mut ConfigBag,
) -> Result<(), BoxError> {
let request = context.request();
if request.method() == "DELETE" && request.uri().contains("uploadId=") {
self.abort_attempts.fetch_add(1, Ordering::SeqCst);
}
Ok(())
}
}
fn customer_key(byte: u8) -> CustomerKey {
let raw = [byte; 32];
CustomerKey {
raw: String::from_utf8_lossy(&raw).into_owned(),
encoded: base64::engine::general_purpose::STANDARD.encode(raw),
md5: base64::engine::general_purpose::STANDARD.encode(md5::compute(raw).0),
}
}
fn assert_secret_absent(error: &str, keys: &[&CustomerKey]) {
for key in keys {
assert!(!error.contains(&key.raw), "error exposed a raw SSE-C key");
assert!(!error.contains(&key.encoded), "error exposed an encoded SSE-C key");
assert!(!error.contains(&key.md5), "error exposed an SSE-C key MD5");
}
}
fn invalid_ssec_cases<'a>(correct_key: &'a CustomerKey, wrong_key: &'a CustomerKey) -> [InvalidSsec<'a>; 5] {
[
InvalidSsec {
algorithm: None,
key: Some(&correct_key.encoded),
md5: Some(&correct_key.md5),
},
InvalidSsec {
algorithm: Some("AES256"),
key: None,
md5: Some(&correct_key.md5),
},
InvalidSsec {
algorithm: Some("AES256"),
key: Some(&correct_key.encoded),
md5: None,
},
InvalidSsec {
algorithm: Some("AES256"),
key: Some(&wrong_key.encoded),
md5: Some(&wrong_key.md5),
},
InvalidSsec {
algorithm: Some("AES256"),
key: Some(&correct_key.encoded),
md5: Some(&wrong_key.md5),
},
]
}
#[tokio::test]
async fn copy_object_rotates_ssec_key_and_drops_source_encryption_metadata() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "ssec-copy-object";
let source = "source.bin";
let plaintext_copy = "plaintext-copy.bin";
let rotated_copy = "rotated-copy.bin";
let source_key = customer_key(0x41);
let destination_key = customer_key(0x42);
let wrong_key = customer_key(0x43);
let body = b"backlog-1467 versioned SSE-C copy payload";
env.create_test_bucket(bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let put = client
.put_object()
.bucket(bucket)
.key(source)
.sse_customer_algorithm("AES256")
.sse_customer_key(&source_key.encoded)
.sse_customer_key_md5(&source_key.md5)
.body(ByteStream::from_static(body))
.send()
.await?;
let source_version = put.version_id().ok_or("versioned PUT returned no version ID")?;
let copy_source = format!("{bucket}/{source}?versionId={source_version}");
let plaintext = client
.copy_object()
.bucket(bucket)
.key(plaintext_copy)
.copy_source(&copy_source)
.copy_source_sse_customer_algorithm("AES256")
.copy_source_sse_customer_key(&source_key.encoded)
.copy_source_sse_customer_key_md5(&source_key.md5)
.send()
.await?;
assert_eq!(plaintext.copy_source_version_id(), Some(source_version));
let plaintext_body = client
.get_object()
.bucket(bucket)
.key(plaintext_copy)
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(plaintext_body.as_ref(), body);
let rotated = client
.copy_object()
.bucket(bucket)
.key(rotated_copy)
.copy_source(&copy_source)
.copy_source_sse_customer_algorithm("AES256")
.copy_source_sse_customer_key(&source_key.encoded)
.copy_source_sse_customer_key_md5(&source_key.md5)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?;
assert_eq!(rotated.sse_customer_algorithm(), Some("AES256"));
assert_eq!(rotated.sse_customer_key_md5(), Some(destination_key.md5.as_str()));
let wrong_key_error = client
.get_object()
.bucket(bucket)
.key(rotated_copy)
.sse_customer_algorithm("AES256")
.sse_customer_key(&source_key.encoded)
.sse_customer_key_md5(&source_key.md5)
.send()
.await
.expect_err("the source key must not read a copy encrypted with the destination key");
assert_secret_absent(&format!("{wrong_key_error:?}"), &[&source_key, &destination_key]);
let rotated_body = client
.get_object()
.bucket(bucket)
.key(rotated_copy)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(rotated_body.as_ref(), body);
for (case_index, case) in invalid_ssec_cases(&source_key, &wrong_key).iter().enumerate() {
let failed_target = format!("failed-copy-{case_index}.bin");
let mut request = client
.copy_object()
.bucket(bucket)
.key(&failed_target)
.copy_source(&copy_source);
if let Some(algorithm) = case.algorithm {
request = request.copy_source_sse_customer_algorithm(algorithm);
}
if let Some(key) = case.key {
request = request.copy_source_sse_customer_key(key);
}
if let Some(md5) = case.md5 {
request = request.copy_source_sse_customer_key_md5(md5);
}
let error = request
.send()
.await
.expect_err("invalid source SSE-C parameters must reject CopyObject");
assert_secret_absent(&format!("{error:?}"), &[&source_key, &wrong_key]);
assert!(
client.head_object().bucket(bucket).key(&failed_target).send().await.is_err(),
"a rejected CopyObject must not create its target"
);
}
env.stop_server();
Ok(())
}
#[tokio::test]
async fn multipart_copy_requires_keys_on_every_stage_and_abort_leaves_no_object() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let response_headers = ResponseHeaderCapture::default();
let credentials = Credentials::new(&env.access_key, &env.secret_key, None, None, "ssec-copy-e2e");
let config = aws_sdk_s3::Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.http_client(SmithyHttpClientBuilder::new().build_http())
.interceptor(response_headers.clone())
.build();
let client = aws_sdk_s3::Client::from_conf(config);
let bucket = "ssec-multipart-copy";
let source = "source.bin";
let destination = "destination.bin";
let aborted_destination = "aborted.bin";
let source_key = customer_key(0x51);
let destination_key = customer_key(0x52);
let wrong_key = customer_key(0x53);
let part_size = 5 * 1024 * 1024;
let body: Vec<u8> = (0..part_size * 2).map(|index| (index % 251) as u8).collect();
env.create_test_bucket(bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let source_put = client
.put_object()
.bucket(bucket)
.key(source)
.sse_customer_algorithm("AES256")
.sse_customer_key(&source_key.encoded)
.sse_customer_key_md5(&source_key.md5)
.body(ByteStream::from(body.clone()))
.send()
.await?;
let source_version = source_put
.version_id()
.ok_or("versioned multipart-copy source returned no version ID")?;
let create = client
.create_multipart_upload()
.bucket(bucket)
.key(destination)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?;
assert_eq!(create.sse_customer_algorithm(), Some("AES256"));
assert_eq!(create.sse_customer_key_md5(), Some(destination_key.md5.as_str()));
let upload_id = create.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
let mut completed = Vec::new();
for part_number in 1..=2 {
let first = (part_number - 1) * part_size;
let last = part_number * part_size - 1;
let copied = client
.upload_part_copy()
.bucket(bucket)
.key(destination)
.upload_id(upload_id)
.part_number(part_number)
.copy_source(format!("{bucket}/{source}"))
.copy_source_range(format!("bytes={first}-{last}"))
.copy_source_sse_customer_algorithm("AES256")
.copy_source_sse_customer_key(&source_key.encoded)
.copy_source_sse_customer_key_md5(&source_key.md5)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?;
assert_eq!(
copied.copy_source_version_id(),
Some(source_version),
"UploadPartCopy must return the actual latest source version"
);
let etag = copied
.copy_part_result()
.and_then(|result| result.e_tag())
.ok_or("UploadPartCopy returned no ETag")?;
completed.push(CompletedPart::builder().part_number(part_number).e_tag(etag).build());
}
client
.complete_multipart_upload()
.bucket(bucket)
.key(destination)
.upload_id(upload_id)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed)).build())
.send()
.await?;
let completed_headers = response_headers.snapshot()?;
assert_eq!(completed_headers.get(SSE_CUSTOMER_ALGORITHM_HEADER).map(String::as_str), Some("AES256"));
assert_eq!(
completed_headers.get(SSE_CUSTOMER_KEY_MD5_HEADER).map(String::as_str),
Some(destination_key.md5.as_str())
);
let downloaded = client
.get_object()
.bucket(bucket)
.key(destination)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(downloaded.as_ref(), body.as_slice());
for (case_index, case) in invalid_ssec_cases(&destination_key, &wrong_key).iter().enumerate() {
let failed_target = format!("{aborted_destination}-{case_index}");
let failed_create = client
.create_multipart_upload()
.bucket(bucket)
.key(&failed_target)
.sse_customer_algorithm("AES256")
.sse_customer_key(&destination_key.encoded)
.sse_customer_key_md5(&destination_key.md5)
.send()
.await?;
let failed_upload_id = failed_create
.upload_id()
.ok_or("CreateMultipartUpload returned no upload ID")?;
let mut request = client
.upload_part_copy()
.bucket(bucket)
.key(&failed_target)
.upload_id(failed_upload_id)
.part_number(1)
.copy_source(format!("{bucket}/{source}"))
.copy_source_sse_customer_algorithm("AES256")
.copy_source_sse_customer_key(&source_key.encoded)
.copy_source_sse_customer_key_md5(&source_key.md5);
if let Some(algorithm) = case.algorithm {
request = request.sse_customer_algorithm(algorithm);
}
if let Some(key) = case.key {
request = request.sse_customer_key(key);
}
if let Some(md5) = case.md5 {
request = request.sse_customer_key_md5(md5);
}
let error = request
.send()
.await
.expect_err("invalid destination SSE-C parameters must reject UploadPartCopy");
assert_secret_absent(&format!("{error:?}"), &[&source_key, &destination_key, &wrong_key]);
let abort_attempts_before = response_headers.abort_attempts();
client
.abort_multipart_upload()
.bucket(bucket)
.key(&failed_target)
.upload_id(failed_upload_id)
.send()
.await?;
assert_eq!(
response_headers.abort_attempts(),
abort_attempts_before + 1,
"each failed multipart copy must issue exactly one wire-level abort attempt"
);
assert!(
client.head_object().bucket(bucket).key(&failed_target).send().await.is_err(),
"an aborted failed multipart copy must leave no completed object"
);
}
env.stop_server();
Ok(())
}
+13
View File
@@ -16,6 +16,8 @@
pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::disk::{VolumeInfo, WalkDirOptions};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers};
pub(crate) use rustfs_ecstore::api::rpc::{TonicInterceptor, node_service_time_out_client_no_auth};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{gen_tonic_signature_interceptor, node_service_time_out_client};
@@ -31,6 +33,17 @@ pub(crate) mod grpc_lock {
pub(crate) use super::{TonicInterceptor, node_service_time_out_client_no_auth};
}
/// Signing/transport surface used by the cross-process internode RPC signature
/// acceptance tests (backlog#1327). The signing helpers are what let a test mint
/// the one legitimately signed request an on-path attacker is assumed to have
/// captured; every attack in that suite then only *reuses* those bytes.
#[cfg(test)]
pub(crate) mod internode_rpc_signature {
pub(crate) use super::{
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers, node_service_time_out_client_no_auth,
};
}
#[cfg(test)]
pub(crate) mod replication_extension {
pub(crate) use super::BucketTargetSys;
@@ -0,0 +1,428 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Truthful storage-class write and discovery contract regressions.
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{ObjectAttributes, StorageClass};
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde_json::Value;
use std::error::Error;
use std::path::Path;
const UNSUPPORTED_AWS_CLASSES: [&str; 9] = [
"DEEP_ARCHIVE",
"EXPRESS_ONEZONE",
"GLACIER",
"GLACIER_IR",
"INTELLIGENT_TIERING",
"ONEZONE_IA",
"OUTPOSTS",
"SNOW",
"STANDARD_IA",
];
async fn assert_object_storage_class(
client: &Client,
bucket: &str,
key: &str,
expected: &str,
body: &[u8],
) -> Result<(), Box<dyn Error + Send + Sync>> {
let head = client.head_object().bucket(bucket).key(key).send().await?;
let expected_head = (expected != "STANDARD").then_some(expected);
assert_eq!(
head.storage_class().map(StorageClass::as_str),
expected_head,
"HeadObject must omit implicit STANDARD and report RRS"
);
let listed = client.list_objects_v2().bucket(bucket).prefix(key).send().await?;
let object = listed
.contents()
.iter()
.find(|object| object.key() == Some(key))
.ok_or("object missing from ListObjectsV2")?;
assert_eq!(object.storage_class().map(|storage_class| storage_class.as_str()), Some(expected));
let get = client.get_object().bucket(bucket).key(key).send().await?;
assert_eq!(
get.storage_class().map(StorageClass::as_str),
expected_head,
"GetObject must report the same effective storage class as HeadObject"
);
let downloaded = get.body.collect().await?.into_bytes();
assert_eq!(downloaded.as_ref(), body, "storage-class selection must not alter object bytes");
Ok(())
}
async fn mutate_xl_meta(
root: &str,
bucket: &str,
key: &str,
mutate: impl FnOnce(&mut rustfs_filemeta::MetaObject),
) -> Result<(), Box<dyn Error + Send + Sync>> {
let path = Path::new(root).join(bucket).join(key).join("xl.meta");
let bytes = tokio::fs::read(&path).await?;
let mut file_meta = rustfs_filemeta::FileMeta::load(&bytes)?;
let (index, mut version) = file_meta.find_version(None)?;
let object = version.object.as_mut().ok_or("fixture version is not an object")?;
mutate(object);
file_meta.versions[index] = rustfs_filemeta::FileMetaShallowVersion::try_from(version)?;
tokio::fs::write(path, file_meta.marshal_msg()?).await?;
Ok(())
}
async fn signed_admin_get(
env: &RustFSTestEnvironment,
path: &str,
) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
let url = format!("{}{path}", env.url);
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let request = http::Request::builder()
.method(http::Method::GET)
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
.body(Body::empty())?;
let signed = sign_v4(request, 0, &env.access_key, &env.secret_key, "", "us-east-1");
let mut request = local_http_client().get(&url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
Ok(request.send().await?)
}
#[tokio::test]
async fn standard_and_rrs_are_supported_across_put_copy_and_multipart() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "storage-class-supported-contract";
env.create_test_bucket(bucket).await?;
client
.put_object()
.bucket(bucket)
.key("copy-source")
.body(ByteStream::from_static(b"copy-source-body"))
.send()
.await?;
for storage_class in [StorageClass::Standard, StorageClass::ReducedRedundancy] {
let class_name = storage_class.as_str().to_string();
let put_key = format!("put-{class_name}");
let put_body = format!("put-body-{class_name}").into_bytes();
client
.put_object()
.bucket(bucket)
.key(&put_key)
.storage_class(storage_class.clone())
.body(ByteStream::from(put_body.clone()))
.send()
.await?;
assert_object_storage_class(&client, bucket, &put_key, &class_name, &put_body).await?;
let copy_key = format!("copy-{class_name}");
client
.copy_object()
.bucket(bucket)
.key(&copy_key)
.copy_source(format!("{bucket}/copy-source"))
.storage_class(storage_class.clone())
.send()
.await?;
assert_object_storage_class(&client, bucket, &copy_key, &class_name, b"copy-source-body").await?;
let multipart_key = format!("multipart-{class_name}");
let created = client
.create_multipart_upload()
.bucket(bucket)
.key(&multipart_key)
.storage_class(storage_class)
.send()
.await?;
let upload_id = created.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
let parts = client
.list_parts()
.bucket(bucket)
.key(&multipart_key)
.upload_id(upload_id)
.send()
.await?;
assert_eq!(parts.storage_class().map(StorageClass::as_str), Some(class_name.as_str()));
client
.abort_multipart_upload()
.bucket(bucket)
.key(&multipart_key)
.upload_id(upload_id)
.send()
.await?;
}
Ok(())
}
#[tokio::test]
async fn label_only_aws_classes_fail_before_put_copy_or_multipart_mutation() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "storage-class-unsupported-contract";
env.create_test_bucket(bucket).await?;
for key in ["put-guard", "copy-source", "copy-guard"] {
client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from(format!("original-{key}").into_bytes()))
.send()
.await?;
}
for unsupported in UNSUPPORTED_AWS_CLASSES {
let put_error = client
.put_object()
.bucket(bucket)
.key("put-guard")
.storage_class(StorageClass::from(unsupported))
.body(ByteStream::from(format!("rejected-put-{unsupported}").into_bytes()))
.send()
.await
.expect_err("label-only PUT storage class must be rejected");
assert_eq!(
put_error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidStorageClass"),
"PUT returned a different error for {unsupported}"
);
let copy_error = client
.copy_object()
.bucket(bucket)
.key("copy-guard")
.copy_source(format!("{bucket}/copy-source"))
.storage_class(StorageClass::from(unsupported))
.send()
.await
.expect_err("label-only CopyObject storage class must be rejected");
assert_eq!(
copy_error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidStorageClass"),
"CopyObject returned a different error for {unsupported}"
);
let multipart_key = format!("multipart-{unsupported}");
let multipart_error = client
.create_multipart_upload()
.bucket(bucket)
.key(&multipart_key)
.storage_class(StorageClass::from(unsupported))
.send()
.await
.expect_err("label-only CreateMultipartUpload storage class must be rejected");
assert_eq!(
multipart_error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidStorageClass"),
"CreateMultipartUpload returned a different error for {unsupported}"
);
}
let put_guard = client
.get_object()
.bucket(bucket)
.key("put-guard")
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(put_guard.as_ref(), b"original-put-guard");
let copy_guard = client
.get_object()
.bucket(bucket)
.key("copy-guard")
.send()
.await?
.body
.collect()
.await?
.into_bytes();
assert_eq!(copy_guard.as_ref(), b"original-copy-guard");
let uploads = client.list_multipart_uploads().bucket(bucket).send().await?;
assert!(uploads.uploads().is_empty(), "unsupported classes must not create multipart sessions");
Ok(())
}
#[tokio::test]
async fn historical_label_only_metadata_is_standard_without_hiding_a_real_transition_tier()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let client = env.create_s3_client();
let bucket = "storage-class-historical-contract";
let legacy_key = "legacy-label-only";
let transitioned_key = "real-transition-tier";
env.create_test_bucket(bucket).await?;
for key in [legacy_key, transitioned_key] {
client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"fixture-body"))
.send()
.await?;
}
env.stop_server();
mutate_xl_meta(&env.temp_dir, bucket, legacy_key, |object| {
object
.meta_user
.insert("x-amz-storage-class".to_string(), "STANDARD_IA".to_string());
})
.await?;
mutate_xl_meta(&env.temp_dir, bucket, transitioned_key, |object| {
object.set_transition(&rustfs_filemeta::FileInfo {
transition_status: rustfs_filemeta::TRANSITION_COMPLETE.to_string(),
transition_tier: "STANDARD_IA".to_string(),
..Default::default()
});
})
.await?;
env.restart_server_preserving_data(Vec::new(), &[]).await?;
assert_object_storage_class(&client, bucket, legacy_key, "STANDARD", b"fixture-body").await?;
let legacy_attributes = client
.get_object_attributes()
.bucket(bucket)
.key(legacy_key)
.object_attributes(ObjectAttributes::StorageClass)
.send()
.await?;
assert_eq!(legacy_attributes.storage_class().map(StorageClass::as_str), Some("STANDARD"));
let versions = client.list_object_versions().bucket(bucket).prefix(legacy_key).send().await?;
let legacy_version = versions
.versions()
.iter()
.find(|version| version.key() == Some(legacy_key))
.ok_or("legacy fixture missing from ListObjectVersions")?;
assert_eq!(legacy_version.storage_class().map(|class| class.as_str()), Some("STANDARD"));
let transitioned_head = client.head_object().bucket(bucket).key(transitioned_key).send().await?;
assert_eq!(transitioned_head.storage_class().map(StorageClass::as_str), Some("STANDARD_IA"));
let transitioned_attributes = client
.get_object_attributes()
.bucket(bucket)
.key(transitioned_key)
.object_attributes(ObjectAttributes::StorageClass)
.send()
.await?;
assert_eq!(transitioned_attributes.storage_class().map(StorageClass::as_str), Some("STANDARD_IA"));
let transitioned_list = client
.list_objects_v2()
.bucket(bucket)
.prefix(transitioned_key)
.send()
.await?;
assert_eq!(
transitioned_list.contents()[0].storage_class().map(|class| class.as_str()),
Some("STANDARD_IA")
);
let transitioned_versions = client
.list_object_versions()
.bucket(bucket)
.prefix(transitioned_key)
.send()
.await?;
assert_eq!(
transitioned_versions.versions()[0]
.storage_class()
.map(|class| class.as_str()),
Some("STANDARD_IA")
);
Ok(())
}
#[tokio::test]
async fn authenticated_runtime_capabilities_publish_the_versioned_storage_class_contract()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
let path = "/rustfs/admin/v4/runtime/capabilities";
let unsigned = local_http_client().get(format!("{}{path}", env.url)).send().await?;
assert_eq!(unsigned.status(), StatusCode::FORBIDDEN);
let unsigned_body = unsigned.text().await?;
assert!(
!unsigned_body.contains("supported_write_classes"),
"the capability contract must not bypass admin authentication"
);
assert!(
!unsigned_body.contains("manual_transition_jobs"),
"manual transition job capabilities must not bypass admin authentication"
);
let response = signed_admin_get(&env, path).await?;
assert_eq!(response.status(), StatusCode::OK);
let body: Value = response.json().await?;
assert_eq!(body["storage_classes"]["contract_version"], 1);
assert_eq!(
body["storage_classes"]["supported_write_classes"],
serde_json::json!(["STANDARD", "REDUCED_REDUNDANCY"])
);
assert_eq!(body["storage_classes"]["unsupported_write_error"], "InvalidStorageClass");
assert_eq!(body["storage_classes"]["legacy_label_behavior"], "normalized_to_effective_class");
assert_eq!(body["summary"]["manual_transition_jobs"]["state"], "supported");
assert_eq!(body["manual_transition_jobs"]["contract_version"], 1);
assert_eq!(body["manual_transition_jobs"]["status"]["state"], "supported");
assert_eq!(body["manual_transition_jobs"]["modes"], serde_json::json!(["enqueue_only", "async"]));
assert_eq!(body["manual_transition_jobs"]["run_route"], "/rustfs/admin/v3/ilm/transition/run");
assert_eq!(
body["manual_transition_jobs"]["status_route"],
"/rustfs/admin/v3/ilm/transition/jobs/{job_id}"
);
assert_eq!(
body["manual_transition_jobs"]["cancel_route"],
"/rustfs/admin/v3/ilm/transition/jobs/{job_id}"
);
assert_eq!(body["manual_transition_jobs"]["job_id_format"], "uuid");
assert_eq!(body["manual_transition_jobs"]["admission_scope"], "bucket");
assert_eq!(
body["manual_transition_jobs"]["mixed_version_policy"],
"fail_closed_when_capability_unknown_or_unsupported"
);
Ok(())
}
}
@@ -0,0 +1,229 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_sts::config::retry::RetryConfig;
use aws_sdk_sts::config::{Credentials, Region};
use aws_sdk_sts::error::ProvideErrorMetadata;
use aws_sdk_sts::operation::RequestId;
use aws_sdk_sts::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
type BoxError = Box<dyn Error + Send + Sync>;
type TestResult = Result<(), BoxError>;
fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
let mut config = Config::builder()
.credentials_provider(Credentials::new(
access_key,
secret_key,
session_token.map(str::to_owned),
None,
"e2e-sts-query-compat",
))
.region(Region::new("us-east-1"))
.endpoint_url(url)
.retry_config(RetryConfig::standard().with_max_attempts(1))
.behavior_version_latest();
if url.starts_with("http://") {
config = config.http_client(SmithyHttpClientBuilder::new().build_http());
}
Client::from_conf(config.build())
}
async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
let path = "/rustfs/admin/v3/add-service-accounts";
let url = format!("{}{path}", env.url);
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let body = serde_json::json!({ "targetUser": env.access_key.clone() }).to_string();
let request = http::Request::builder()
.method(http::Method::PUT)
.uri(uri)
.header(HOST, authority)
.header(CONTENT_TYPE, "application/json")
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
.body(Body::empty())?;
let content_length = i64::try_from(body.len()).map_err(|_| "service account request body is too large")?;
let signed = sign_v4(request, content_length, &env.access_key, &env.secret_key, "", "us-east-1");
let mut request = local_http_client().put(&url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
let response = request.body(body).send().await?;
let status = response.status();
let body = response.text().await?;
if !status.is_success() {
return Err(format!("create service account failed: {status} {body}").into());
}
let response: serde_json::Value = serde_json::from_str(&body)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
.to_owned();
let secret_key = response["credentials"]["secretKey"]
.as_str()
.ok_or("service account response should contain credentials.secretKey")?
.to_owned();
Ok((access_key, secret_key))
}
async fn assert_chaining_denied(client: &Client, credential_kind: &str) -> TestResult {
let error = client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-compat-e2e")
.send()
.await
.expect_err("credential chaining must be denied");
let service_error = error
.as_service_error()
.ok_or_else(|| format!("{credential_kind} denial should deserialize as an STS service error: {error:?}"))?;
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
assert_eq!(service_error.code(), Some("AccessDenied"));
assert_eq!(service_error.message(), Some("Access Denied"));
assert!(
error.request_id().is_some_and(|request_id| !request_id.is_empty()),
"{credential_kind} denial should include a request ID"
);
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let assumed = sts_client(&env.url, &env.access_key, &env.secret_key, None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-compat-e2e")
.send()
.await?;
assert!(
assumed.request_id().is_some_and(|request_id| !request_id.is_empty()),
"successful AssumeRole should include a request ID"
);
let temporary = assumed
.credentials()
.ok_or("successful AssumeRole response should contain credentials")?;
let invalid_signature = sts_client(&env.url, &env.access_key, "incorrect-secret-key", None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-invalid-signature")
.send()
.await
.expect_err("an invalid signature must be rejected");
assert_eq!(invalid_signature.raw_response().map(|response| response.status().as_u16()), Some(403));
let invalid_signature_service_error = invalid_signature
.as_service_error()
.ok_or_else(|| format!("invalid signature should deserialize as an STS service error: {invalid_signature:?}"))?;
assert_eq!(invalid_signature_service_error.code(), Some("SignatureDoesNotMatch"));
assert!(
invalid_signature_service_error
.message()
.is_some_and(|message| message.starts_with("The request signature we calculated does not match")),
"signature rejection should preserve the canonical error message"
);
assert!(
invalid_signature
.request_id()
.is_some_and(|request_id| !request_id.is_empty()),
"signature rejection should include a request ID"
);
assert_chaining_denied(
&sts_client(
&env.url,
temporary.access_key_id(),
temporary.secret_access_key(),
Some(temporary.session_token()),
),
"temporary credential",
)
.await?;
let (service_access_key, service_secret_key) = create_root_service_account(&env).await?;
assert_chaining_denied(&sts_client(&env.url, &service_access_key, &service_secret_key, None), "service account").await?;
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_query_rate_limit_error_is_aws_sdk_compatible() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
("RUSTFS_API_RATE_LIMIT_RPM", "60"),
("RUSTFS_API_RATE_LIMIT_BURST", "1"),
],
)
.await?;
let client = sts_client(&env.url, &env.access_key, &env.secret_key, None);
let mut throttled = None;
let request = || {
client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-rate-limit")
.send()
};
let (first, second, third, fourth) = tokio::join!(request(), request(), request(), request());
for result in [first, second, third, fourth] {
if let Err(error) = result
&& error.raw_response().map(|response| response.status().as_u16()) == Some(429)
{
throttled = Some(error);
break;
}
}
let error = throttled.ok_or("at least one concurrent STS request should be throttled at burst one")?;
let service_error = error
.as_service_error()
.ok_or_else(|| format!("rate limit response should deserialize as an STS service error: {error:?}"))?;
assert_eq!(service_error.code(), Some("TooManyRequests"));
assert!(
service_error
.message()
.is_some_and(|message| message.starts_with("Request rate limit exceeded")),
"rate limit response should preserve the server message"
);
assert!(
error.request_id().is_some_and(|request_id| !request_id.is_empty()),
"rate limit response should include a request ID"
);
env.stop_server();
Ok(())
}
+60 -38
View File
@@ -43,10 +43,27 @@ pub mod bucket {
pub mod bucket_lifecycle_ops {
pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
ExpiryState, LifecycleOps, RestoreRequestOps, TransitionState, TransitionedObject, apply_expiry_rule,
ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
ManualTransitionRunReport, RestoreRequestOps, TransitionState, TransitionedObject, apply_expiry_rule,
apply_transition_rule, enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects,
enqueue_transition_for_existing_objects_scoped, enqueue_transition_for_existing_objects_scoped_with_cancel,
enqueue_transition_immediate, expire_transitioned_object, get_global_expiry_state, get_global_transition_state,
init_background_expiry, post_restore_opts, run_stale_multipart_upload_cleanup_once, validate_transition_tier,
init_background_expiry, manual_transition_queue_snapshot, post_restore_opts,
run_stale_multipart_upload_cleanup_once, validate_transition_tier,
};
}
pub mod manual_transition_job {
pub use crate::bucket::lifecycle::manual_transition_job::{
ManualTransitionJobRecord, ManualTransitionJobState, ManualTransitionScopeAdmission,
ManualTransitionScopeAdmissionClaim, claim_manual_transition_scope_admission,
delete_manual_transition_scope_admission_if_current, load_manual_transition_job_record,
load_manual_transition_job_record_with_etag, load_manual_transition_scope_admission,
manual_transition_job_lease_expired, manual_transition_scope_admission_lease_expired,
manual_transition_scope_key, persist_manual_transition_job_progress, renew_manual_transition_job_lease,
request_manual_transition_job_cancel, save_manual_transition_job_record,
save_manual_transition_job_record_if_current, save_manual_transition_scope_admission_if_absent,
};
}
@@ -105,12 +122,13 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BucketMetadataSys, delete, get, get_accelerate_config, get_bucket_policy, get_bucket_policy_raw,
get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
BucketMetadataSys, acquire_bucket_targets_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, remove_bucket_metadata, set_bucket_metadata, update,
update_bucket_targets_under_transaction_lock, update_config_with,
};
}
@@ -147,18 +165,19 @@ pub mod bucket {
pub mod replication {
pub use crate::bucket::replication::{
BucketReplicationResyncStatus, BucketStats, DeletedObjectReplicationInfo, DynReplicationPool, MustReplicateOptions,
ObjectOpts, REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationConfig,
ReplicationConfigurationExt, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource,
ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait,
ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge, ReplicationState, ReplicationStats,
ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError, ReplicationType, ResyncOpts,
ResyncStatusType, TargetReplicationResyncStatus, VersionPurgeStatusType, delete_replication_state_from_config,
delete_replication_version_id, get_global_replication_pool, get_global_replication_stats,
init_background_replication, replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map,
replication_target_arns, should_remove_replication_target, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
validate_replication_config_target_arns, version_purge_status_to_filemeta,
BucketReplicationResyncStatus, BucketStats, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool,
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REPLICATE_INCOMING_DELETE, ReplicateDecision,
ReplicateObjectInfo, ReplicationConfig, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
ReplicationType, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus, VersionPurgeStatusType,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
};
}
@@ -240,21 +259,23 @@ pub mod config {
pub mod com {
pub use crate::config::com::{
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
ServerConfigCorruptError, delete_config, is_server_config_corrupt_error, lookup_configs, read_config,
read_config_no_lock, read_config_with_metadata, read_config_without_migrate, read_config_without_migrate_no_lock,
read_existing_server_config_no_lock, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
save_server_config_no_lock, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
with_server_config_read_lock, with_server_config_write_lock,
ServerConfigCorruptError, ServerConfigSnapshot, delete_config, is_server_config_corrupt_error, lookup_configs,
read_config, read_config_no_lock, read_config_with_metadata, read_config_without_migrate,
read_config_without_migrate_no_lock, read_existing_server_config_no_lock, read_server_config_snapshot, save_config,
save_config_no_lock, save_config_with_opts, save_server_config, save_server_config_no_lock,
save_server_config_snapshot, server_config_path, try_migrate_server_config, with_config_object_read_lock,
with_config_object_write_lock, with_server_config_read_lock, with_server_config_write_lock,
};
}
pub mod storageclass {
pub use crate::config::storageclass::{
CLASS_RRS, CLASS_STANDARD, Config, DEEP_ARCHIVE, DEFAULT_INLINE_BLOCK, DEFAULT_KVS, DEFAULT_RRS_PARITY,
EXPRESS_ONEZONE, GLACIER, GLACIER_IR, INLINE_BLOCK, INLINE_BLOCK_ENV, INTELLIGENT_TIERING, MIN_PARITY_DRIVES,
ONEZONE_IA, OPTIMIZE, OPTIMIZE_ENV, OUTPOSTS, RRS, RRS_ENV, SCHEME_PREFIX, SNOW, STANDARD, STANDARD_ENV, STANDARD_IA,
StorageClass, default_parity_count, lookup_config, lookup_config_for_pools, parse_storage_class, validate_parity,
validate_parity_inner,
CAPABILITY_CONTRACT_VERSION, CLASS_RRS, CLASS_STANDARD, Config, DEEP_ARCHIVE, DEFAULT_INLINE_BLOCK, DEFAULT_KVS,
DEFAULT_RRS_PARITY, EXPRESS_ONEZONE, GLACIER, GLACIER_IR, INLINE_BLOCK, INLINE_BLOCK_ENV, INTELLIGENT_TIERING,
LEGACY_LABEL_BEHAVIOR, MIN_PARITY_DRIVES, ONEZONE_IA, OPTIMIZE, OPTIMIZE_ENV, OUTPOSTS, RRS, RRS_ENV, SCHEME_PREFIX,
SNOW, STANDARD, STANDARD_ENV, STANDARD_IA, SUPPORTED_WRITE_CLASSES, StorageClass, UNSUPPORTED_WRITE_ERROR,
default_parity_count, effective_class, is_supported_write_class, lookup_config, lookup_config_for_pools,
parse_storage_class, validate_parity, validate_parity_inner,
};
}
@@ -266,10 +287,10 @@ pub mod config {
pub mod data_usage {
pub use crate::data_usage::{
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, live_bucket_usage_computations, load_compression_total_from_memory,
load_data_usage_from_backend, load_data_usage_from_backend_cached, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
init_compression_total_memory_from_backend, invalidate_data_usage_snapshot_cache, live_bucket_usage_computations,
load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached,
record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
store_data_usage_in_backend,
@@ -282,9 +303,9 @@ pub mod disk {
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, OldCurrentSize, RUSTFS_META_BUCKET, ReadMultipleReq,
ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
new_disk, validate_batch_read_version_item_count,
FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count,
};
pub use bytes::Bytes;
pub use endpoint::Endpoint;
@@ -385,11 +406,12 @@ pub mod rio {
pub mod rpc {
pub use crate::cluster::rpc::{
LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor,
gen_signature_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest, verify_tonic_rpc_response_proof,
LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerPeerActivity,
TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, verify_rpc_signature,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature,
};
}
+23 -9
View File
@@ -807,15 +807,29 @@ impl BucketTargetSys {
&& !new_targets.is_empty()
{
for target in &new_targets.targets {
if let Ok(client) = self.get_remote_target_client_internal(target).await {
arn_remotes_map.insert(
target.arn.clone(),
ArnTarget {
client: Some(Arc::new(client)),
last_refresh: OffsetDateTime::now_utc(),
},
);
self.update_bandwidth_limit(bucket, &target.arn, target.bandwidth_limit);
match self.get_remote_target_client_internal(target).await {
Ok(client) => {
arn_remotes_map.insert(
target.arn.clone(),
ArnTarget {
client: Some(Arc::new(client)),
last_refresh: OffsetDateTime::now_utc(),
},
);
self.update_bandwidth_limit(bucket, &target.arn, target.bandwidth_limit);
}
// The target stays in `targets_map`, so it keeps showing up in
// `bucket remote ls` while no client exists to replicate through it —
// replication then drops every object for this ARN. Without this the
// rejection (loopback endpoint, bad CA, unparseable URL) left no trace
// anywhere.
Err(err) => warn!(
bucket = %bucket,
arn = %target.arn,
endpoint = %target.endpoint,
error = %err,
"replication target client unavailable; objects for this ARN will not replicate"
),
}
}
targets_map.insert(bucket.to_string(), new_targets.targets.clone());
File diff suppressed because it is too large Load Diff
@@ -18,10 +18,11 @@ use http::HeaderMap;
use rustfs_filemeta::FileInfo;
use crate::config::com;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result};
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
use crate::storage_api_contracts::{
object::{DeletedObject, ObjectIO, ObjectOperations, ObjectToDelete},
object::{DeletedObject, HTTPPreconditions, ObjectIO, ObjectOperations, ObjectToDelete},
range::HTTPRangeSpec,
};
@@ -40,6 +41,21 @@ where
com::read_config(api, file).await
}
pub(crate) async fn read_config_with_metadata<S>(api: Arc<S>, file: &str, opts: &ObjectOptions) -> Result<(Vec<u8>, ObjectInfo)>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
com::read_config_with_metadata(api, file, opts).await
}
pub(crate) async fn save_config<S>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()>
where
S: ObjectIO<
@@ -55,6 +71,21 @@ where
com::save_config(api, file, data).await
}
pub(crate) async fn save_config_with_opts<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
com::save_config_with_opts(api, file, data, opts).await
}
pub(crate) async fn delete_config<S>(api: Arc<S>, file: &str) -> Result<()>
where
S: ObjectOperations<
@@ -68,3 +99,39 @@ where
{
com::delete_config(api, file).await
}
pub(crate) async fn delete_config_if_match<S>(api: Arc<S>, file: &str, etag: &str) -> Result<()>
where
S: ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
match api
.delete_object(
RUSTFS_META_BUCKET,
file,
ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(etag.to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
{
Ok(_) => Ok(()),
Err(err) => {
if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
Err(Error::ConfigNotFound)
} else {
Err(err)
}
}
}
}
File diff suppressed because it is too large Load Diff
@@ -17,6 +17,7 @@ pub mod bucket_lifecycle_ops;
mod config_boundary;
pub mod core;
pub mod evaluator;
pub mod manual_transition_job;
mod metadata_boundary;
mod object_lock_boundary;
pub use self::core as lifecycle;
@@ -33,7 +33,6 @@ use rustfs_filemeta::FileInfo;
pub const DEFAULT_FREE_VERSION_RECOVERY_LIMIT: usize = 1_000;
const DEFAULT_FREE_VERSION_RECOVERY_SCAN_LIMIT: usize = 10_000;
const BACKGROUND_WALKDIR_TIMEOUT: Duration = Duration::from_secs(60);
#[cfg(not(test))]
const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(5);
#[cfg(test)]
@@ -42,6 +41,21 @@ const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_millis(100);
type ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, crate::error::Error>;
type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
fn recovery_walk_options(limit: usize, marker: Option<String>) -> WalkOptions {
WalkOptions {
include_free_versions: true,
limit,
marker,
// Total walk time scales with bucket size, so it is left unbounded
// (Duration::ZERO disables the wall-clock budget). Per-call progress
// stalls stay bounded by the drive-level stall budget inherited from
// `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS`.
walkdir_timeout: Some(Duration::ZERO),
walkdir_stall_timeout: None,
..Default::default()
}
}
#[cfg(test)]
pub(super) enum RecoveryWalkTestAction {
SendItemsThenError(Vec<ObjectInfo>, crate::error::Error),
@@ -406,20 +420,8 @@ pub(super) async fn list_tier_free_versions(
}
}
api.walk(
cancel,
&bucket_name,
"",
tx,
WalkOptions {
include_free_versions: true,
limit: walk_scan_limit,
marker: object_marker,
..Default::default()
}
.with_walkdir_timeouts(BACKGROUND_WALKDIR_TIMEOUT),
)
.await
api.walk(cancel, &bucket_name, "", tx, recovery_walk_options(walk_scan_limit, object_marker))
.await
}
});
@@ -695,6 +697,16 @@ mod tests {
);
}
#[test]
fn recovery_walk_disables_total_timeout_and_inherits_stall_timeout() {
let opts = recovery_walk_options(123, Some("marker".to_string()));
assert_eq!(opts.limit, 123);
assert_eq!(opts.marker.as_deref(), Some("marker"));
assert_eq!(opts.walkdir_timeout, Some(Duration::ZERO));
assert_eq!(opts.walkdir_stall_timeout, None);
}
#[test]
fn scan_truncation_keeps_marker_after_nonrecoverable_window() {
let mut page = FreeVersionRecoveryPage {
+43 -7
View File
@@ -649,7 +649,7 @@ impl BucketMetadata {
Ok(())
}
fn default_timestamps(&mut self) {
pub(crate) fn default_timestamps(&mut self) {
if self.policy_config_updated_at == OffsetDateTime::UNIX_EPOCH {
self.policy_config_updated_at = self.created
}
@@ -737,6 +737,9 @@ impl BucketMetadata {
}
BUCKET_TAGGING_CONFIG => {
self.tagging_config_xml = data;
// Drop the parsed form (like lifecycle above) so clearing the
// payload can't leave stale parsed tags to be cached.
self.tagging_config = None;
self.tagging_config_updated_at = updated;
}
BUCKET_QUOTA_CONFIG_FILE => {
@@ -1093,16 +1096,25 @@ pub async fn load_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<Buc
}
pub async fn load_bucket_metadata_parse(api: Arc<ECStore>, bucket: &str, parse: bool) -> Result<BucketMetadata> {
let mut bm = match read_bucket_metadata(api.clone(), bucket).await {
Ok(res) => res,
Ok(load_bucket_metadata_parse_with_presence(api, bucket, parse).await?.0)
}
/// The returned `bool` reports whether the metadata was actually read from
/// persisted storage; `false` means no metadata exists for this bucket on this
/// store and the returned value is a fabricated in-memory default.
pub(crate) async fn load_bucket_metadata_parse_with_presence(
api: Arc<ECStore>,
bucket: &str,
parse: bool,
) -> Result<(BucketMetadata, bool)> {
let (mut bm, persisted) = match read_bucket_metadata(api.clone(), bucket).await {
Ok(res) => (res, true),
Err(err) => {
if err != Error::ConfigNotFound {
return Err(err);
}
// info!("bucketmeta {} not found with err {:?}, start to init ", bucket, &err);
BucketMetadata::new(bucket)
(BucketMetadata::new(bucket), false)
}
};
@@ -1112,7 +1124,7 @@ pub async fn load_bucket_metadata_parse(api: Arc<ECStore>, bucket: &str, parse:
bm.parse_all_configs()?;
}
Ok(bm)
Ok((bm, persisted))
}
async fn read_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketMetadata> {
@@ -1309,6 +1321,30 @@ mod test {
assert!(bm.lifecycle_config.is_none());
}
/// Companion to the lifecycle case above. `parse_all_configs` skips empty
/// XML rather than clearing, so without the explicit reset a cleared
/// tagging config would keep serving the previously parsed tags.
#[test]
fn tagging_update_config_clears_parsed_config_on_delete() {
let mut bm = BucketMetadata::new("test-bucket");
let tagging_xml = br#"<Tagging><TagSet><Tag><Key>env</Key><Value>prod</Value></Tag></TagSet></Tagging>"#;
bm.update_config(BUCKET_TAGGING_CONFIG, tagging_xml.to_vec())
.expect("tagging config should update");
bm.parse_all_configs().expect("tagging config should parse");
assert!(bm.tagging_config.is_some());
bm.update_config(BUCKET_TAGGING_CONFIG, Vec::new())
.expect("tagging config delete should update metadata");
assert!(bm.tagging_config_xml.is_empty());
assert!(bm.tagging_config.is_none());
// A re-parse must not resurrect them either.
bm.parse_all_configs().expect("cleared tagging should parse");
assert!(bm.tagging_config.is_none());
}
#[tokio::test]
async fn marshal_msg_complete_example() {
// Create a complete BucketMetadata with various configurations
+418 -32
View File
@@ -12,15 +12,17 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::metadata::{BucketMetadata, load_bucket_metadata};
use super::metadata::{BUCKET_TARGETS_FILE, BucketMetadata, load_bucket_metadata};
use super::quota::BucketQuota;
use super::target::BucketTargets;
use crate::bucket::bucket_target_sys::BucketTargetSys;
use crate::bucket::metadata::load_bucket_metadata_parse;
use crate::bucket::metadata::{load_bucket_metadata_parse, load_bucket_metadata_parse_with_presence};
use crate::bucket::utils::is_meta_bucketname;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result, is_err_bucket_not_found};
use crate::runtime::sources as runtime_sources;
use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
use crate::store::ECStore;
use futures::future::join_all;
use rustfs_common::heal_channel::HealOpts;
@@ -188,7 +190,7 @@ pub async fn get(bucket: &str) -> Result<Arc<BucketMetadata>> {
// instance cell is not initialized yet (early startup) they fall back to the
// ambient default — the single-instance legacy behavior.
fn bucket_metadata_sys_of(ctx: &crate::runtime::instance::InstanceContext) -> Result<Arc<RwLock<BucketMetadataSys>>> {
pub(crate) fn bucket_metadata_sys_of(ctx: &crate::runtime::instance::InstanceContext) -> Result<Arc<RwLock<BucketMetadataSys>>> {
if let Some(sys) = ctx.bucket_metadata_sys() {
return Ok(sys);
}
@@ -221,11 +223,64 @@ pub(crate) async fn remove_bucket_metadata_in(ctx: &crate::runtime::instance::In
pub async fn update(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let _targets_guard = if config_file == BUCKET_TARGETS_FILE {
Some(acquire_bucket_targets_transaction_lock(bucket).await?)
} else {
None
};
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update(bucket, config_file, data).await
}
pub async fn update_bucket_targets_under_transaction_lock(bucket: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update(bucket, BUCKET_TARGETS_FILE, data).await
}
/// Read-modify-write one bucket config file under the metadata system's
/// outer write guard.
///
/// `mutate` sees the freshly loaded on-disk metadata and returns the
/// replacement payload for `config_file` (empty clears it, like
/// [`delete`]). Both the read and the persisted write happen inside the
/// same guard that [`update`] uses, so within this process the rewrite can
/// neither clobber a concurrent update to another config file nor lose a
/// concurrent write to the same one — unlike caching a mutated clone of
/// previously read metadata.
///
/// This guard is process-local. Writers on other nodes still race, exactly
/// as they do for [`update`]: each rewrites the whole metadata file, so the
/// later save wins. What this narrows is the window — from "as stale as the
/// local cache" down to a single metadata read plus write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let _targets_guard = if config_file == BUCKET_TARGETS_FILE {
Some(acquire_bucket_targets_transaction_lock(bucket).await?)
} else {
None
};
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update_config_with(bucket, config_file, mutate).await
}
pub async fn acquire_bucket_targets_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let api = bucket_meta_sys_lock.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_targets_transaction_lock_key(bucket))
.await?;
Ok(lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout()).await?)
}
fn bucket_targets_transaction_lock_key(bucket: &str) -> String {
format!("bucket-targets/{bucket}/transaction.lock")
}
pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
@@ -396,9 +451,23 @@ pub async fn list_bucket_targets(bucket: &str) -> Result<BucketTargets> {
bucket_meta_sys.get_bucket_targets_config(bucket).await
}
/// Bound and lifetime of the negative cache for buckets with no persisted
/// metadata. Entries are invalidated the moment real metadata is cached, so
/// the TTL only bounds staleness for out-of-band creations whose reload
/// notification was lost; the capacity bounds memory under bogus-name floods.
const ABSENT_BUCKET_METADATA_TTL: Duration = Duration::from_secs(30);
const ABSENT_BUCKET_METADATA_MAX_ENTRIES: u64 = 10_000;
#[derive(Debug)]
pub struct BucketMetadataSys {
metadata_map: RwLock<HashMap<String, Arc<BucketMetadata>>>,
/// Buckets recently observed to have no persisted metadata. Serving the
/// fabricated default from here (instead of re-reading disk) keeps the
/// per-request cost of repeated lookups for such names bounded — without
/// this, every request naming a nonexistent bucket pays a namespace-lock
/// acquisition plus a full erasure-set metadata fanout (reachable
/// pre-auth via CORS preflight, and per-key in DeleteObjects).
absent_metadata: moka::future::Cache<String, ()>,
api: Arc<ECStore>,
initialized: RwLock<bool>,
}
@@ -407,6 +476,10 @@ impl BucketMetadataSys {
pub fn new(api: Arc<ECStore>) -> Self {
Self {
metadata_map: RwLock::new(HashMap::new()),
absent_metadata: moka::future::Cache::builder()
.max_capacity(ABSENT_BUCKET_METADATA_MAX_ENTRIES)
.time_to_live(ABSENT_BUCKET_METADATA_TTL)
.build(),
api,
initialized: RwLock::new(false),
}
@@ -461,7 +534,7 @@ impl BucketMetadataSys {
},
)
.await;
load_bucket_metadata(self.api.clone(), bucket.as_str()).await
load_bucket_metadata_parse_with_presence(self.api.clone(), bucket.as_str(), true).await
});
}
@@ -469,9 +542,24 @@ impl BucketMetadataSys {
for (idx, res) in results.into_iter().enumerate() {
match res {
Ok(res) => {
Ok((bm, persisted)) => {
if let Some(bucket) = buckets.get(idx) {
self.set(bucket.clone(), Arc::new(res)).await;
if persisted {
self.set(bucket.clone(), Arc::new(bm)).await;
} else {
// A fabricated default (no persisted metadata
// readable right now) must never REPLACE an
// existing entry: the periodic refresh would
// otherwise downgrade a lock-enabled bucket to an
// authoritative "no lock" default on a transient
// ConfigNotFound, disabling the object-lock
// delete gate and wiping its target/durability
// sync state. Insert-if-vacant keeps the startup
// behavior for legacy buckets without a metadata
// file, atomically under the map write lock.
let mut map = self.metadata_map.write().await;
map.entry(bucket.clone()).or_insert_with(|| Arc::new(bm));
}
}
}
Err(e) => {
@@ -502,6 +590,8 @@ impl BucketMetadataSys {
let mut map = self.metadata_map.write().await;
map.insert(bucket.clone(), bm.clone());
drop(map);
// Real metadata supersedes any recorded absence immediately.
self.absent_metadata.invalidate(&bucket).await;
sync_bucket_target_sys(&bucket, &bm).await;
sync_bucket_durability(&bucket, &bm);
}
@@ -542,24 +632,7 @@ impl BucketMetadataSys {
return Err(Error::other("errServerNotInitialized"));
};
if is_meta_bucketname(bucket) {
return Err(Error::other("errInvalidArgument"));
}
let mut bm = match load_bucket_metadata_parse(store, bucket, parse).await {
Ok(res) => res,
Err(err) => {
if !runtime_sources::setup_is_erasure().await
&& !runtime_sources::setup_is_dist_erasure().await
&& is_err_bucket_not_found(&err)
{
BucketMetadata::new(bucket)
} else {
error!("load bucket metadata failed: {}", err);
return Err(err);
}
}
};
let mut bm = Self::load_bucket_metadata_for_update(store, bucket, parse).await?;
let updated = bm.update_config(config_file, data)?;
@@ -568,6 +641,49 @@ impl BucketMetadataSys {
Ok(updated)
}
/// See the free [`update_config_with`]: same load-mutate-persist cycle as
/// [`Self::update`], with the payload computed from the loaded metadata
/// instead of supplied up front. Loads through this system's own store so
/// the read and the persisted write target the same instance.
async fn update_config_with<F>(&mut self, bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let mut bm = Self::load_bucket_metadata_for_update(self.api.clone(), bucket, true).await?;
let data = mutate(&bm)?;
let updated = bm.update_config(config_file, data)?;
self.save(bm).await?;
Ok(updated)
}
/// Load a bucket's on-disk metadata as the base of a config rewrite.
/// Outside erasure setups a missing metadata file degrades to a fresh
/// default (legacy buckets without one); erasure setups fail instead of
/// fabricating state that a quorum may still hold.
async fn load_bucket_metadata_for_update(store: Arc<ECStore>, bucket: &str, parse: bool) -> Result<BucketMetadata> {
if is_meta_bucketname(bucket) {
return Err(Error::other("errInvalidArgument"));
}
match load_bucket_metadata_parse(store, bucket, parse).await {
Ok(res) => Ok(res),
Err(err) => {
if !runtime_sources::setup_is_erasure().await
&& !runtime_sources::setup_is_dist_erasure().await
&& is_err_bucket_not_found(&err)
{
Ok(BucketMetadata::new(bucket))
} else {
error!("load bucket metadata failed: {}", err);
Err(err)
}
}
}
}
async fn save(&self, bm: BucketMetadata) -> Result<()> {
if is_meta_bucketname(&bm.name) {
return Err(Error::other("errInvalidArgument"));
@@ -604,13 +720,22 @@ impl BucketMetadataSys {
pub async fn get_config(&self, bucket: &str) -> Result<(Arc<BucketMetadata>, bool)> {
let has_bm = {
let map = self.metadata_map.read().await;
map.get(&bucket.to_string()).cloned()
map.get(bucket).cloned()
};
if let Some(bm) = has_bm {
Ok((bm, false))
} else {
let bm = match load_bucket_metadata(self.api.clone(), bucket).await {
// A recent lookup already established there is no persisted
// metadata: serve the fabricated default without another
// namespace-lock + erasure-set fanout.
if self.absent_metadata.get(bucket).await.is_some() {
let mut bm = BucketMetadata::new(bucket);
bm.default_timestamps();
return Ok((Arc::new(bm), true));
}
let (bm, persisted) = match load_bucket_metadata_parse_with_presence(self.api.clone(), bucket, true).await {
Ok(res) => res,
Err(err) => {
return if *self.initialized.read().await {
@@ -621,13 +746,27 @@ impl BucketMetadataSys {
}
};
let mut map = self.metadata_map.write().await;
let bm = Arc::new(bm);
map.insert(bucket.to_string(), bm.clone());
drop(map);
sync_bucket_target_sys(bucket, &bm).await;
sync_bucket_durability(bucket, &bm);
// This lazy path caches only metadata that actually exists on
// this store. A fabricated default must not enter the map:
// `get()` is map-only and fail-closed — the object-lock delete
// gate (`object_lock_delete_check_required`) skips its per-object
// protection stat exactly when the map serves metadata saying the
// bucket has no Object Lock, so caching a fabricated default here
// would turn a metadata miss into an authoritative "no lock"
// answer. (Startup `concurrent_load` still caches fabricated
// defaults for buckets listed on disk — legacy buckets without a
// metadata file — but never lets one replace an existing entry.)
if persisted {
let mut map = self.metadata_map.write().await;
map.insert(bucket.to_string(), bm.clone());
drop(map);
sync_bucket_target_sys(bucket, &bm).await;
sync_bucket_durability(bucket, &bm);
} else {
self.absent_metadata.insert(bucket.to_string(), ()).await;
}
Ok((bm, true))
}
@@ -657,6 +796,8 @@ impl BucketMetadataSys {
if let Some(config) = &bm.policy_config {
Ok((config.clone(), bm.policy_config_updated_at))
} else if !bm.policy_config_json.is_empty() {
Ok((serde_json::from_slice(&bm.policy_config_json)?, bm.policy_config_updated_at))
} else {
Err(Error::ConfigNotFound)
}
@@ -847,13 +988,258 @@ impl BucketMetadataSys {
}
}
/// Test-only fixture shared with sibling modules (e.g. the quota checker
/// tests): a 4-disk `ECStore` on an isolated instance context, so tests
/// exercising the metadata system never touch ambient process state.
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
use crate::disk::endpoint::Endpoint;
use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
use crate::runtime::instance::InstanceContext;
use crate::store::init_local_disks_with_instance_ctx;
pub(crate) async fn isolated_store_over_temp_disks() -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let mut dirs = Vec::with_capacity(4);
let mut endpoints = Vec::with_capacity(4);
for disk_idx in 0..4 {
let dir = tempfile::tempdir().expect("tempdir should be created");
let mut endpoint =
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_idx);
dirs.push(dir);
endpoints.push(endpoint);
}
let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 4,
endpoints: Endpoints::from(endpoints),
cmd_line: "metadata-sys-cache-test".to_string(),
platform: "test".to_string(),
}]);
let instance_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone())
.await
.expect("local disks should initialize");
let ecstore = ECStore::new_with_instance_ctx(
"127.0.0.1:0".parse().expect("test address"),
endpoint_pools,
CancellationToken::new(),
instance_ctx,
)
.await
.expect("ECStore should initialize");
(dirs, ecstore)
}
}
#[cfg(test)]
mod tests {
use super::test_support::isolated_store_over_temp_disks;
use super::*;
use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
use serial_test::serial;
use tokio::time::timeout;
/// Pins the fail-closed caching contract of the lazy `get_config` path
/// and the refresh no-replace rule: fabricated defaults are returned but
/// never served by the map-only `get()`, persisted metadata is cached on
/// lazy load (superseding a recorded absence), and a refresh-load miss
/// never replaces an existing entry.
#[tokio::test]
async fn get_config_never_caches_fabricated_defaults_as_authoritative() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
// (a) Miss: the fabricated default is returned but not cached.
let (bm, _) = sys
.get_config("absent-bucket")
.await
.expect("fabricated default should be returned");
assert!(bm.object_lock_config_xml.is_empty());
assert!(
sys.get("absent-bucket").await.is_err(),
"a fabricated default must never be served by the map-only get()"
);
// The repeat lookup is served from the negative cache, same answer.
let (bm, _) = sys
.get_config("absent-bucket")
.await
.expect("negative-cached default should be returned");
assert!(bm.object_lock_config_xml.is_empty());
assert!(sys.get("absent-bucket").await.is_err());
// (b) Persisting real metadata supersedes the recorded absence, and a
// lazy reload after a map wipe re-caches it.
let mut persisted = BucketMetadata::new("absent-bucket");
persisted.policy_config_json = b"persisted-marker".to_vec();
sys.persist_and_set(persisted).await.expect("metadata should persist");
sys.metadata_map.write().await.clear();
let _ = sys
.get_config("absent-bucket")
.await
.expect("persisted metadata should lazily reload");
let cached = sys
.get("absent-bucket")
.await
.expect("lazily loaded persisted metadata must be cached");
assert_eq!(cached.policy_config_json, b"persisted-marker".to_vec());
// (c) A refresh-load miss (no persisted metadata readable) must not
// replace an existing entry.
let mut kept = BucketMetadata::new("kept-bucket");
kept.policy_config_json = b"kept-marker".to_vec();
sys.set("kept-bucket".to_string(), Arc::new(kept)).await;
let mut failed = HashSet::new();
let refresh_targets = vec!["kept-bucket".to_string()];
sys.concurrent_load(&refresh_targets, &mut failed).await;
let kept = sys
.get("kept-bucket")
.await
.expect("existing entry must survive a refresh miss");
assert_eq!(
kept.policy_config_json,
b"kept-marker".to_vec(),
"a fabricated refresh default must not replace real metadata"
);
}
#[tokio::test]
async fn get_bucket_policy_rejects_malformed_cached_policy() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let mut metadata = BucketMetadata::new("malformed-policy");
metadata.policy_config_json = b"{".to_vec();
sys.set("malformed-policy".to_string(), Arc::new(metadata)).await;
let err = sys
.get_bucket_policy("malformed-policy")
.await
.expect_err("malformed persisted policy must not be treated as missing");
assert!(matches!(err, Error::Io(_)), "malformed persisted policy must surface its parse failure");
}
/// A tagging rewrite through `update_config_with` (the Swift metadata
/// POST path) is persisted: it survives a metadata reload from disk, and
/// an emptied rewrite clears the config in the cached copy too instead of
/// leaving stale parsed tags behind.
#[tokio::test]
async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let mut sys = BucketMetadataSys::new(ecstore);
let bucket = "swift-tagging-bucket";
sys.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
let tagging = Tagging {
tag_set: vec![Tag {
key: Some("swift-meta-color".to_string()),
value: Some("blue".to_string()),
}],
};
let xml = crate::bucket::utils::serialize::<Tagging>(&tagging).expect("tagging should serialize");
sys.update_config_with(bucket, BUCKET_TAGGING_CONFIG, move |bm| {
assert!(bm.tagging_config.is_none(), "rewrite must see the on-disk state");
Ok(xml)
})
.await
.expect("tagging rewrite should persist");
// Simulate the disk-truth reload that used to lose Swift writes: drop
// the cached entry and lazily re-load from the metadata file.
sys.metadata_map.write().await.clear();
let (tags, _) = sys
.get_tagging_config(bucket)
.await
.expect("tagging must survive a reload from disk");
assert_eq!(tags.tag_set.len(), 1);
assert_eq!(tags.tag_set[0].key.as_deref(), Some("swift-meta-color"));
assert_eq!(tags.tag_set[0].value.as_deref(), Some("blue"));
// An emptied rewrite clears the config everywhere.
sys.update_config_with(bucket, BUCKET_TAGGING_CONFIG, |bm| {
assert!(bm.tagging_config.is_some(), "rewrite must see the persisted tags");
Ok(Vec::new())
})
.await
.expect("clearing rewrite should persist");
assert_eq!(
sys.get_tagging_config(bucket).await.unwrap_err(),
Error::ConfigNotFound,
"cleared tagging must not be served from the cache"
);
sys.metadata_map.write().await.clear();
assert_eq!(
sys.get_tagging_config(bucket).await.unwrap_err(),
Error::ConfigNotFound,
"cleared tagging must not reappear after a reload from disk"
);
}
/// The load and the persisted write share one write guard, so concurrent
/// rewrites of the same config compose instead of clobbering each other.
/// Moving the load outside that guard loses all but the last tag.
#[tokio::test]
async fn concurrent_update_config_with_calls_do_not_lose_writes() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
let bucket = "swift-tagging-concurrent";
sys.read()
.await
.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
const WRITERS: usize = 8;
let mut handles = Vec::with_capacity(WRITERS);
for idx in 0..WRITERS {
let sys = sys.clone();
handles.push(tokio::spawn(async move {
sys.write()
.await
.update_config_with(bucket, BUCKET_TAGGING_CONFIG, move |bm| {
// Each writer merges its own tag onto whatever is
// currently persisted — the Swift rewrite shape.
let mut tagging = bm.tagging_config.clone().unwrap_or_else(|| Tagging { tag_set: vec![] });
tagging.tag_set.push(Tag {
key: Some(format!("swift-meta-key{idx}")),
value: Some(idx.to_string()),
});
crate::bucket::utils::serialize::<Tagging>(&tagging).map_err(|e| Error::other(e.to_string()))
})
.await
}));
}
for handle in handles {
handle
.await
.expect("writer task should join")
.expect("rewrite should persist");
}
let (tags, _) = sys
.read()
.await
.get_tagging_config(bucket)
.await
.expect("tagging should be readable");
assert_eq!(tags.tag_set.len(), WRITERS, "every concurrent rewrite must survive: {tags:?}");
}
fn target(bucket: &str, id: &str) -> BucketTarget {
BucketTarget {
source_bucket: bucket.to_string(),
+15 -3
View File
@@ -538,10 +538,12 @@ mod tests {
use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
use crate::object_api::{ObjectOptions, PutObjReader};
use crate::runtime::instance::InstanceContext;
use crate::storage_api_contracts::bucket::{BucketOperations, BucketOptions, MakeBucketOptions};
use crate::storage_api_contracts::object::{ObjectIO, ObjectOperations};
use crate::store::{ECStore, init_local_disks};
use crate::store::{ECStore, init_local_disks_with_instance_ctx};
use rustfs_utils::path::SLASH_SEPARATOR;
use std::sync::Arc;
use tokio::fs;
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
@@ -570,10 +572,20 @@ mod tests {
cmd_line: "minio-migrate-test".to_string(),
platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
}]);
init_local_disks(endpoint_pools.clone()).await.unwrap();
let ecstore = ECStore::new("127.0.0.1:0".parse().unwrap(), endpoint_pools, CancellationToken::new())
// Isolated instance context: this test deletes its disks at the end,
// and dead entries in the shared registry break other cached envs.
let instance_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone())
.await
.unwrap();
let ecstore = ECStore::new_with_instance_ctx(
"127.0.0.1:0".parse().unwrap(),
endpoint_pools,
CancellationToken::new(),
instance_ctx,
)
.await
.unwrap();
let existing: Vec<String> = ecstore
.list_bucket(&BucketOptions {
no_metadata: true,
+102 -11
View File
@@ -15,23 +15,26 @@
use super::metadata_sys::get_bucket_metadata_sys;
use crate::error::{Result, StorageError};
use rustfs_policy::policy::{BucketPolicy, BucketPolicyArgs};
use tracing::info;
pub struct PolicySys {}
impl PolicySys {
pub async fn is_allowed(args: &BucketPolicyArgs<'_>) -> bool {
match Self::get(args.bucket).await {
Ok(cfg) => return cfg.is_allowed(args).await,
Err(err) => {
if err != StorageError::ConfigNotFound {
info!("config get err {:?}", err);
}
}
}
args.is_owner
matches!(Self::try_is_allowed(args).await, Ok(true))
}
pub async fn try_is_allowed(args: &BucketPolicyArgs<'_>) -> Result<bool> {
Self::is_allowed_with_policy(args, Self::get(args.bucket).await).await
}
async fn is_allowed_with_policy(args: &BucketPolicyArgs<'_>, policy: Result<BucketPolicy>) -> Result<bool> {
match policy {
Ok(policy) => Ok(policy.is_allowed(args).await),
Err(StorageError::ConfigNotFound) => Ok(args.is_owner),
Err(err) => Err(err),
}
}
pub async fn get(bucket: &str) -> Result<BucketPolicy> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -41,3 +44,91 @@ impl PolicySys {
Ok(cfg)
}
}
#[cfg(test)]
mod tests {
use super::{PolicySys, StorageError};
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_policy::policy::{BucketPolicy, BucketPolicyArgs};
use std::collections::HashMap;
fn args<'a>(
is_owner: bool,
groups: &'a Option<Vec<String>>,
conditions: &'a HashMap<String, Vec<String>>,
) -> BucketPolicyArgs<'a> {
BucketPolicyArgs {
bucket: "bucket",
action: Action::S3Action(S3Action::GetObjectAction),
is_owner,
account: "account",
groups,
conditions,
object: "object",
}
}
#[tokio::test]
async fn missing_policy_preserves_owner_and_iam_fallback_semantics() {
let groups = None;
let conditions = HashMap::new();
assert!(
PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Err(StorageError::ConfigNotFound),)
.await
.expect("missing policy should preserve owner access")
);
assert!(
!PolicySys::is_allowed_with_policy(&args(false, &groups, &conditions), Err(StorageError::ConfigNotFound),)
.await
.expect("missing policy should defer non-owner access to IAM")
);
}
#[tokio::test]
async fn policy_load_failures_propagate() {
let groups = None;
let conditions = HashMap::new();
for (failure, expected_message) in [
(StorageError::Io(std::io::Error::other("policy read failed")), "policy read failed"),
(
StorageError::other("bucket metadata sys not initialized for this instance"),
"bucket metadata sys not initialized for this instance",
),
] {
let result = PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Err(failure)).await;
assert!(
matches!(result, Err(StorageError::Io(ref err)) if err.to_string().contains(expected_message)),
"policy I/O and uninitialized metadata failures must propagate instead of granting owner access"
);
}
}
#[tokio::test]
async fn explicit_bucket_deny_precedes_iam_allow() {
let groups = None;
let conditions = HashMap::new();
let policy: BucketPolicy = serde_json::from_str(
r#"{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Deny",
"Principal":{"AWS":"*"},
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::bucket/*"]
}]
}"#,
)
.expect("deny policy should parse");
let bucket_allowed = PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Ok(policy))
.await
.expect("loaded bucket policy should evaluate");
let iam_allowed = true;
let request_allowed = bucket_allowed && iam_allowed;
assert!(iam_allowed, "test precondition: IAM grants the action");
assert!(!bucket_allowed, "test precondition: bucket policy explicitly denies the action");
assert!(!request_allowed, "explicit bucket Deny must reject before IAM Allow fallback");
}
}
+56 -4
View File
@@ -118,11 +118,17 @@ impl QuotaChecker {
}
pub async fn get_quota_config(&self, bucket: &str) -> Result<BucketQuota, QuotaError> {
let meta = self
// `get_config`, not the map-only `get()`: a bucket with no persisted
// metadata must resolve to the fabricated default (no quota
// configured) so the admission check passes and the request reaches
// the NoSuchBucket answer — a map-only miss would fail every such
// PUT closed with 503 before the 404 could be produced. Real read
// faults still surface as errors and keep the fail-closed behavior.
let (meta, _) = self
.metadata_sys
.read()
.await
.get(bucket)
.get_config(bucket)
.await
.map_err(QuotaError::StorageError)?;
@@ -161,7 +167,7 @@ impl QuotaChecker {
}
let quota = self.get_quota_config(bucket).await?;
let current_usage = self.get_real_time_usage(bucket).await.unwrap_or(0);
let current_usage = self.get_real_time_usage(bucket).await?;
Ok((quota, Some(current_usage)))
}
@@ -171,13 +177,59 @@ impl QuotaChecker {
}
pub async fn get_real_time_usage(&self, bucket: &str) -> Result<u64, QuotaError> {
Ok(get_bucket_usage_memory(bucket).await.unwrap_or(0))
get_bucket_usage_memory(bucket)
.await
.ok_or_else(|| QuotaError::UsageUnavailable {
bucket: bucket.to_string(),
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use serial_test::serial;
use uuid::Uuid;
/// Regression (PR #5307 / s3-tests `test_100_continue_error_retry`): a
/// bucket with no persisted metadata has no quota, so the admission check
/// must pass and let the request reach its NoSuchBucket answer. With the
/// map-only `get()` this failed closed as a retryable 503 on every PUT to
/// a nonexistent bucket.
#[tokio::test]
async fn quota_check_allows_bucket_without_persisted_metadata() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
let checker = QuotaChecker::new(sys);
let result = checker
.check_quota("no-such-bucket", QuotaOperation::PutObject, 1024)
.await
.expect("a bucket with no persisted metadata has no quota and must not fail the check");
assert!(result.allowed);
assert_eq!(result.quota_limit, None);
}
#[tokio::test]
#[serial]
async fn quota_usage_rejects_an_unknown_mutation_baseline() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
let checker = QuotaChecker::new(sys);
let bucket = format!("quota-unknown-{}", Uuid::new_v4().simple());
crate::data_usage::record_bucket_object_write_memory(&bucket, None, 42).await;
let result = checker.get_real_time_usage(&bucket).await;
crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None)
.await
.expect("test usage cache cleanup should succeed");
assert!(
matches!(result, Err(QuotaError::UsageUnavailable { bucket: failed_bucket }) if failed_bucket == bucket),
"quota decisions must fail closed without an authoritative usage baseline"
);
}
#[tokio::test]
async fn test_quota_check_no_limit() {
+3 -1
View File
@@ -110,6 +110,8 @@ pub enum QuotaError {
QuotaExceeded { current: u64, limit: u64, operation: u64 },
#[error("Quota configuration not found for bucket: {bucket}")]
ConfigNotFound { bucket: String },
#[error("Authoritative data usage is unavailable for bucket: {bucket}")]
UsageUnavailable { bucket: String },
#[error("Invalid quota configuration: {reason}")]
InvalidConfig { reason: String },
#[error("Storage error: {0}")]
@@ -155,7 +157,7 @@ impl QuotaErrorResponse {
request_id: request_id.to_string(),
host_id: host_id.to_string(),
},
QuotaError::StorageError(_) => Self {
QuotaError::UsageUnavailable { .. } | QuotaError::StorageError(_) => Self {
code: QUOTA_INTERNAL_ERROR_CODE.to_string(),
message: quota_error.to_string(),
resource: QUOTA_API_PATH.to_string(),
+5 -5
View File
@@ -52,9 +52,9 @@ pub use replication_config_boundary::{
pub(crate) use replication_filemeta_boundary::ReplicateTargetDecision;
pub(crate) use replication_filemeta_boundary::version_purge_statuses_map;
pub use replication_filemeta_boundary::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationState, ReplicationStatusType, ReplicationType,
VersionPurgeStatusType, replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map,
version_purge_status_to_filemeta,
MrfOpKind, MrfReplicateEntry, REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationState,
ReplicationStatusType, ReplicationType, VersionPurgeStatusType, replication_state_to_filemeta,
replication_status_to_filemeta, replication_statuses_map, version_purge_status_to_filemeta,
};
pub(crate) use replication_filemeta_boundary::{
replication_state_from_filemeta, replication_status_from_filemeta, version_purge_status_from_filemeta,
@@ -69,8 +69,8 @@ pub use replication_object_decision_boundary::{
should_use_existing_delete_replication_source,
};
pub use replication_pool::{
DynReplicationPool, ReplicationPoolTrait, get_global_replication_pool, get_global_replication_stats,
init_background_replication,
DurableMrfBacklog, DynReplicationPool, ReplicationPoolTrait, get_global_replication_pool, get_global_replication_stats,
init_background_replication, read_durable_mrf_backlog, resync_start_conflict_id,
};
pub use replication_queue_boundary::{
DeletedObjectReplicationInfo, ReplicationHealQueueResult, ReplicationOperation, ReplicationPriority,
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
pub(crate) use rustfs_replication::{
REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos,
ReplicatedTargetInfo, ReplicationAction, ReplicationWorkerOperation, ResyncDecision, get_replication_state,
@@ -53,6 +53,10 @@ impl ReplicationMetadataStore {
format!("{REPLICATION_DIR}/{bucket}/{arn}")
}
pub(crate) fn resync_admission_lock_key(bucket: &str) -> String {
format!("{REPLICATION_DIR}/{bucket}/admission.lock")
}
pub(crate) fn bucket_resync_dir_path(bucket: &str) -> String {
path_join_buf(&[BUCKET_META_PREFIX, bucket, REPLICATION_DIR])
}
@@ -73,6 +77,10 @@ mod tests {
ReplicationMetadataStore::resync_lock_key("bucket-a", "arn-a"),
".replication/bucket-a/arn-a"
);
assert_eq!(
ReplicationMetadataStore::resync_admission_lock_key("bucket-a"),
".replication/bucket-a/admission.lock"
);
assert_eq!(
ReplicationMetadataStore::bucket_resync_dir_path("bucket-a"),
"buckets/bucket-a/.replication"
@@ -68,6 +68,51 @@ const EVENT_REPLICATION_RESYNC_LOAD_SKIPPED: &str = "replication_resync_load_ski
const EVENT_REPLICATION_RESYNC_RECOVERED: &str = "replication_resync_recovered";
const EVENT_REPLICATION_MRF_QUEUE_UNAVAILABLE: &str = "replication_mrf_queue_unavailable";
#[derive(Debug, Default)]
pub struct DurableMrfBacklog {
pub available: bool,
pub entries: Vec<MrfReplicateEntry>,
}
fn durable_mrf_backlog_from_read(result: Result<Vec<u8>, EcstoreError>) -> DurableMrfBacklog {
match result {
Ok(data) => match decode_mrf_file(&data) {
Ok(entries) if entries.iter().all(|entry| entry.size >= 0) => DurableMrfBacklog {
available: true,
entries,
},
Ok(_) | Err(_) => DurableMrfBacklog::default(),
},
Err(EcstoreError::ConfigNotFound) => DurableMrfBacklog {
available: true,
entries: Vec::new(),
},
Err(_) => DurableMrfBacklog::default(),
}
}
pub async fn read_durable_mrf_backlog<S: ReplicationObjectIO>(storage: Arc<S>) -> DurableMrfBacklog {
durable_mrf_backlog_from_read(ReplicationConfigStore::read(storage, ReplicationMetadataStore::MRF_REPLICATION_FILE).await)
}
#[derive(Debug, thiserror::Error)]
#[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
struct ResyncActiveConflictError {
bucket: String,
arn: String,
active_resync_id: String,
}
pub fn resync_start_conflict_id(error: &EcstoreError) -> Option<&str> {
match error {
EcstoreError::Io(io_error) => io_error
.get_ref()?
.downcast_ref::<ResyncActiveConflictError>()
.map(|conflict| conflict.active_resync_id.as_str()),
_ => None,
}
}
/// Main replication pool structure
#[derive(Debug)]
pub struct ReplicationPool<S: ReplicationStorage> {
@@ -948,47 +993,123 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
}
pub async fn start_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<(), EcstoreError> {
let now = OffsetDateTime::now_utc();
let bucket_status = {
let mut status_map = self.resyncer.status_map.write().await;
let bucket_status = status_map.entry(opts.bucket.clone()).or_insert_with(|| {
let mut status = BucketReplicationResyncStatus::new();
status.id = 0;
status
});
let new_run = self.clone().admit_bucket_resync(opts.clone()).await?;
self.activate_bucket_resync(opts, !new_run).await
}
bucket_status.last_update = Some(now);
bucket_status.targets_map.insert(
opts.arn.clone(),
TargetReplicationResyncStatus {
start_time: Some(now),
last_update: Some(now),
resync_id: opts.resync_id.clone(),
resync_before_date: opts.resync_before,
resync_status: ResyncStatusType::ResyncPending,
failed_size: 0,
failed_count: 0,
replicated_size: 0,
replicated_count: 0,
bucket: opts.bucket.clone(),
object: String::new(),
error: None,
},
);
pub async fn admit_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<bool, EcstoreError> {
tokio::spawn(async move { self.admit_bucket_resync_transaction(opts).await })
.await
.map_err(|error| EcstoreError::other(format!("replication resync admission task failed: {error}")))?
}
bucket_status.clone()
async fn admit_bucket_resync_transaction(self: Arc<Self>, opts: ResyncOpts) -> Result<bool, EcstoreError> {
let admission_lock_key = ReplicationMetadataStore::resync_admission_lock_key(&opts.bucket);
let admission_lock = self
.storage
.new_ns_lock(ReplicationMetadataStore::rustfs_meta_bucket(), &admission_lock_key)
.await?;
// Lock order: bucket resync admission lock -> resync status config-object lock.
let _admission_guard = match admission_lock.get_write_lock(ReplicationLockTiming::acquire_timeout()).await {
Ok(guard) => guard,
Err(lock_error) => {
if let Ok(status) = load_bucket_resync_metadata(&opts.bucket, self.storage.clone()).await {
self.resyncer.status_map.write().await.insert(opts.bucket.clone(), status);
}
return Err(EcstoreError::from(lock_error));
}
};
let mut bucket_status = load_bucket_resync_metadata(&opts.bucket, self.storage.clone()).await?;
if let Some(active) = bucket_status.targets_map.get(&opts.arn) {
if active.resync_id == opts.resync_id {
self.resyncer
.status_map
.write()
.await
.insert(opts.bucket.clone(), bucket_status);
return Ok(false);
}
if should_auto_resume_resync(active.resync_status) {
let active_resync_id = active.resync_id.clone();
self.resyncer
.status_map
.write()
.await
.insert(opts.bucket.clone(), bucket_status);
return Err(EcstoreError::other(ResyncActiveConflictError {
bucket: opts.bucket.clone(),
arn: opts.arn.clone(),
active_resync_id,
}));
}
}
let now = OffsetDateTime::now_utc();
bucket_status.last_update = Some(now);
bucket_status.targets_map.insert(
opts.arn.clone(),
TargetReplicationResyncStatus {
start_time: Some(now),
last_update: Some(now),
resync_id: opts.resync_id.clone(),
resync_before_date: opts.resync_before,
resync_status: ResyncStatusType::ResyncPending,
failed_size: 0,
failed_count: 0,
replicated_size: 0,
replicated_count: 0,
bucket: opts.bucket.clone(),
object: String::new(),
error: None,
},
);
save_resync_status(&opts.bucket, &bucket_status, self.storage.clone()).await?;
self.resyncer
.status_map
.write()
.await
.insert(opts.bucket.clone(), bucket_status);
Ok(true)
}
pub async fn activate_bucket_resync(self: Arc<Self>, opts: ResyncOpts, recovering: bool) -> Result<(), EcstoreError> {
let bucket_status = load_bucket_resync_metadata(&opts.bucket, self.storage.clone()).await?;
let Some(target_status) = bucket_status.targets_map.get(&opts.arn) else {
return Err(EcstoreError::other("replication resync admission is missing"));
};
if target_status.resync_id != opts.resync_id {
return Err(EcstoreError::other(ResyncActiveConflictError {
bucket: opts.bucket.clone(),
arn: opts.arn.clone(),
active_resync_id: target_status.resync_id.clone(),
}));
}
if !should_auto_resume_resync(target_status.resync_status) {
return Ok(());
}
self.resyncer
.status_map
.write()
.await
.insert(opts.bucket.clone(), bucket_status);
let resyncer = self.resyncer.clone();
let storage = self.storage.clone();
let cancel_token = CancellationToken::new();
resyncer.register_cancel_token(&opts, cancel_token.clone()).await;
tokio::spawn(async move {
Box::pin(resyncer.clone().resync_bucket(cancel_token, storage, false, opts.clone())).await;
resyncer.clear_cancel_token(&opts).await;
});
if resyncer.register_cancel_token(&opts, cancel_token.clone()).await {
tokio::spawn(async move {
Box::pin(
resyncer
.clone()
.resync_bucket(cancel_token, storage, recovering, opts.clone()),
)
.await;
resyncer.clear_cancel_token(&opts).await;
});
}
Ok(())
}
@@ -1141,9 +1262,10 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let resync = self.resyncer.clone();
let storage = self.storage.clone();
tokio::spawn(async move {
resync.register_cancel_token(&opts, ctx.clone()).await;
Box::pin(resync.clone().resync_bucket(ctx, storage, true, opts.clone())).await;
resync.clear_cancel_token(&opts).await;
if resync.register_cancel_token(&opts, ctx.clone()).await {
Box::pin(resync.clone().resync_bucket(ctx, storage, true, opts.clone())).await;
resync.clear_cancel_token(&opts).await;
}
});
}
@@ -1247,6 +1369,8 @@ pub trait ReplicationPoolTrait: std::fmt::Debug {
async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize);
async fn get_bucket_resync_status(&self, bucket: &str) -> Result<BucketReplicationResyncStatus, EcstoreError>;
async fn cancel_bucket_resync(&self, opts: ResyncOpts) -> Result<(), EcstoreError>;
async fn admit_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<bool, EcstoreError>;
async fn activate_bucket_resync(self: Arc<Self>, opts: ResyncOpts, recovering: bool) -> Result<(), EcstoreError>;
async fn start_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<(), EcstoreError>;
async fn init_resync(
self: Arc<Self>,
@@ -1290,6 +1414,14 @@ impl<S: ReplicationStorage> ReplicationPoolTrait for ReplicationPool<S> {
self.cancel_bucket_resync(opts).await
}
async fn admit_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<bool, EcstoreError> {
self.admit_bucket_resync(opts).await
}
async fn activate_bucket_resync(self: Arc<Self>, opts: ResyncOpts, recovering: bool) -> Result<(), EcstoreError> {
self.activate_bucket_resync(opts, recovering).await
}
async fn start_bucket_resync(self: Arc<Self>, opts: ResyncOpts) -> Result<(), EcstoreError> {
self.start_bucket_resync(opts).await
}
@@ -1553,7 +1685,9 @@ mod tests {
use std::collections::HashMap;
use std::fmt::{Debug, Formatter};
use std::io::Cursor;
use std::sync::atomic::AtomicUsize;
use std::sync::Mutex as StdMutex;
use std::sync::atomic::{AtomicBool, AtomicUsize};
use tokio::io::AsyncReadExt;
use tokio::sync::Notify;
use uuid::Uuid;
@@ -1562,10 +1696,16 @@ mod tests {
type TestObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, EcstoreError>;
struct LoadResyncSharedState {
data: Vec<u8>,
data: StdMutex<Vec<u8>>,
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
first_read_started: Notify,
delay_first_read: AtomicBool,
read_count: AtomicUsize,
write_count: AtomicUsize,
fail_next_write: AtomicBool,
block_next_write: AtomicBool,
write_started: Notify,
allow_write: Notify,
}
struct LoadResyncNodeStore {
@@ -1606,22 +1746,31 @@ mod tests {
_h: Self::HeaderMap,
_opts: &Self::ObjectOptions,
) -> Result<Self::GetObjectReader, Self::Error> {
if object != ReplicationMetadataStore::bucket_resync_file_path("load-resync-lock") {
if !object.ends_with("/.replication/resync.bin") {
return Err(EcstoreError::FileNotFound);
}
let read_index = self.shared.read_count.fetch_add(1, Ordering::SeqCst);
if read_index == 0 {
if read_index == 0 && self.shared.delay_first_read.load(Ordering::SeqCst) {
self.shared.first_read_started.notify_waiters();
tokio::time::sleep(Duration::from_millis(1_500)).await;
}
let data = self.shared.data.clone();
let data = self
.shared
.data
.lock()
.expect("test data lock should not be poisoned")
.clone();
if data.is_empty() {
return Err(EcstoreError::FileNotFound);
}
let size = i64::try_from(data.len()).expect("test metadata length should fit i64");
Ok(Self::GetObjectReader {
stream: Box::new(Cursor::new(data.clone())),
stream: Box::new(Cursor::new(data)),
object_info: ObjectInfo {
size: data.len() as i64,
actual_size: data.len() as i64,
size,
actual_size: size,
..Default::default()
},
buffered_body: None,
@@ -1633,9 +1782,20 @@ mod tests {
&self,
_bucket: &str,
_object: &str,
_data: &mut Self::PutObjectReader,
data: &mut Self::PutObjectReader,
_opts: &Self::ObjectOptions,
) -> Result<Self::ObjectInfo, Self::Error> {
if self.shared.fail_next_write.swap(false, Ordering::SeqCst) {
return Err(EcstoreError::Unexpected);
}
if self.shared.block_next_write.swap(false, Ordering::SeqCst) {
self.shared.write_started.notify_one();
self.shared.allow_write.notified().await;
}
let mut encoded = Vec::new();
data.stream.read_to_end(&mut encoded).await.map_err(EcstoreError::from)?;
*self.shared.data.lock().expect("test data lock should not be poisoned") = encoded;
self.shared.write_count.fetch_add(1, Ordering::SeqCst);
Ok(ObjectInfo::default())
}
}
@@ -1869,6 +2029,267 @@ mod tests {
encode_resync_file(&status).expect("test resync metadata should encode")
}
fn empty_resync_shared_state() -> Arc<LoadResyncSharedState> {
Arc::new(LoadResyncSharedState {
data: StdMutex::new(Vec::new()),
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
first_read_started: Notify::new(),
delay_first_read: AtomicBool::new(false),
read_count: AtomicUsize::new(0),
write_count: AtomicUsize::new(0),
fail_next_write: AtomicBool::new(false),
block_next_write: AtomicBool::new(false),
write_started: Notify::new(),
allow_write: Notify::new(),
})
}
async fn hold_resync_runtime_lock(
shared: &Arc<LoadResyncSharedState>,
bucket: &str,
arn: &str,
) -> rustfs_lock::NamespaceLockGuard {
let lock =
rustfs_lock::NamespaceLock::with_local_manager("resync-start-blocker".to_string(), shared.lock_manager.clone());
let lock = rustfs_lock::NamespaceLockWrapper::new(
lock,
rustfs_lock::ObjectKey::new(
ReplicationMetadataStore::rustfs_meta_bucket().to_string(),
ReplicationMetadataStore::resync_lock_key(bucket, arn),
),
"blocker".to_string(),
);
lock.get_write_lock(Duration::from_secs(1))
.await
.expect("test should hold the runtime resync lock")
}
fn test_resync_opts(bucket: &str, arn: &str, id: &str) -> ResyncOpts {
ResyncOpts {
bucket: bucket.to_string(),
arn: arn.to_string(),
resync_id: id.to_string(),
resync_before: Some(OffsetDateTime::UNIX_EPOCH),
}
}
#[tokio::test]
async fn concurrent_resync_starts_accept_one_id_and_reject_the_other() {
let shared = empty_resync_shared_state();
let first_pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let second_pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-b", shared.clone()))).await;
let _runtime_guard = hold_resync_runtime_lock(&shared, "atomic-start", "arn:test").await;
let first = first_pool
.clone()
.start_bucket_resync(test_resync_opts("atomic-start", "arn:test", "run-a"));
let second = second_pool
.clone()
.start_bucket_resync(test_resync_opts("atomic-start", "arn:test", "run-b"));
let (first, second) = tokio::join!(first, second);
let (accepted_id, conflict) = match (first, second) {
(Ok(()), Err(conflict)) => ("run-a", conflict),
(Err(conflict), Ok(())) => ("run-b", conflict),
outcome => panic!("exactly one concurrent start should be accepted: {outcome:?}"),
};
assert_eq!(resync_start_conflict_id(&conflict), Some(accepted_id));
let persisted = decode_resync_file(&shared.data.lock().expect("test data lock should not be poisoned"))
.expect("accepted status should be persisted");
assert_eq!(persisted.targets_map["arn:test"].resync_id, accepted_id);
assert_eq!(persisted.targets_map["arn:test"].resync_status, ResyncStatusType::ResyncPending);
assert_eq!(
first_pool.resyncer.status_map.read().await["atomic-start"].targets_map["arn:test"].resync_id,
accepted_id
);
assert_eq!(
second_pool.resyncer.status_map.read().await["atomic-start"].targets_map["arn:test"].resync_id,
accepted_id
);
}
#[tokio::test]
async fn same_resync_id_retry_is_idempotent_without_rewriting_status() {
let shared = empty_resync_shared_state();
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let _runtime_guard = hold_resync_runtime_lock(&shared, "same-id", "arn:test").await;
let opts = test_resync_opts("same-id", "arn:test", "run-a");
pool.clone()
.start_bucket_resync(opts.clone())
.await
.expect("first start should be accepted");
let first_status = pool
.resyncer
.status_map
.read()
.await
.get("same-id")
.expect("accepted status should be published")
.targets_map["arn:test"]
.clone();
pool.clone()
.start_bucket_resync(opts)
.await
.expect("same ID retry should be accepted idempotently");
let retried_status = pool
.resyncer
.status_map
.read()
.await
.get("same-id")
.expect("retried status should remain published")
.targets_map["arn:test"]
.clone();
assert_eq!(shared.write_count.load(Ordering::SeqCst), 1);
assert_eq!(retried_status.resync_id, first_status.resync_id);
assert_eq!(retried_status.start_time, first_status.start_time);
assert_eq!(retried_status.resync_status, ResyncStatusType::ResyncPending);
assert_eq!(pool.resyncer.cancel_tokens.read().await.len(), 1);
}
#[tokio::test]
async fn admitted_resync_waits_for_target_metadata_commit_before_activation() {
let shared = empty_resync_shared_state();
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let _runtime_guard = hold_resync_runtime_lock(&shared, "two-phase-start", "arn:test").await;
let opts = test_resync_opts("two-phase-start", "arn:test", "run-a");
let new_run = pool
.clone()
.admit_bucket_resync(opts.clone())
.await
.expect("admission should persist the intent");
assert!(new_run);
assert!(pool.resyncer.cancel_tokens.read().await.is_empty());
assert_eq!(shared.write_count.load(Ordering::SeqCst), 1);
pool.clone()
.activate_bucket_resync(opts, false)
.await
.expect("activation should start the admitted run");
assert_eq!(pool.resyncer.cancel_tokens.read().await.len(), 1);
}
#[tokio::test]
async fn same_id_retry_after_restart_recreates_missing_runtime_task() {
let shared = empty_resync_shared_state();
let mut persisted = BucketReplicationResyncStatus::new();
persisted.targets_map.insert(
"arn:test".to_string(),
TargetReplicationResyncStatus {
bucket: "restart-retry".to_string(),
resync_id: "run-a".to_string(),
resync_status: ResyncStatusType::ResyncPending,
..Default::default()
},
);
*shared.data.lock().expect("test data lock should not be poisoned") =
encode_resync_file(&persisted).expect("restart status should encode");
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let _runtime_guard = hold_resync_runtime_lock(&shared, "restart-retry", "arn:test").await;
pool.clone()
.start_bucket_resync(test_resync_opts("restart-retry", "arn:test", "run-a"))
.await
.expect("same ID retry should recover an accepted run");
assert_eq!(shared.write_count.load(Ordering::SeqCst), 0);
assert_eq!(pool.resyncer.cancel_tokens.read().await.len(), 1);
assert_eq!(
pool.resyncer.status_map.read().await["restart-retry"].targets_map["arn:test"].resync_id,
"run-a"
);
}
#[tokio::test]
async fn same_completed_resync_id_retry_does_not_restart_work() {
let shared = empty_resync_shared_state();
let mut persisted = BucketReplicationResyncStatus::new();
persisted.targets_map.insert(
"arn:test".to_string(),
TargetReplicationResyncStatus {
bucket: "completed-retry".to_string(),
resync_id: "run-a".to_string(),
resync_status: ResyncStatusType::ResyncCompleted,
..Default::default()
},
);
*shared.data.lock().expect("test data lock should not be poisoned") =
encode_resync_file(&persisted).expect("completed status should encode");
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
pool.clone()
.start_bucket_resync(test_resync_opts("completed-retry", "arn:test", "run-a"))
.await
.expect("completed same ID retry should remain idempotent");
assert_eq!(shared.write_count.load(Ordering::SeqCst), 0);
assert!(pool.resyncer.cancel_tokens.read().await.is_empty());
assert_eq!(
pool.resyncer.status_map.read().await["completed-retry"].targets_map["arn:test"].resync_status,
ResyncStatusType::ResyncCompleted
);
}
#[tokio::test]
async fn start_failure_does_not_publish_or_persist_requested_id() {
let shared = empty_resync_shared_state();
shared.fail_next_write.store(true, Ordering::SeqCst);
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let error = pool
.clone()
.start_bucket_resync(test_resync_opts("failed-start", "arn:test", "run-a"))
.await
.expect_err("metadata save failure should reject the start");
assert!(matches!(error, EcstoreError::Unexpected));
assert!(shared.data.lock().expect("test data lock should not be poisoned").is_empty());
assert!(!pool.resyncer.status_map.read().await.contains_key("failed-start"));
assert_eq!(shared.write_count.load(Ordering::SeqCst), 0);
}
#[tokio::test]
async fn canceled_start_request_finishes_accepted_transaction() {
let shared = empty_resync_shared_state();
shared.block_next_write.store(true, Ordering::SeqCst);
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let _runtime_guard = hold_resync_runtime_lock(&shared, "canceled-start", "arn:test").await;
let start_pool = pool.clone();
let start = tokio::spawn(async move {
start_pool
.start_bucket_resync(test_resync_opts("canceled-start", "arn:test", "run-a"))
.await
});
tokio::time::timeout(Duration::from_secs(10), shared.write_started.notified())
.await
.expect("start transaction should reach the durable write");
start.abort();
assert!(start.await.expect_err("caller task should be canceled").is_cancelled());
shared.allow_write.notify_one();
tokio::time::timeout(Duration::from_secs(10), async {
loop {
if pool.resyncer.status_map.read().await.contains_key("canceled-start") {
break;
}
tokio::task::yield_now().await;
}
})
.await
.expect("detached admission transaction should finish after caller cancellation");
assert_eq!(shared.write_count.load(Ordering::SeqCst), 1);
assert_eq!(
pool.resyncer.status_map.read().await["canceled-start"].targets_map["arn:test"].resync_id,
"run-a"
);
}
#[test]
fn replication_queue_admission_combines_target_results() {
let mut admission = ReplicationQueueAdmission::Skipped;
@@ -1958,10 +2379,16 @@ mod tests {
async fn load_resync_leader_lock_allows_only_one_startup_recovery() {
temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("1"))], async {
let shared = Arc::new(LoadResyncSharedState {
data: load_resync_test_metadata(),
data: StdMutex::new(load_resync_test_metadata()),
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
first_read_started: Notify::new(),
delay_first_read: AtomicBool::new(true),
read_count: AtomicUsize::new(0),
write_count: AtomicUsize::new(0),
fail_next_write: AtomicBool::new(false),
block_next_write: AtomicBool::new(false),
write_started: Notify::new(),
allow_write: Notify::new(),
});
let leader_pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", shared.clone()))).await;
let skipped_pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-b", shared.clone()))).await;
@@ -2233,4 +2660,53 @@ mod tests {
// None so replay falls back to the current time (backlog#867 backward compatibility).
assert_eq!(entry.delete_marker_mtime, None, "missing deleteMarkerMtime key must default to None");
}
#[test]
fn durable_mrf_snapshot_reads_restart_backlog_and_valid_empty_state() {
let entries = vec![MrfReplicateEntry {
bucket: "restart-bucket".to_string(),
object: "object".to_string(),
version_id: None,
retry_count: 1,
size: 512,
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
delete_marker_mtime: None,
}];
let encoded = encode_mrf_file(&entries).expect("durable MRF backlog should encode");
let recovered = durable_mrf_backlog_from_read(Ok(encoded));
assert!(recovered.available);
assert_eq!(recovered.entries.len(), 1);
assert_eq!(recovered.entries[0].bucket, "restart-bucket");
assert_eq!(recovered.entries[0].size, 512);
let missing_file = durable_mrf_backlog_from_read(Err(EcstoreError::ConfigNotFound));
assert!(missing_file.available);
assert!(missing_file.entries.is_empty());
}
#[test]
fn durable_mrf_snapshot_marks_corrupt_or_invalid_data_unavailable() {
let corrupt = durable_mrf_backlog_from_read(Ok(vec![0, 1, 2]));
assert!(!corrupt.available);
assert!(corrupt.entries.is_empty());
let negative = encode_mrf_file(&[MrfReplicateEntry {
bucket: "bucket".to_string(),
object: "object".to_string(),
version_id: None,
retry_count: 0,
size: -1,
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
delete_marker_mtime: None,
}])
.expect("invalid persisted entry should still encode for boundary testing");
let invalid = durable_mrf_backlog_from_read(Ok(negative));
assert!(!invalid.available);
assert!(invalid.entries.is_empty());
}
}
@@ -17,7 +17,8 @@ use super::replication_filemeta_boundary::MrfReplicateEntry;
pub use rustfs_replication::{BucketReplicationResyncStatus, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus};
pub(crate) use rustfs_replication::{
is_version_id_mismatch, resync_state_accepts_update, should_auto_resume_resync, should_count_head_proxy_failure,
is_version_id_mismatch, resync_state_accepts_update, sanitize_resync_error_detail, should_auto_resume_resync,
should_count_head_proxy_failure,
};
pub(crate) const RESYNC_META_FORMAT: u16 = rustfs_replication::resync::RESYNC_META_FORMAT;
@@ -37,7 +37,7 @@ use super::replication_queue_boundary::DeletedObjectReplicationInfo;
use super::replication_resync_boundary::ResyncStatusType;
use super::replication_resync_boundary::{
BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus, encode_resync_file, is_version_id_mismatch,
resync_state_accepts_update, should_count_head_proxy_failure,
resync_state_accepts_update, sanitize_resync_error_detail, should_count_head_proxy_failure,
};
#[cfg(test)]
use super::replication_resync_boundary::{RESYNC_META_FORMAT, RESYNC_META_VERSION, WIRE_ZERO_TIME_UNIX, decode_resync_file};
@@ -117,6 +117,20 @@ const RESYNC_TIME_INTERVAL: TokioDuration = TokioDuration::from_secs(60);
static WARNED_MONITOR_UNINIT: std::sync::Once = std::sync::Once::new();
fn resync_target_error_detail<E, R>(error: &SdkError<E, R>) -> Option<String>
where
E: ProvideErrorMetadata,
{
sanitize_resync_error_detail(error.code().unwrap_or(match error {
SdkError::ConstructionFailure(_) => "failed to construct target request",
SdkError::TimeoutError(_) => "target request timed out",
SdkError::DispatchFailure(_) => "target dispatch failed",
SdkError::ResponseError(_) => "invalid target response",
SdkError::ServiceError(_) => "target service error",
_ => "target request failed",
}))
}
async fn finish_resync_workers(
worker_txs: Vec<tokio::sync::mpsc::Sender<ReplicateObjectInfo>>,
results_tx: tokio::sync::mpsc::Sender<TargetReplicationResyncStatus>,
@@ -241,11 +255,13 @@ fn resync_status_duration(
Some(std::time::Duration::from_millis(millis))
}
type ResyncCancelKey = (String, String, String);
#[derive(Debug)]
pub struct ReplicationResyncer {
pub status_map: Arc<RwLock<HashMap<String, BucketReplicationResyncStatus>>>,
pub worker_size: usize,
pub cancel_tokens: Arc<RwLock<HashMap<String, CancellationToken>>>,
pub(crate) cancel_tokens: Arc<RwLock<HashMap<ResyncCancelKey, CancellationToken>>>,
}
impl ReplicationResyncer {
@@ -257,12 +273,19 @@ impl ReplicationResyncer {
}
}
fn cancel_key(opts: &ResyncOpts) -> String {
format!("{}:{}", opts.bucket, opts.arn)
fn cancel_key(opts: &ResyncOpts) -> ResyncCancelKey {
(opts.bucket.clone(), opts.arn.clone(), opts.resync_id.clone())
}
pub async fn register_cancel_token(&self, opts: &ResyncOpts, token: CancellationToken) {
self.cancel_tokens.write().await.insert(Self::cancel_key(opts), token);
pub async fn register_cancel_token(&self, opts: &ResyncOpts, token: CancellationToken) -> bool {
let mut cancel_tokens = self.cancel_tokens.write().await;
match cancel_tokens.entry(Self::cancel_key(opts)) {
std::collections::hash_map::Entry::Vacant(entry) => {
entry.insert(token);
true
}
std::collections::hash_map::Entry::Occupied(_) => false,
}
}
pub async fn clear_cancel_token(&self, opts: &ResyncOpts) {
@@ -428,6 +451,9 @@ impl ReplicationResyncer {
state.replicated_size += status.replicated_size;
state.failed_count += status.failed_count;
state.failed_size += status.failed_size;
if state.error.is_none() && status.failed_count > 0 {
state.error = status.error.as_deref().and_then(sanitize_resync_error_detail);
}
state.last_update = Some(now);
bucket_status.last_update = Some(now);
}
@@ -885,6 +911,7 @@ impl ReplicationResyncer {
"Processed resync object"
);
}
st.error = err.as_ref().and_then(resync_target_error_detail);
if cancel_token.is_cancelled() {
return;
@@ -2071,14 +2098,20 @@ pub async fn replicate_object<S: ReplicationStorage>(roi: ReplicateObjectInfo, s
for arn in tgt_arns {
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(&bucket, &arn).await else {
// Deliberately debug: this fires once per object per ARN, so a target that
// stays unreachable would flood the log from the replication hot path. The
// condition is reported once per pass by the site-replication reconciler and
// once per rebuild by `update_all_targets`, which is where an operator can act
// on it; the per-object event below still records each dropped object.
debug!(
event = EVENT_RESYNC_RUNTIME_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
object = %object,
arn = %arn,
reason = "target_client_missing",
"Skipping replication object target"
"Replication rule has no bucket target for its destination ARN; object not replicated"
);
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
@@ -3227,6 +3260,7 @@ mod tests {
assert_eq!(tgt.start_time, Some(start));
assert_eq!(tgt.last_update, Some(last));
assert_eq!(tgt.resync_before_date, Some(before));
assert_eq!(tgt.error, None);
}
#[test]
@@ -3681,6 +3715,59 @@ mod tests {
assert!(resyncer.target_has_resync_failures(&opts).await);
}
#[tokio::test]
async fn test_inc_stats_retains_first_sanitized_error_across_success() {
let resyncer = ReplicationResyncer::new().await;
let opts = ResyncOpts {
bucket: "bucket".to_string(),
arn: "arn:replication::dest".to_string(),
resync_id: "run-new".to_string(),
resync_before: None,
};
let failed = TargetReplicationResyncStatus {
failed_count: 1,
object: "failed-object".to_string(),
error: Some("Authorization: Bearer status-secret".to_string()),
..Default::default()
};
let later_failure = TargetReplicationResyncStatus {
failed_count: 1,
object: "later-failed-object".to_string(),
error: Some("AccessDenied".to_string()),
..Default::default()
};
let succeeded = TargetReplicationResyncStatus {
replicated_count: 1,
object: "successful-object".to_string(),
..Default::default()
};
resyncer.inc_stats(&failed, opts.clone()).await;
resyncer.inc_stats(&later_failure, opts.clone()).await;
resyncer.inc_stats(&succeeded, opts.clone()).await;
let status_map = resyncer.status_map.read().await;
let target = &status_map["bucket"].targets_map["arn:replication::dest"];
assert_eq!(target.failed_count, 2);
assert_eq!(target.replicated_count, 1);
assert_eq!(target.object, "successful-object");
assert_eq!(target.error.as_deref(), Some("[redacted sensitive resync error detail]"));
}
#[test]
fn test_resync_target_error_detail_uses_safe_service_code_and_fallback() {
let metadata = aws_smithy_types::error::ErrorMetadata::builder()
.code("AccessDenied")
.message("Authorization: Bearer status-secret")
.build();
let service_error = SdkError::service_error(HeadObjectError::generic(metadata), ());
let timeout_error =
SdkError::<HeadObjectError, ()>::timeout_error(std::io::Error::new(std::io::ErrorKind::TimedOut, "status-secret"));
assert_eq!(resync_target_error_detail(&service_error).as_deref(), Some("AccessDenied"));
assert_eq!(resync_target_error_detail(&timeout_error).as_deref(), Some("target request timed out"));
}
#[test]
fn test_resync_state_accepts_update_only_for_matching_run() {
let current = TargetReplicationResyncStatus {
@@ -15,9 +15,11 @@
use super::replication_error_boundary::Error;
use super::replication_filemeta_boundary::{ReplicatedTargetInfo, ReplicationStatusType, ReplicationType};
use super::replication_resync_boundary::ResyncStatusType;
#[cfg(test)]
use super::replication_stats_boundary::FailStats;
use super::replication_stats_boundary::{
ActiveWorkerStat, BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, ProxyMetric, ProxyStatsCache,
QueueCache, SRMetricsSummary, XferStats,
QueueCache, ReplicationMetricScope, SRMetricsSummary, XferStats,
};
use super::runtime_boundary as runtime_sources;
use std::collections::HashMap;
@@ -361,10 +363,12 @@ impl ReplicationStats {
if rs.transfer_duration > Duration::default() {
stat.latency.update(rs.transfer_size, rs.transfer_duration);
stat.update_xfer_rate(rs.transfer_size, rs.transfer_duration);
stat.latency_scope = ReplicationMetricScope::NodeLocal;
}
}
(false, true, false) => {
stat.fail_stats.add_size(rs.transfer_size, rs.err.as_ref());
stat.failed = stat.fail_stats.to_metric();
}
(false, false, true) => {
// Pending status, no processing for now
@@ -379,7 +383,10 @@ impl ReplicationStats {
let mut result = HashMap::with_capacity(cache.len());
for (bucket, stats) in cache.iter() {
result.insert(bucket.clone(), stats.clone_stats());
let mut snapshot = stats.clone_stats();
snapshot.mark_node_local_provider_available();
snapshot.queue_scope = ReplicationMetricScope::NodeLocal;
result.insert(bucket.clone(), snapshot);
}
drop(cache);
@@ -388,6 +395,8 @@ impl ReplicationStats {
for (bucket, queue_stats) in &q_cache.bucket_stats {
let bucket_stats = result.entry(bucket.clone()).or_insert_with(BucketReplicationStats::new);
bucket_stats.q_stat = queue_stats.snapshot();
bucket_stats.mark_node_local_provider_available();
bucket_stats.queue_scope = ReplicationMetricScope::NodeLocal;
}
}
@@ -405,9 +414,13 @@ impl ReplicationStats {
pub async fn get(&self, bucket: &str) -> BucketReplicationStats {
let cache = self.cache.read().await;
if let Some(stats) = cache.get(bucket) {
stats.clone_stats()
let mut snapshot = stats.clone_stats();
snapshot.mark_node_local_provider_available();
snapshot
} else {
BucketReplicationStats::new()
let mut snapshot = BucketReplicationStats::new();
snapshot.mark_node_local_provider_available();
snapshot
}
}
@@ -453,11 +466,15 @@ impl ReplicationStats {
let mut tq = InQueueMetric::default();
for bucket_stat in &bucket_stats {
tot_replica_size += bucket_stat.replication_stats.replica_size;
tot_replica_count += bucket_stat.replication_stats.replica_count;
tot_replica_size = tot_replica_size.saturating_add(bucket_stat.replication_stats.replica_size);
tot_replica_count = tot_replica_count.saturating_add(bucket_stat.replication_stats.replica_count);
for q in &bucket_stat.queue_stats.nodes {
tq = tq.merge(&q.q_stats);
if bucket_stat.replication_stats.queue_scope != ReplicationMetricScope::Unavailable {
tq = tq.merge(&bucket_stat.replication_stats.q_stat);
} else {
for q in &bucket_stat.queue_stats.nodes {
tq = tq.merge(&q.q_stats);
}
}
for (arn, stat) in &bucket_stat.replication_stats.stats {
@@ -470,22 +487,38 @@ impl ReplicationStats {
let f_stats = stat.fail_stats.merge(&old_stat.fail_stats);
let lrg = old_stat.xfer_rate_lrg.merge(&stat.xfer_rate_lrg);
let sml = old_stat.xfer_rate_sml.merge(&stat.xfer_rate_sml);
let latency_available = stat.latency_scope != ReplicationMetricScope::Unavailable
|| old_stat.latency_scope != ReplicationMetricScope::Unavailable;
let bandwidth_available = stat.bandwidth_scope != ReplicationMetricScope::Unavailable
|| old_stat.bandwidth_scope != ReplicationMetricScope::Unavailable;
*old_stat = BucketReplicationStat {
failed: f_stats.to_metric(),
fail_stats: f_stats,
replicated_size: stat.replicated_size + old_stat.replicated_size,
replicated_count: stat.replicated_count + old_stat.replicated_count,
replicated_size: stat.replicated_size.saturating_add(old_stat.replicated_size),
replicated_count: stat.replicated_count.saturating_add(old_stat.replicated_count),
latency: stat.latency.merge(&old_stat.latency),
xfer_rate_lrg: lrg,
xfer_rate_sml: sml,
bandwidth_limit_bytes_per_sec: stat.bandwidth_limit_bytes_per_sec,
bandwidth_limit_bytes_per_sec: stat
.bandwidth_limit_bytes_per_sec
.saturating_add(old_stat.bandwidth_limit_bytes_per_sec),
current_bandwidth_bytes_per_sec: stat.current_bandwidth_bytes_per_sec
+ old_stat.current_bandwidth_bytes_per_sec,
latency_scope: if latency_available {
ReplicationMetricScope::ClusterAggregated
} else {
ReplicationMetricScope::Unavailable
},
bandwidth_scope: if bandwidth_available {
ReplicationMetricScope::ClusterAggregated
} else {
ReplicationMetricScope::Unavailable
},
};
tot_replicated_size += stat.replicated_size;
tot_replicated_count += stat.replicated_count;
tot_replicated_size = tot_replicated_size.saturating_add(stat.replicated_size);
tot_replicated_count = tot_replicated_count.saturating_add(stat.replicated_count);
}
}
@@ -499,23 +532,28 @@ impl ReplicationStats {
resync_started_count: bucket_stats
.iter()
.map(|stats| stats.replication_stats.resync_started_count)
.sum(),
.fold(0i64, i64::saturating_add),
resync_completed_count: bucket_stats
.iter()
.map(|stats| stats.replication_stats.resync_completed_count)
.sum(),
.fold(0i64, i64::saturating_add),
resync_failed_count: bucket_stats
.iter()
.map(|stats| stats.replication_stats.resync_failed_count)
.sum(),
.fold(0i64, i64::saturating_add),
resync_canceled_count: bucket_stats
.iter()
.map(|stats| stats.replication_stats.resync_canceled_count)
.sum(),
.fold(0i64, i64::saturating_add),
resync_duration_ms: bucket_stats
.iter()
.map(|stats| stats.replication_stats.resync_duration_ms)
.sum(),
.fold(0i64, i64::saturating_add),
provider_available: true,
cluster_complete: true,
observed_node_count: u32::try_from(bucket_stats.len()).unwrap_or(u32::MAX),
expected_node_count: u32::try_from(bucket_stats.len()).unwrap_or(u32::MAX),
queue_scope: ReplicationMetricScope::ClusterAggregated,
};
let qs = Default::default();
@@ -547,6 +585,33 @@ impl ReplicationStats {
bs
}
pub async fn aggregate_bucket_replication_stats(
&self,
bucket: &str,
bucket_stats: Vec<BucketStats>,
expected_node_count: u32,
) -> BucketStats {
let mut aggregated = self.calculate_bucket_replication_stats(bucket, bucket_stats).await;
let observed_node_count = aggregated.replication_stats.observed_node_count;
let complete = observed_node_count == expected_node_count;
aggregated.replication_stats.expected_node_count = expected_node_count;
aggregated.replication_stats.cluster_complete = complete;
aggregated.replication_stats.queue_scope = if complete {
ReplicationMetricScope::ClusterAggregated
} else {
ReplicationMetricScope::PartialCluster
};
for stat in aggregated.replication_stats.stats.values_mut() {
if stat.latency_scope != ReplicationMetricScope::Unavailable {
stat.latency_scope = aggregated.replication_stats.queue_scope;
}
if stat.bandwidth_scope != ReplicationMetricScope::Unavailable {
stat.bandwidth_scope = aggregated.replication_stats.queue_scope;
}
}
aggregated
}
/// Get latest replication statistics
pub async fn get_latest_replication_stats(&self, bucket: &str) -> BucketStats {
// In actual implementation, statistics would be obtained from cluster
@@ -567,6 +632,15 @@ impl ReplicationStats {
};
drop(cache);
{
let q_cache = self.q_cache.lock().await;
if let Some(queue_stats) = q_cache.bucket_stats.get(bucket) {
replication_stats.q_stat = queue_stats.snapshot();
}
}
replication_stats.mark_node_local_provider_available();
replication_stats.queue_scope = ReplicationMetricScope::NodeLocal;
if let Some(monitor) = runtime_sources::bucket_monitor() {
let bw_report = monitor.get_report(|name| name == bucket);
for (opts, bw) in bw_report.bucket_stats {
@@ -578,8 +652,7 @@ impl ReplicationStats {
xfer_rate_sml: XferStats::new(),
..Default::default()
});
stat.bandwidth_limit_bytes_per_sec = bw.limit_bytes_per_sec;
stat.current_bandwidth_bytes_per_sec = bw.current_bandwidth_bytes_per_sec;
stat.set_node_local_bandwidth(bw.limit_bytes_per_sec, bw.current_bandwidth_bytes_per_sec);
}
}
@@ -724,6 +797,132 @@ mod tests {
assert_eq!(stat.replicated_count, 1);
}
#[tokio::test]
async fn latest_stats_include_queue_until_drained() {
let stats = ReplicationStats::new();
stats.inc_q("queued-bucket", 4096, false, ReplicationType::Object).await;
let queued = stats.get_latest_replication_stats("queued-bucket").await;
assert!(queued.replication_stats.provider_available);
assert_eq!(queued.replication_stats.q_stat.curr.count, 1);
assert_eq!(queued.replication_stats.q_stat.curr.bytes, 4096);
assert_eq!(queued.replication_stats.queue_scope, ReplicationMetricScope::NodeLocal);
stats.dec_q("queued-bucket", 4096, false, ReplicationType::Object).await;
let drained = stats.get_latest_replication_stats("queued-bucket").await;
assert_eq!(drained.replication_stats.q_stat.curr.count, 0);
assert_eq!(drained.replication_stats.q_stat.curr.bytes, 0);
}
#[tokio::test]
async fn failed_metric_matches_authoritative_fail_stats() {
let stats = ReplicationStats::new();
let target_info = ReplicatedTargetInfo {
arn: "failed-arn".to_string(),
size: 2048,
duration: Duration::from_millis(25),
op_type: ReplicationType::Object,
error: Some("target unavailable".to_string()),
..Default::default()
};
stats
.update(
"failed-bucket",
&target_info,
ReplicationStatusType::Failed,
ReplicationStatusType::Pending,
)
.await;
let snapshot = stats.get_latest_replication_stats("failed-bucket").await;
let target = &snapshot.replication_stats.stats["failed-arn"];
assert_eq!(target.failed.count, target.fail_stats.count);
assert_eq!(target.failed.size, target.fail_stats.size);
assert_eq!(target.failed.count, 1);
assert_eq!(target.failed.size, 2048);
}
#[tokio::test]
async fn valid_empty_provider_is_not_reported_as_unavailable() {
let stats = ReplicationStats::new();
let snapshot = stats.get_latest_replication_stats("empty-bucket").await;
assert!(snapshot.replication_stats.provider_available);
assert!(snapshot.replication_stats.cluster_complete);
assert_eq!(snapshot.replication_stats.observed_node_count, 1);
assert_eq!(snapshot.replication_stats.expected_node_count, 1);
assert!(snapshot.replication_stats.stats.is_empty());
}
#[tokio::test]
async fn cluster_aggregation_counts_each_node_once_and_marks_partial() {
let stats = ReplicationStats::new();
let node = |failed_count, failed_size, queued_count, queued_size| {
let mut fail_stats = FailStats::new();
fail_stats.count = failed_count;
fail_stats.size = failed_size;
let mut targets = HashMap::new();
targets.insert(
"arn".to_string(),
BucketReplicationStat {
fail_stats,
latency_scope: ReplicationMetricScope::NodeLocal,
..Default::default()
},
);
let q_stat = InQueueMetric::default();
q_stat.curr.now_count.store(queued_count, Ordering::Relaxed);
q_stat.curr.now_bytes.store(queued_size, Ordering::Relaxed);
let q_stat = q_stat.snapshot();
BucketStats {
replication_stats: BucketReplicationStats {
stats: targets,
q_stat,
provider_available: true,
queue_scope: ReplicationMetricScope::NodeLocal,
..Default::default()
},
..Default::default()
}
};
let aggregated = stats
.aggregate_bucket_replication_stats("bucket", vec![node(1, 10, 2, 20), node(3, 30, 4, 40)], 3)
.await;
let target = &aggregated.replication_stats.stats["arn"];
assert_eq!(target.failed.count, 4);
assert_eq!(target.failed.size, 40);
assert_eq!(aggregated.replication_stats.q_stat.curr.count, 6);
assert_eq!(aggregated.replication_stats.q_stat.curr.bytes, 60);
assert_eq!(aggregated.replication_stats.observed_node_count, 2);
assert_eq!(aggregated.replication_stats.expected_node_count, 3);
assert!(!aggregated.replication_stats.cluster_complete);
assert_eq!(aggregated.replication_stats.queue_scope, ReplicationMetricScope::PartialCluster);
assert_eq!(target.latency_scope, ReplicationMetricScope::PartialCluster);
}
#[tokio::test]
async fn concurrent_queue_updates_are_visible_without_lost_counts() {
let stats = Arc::new(ReplicationStats::new());
let mut tasks = Vec::with_capacity(32);
for _ in 0..32 {
let stats = Arc::clone(&stats);
tasks.push(tokio::spawn(async move {
stats.inc_q("concurrent-bucket", 7, false, ReplicationType::Object).await;
}));
}
for task in tasks {
task.await.expect("queue update task should complete");
}
let snapshot = stats.get_latest_replication_stats("concurrent-bucket").await;
assert_eq!(snapshot.replication_stats.q_stat.curr.count, 32);
assert_eq!(snapshot.replication_stats.q_stat.curr.bytes, 224);
}
#[tokio::test]
async fn test_get_all_includes_proxy_only_bucket() {
let stats = ReplicationStats::new();
@@ -13,7 +13,9 @@
// limitations under the License.
pub use rustfs_replication::BucketStats;
#[cfg(test)]
pub(crate) use rustfs_replication::FailStats;
pub(crate) use rustfs_replication::{
ActiveWorkerStat, BucketReplicationStat, BucketReplicationStats, InQueueMetric, ProxyMetric, ProxyStatsCache, QueueCache,
SRMetricsSummary, XferStats,
ReplicationMetricScope, SRMetricsSummary, XferStats,
};
+7 -4
View File
@@ -310,10 +310,8 @@ pub fn check_list_multipart_args(
) -> Result<()> {
check_list_objs_args(bucket, prefix, key_marker)?;
if let Some(upload_id_marker) = upload_id_marker {
if let Some(key_marker) = key_marker
&& key_marker.ends_with('/')
{
if let (Some(key_marker), Some(upload_id_marker)) = (key_marker, upload_id_marker) {
if key_marker.ends_with('/') {
return Err(StorageError::InvalidUploadIDKeyCombination(
upload_id_marker.to_string(),
key_marker.to_string(),
@@ -629,6 +627,11 @@ mod tests {
assert!(check_list_objs_args("INVALID", "", &None).is_err());
}
#[test]
fn test_list_multipart_upload_marker_is_ignored_without_key_marker() {
assert!(check_list_multipart_args("valid-bucket", "", &None, &Some("not-base64!".to_string()), &None,).is_ok());
}
#[test]
fn test_check_multipart_args() {
assert!(check_new_multipart_args("valid-bucket", "valid-object").is_ok());
@@ -85,4 +85,21 @@ impl BucketVersioningSys {
Ok(cfg)
}
/// Instance-scoped variant of [`Self::get`] (backlog#1052): resolves the
/// caller's own instance context so a second in-process store never
/// answers with the first instance's versioning state; falls back to the
/// ambient system when the instance cell is not initialized.
pub(crate) async fn get_in(ctx: &crate::runtime::instance::InstanceContext, bucket: &str) -> Result<VersioningConfiguration> {
if bucket == RUSTFS_META_BUCKET || bucket.starts_with(RUSTFS_META_BUCKET) {
return Ok(VersioningConfiguration::default());
}
let bucket_meta_sys_lock = crate::bucket::metadata_sys::bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
let (cfg, _) = bucket_meta_sys.get_versioning_config(bucket).await?;
Ok(cfg)
}
}
+32 -35
View File
@@ -267,6 +267,23 @@ impl Clone for ListPathRawOptions {
}
}
fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
WalkDirOptions {
bucket: opts.bucket.clone(),
base_dir: opts.path.clone(),
recursive: opts.recursive,
incl_deleted: opts.incl_deleted,
report_notfound: opts.report_not_found,
filter_prefix: opts.filter_prefix.clone(),
forward_to: opts.forward_to.clone(),
limit: opts.per_disk_limit,
skip_total_timeout: opts.skip_walkdir_total_timeout,
timeout_ms: opts.walkdir_timeout.map(duration_millis),
stall_timeout_ms: opts.walkdir_stall_timeout.map(duration_millis),
..Default::default()
}
}
pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> disk::error::Result<()> {
let rx = rx.child_token();
let _cancel_guard = rx.clone().drop_guard();
@@ -373,20 +390,7 @@ async fn list_path_raw_inner(
None
};
let wakl_opts = WalkDirOptions {
bucket: opts_clone.bucket.clone(),
base_dir: opts_clone.path.clone(),
recursive: opts_clone.recursive,
incl_deleted: opts_clone.incl_deleted,
report_notfound: opts_clone.report_not_found,
filter_prefix: opts_clone.filter_prefix.clone(),
forward_to: opts_clone.forward_to.clone(),
limit: opts_clone.per_disk_limit,
skip_total_timeout: opts_clone.skip_walkdir_total_timeout,
timeout_ms: opts_clone.walkdir_timeout.map(duration_millis),
stall_timeout_ms: opts_clone.walkdir_stall_timeout.map(duration_millis),
..Default::default()
};
let wakl_opts = walk_dir_options(&opts_clone);
let mut need_fallback = false;
let mut last_err = None;
@@ -559,27 +563,7 @@ async fn list_path_raw_inner(
}
let fallback_walk_started = std::time::Instant::now();
match disk
.as_ref()
.walk_dir(
WalkDirOptions {
bucket: opts_clone.bucket.clone(),
base_dir: opts_clone.path.clone(),
recursive: opts_clone.recursive,
incl_deleted: opts_clone.incl_deleted,
report_notfound: opts_clone.report_not_found,
filter_prefix: opts_clone.filter_prefix.clone(),
forward_to: opts_clone.forward_to.clone(),
limit: opts_clone.per_disk_limit,
skip_total_timeout: opts_clone.skip_walkdir_total_timeout,
timeout_ms: opts_clone.walkdir_timeout.map(duration_millis),
stall_timeout_ms: opts_clone.walkdir_stall_timeout.map(duration_millis),
..Default::default()
},
&mut wr,
)
.await
{
match disk.as_ref().walk_dir(walk_dir_options(&opts_clone), &mut wr).await {
Ok(_r) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_fallback",
@@ -1091,6 +1075,19 @@ mod tests {
assert!(!is_benign_not_found_listing_failure(&[DiskError::DiskNotFound]));
}
#[test]
fn walk_dir_options_preserve_zero_total_and_inherited_stall_timeouts() {
let options = walk_dir_options(&ListPathRawOptions {
walkdir_timeout: Some(Duration::ZERO),
walkdir_stall_timeout: None,
..Default::default()
});
assert_eq!(options.timeout_ms, Some(0));
assert_eq!(options.stall_timeout_ms, None);
assert!(!options.skip_total_timeout);
}
#[tokio::test]
async fn list_path_raw_empty_disks_returns_read_quorum() {
let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
@@ -417,7 +417,7 @@ impl TransitionClient {
bucket: complete_multipart_upload_result.bucket,
key: complete_multipart_upload_result.key,
etag: trim_etag(&complete_multipart_upload_result.etag),
version_id: self.raw_version_id(&h)?.unwrap_or_default().to_string(),
version_id: self.legacy_remote_version_id(&h)?,
location: complete_multipart_upload_result.location,
expiration: exp_time,
expiration_rule_id: rule_id,
@@ -22,7 +22,7 @@ use bytes::Bytes;
use futures::future::join_all;
use http::{HeaderMap, HeaderName, HeaderValue, StatusCode};
use std::io::Error;
use std::sync::RwLock;
use std::sync::{Mutex, MutexGuard, RwLock};
use std::{collections::HashMap, sync::Arc};
use time::{OffsetDateTime, format_description};
use tokio::io::AsyncReadExt;
@@ -46,6 +46,14 @@ use crate::client::utils::base64_encode;
use rustfs_utils::path::trim_etag;
use s3s::header::X_AMZ_EXPIRATION;
fn lock_md5_hasher(
md5_hasher: &Mutex<Option<rustfs_utils::hash::HashAlgorithm>>,
) -> Result<MutexGuard<'_, Option<rustfs_utils::hash::HashAlgorithm>>, std::io::Error> {
md5_hasher
.lock()
.map_err(|_| std::io::Error::other("MD5 hasher state is unavailable"))
}
/// Read exactly `want` bytes for a single multipart part, or fewer if the reader
/// reaches EOF first. Advances the reader so the next call returns the following
/// part. Replaces the previous per-part `read_all()`/`to_vec()`, which drained
@@ -177,7 +185,7 @@ impl TransitionClient {
let length = buf.len();
if opts.send_content_md5 {
let mut md5_hasher = self.md5_hasher.lock().unwrap();
let mut md5_hasher = lock_md5_hasher(&self.md5_hasher)?;
let md5_hash = match md5_hasher.as_mut() {
Some(hasher) => hasher,
None => return Err(std::io::Error::other("MD5 hasher not initialized")),
@@ -370,7 +378,7 @@ impl TransitionClient {
let mut md5_base64: String = "".to_string();
if opts.send_content_md5 {
let mut md5_hasher = clone_self.md5_hasher.lock().unwrap();
let mut md5_hasher = lock_md5_hasher(&clone_self.md5_hasher)?;
let md5_hash = match md5_hasher.as_mut() {
Some(hasher) => hasher,
None => {
@@ -418,6 +426,9 @@ impl TransitionClient {
}
let results = join_all(futures).await;
for result in results {
result?;
}
select! {
err = err_rx.recv() => {
@@ -567,7 +578,7 @@ impl TransitionClient {
key: object_name.to_string(),
etag: trim_etag(h.get("ETag").and_then(|v| v.to_str().ok()).unwrap_or("")),
version_id: self.raw_version_id(h)?.unwrap_or_default().to_string(),
version_id: self.legacy_remote_version_id(h)?,
size,
expiration: exp_time,
expiration_rule_id: rule_id,
@@ -620,10 +631,12 @@ fn collect_complete_parts(parts_info: &HashMap<i64, ObjectPart>, total_parts_cou
#[cfg(test)]
mod tests {
use super::{ObjectPart, ReaderImpl, collect_complete_parts, read_multipart_part};
use super::{ObjectPart, ReaderImpl, collect_complete_parts, lock_md5_hasher, read_multipart_part};
use crate::object_api::GetObjectReader;
use bytes::Bytes;
use rustfs_utils::hash::HashAlgorithm;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
// Drive a reader through the same per-part loop the multipart stream uses and
// collect the size of every part. Regression for rustfs/rustfs#4811: the old
@@ -733,4 +746,18 @@ mod tests {
"a gap in the parts map must be an error, not a panic"
);
}
#[test]
fn poisoned_md5_state_fails_closed() {
let hasher = Arc::new(Mutex::new(Some(HashAlgorithm::Md5)));
let poison_target = Arc::clone(&hasher);
let _ = std::thread::spawn(move || {
let _guard = poison_target.lock().expect("fresh mutex should lock");
panic!("poison MD5 state");
})
.join();
let error = lock_md5_hasher(&hasher).expect_err("poisoned hash state must not be reused");
assert_eq!(error.kind(), std::io::ErrorKind::Other);
}
}
+1 -1
View File
@@ -201,7 +201,7 @@ impl TransitionClient {
object_name: object_name.to_string(),
object_version_id: opts.version_id,
delete_marker: resp.headers().get(X_AMZ_DELETE_MARKER).map_or(false, |v| v == "true"),
delete_marker_version_id: self.raw_version_id(resp.headers())?.unwrap_or_default().to_string(),
delete_marker_version_id: self.legacy_remote_version_id(resp.headers())?,
..Default::default()
})
}
+119 -1
View File
@@ -46,11 +46,33 @@ impl RemoteVersion {
Self::Unknown | Self::Disabled => None,
}
}
pub(crate) fn exact_request_id(&self) -> Result<Option<&str>, Error> {
match self {
Self::Unknown => Err(Error::new(
ErrorKind::InvalidData,
"remote object version is unknown; exact version routing is unsafe",
)),
Self::Disabled => Ok(None),
Self::SuspendedNull => Ok(Some("null")),
Self::Exact(version_id) => Ok(Some(version_id)),
}
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum ConditionalCreateCapability {
Unsupported,
IfNoneMatchStar,
GenerationMatchZero,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) struct ProviderVersionCapabilities {
raw_version_header: Option<&'static str>,
pub(crate) bucket_versioning_state: bool,
pub(crate) list_object_versions: bool,
pub(crate) conditional_create: ConditionalCreateCapability,
pub(crate) exact_get_delete: bool,
}
@@ -62,28 +84,59 @@ impl ProviderVersionCapabilities {
|| tier_type.eq_ignore_ascii_case("r2")
|| tier_type.eq_ignore_ascii_case("wasabi")
{
let list_object_versions = tier_type.eq_ignore_ascii_case("s3")
|| tier_type.eq_ignore_ascii_case("rustfs")
|| tier_type.eq_ignore_ascii_case("minio")
|| tier_type.eq_ignore_ascii_case("r2");
Self {
raw_version_header: Some(X_AMZ_VERSION_ID),
bucket_versioning_state: list_object_versions,
list_object_versions,
conditional_create: if tier_type.eq_ignore_ascii_case("s3") || tier_type.eq_ignore_ascii_case("r2") {
ConditionalCreateCapability::IfNoneMatchStar
} else {
ConditionalCreateCapability::Unsupported
},
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("aliyun") {
Self {
raw_version_header: Some(X_OSS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("tencent") {
Self {
raw_version_header: Some(X_COS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("huaweicloud") {
Self {
raw_version_header: Some(X_OBS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("gcs") {
Self {
raw_version_header: None,
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::GenerationMatchZero,
exact_get_delete: false,
}
} else {
Self {
raw_version_header: None,
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: false,
}
}
@@ -143,7 +196,7 @@ fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
#[cfg(test)]
mod tests {
use super::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
use super::{BucketVersioningState, ConditionalCreateCapability, ProviderVersionCapabilities, RemoteVersion};
use http::{HeaderMap, HeaderValue};
#[test]
@@ -219,6 +272,71 @@ mod tests {
);
}
#[test]
fn provider_capability_matrix_is_conservative_and_provider_specific() {
for (tier_type, state, list, conditional_create, exact_get_delete) in [
("s3", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
("rustfs", true, true, ConditionalCreateCapability::Unsupported, true),
("minio", true, true, ConditionalCreateCapability::Unsupported, true),
("r2", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
("wasabi", false, false, ConditionalCreateCapability::Unsupported, true),
("aliyun", false, false, ConditionalCreateCapability::Unsupported, true),
("tencent", false, false, ConditionalCreateCapability::Unsupported, true),
("huaweicloud", false, false, ConditionalCreateCapability::Unsupported, true),
("gcs", false, false, ConditionalCreateCapability::GenerationMatchZero, false),
("azure", false, false, ConditionalCreateCapability::Unsupported, false),
("unsupported", false, false, ConditionalCreateCapability::Unsupported, false),
] {
let capabilities = ProviderVersionCapabilities::for_tier_type(tier_type);
assert_eq!(capabilities.bucket_versioning_state, state, "{tier_type} versioning state");
assert_eq!(capabilities.list_object_versions, list, "{tier_type} version listing");
assert_eq!(capabilities.conditional_create, conditional_create, "{tier_type} conditional create");
assert_eq!(capabilities.exact_get_delete, exact_get_delete, "{tier_type} exact routing");
}
}
#[test]
fn remote_version_states_preserve_unknown_disabled_suspended_and_exact() {
let capabilities = ProviderVersionCapabilities::for_tier_type("s3");
let empty = HeaderMap::new();
let mut null = HeaderMap::new();
null.insert("x-amz-version-id", HeaderValue::from_static("null"));
let mut exact = HeaderMap::new();
exact.insert("x-amz-version-id", HeaderValue::from_static("opaque.generation-7"));
for (headers, state, expected) in [
(&empty, BucketVersioningState::Unknown, RemoteVersion::Unknown),
(&empty, BucketVersioningState::Disabled, RemoteVersion::Disabled),
(&empty, BucketVersioningState::Suspended, RemoteVersion::Unknown),
(&empty, BucketVersioningState::Enabled, RemoteVersion::Unknown),
(&null, BucketVersioningState::Suspended, RemoteVersion::SuspendedNull),
(
&exact,
BucketVersioningState::Enabled,
RemoteVersion::Exact("opaque.generation-7".to_string()),
),
] {
assert_eq!(
capabilities
.remote_version(headers, state)
.expect("version state should normalize"),
expected
);
}
}
#[test]
fn exact_request_routing_fails_closed_for_unknown_versions() {
for (version, expected) in [
(RemoteVersion::Disabled, None),
(RemoteVersion::SuspendedNull, Some("null")),
(RemoteVersion::Exact("opaque-v1".to_string()), Some("opaque-v1")),
] {
assert_eq!(version.exact_request_id().expect("known version state"), expected);
}
assert!(RemoteVersion::Unknown.exact_request_id().is_err());
}
#[test]
fn provider_version_rejects_empty_or_oversized_headers() {
let oversized = "v".repeat(1025);
+33 -2
View File
@@ -31,7 +31,7 @@ use crate::client::{
},
constants::{UNSIGNED_PAYLOAD, UNSIGNED_PAYLOAD_TRAILER},
credentials::{CredContext, Credentials, SignatureType, Static},
provider_versions::{BucketVersioningState, ProviderVersionCapabilities},
provider_versions::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion},
signer_error,
};
use crate::{client::checksum::ChecksumMode, object_api::GetObjectReader};
@@ -332,6 +332,22 @@ impl TransitionClient {
self.provider_version_capabilities().raw_version_id(headers)
}
pub(crate) fn remote_version(
&self,
headers: &HeaderMap,
versioning: BucketVersioningState,
) -> Result<RemoteVersion, std::io::Error> {
self.provider_version_capabilities().remote_version(headers, versioning)
}
pub(crate) fn legacy_remote_version_id(&self, headers: &HeaderMap) -> Result<String, std::io::Error> {
Ok(self
.remote_version(headers, BucketVersioningState::Unknown)?
.exact_id()
.unwrap_or_default()
.to_string())
}
fn trace_errors_only_off(&self) {
if let Ok(mut trace_errors_only) = self.trace_errors_only.lock() {
*trace_errors_only = false;
@@ -1095,6 +1111,16 @@ impl Default for ObjectInfo {
}
}
impl ObjectInfo {
pub(crate) fn remote_version(
&self,
capabilities: ProviderVersionCapabilities,
versioning: BucketVersioningState,
) -> Result<RemoteVersion, std::io::Error> {
capabilities.remote_version(&self.metadata, versioning)
}
}
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct RestoreInfo {
ongoing_restore: bool,
@@ -1414,7 +1440,7 @@ mod tests {
MAX_S3_CLIENT_RESPONSE_SIZE, MAX_S3_ERROR_RESPONSE_SIZE, SignatureType, build_tls_config, collect_response_body,
signer_error_to_io_error, to_object_info_for_provider, validate_header_values, with_rustls_init_guard,
};
use crate::client::provider_versions::ProviderVersionCapabilities;
use crate::client::provider_versions::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
use http::{HeaderMap, HeaderValue};
use http_body_util::Full;
use hyper::body::Bytes;
@@ -1539,6 +1565,11 @@ mod tests {
.expect("opaque provider version should parse");
assert_eq!(info.version_id, None);
assert_eq!(
info.remote_version(ProviderVersionCapabilities::for_tier_type("tencent"), BucketVersioningState::Enabled,)
.expect("opaque response version should remain available"),
RemoteVersion::Exact("opaque.version_01".to_string())
);
assert_eq!(
info.metadata.get("x-cos-version-id").and_then(|value| value.to_str().ok()),
Some("opaque.version_01")
+361 -3
View File
@@ -27,6 +27,7 @@
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q>
use crate::cluster::rpc::context_propagation::{inject_request_id_into_http_headers, inject_trace_context_into_http_headers};
use crate::storage_api_contracts::internode::NS_SCANNER_PROTOCOL_VERSION;
use base64::Engine as _;
use base64::engine::general_purpose;
use hmac::{Hmac, KeyInit, Mac};
@@ -35,6 +36,8 @@ use http::{HeaderMap, HeaderValue, Method, Uri};
#[cfg(test)]
use rustfs_credentials::{DEFAULT_SECRET_KEY, RPC_SECRET_REQUIRED_MESSAGE};
use rustfs_credentials::{RPC_SECRET_REQUIRED_OPERATOR_MESSAGE, try_get_rpc_token};
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_utils::get_env_bool;
use sha2::Digest as _;
use sha2::Sha256;
use std::collections::{HashSet, VecDeque};
@@ -58,8 +61,30 @@ const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(601);
const MAX_REPLAY_PROTECTED_NONCES: usize = 65_536;
const NS_SCANNER_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-capability-v3";
pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
static INTERNODE_RPC_SIGNATURE_STRICT: LazyLock<bool> = LazyLock::new(|| {
get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_SIGNATURE_STRICT,
rustfs_config::DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT,
)
});
static INTERNODE_RPC_BODY_DIGEST_STRICT: LazyLock<bool> = LazyLock::new(|| {
get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_BODY_DIGEST_STRICT,
rustfs_config::DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT,
)
});
// Sized for peak legitimate body-bound mutation RPS x the retention window; overflow fails closed
// and increments the replay-cache overflow counter. Clamped to at least 1 so a misconfigured zero
// cannot disable replay protection by rejecting every body-bound request.
static REPLAY_CACHE_CAPACITY: LazyLock<usize> = LazyLock::new(|| {
rustfs_utils::get_env_usize(
rustfs_config::ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY,
rustfs_config::DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY,
)
.max(1)
});
static RPC_SECRET_RESOLUTION_LOG_ONCE: Once = Once::new();
#[derive(Default)]
@@ -100,6 +125,10 @@ impl RpcNonceCache {
return Err(std::io::Error::other("RPC request replay detected"));
}
if self.nonces.len() >= capacity {
// Fail closed and alert: only legitimately signed traffic can fill the cache, so a
// sustained overflow means RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY is undersized
// for this node's peak mutation rate and writes are being refused.
global_internode_metrics().record_replay_cache_overflow();
return Err(std::io::Error::other("RPC replay cache capacity exceeded"));
}
self.nonces.insert(nonce);
@@ -201,6 +230,42 @@ fn verify_signature(secret: &str, url: &str, method: &Method, timestamp: i64, si
mac.verify_slice(&signature).is_ok()
}
fn update_ns_scanner_capability_mac(mac: &mut HmacSha256, challenge: Uuid, server_epoch: Uuid) {
mac.update(NS_SCANNER_CAPABILITY_AUTH_DOMAIN);
mac.update(&NS_SCANNER_PROTOCOL_VERSION.to_be_bytes());
mac.update(challenge.as_bytes());
mac.update(server_epoch.as_bytes());
}
fn generate_ns_scanner_capability_proof(secret: &str, challenge: Uuid, server_epoch: Uuid) -> std::io::Result<Vec<u8>> {
if challenge.is_nil() || server_epoch.is_nil() {
return Err(std::io::Error::other("Invalid namespace scanner capability scope"));
}
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_ns_scanner_capability_mac(&mut mac, challenge, server_epoch);
Ok(mac.finalize().into_bytes().to_vec())
}
fn verify_ns_scanner_capability_proof(secret: &str, challenge: Uuid, server_epoch: Uuid, proof: &[u8]) -> std::io::Result<()> {
if challenge.is_nil() || server_epoch.is_nil() {
return Err(std::io::Error::other("Invalid namespace scanner capability scope"));
}
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_ns_scanner_capability_mac(&mut mac, challenge, server_epoch);
mac.verify_slice(proof)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid namespace scanner capability proof"))
}
pub fn sign_ns_scanner_capability(challenge: Uuid, server_epoch: Uuid) -> std::io::Result<Vec<u8>> {
generate_ns_scanner_capability_proof(&get_shared_secret()?, challenge, server_epoch)
}
pub fn verify_ns_scanner_capability(challenge: Uuid, server_epoch: Uuid, proof: &[u8]) -> std::io::Result<()> {
verify_ns_scanner_capability_proof(&get_shared_secret()?, challenge, server_epoch, proof)
}
#[derive(Clone, Copy)]
struct SignatureV2Scope<'a> {
audience: &'a str,
@@ -288,7 +353,7 @@ fn check_and_record_nonce(nonce: Uuid, signed_at: i64) -> std::io::Result<()> {
let expires_at = now
.checked_add(REPLAY_CACHE_RETENTION)
.ok_or_else(|| std::io::Error::other("RPC replay expiry overflow"))?;
cache.check_and_record(nonce, signed_at, now, wall_time, expires_at, MAX_REPLAY_PROTECTED_NONCES)
cache.check_and_record(nonce, signed_at, now, wall_time, expires_at, *REPLAY_CACHE_CAPACITY)
}
/// Build headers with authentication signature
@@ -401,6 +466,50 @@ pub fn verify_tonic_canonical_body_digest<T>(request: &tonic::Request<T>, canoni
Ok(())
}
/// Verify a mutating disk RPC's canonical body digest with a rolling-upgrade fallback.
///
/// When the request carries a real (non-`UNSIGNED-PAYLOAD`) content SHA-256 it is verified exactly
/// like [`verify_tonic_canonical_body_digest`]. The digest value is a member of the signed v2
/// scope, so within the v2 lane it cannot be stripped or altered without invalidating the signature
/// `check_auth` already enforced. When the request carries no digest — a peer that predates
/// body-digest signing, or an attacker who downgraded the request to the legacy signature by
/// dropping every v2 header — the request is accepted and counted on the body-digest fallback
/// counter unless `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT` is enabled. That switch is what actually
/// closes on-path body tampering for covered handlers: it rejects every digestless mutation,
/// including v1-downgraded ones. It converges independently of the signature-strict switch
/// (<https://github.com/rustfs/backlog/issues/1327>).
pub fn verify_tonic_mutation_body_digest<T>(request: &tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
verify_tonic_mutation_body_digest_with_strictness(request, canonical_body, *INTERNODE_RPC_BODY_DIGEST_STRICT)
}
/// [`verify_tonic_mutation_body_digest`] with the strict gate injected as a parameter, so both
/// rollout postures are unit-testable without racing on process-global environment variables.
fn verify_tonic_mutation_body_digest_with_strictness<T>(
request: &tonic::Request<T>,
canonical_body: &[u8],
strict: bool,
) -> std::io::Result<()> {
let digest = request
.metadata()
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok());
match digest {
Some(digest) if digest != UNSIGNED_PAYLOAD => verify_tonic_canonical_body_digest(request, canonical_body),
_ => {
// RUSTFS_COMPAT_TODO(disk-mutation-body-digest): accept digestless peers during rolling upgrades. Remove after the
// minimum supported RustFS peer version body-binds every mutating disk RPC.
if strict {
return Err(std::io::Error::other("RPC mutation requires a body-bound v2 signature"));
}
// Count only ACCEPTED digestless mutations: this counter is the convergence gate that
// must read zero fleet-wide across a release window before
// `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT` may be enabled.
global_internode_metrics().record_body_digest_fallback();
Ok(())
}
}
}
fn has_v2_auth_headers(headers: &HeaderMap) -> bool {
[
RPC_AUTH_VERSION_HEADER,
@@ -412,12 +521,40 @@ fn has_v2_auth_headers(headers: &HeaderMap) -> bool {
.any(|name| headers.contains_key(*name))
}
/// Whether the server requires target-bound v2 authentication on every internode gRPC request,
/// rejecting the legacy constant-target fallback instead of accepting it. Default-off rollout
/// lever gated on the v1-fallback counter reading zero fleet-wide; see
/// [`rustfs_config::ENV_INTERNODE_RPC_SIGNATURE_STRICT`] and
/// <https://github.com/rustfs/backlog/issues/1327>.
fn internode_rpc_signature_strict() -> bool {
*INTERNODE_RPC_SIGNATURE_STRICT
}
/// Verify gRPC authentication, preferring v2 without downgrade on malformed v2 metadata.
pub fn verify_tonic_rpc_signature(audience: &str, path: &str, headers: &HeaderMap) -> std::io::Result<()> {
verify_tonic_rpc_signature_with_strictness(audience, path, headers, internode_rpc_signature_strict())
}
/// [`verify_tonic_rpc_signature`] with the strict gate injected as a parameter, so both rollout
/// postures are unit-testable without racing on process-global environment variables.
fn verify_tonic_rpc_signature_with_strictness(
audience: &str,
path: &str,
headers: &HeaderMap,
strict: bool,
) -> std::io::Result<()> {
if !has_v2_auth_headers(headers) {
// RUSTFS_COMPAT_TODO(heal-rpc-auth-v2): accept old peers during rolling upgrades. Remove after the minimum
// supported RustFS peer version sends v2 authentication on every internode gRPC request.
return verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, headers);
if strict {
return Err(std::io::Error::other("RPC v2 authentication required"));
}
verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, headers)?;
// Count only ACCEPTED legacy-only requests: this counter is the convergence gate that must
// read zero fleet-wide across a release window before
// `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT` may be enabled.
global_internode_metrics().record_signature_v1_fallback();
return Ok(());
}
let path = path
@@ -593,6 +730,20 @@ mod tests {
runtime_sources::ensure_test_rpc_secret();
}
#[test]
fn namespace_scanner_capability_proof_binds_challenge_and_server_epoch() {
let secret = "test-scanner-capability-secret";
let challenge = Uuid::new_v4();
let server_epoch = Uuid::new_v4();
let proof =
generate_ns_scanner_capability_proof(secret, challenge, server_epoch).expect("capability proof should be generated");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), server_epoch, &proof).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, Uuid::new_v4(), &proof).is_err());
assert!(verify_ns_scanner_capability_proof("different-secret", challenge, server_epoch, &proof).is_err());
}
/// Security regression for GHSA-r5qv-rc46-hv8q (internode RPC fail-closed,
/// fixed in rustfs/rustfs#4402): secret resolution must never silently fall
/// back to a default/empty shared secret. Missing and default secrets both
@@ -1111,7 +1262,10 @@ mod tests {
assert_eq!(error.to_string(), "Invalid RPC v2 signature");
}
// The `rpc_v1_fallback_counter` serial group covers every test that drives (or asserts on) the
// process-global v1-fallback counter, so exact-delta assertions cannot race with each other.
#[test]
#[serial_test::serial(rpc_v1_fallback_counter)]
fn legacy_tonic_signature_remains_accepted_during_rolling_upgrade() {
ensure_test_rpc_secret();
let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
@@ -1119,6 +1273,87 @@ mod tests {
assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers).is_ok());
}
#[test]
#[serial_test::serial(rpc_v1_fallback_counter)]
fn accepted_legacy_fallback_increments_v1_fallback_counter() {
ensure_test_rpc_secret();
let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
let before = global_internode_metrics().snapshot().signature_v1_fallback_total;
assert!(
verify_tonic_rpc_signature_with_strictness("node-a:9000", "/node_service.NodeService/Ping", &headers, false).is_ok(),
"a legacy-only peer must keep authenticating while the strict gate is off"
);
let after = global_internode_metrics().snapshot().signature_v1_fallback_total;
assert_eq!(
after,
before + 1,
"an accepted legacy-only request must increment the v1 fallback counter exactly once"
);
}
#[test]
#[serial_test::serial(rpc_v1_fallback_counter)]
fn rejected_legacy_fallback_does_not_count_as_v1_fallback() {
ensure_test_rpc_secret();
// Legacy-shaped headers with a forged signature: the fallback path runs but must reject,
// and a rejected request is not a rollout-convergence signal.
let mut headers = HeaderMap::new();
let now = OffsetDateTime::now_utc().unix_timestamp();
headers.insert(SIGNATURE_HEADER, HeaderValue::from_static("not-a-real-signature"));
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&now.to_string()).unwrap());
let before = global_internode_metrics().snapshot().signature_v1_fallback_total;
assert!(
verify_tonic_rpc_signature_with_strictness("node-a:9000", "/node_service.NodeService/Ping", &headers, false).is_err(),
"a forged legacy signature must still be rejected"
);
let after = global_internode_metrics().snapshot().signature_v1_fallback_total;
assert_eq!(after, before, "a rejected legacy request must not count as an accepted fallback");
}
#[test]
#[serial_test::serial(rpc_v1_fallback_counter)]
fn strict_gate_rejects_legacy_only_auth_but_keeps_v2() {
ensure_test_rpc_secret();
let legacy = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
let before = global_internode_metrics().snapshot().signature_v1_fallback_total;
let error = verify_tonic_rpc_signature_with_strictness("node-a:9000", "/node_service.NodeService/Ping", &legacy, true)
.expect_err("strict mode must reject legacy-only authentication");
assert_eq!(error.to_string(), "RPC v2 authentication required");
let v2 = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
.expect("tonic auth headers should build");
assert!(
verify_tonic_rpc_signature_with_strictness("node-a:9000", "/node_service.NodeService/Ping", &v2, true).is_ok(),
"strict mode must keep accepting v2-authenticated peers"
);
let after = global_internode_metrics().snapshot().signature_v1_fallback_total;
assert_eq!(after, before, "neither a strict rejection nor a v2 acceptance is a legacy fallback");
}
#[test]
#[serial_test::serial(rpc_v1_fallback_counter)]
fn strict_gate_default_posture_is_fail_open_legacy_accept() {
ensure_test_rpc_secret();
// The public entry point resolves strictness from the environment, whose compile-time
// default is pinned to false in `rustfs_config`. A legacy-only peer therefore keeps
// authenticating through the default build with no configuration at all.
let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
assert!(
verify_tonic_rpc_signature_with_strictness(
"node-a:9000",
"/node_service.NodeService/Ping",
&headers,
rustfs_config::DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT,
)
.is_ok(),
"the default strict posture must accept legacy-only peers"
);
}
#[test]
fn body_bound_tonic_request_rejects_replay_and_body_tampering() {
ensure_test_rpc_secret();
@@ -1238,6 +1473,129 @@ mod tests {
assert!(cache.nonces.contains(&nonce_b));
}
// The `rpc_body_digest_fallback_counter` serial group covers every test that drives (or
// asserts on) the process-global body-digest fallback counter, so exact-delta assertions
// cannot race with each other.
#[test]
#[serial_test::serial(rpc_body_digest_fallback_counter)]
fn digestless_mutation_is_accepted_and_counted_while_strict_gate_is_off() {
let request = tonic::Request::new(());
let before = global_internode_metrics().snapshot().body_digest_fallback_total;
assert!(
verify_tonic_mutation_body_digest_with_strictness(&request, b"canonical-mutation-body", false).is_ok(),
"a digestless peer must keep mutating while the strict gate is off"
);
let after = global_internode_metrics().snapshot().body_digest_fallback_total;
assert_eq!(
after,
before + 1,
"an accepted digestless mutation must increment the body-digest fallback counter exactly once"
);
}
#[test]
#[serial_test::serial(rpc_body_digest_fallback_counter)]
fn strict_mutation_gate_rejects_digestless_but_keeps_body_bound() {
let before = global_internode_metrics().snapshot().body_digest_fallback_total;
let digestless = tonic::Request::new(());
let error = verify_tonic_mutation_body_digest_with_strictness(&digestless, b"body", true)
.expect_err("strict mode must reject a mutation without a body digest");
assert_eq!(error.to_string(), "RPC mutation requires a body-bound v2 signature");
let mut unsigned = tonic::Request::new(());
unsigned
.metadata_mut()
.as_mut()
.insert(RPC_CONTENT_SHA256_HEADER, HeaderValue::from_static(UNSIGNED_PAYLOAD));
let error = verify_tonic_mutation_body_digest_with_strictness(&unsigned, b"body", true)
.expect_err("strict mode must reject an explicitly unsigned mutation payload");
assert_eq!(error.to_string(), "RPC mutation requires a body-bound v2 signature");
let mut bound = tonic::Request::new(());
set_tonic_canonical_body_digest(&mut bound, b"body").expect("digest metadata should encode");
bound
.metadata_mut()
.as_mut()
.insert(RPC_AUTH_VERSION_HEADER, HeaderValue::from_static(RPC_AUTH_VERSION_V2));
assert!(
verify_tonic_mutation_body_digest_with_strictness(&bound, b"body", true).is_ok(),
"strict mode must keep accepting body-bound mutations"
);
let tampered = verify_tonic_mutation_body_digest_with_strictness(&bound, b"tampered-body", true)
.expect_err("a tampered canonical body must fail even in strict mode");
assert_eq!(tampered.to_string(), "RPC content SHA-256 mismatch");
let after = global_internode_metrics().snapshot().body_digest_fallback_total;
assert_eq!(
after, before,
"neither strict rejections nor bound verifications are digestless fallbacks"
);
}
#[test]
#[serial_test::serial(rpc_body_digest_fallback_counter)]
fn mutation_digest_default_posture_is_fail_open_digestless_accept() {
// The public entry point resolves strictness from the environment, whose compile-time
// default is pinned to false in `rustfs_config`. A digestless peer therefore keeps
// mutating through the default build with no configuration at all.
let request = tonic::Request::new(());
assert!(
verify_tonic_mutation_body_digest_with_strictness(
&request,
b"canonical-mutation-body",
rustfs_config::DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT,
)
.is_ok(),
"the default strict posture must accept digestless mutations"
);
}
#[test]
fn rename_data_mutation_contract_binds_method_nonce_and_body() {
ensure_test_rpc_secret();
let message = rustfs_protos::proto_gen::node_service::RenameDataRequest {
disk: "http://node-a:9000/data/rustfs0".to_string(),
src_volume: ".rustfs.sys/multipart".to_string(),
src_path: "uploads/object".to_string(),
file_info: "{\"volume\":\"bucket\"}".to_string(),
dst_volume: "bucket".to_string(),
dst_path: "object".to_string(),
file_info_bin: vec![0x81, 0xA1, 0x76, 0x01].into(),
};
let body = rustfs_protos::canonical_rename_data_request_body(&message).expect("small request should encode");
let mut request = tonic::Request::new(());
set_tonic_canonical_body_digest(&mut request, &body).expect("canonical body digest should be attached");
let content_sha256 = request
.metadata()
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok());
let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "RenameData", content_sha256)
.expect("body-bound auth headers should build");
request.metadata_mut().as_mut().extend(headers.clone());
assert!(
verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/RenameData", &headers).is_ok(),
"the rename_data signature must bind destination, method, nonce, and body digest"
);
let replay = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/RenameData", &headers)
.expect_err("reusing a consumed rename_data nonce must fail");
assert_eq!(replay.to_string(), "RPC request replay detected");
let transplant = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/DeleteVersion", &headers)
.expect_err("a rename_data signature must not authenticate a different method");
assert_eq!(transplant.to_string(), "Invalid RPC v2 signature");
assert!(verify_tonic_mutation_body_digest(&request, &body).is_ok());
let mut tampered = message;
tampered.file_info_bin = Vec::new().into();
let tampered_body = rustfs_protos::canonical_rename_data_request_body(&tampered).expect("small request should encode");
let stripped = verify_tonic_mutation_body_digest(&request, &tampered_body)
.expect_err("stripping the msgpack payload to force the JSON fallback decode must fail");
assert_eq!(stripped.to_string(), "RPC content SHA-256 mismatch");
}
#[test]
fn nonce_cache_rejects_replay_after_wall_clock_regression() {
let now = Instant::now();
@@ -12,11 +12,14 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::cluster::rpc::build_auth_headers;
use crate::cluster::rpc::{build_auth_headers, verify_ns_scanner_capability};
use crate::disk::error::{Error, Result};
use crate::disk::{FileReader, FileWriter};
use crate::storage_api_contracts::internode::{
WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY,
NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY, NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY,
NS_SCANNER_SESSION_ID_QUERY, NS_SCANNER_SESSION_SEQUENCE_QUERY, NsScannerCapabilityResponse, WALK_DIR_BODY_SHA256_QUERY,
WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
};
use async_trait::async_trait;
use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
@@ -28,13 +31,18 @@ use rustfs_rio::{HttpReader, HttpWriter};
use sha2::{Digest, Sha256};
use std::sync::{Arc, OnceLock};
use std::time::Duration;
use tokio::io::AsyncReadExt;
use uuid::Uuid;
static INTERNODE_DATA_TRANSPORT: OnceLock<std::result::Result<Arc<dyn InternodeDataTransport>, String>> = OnceLock::new();
const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
const PUT_FILE_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream";
const WALK_DIR_PATH: &str = "/rustfs/rpc/walk_dir";
const NS_SCANNER_PATH: &str = "/rustfs/rpc/ns_scanner";
const NS_SCANNER_MAX_CAPABILITY_RESPONSE_SIZE: usize = 1024;
const CONTENT_TYPE_JSON: &str = "application/json";
const CONTENT_TYPE_MSGPACK: &str = "application/msgpack";
fn unsupported_transport_message(transport: &str) -> String {
format!(
@@ -101,6 +109,25 @@ pub struct WalkDirStreamRequest {
pub stall_timeout: Option<Duration>,
}
#[derive(Debug, Clone)]
pub struct NsScannerStreamRequest {
pub endpoint: String,
pub disk: String,
pub request_id: Uuid,
pub server_epoch: Uuid,
pub session_id: Uuid,
pub session_sequence: u64,
pub next_cycle: u64,
pub leader_epoch: u64,
pub body: Vec<u8>,
pub stall_timeout: Option<Duration>,
}
#[derive(Debug, Clone)]
pub struct NsScannerCapabilityRequest {
pub endpoint: String,
}
/// Data-plane stream opener used by `RemoteDisk`.
///
/// This boundary is limited to remote disk streams that can move large payloads.
@@ -114,6 +141,12 @@ pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
async fn open_ns_scanner(&self, _request: NsScannerStreamRequest) -> Result<FileReader> {
Err(Error::MethodNotAllowed)
}
async fn probe_ns_scanner(&self, _request: NsScannerCapabilityRequest) -> Result<Uuid> {
Err(Error::MethodNotAllowed)
}
fn name(&self) -> &'static str;
fn capabilities(&self) -> InternodeDataTransportCapabilities;
}
@@ -148,6 +181,39 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
))
}
async fn open_ns_scanner(&self, request: NsScannerStreamRequest) -> Result<FileReader> {
let url = build_ns_scanner_url(&request);
let mut headers = msgpack_headers();
build_auth_headers(&url, &Method::POST, &mut headers)?;
Ok(Box::new(
HttpReader::new_with_stall_timeout(url, Method::POST, headers, Some(request.body), request.stall_timeout).await?,
))
}
async fn probe_ns_scanner(&self, request: NsScannerCapabilityRequest) -> Result<Uuid> {
let challenge = Uuid::new_v4();
let url = build_ns_scanner_capability_url(&request, challenge);
let mut headers = msgpack_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
let reader = HttpReader::new(url, Method::GET, headers, None).await?;
let mut body = Vec::new();
reader
.take(u64::try_from(NS_SCANNER_MAX_CAPABILITY_RESPONSE_SIZE + 1).unwrap_or(u64::MAX))
.read_to_end(&mut body)
.await?;
if body.is_empty() || body.len() > NS_SCANNER_MAX_CAPABILITY_RESPONSE_SIZE {
return Err(Error::other("invalid remote namespace scanner capability response size"));
}
let response: NsScannerCapabilityResponse =
rmp_serde::from_slice(&body).map_err(|_| Error::other("invalid remote namespace scanner capability response"))?;
if response.version != NS_SCANNER_PROTOCOL_VERSION || response.server_epoch.is_nil() {
return Err(Error::other("incompatible remote namespace scanner capability response"));
}
verify_ns_scanner_capability(challenge, response.server_epoch, &response.proof)
.map_err(|err| Error::other(format!("remote namespace scanner capability authentication failed: {err}")))?;
Ok(response.server_epoch)
}
fn name(&self) -> &'static str {
DEFAULT_INTERNODE_DATA_TRANSPORT
}
@@ -197,12 +263,54 @@ fn build_walk_dir_url(request: &WalkDirStreamRequest) -> String {
)
}
fn build_ns_scanner_url(request: &NsScannerStreamRequest) -> String {
let body_sha256 = hex_simd::encode_to_string(Sha256::digest(&request.body), hex_simd::AsciiCase::Lower);
format!(
"{}{}?disk={}&{}={}&{}={}&{}={}&{}={}&{}={}&{}={}&{}={}",
request.endpoint,
NS_SCANNER_PATH,
urlencoding::encode(&request.disk),
NS_SCANNER_REQUEST_ID_QUERY,
request.request_id,
NS_SCANNER_SERVER_EPOCH_QUERY,
request.server_epoch,
NS_SCANNER_SESSION_ID_QUERY,
request.session_id,
NS_SCANNER_SESSION_SEQUENCE_QUERY,
request.session_sequence,
NS_SCANNER_CYCLE_QUERY,
request.next_cycle,
NS_SCANNER_LEADER_EPOCH_QUERY,
request.leader_epoch,
NS_SCANNER_BODY_SHA256_QUERY,
body_sha256
)
}
fn build_ns_scanner_capability_url(request: &NsScannerCapabilityRequest, challenge: Uuid) -> String {
format!(
"{}{}?{}={}&{}={}",
request.endpoint,
NS_SCANNER_PATH,
NS_SCANNER_PROTOCOL_VERSION_QUERY,
NS_SCANNER_PROTOCOL_VERSION,
NS_SCANNER_CAPABILITY_CHALLENGE_QUERY,
challenge
)
}
fn json_headers() -> HeaderMap {
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static(CONTENT_TYPE_JSON));
headers
}
fn msgpack_headers() -> HeaderMap {
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static(CONTENT_TYPE_MSGPACK));
headers
}
fn build_internode_data_transport_result(
configured_transport: Option<&str>,
) -> std::result::Result<Arc<dyn InternodeDataTransport>, String> {
@@ -241,6 +349,65 @@ pub fn build_internode_data_transport_from_env() -> Result<Arc<dyn InternodeData
mod tests {
use super::*;
#[derive(Debug)]
struct LegacyTestTransport;
#[async_trait::async_trait]
impl InternodeDataTransport for LegacyTestTransport {
async fn open_read(&self, _request: ReadStreamRequest) -> Result<FileReader> {
Ok(Box::new(tokio::io::empty()))
}
async fn open_write(&self, _request: WriteStreamRequest) -> Result<FileWriter> {
Ok(Box::new(tokio::io::sink()))
}
async fn open_walk_dir(&self, _request: WalkDirStreamRequest) -> Result<FileReader> {
Ok(Box::new(tokio::io::empty()))
}
fn name(&self) -> &'static str {
"legacy-test"
}
fn capabilities(&self) -> InternodeDataTransportCapabilities {
InternodeDataTransportCapabilities::tcp_http()
}
}
#[tokio::test]
async fn legacy_transport_defaults_namespace_scanner_to_unsupported() {
let transport = LegacyTestTransport;
let probe_err = transport
.probe_ns_scanner(NsScannerCapabilityRequest {
endpoint: "http://node1:9000".to_string(),
})
.await
.expect_err("legacy transport should report namespace scanner as unsupported");
assert!(matches!(probe_err, Error::MethodNotAllowed));
let open_result = transport
.open_ns_scanner(NsScannerStreamRequest {
endpoint: "http://node1:9000".to_string(),
disk: "http://node1:9000/data/rustfs0".to_string(),
request_id: Uuid::new_v4(),
server_epoch: Uuid::new_v4(),
session_id: Uuid::new_v4(),
session_sequence: 0,
next_cycle: 7,
leader_epoch: 9,
body: Vec::new(),
stall_timeout: None,
})
.await;
let open_err = match open_result {
Ok(_) => panic!("legacy transport should not open namespace scanner streams"),
Err(err) => err,
};
assert!(matches!(open_err, Error::MethodNotAllowed));
}
#[test]
fn tcp_http_capabilities_are_behavior_preserving() {
let transport = TcpHttpInternodeDataTransport;
@@ -322,6 +489,57 @@ mod tests {
);
}
#[test]
fn ns_scanner_url_binds_body_and_encodes_disk_ref() {
let request_id = Uuid::parse_str("11111111-2222-4333-8444-555555555555").expect("request ID");
let server_epoch = Uuid::parse_str("aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee").expect("server epoch");
let session_id = Uuid::parse_str("99999999-8888-4777-8666-555555555555").expect("session ID");
let url = build_ns_scanner_url(&NsScannerStreamRequest {
endpoint: "http://node1:9000".to_string(),
disk: "http://node1:9000/data/rustfs0".to_string(),
request_id,
server_epoch,
session_id,
session_sequence: 3,
next_cycle: 7,
leader_epoch: 9,
body: b"scanner-request".to_vec(),
stall_timeout: None,
});
assert_eq!(
url,
concat!(
"http://node1:9000/rustfs/rpc/ns_scanner?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0",
"&ns_scanner_request_id=11111111-2222-4333-8444-555555555555",
"&ns_scanner_server_epoch=aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee",
"&ns_scanner_session_id=99999999-8888-4777-8666-555555555555",
"&ns_scanner_session_sequence=3",
"&ns_scanner_cycle=7",
"&ns_scanner_leader_epoch=9",
"&ns_scanner_body_sha256=c958f15ca28422275c1245399f4c44eaba628ca453fcd77d6b3d4484573e4387"
)
);
}
#[test]
fn ns_scanner_capability_url_binds_version_and_challenge() {
let challenge = Uuid::parse_str("12345678-1234-4234-8234-123456789abc").expect("challenge");
let url = build_ns_scanner_capability_url(
&NsScannerCapabilityRequest {
endpoint: "http://node1:9000".to_string(),
},
challenge,
);
assert_eq!(
url,
format!(
"http://node1:9000/rustfs/rpc/ns_scanner?ns_scanner_protocol={NS_SCANNER_PROTOCOL_VERSION}&ns_scanner_challenge={challenge}"
)
);
}
#[test]
fn transport_config_defaults_to_tcp_http() {
let transport = build_internode_data_transport(None).unwrap();
+5 -3
View File
@@ -30,17 +30,19 @@ pub use client::{
};
pub use http_auth::{
TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience,
set_tonic_canonical_body_digest, sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, verify_ns_scanner_capability,
verify_rpc_signature, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
pub use internode_data_transport::build_internode_data_transport_from_env;
pub(crate) use peer_rest_client::TierConfigReloadOutcome;
pub use peer_rest_client::{
PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG,
SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerPeerActivity,
};
pub(crate) use peer_s3_client::heal_bucket_local_on_disks;
pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, S3PeerSys};
pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, S3PeerSys, ScannerBucketListing, ScannerSetBucketListing};
pub use remote_disk::RemoteDisk;
pub use remote_locker::RemoteClient;
@@ -18,7 +18,11 @@ use crate::cluster::rpc::client::{
};
use crate::cluster::rpc::{set_tonic_canonical_body_digest, verify_tonic_rpc_response_proof};
use crate::error::{Error, Result};
use crate::storage_api_contracts::internode::{
SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION, SCANNER_ACTIVITY_PROTOCOL_VERSION,
};
use crate::{
bucket::replication::BucketStats,
disk::disk_store::{get_drive_active_check_interval, get_drive_active_check_timeout},
layout::endpoints::EndpointServerPools,
runtime::sources as runtime_sources,
@@ -26,6 +30,7 @@ use crate::{
};
use bytes::Bytes;
use rmp_serde::{Deserializer, Serializer};
use rustfs_config::{HEAL_SUB_SYS, SCANNER_SUB_SYS};
use rustfs_madmin::{
ServerProperties,
health::{Cpus, MemInfo, OsInfo, Partitions, ProcInfo, SysConfig, SysErrors, SysServices},
@@ -34,14 +39,15 @@ use rustfs_madmin::{
};
use rustfs_protos::proto_gen::node_service::{
BackgroundHealStatusRequest, CancelDecommissionRequest, ClearDecommissionRequest, DeleteBucketMetadataRequest,
DeletePolicyRequest, DeleteServiceAccountRequest, DeleteUserRequest, GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest,
GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest,
GetSysConfigRequest, GetSysErrorsRequest, HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest,
LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest,
LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest,
ReloadSiteReplicationConfigRequest, ScannerActivityRequest, ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest,
StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest,
TierMutationControlResponse, TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient,
DeletePolicyRequest, DeleteServiceAccountRequest, DeleteUserRequest, GetBucketStatsDataRequest, GetBucketStatsDataResponse,
GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest, GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest,
GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSysConfigRequest, GetSysErrorsRequest,
HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest,
LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest,
LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, ScannerActivityRequest,
ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest, StartDecommissionRequest, StartProfilingRequest,
StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest, TierMutationControlResponse,
TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient,
tier_mutation_control_service_client::TierMutationControlServiceClient,
};
use rustfs_protos::{TierMutationRpcPhase, evict_failed_connection};
@@ -74,16 +80,55 @@ const HEAL_CONTROL_PAYLOAD_MAX_SIZE: usize = 64 * 1024;
const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
const SCANNER_ACTIVITY_MAX_MESSAGE_SIZE: usize = 1024;
const REPLICATION_STATS_MAX_MESSAGE_SIZE: usize = 8 * 1024 * 1024;
fn decode_bucket_stats_response(response: GetBucketStatsDataResponse) -> Result<BucketStats> {
if !response.success {
return Err(Error::other(
response
.error_info
.unwrap_or_else(|| "peer replication statistics provider is unavailable".to_string()),
));
}
if response.bucket_stats.len() > REPLICATION_STATS_MAX_MESSAGE_SIZE {
return Err(Error::other("peer replication statistics response exceeds size limit"));
}
let mut buf = Deserializer::new(Cursor::new(response.bucket_stats));
let stats = BucketStats::deserialize(&mut buf).map_err(Error::from)?;
if !stats.replication_stats.provider_available {
return Err(Error::other("peer replication statistics provider is unavailable"));
}
Ok(stats)
}
fn validate_signal_service_protocol(sig: u64, sub_sys: &str, protocol_version: u32) -> Result<()> {
if sig == SERVICE_SIGNAL_RELOAD_DYNAMIC
&& matches!(sub_sys, SCANNER_SUB_SYS | HEAL_SUB_SYS)
&& protocol_version < rustfs_protos::DYNAMIC_CONFIG_PROTOCOL_VERSION
{
return Err(Error::other(format!("peer does not support dynamic {sub_sys} config convergence")));
}
Ok(())
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ScannerPeerActivity {
pub instance_id: String,
pub namespace_generation: u64,
pub maintenance_generation: u64,
pub protocol_version: u32,
pub topology_digest: Option<[u8; 32]>,
pub data_movement_active: Option<bool>,
pub dirty_usage_generation: Option<u64>,
pub dirty_usage_pending: Option<bool>,
}
fn decode_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerPeerActivity> {
let instance_id = response.instance_id;
fn decode_scanner_activity_with_verifier(
response: ScannerActivityResponse,
challenge: &[u8; 16],
verify_proof: impl FnOnce(&[u8], &[u8]) -> Result<()>,
) -> Result<ScannerPeerActivity> {
let instance_id = &response.instance_id;
if instance_id.len() != 32
|| !instance_id
.as_bytes()
@@ -92,10 +137,80 @@ fn decode_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerP
{
return Err(Error::other("peer returned an invalid scanner activity instance ID"));
}
let (topology_digest, data_movement_active, dirty_usage_generation, dirty_usage_pending) = match response.protocol_version {
// RUSTFS_COMPAT_TODO(ns-scanner-rpc-v3): legacy response fields are unauthenticated. Remove after protocol v0 peers are unsupported.
SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION
if response.topology_digest.is_empty()
&& response.response_proof.is_empty()
&& !response.data_movement_active
&& response.dirty_usage_generation == 0
&& !response.dirty_usage_pending =>
{
(None, None, None, None)
}
SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION => {
return Err(Error::other("legacy scanner activity peer returned unexpected extended fields"));
}
SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION => {
if response.dirty_usage_generation != 0 || response.dirty_usage_pending {
return Err(Error::other("scanner activity protocol v4 peer returned unauthenticated v5 fields"));
}
let canonical = rustfs_protos::canonical_scanner_activity_v4_response_body(challenge, &response)
.map_err(|_| Error::other("peer scanner activity response is too large to authenticate"))?;
verify_proof(&canonical, &response.response_proof)?;
(
Some(
response
.topology_digest
.as_ref()
.try_into()
.map_err(|_| Error::other("peer returned an invalid scanner topology digest"))?,
),
Some(response.data_movement_active),
None,
None,
)
}
SCANNER_ACTIVITY_PROTOCOL_VERSION => {
if response.dirty_usage_pending && response.dirty_usage_generation == 0 {
return Err(Error::other("scanner activity peer returned pending dirty usage without a generation"));
}
let canonical = rustfs_protos::canonical_scanner_activity_response_body(challenge, &response)
.map_err(|_| Error::other("peer scanner activity response is too large to authenticate"))?;
verify_proof(&canonical, &response.response_proof)?;
(
Some(
response
.topology_digest
.as_ref()
.try_into()
.map_err(|_| Error::other("peer returned an invalid scanner topology digest"))?,
),
Some(response.data_movement_active),
Some(response.dirty_usage_generation),
Some(response.dirty_usage_pending),
)
}
version => {
return Err(Error::other(format!("peer returned unsupported scanner activity protocol {version}")));
}
};
Ok(ScannerPeerActivity {
instance_id,
instance_id: response.instance_id,
namespace_generation: response.namespace_generation,
maintenance_generation: response.maintenance_generation,
protocol_version: response.protocol_version,
topology_digest,
data_movement_active,
dirty_usage_generation,
dirty_usage_pending,
})
}
fn decode_scanner_activity(response: ScannerActivityResponse, challenge: &[u8; 16]) -> Result<ScannerPeerActivity> {
decode_scanner_activity_with_verifier(response, challenge, |canonical, proof| {
verify_tonic_rpc_response_proof(canonical, proof)
.map_err(|_| Error::other("peer returned an invalid scanner activity response proof"))
})
}
@@ -819,9 +934,26 @@ impl PeerRestClient {
Err(Error::NotImplemented)
}
pub async fn get_bucket_stats(&self) -> Result<()> {
warn!("get_bucket_stats is not implemented in PeerRestClient");
Err(Error::NotImplemented)
pub async fn get_bucket_stats(&self, bucket: &str) -> Result<BucketStats> {
let response = self
.finalize_result(
async {
let mut client = self
.get_client()
.await?
.max_decoding_message_size(REPLICATION_STATS_MAX_MESSAGE_SIZE);
let response = client
.get_bucket_stats(Request::new(GetBucketStatsDataRequest {
bucket: bucket.to_string(),
}))
.await?
.into_inner();
Ok(response)
}
.await,
)
.await?;
decode_bucket_stats_response(response)
}
pub async fn get_sr_metrics(&self) -> Result<()> {
@@ -1292,6 +1424,7 @@ impl PeerRestClient {
}
return Err(Error::other(""));
}
validate_signal_service_protocol(sig, sub_sys, response.protocol_version)?;
Ok(())
}
.await,
@@ -1299,25 +1432,44 @@ impl PeerRestClient {
.await
}
pub async fn scanner_activity(&self) -> Result<ScannerPeerActivity> {
async fn scanner_activity_request(
&self,
acknowledge_instance_id: String,
acknowledge_dirty_usage_generation: u64,
) -> Result<ScannerPeerActivity> {
self.finalize_result(
async {
let challenge = Uuid::new_v4();
let mut client = self
.get_client()
.await?
.max_decoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE)
.max_encoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE);
let response = client
.scanner_activity(Request::new(ScannerActivityRequest {}))
.await?
.into_inner();
decode_scanner_activity(response)
let mut request = Request::new(ScannerActivityRequest {
challenge: challenge.as_bytes().to_vec().into(),
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
acknowledge_instance_id,
acknowledge_dirty_usage_generation,
});
let canonical = rustfs_protos::canonical_scanner_activity_request_body(request.get_ref())
.map_err(|_| Error::other("scanner activity request is too large to authenticate"))?;
set_tonic_canonical_body_digest(&mut request, &canonical)?;
let response = client.scanner_activity(request).await?.into_inner();
decode_scanner_activity(response, challenge.as_bytes())
}
.await,
)
.await
}
pub async fn scanner_activity(&self) -> Result<ScannerPeerActivity> {
self.scanner_activity_request(String::new(), 0).await
}
pub async fn acknowledge_scanner_dirty_usage(&self, instance_id: String, generation: u64) -> Result<ScannerPeerActivity> {
self.scanner_activity_request(instance_id, generation).await
}
pub async fn get_metacache_listing(&self) -> Result<()> {
warn!("get_metacache_listing is not implemented in PeerRestClient");
Err(Error::NotImplemented)
@@ -1479,35 +1631,177 @@ impl PeerRestClient {
}
pub async fn load_transition_tier_config(&self) -> Result<()> {
self.finalize_result(
async {
let mut client = self.get_client().await?;
let request = Request::new(LoadTransitionTierConfigRequest {});
let response = client.load_transition_tier_config(request).await?.into_inner();
if !response.success {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
}
Ok(())
match self.load_transition_tier_config_outcome().await {
TierConfigReloadOutcome::Success => Ok(()),
TierConfigReloadOutcome::TransientReconnect(err) | TierConfigReloadOutcome::TransientRetrySameChannel(err) => {
self.finalize_result(Err(err)).await
}
.await,
)
.await
TierConfigReloadOutcome::Terminal(err) => Err(err),
}
}
pub(crate) async fn load_transition_tier_config_outcome(&self) -> TierConfigReloadOutcome {
let outcome = self.load_transition_tier_config_single_attempt_outcome().await;
if outcome.is_transient() {
return self.load_transition_tier_config_once_outcome().await;
}
outcome
}
pub(crate) async fn load_transition_tier_config_single_attempt_outcome(&self) -> TierConfigReloadOutcome {
let outcome = self.load_transition_tier_config_once_outcome().await;
if outcome.requires_reconnect() {
self.prepare_retry().await;
}
outcome
}
pub(crate) async fn load_transition_tier_config_once_outcome(&self) -> TierConfigReloadOutcome {
let mut client = match self.get_client().await {
Ok(client) => client,
Err(err) => return tier_config_reload_connection_outcome(err),
};
let mut request = Request::new(LoadTransitionTierConfigRequest {});
request.set_timeout(rustfs_protos::heal_control_execution_timeout());
let response = match client.load_transition_tier_config(request).await {
Ok(response) => response.into_inner(),
Err(status) => return tier_config_reload_status_outcome(status),
};
if !response.success {
return tier_config_reload_remote_failure(response.error_info);
}
TierConfigReloadOutcome::Success
}
}
pub(crate) enum TierConfigReloadOutcome {
Success,
TransientReconnect(Error),
TransientRetrySameChannel(Error),
Terminal(Error),
}
impl TierConfigReloadOutcome {
fn is_transient(&self) -> bool {
matches!(self, Self::TransientReconnect(_) | Self::TransientRetrySameChannel(_))
}
fn requires_reconnect(&self) -> bool {
matches!(self, Self::TransientReconnect(_))
}
}
fn tier_config_reload_connection_outcome(err: Error) -> TierConfigReloadOutcome {
if is_tier_config_reload_connection_failure(&err) {
TierConfigReloadOutcome::TransientReconnect(err)
} else {
TierConfigReloadOutcome::Terminal(err)
}
}
fn is_tier_config_reload_connection_failure(err: &Error) -> bool {
let message = err.to_string().to_ascii_lowercase();
if message
.split_once("can not get client, err:")
.is_some_and(|(_, local_error)| local_error.contains("unavailable"))
{
return true;
}
[
"temporarily offline",
"transport error",
"error trying to connect",
"connection refused",
"connection reset",
"connection closed",
"connection aborted",
"broken pipe",
"not connected",
"unexpected eof",
"timed out",
"deadline has elapsed",
"tcp connect error",
]
.iter()
.any(|needle| message.contains(needle))
}
fn tier_config_reload_remote_failure(error_info: Option<String>) -> TierConfigReloadOutcome {
let error_info = error_info.unwrap_or_default();
if matches!(error_info.as_str(), "errServerNotInitialized" | "ServerNotInitialized") {
TierConfigReloadOutcome::TransientRetrySameChannel(Error::other(error_info))
} else {
TierConfigReloadOutcome::Terminal(Error::other(error_info))
}
}
fn tier_config_reload_status_outcome(status: tonic::Status) -> TierConfigReloadOutcome {
use tonic::Code;
if matches!(status.code(), Code::Unavailable | Code::DeadlineExceeded) {
TierConfigReloadOutcome::TransientReconnect(status.into())
} else if status.code() == Code::Unknown && status.message().starts_with("Service was not ready:") {
TierConfigReloadOutcome::TransientRetrySameChannel(status.into())
} else {
TierConfigReloadOutcome::Terminal(status.into())
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::com::STORAGE_CLASS_SUB_SYS;
use serde_json::Value;
use std::io::{self, Write};
use std::sync::{Arc, Mutex};
use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
#[test]
fn replication_stats_response_decodes_valid_empty_provider() {
let mut stats = BucketStats::default();
stats.replication_stats.provider_available = true;
let payload = rmp_serde::to_vec_named(&stats).expect("bucket statistics should encode");
let decoded = decode_bucket_stats_response(GetBucketStatsDataResponse {
success: true,
bucket_stats: payload.into(),
error_info: None,
})
.expect("valid bucket statistics should decode");
assert!(decoded.replication_stats.provider_available);
assert!(decoded.replication_stats.stats.is_empty());
}
#[test]
fn replication_stats_response_rejects_unavailable_malformed_and_oversized_payloads() {
let unavailable = decode_bucket_stats_response(GetBucketStatsDataResponse {
success: false,
bucket_stats: Bytes::new(),
error_info: Some("provider unavailable".to_string()),
})
.expect_err("unavailable provider must not become a zero snapshot");
assert!(unavailable.to_string().contains("provider unavailable"));
let malformed = decode_bucket_stats_response(GetBucketStatsDataResponse {
success: true,
bucket_stats: Bytes::from_static(b"not-msgpack"),
error_info: None,
})
.expect_err("malformed peer statistics must fail closed");
assert!(!malformed.to_string().is_empty());
let oversized = decode_bucket_stats_response(GetBucketStatsDataResponse {
success: true,
bucket_stats: Bytes::from(vec![0; REPLICATION_STATS_MAX_MESSAGE_SIZE + 1]),
error_info: None,
})
.expect_err("oversized peer statistics must fail closed");
assert!(oversized.to_string().contains("size limit"));
}
#[derive(Clone, Default)]
struct CapturedLogs {
buffer: Arc<Mutex<Vec<u8>>>,
@@ -1567,6 +1861,14 @@ mod tests {
)
}
fn decode_test_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerPeerActivity> {
decode_scanner_activity_with_verifier(response, &[9; 16], |_canonical, proof| {
(proof == b"proof")
.then_some(())
.ok_or_else(|| Error::other("peer returned an invalid scanner activity response proof"))
})
}
#[test]
fn build_clients_from_slots_preserves_missing_remote_topology_slots() {
let slots = vec![
@@ -1599,13 +1901,89 @@ mod tests {
#[test]
fn scanner_activity_requires_restart_safe_peer_identity() {
let legacy = decode_test_scanner_activity(ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION,
topology_digest: Vec::new().into(),
data_movement_active: false,
response_proof: Vec::new().into(),
dirty_usage_generation: 0,
dirty_usage_pending: false,
})
.expect("legacy peers should retain their activity generations during a rolling upgrade");
assert_eq!(
legacy,
ScannerPeerActivity {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION,
topology_digest: None,
data_movement_active: None,
dirty_usage_generation: None,
dirty_usage_pending: None,
}
);
let previous = decode_test_scanner_activity(ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: true,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 0,
dirty_usage_pending: false,
})
.expect("protocol v4 peers should remain observable during a rolling upgrade");
assert_eq!(
previous,
ScannerPeerActivity {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
topology_digest: Some([7; 32]),
data_movement_active: Some(true),
dirty_usage_generation: None,
dirty_usage_pending: None,
}
);
let malformed_topology = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 31].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_test_scanner_activity(malformed_topology)
.expect_err("activity topology digests must have the protocol-defined length")
.to_string()
.contains("topology digest")
);
let missing_instance = ScannerActivityResponse {
instance_id: String::new(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_scanner_activity(missing_instance)
decode_test_scanner_activity(missing_instance)
.expect_err("an empty instance ID is not restart safe")
.to_string()
.contains("instance ID")
@@ -1615,18 +1993,30 @@ mod tests {
instance_id: "ABCDEF0123456789ABCDEF0123456789".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_scanner_activity(malformed_instance)
decode_test_scanner_activity(malformed_instance)
.expect_err("activity instance IDs must use the canonical lowercase hex form")
.to_string()
.contains("instance ID")
);
let activity = decode_scanner_activity(ScannerActivityResponse {
let activity = decode_test_scanner_activity(ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: true,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
})
.expect("complete activity responses should be accepted");
assert_eq!(
@@ -1635,8 +2025,123 @@ mod tests {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: Some([7; 32]),
data_movement_active: Some(true),
dirty_usage_generation: Some(11),
dirty_usage_pending: Some(true),
}
);
let pending_without_generation = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 0,
dirty_usage_pending: true,
};
assert!(
decode_test_scanner_activity(pending_without_generation)
.expect_err("pending dirty usage must carry a nonzero generation")
.to_string()
.contains("without a generation")
);
let previous_with_dirty_usage = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_test_scanner_activity(previous_with_dirty_usage)
.expect_err("protocol v4 responses must not claim unauthenticated dirty usage fields")
.to_string()
.contains("unauthenticated v5 fields")
);
let legacy_with_topology = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 0,
dirty_usage_pending: false,
};
assert!(
decode_test_scanner_activity(legacy_with_topology)
.expect_err("legacy protocol responses must not claim extended fields")
.to_string()
.contains("unexpected extended fields")
);
let unsupported_protocol = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION + 1,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: b"proof".to_vec().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_test_scanner_activity(unsupported_protocol)
.expect_err("unknown activity protocols must fail closed")
.to_string()
.contains("unsupported scanner activity protocol")
);
let missing_proof = ScannerActivityResponse {
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
namespace_generation: 7,
maintenance_generation: 3,
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
topology_digest: vec![7; 32].into(),
data_movement_active: false,
response_proof: Vec::new().into(),
dirty_usage_generation: 11,
dirty_usage_pending: true,
};
assert!(
decode_test_scanner_activity(missing_proof)
.expect_err("unsigned scanner activity responses must fail closed")
.to_string()
.contains("response proof")
);
}
#[test]
fn dynamic_scanner_config_requires_versioned_peer_acknowledgement() {
for sub_system in [SCANNER_SUB_SYS, HEAL_SUB_SYS] {
let err = validate_signal_service_protocol(SERVICE_SIGNAL_RELOAD_DYNAMIC, sub_system, 0)
.expect_err("an unversioned peer must not claim scanner config convergence");
assert!(err.to_string().contains("does not support dynamic"));
validate_signal_service_protocol(
SERVICE_SIGNAL_RELOAD_DYNAMIC,
sub_system,
rustfs_protos::DYNAMIC_CONFIG_PROTOCOL_VERSION,
)
.expect("a current peer should support dynamic scanner config");
}
validate_signal_service_protocol(SERVICE_SIGNAL_RELOAD_DYNAMIC, STORAGE_CLASS_SUB_SYS, 0)
.expect("unrelated dynamic config keeps its existing compatibility contract");
validate_signal_service_protocol(SERVICE_SIGNAL_REFRESH_CONFIG, SCANNER_SUB_SYS, 0)
.expect("full refresh compatibility is guarded by its scanner preflight");
}
#[test]
@@ -1646,6 +2151,73 @@ mod tests {
assert!(!PeerRestClient::is_network_like_error(&Error::NotImplemented));
}
#[test]
fn tier_config_reload_outcome_keeps_tonic_and_remote_errors_typed() {
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::unavailable("peer offline")),
TierConfigReloadOutcome::TransientReconnect(_)
));
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::deadline_exceeded("peer timeout")),
TierConfigReloadOutcome::TransientReconnect(_)
));
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::permission_denied("bad signature")),
TierConfigReloadOutcome::Terminal(_)
));
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::unknown("Service was not ready: test client")),
TierConfigReloadOutcome::TransientRetrySameChannel(_)
));
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::unknown("peer response unknown")),
TierConfigReloadOutcome::Terminal(_)
));
assert!(matches!(
tier_config_reload_status_outcome(tonic::Status::cancelled("request cancelled")),
TierConfigReloadOutcome::Terminal(_)
));
assert!(matches!(
tier_config_reload_remote_failure(Some("backend unavailable".to_string())),
TierConfigReloadOutcome::Terminal(_)
));
assert!(matches!(
tier_config_reload_remote_failure(Some("errServerNotInitialized".to_string())),
TierConfigReloadOutcome::TransientRetrySameChannel(_)
));
assert!(matches!(
tier_config_reload_connection_outcome(Error::other("backend unavailable")),
TierConfigReloadOutcome::Terminal(_)
));
assert!(matches!(
tier_config_reload_connection_outcome(Error::other("can not get client, err: connection unavailable")),
TierConfigReloadOutcome::TransientReconnect(_)
));
}
#[tokio::test]
async fn tier_config_reload_single_attempt_clears_offline_gate_without_redial() {
let client = test_peer_client();
client.offline.store(true, Ordering::Release);
let outcome = client.load_transition_tier_config_single_attempt_outcome().await;
assert!(matches!(outcome, TierConfigReloadOutcome::TransientReconnect(_)));
assert!(!client.offline.load(Ordering::Acquire));
}
#[test]
fn tier_config_reload_readiness_retry_does_not_require_reconnect() {
let client = test_peer_client();
client.offline.store(true, Ordering::Release);
let outcome = tier_config_reload_status_outcome(tonic::Status::unknown("Service was not ready: startup"));
assert!(matches!(outcome, TierConfigReloadOutcome::TransientRetrySameChannel(_)));
assert!(!outcome.requires_reconnect());
assert!(client.offline.load(Ordering::Acquire));
}
#[tokio::test]
async fn peer_rest_client_fast_fails_when_marked_offline() {
let client = test_peer_client();
@@ -49,6 +49,20 @@ use tracing::{debug, info, warn};
type Client = Arc<Box<dyn PeerS3Client>>;
#[derive(Clone, Debug)]
pub struct ScannerBucketListing {
pub buckets: Vec<BucketInfo>,
pub set_buckets: Vec<ScannerSetBucketListing>,
pub topology_complete: bool,
}
#[derive(Clone, Debug)]
pub struct ScannerSetBucketListing {
pub pool_index: usize,
pub set_index: usize,
pub buckets: Vec<BucketInfo>,
}
fn pool_participant_errors(clients: &[Client], errors: &[Option<Error>], pool_idx: usize) -> Vec<Option<Error>> {
clients
.iter()
@@ -216,6 +230,10 @@ impl S3PeerSys {
Ok(())
}
pub async fn list_bucket(&self, opts: &BucketOptions) -> Result<Vec<BucketInfo>> {
Ok(self.list_bucket_for_scanner(opts).await?.buckets)
}
pub async fn list_bucket_for_scanner(&self, opts: &BucketOptions) -> Result<ScannerBucketListing> {
let mut futures = Vec::with_capacity(self.clients.len());
for cli in self.clients.iter() {
futures.push(cli.list_bucket(opts));
@@ -239,9 +257,12 @@ impl S3PeerSys {
}
let mut result_map: HashMap<&String, BucketInfo> = HashMap::new();
let mut topology_complete = true;
for i in 0..self.pools_count {
let per_pool_errs = pool_participant_errors(&self.clients, &errors, i);
let quorum = pool_write_quorum(per_pool_errs.len());
topology_complete &=
!per_pool_errs.is_empty() && per_pool_errs.iter().all(|participant_error| participant_error.is_none());
if let Some(pool_err) = reduce_pool_write_quorum_errs(&per_pool_errs) {
tracing::error!("list_bucket per_pool_errs: {per_pool_errs:?}");
@@ -261,20 +282,17 @@ impl S3PeerSys {
}
for bucket in buckets.iter() {
if result_map.contains_key(&bucket.name) {
continue;
}
// incr bucket_map count create if not exists
let count = bucket_map.entry(&bucket.name).or_insert(0usize);
*count += 1;
if *count >= quorum {
result_map.insert(&bucket.name, bucket.clone());
result_map.entry(&bucket.name).or_insert_with(|| bucket.clone());
}
}
}
}
topology_complete &= bucket_map.values().all(|count| *count >= quorum);
// TODO: MRF
}
@@ -282,7 +300,11 @@ impl S3PeerSys {
buckets.sort_by_key(|b| b.name.clone());
Ok(buckets)
Ok(ScannerBucketListing {
buckets,
set_buckets: Vec::new(),
topology_complete,
})
}
pub async fn delete_bucket(&self, bucket: &str, opts: &DeleteBucketOptions) -> Result<()> {
let mut futures = Vec::with_capacity(self.clients.len());
@@ -1645,6 +1667,86 @@ mod tests {
assert_eq!(buckets[0].name, bucket.name);
}
#[tokio::test]
async fn scanner_bucket_listing_marks_quorum_result_incomplete_when_a_peer_is_missing() {
let bucket = BucketInfo {
name: "bucket-hidden-by-quorum".to_string(),
..Default::default()
};
let peer_sys = S3PeerSys {
clients: vec![
test_peer_with_list_bucket(&[0], Ok(vec![bucket])),
test_peer_with_list_bucket(&[0], Ok(Vec::new())),
test_peer_with_list_bucket(&[0], Ok(Vec::new())),
test_peer_with_list_bucket(&[0], Err(Error::DiskAccessDenied)),
],
pools_count: 1,
};
let listing = peer_sys
.list_bucket_for_scanner(&BucketOptions::default())
.await
.expect("peer quorum should still produce a scanner candidate listing");
assert!(listing.buckets.is_empty());
assert!(!listing.topology_complete);
}
#[tokio::test]
async fn scanner_bucket_listing_marks_divergent_successful_peers_incomplete() {
let bucket = BucketInfo {
name: "bucket-below-quorum".to_string(),
..Default::default()
};
let peer_sys = S3PeerSys {
clients: vec![
test_peer_with_list_bucket(&[0], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[0], Ok(vec![bucket])),
test_peer_with_list_bucket(&[0], Ok(Vec::new())),
test_peer_with_list_bucket(&[0], Ok(Vec::new())),
],
pools_count: 1,
};
let listing = peer_sys
.list_bucket_for_scanner(&BucketOptions::default())
.await
.expect("successful peer responses should still produce a scanner candidate listing");
assert!(listing.buckets.is_empty());
assert!(!listing.topology_complete);
}
#[tokio::test]
async fn scanner_bucket_listing_checks_same_bucket_in_every_pool() {
let bucket = BucketInfo {
name: "shared-bucket".to_string(),
..Default::default()
};
let peer_sys = S3PeerSys {
clients: vec![
test_peer_with_list_bucket(&[0], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[0], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[0], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[0], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[1], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[1], Ok(vec![bucket.clone()])),
test_peer_with_list_bucket(&[1], Ok(Vec::new())),
test_peer_with_list_bucket(&[1], Ok(Vec::new())),
],
pools_count: 2,
};
let listing = peer_sys
.list_bucket_for_scanner(&BucketOptions::default())
.await
.expect("a bucket visible in one pool should remain a scan candidate");
assert_eq!(listing.buckets.len(), 1);
assert_eq!(listing.buckets[0].name, bucket.name);
assert!(!listing.topology_complete);
}
#[tokio::test]
async fn test_delete_bucket_fails_when_any_pool_misses_write_quorum() {
let peer_sys = S3PeerSys {
File diff suppressed because it is too large Load Diff
@@ -13,9 +13,10 @@
// limitations under the License.
use rustfs_io_metrics::internode_metrics::{
INTERNODE_MSGPACK_DIRECTION_RESPONSE, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE,
INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_WRITE_ALL,
INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_TRANSPORT_BACKEND_GRPC,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
};
#[cfg(test)]
@@ -126,6 +127,38 @@ pub(crate) fn record_response_json_fallback(message: &'static str) {
global_internode_metrics().record_msgpack_json_fallback(INTERNODE_MSGPACK_DIRECTION_RESPONSE, message);
}
pub(crate) fn record_response_msgpack_decode(message: &'static str) {
global_internode_metrics().record_msgpack_json_decode(
INTERNODE_MSGPACK_DIRECTION_RESPONSE,
message,
INTERNODE_MSGPACK_CODEC_MSGPACK,
);
}
pub(crate) fn record_response_json_decode(message: &'static str) {
global_internode_metrics().record_msgpack_json_decode(
INTERNODE_MSGPACK_DIRECTION_RESPONSE,
message,
INTERNODE_MSGPACK_CODEC_JSON,
);
}
pub(crate) fn record_response_msgpack_decode_error(message: &'static str) {
global_internode_metrics().record_msgpack_json_decode_error(
INTERNODE_MSGPACK_DIRECTION_RESPONSE,
message,
INTERNODE_MSGPACK_CODEC_MSGPACK,
);
}
pub(crate) fn record_response_json_decode_error(message: &'static str) {
global_internode_metrics().record_msgpack_json_decode_error(
INTERNODE_MSGPACK_DIRECTION_RESPONSE,
message,
INTERNODE_MSGPACK_CODEC_JSON,
);
}
#[cfg(test)]
pub(crate) fn reset_internode_metrics_for_test() {
global_internode_metrics().reset_for_test();
@@ -135,3 +168,8 @@ pub(crate) fn reset_internode_metrics_for_test() {
pub(crate) fn internode_metrics_snapshot_for_test() -> InternodeMetricsSnapshot {
global_internode_metrics().snapshot()
}
#[cfg(test)]
pub(crate) fn internode_msgpack_json_decode_error_total_for_test() -> u64 {
global_internode_metrics().msgpack_json_decode_error_total_for_test()
}
File diff suppressed because it is too large Load Diff
+6 -1
View File
@@ -18,7 +18,7 @@ use rustfs_config::{
oidc::{
OIDC_CLAIM_NAME, OIDC_CLAIM_PREFIX, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_CONFIG_URL, OIDC_DEFAULT_CLAIM_NAME,
OIDC_DEFAULT_EMAIL_CLAIM, OIDC_DEFAULT_GROUPS_CLAIM, OIDC_DEFAULT_ROLES_CLAIM, OIDC_DEFAULT_SCOPES,
OIDC_DEFAULT_USERNAME_CLAIM, OIDC_DISPLAY_NAME, OIDC_EMAIL_CLAIM, OIDC_GROUPS_CLAIM, OIDC_OTHER_AUDIENCES,
OIDC_DEFAULT_USERNAME_CLAIM, OIDC_DISPLAY_NAME, OIDC_EMAIL_CLAIM, OIDC_GROUPS_CLAIM, OIDC_ISSUER, OIDC_OTHER_AUDIENCES,
OIDC_REDIRECT_URI, OIDC_REDIRECT_URI_DYNAMIC, OIDC_ROLE_POLICY, OIDC_ROLES_CLAIM, OIDC_SCOPES, OIDC_USERNAME_CLAIM,
},
};
@@ -37,6 +37,11 @@ pub static DEFAULT_IDENTITY_OPENID_KVS: LazyLock<KVS> = LazyLock::new(|| {
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: OIDC_ISSUER.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: OIDC_CLIENT_ID.to_owned(),
value: "".to_owned(),
+113
View File
@@ -46,6 +46,34 @@ pub const OUTPOSTS: &str = "OUTPOSTS";
pub const SNOW: &str = "SNOW";
pub const STANDARD_IA: &str = "STANDARD_IA";
/// Version of the client-discoverable storage-class write contract.
pub const CAPABILITY_CONTRACT_VERSION: u32 = 1;
/// Storage classes whose write semantics RustFS implements.
pub const SUPPORTED_WRITE_CLASSES: [&str; 2] = [STANDARD, RRS];
/// Stable S3 error code returned for unsupported write classes.
pub const UNSUPPORTED_WRITE_ERROR: &str = "InvalidStorageClass";
/// Compatibility behavior applied to historical label-only object metadata.
pub const LEGACY_LABEL_BEHAVIOR: &str = "normalized_to_effective_class";
/// Returns whether a client may select this storage class for a write.
pub fn is_supported_write_class(storage_class: &str) -> bool {
SUPPORTED_WRITE_CLASSES.contains(&storage_class)
}
/// Resolves the storage class that truthfully describes the stored object.
///
/// A completed lifecycle transition is a real storage tier and therefore keeps
/// its tier name. For local objects, only RRS has distinct layout semantics;
/// historical AWS class labels otherwise describe the effective STANDARD
/// layout.
pub fn effective_class<'a>(stored_class: Option<&'a str>, transitioned_tier: Option<&'a str>) -> &'a str {
if let Some(tier) = transitioned_tier.filter(|tier| !tier.is_empty()) {
return tier;
}
if stored_class == Some(RRS) { RRS } else { STANDARD }
}
// Standard constants for config info storage class
pub const CLASS_STANDARD: &str = "standard";
pub const CLASS_RRS: &str = "rrs";
@@ -512,6 +540,91 @@ mod tests {
StorageClassEnvOverrides::default()
}
#[test]
fn should_inline_preserves_exact_default_shard_boundaries() {
let config = Config::default();
for (case, shard_size, versioned, expected) in [
("unversioned below", 128 * 1024 - 1, false, true),
("unversioned exact", 128 * 1024, false, true),
("unversioned above", 128 * 1024 + 1, false, false),
("versioned below", 16 * 1024 - 1, true, true),
("versioned exact", 16 * 1024, true, true),
("versioned above", 16 * 1024 + 1, true, false),
("negative", -1, false, false),
] {
assert_eq!(
config.should_inline(shard_size, versioned),
expected,
"{case}: shard_size={shard_size}, versioned={versioned}"
);
}
}
#[test]
fn should_inline_preserves_exact_default_ec_2_2_object_boundaries() {
let config = Config::default();
let erasure = crate::erasure::coding::Erasure::new(2, 2, 1024 * 1024);
for (case, object_size, versioned, expected_shard_size, expected) in [
("unversioned below", 256 * 1024 - 1, false, 128 * 1024, true),
("unversioned exact", 256 * 1024, false, 128 * 1024, true),
("unversioned above", 256 * 1024 + 1, false, 128 * 1024 + 1, false),
("versioned below", 32 * 1024 - 1, true, 16 * 1024, true),
("versioned exact", 32 * 1024, true, 16 * 1024, true),
("versioned above", 32 * 1024 + 1, true, 16 * 1024 + 1, false),
] {
let shard_size = erasure.shard_file_size(object_size);
assert_eq!(shard_size, expected_shard_size, "{case}: object_size={object_size}");
assert_eq!(
config.should_inline(shard_size, versioned),
expected,
"{case}: object_size={object_size}, shard_size={shard_size}, versioned={versioned}"
);
}
}
#[test]
fn write_capability_contract_only_accepts_implemented_layouts() {
assert_eq!(SUPPORTED_WRITE_CLASSES, [STANDARD, RRS]);
assert!(is_supported_write_class(STANDARD));
assert!(is_supported_write_class(RRS));
for label_only_class in [
DEEP_ARCHIVE,
EXPRESS_ONEZONE,
GLACIER,
GLACIER_IR,
INTELLIGENT_TIERING,
ONEZONE_IA,
OUTPOSTS,
SNOW,
STANDARD_IA,
] {
assert!(
!is_supported_write_class(label_only_class),
"{label_only_class} must not be advertised as a supported write class"
);
}
assert!(!is_supported_write_class(""));
assert!(!is_supported_write_class("standard"));
assert!(!is_supported_write_class("UNKNOWN"));
}
#[test]
fn effective_class_normalizes_legacy_labels_and_preserves_real_tiers() {
assert_eq!(effective_class(None, None), STANDARD);
assert_eq!(effective_class(Some(STANDARD), None), STANDARD);
assert_eq!(effective_class(Some(RRS), None), RRS);
assert_eq!(effective_class(Some(STANDARD_IA), None), STANDARD);
assert_eq!(effective_class(Some(GLACIER), None), STANDARD);
assert_eq!(effective_class(Some("UNKNOWN"), None), STANDARD);
assert_eq!(effective_class(Some(STANDARD_IA), Some("WARM-TIER")), "WARM-TIER");
assert_eq!(effective_class(Some(STANDARD), Some(STANDARD_IA)), STANDARD_IA);
assert_eq!(effective_class(Some(RRS), Some("CUSTOM-RRS-TIER")), "CUSTOM-RRS-TIER");
}
#[test]
fn automatic_parity_is_resolved_per_pool() {
let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides())
+242 -4
View File
@@ -16,6 +16,7 @@
use crate::disk::error_reduce::count_errs;
use crate::error::{Error, Result};
use crate::layout::set_heal::{formats_to_drives_info, new_heal_format_sets};
use crate::multipart_listing::paginate_multipart_listing;
use crate::storage_api_contracts::{
bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions},
list::{StorageListObjectVersionsInfo, StorageListObjectsV2Info, StorageObjectInfoOrErr, StorageWalkOptions},
@@ -49,7 +50,10 @@ use rustfs_filemeta::FileInfo;
use rustfs_lock::NamespaceLockWrapper;
use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_utils::{crc_hash, path::path_join_buf, sip_hash};
use std::{collections::HashMap, sync::Arc};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
use tokio::sync::RwLock;
use tokio::sync::broadcast::{Receiver, Sender};
use tokio::time::Duration;
@@ -63,6 +67,8 @@ type ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>;
type ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, Error>;
type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
#[derive(Debug, Clone)]
pub struct Sets {
pub id: Uuid,
@@ -799,9 +805,52 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for Sets {
delimiter: Option<String>,
max_uploads: usize,
) -> Result<ListMultipartsInfo> {
self.get_disks_by_key(prefix)
.list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, max_uploads)
.await
let per_set_limit = max_uploads.saturating_add(1);
let results = futures::stream::iter(self.disk_set.iter().cloned())
.map(|set| {
let key_marker = key_marker.clone();
let upload_id_marker = upload_id_marker.clone();
let delimiter = delimiter.clone();
async move {
set.list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, per_set_limit)
.await
}
})
.buffer_unordered(LIST_MULTIPART_SETS_CONCURRENCY)
.collect::<Vec<_>>()
.await;
let mut uploads = Vec::new();
let mut common_prefixes = HashSet::new();
let mut source_truncated = false;
for result in results {
let page = result?;
uploads.extend(page.uploads);
common_prefixes.extend(page.common_prefixes);
source_truncated |= page.is_truncated;
}
let page = paginate_multipart_listing(
uploads,
common_prefixes.into_iter().collect(),
key_marker.as_deref(),
key_marker.as_ref().and(upload_id_marker.as_deref()),
max_uploads,
source_truncated,
);
Ok(ListMultipartsInfo {
key_marker,
upload_id_marker,
next_key_marker: page.next_key_marker,
next_upload_id_marker: page.next_upload_id_marker,
max_uploads,
is_truncated: page.is_truncated,
uploads: page.uploads,
common_prefixes: page.common_prefixes,
prefix: prefix.to_owned(),
delimiter,
})
}
#[tracing::instrument(skip(self))]
async fn new_multipart_upload(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<MultipartUploadResult> {
@@ -1111,6 +1160,7 @@ mod tests {
use crate::layout::endpoints::SetupType;
use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::list::ListOperations as _;
use crate::storage_api_contracts::multipart::MultipartOperations as _;
use rustfs_lock::client::local::LocalClient;
use serial_test::serial;
@@ -1248,6 +1298,194 @@ mod tests {
assert_eq!(result, (Some(3), Some(1), Some(0)));
}
async fn multipart_listing_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
let format = FormatV3::new(2, 2);
let mut temp_dirs = Vec::new();
let mut all_endpoints = Vec::new();
let mut disk_sets = Vec::new();
for set_index in 0..2 {
let mut endpoints = Vec::new();
let mut disks = Vec::new();
for disk_index in 0..2 {
let temp_dir = tempfile::tempdir().expect("tempdir should be created");
let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("tempdir path should be utf8"))
.expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(set_index);
endpoint.set_disk_index(disk_index);
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("disk should be created");
let mut disk_format = format.clone();
disk_format.erasure.this = format.erasure.sets[set_index][disk_index];
save_format_file(&Some(disk.clone()), &Some(disk_format))
.await
.expect("format should be saved");
temp_dirs.push(temp_dir);
all_endpoints.push(endpoint.clone());
endpoints.push(endpoint);
disks.push(Some(disk));
}
disk_sets.push(
SetDisks::new(
"test-owner".to_string(),
Arc::new(RwLock::new(disks)),
2,
1,
0,
set_index,
endpoints,
format.clone(),
vec![Arc::new(LocalClient::new()), Arc::new(LocalClient::new())],
)
.await,
);
}
let sets = Arc::new(Sets {
id: format.id,
disk_set: disk_sets,
pool_idx: 0,
endpoints: PoolEndpoints {
legacy: false,
set_count: 2,
drives_per_set: 2,
endpoints: Endpoints::from(all_endpoints),
cmd_line: String::new(),
platform: String::new(),
},
format,
parity_count: 1,
set_count: 2,
set_drive_count: 2,
default_parity_count: 1,
distribution_algo: DistributionAlgoVersion::V1,
exit_signal: None,
ctx: bootstrap_ctx(),
});
(temp_dirs, sets)
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn list_multipart_uploads_merges_all_sets_without_pagination_loss() {
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::Erasure).await;
let (_temp_dirs, sets) = multipart_listing_test_sets().await;
let bucket = format!("multipart-list-{}", Uuid::new_v4().simple());
sets.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut keys_by_set = [Vec::new(), Vec::new()];
for index in 0..100 {
let key = format!("logs/{index:03}.bin");
let set_index = sets.get_hashed_set_index(&key);
if keys_by_set[set_index].len() < 2 {
keys_by_set[set_index].push(key);
}
if keys_by_set.iter().all(|keys| keys.len() == 2) {
break;
}
}
assert!(keys_by_set.iter().all(|keys| keys.len() == 2), "test keys must span both sets");
let repeated_key = keys_by_set[0][0].clone();
let mut expected = Vec::new();
for key in keys_by_set.iter().flatten() {
let upload = sets
.new_multipart_upload(&bucket, key, &ObjectOptions::default())
.await
.expect("multipart upload should be created");
expected.push((key.clone(), upload.upload_id));
}
let second = sets
.new_multipart_upload(&bucket, &repeated_key, &ObjectOptions::default())
.await
.expect("second upload for the same key should be created");
expected.push((repeated_key, second.upload_id));
expected.sort();
let mut actual = Vec::new();
let mut key_marker = None;
let mut upload_id_marker = None;
for _ in 0..expected.len() + 1 {
let page = sets
.list_multipart_uploads(&bucket, "logs/", key_marker.clone(), upload_id_marker.clone(), None, 2)
.await
.expect("multipart page should list across every set");
assert!(page.uploads.len() <= 2);
actual.extend(
page.uploads
.iter()
.map(|upload| (upload.object.clone(), upload.upload_id.clone())),
);
if !page.is_truncated {
break;
}
key_marker = page.next_key_marker;
upload_id_marker = page.next_upload_id_marker;
}
assert_eq!(actual, expected, "set-level merge must return every upload exactly once");
let mut deduped = actual.clone();
deduped.dedup();
assert_eq!(deduped.len(), actual.len(), "set-level pagination must not duplicate uploads");
let mut nested_by_set = [None, None];
for index in 0..100 {
let key = format!("nested/group-{index:03}/file.bin");
let set_index = sets.get_hashed_set_index(&key);
nested_by_set[set_index].get_or_insert(key);
if nested_by_set.iter().all(Option::is_some) {
break;
}
}
for key in nested_by_set.iter().flatten() {
sets.new_multipart_upload(&bucket, key, &ObjectOptions::default())
.await
.expect("nested multipart upload should be created");
}
let mut expected_prefixes = nested_by_set
.iter()
.flatten()
.map(|key| {
key.rsplit_once('/')
.expect("nested key should contain a delimiter")
.0
.to_string()
+ "/"
})
.collect::<Vec<_>>();
expected_prefixes.sort();
let first = sets
.list_multipart_uploads(&bucket, "nested/", None, None, Some("/".to_string()), 1)
.await
.expect("first delimiter page should list across every set");
assert!(first.is_truncated);
assert_eq!(first.common_prefixes, expected_prefixes[..1]);
let second = sets
.list_multipart_uploads(
&bucket,
"nested/",
first.next_key_marker,
first.next_upload_id_marker,
Some("/".to_string()),
1,
)
.await
.expect("second delimiter page should list across every set");
assert!(!second.is_truncated);
assert_eq!(second.common_prefixes, expected_prefixes[1..]);
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn sets_list_objects_v2_lists_objects_within_the_pool() {
+16 -11
View File
@@ -560,17 +560,22 @@ pub(crate) async fn cleanup_source_entry_if_unchanged(
ensure_source_cleanup_versions_unchanged(set.clone(), bucket, object, expected, allowed_missing, op_label).await?;
set.delete_object(
bucket,
cleanup_key.as_str(),
ObjectOptions {
delete_prefix: true,
delete_prefix_object: true,
no_lock: true,
..Default::default()
},
)
.await
let result = set
.delete_object(
bucket,
cleanup_key.as_str(),
ObjectOptions {
delete_prefix: true,
delete_prefix_object: true,
no_lock: true,
..Default::default()
},
)
.await;
if result.is_ok() {
crate::store::list_objects::observe_scanner_namespace_mutations(bucket, 1);
}
result
}
fn should_check_data_movement_resume_target(src_pool_idx: usize, target_pool_idx: usize) -> bool {
File diff suppressed because it is too large Load Diff
@@ -57,14 +57,14 @@ fn apply_data_usage_result(
usage: &mut rustfs_madmin::Usage,
) {
match result {
Ok(info) => {
Ok(info) if info.is_complete_bucket_usage_snapshot() => {
buckets.count = info.buckets_count;
objects.count = info.objects_total_count;
versions.count = info.versions_total_count;
delete_markers.count = info.delete_markers_total_count;
usage.size = info.objects_total_size;
}
Err(_) => {
Ok(_) | Err(_) => {
buckets.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
objects.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
versions.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
@@ -649,18 +649,15 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
if erasure_set.id == 0 {
erasure_set.id = d.set_index;
if let Ok(cache) = load_data_usage_cache(
match load_data_usage_cache(
&store.pools[d.pool_index as usize].disk_set[d.set_index as usize].clone(),
DATA_USAGE_CACHE_NAME,
)
.await
{
let data_usage_info = cache.dui(DATA_USAGE_ROOT, &Vec::<String>::new());
erasure_set.objects_count = data_usage_info.objects_total_count;
erasure_set.versions_count = data_usage_info.versions_total_count;
erasure_set.delete_markers_count = data_usage_info.delete_markers_total_count;
erasure_set.usage = data_usage_info.objects_total_size;
};
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
}
erasure_set.raw_capacity += d.total_space;
@@ -672,6 +669,20 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
Ok(pools_info)
}
fn apply_erasure_set_usage(cache: &rustfs_data_usage::DataUsageCache, erasure_set: &mut ErasureSetInfo) {
let data_usage_info = cache.dui(DATA_USAGE_ROOT, &[]);
if !data_usage_info.is_complete_bucket_usage_snapshot() {
erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
return;
}
erasure_set.objects_count = data_usage_info.objects_total_count;
erasure_set.versions_count = data_usage_info.versions_total_count;
erasure_set.delete_markers_count = data_usage_info.delete_markers_total_count;
erasure_set.usage = data_usage_info.objects_total_size;
erasure_set.usage_error = None;
}
#[allow(clippy::const_is_empty)]
pub fn get_commit_id() -> String {
let ver = if !build::TAG.is_empty() {
@@ -697,8 +708,9 @@ mod tests {
use rustfs_madmin::{Disk, ITEM_OFFLINE, ITEM_ONLINE, ITEM_UNKNOWN, ServerProperties};
use super::{
DATA_USAGE_UNAVAILABLE_ERROR, apply_data_usage_result, get_local_server_property, get_online_offline_disks_stats,
get_server_info, reconcile_servers_with_endpoint_topology, server_topology_completeness_report,
DATA_USAGE_ROOT, DATA_USAGE_UNAVAILABLE_ERROR, apply_data_usage_result, apply_erasure_set_usage,
get_local_server_property, get_online_offline_disks_stats, get_server_info, reconcile_servers_with_endpoint_topology,
server_topology_completeness_report,
};
fn disk_with_state(endpoint: &str, state: &str) -> Disk {
@@ -900,11 +912,19 @@ mod tests {
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
let info = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
buckets_count: 2,
objects_total_count: 3,
versions_total_count: 4,
delete_markers_total_count: 5,
objects_total_size: 6,
buckets_usage: [
("bucket-a".to_string(), rustfs_data_usage::BucketUsageInfo::default()),
("bucket-b".to_string(), rustfs_data_usage::BucketUsageInfo::default()),
]
.into_iter()
.collect(),
usage_snapshot_complete: true,
..Default::default()
};
@@ -917,6 +937,71 @@ mod tests {
assert_eq!(usage.size, 6);
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
let info = rustfs_data_usage::DataUsageInfo {
buckets_count: 1,
objects_total_count: 100,
objects_total_size: 1024,
..Default::default()
};
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
}
#[test]
fn incomplete_erasure_set_cache_is_not_reported_as_zero() {
let mut cache = rustfs_data_usage::DataUsageCache::default();
cache.info.name = DATA_USAGE_ROOT.to_string();
cache.replace(DATA_USAGE_ROOT, "", rustfs_data_usage::DataUsageEntry::default());
let mut set = rustfs_madmin::ErasureSetInfo::default();
apply_erasure_set_usage(&cache, &mut set);
assert_eq!(set.usage_error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(set.objects_count, 0);
assert_eq!(set.usage, 0);
}
#[test]
fn complete_erasure_set_cache_is_reported() {
let mut cache = rustfs_data_usage::DataUsageCache::default();
cache.info.name = DATA_USAGE_ROOT.to_string();
cache.info.last_update = Some(std::time::SystemTime::UNIX_EPOCH);
cache.info.snapshot_complete = true;
cache.replace(
DATA_USAGE_ROOT,
"",
rustfs_data_usage::DataUsageEntry {
size: 512,
objects: 3,
versions: 4,
delete_markers: 1,
..Default::default()
},
);
let mut set = rustfs_madmin::ErasureSetInfo::default();
apply_erasure_set_usage(&cache, &mut set);
assert_eq!(set.objects_count, 3);
assert_eq!(set.versions_count, 4);
assert_eq!(set.delete_markers_count, 1);
assert_eq!(set.usage, 512);
assert!(set.usage_error.is_none());
}
#[serial]
#[tokio::test]
async fn server_info_includes_global_deployment_id() {
+60 -5
View File
@@ -14,8 +14,8 @@
use crate::disk::error::DiskError;
use crate::error::StorageError;
use std::io;
use std::time::Instant;
use std::{error::Error as StdError, fmt, io};
pub(crate) const GET_OBJECT_PATH_CODEC_STREAMING: &str = "codec_streaming";
pub(crate) const GET_OBJECT_PATH_CODEC_STREAMING_LEGACY_ENGINE: &str = "codec_streaming_legacy_engine";
@@ -207,9 +207,42 @@ pub(crate) fn classify_disk_error(err: &DiskError) -> GetObjectFailureReason {
}
}
pub(crate) fn classify_io_error(err: &io::Error) -> GetObjectFailureReason {
#[derive(Debug)]
struct GetObjectDownstreamClosed {
source: io::Error,
}
impl fmt::Display for GetObjectDownstreamClosed {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "GetObject output closed: {}", self.source)
}
}
impl StdError for GetObjectDownstreamClosed {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
Some(&self.source)
}
}
pub(crate) fn mark_get_object_downstream_closed(err: io::Error) -> io::Error {
match err.kind() {
io::ErrorKind::BrokenPipe | io::ErrorKind::ConnectionReset | io::ErrorKind::ConnectionAborted => {
io::Error::new(err.kind(), GetObjectDownstreamClosed { source: err })
}
_ => err,
}
}
pub(crate) fn classify_io_error(err: &io::Error) -> GetObjectFailureReason {
let mut source = err.get_ref().map(|source| source as &(dyn StdError + 'static));
while let Some(error) = source {
if error.is::<GetObjectDownstreamClosed>() {
return GetObjectFailureReason::DownstreamClosed;
}
source = error.source();
}
match err.kind() {
io::ErrorKind::BrokenPipe | io::ErrorKind::ConnectionReset => GetObjectFailureReason::DownstreamClosed,
io::ErrorKind::TimedOut => GetObjectFailureReason::Timeout,
io::ErrorKind::UnexpectedEof => GetObjectFailureReason::ShortRead,
io::ErrorKind::InvalidInput | io::ErrorKind::InvalidData => GetObjectFailureReason::RangeOrLengthInvalid,
@@ -260,6 +293,16 @@ mod tests {
classify_storage_error(&StorageError::InvalidRangeSpec("bad range".to_string())),
GetObjectFailureReason::RangeOrLengthInvalid
);
let internal_broken_pipe = StorageError::Io(io::Error::from(io::ErrorKind::BrokenPipe));
assert_eq!(classify_storage_error(&internal_broken_pipe), GetObjectFailureReason::Io);
let downstream_closed = StorageError::Io(mark_get_object_downstream_closed(io::Error::from(io::ErrorKind::BrokenPipe)));
assert_eq!(
classify_storage_error(&downstream_closed),
GetObjectFailureReason::DownstreamClosed,
"the legacy duplex producer must suppress only explicitly tagged output closes"
);
}
#[test]
@@ -277,8 +320,8 @@ mod tests {
#[test]
fn classifies_io_errors_for_get_pipeline() {
let cases = [
(io::ErrorKind::BrokenPipe, GetObjectFailureReason::DownstreamClosed),
(io::ErrorKind::ConnectionReset, GetObjectFailureReason::DownstreamClosed),
(io::ErrorKind::BrokenPipe, GetObjectFailureReason::Io),
(io::ErrorKind::ConnectionReset, GetObjectFailureReason::Io),
(io::ErrorKind::TimedOut, GetObjectFailureReason::Timeout),
(io::ErrorKind::UnexpectedEof, GetObjectFailureReason::ShortRead),
(io::ErrorKind::InvalidInput, GetObjectFailureReason::RangeOrLengthInvalid),
@@ -292,6 +335,18 @@ mod tests {
}
}
#[test]
fn classifies_marked_output_close_as_downstream_closed() {
for kind in [
io::ErrorKind::BrokenPipe,
io::ErrorKind::ConnectionReset,
io::ErrorKind::ConnectionAborted,
] {
let err = mark_get_object_downstream_closed(io::Error::from(kind));
assert_eq!(classify_io_error(&err), GetObjectFailureReason::DownstreamClosed, "kind={kind:?}");
}
}
#[test]
fn keeps_metric_labels_stable() {
assert_eq!(GetObjectFailureReason::ReadQuorum.as_str(), "read_quorum");
+146 -53
View File
@@ -35,7 +35,7 @@ use std::{
Arc,
atomic::{AtomicI64, AtomicU32, AtomicU64, Ordering},
},
time::Duration,
time::{Duration, SystemTime, UNIX_EPOCH},
};
use tokio::{sync::RwLock, time};
use tokio_util::sync::CancellationToken;
@@ -339,21 +339,19 @@ impl Drop for DiskHealthWaitingGuard<'_> {
impl DiskHealthTracker {
/// Create a new disk health tracker
pub fn new() -> Self {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
let now = current_unix_time();
let now_nanos = unix_nanos(now);
Self {
last_success: AtomicI64::new(now),
last_started: AtomicI64::new(now),
last_success: AtomicI64::new(now_nanos),
last_started: AtomicI64::new(now_nanos),
status: AtomicU32::new(DISK_HEALTH_OK),
waiting: AtomicU32::new(0),
runtime_state: AtomicU32::new(RuntimeDriveHealthState::Online as u32),
consecutive_failures: AtomicU32::new(0),
consecutive_successes: AtomicU32::new(0),
offline_since_unix_secs: AtomicI64::new(0),
last_transition_unix_secs: AtomicI64::new(now / 1_000_000_000),
last_transition_unix_secs: AtomicI64::new(unix_secs_i64(now)),
last_capacity_total: AtomicU64::new(0),
last_capacity_used: AtomicU64::new(0),
last_capacity_free: AtomicU64::new(0),
@@ -363,11 +361,7 @@ impl DiskHealthTracker {
/// Log a successful operation
pub fn log_success(&self) {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.last_success.store(now, Ordering::Relaxed);
self.last_success.store(current_unix_nanos(), Ordering::Relaxed);
}
pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
@@ -429,11 +423,14 @@ impl DiskHealthTracker {
}
pub fn offline_duration(&self) -> Option<Duration> {
self.offline_duration_at(current_unix_secs())
}
fn offline_duration_at(&self, now: u64) -> Option<Duration> {
let offline_since = self.offline_since_unix_secs.load(Ordering::Acquire);
if offline_since <= 0 {
return None;
}
let now = current_unix_secs();
Some(Duration::from_secs(now.saturating_sub(offline_since as u64)))
}
@@ -492,6 +489,12 @@ impl DiskHealthTracker {
/// Remote disks are marked faulty on timeout/network errors; the init loop retries with the
/// same [`DiskStore`] handles, which would otherwise fail immediately at `is_faulty()`.
pub fn reset_for_store_init_retry(&self, endpoint: &Endpoint) {
self.reset_for_store_init_retry_at(endpoint, current_unix_time());
}
fn reset_for_store_init_retry_at(&self, endpoint: &Endpoint, now: Duration) {
let now_nanos = unix_nanos(now);
let now_secs = unix_secs_i64(now);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.runtime_state
.store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
@@ -499,11 +502,9 @@ impl DiskHealthTracker {
self.consecutive_successes.store(0, Ordering::Release);
self.offline_since_unix_secs.store(0, Ordering::Release);
self.waiting.store(0, Ordering::Release);
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap();
let now_nanos = now.as_nanos() as i64;
self.last_success.store(now_nanos, Ordering::Relaxed);
self.last_started.store(now_nanos, Ordering::Relaxed);
self.last_transition_unix_secs.store(now.as_secs() as i64, Ordering::Release);
self.last_transition_unix_secs.store(now_secs, Ordering::Release);
record_drive_runtime_state(endpoint, RuntimeDriveHealthState::Online);
}
@@ -612,10 +613,33 @@ impl DiskHealthTracker {
}
fn current_unix_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_secs()
// Zero is reserved as "not recorded" by health timestamp atomics.
current_unix_time().as_secs().max(1)
}
fn current_unix_nanos() -> i64 {
unix_nanos(current_unix_time())
}
fn current_unix_time() -> Duration {
unix_time_since_epoch(SystemTime::now())
}
fn unix_time_since_epoch(time: SystemTime) -> Duration {
time.duration_since(UNIX_EPOCH).unwrap_or(Duration::ZERO)
}
fn unix_nanos(time: Duration) -> i64 {
i64::try_from(time.as_nanos()).unwrap_or(i64::MAX)
}
fn unix_secs_i64(time: Duration) -> i64 {
i64::try_from(time.as_secs()).unwrap_or(i64::MAX)
}
fn elapsed_since(last_nanos: i64, now_nanos: i64) -> Duration {
let elapsed_nanos = now_nanos.saturating_sub(last_nanos).max(0);
Duration::from_nanos(u64::try_from(elapsed_nanos).unwrap_or(u64::MAX))
}
impl Default for DiskHealthTracker {
@@ -635,11 +659,7 @@ struct HealthDiskCtxValue {
impl HealthDiskCtxValue {
fn log_success(&self) {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.last_success.store(now, Ordering::Relaxed);
self.last_success.store(current_unix_nanos(), Ordering::Relaxed);
}
}
@@ -774,12 +794,7 @@ impl LocalDiskWrapper {
}
let last_success_nanos = health.last_success.load(Ordering::Relaxed);
let elapsed = Duration::from_nanos(
(std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64 - last_success_nanos) as u64
);
let elapsed = elapsed_since(last_success_nanos, current_unix_nanos());
if elapsed < SKIP_IF_SUCCESS_BEFORE {
continue;
@@ -1091,11 +1106,7 @@ impl LocalDiskWrapper {
self.check_disk_stale().await?;
// Record operation start
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.health.last_started.store(now, Ordering::Relaxed);
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
let _waiting_guard = self.health.waiting_guard();
if timeout_duration == Duration::ZERO {
@@ -1317,11 +1328,7 @@ impl DiskAPI for LocalDiskWrapper {
}
// Record operation start
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.health.last_started.store(now, Ordering::Relaxed);
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
self.health.increment_waiting();
// Execute the operation
@@ -1473,6 +1480,33 @@ impl DiskAPI for LocalDiskWrapper {
.await
}
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
self.track_disk_health(
|| async {
self.disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
},
get_max_timeout_duration(),
)
.await
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: crate::disk::PartTransactionAction) -> Result<()> {
self.track_disk_health(
|| async { self.disk.settle_part_transaction(volume, path, action).await },
get_max_timeout_duration(),
)
.await
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
self.track_disk_health(|| async { self.disk.delete(volume, path, opt).await }, get_max_timeout_duration())
.await
@@ -2063,7 +2097,7 @@ mod tests {
}
#[tokio::test]
async fn walk_dir_skip_total_timeout_keeps_stream_pending() {
async fn walk_dir_total_timeout_disable_modes_keep_stream_pending() {
temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
let dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint =
@@ -2086,24 +2120,34 @@ mod tests {
.await
.expect("object metadata should be written");
let mut writer = PendingWriter;
let result = tokio::time::timeout(
Duration::from_millis(20),
wrapper.walk_dir(
for (reason, options) in [
(
"skip_total_timeout",
WalkDirOptions {
bucket: bucket.to_string(),
recursive: true,
skip_total_timeout: true,
..Default::default()
},
&mut writer,
),
)
.await;
(
"zero per-request timeout",
WalkDirOptions {
bucket: bucket.to_string(),
recursive: true,
timeout_ms: Some(0),
stall_timeout_ms: None,
..Default::default()
},
),
] {
let mut writer = PendingWriter;
let result = tokio::time::timeout(Duration::from_millis(1_100), wrapper.walk_dir(options, &mut writer)).await;
assert!(result.is_err(), "skip_total_timeout should leave backpressured walk pending");
assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
assert!(!wrapper.health.is_faulty());
assert!(result.is_err(), "{reason} should leave backpressured walk pending");
assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
assert!(!wrapper.health.is_faulty());
}
})
.await;
}
@@ -2233,4 +2277,53 @@ mod tests {
assert!(health.mark_offline(&endpoint, "again"));
assert!(health.is_faulty());
}
#[test]
fn unix_time_clamps_epoch_and_pre_epoch_to_zero() {
let before_epoch = UNIX_EPOCH
.checked_sub(Duration::from_nanos(1))
.expect("one nanosecond before the Unix epoch should be representable");
assert_eq!(unix_time_since_epoch(UNIX_EPOCH), Duration::ZERO);
assert_eq!(unix_time_since_epoch(before_epoch), Duration::ZERO);
}
#[test]
fn elapsed_and_offline_duration_saturate_on_clock_rollback() {
let health = DiskHealthTracker::new();
health.offline_since_unix_secs.store(10, Ordering::Release);
assert_eq!(elapsed_since(10, 12), Duration::from_nanos(2));
assert_eq!(elapsed_since(10, 9), Duration::ZERO);
assert_eq!(health.offline_duration_at(9), Some(Duration::ZERO));
}
#[test]
fn pre_epoch_retry_reset_updates_the_complete_health_state() {
let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry-pre-epoch").expect("endpoint should parse");
let health = DiskHealthTracker::new();
health.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
health
.runtime_state
.store(RuntimeDriveHealthState::Offline as u32, Ordering::Release);
health.consecutive_failures.store(3, Ordering::Release);
health.consecutive_successes.store(2, Ordering::Release);
health.offline_since_unix_secs.store(11, Ordering::Release);
health.waiting.store(4, Ordering::Release);
health.last_success.store(12, Ordering::Release);
health.last_started.store(13, Ordering::Release);
health.last_transition_unix_secs.store(14, Ordering::Release);
health.reset_for_store_init_retry_at(&endpoint, unix_time_since_epoch(UNIX_EPOCH - Duration::from_nanos(1)));
assert_eq!(health.status.load(Ordering::Acquire), DISK_HEALTH_OK);
assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Online);
assert_eq!(health.consecutive_failures.load(Ordering::Acquire), 0);
assert_eq!(health.consecutive_successes.load(Ordering::Acquire), 0);
assert_eq!(health.offline_since_unix_secs.load(Ordering::Acquire), 0);
assert_eq!(health.waiting.load(Ordering::Acquire), 0);
assert_eq!(health.last_success.load(Ordering::Acquire), 0);
assert_eq!(health.last_started.load(Ordering::Acquire), 0);
assert_eq!(health.last_transition_unix_secs.load(Ordering::Acquire), 0);
}
}
+18
View File
@@ -231,6 +231,13 @@ impl DiskError {
}
}
pub fn is_internode_http_status(&self, status: u16) -> bool {
matches!(
self.internode_http_error_kind(),
Some(InternodeHttpErrorKind::HttpStatus(actual)) if actual.as_u16() == status
)
}
// /// If all errors are of the same fatal disk error type, returns the corresponding error.
// /// Otherwise, returns Ok.
// pub fn check_disk_fatal_errs(errs: &[Option<Error>]) -> Result<()> {
@@ -981,6 +988,17 @@ mod tests {
assert!(!disk_error.contains_io_error_kind(std::io::ErrorKind::TimedOut));
}
#[test]
fn test_internode_http_status_classification() {
let too_many_requests = DiskError::from(rustfs_rio::new_test_internode_http_io_error(
rustfs_rio::InternodeHttpErrorKind::HttpStatus(http::StatusCode::TOO_MANY_REQUESTS),
));
assert!(too_many_requests.is_internode_http_status(429));
assert!(!too_many_requests.is_internode_http_status(500));
assert!(!DiskError::FileNotFound.is_internode_http_status(429));
}
#[test]
fn test_metacache_output_stream_closed_classification_survives_clone() {
let disk_error = DiskError::metacache_output_stream_closed();
+643 -100
View File
@@ -19,7 +19,8 @@ use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_r
use crate::disk::{
BUCKET_META_PREFIX, CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN,
CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskMetrics,
FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET,
FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize, PART_TRANSACTION_NEW_META,
PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET,
RUSTFS_META_TMP_DELETED_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE,
STORAGE_FORMAT_FILE_BACKUP, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, conv_part_err_to_int,
endpoint::Endpoint,
@@ -80,6 +81,40 @@ const ENV_BITROT_SIZE_MISMATCH_RETRY_COUNT: &str = "RUSTFS_BITROT_SIZE_MISMATCH_
const ENV_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS: &str = "RUSTFS_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS";
const DEFAULT_BITROT_SIZE_MISMATCH_RETRY_COUNT: u64 = 2;
const DEFAULT_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS: u64 = 100;
const PART_TRANSACTION_OLD_DATA: &str = "old.data";
const PART_TRANSACTION_OLD_DATA_ABSENT: &str = "old.data.absent";
const PART_TRANSACTION_OLD_META_ABSENT: &str = "old.meta.absent";
const PART_TRANSACTION_PUBLISH_META: &str = "publish.meta";
enum ReadAllError {
Open(std::io::Error),
Disk(DiskError),
}
struct ListingMetadataRead {
bytes: Vec<u8>,
file_meta: Option<FileMeta>,
data_dirs: HashSet<String>,
has_namespace_child_candidate: bool,
}
fn read_all_data_std(path: &Path) -> core::result::Result<(Vec<u8>, Option<OffsetDateTime>), ReadAllError> {
let mut file = std::fs::File::open(path).map_err(ReadAllError::Open)?;
let metadata = file.metadata().map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
if metadata.is_dir() {
return Err(ReadAllError::Disk(DiskError::FileNotFound));
}
let size = usize::try_from(metadata.len()).map_err(|err| ReadAllError::Disk(DiskError::other(err)))?;
let mut bytes = Vec::new();
bytes
.try_reserve_exact(size)
.map_err(|err| ReadAllError::Disk(Error::other(err)))?;
std::io::Read::read_to_end(&mut file, &mut bytes).map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
let modtime = metadata.modified().ok().map(OffsetDateTime::from);
Ok((bytes, modtime))
}
fn inline_metadata_rollback_dir(version_id: Uuid, meta: &FileMeta) -> Uuid {
let used_data_dirs: HashSet<Uuid> = meta.get_data_dirs().unwrap_or_default().into_iter().flatten().collect();
@@ -111,6 +146,28 @@ fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
}
}
fn snapshot_part_transaction_file(src: &Path, backup: &Path, absent: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(src) {
Ok(metadata) if metadata.is_file() => std::fs::hard_link(src, backup),
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound => std::fs::write(absent, []),
Err(err) => Err(err),
}
}
fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, restore: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(backup) {
Ok(metadata) if metadata.is_file() => {
remove_file_if_exists(restore)?;
std::fs::hard_link(backup, restore)?;
std::fs::rename(restore, current)
}
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction backup is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound && absent.is_file() => remove_file_if_exists(current),
Err(err) => Err(err),
}
}
fn rollback_committed_rename_std(
dst_file_path: &Path,
new_data_path: Option<&Path>,
@@ -1859,6 +1916,46 @@ where
with_walk_stall_deadline(stall, fut).await?
}
async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option<Duration>) -> Result<Vec<String>> {
let mut entries = with_walk_stall_deadline(stall, fs::read_dir(path))
.await?
.map_err(to_file_error)?;
let mut names = Vec::new();
let mut remaining = count;
loop {
let Some(entry) = with_walk_stall_deadline(stall, entries.next_entry())
.await?
.map_err(to_file_error)?
else {
break;
};
let name = entry.file_name().to_string_lossy().to_string();
if name.is_empty() || name == "." || name == ".." {
continue;
}
let file_type = with_walk_stall_deadline(stall, entry.file_type())
.await?
.map_err(to_file_error)?;
if file_type.is_file() {
names.push(name);
} else if file_type.is_dir() {
names.push(format!("{name}{SLASH_SEPARATOR}"));
} else {
continue;
}
remaining -= 1;
if remaining == 0 {
break;
}
}
Ok(names)
}
impl FileCacheReclaimReader {
fn new(inner: File, reclaim_offset: u64, reclaim_len: usize, reclaim_on_drop: bool) -> Self {
#[cfg(target_os = "macos")]
@@ -4651,6 +4748,80 @@ impl LocalDisk {
Ok(data)
}
async fn read_listing_metadata(&self, volume: &str, object_name: &str) -> Result<ListingMetadataRead> {
let object_dir = self.get_object_path(volume, object_name)?;
let metadata_path = object_dir.join(STORAGE_FORMAT_FILE);
let volume_dir = self.get_bucket_path(volume)?;
let result = tokio::task::spawn_blocking(move || {
let (bytes, _) = read_all_data_std(&metadata_path)?;
let file_meta = FileMeta::load(&bytes).ok();
let data_dirs: HashSet<String> = file_meta
.as_ref()
.and_then(|meta| meta.get_data_dirs().ok())
.into_iter()
.flatten()
.flatten()
.map(|data_dir| data_dir.to_string())
.collect();
let probe = match os::read_dir_probe(&object_dir, data_dirs.len().saturating_add(2)) {
Ok(probe) => probe,
Err(err) if err.kind() == ErrorKind::NotFound => os::ReadDirProbe {
entries: Vec::new(),
complete: true,
},
Err(err) => return Err(ReadAllError::Disk(to_file_error(err).into())),
};
let found_child = probe.entries.into_iter().any(|entry| {
entry
.strip_suffix(SLASH_SEPARATOR)
.is_some_and(|child| !child.is_empty() && !data_dirs.contains(child))
});
let has_namespace_child_candidate = found_child || !probe.complete;
Ok(ListingMetadataRead {
bytes,
file_meta,
data_dirs,
has_namespace_child_candidate,
})
})
.await
.map_err(DiskError::from)?;
self.resolve_read_all_result(volume, &volume_dir, result).await
}
async fn resolve_read_all_result<T>(
&self,
volume: &str,
volume_dir: &Path,
result: core::result::Result<T, ReadAllError>,
) -> Result<T> {
match result {
Ok(value) => Ok(value),
Err(ReadAllError::Disk(err)) => Err(err),
Err(ReadAllError::Open(err)) => {
if err.kind() == ErrorKind::NotFound
&& !skip_access_checks(volume)
&& let Err(access_err) = access(volume_dir).await
&& access_err.kind() == ErrorKind::NotFound
{
warn!(
event = EVENT_DISK_LOCAL_READ_VERSION_FALLBACK,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "read_all_data_with_dmtime_volume_not_found",
error = ?access_err,
"Disk local read fallback failed"
);
return Err(DiskError::VolumeNotFound);
}
Err(to_file_error(err).into())
}
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn read_all_data_with_dmtime(
&self,
@@ -4671,66 +4842,12 @@ impl LocalDisk {
// succeeds the fd is valid, so fstat/read never return ENOENT. Hence
// gating the volume fallback on the open error alone is equivalent to
// the original code, where the fallback lived solely in the open match arm.
enum ReadAllError {
/// Raw, unmapped open() error; the async side decides the fallback.
Open(std::io::Error),
/// Already-mapped post-open error.
Disk(DiskError),
}
let path = file_path.as_ref().to_path_buf();
let res =
tokio::task::spawn_blocking(move || -> core::result::Result<(Vec<u8>, Option<OffsetDateTime>), ReadAllError> {
// Read-only open, equivalent to O_RDONLY (get_readonly_options only sets read(true)).
let mut f = std::fs::File::open(&path).map_err(ReadAllError::Open)?;
let meta = f.metadata().map_err(|e| ReadAllError::Disk(to_file_error(e).into()))?;
if meta.is_dir() {
return Err(ReadAllError::Disk(DiskError::FileNotFound));
}
let size = meta.len() as usize;
let mut bytes = Vec::new();
bytes
.try_reserve_exact(size)
.map_err(|e| ReadAllError::Disk(Error::other(e)))?;
std::io::Read::read_to_end(&mut f, &mut bytes).map_err(|e| ReadAllError::Disk(to_file_error(e).into()))?;
let modtime = match meta.modified() {
Ok(md) => Some(OffsetDateTime::from(md)),
Err(_) => None,
};
Ok((bytes, modtime))
})
let res = tokio::task::spawn_blocking(move || read_all_data_std(&path))
.await
.map_err(DiskError::from)?;
let (bytes, modtime) = match res {
Ok(v) => v,
Err(ReadAllError::Disk(e)) => return Err(e),
Err(ReadAllError::Open(e)) => {
if e.kind() == ErrorKind::NotFound
&& !skip_access_checks(volume)
&& let Err(er) = access(volume_dir.as_ref()).await
&& er.kind() == ErrorKind::NotFound
{
warn!(
event = EVENT_DISK_LOCAL_READ_VERSION_FALLBACK,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "read_all_data_with_dmtime_volume_not_found",
error = ?er,
"Disk local read fallback failed"
);
return Err(DiskError::VolumeNotFound);
}
return Err(to_file_error(e).into());
}
};
let (bytes, modtime) = self.resolve_read_all_result(volume, volume_dir.as_ref(), res).await?;
Ok((bytes, modtime))
}
@@ -5163,17 +5280,24 @@ impl LocalDisk {
let stall = opts.stall_timeout_duration();
// `count = -1` enumerates the whole directory in one `list_dir` call, and
// the stall budget bounds that entire enumeration as a single unit. On a
// WIDE, FLAT directory (millions of immediate children) that one readdir
// can exceed `stall` on a healthy disk and fail the whole walk -- see the
// wide-directory stall hazard documented on `list_dir`
// (rustfs/backlog#1216, a #2999 sub-class). Mitigate operationally with a
// larger `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS` or the high-latency
// drive-timeout profile; a streaming readdir rewrite is a separate,
// higher-risk follow-up and is intentionally not done here.
// Keep the existing in-memory sort contract, but bound each directory-entry
// read rather than treating the whole enumeration as one stalled disk
// operation. Object listing keeps using per-entry stall deadlines through
// `read_dir_entries_with_walk_stall` so wide prefixes can still be handled
// as a single logical read in API semantics.
let read_dir_started = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
let read_dir_result = with_walk_stall_timeout(stall, self.list_dir("", &opts.bucket, &current, -1)).await;
let dir_path_abs = self.get_object_path(&opts.bucket, current.trim_start_matches(SLASH_SEPARATOR))?;
let read_dir_result = match read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await {
Err(err) if err == Error::FileNotFound && !skip_access_checks(&opts.bucket) => {
let volume_dir = self.get_bucket_path(&opts.bucket)?;
if let Err(access_err) = access(&volume_dir).await {
Err(to_access_error(access_err, DiskError::VolumeAccessDenied).into())
} else {
Err(err)
}
}
result => result,
};
if let Some(started) = read_dir_started {
rustfs_io_metrics::record_list_objects_local_read_dir(rustfs_io_metrics::ListObjectsLocalReadDirObservation {
outcome: if read_dir_result.is_ok() {
@@ -5307,16 +5431,19 @@ impl LocalDisk {
}
}
let mut dir_stack: Vec<(String, bool, Option<HashSet<String>>)> = Vec::with_capacity(5);
let mut dir_stack: Vec<(String, bool, Option<HashSet<String>>, bool)> = Vec::with_capacity(5);
// Explicit directory markers and real directories can resolve to the same logical path.
let schedule_dir = |dir_stack: &mut Vec<(String, bool, Option<HashSet<String>>)>,
let schedule_dir = |dir_stack: &mut Vec<(String, bool, Option<HashSet<String>>, bool)>,
dir_name: String,
skip_object: bool,
dir_to_skip: Option<HashSet<String>>| {
if let Some((last_dir_name, existing_skip_object, existing_dir_to_skip)) = dir_stack.last_mut()
dir_to_skip: Option<HashSet<String>>,
scan_required: bool| {
if let Some((last_dir_name, existing_skip_object, existing_dir_to_skip, existing_scan_required)) =
dir_stack.last_mut()
&& *last_dir_name == dir_name
{
*existing_skip_object |= skip_object;
*existing_scan_required |= scan_required;
if let Some(existing_dir_to_skip) = existing_dir_to_skip {
if let Some(new_dir_to_skip) = &dir_to_skip {
existing_dir_to_skip.extend(new_dir_to_skip.iter().cloned());
@@ -5325,7 +5452,7 @@ impl LocalDisk {
*existing_dir_to_skip = dir_to_skip;
}
} else {
dir_stack.push((dir_name, skip_object, dir_to_skip));
dir_stack.push((dir_name, skip_object, dir_to_skip, scan_required));
}
};
prefix = "".to_owned();
@@ -5341,10 +5468,10 @@ impl LocalDisk {
let name = path_join_buf(&[current.as_str(), entry.as_str()]);
while let Some((last_name, _, _)) = dir_stack.last()
while let Some((last_name, _, _, _)) = dir_stack.last()
&& *last_name < name
{
let (pop, skip_object, dir_to_skip) = dir_stack.pop().expect("operation should succeed");
let (pop, skip_object, dir_to_skip, scan_required) = dir_stack.pop().expect("operation should succeed");
write_metacache_obj(
out,
&MetaCacheEntry {
@@ -5356,6 +5483,7 @@ impl LocalDisk {
let scan_path = pop.clone();
if opts.recursive
&& scan_required
&& let Err(er) =
Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
{
@@ -5389,9 +5517,25 @@ impl LocalDisk {
}
let fname = format!("{}/{}", meta.name, STORAGE_FORMAT_FILE);
let metadata_read = if opts.recursive && !is_dir_obj {
with_walk_stall_timeout(stall, self.read_listing_metadata(&opts.bucket, &meta.name))
.await
.map(|read| {
(
Bytes::from(read.bytes),
read.file_meta,
Some(read.data_dirs),
read.has_namespace_child_candidate,
)
})
} else {
with_walk_stall_timeout(stall, self.read_metadata(&opts.bucket, fname.as_str()))
.await
.map(|metadata| (metadata, None, None, true))
};
match with_walk_stall_timeout(stall, self.read_metadata(&opts.bucket, fname.as_str())).await {
Ok(res) => {
match metadata_read {
Ok((res, prefetched_file_meta, prefetched_data_dirs, has_namespace_child_candidate)) => {
if is_dir_obj {
meta.name = meta.name.trim_end_matches(GLOBAL_DIR_SUFFIX_WITH_SLASH).to_owned();
meta.name.push_str(SLASH_SEPARATOR);
@@ -5401,24 +5545,29 @@ impl LocalDisk {
write_metacache_obj(out, &meta).await?;
let file_meta = if opts.limit > 0 || opts.recursive || !is_dir_obj {
FileMeta::load(&res).ok()
} else {
None
let file_meta = match prefetched_file_meta {
Some(file_meta) => Some(file_meta),
None if opts.limit > 0 || opts.recursive || !is_dir_obj => FileMeta::load(&res).ok(),
None => None,
};
if opts.limit <= 0 || file_meta.as_ref().is_none_or(file_meta_counts_toward_limit) {
*objs_returned += 1;
}
let mut dir_to_skip = HashSet::new();
if let Some(file_meta) = file_meta.as_ref()
&& let Ok(data_dirs) = file_meta.get_data_dirs()
{
for data_dir in data_dirs.iter().flatten() {
dir_to_skip.insert(data_dir.to_string());
let dir_to_skip = if let Some(data_dirs) = prefetched_data_dirs {
data_dirs
} else {
let mut data_dirs_to_skip = HashSet::new();
if let Some(file_meta) = file_meta.as_ref()
&& let Ok(data_dirs) = file_meta.get_data_dirs()
{
for data_dir in data_dirs.iter().flatten() {
data_dirs_to_skip.insert(data_dir.to_string());
}
}
}
data_dirs_to_skip
};
if opts.recursive {
let mut dir_name = meta.name.clone();
@@ -5430,6 +5579,7 @@ impl LocalDisk {
dir_name,
true,
if dir_to_skip.is_empty() { None } else { Some(dir_to_skip) },
has_namespace_child_candidate,
);
} else if !is_dir_obj
&& self
@@ -5444,7 +5594,7 @@ impl LocalDisk {
// separate real directory entry handled above.
let mut dir_name = meta.name.clone();
dir_name.push_str(SLASH_SEPARATOR);
schedule_dir(&mut dir_stack, dir_name, true, None);
schedule_dir(&mut dir_stack, dir_name, true, None, true);
}
}
Err(err) => {
@@ -5462,7 +5612,7 @@ impl LocalDisk {
.directory_has_listing_entry(&opts.bucket, &meta.name, opts.incl_deleted, stall)
.await?
{
schedule_dir(&mut dir_stack, meta.name, false, None);
schedule_dir(&mut dir_stack, meta.name, false, None, true);
}
}
@@ -5483,7 +5633,7 @@ impl LocalDisk {
};
}
while let Some((dir, skip_object, dir_to_skip)) = dir_stack.pop() {
while let Some((dir, skip_object, dir_to_skip, scan_required)) = dir_stack.pop() {
if opts.limit > 0 && *objs_returned >= opts.limit {
return Ok(());
}
@@ -5499,6 +5649,7 @@ impl LocalDisk {
let scan_path = dir.clone();
if opts.recursive
&& scan_required
&& let Err(er) =
Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
{
@@ -6323,6 +6474,151 @@ impl DiskAPI for LocalDisk {
Ok(resp)
}
#[tracing::instrument(level = "trace", skip_all)]
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
let src_volume_dir = self.get_bucket_path(src_volume)?;
let dst_volume_dir = self.get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
let src_file_path = self.get_object_path(src_volume, src_path)?;
let dst_file_path = self.get_object_path(dst_volume, dst_path)?;
let dst_meta_path = self.get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
for path in [&src_file_path, &dst_file_path, &dst_meta_path, &transaction_path] {
check_path_length(path.to_string_lossy().as_ref())?;
}
let durability = effective_durability(dst_volume);
tokio::task::spawn_blocking(move || {
let source = std::fs::symlink_metadata(&src_file_path).map_err(to_file_error)?;
if !source.is_file() {
return Err(DiskError::FileAccessDenied);
}
if transaction_path.exists() {
return Err(DiskError::FileAccessDenied);
}
let Some(transaction_parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
};
std::fs::create_dir_all(transaction_parent).map_err(to_file_error)?;
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
std::fs::create_dir(&staging_path).map_err(to_file_error)?;
let prepare_result = (|| -> std::io::Result<()> {
snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
)?;
snapshot_part_transaction_file(
&dst_meta_path,
&staging_path.join(PART_TRANSACTION_OLD_META),
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
)?;
let mut new_meta = std::fs::OpenOptions::new()
.create_new(true)
.write(true)
.open(staging_path.join(PART_TRANSACTION_NEW_META))?;
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
os::fsync_dir_std(&staging_path)?;
}
std::fs::rename(&staging_path, &transaction_path)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(transaction_parent)?;
}
Ok(())
})();
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Err(to_file_error(err).into());
}
Ok(())
})
.await
.map_err(DiskError::from)?
}
#[tracing::instrument(level = "trace", skip_all)]
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.get_bucket_path(volume)?;
let current_data_path = self.get_object_path(volume, path)?;
let current_meta_path = self.get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.get_object_path(volume, &crate::disk::part_transaction_path(path))?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let durability = effective_durability(volume);
tokio::task::spawn_blocking(move || {
match std::fs::symlink_metadata(&transaction_path) {
Ok(metadata) if metadata.is_dir() => {}
Ok(_) => return Err(DiskError::FileCorrupt),
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()),
Err(err) => return Err(to_file_error(err).into()),
}
if action == PartTransactionAction::Rollback {
std::fs::write(transaction_path.join(PART_TRANSACTION_ROLLBACK), []).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(&transaction_path).map_err(to_file_error)?;
}
restore_part_transaction_file(
&current_data_path,
&transaction_path.join(PART_TRANSACTION_OLD_DATA),
&transaction_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
&transaction_path.join("restore.data"),
)
.map_err(to_file_error)?;
restore_part_transaction_file(
&current_meta_path,
&transaction_path.join(PART_TRANSACTION_OLD_META),
&transaction_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&transaction_path.join("restore.meta"),
)
.map_err(to_file_error)?;
if durability.syncs_commit_metadata()
&& let Some(parent) = current_data_path.parent()
{
os::fsync_dir_std(parent).map_err(to_file_error)?;
}
}
let Some(parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
};
let cleanup_path = parent.join(format!(".part-txn-settled-{}", Uuid::new_v4()));
std::fs::rename(&transaction_path, &cleanup_path).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(parent).map_err(to_file_error)?;
}
remove_dir_all_if_exists(&cleanup_path).map_err(to_file_error)?;
Ok(())
})
.await
.map_err(DiskError::from)??;
self.io_backend.invalidate_cached_fd(volume, path).await;
Ok(())
}
#[tracing::instrument(level = "trace", skip_all)]
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()> {
let src_volume_dir = self.get_bucket_path(src_volume)?;
@@ -6404,6 +6700,42 @@ impl DiskAPI for LocalDisk {
}
}
let transaction_publish_meta = if src_is_dir {
None
} else {
let transaction_path = self.get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match fs::read(&transaction_meta_path).await {
Ok(expected_meta) => {
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt);
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
let source_meta_path = transaction_meta_path.clone();
let publish_path = publish_meta_path.clone();
tokio::task::spawn_blocking(move || {
remove_file_if_exists(&publish_path)?;
std::fs::hard_link(source_meta_path, &publish_path)
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match fs::metadata(&transaction_path).await {
Ok(_) => return Err(DiskError::FileCorrupt),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err).into()),
},
Err(err) => return Err(to_file_error(err).into()),
}
};
// UploadPart is acknowledged once this rename lands, so the part data and
// its directory entry must be durable before we return. Relaxed keeps the
// part payload fdatasync but leaves the directory entry to the page cache.
@@ -6437,7 +6769,17 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
if let Some(transaction_publish_meta) = transaction_publish_meta {
let dst_meta_path = self.get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
rename_all(&transaction_publish_meta, &dst_meta_path, &dst_volume_dir).await?;
if durability.syncs_commit_metadata()
&& let Some(parent) = dst_meta_path.parent()
{
os::fsync_dir(parent).await.map_err(to_file_error)?;
}
} else {
self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
}
if let Some(parent) = src_file_path.parent() {
self.delete_file(&src_volume_dir, &parent.to_path_buf(), false, false).await?;
@@ -6590,15 +6932,20 @@ impl DiskAPI for LocalDisk {
/// escalate to a quorum failure and surface to the client as a ListObjects
/// 500, even though nothing is actually wrong with the drive.
///
/// This is deliberately NOT fixed here by rewriting the one-shot
/// `os::read_dir` into a streaming/batched readdir that would refresh the
/// stall deadline between chunks: that is an architecture-level change with
/// high regression surface (ordering, the `count` contract, quorum merge
/// semantics) and is tracked as a separate follow-up. The supported
/// mitigation for wide-directory deployments today is operational -- raise
/// `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS` or run with the high-latency
/// drive-timeout profile (see `get_drive_walkdir_stall_timeout`), both of
/// which widen the budget without any code change.
/// This path keeps the legacy one-shot call contract and does not attempt
/// per-entry timeout segmentation for compatibility reasons. Wide prefix
/// listing workarounds therefore remain operational: raise
/// `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS` or use the
/// high-latency drive-timeout profile from
/// `get_drive_walkdir_stall_timeout`.
///
/// This is deliberately NOT fixed here by rewriting this one-shot
/// `os::read_dir` into a streaming/batched readdir with per-chunk timeout
/// refresh: that is an architecture-level change with high regression
/// surface (ordering, the `count` contract, quorum merge semantics) and is
/// tracked as a separate follow-up. Object listing now uses
/// `scan_dir` + `read_dir_entries_with_walk_stall` to keep wide-prefix
/// listing resilient while preserving existing full-enumeration behavior.
#[tracing::instrument(level = "trace", skip_all)]
async fn list_dir(&self, origvolume: &str, volume: &str, dir_path: &str, count: i32) -> Result<Vec<String>> {
if !origvolume.is_empty() {
@@ -8153,6 +8500,12 @@ mod test {
file_info.data_dir = data_dir;
file_info.data = data;
file_info.size = size;
file_info.parts = vec![ObjectPartInfo {
number: 1,
size: usize::try_from(size).expect("test object size should fit usize"),
actual_size: size,
..Default::default()
}];
file_info.mod_time = Some(OffsetDateTime::now_utc());
file_info
}
@@ -9241,9 +9594,15 @@ mod test {
.await
.expect("source part should be written");
disk.prepare_part_transaction(tmp_volume, "upload/part.1", bucket, "object/part.1", meta.clone())
.await
.expect("part transaction should be prepared");
disk.rename_part(tmp_volume, "upload/part.1", bucket, "object/part.1", meta.clone())
.await
.expect("rename_part should commit part");
disk.settle_part_transaction(bucket, "object/part.1", PartTransactionAction::Commit)
.await
.expect("part transaction should be committed");
assert_eq!(
disk.read_all(bucket, "object/part.1")
@@ -9261,6 +9620,71 @@ mod test {
matches!(disk.read_all(tmp_volume, "upload/part.1").await, Err(DiskError::FileNotFound)),
"source part must be removed after a successful commit"
);
let legacy_payload = Bytes::from_static(b"legacy peer payload");
let legacy_meta = Bytes::from_static(b"legacy peer metadata");
disk.write_all(tmp_volume, "legacy/part.1", legacy_payload.clone())
.await
.expect("legacy source part should be written");
disk.rename_part(tmp_volume, "legacy/part.1", bucket, "object/part.1", legacy_meta.clone())
.await
.expect("pre-transaction peer RenamePart should remain supported");
assert_eq!(
disk.read_all(bucket, "object/part.1")
.await
.expect("legacy destination part should be readable"),
legacy_payload
);
assert_eq!(
disk.read_all(bucket, "object/part.1.meta")
.await
.expect("legacy destination metadata should be readable"),
legacy_meta
);
}
#[tokio::test]
async fn test_part_transaction_rolls_back_data_published_before_metadata() {
use tempfile::tempdir;
let dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
ensure_test_volume(&disk, "tmp").await;
ensure_test_volume(&disk, "bucket").await;
disk.write_all("tmp", "upload/part.1", Bytes::from_static(b"new data"))
.await
.expect("new part should be staged");
disk.write_all("bucket", "object/part.1", Bytes::from_static(b"old data"))
.await
.expect("old part data should be staged");
disk.write_all("bucket", "object/part.1.meta", Bytes::from_static(b"old metadata"))
.await
.expect("old part metadata should be staged");
disk.prepare_part_transaction("tmp", "upload/part.1", "bucket", "object/part.1", Bytes::from_static(b"new metadata"))
.await
.expect("part transaction should be prepared");
disk.rename_file("tmp", "upload/part.1", "bucket", "object/part.1")
.await
.expect("data publication should succeed");
disk.settle_part_transaction("bucket", "object/part.1", PartTransactionAction::Rollback)
.await
.expect("part transaction should roll back");
assert_eq!(
disk.read_all("bucket", "object/part.1")
.await
.expect("old part data should be restored"),
Bytes::from_static(b"old data")
);
assert_eq!(
disk.read_all("bucket", "object/part.1.meta")
.await
.expect("old part metadata should be restored"),
Bytes::from_static(b"old metadata")
);
}
struct BlockingScanWriter {
@@ -11916,6 +12340,52 @@ mod test {
assert!(err.is_metacache_output_stream_closed());
}
#[tokio::test]
async fn listing_metadata_probe_distinguishes_data_dirs_from_namespace_children() {
use tempfile::tempdir;
let dir = tempdir().expect("create temp dir");
let bucket = "test-bucket";
let object_dir = dir.path().join(bucket).join("object");
let data_dir = Uuid::parse_str("11111111-1111-1111-1111-111111111111").expect("parse data dir");
let version_id = Uuid::parse_str("22222222-2222-2222-2222-222222222222").expect("parse version id");
let mut file_meta = FileMeta::default();
let mut file_info = FileInfo::new("object", 1, 1);
file_info.data_dir = Some(data_dir);
file_info.version_id = Some(version_id);
file_info.mod_time = Some(OffsetDateTime::now_utc());
file_meta.add_version(file_info).expect("add object version");
fs::create_dir_all(object_dir.join(data_dir.to_string()))
.await
.expect("create version data dir");
fs::write(
object_dir.join(STORAGE_FORMAT_FILE),
file_meta.marshal_msg().expect("encode object metadata"),
)
.await
.expect("write object metadata");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("parse endpoint");
let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk");
let leaf = disk
.read_listing_metadata(bucket, "object")
.await
.expect("read leaf metadata");
assert!(!leaf.has_namespace_child_candidate);
assert!(leaf.data_dirs.contains(&data_dir.to_string()));
fs::create_dir(object_dir.join("child"))
.await
.expect("create namespace child");
let parent = disk
.read_listing_metadata(bucket, "object")
.await
.expect("read parent metadata");
assert!(parent.has_namespace_child_candidate);
}
#[tokio::test]
async fn test_scan_dir_includes_nested_object_dirs() {
use rustfs_filemeta::MetacacheReader;
@@ -11969,6 +12439,16 @@ mod test {
let mut reader = MetacacheReader::new(reader);
let entries = reader.read_all().await.expect("operation should succeed");
assert!(
entries.iter().any(|entry| entry.name == "asdf/" && entry.metadata.is_empty()),
"leaf object traversal markers must remain in the metacache stream"
);
assert!(
entries
.iter()
.any(|entry| entry.name == "foo/bar/" && entry.metadata.is_empty()),
"objects with namespace children must still emit a traversal prefix"
);
let names: Vec<String> = entries
.into_iter()
.filter(|entry| !entry.metadata.is_empty())
@@ -12065,6 +12545,69 @@ mod test {
);
}
#[test]
#[serial_test::serial]
fn scan_dir_leaf_object_avoids_recursive_scan() {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime should be created");
let recorder = crate::test_metrics::CapturingRecorder::default();
let previous_gate = rustfs_io_metrics::get_stage_metrics_enabled();
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
metrics::with_local_recorder(&recorder, || {
runtime.block_on(async {
use tempfile::tempdir;
let dir = tempdir().expect("tempdir should be created");
let bucket = "test-bucket";
let object_dir = dir.path().join(bucket).join("object");
fs::create_dir_all(&object_dir)
.await
.expect("object directory should be created");
fs::write(object_dir.join(STORAGE_FORMAT_FILE), b"meta")
.await
.expect("object metadata should be written");
let endpoint =
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let mut sink = tokio::io::sink();
let mut out = MetacacheWriter::new(&mut sink);
let opts = WalkDirOptions {
bucket: bucket.to_owned(),
recursive: true,
..Default::default()
};
let mut objects_returned = 0;
disk.scan_dir(String::new(), String::new(), &opts, &mut out, &mut objects_returned, false, None)
.await
.expect("scan_dir should succeed");
out.close().await.expect("metacache writer should close");
assert_eq!(objects_returned, 1);
});
});
rustfs_io_metrics::set_get_stage_metrics_enabled(previous_gate);
assert_eq!(
recorder.counter_value(
"rustfs_s3_list_objects_local_read_dir_total",
&[("outcome", "ok"), ("count_mode", "whole"), ("is_root", "true")]
),
1
);
assert_eq!(
recorder.counter_value(
"rustfs_s3_list_objects_local_read_dir_total",
&[("outcome", "ok"), ("count_mode", "whole"), ("is_root", "false")]
),
0
);
}
#[tokio::test]
async fn test_scan_dir_reports_base_dir_object_metadata() {
use rustfs_filemeta::MetacacheReader;
+85 -1
View File
@@ -38,6 +38,16 @@ pub const FORMAT_CONFIG_FILE: &str = "format.json";
pub const HEALING_MARKER_PATH: &str = "healing.bin";
pub const STORAGE_FORMAT_FILE: &str = "xl.meta";
pub const STORAGE_FORMAT_FILE_BACKUP: &str = "xl.meta.bkp";
pub const PART_TRANSACTION_NEW_META: &str = "new.meta";
pub const PART_TRANSACTION_OLD_META: &str = "old.meta";
pub const PART_TRANSACTION_ROLLBACK: &str = "rollback";
pub fn part_transaction_path(part_path: &str) -> String {
match part_path.rsplit_once('/') {
Some((parent, name)) => format!("{parent}/.{name}.rustfs-txn"),
None => format!(".{part_path}.rustfs-txn"),
}
}
use crate::cluster::rpc::RemoteDisk;
use crate::cluster::rpc::build_internode_data_transport_from_env;
@@ -52,7 +62,7 @@ use local::LocalDisk;
use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
use rustfs_madmin::info_commands::DiskMetrics;
use serde::{Deserialize, Serialize};
use std::{fmt::Debug, path::PathBuf, sync::Arc};
use std::{fmt::Debug, path::PathBuf, sync::Arc, time::Duration};
use time::OffsetDateTime;
use tokio::io::{AsyncRead, AsyncWrite};
use uuid::Uuid;
@@ -62,6 +72,12 @@ pub type DiskStore = Arc<Disk>;
pub type FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>;
pub type FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum PartTransactionAction {
Commit,
Rollback,
}
#[derive(Clone, Copy, Debug)]
pub struct MmapCopyStageMetrics {
pub(crate) path: &'static str,
@@ -381,6 +397,35 @@ impl DiskAPI for Disk {
}
}
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
match self {
Disk::Local(local_disk) => {
local_disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
}
Disk::Remote(remote_disk) => {
remote_disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
}
}
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
match self {
Disk::Local(local_disk) => local_disk.settle_part_transaction(volume, path, action).await,
Disk::Remote(remote_disk) => remote_disk.settle_part_transaction(volume, path, action).await,
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
match self {
@@ -453,6 +498,20 @@ impl DiskAPI for Disk {
}
impl Disk {
pub async fn ns_scanner_server_epoch(&self) -> Result<Option<Uuid>> {
match self {
Disk::Local(_) => Ok(None),
Disk::Remote(remote_disk) => remote_disk.ns_scanner_server_epoch().await,
}
}
pub async fn open_ns_scanner_stream(&self, request: NsScannerOpenRequest) -> Result<FileReader> {
match self {
Disk::Remote(remote_disk) => remote_disk.open_ns_scanner_stream(request).await,
Disk::Local(_) => Err(Error::other("namespace scanner stream requires a remote disk")),
}
}
pub fn runtime_state(&self) -> RuntimeDriveHealthState {
match self {
Disk::Local(local_disk) => local_disk.runtime_state(),
@@ -498,6 +557,18 @@ impl Disk {
}
}
#[derive(Debug)]
pub struct NsScannerOpenRequest {
pub request_id: Uuid,
pub server_epoch: Uuid,
pub session_id: Uuid,
pub session_sequence: u64,
pub next_cycle: u64,
pub leader_epoch: u64,
pub body: Vec<u8>,
pub stall_timeout: Option<Duration>,
}
impl Disk {
/// Reset drive health so `connect_load_init_formats` retries are not blocked by a prior
/// transient mark-faulty (same disk handles are reused across retries).
@@ -633,6 +704,19 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// ReadFileStream
async fn rename_file(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()>;
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()>;
async fn prepare_part_transaction(
&self,
_src_volume: &str,
_src_path: &str,
_dst_volume: &str,
_dst_path: &str,
_meta: Bytes,
) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn settle_part_transaction(&self, _volume: &str, _path: &str, _action: PartTransactionAction) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
// VerifyFile
async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
+59
View File
@@ -452,6 +452,49 @@ pub async fn is_empty_dir(path: impl AsRef<Path>) -> bool {
read_dir(path.as_ref(), 1).await.is_ok_and(|v| v.is_empty())
}
const READ_DIR_PROBE_RAW_LIMIT: usize = 256;
pub(crate) struct ReadDirProbe {
pub entries: Vec<String>,
pub complete: bool,
}
pub(crate) fn read_dir_probe(path: impl AsRef<Path>, entry_limit: usize) -> io::Result<ReadDirProbe> {
let mut dir = std::fs::read_dir(path)?;
let mut entries = Vec::with_capacity(entry_limit.min(READ_DIR_PROBE_RAW_LIMIT));
for _ in 0..READ_DIR_PROBE_RAW_LIMIT {
let Some(entry) = dir.next() else {
return Ok(ReadDirProbe { entries, complete: true });
};
let entry = entry?;
let name = entry.file_name().to_string_lossy().to_string();
if name.is_empty() || name == "." || name == ".." {
continue;
}
let file_type = entry.file_type()?;
if file_type.is_file() {
entries.push(name);
} else if file_type.is_dir() {
entries.push(format!("{name}{SLASH_SEPARATOR}"));
} else {
continue;
}
if entries.len() == entry_limit {
return Ok(ReadDirProbe {
entries,
complete: false,
});
}
}
Ok(ReadDirProbe {
entries,
complete: false,
})
}
// read_dir count read limit. when count == 0 unlimit.
/// Return file names in the directory.
#[tracing::instrument(level = "debug", skip_all)]
@@ -742,6 +785,22 @@ mod tests {
(logs, guard)
}
#[cfg(unix)]
#[test]
fn read_dir_probe_bounds_unsupported_entries() {
use std::os::unix::fs::symlink;
let temp_dir = tempdir().expect("create temp dir");
for index in 0..=READ_DIR_PROBE_RAW_LIMIT {
symlink("missing", temp_dir.path().join(format!("ignored-link-{index:04}"))).expect("create symlink");
}
let probe = read_dir_probe(temp_dir.path(), 1).expect("probe directory");
assert!(probe.entries.is_empty());
assert!(!probe.complete, "a bounded probe must not claim that an oversized directory is complete");
}
#[test]
fn global_file_sync_limit_scales_and_preserves_blocking_capacity() {
assert_eq!(default_global_file_sync_limit(1, 1024), MIN_GLOBAL_FILE_SYNCS);
@@ -21,6 +21,19 @@ use tokio::sync::RwLock;
/// Returns the backing [`tempfile::TempDir`]s alongside the set so callers keep
/// them alive for the test's duration and the directories are removed on drop.
pub(crate) async fn make_local_set_disks(drive_count: usize, parity_count: usize) -> (Vec<tempfile::TempDir>, Arc<SetDisks>) {
make_local_set_disks_with_ctx(drive_count, parity_count, crate::runtime::instance::bootstrap_ctx()).await
}
/// Like [`make_local_set_disks`], but binds the set to an explicit instance
/// context. Tests whose assertions depend on ambient runtime state that other
/// tests mutate (e.g. `SetupTypeGuard` flipping the shared setup type to
/// DistErasure from `#[serial]` tests in a different serial group) pass an
/// isolated context here so that state cannot leak into their code path.
pub(crate) async fn make_local_set_disks_with_ctx(
drive_count: usize,
parity_count: usize,
ctx: Arc<crate::runtime::instance::InstanceContext>,
) -> (Vec<tempfile::TempDir>, Arc<SetDisks>) {
let format = FormatV3::new(1, drive_count);
let mut dirs = Vec::with_capacity(drive_count);
let mut endpoints = Vec::with_capacity(drive_count);
@@ -55,7 +68,7 @@ pub(crate) async fn make_local_set_disks(drive_count: usize, parity_count: usize
disks.push(Some(disk));
}
let set_disks = SetDisks::new(
let set_disks = SetDisks::new_with_instance_ctx(
"ecstore-validation-blackbox".to_string(),
Arc::new(RwLock::new(disks)),
drive_count,
@@ -65,6 +78,7 @@ pub(crate) async fn make_local_set_disks(drive_count: usize, parity_count: usize
endpoints,
format,
Vec::new(),
ctx,
)
.await;

Some files were not shown because too many files have changed in this diff Show More