Compare commits

..

74 Commits

Author SHA1 Message Date
Zhengchao An 8c42499e83 Merge branch 'main' into reatang/fix-replication-switches 2026-08-01 10:44:38 +08:00
houseme 4b6b6f14bd feat(hotpath): use mimalloc in inner counting allocator (#5523)
* feat(hotpath): use mimalloc in counting allocator

* fix(hotpath): adapt mimalloc for allocation counting

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-01 10:43:52 +08:00
Zhengchao An 9080ea8ea0 test(object-lock): cover unretained version cleanup (#5504)
Co-authored-by: cxymds <cxymds@gmail.com>
2026-08-01 10:14:32 +08:00
Zhengchao An 739efaaea1 feat(kms): restore local backend key material from sealed backup bundles (#5522)
* feat(kms): add link_durably primitive and restore-marker startup guard

The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.

* feat(kms): record a master-key verifier and pre-seal decrypt probe in export

Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.

Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.

* feat(kms): restore local backend key material from sealed backup bundles

The consumer side of the Local bundle export, as a four-phase protocol:

- Dry-run: full in-memory bundle decode (digest and AEAD verification of
  every artifact), KDF-drift detection against the compiled-in
  derivation, deployment and injected-generation checks (strictly lower
  is rejected, equal stays allowed for repeated drills), master-key
  verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
  subdirectory (invisible to the backend's key scan and orphan-temp
  matcher) and every record is decryption-probed with the derived
  master key both in memory before staging and again from the staged
  bytes.
- Commit marker + cutover: the durably published .restore-commit.json
  marker is the single commit point; cutover publishes staged files via
  link_durably (salt first, keys after), then durably removes the
  marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
  and a re-run starts over; with the marker published, backend startup
  fails closed and a re-run with the same bundle rolls forward while
  abort_local_restore rolls back. Every interruption converges to the
  complete old or complete new state.

Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.

Refs rustfs/backlog#1572
2026-08-01 10:05:47 +08:00
Zhengchao An b6d4689c75 feat(policy): parse and match KMS key resources with a kms:Decrypt action (#5515)
Bring the dormant Resource::Kms variant to life: arn:aws:kms:::key/<key_id>
patterns (empty-account form, wildcards allowed in the id, alias/<name>
reserved as parse-only) now parse, validate, serialize back, and are matched
by KMS statements against the requested key id carried in Args::object.
Statements without KMS resources keep the legacy match-every-key behaviour,
as do call sites that pass no key resource, so nothing changes until the
admin/SSE authorization paths start passing key ids.

Statement validation rejects KMS resources on non-KMS statements, and bucket
policy validation rejects KMS actions and resources outright while stored
policies keep deserializing; evaluation skips pure-KMS bucket policy
statements with a warning. A kms:Decrypt action is added for the upcoming
SSE-KMS read path.

Refs rustfs/backlog#1582 (part of rustfs/backlog#1562)
2026-08-01 10:05:40 +08:00
Zhengchao An 8763cd0c67 fix(kms): CAS Transit metadata writes and bound the metadata cache (#5520)
* fix(kms): drop the KmsClient trait import removed by #5501

The local backup export tests (#5499) merged after #5501 folded the
KmsClient trait into KmsBackend, leaving a dead trait import that breaks
'cargo test -p rustfs-kms' compilation on main. create_key is an inherent
method on LocalKmsClient since #5501, so the import is unnecessary.

* fix(kms): CAS Transit metadata writes and bound the metadata cache

Transit KV metadata writes were whole-record overwrites with no
precondition, so two nodes mutating the same key could silently clobber
each other's lifecycle state, and the process-local metadata cache had
neither a TTL nor a capacity bound, so a key disabled or scheduled for
deletion on one node stayed usable on every other node until restart.

- Replace write_metadata_to_kv with a versioned read
  (read_metadata_from_kv_versioned) plus a check-and-set write
  (cas_write_metadata_to_kv); mutate_key_metadata re-reads the
  authoritative record and re-runs the state gate on every attempt, and a
  lost CAS race retries with a fresh snapshot (bounded budget) instead of
  replaying the stale one.
- Migrate every read-modify-write caller: enable, disable, schedule and
  cancel deletion, rotate version bump, the expired-key tombstone, and
  both create paths (create-only CAS that read-confirms the winner on a
  lost race).
- Bound the metadata cache with moka (300s TTL, 1024 entries) and drop a
  key's entry when a transit data call reports it gone server-side.
- Fail closed when the synthesized-metadata fallback cannot be read or
  persisted: the fabricated Enabled record is only served after a durable
  create-only CAS write, closing the gate weakening documented as a KNOWN
  RISK; the persistence fallback for pre-metadata keys is kept.

Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
2026-08-01 10:05:33 +08:00
Zhengchao An fdac60b0e2 fix(kms): make Vault KV2 lifecycle writes check-and-set (#5518)
* fix(kms): drop stale KmsClient trait import in local_export tests

The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.

* fix(kms): make Vault KV2 lifecycle writes check-and-set

Every KV2 lifecycle write used to be a blind whole-record overwrite, so
two nodes racing on the same key could lose updates: a disable racing a
rotation wrote the pre-rotation record back (rolling back the version
and material of a committed rotation), concurrent same-name creates let
the later material win (orphaning DEKs wrapped under the earlier one),
and a cancellation racing the deletion sweep could be overwritten by
the tombstone (or resurrect an already tombstoned key).

All lifecycle mutations now go through a bounded check-and-set
read-modify-write loop: each attempt re-reads the record pinned to its
KV2 secret version, re-runs the state gate against the fresh snapshot,
and writes back check-and-set against exactly that version; after
LIFECYCLE_CAS_ATTEMPTS lost races the typed conflict error surfaces.
The loop composes with the operation policy's single-attempt rule for
non-idempotent writes: each write is still sent at most once, only the
whole read-gate-write cycle repeats. create_key becomes a create-only
write (cas=0) so exactly one of two concurrent creates commits and the
loser reports KeyAlreadyExists. The blind store_key_data primitive is
now test-only.

Reads and rotation additionally fail closed when the version history is
inconsistent: resolving material through a version record above the
current pointer is refused (that state only arises when a lost update
rolled back a committed rotation), and rotation refuses to extend a
history whose records reach more than one step past the current pointer
(one step ahead is the footprint of an interrupted rotation and still
recovers through the adopt path).

Refs rustfs/backlog#1581
2026-08-01 09:36:12 +08:00
Zhengchao An 98b20b4231 test(s3): cover delete marker list visibility (#5503) 2026-08-01 09:31:21 +08:00
Zhengchao An ffc9de72cb docs: make validation proportional to change risk (#5527) 2026-08-01 09:31:03 +08:00
cxymds c09d11ff3b fix(config): fence persisted config updates and reloads (#5512)
* fix(config): fence persisted config updates and reloads

* fix(ci): unblock config and e2e checks

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-01 00:33:46 +00:00
Zhengchao An 792f2ef204 docs(kms): add observability dashboard, alert rules and runbook (#5517)
Add a Grafana dashboard for the four KMS backend operation metrics
emitted at the policy choke point, Prometheus alert rules with
conservative default thresholds (pending staging baseline calibration),
and an operations runbook documenting the metric contract and the
response procedure for each alert. Register the new alert rules file in
the observability READMEs.

Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
2026-07-31 21:28:10 +00:00
Zhengchao An 74019845c4 fix(kms): drop stale KmsClient trait import in local_export tests (#5519)
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
2026-07-31 20:24:04 +00:00
houseme 04c5921850 fix(s3): allow CopyObject COPY request metadata (#5514)
* fix(s3): allow CopyObject COPY request metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(kms): remove stale KmsClient import

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 17:52:00 +00:00
Zhengchao An db8039dece feat(kms): export local backend key material as sealed backup bundles (#5499)
* feat(kms): export local backend key material as sealed backup bundles

Adds the producer side of the Local backup series on top of the #5483
contract: a directory-wide export fence gives the snapshot a single
consistent generation, every artifact is AEAD-wrapped under a
caller-supplied backup KEK that is separate from the business trust
hierarchy, and the sealed manifest with completeness marker is written
last so an interrupted export can never be mistaken for a restorable
bundle. Restore and the admin API land in follow-up changes.

* fix(kms): verify manifest digest against raw bytes, not re-serialized fields

The decode path recomputed the digest by re-serializing the parsed
manifest, which silently assumes every field's stored spelling survives
a parse-and-reprint round trip. Timestamps do not guarantee that: the
time zone annotation jiff emits depends on the host (IANA name, POSIX
TZ string, Etc/Unknown), and the legacy-compat parser rewrites
bracket-less spellings to +00:00[UTC]. On CI this made freshly written
bundles fail digest verification while passing locally.

Digest verification now operates on the raw stored bytes, normalized
only through the JSON value layer with the digest slot emptied in
place; parsed typed fields are never re-serialized on the decode path.
Sealing uses the same value-layer canonical form, and the export
additionally pins created_at to UTC so bundles are host-independent. A
regression test seals a manifest whose created_at spelling cannot
round-trip and proves decoding still verifies. One behavior sharpens:
inserting an explicit null reserved slot after sealing is now rejected
as a digest mismatch instead of being tolerated.

* fix(kms): make manifest digest canonicalization independent of map ordering

The canonical digest form serialized serde_json values directly, which
inherits the key order of serde_json's map type: sorted by default, but
insertion-ordered when any crate in the unified build graph enables the
preserve_order feature. The workspace-wide CI build unified that
feature while a per-crate local build did not, so the frozen fixture
digest matched in one environment and not the other — and a bundle
sealed by one build flavor would fail verification in the other.

Canonicalization now rebuilds every JSON object with bytewise-sorted
keys (array order preserved) before hashing, so the digest bytes are
identical regardless of feature unification. Reproduced by enabling
preserve_order in dev-dependencies (fixture test red), then verified
green with the fix under both map flavors.
2026-07-31 23:57:39 +08:00
houseme 40eee6177a test: add formal hotpath ABBA runner (#5507)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 11:33:10 +00:00
Zhengchao An 76b3c085b5 refactor(kms): fold the KmsClient layer into KmsBackend and complete lifecycle overrides (#5501) 2026-07-31 08:31:21 +00:00
houseme 78d6918c52 feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.

Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 05:42:19 +00:00
Zhengchao An dd11145a26 feat(kms): record operation metrics in the retry policy engine (#5500)
* feat(kms): record operation metrics in the retry policy engine

Instrument policy::execute — the single choke point every outbound Vault
call and credential exchange already flows through — so no call site
needs its own instrumentation:

- rustfs_kms_backend_operations_total (counter): operation, op_class,
  outcome (success / fatal / budget_exhausted / deadline_exceeded /
  cancelled)
- rustfs_kms_backend_attempt_failures_total (counter): operation,
  error_class (retryable_conn / retryable_status / fatal /
  attempt_timeout)
- rustfs_kms_backend_operation_duration_seconds (histogram): wall-clock
  duration including retries and backoff
- rustfs_kms_backend_operation_attempts (histogram): attempts used

Metric labels carry only static enum values (operation names, classes,
outcomes) — never key identifiers, key material, ciphertext, or tokens.
Emission goes through the process-global metrics facade recorder, the
same pattern the rest of the workspace uses, so no new wiring is needed
in rustfs/src.

Tests drive a paused-clock runtime under a thread-local debugging
recorder, so counts, attempts, and even the recorded (virtual-clock)
durations are asserted deterministically with zero real sleeps.

Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)

* test(kms): add Vault fault-injection matrix

Offline cases inject transport faults locally and are fully
deterministic: a refused connection is retried up to the configured
budget, and a stalled connection is cut off by the per-attempt timeout
instead of hanging. Ignored cases run against a real dev Vault
(RUSTFS_KMS_VAULT_ADDR) and pin the fail-closed auth behavior: an
invalid token and a missing key each resolve in exactly one attempt.

Every case asserts through the policy metrics recorded by a
thread-local debugging recorder, which doubles as the request-count
assertion even against a real server. Throttling and recoverable 5xx
responses cannot be forced on a stock dev Vault; those paths stay
pinned by the scripted-Vault wiring tests and the engine tests.

Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
2026-07-31 02:56:44 +00:00
Zhengchao An f718e72e24 perf(ecstore): deduplicate concurrent lazy bucket-metadata loads (#5498) 2026-07-31 02:27:03 +00:00
houseme e06c9c02c6 feat: add hotpath primitive profiling coverage (#5492)
* chore(deps): refresh google cloud dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat: add hotpath primitive coverage

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat: extend hotpath profiling features

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: gate OPA hotpath client wrapping

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: avoid request-scoped hotpath primitive wrappers

Preserve the original bounded channel and stream semantics in EC and RIO request paths while keeping hotpath CPU profiling as a separate opt-in feature that implies base hotpath instrumentation.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 01:59:13 +00:00
Zhengchao An a2fe5d7d88 feat(admin): add KMS key lifecycle endpoints and deletion reference gate (#5496)
* feat(kms): add key lifecycle operations to the backend contract

Add enable_key/disable_key/rotate_key to KmsBackend with conservative
defaults returning the typed UnsupportedCapability error, mirroring
remove_expired_key. KmsManager gains matching pass-through methods and
drops cached key metadata after every successful state mutation so the
next describe observes backend truth. The local backend overrides
enable/disable, delegating to its state-machine-gated client methods;
rotation stays rejected, matching its advertised capabilities. New
dedicated policy actions kms:EnableKey and kms:DisableKey complete the
KMS action taxonomy alongside the existing kms:RotateKey.

* feat(admin): add KMS key enable/disable/rotate endpoints

POST /v3/kms/keys/enable, /v3/kms/keys/disable and /v3/kms/keys/rotate,
following the existing /v3/kms/keys handler conventions: key_id body
with keyId query fallback, {success, message, key_id, key_metadata}
responses, and 503 JSON while the KMS service is absent. Error mapping
keeps InvalidOperation/ValidationError at 400 like the sibling handlers
and surfaces UnsupportedCapability as 501 so a backend capability gap is
never mistaken for a missing key. Existing /v3/kms/keys handlers are
untouched apart from a visibility change on a private query helper.

* feat(kms): gate scheduled key deletion on bucket encryption references

Implement the DeletionReferenceChecker seam left by the deletion worker:
before any material is destroyed, every bucket's SSE configuration is
checked for a default KMS key reference and a hit blocks the removal.
The gate fails closed - an unpublished object store, a failed bucket
listing or an unreadable per-bucket encryption config all report a
blocking reference - because destroying key material is irreversible
while a blocked removal is simply retried on the next sweep. Registered
during init_kms_system before the service can start, so every worker
spawn observes it. Storage access goes through a new kms section of the
root storage facade.
2026-07-31 01:37:07 +00:00
Zhengchao An cad8246ffb feat(kms): route Vault operations through the retry policy engine (#5495)
* test(kms): add a scripted loopback Vault for policy wiring tests

A minimal HTTP/1.1 responder that serves canned Vault responses in order
and records the method/path sequence, so wiring tests can assert exactly
how many requests a code path performed (retries, read-confirm) without
a live Vault server.

* feat(kms): route Vault operations through the retry policy engine

Wire every outbound vaultrs call in the KV2 and Transit backends through
policy::execute, completing the wiring half of the operation policy work
(the engine landed separately):

- Reads (KV2 read/read_metadata/read_version/list, transit read/list/
  encrypt/decrypt, health checks) run as ReadIdempotent: bounded retries
  with exponential backoff and jitter on 429, recoverable 5xx, and
  connection-level failures; 400/401/403/404 stay fatal.
- Writes (KV2 set/CAS set/delete_metadata, transit create/update/rotate/
  delete, metadata writes) run as MutatingNonIdempotent: exactly one
  attempt under the per-attempt timeout, never replayed. CAS conflicts
  in the rotation protocol pass through unchanged as the concurrency
  signal they are.
- Each attempt takes a fresh credential snapshot, so a retry after a
  credential rotation uses the new token.
- Read-confirm recovery for lost create responses: when a create finds
  an existing key that is exactly what it would have produced (same
  algorithm, enabled, usable material, and for request-level creates the
  same usage/description/tags), it reports the stored key as the create
  result instead of KeyAlreadyExists. Any divergence keeps failing.
- Deletes treat already-deleted records as completed deletes (KV2
  version records; transit metadata already did), so re-running an
  interrupted deletion converges.
- A failed existence pre-check inside create now fails the create
  instead of falling through to a blind overwrite (fail closed).
- The policy module sheds its allow(dead_code) now that it is wired.

Wiring tests run against a scripted loopback Vault and assert request
counts and endpoints for the retry, single-attempt, CAS-conflict, and
read-confirm paths.

Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
2026-07-31 00:51:02 +00:00
Zhengchao An b4901abd17 fix(admin): keep data usage endpoint available (#5494) 2026-07-31 00:11:14 +00:00
Zhengchao An 1d383e239d fix(iam): separate OIDC role and claim policies (#5493) 2026-07-30 23:53:06 +00:00
Zhengchao An 2e29c330a9 feat(kms): enforce shared key state machine across backends (#5489)
* feat(kms): enforce shared key state machine across backends

Unify the key state x operation matrix behind a single gate in
backends/mod.rs and wire it into the Local, Vault KV2 and Vault Transit
backends: Disabled keys reject encryption, data key generation and
rotation while still allowing decryption and lifecycle recovery;
PendingDeletion keys reject everything except decryption and
cancellation (including repeated deletion scheduling); cancellation now
requires an actual pending deletion everywhere. This closes the missing
gates on KV2 encrypt/generate and Local generate_data_key, and stops
enable_key from silently reverting a pending deletion.

Decryption is deliberately left ungated in Disabled/PendingDeletion — an
explicit, documented and tested deviation from AWS KMS, since gating it
would break reads of existing objects the moment a key is disabled.

Add shared contract tests driving the full matrix offline for Local (and
via ignored tests against a live Vault for KV2/Transit), a stateless
contract for Static, an SSE-shaped regression proving existing envelopes
stay decryptable after disable, and a pin on the known-risk Enabled
default of Transit's synthesized metadata fallback.

Refs rustfs/backlog#1571 (part of rustfs/backlog#1562)

* feat(kms): persist deletion deadlines and run a restartable deletion worker (#5491)
2026-07-30 23:24:39 +00:00
Zhengchao An 3921336b23 feat(kms): AppRole login with background token renewal and fail-closed expiry (#5487)
* feat(kms): add AppRole configuration surface for Vault auth

Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every
login so external rotation is picked up), a configurable auth mount
(default "approle"), and an optional fail-closed safety window. All new
fields are serde(default) so previously persisted configurations keep
deserializing, and the strict admin-configure deserializer accepts them
as optional.

Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches
both Vault backends to AppRole; the secret_id comes from
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file
precedent. validate() rejects AppRole configs without a role_id, without
any secret_id source, or with an empty mount.

Also append the CredentialsUnavailable error variant used by the
fail-closed credential gate.

* feat(kms): implement AppRole login with background renewal and fail-closed expiry

Implement the AppRoleLogin token source (vaultrs approle login +
renew-self) and wire lease-bound credentials through the provider:

- Each successful login/renewal installs a new client generation in the
  ArcSwap; in-flight requests finish on the generation they captured.
- A background renewal task refreshes at half the lease TTL: renewable
  tokens are renewed in place, everything else (or a failed renewal)
  falls back to a fresh login. Auth exchanges run under the typed retry
  policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so
  the provider recovers once Vault does.
- Fail-closed: current() refuses to hand out a token inside the
  configured safety window of its expiry (default: one attempt timeout),
  returning CredentialsUnavailable instead of sending a request whose
  token may lapse mid-flight.
- Refreshes are single-flight: concurrent triggers for the same
  generation coalesce into one login.
- The renewal task's owner handle lives on the KMS service version:
  stop() shuts it down explicitly and reconfigure recycles it via
  cancel-on-drop when the old version is discarded.
- The secret_id file is re-read on every login attempt; missing or empty
  files fail the attempt without contacting Vault. Crate-owned copies of
  tokens and secret_ids are zeroized on drop, and Debug output of every
  credential-carrying type stays redacted (leak regression tests).

The renewal machinery is covered by paused-clock tests driving a
scripted token source: renew-at-half-TTL timing, login fallback,
fail-closed window entry and recovery, prompt task recycling, and
coalesced concurrent refreshes.

* feat(kms): add Vault Agent token file authentication

Add the TokenFile source: the token is read from an agent-managed sink
file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and
re-read once per poll interval (default 30s) through the existing
renewal loop, so a token rotated by the agent installs a new client
generation within one poll of the atomic replace. Each successful read
extends the token's observed validity to twice the poll interval; a
file that disappears or turns empty keeps failing the refresh until the
fail-closed window trips, and heals the provider as soon as it is
restored.

Reads are strict and never contact Vault on failure: the file must be
non-empty after trimming, and on Unix group/other permission bits are a
hard error (mirroring the SFTP host-key rule). Rotation detection uses
a content digest; the token itself is never stored on the source and
the crate-owned copy is zeroized.

Configuring the token file together with AppRole or an explicit static
token is rejected as a configuration error. All new config fields are
serde(default) and the strict admin-configure deserializer accepts the
new variant.

Covered by paused-clock tests (atomic replacement installs a new
generation next cycle, deletion fails closed and recovers, prompt task
recycling) plus negatives for missing/empty/over-permissive files and a
Debug leak regression.

* docs(kms): add Vault authentication and credential lifecycle runbook

Cover choosing between static token, AppRole, and Vault Agent token
file auth; AppRole role setup with SecretID delivery and rotation;
Agent sink deployment with the permission requirements; and the
fail-closed window semantics with a troubleshooting table keyed on the
renewal task's log lines.
2026-07-30 22:29:49 +00:00
Zhengchao An 342ee1df78 feat(kms): add backend capability discovery (#5485) 2026-07-31 06:09:43 +08:00
Zhengchao An 40ef0db9cc test(kms): pin rotation contracts across backends and document retention (#5486)
* feat(kms): retain historical master key versions for Vault KV2 rotation

Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:

- decrypt resolves the envelope's master_key_version to its version
  record; a missing version fails closed with KeyVersionNotFound and
  never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
  baseline_version frozen at the key's first rotation, so never-rotated
  keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
  snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
  the next version's material, then switch the current pointer; any
  failure leaves the current pointer untouched, and concurrent rotations
  serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
  deletion purges version records before the key record

Refs rustfs/backlog#1565

* test(kms): pin rotation contracts across backends and document retention

Completes the backlog#1565 series with the cross-backend regression net
and operator documentation:

- Vault Transit: ignored integration test proving version-prefixed
  historical ciphertext still decrypts after rotation, with no
  RustFS-side version bookkeeping in the envelope
- Local: offline mixed-format test interleaving pre-versioning and
  versioned envelopes through a rejected rotation; the existing
  rotation-rejection pinning test already covers material immutability
- docs: kms-backend-security.md gains the KV2 versioned retention
  model, version record retention/destruction preconditions, and the
  upgrade-before-first-rotation cluster constraint

Refs rustfs/backlog#1565
2026-07-31 01:49:55 +08:00
Zhengchao An b457c6abcc feat(kms): add backup manifest and responsibility contract types (#5483)
Contract-only module for KMS backup/restore (no handler or backend
wiring): versioned manifest schema with completeness marker and sealed
digest, the (backend, at-rest protection) responsibility matrix, typed
fail-closed errors, and the zero-write restore dry-run report. Fields
whose shape depends on in-flight contracts are reserved and reject data
in format version 1.
2026-07-31 01:49:21 +08:00
houseme 7051a5ce41 feat: add opt-in hotpath profiling (#5488)
* feat: add opt-in hotpath profiling

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: fix vault kms client construction

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 01:49:19 +08:00
Zhengchao An 1d3ba1eb8b feat(kms): retain historical master key versions for Vault KV2 rotation (#5484)
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:

- decrypt resolves the envelope's master_key_version to its version
  record; a missing version fails closed with KeyVersionNotFound and
  never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
  baseline_version frozen at the key's first rotation, so never-rotated
  keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
  snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
  the next version's material, then switch the current pointer; any
  failure leaves the current pointer untouched, and concurrent rotations
  serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
  deletion purges version records before the key record

Refs rustfs/backlog#1565
2026-07-31 01:48:10 +08:00
Zhengchao An 8368017fb2 refactor(kms): route Vault clients through a rotatable credential provider (#5481)
* fix(kms): repair vault test call sites missed by the timeout refactor

Three offline tests still constructed VaultKmsClient with the pre-#5472
single-argument signature, leaving cargo test -p rustfs-kms unable to
compile. Pass the same 30s attempt timeout the neighbouring tests use.

* refactor(kms): route Vault clients through a rotatable credential provider

Both Vault backends previously built a VaultClient in their constructor
and held it for the lifetime of the backend, which leaves no seam for
re-authentication: rotating credentials would require tearing down the
whole backend.

Introduce backends/vault_credentials with a TokenSource trait (only
StaticToken for now; AppRole login and agent token files land in
follow-ups) and a VaultCredentialProvider that owns the authenticated
client behind an ArcSwap. Request paths take a per-call snapshot via
current(), so a future rotation swaps in a new client generation without
interrupting calls already in flight. Tokens held by this crate are
zeroized on drop, and Debug output of every credential-carrying type is
redacted (covered by a leak regression test).

Behavior is unchanged: static token, namespace, and per-attempt timeout
feed the same VaultClientSettings as before, and AppRole configurations
are still rejected at construction with the same message.
2026-07-30 16:48:24 +00:00
Zhengchao An 699ef14ddd fix(kms): pass attempt timeout in vault kv2 tests (#5482)
PR #5472 added a required attempt_timeout parameter to
VaultKmsClient::new, while PR #5474 (developed in parallel) added
three tests calling it with the old single-argument signature,
breaking compilation of cargo test -p rustfs-kms. Pass the same
30-second timeout the neighboring tests use.
2026-07-30 16:47:06 +00:00
Zhengchao An 704ea43da5 fix(kms): pass attempt timeout to Vault test clients (#5479)
PR #5472 added an attempt_timeout parameter to VaultKmsClient::new while
PR #5474 was developed in parallel and added three tests using the old
single-argument signature. Both merged cleanly at the text level, leaving
main unable to compile the rustfs-kms test target. Align the three call
sites with the current constructor signature.
2026-07-30 16:44:21 +00:00
Zhengchao An 35a20622f1 feat(kms): add master key version to data key envelope contract (#5480)
* fix(kms): restore vault backend test compilation after timeout parameter

PR #5472 added an attempt_timeout parameter to VaultKmsClient::new while
PR #5474 landed tests still using the one-argument form, leaving
'cargo test -p rustfs-kms' unable to compile on main. Pass the same
30-second timeout the surrounding integration tests already use.

* feat(kms): add master key version to data key envelope contract

DataKeyEnvelope gains an optional master_key_version field recording
which KEK version wrapped the DEK, so rotation-aware backends can load
the matching historical material on decrypt. The field is skipped when
None, keeping envelopes from non-rotating backends byte-identical to
the historical seven-field JSON shape, and legacy envelopes without the
field deserialize to None. The envelope discriminator marker is
untouched, so mixed-format routing is unchanged in both directions.

Adds the KeyVersionNotFound typed error for version-addressed material
lookups that must fail closed instead of falling back to the current
version.

Refs rustfs/backlog#1565
2026-07-30 16:43:10 +00:00
Zhengchao An 7662b2436a docs(kms): document local backend durability and deployment support matrix (#5478)
* docs(kms): document local backend durability and deployment support matrix

* docs(kms): reflow backend security doc to one line per paragraph
2026-07-31 00:14:45 +08:00
Zhengchao An d4f2efa2ad fix(kms): report accurate Vault KV2 security contract and disable unsafe rotation (#5474) 2026-07-30 22:56:43 +08:00
Zhengchao An 19cdd806a2 fix(kms): fail closed on missing or corrupt key material (#5475) 2026-07-30 22:56:30 +08:00
Zhengchao An 6e5f330ff5 feat(kms): add operation timeout and typed retry policy engine (#5472) 2026-07-30 11:20:17 +00:00
Zhengchao An e86d4cb579 fix(kms): make local backend persistence crash-durable (#5471) 2026-07-30 11:13:37 +00:00
houseme e08847d2b6 docs(obs): add Grafana server-label dashboard (#5468)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-30 17:51:34 +08:00
Zhengchao An 145d38133b fix(ci): generate release notes with the correct baseline (#5467)
fix(ci): generate release notes with correct baseline
2026-07-30 14:09:13 +08:00
houseme 30dc04c94b fix(obs): label node-local metrics by server (#5465)
Add stable server labels to node-local Prometheus metrics and OTLP resource attributes so dashboards can distinguish per-node CPU, memory, host network, and internode traffic series.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-30 04:57:22 +00:00
唐小鸭 ad7663afd1 refactor(sse): decouple ecstore and harden KMS lifecycle (#5435)
* refactor(sse): decouple encryption from ecstore

* feat(kms): enhance KMS service manager with runtime state and persistence support

* feat(kms): add local key export functionality for SSE-S3 migration tests

* fix(kms): keep local key export narrowly scoped

* fix(sse): validate copy source customer algorithm

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-07-30 12:39:25 +08:00
houseme 6e6b38ad8e test(e2e): accept backpressure unknown terminal status (#5464)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-30 04:38:30 +00:00
唐小鸭 cbfd29ccdc fix(replication): restore MinIO tag filter behavior 2026-07-30 12:38:18 +08:00
唐小鸭 327f7118d6 fix(replication): satisfy delete admission clippy lint 2026-07-30 12:21:35 +08:00
cxymds 8601179c39 fix(ecstore): quarantine rejected format members (#5463) 2026-07-30 04:09:08 +00:00
Zhengchao An b83c9c4663 ci(release): isolate preview releases from latest channels (#5462) 2026-07-30 11:28:48 +08:00
cxymds 67904a6c18 fix(ecstore): start with unresolved Kubernetes peers (#5460)
* fix(ecstore): start with unresolved Kubernetes peers

* fix(ecstore): infer Kubernetes endpoint identity safely

* fix(ecstore): fail closed on unsafe format migration

* fix(ecstore): reject poisoned format heal candidates

* fix(ecstore): reject unsafe legacy migration outliers

* fix(ecstore): resume interrupted format migrations

* fix(ecstore): preserve Kubernetes startup compatibility
2026-07-30 11:14:38 +08:00
Zhengchao An 2e5cef513f chore(release): prepare 1.0.0-beta.12 (#5461)
* chore(release): prepare 1.0.0-beta.12

* chore(release): align release assets for 1.0.0-beta.12

* chore(deps): refresh release dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-30 01:31:28 +00:00
Zhengchao An 2d4f77fd3b fix(iam): add correctly named policy APIs (#5457) 2026-07-30 00:55:23 +00:00
Zhengchao An 920705417c refactor(ecstore): correct bucket config static names (#5458)
refactor(ecstore): fix bucket config static names
2026-07-30 00:53:16 +00:00
Zhengchao An b097c94c59 ci: model server as a composition root (#5459)
* ci: model server as a composition root

* test(ci): cover server to storage layer flow
2026-07-30 00:52:37 +00:00
Zhengchao An 3991a1d73c test(ecstore): stabilize late snapshot lease cleanup (#5456)
test(ecstore): wait for late snapshot lease cleanup
2026-07-30 00:10:38 +00:00
Zhengchao An 4bd765a2c0 Merge branch 'main' into reatang/fix-replication-switches 2026-07-30 07:27:05 +08:00
Zhengchao An 422e0ad768 test(ecstore): fix rename_all WARN flake from callsite-interest poisoning (#5448)
rename_all_missing_source_still_warns and rename_all_real_failure_still_warns
assert that `warn_reliable_rename_failure` emitted its WARN, but that is a
single production callsite shared with tests that call rename_all *without*
installing a subscriber — rename_all_missing_source_returns_file_not_found,
two tests above, is one of them.

tracing caches each callsite's Interest process-globally and the first thread
to reach a callsite fixes that value; while at most one dispatcher is
registered, tracing-core derives it from the registering thread's own
subscriber, and registration is once-only. When the subscriber-less sibling
wins, the callsite is cached as Interest::never() and the WARN never fires,
so the assertion sees empty output:

    ordinary missing-source failures must keep the WARN, got:

Reproduced at 3/25 with `disk::os::tests::rename_all_missing_source` (both
tests), against 0/20 for the victim alone. Fixed by pinning callsite interest
inside warn_capture(), so every current and future user of that helper is
covered rather than just the two tests that happen to fail today.

pin_callsite_interest_for_test() moves from cluster::rpc::background_monitor
to a new crate-level test_tracing module: it is domain-neutral and now has
consumers in two unrelated subsystems, and disk::os should not have to reach
into a cluster::rpc test helper.

Verified: repro filter 0/30 (was 3/25); disk::os:: 0/12; cluster::rpc:: 0/12
and its poisoner pair 0/15, confirming the moved helper still holds.

Follow-up to #5438. Closes the last item in #5439.
2026-07-30 07:26:12 +08:00
Zhengchao An 3a6212f597 fix(admin): fail closed for empty server context slots (#5452)
* fix(admin): fail closed for empty server context slots

* test(embedded): cover uninitialized context slot window

* test(embedded): authenticate startup probe as server B

* fix(admin): satisfy strict context-slot test lints
2026-07-30 07:25:33 +08:00
Zhengchao An ba964c82c7 fix(data-usage): classify 1024-byte objects correctly (#5451) 2026-07-30 07:24:45 +08:00
Zhengchao An d77439929c fix(server): preserve non-S3 trace context (#5450) 2026-07-30 07:24:23 +08:00
Henry Guo abc5f2e818 fix(scanner): persist portable usage cache keys (#5444)
* fix(scanner): persist portable usage cache keys

* fix(scanner): validate complete bucket cache graphs

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 23:22:30 +00:00
Zhengchao An 719c0d6ef0 feat(rpc): add replay-scoped internode authentication (#5455) 2026-07-29 23:12:38 +00:00
Zhengchao An 83f3a7320d test(ecstore): stabilize multipart lock ordering test (#5454) 2026-07-29 23:11:04 +00:00
Zhengchao An 2ed28f9c5f fix(ecstore): prevent recursive delete after empty bucket scan (#5453) 2026-07-29 21:06:28 +00:00
cxymds 0247c48ce0 fix(tiering): bind recovery to transaction metadata (#5409)
* fix(tiering): bind recovery to transaction metadata

* fix(tier): add operator transition reconciliation (#5410)

* fix(tier): add operator transition reconciliation

* fix(tiering): require live fleet capability proof (#5423)
2026-07-29 18:44:44 +00:00
唐小鸭 310ff96460 fix(replication): enforce bucket replication switches 2026-07-30 01:25:20 +08:00
Zhengchao An 88fa3877c1 fix(ecstore): serialize every bucket config write under the transaction lock (#5445)
Only replication-target writes took the bucket transaction lock. Every other
config write (policy, tagging, lifecycle, versioning, ...) went straight to
the process-local metadata-system guard, which serializes nothing across
nodes.

Each config write is a read-modify-write of one whole BucketMetadata blob:
load the blob, replace one field, save the blob back. The namespace locks
inside read_config/save_config are taken and released separately, so they do
not span that cycle. Two nodes updating different config files of the same
bucket therefore both load the same blob, each set their own field, and the
later save drops the other's -- with both clients already told 2xx. This is
not last-writer-wins on one document; an orthogonal config silently vanishes.

Route update(), delete() and update_config_with() through
acquire_config_write_guards(), which takes the cluster-wide transaction lock
first and the metadata-system write guard second. That order is load-bearing:
taking the process-local guard first would park every local reader and writer
of every bucket behind a lock whose holder may be another node, turning
remote contention into a local stall.

The lock is per bucket rather than per config file, since a per-file key
would let exactly the offending pair run concurrently. Rename the helper to
acquire_bucket_metadata_transaction_lock to match, but deliberately keep the
lock resource string as "bucket-targets/{bucket}/transaction.lock": the key
is what nodes agree on, so renaming it would leave a mixed-version cluster
with two disjoint keys and stop old and new nodes from excluding each other
on the very writes that are serialized today.

update_config_with() already narrowed its staleness window to a single load
and save, but its exclusion was explicitly process-local; it is now
cluster-wide, so its doc comment no longer disclaims cross-node races.

Also make update_and_parse load through self.api instead of the ambient store
handle, so the read and the write of one read-modify-write cannot resolve to
different instances.

The new tests drive two BucketMetadataSys instances over one ECStore -- the
in-process stand-in for two nodes, since they share no RwLock and can only be
serialized by the namespace lock. Verified the lost-update test has teeth by
removing the lock and confirming it fails on round 0, with the tagging config
clobbered to empty by the concurrent policy write.
2026-07-29 17:17:09 +00:00
GatewayJ 2dea4a9acf fix(s3): correlate server-owned request IDs (#5433)
* fix(s3): correlate server-owned request IDs

* fix(server): preserve trace context and Swift routing

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 23:50:59 +08:00
Zhengchao An 94ee597721 ci(s3select-query): inherit workspace lint policy (#5443) 2026-07-29 23:31:11 +08:00
Zhengchao An 962c11e6db ci(s3select-api): inherit workspace lint policy (#5441) 2026-07-29 23:06:33 +08:00
cxymds ae11bcf2be fix(tier): reconcile paginated remote versions (#5405)
* feat(tiering): model provider version capabilities

* feat(tiering): persist opaque remote versions

* fix(tiering): use exact GCS generations

* fix(tiering): gate remote version state safely

* fix(tiering): gate remote version state writes

* fix(tiering): preserve remote version state on delete

* fix(tiering): accept unversioned transition responses

* fix(tiering): replay exact cleanup journals

* test(tiering): pin empty exact cleanup guard

* test(tiering): accept strict missing journal errors

* test(tiering): exercise free-version identity guard

* test(tiering): reach destination identity guard

* test(tiering): persist version identity drift

* fix(tier): reconcile paginated remote versions

* style(tier): format candidate validation test

* test(tiering): bind version drift fixture

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-07-29 23:04:36 +08:00
houseme 09157485aa fix(notify): reconcile persisted bucket rules (#5437)
Restore persisted bucket notification rules after the notification target runtime converges so restarted nodes rebuild their local rule engine without requiring an unchanged PUT bucket notification request.

Fixes #5428

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 15:00:13 +00:00
Zhengchao An e2257325a2 test(ecstore): fix two cluster::rpc flakes from process-global test state (#5438)
peer_rest_recovery_probe_logs_keep_request_id_span_context failed ~10% of
`cargo test -p rustfs-ecstore --lib -- cluster::rpc::` runs with
left: "request-span", right: "recovery-monitor". The recovery-monitor
info_span! was evaluating to Span::none(), so the probe's log line landed
under the caller's span.

tracing caches each callsite's Interest in process-global state, and the
first thread to reach a callsite fixes that value. While at most one
dispatcher is registered, tracing-core takes a fast path that derives the
interest from the registering thread's own subscriber, and registration is
once-only (CAS). Under libtest a sibling test reaches recovery_monitor_span
via mark_offline_and_spawn_recovery from a thread with no subscriber, so
the interest is derived from NoSubscriber and cached as Interest::never()
for the whole process.

Add pin_callsite_interest_for_test(): registering a second, inert
dispatcher rebuilds every registered callsite's interest against the live
dispatcher set (repairing a poisoned value) and keeps tracing-core off the
single-dispatcher fast path (preventing new ones). This also covers the
production marked_suspect / recovery_monitor_started event callsites that
remote_disk_network_error_starts_recovery_monitor_with_request_context
asserts on.

rename_data_response_accepts_legacy_json_without_decode_error is a
separate root cause: it snapshots the process-global internode metrics and
asserts the decode-error counter did not move, which siblings that record
decode errors (or reset the counters) invalidate. Put the 11 tests that
observe those counters in one #[serial(internode_metrics)] group.

Both races are impossible under nextest, which runs each test in its own
process, so neither test belongs in the ecstore-serial-flaky test-group
(that serializes across process boundaries) nor in the ci-profile
quarantine (they never redden CI).

Verified: cluster::rpc:: subset 0/30 failures under libtest (was 3/30);
target test paired with its poisoner 0/30 (was 4/20); 5/5 clean under
nextest at 179/179.
2026-07-29 14:59:41 +00:00
Zhengchao An c9397405ed ci(protocols): inherit workspace lint policy (#5436) 2026-07-29 22:35:02 +08:00
307 changed files with 38444 additions and 5637 deletions
+8 -2
View File
@@ -10,10 +10,16 @@ never weaken a check to get green.
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
upward imports. Known legacy violations live in
Enforces `composition (server, startup/init) → interface (admin,
storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
files are composition roots, while imports of their exported HTTP contracts
are classified as interface dependencies. Known legacy violations live in
`scripts/layer-dependency-baseline.txt`.
Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
Inline `#[cfg(test)]` imports remain checked under their source file's layer;
move architecture-crossing test scaffolding into a dedicated test module.
- **New violation**: restructure your change so the dependency points
downward (move the shared type/function to the lower layer).
- **You legitimately removed a baseline entry**: run
+80 -16
View File
@@ -1,19 +1,21 @@
---
name: rustfs-release-publish
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
bump version files to <target> (final version, ONE commit) -> merge
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify release artifacts -> run binary locally + console checks
-> verify preview Release assets -> run binary locally + console checks
-> validate with latest rc client
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
@@ -23,7 +25,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` after `git fetch --tags`).
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
@@ -45,14 +47,18 @@ Rules:
## Preview tag naming
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N``build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -63,6 +69,61 @@ Rules:
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
```bash
CONSOLE_REPO="rustfs/console"
CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
--jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
```
- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
```bash
CONSOLE_SCRATCH=$(mktemp -d)
gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
```
If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
```bash
git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
```
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
3. Find the exact tag run and wait for completion:
```bash
gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
```
4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
```bash
gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
```
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -85,16 +146,17 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and artifact verification
## Phase 3 — CI and preview Release verification
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
- `isPrerelease` must be `true`.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
## Phase 4 — Run the artifact locally, verify the console
@@ -157,13 +219,15 @@ git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
+10
View File
@@ -60,6 +60,16 @@ The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-con
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
| Alert | Severity | Condition |
|-------|----------|-----------|
| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
### Enabling Alert Rules
Add the alert rules file to your Prometheus configuration:
+10
View File
@@ -60,6 +60,16 @@
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线,持续 10 分钟 |
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%,持续 5 分钟 |
文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值,待 staging 基线校准;响应流程见 `docs/operations/kms-observability-runbook.md`,配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`
| 告警 | 级别 | 条件 |
|------|------|------|
| `KmsBackendFatalErrors` | 严重 | fatal(不可重试)尝试失败 > 0,持续 5 分钟 |
| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%,持续 10 分钟(含流量下限保护) |
| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时(含重试)> 2s,持续 10 分钟 |
| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s,持续 10 分钟 |
| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s,持续 10 分钟 |
### 启用告警规则
在 Prometheus 配置中添加告警规则文件:
@@ -0,0 +1,188 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =============================================================================
# RustFS KMS backend — Prometheus alerting rules
# =============================================================================
#
# Metric source: the KMS operation-policy choke point in
# crates/kms/src/policy.rs. All label values are static enum strings
# (operation, op_class, outcome, error_class); key identifiers, key material,
# and tokens never appear in labels.
#
# Response procedures: docs/operations/kms-observability-runbook.md
#
# IMPORTANT — threshold status: every numeric threshold below is a
# conservative default chosen without a production baseline. Calibrate against
# a staging baseline before relying on these alerts for paging, and prefer
# loosening over tightening until the baseline exists. Formal SLO targets are
# deliberately not encoded here (see rustfs/backlog#1584).
#
# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
# extension or the file is silently ignored by the docker-compose stack.
#
# Validate: promtool check rules rustfs-kms-alerts.yml
# =============================================================================
groups:
# ==========================================================================
# Critical alerts — immediate action required
# ==========================================================================
- name: rustfs-kms-critical
interval: 30s
rules:
# ------------------------------------------------------------------
# 1. KmsBackendFatalErrors
# Any attempt failure classified as fatal (non-retryable): auth
# or permission errors, malformed requests, missing keys. The
# policy never retries these, so even a low rate means real
# operations are failing right now.
# ------------------------------------------------------------------
- alert: KmsBackendFatalErrors
expr: |
sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
for: 5m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
description: >-
Attempt failures classified as fatal are occurring at
{{ $value | printf "%.3f" }}/s on operation
{{ $labels.operation }}. Fatal failures are not retried:
each one is a KMS backend call that failed permanently
(authentication, permissions, malformed request, or a
missing key/version).
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
# ------------------------------------------------------------------
# 2. KmsBackendHighErrorRate
# Sustained share of operations terminating without success
# (fatal, budget_exhausted, deadline_exceeded). The cancelled
# outcome is excluded because shutdowns legitimately produce it.
# The traffic guard keeps a single failure on a near-idle
# cluster from firing the alert.
# Threshold: 5% for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendHighErrorRate
expr: |
(
sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
/
clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
) > 0.05
and
sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
for: 10m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend non-success ratio above 5% for 10m"
description: >-
{{ $value | humanizePercentage }} of KMS backend operations
are terminating in fatal, budget_exhausted, or
deadline_exceeded. Object encryption and decryption paths
depending on the KMS are degraded or failing.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
# ==========================================================================
# Warning alerts — investigation needed
# ==========================================================================
- name: rustfs-kms-warning
interval: 30s
rules:
# ------------------------------------------------------------------
# 3. KmsBackendP99LatencyHigh
# p99 wall-clock duration of whole operations (attempts plus
# backoff) is sustained above 2 seconds. Because the histogram
# includes retries, a high p99 usually means the retry policy
# is absorbing backend failures, not that every call is slow.
# Threshold: 2s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendP99LatencyHigh
expr: |
histogram_quantile(0.99,
sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
) > 2
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operation p99 latency above 2s for 10m"
description: >-
The 99th-percentile KMS backend operation duration is
{{ $value | humanizeDuration }}, including retries and
backoff. Encryption and decryption latency is leaking into
S3 request latency.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
# ------------------------------------------------------------------
# 4. KmsBackendAttemptFailureSpike
# Aggregate attempt-failure rate (all error classes) sustained
# above an absolute floor. An absolute threshold is used instead
# of an offset-1d baseline ratio because fresh deployments have
# no baseline and an empty offset vector would keep a ratio
# alert from ever firing; switch to a baseline-relative form
# (see rustfs-get-optimization-alerts.yaml for the pattern)
# once a stable staging baseline exists.
# Threshold: 0.5/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendAttemptFailureSpike
expr: |
sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend attempt failures above 0.5/s for 10m"
description: >-
KMS backend attempts are failing at
{{ $value | printf "%.2f" }}/s across all error classes.
The retry policy may still be masking these from callers —
check the error-class breakdown before it stops absorbing
them.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
# ------------------------------------------------------------------
# 5. KmsBackendRetryBudgetExhausted
# Operations are running out of retry budget (budget_exhausted)
# or operation deadline (deadline_exceeded). These surface to
# callers as failed KMS operations even though every individual
# failure was retryable — the backend is unhealthy for longer
# than the policy can bridge.
# Threshold: 0.05/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendRetryBudgetExhausted
expr: |
sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
description: >-
KMS backend operations are terminating as
{{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
Retryable failures are outlasting the retry budget, so
callers are seeing hard failures.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
+7
View File
@@ -23,6 +23,8 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
pull_request:
types: [ opened, synchronize, reopened, closed ]
@@ -33,6 +35,8 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
schedule:
- cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
@@ -96,6 +100,9 @@ jobs:
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+50 -81
View File
@@ -107,13 +107,21 @@ jobs:
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - release or prerelease
# Tag push - preview, release, or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
# Preview tags publish a GitHub prerelease for validation, but
# must not update any latest channel.
if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Preview build detected: $tag_name"
elif [[ "$tag_name" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
@@ -714,6 +722,10 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
@@ -732,7 +744,9 @@ jobs:
echo ""
echo "🐳 Docker Images:"
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -740,11 +754,11 @@ jobs:
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release (only for tag pushes)
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
permissions:
contents: write
@@ -767,9 +781,12 @@ jobs:
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
@@ -783,54 +800,24 @@ jobs:
RELEASE_TYPE="release"
fi
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
TITLE="RustFS $VERSION"
fi
echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
echo "Created release: $RELEASE_URL"
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
permissions:
contents: write
@@ -920,8 +907,8 @@ jobs:
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
steps:
- name: Update latest.json
@@ -980,51 +967,33 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Update release notes and publish
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
RELEASE_TYPE="release"
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
+4
View File
@@ -173,6 +173,10 @@ jobs:
run: cargo clippy --all-targets -- -D warnings
- name: Run nextest tests
env:
# Three concurrent workspace test links saturate the self-hosted
# runner's overlay I/O and can wedge Cargo until the 75m timeout.
CARGO_BUILD_JOBS: "2"
run: |
mkdir -p artifacts/test-and-lint
# Evidence sampler for issue #5394: the post-mortem pgrep below runs
+9 -1
View File
@@ -82,7 +82,8 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
runs-on: ubuntu-latest
outputs:
should_build: ${{ steps.check.outputs.should_build }}
@@ -220,6 +221,13 @@ jobs:
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
+3 -2
View File
@@ -33,11 +33,12 @@ jobs:
build-helm-package:
runs-on: ubuntu-latest
if: |
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.')
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
+64 -19
View File
@@ -105,33 +105,78 @@ CI) fails the build if anything is committed under `docs/superpowers/`, even via
## Verification Before PR
Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
Convert changes into independently verifiable outcomes. This section controls
agent-run local validation; preparing a commit or PR does not by itself require
the broadest gate. Inspect only the final task-owned diff, classify it by
behavioral impact rather than line count or path alone, and run the smallest
set of checks that provides meaningful coverage. Do not let unrelated
worktree changes or a generic contributor checklist expand the scope.
Non-exempt changes must also pass Adversarial Validation (next section) before
the checks below count as completion.
For code changes, run and pass the following before opening a PR:
### Validation floor
```bash
make pre-pr
```
- Every change that is not documentation-only must finish with
`cargo fmt --all --check` passing. An umbrella gate that runs this exact
check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
only when formatting needs to be fixed. Run the configured formatter or
validator for other changed languages when one exists.
- Documentation-only or instruction-only means all task-owned changes are
prose or documentation assets and cannot affect runtime, builds, CI,
dependencies, generated code, or tests. Run `git diff --check` and any
relevant documentation guard, but skip Cargo formatting, compilation,
Clippy, tests, `make pre-commit`, and `make pre-pr`.
- Behavior changes require relevant existing or new tests. Prefer the most
focused test or affected package. A passing targeted test can also provide
sufficient compilation coverage when it builds every changed target and
feature involved; do not add a redundant `cargo check` in that case.
- `cargo check` supplements compilation coverage; it never substitutes for a
behavioral test. If a relevant test cannot reasonably be added or run, use
the narrowest compilation check and report the reason and remaining risk.
Before committing code changes, prefer focused verification for the touched
surface and use the faster local gate when a broad smoke check is needed:
### Validation tiers
```bash
make pre-commit
```
1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
such as `make doc-paths-check` only when it is relevant to the edited text.
2. **Non-behavioral source change:** For comments, formatting, or another
demonstrably non-executable change, run the formatting floor. Compilation,
Clippy, and tests may be skipped only when the edit cannot affect
compilation or runtime behavior; run targeted doctests if executable
documentation examples changed.
3. **Localized or bounded behavior change:** Run the formatting floor and the
narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
for changed targets, features, APIs, error handling, async behavior, or
control flow not already covered. When several crates are affected but the
dependency set is identifiable, validate those packages and known
dependents instead of the whole workspace. Use `make pre-commit` only when
a repository-wide fast gate adds useful confidence beyond those checks.
4. **Broad or high-risk change:** Run `make pre-pr` only when targeted coverage
cannot bound the impact, including:
- dependency, feature, build-script, procedural-macro, code-generation,
toolchain, or CI changes that alter compilation or the test matrix;
- cross-crate public APIs, shared foundational code, or broad refactors with
an unbounded dependent set;
- locking, storage durability or formats, erasure coding, replication,
RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
security-sensitive behavior;
- a targeted check that reveals wider impact, an explicit user request, or
a release policy that requires the full gate.
For migration batches, do not run the full `make pre-pr` gate before every
intermediate commit. Use focused tests and `make pre-commit` during
development, then reserve `make pre-pr` for the final PR-ready branch.
Documentation-only and non-behavioral classifications take precedence over
path-based triggers. A small diff can still be high-risk, while a CI comment,
manifest comment, or release-note edit does not require full validation.
Before pushing code changes, make sure formatting is clean:
`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
unchanged diff, and do not repeat equivalent checks during PR preparation or
because a local hook already ran them. Rerun only checks whose scope is affected
by later edits. Full workspace checks do not replace a relevant integration or
E2E test for changed behavior; run that focused test when required and
available, or report why it was not run and the remaining risk.
- Run `cargo fmt --all`.
- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
If `make` is unavailable, run the equivalent checks defined under
`.config/make/`. At handoff, list the checks actually run, checks intentionally
skipped, and the reason for the selected tier.
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
Do not open a PR with code changes when the required checks fail.
Make a failing check pass by fixing the cause, never by weakening the gate:
Generated
+254 -159
View File
File diff suppressed because it is too large Load Diff
+59 -54
View File
@@ -69,7 +69,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.97.1"
version = "1.0.0-beta.11"
version = "1.0.0-beta.12"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,58 +86,58 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.0-beta.11" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.11" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.11" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.11" }
rustfs-common = { path = "crates/common", version = "1.0.0-beta.11" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.11" }
rustfs-config = { path = "./crates/config", version = "1.0.0-beta.11" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.11" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.11" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.11" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.11" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.11" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.11" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.11" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.11" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.11" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.11" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.11" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.11" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.11" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.11" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.11" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.11" }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.11" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.11" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.11" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.11" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.11" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.11" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.11" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.11" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.11" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.11" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.11" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.11" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.11" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.11" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.11" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.11" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.11" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.11" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.11" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.11" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.11" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.11" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.11" }
rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.18" }
mysql_async = { default-features = false, version = "0.37" }
async-compression = { version = "0.4.42" }
async-compression = { version = "0.4.43" }
async-recursion = "1.1.1"
async-trait = "0.1.91"
async-nats = { version = "0.50.0", default-features = false }
@@ -152,7 +152,7 @@ lapin = { default-features = false, version = "4.10.0" }
hyper = { version = "1.11.0" }
hyper-rustls = { default-features = false, version = "0.27.9" }
hyper-util = { version = "0.1.20" }
http = "1.4.2"
http = "1.5.0"
http-body = "1.1.0"
http-body-util = "0.1.4"
minlz = "1.2.3"
@@ -200,7 +200,7 @@ jsonwebtoken = { version = "11.0.0" }
openidconnect = { default-features = false, version = "4.0" }
pbkdf2 = "0.13.0"
rsa = { version = "=0.10.0-rc.18" }
rustls = { default-features = false, version = "0.23.42" }
rustls = { default-features = false, version = "0.23.43" }
rustls-native-certs = "0.8"
rustls-pki-types = "1.15.1"
sha1 = "0.11.0"
@@ -250,12 +250,13 @@ enumset = "1.1.14"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.4"
google-cloud-storage = "1.16.0"
google-cloud-auth = "1.14.0"
google-cloud-storage = "1.17.0"
google-cloud-auth = "1.15.0"
hashbrown = { version = "0.17.1" }
hex = "0.4.3"
hex-simd = "0.8.0"
highway = { version = "1.3.0" }
hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.0"
libc = "0.2.189"
@@ -283,7 +284,8 @@ reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.4.1" }
redis = { version = "1.5.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
@@ -303,6 +305,7 @@ test-case = "3.3.1"
thiserror = "2.0.19"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.33" }
tracing-subscriber = { version = "0.3.23" }
@@ -313,7 +316,9 @@ uuid = { version = "1.24.0" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.18" }
zip = "8.6.0"
zstd = "0.13.3"
@@ -343,7 +348,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1.52"
hotpath = "0.22.0"
hotpath = { version = "0.22.0", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+1 -1
View File
@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
+1 -1
View File
@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
+26
View File
@@ -25,7 +25,33 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
keywords = ["audit", "target", "management", "fan-out", "RustFS"]
categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-config/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-targets/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-targets/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-targets/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-targets = { workspace = true }
rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
rustfs-s3-types = { workspace = true }
+7
View File
@@ -28,7 +28,14 @@ documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
+7
View File
@@ -27,7 +27,14 @@ categories = ["web-programming", "development-tools", "data-structures"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
+7
View File
@@ -13,7 +13,14 @@ categories = ["concurrency", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
# Internal crates
rustfs-io-core = { workspace = true }
serde = { workspace = true, features = ["derive"] }
+4
View File
@@ -25,6 +25,7 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -34,6 +35,9 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
+4
View File
@@ -66,6 +66,10 @@ Current guidance:
- `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node.
## Distributed endpoint locality
- `RUSTFS_LOCAL_ENDPOINT_HOST` identifies this server's host in a distributed `RUSTFS_VOLUMES` topology without resolving every peer during startup. Set it to exactly one host, without a scheme, port, or path. It is accepted only for orchestrated URL topologies and must match at least one endpoint on the RustFS server port; invalid or unmatched values fail startup. Leave it unset to retain DNS-based locality discovery.
## Scanner environment aliases
- `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
+4
View File
@@ -131,6 +131,10 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
/// Environment variable for server volumes.
pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
/// Environment variable identifying this server's host in distributed endpoint
/// lists without relying on DNS locality discovery.
pub const ENV_LOCAL_ENDPOINT_HOST: &str = "RUSTFS_LOCAL_ENDPOINT_HOST";
/// Environment variable to explicitly bypass local physical disk independence checks.
pub const ENV_UNSAFE_BYPASS_DISK_CHECK: &str = "RUSTFS_UNSAFE_BYPASS_DISK_CHECK";
+31 -9
View File
@@ -158,17 +158,33 @@ pub const DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT: bool = false;
// rolling upgrades until the fleet-wide body-digest fallback counter reads zero.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of body-bound v2 signatures.
/// Require the replay-scoped internode RPC signature after the fleet has converged on it.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope, so the steady
/// state holds roughly `mutating RPS x 601s` entries; the default sustains ~1,700 body-bound
/// mutating RPCs per second (about 120 MiB worst case, allocated only under sustained load).
/// Overflow fails closed — legitimate signed traffic is the only thing that can fill the cache
/// (replays are rejected before insertion, and an attacker cannot mint valid nonces without the
/// shared secret) — and increments
/// The default keeps v1/v2 peers available during a rolling upgrade. Operators may set this only
/// after `rustfs_system_network_internode_replay_scope_fallback_total` remains zero for a full
/// release window. The node still accepts a v2-authenticated `Ping` carrying an epoch challenge:
/// that narrowly scoped bootstrap lets an upgraded client learn the receiving process epoch and
/// immediately retry with the replay-scoped signature after a peer restart.
pub const ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT: &str = "RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT";
pub const DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT: bool = false;
// Compile-time invariant: mixed-version clusters must remain available until operators make the
// observed fallback counter an explicit strictness decision.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. The default sustains about 1,700
/// authenticated RPCs per second (about 120 MiB worst case, allocated only under sustained load);
/// operators must size it for the node's aggregate peak RPC rate before enabling strict replay
/// scope. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak mutation rate.
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
@@ -354,6 +370,12 @@ mod tests {
assert_eq!(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT");
}
#[test]
fn internode_replay_scope_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT, "RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT");
}
#[test]
fn internode_replay_cache_capacity_defaults_and_env_name() {
assert_eq!(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY");
+7
View File
@@ -24,7 +24,14 @@ description = "Credentials management utilities for RustFS, enabling secure hand
keywords = ["rustfs", "Minio", "credentials", "authentication", "authorization"]
categories = ["web-programming", "development-tools", "data-structures", "security"]
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
+4
View File
@@ -29,6 +29,7 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -49,6 +50,9 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
fips = []
crypto = [
"dep:aes-gcm",
+7 -1
View File
@@ -27,9 +27,15 @@ categories = ["data-structures", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "rustfs-filemeta/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-filemeta/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
path-clean = { workspace = true }
rmp-serde = { workspace = true }
async-trait = { workspace = true }
rustfs-filemeta = { workspace = true }
+94 -6
View File
@@ -12,12 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use path_clean::PathClean;
use serde::{Deserialize, Serialize};
use std::{
collections::{HashMap, HashSet},
hash::{DefaultHasher, Hash, Hasher},
path::Path,
time::{Duration, SystemTime},
};
@@ -334,7 +332,7 @@ impl<'de> Deserialize<'de> for SizeHistogram {
impl SizeHistogram {
pub fn add(&mut self, size: u64) {
let intervals = [
(0, 1024), // LESS_THAN_1024_B
(0, 1024 - 1), // LESS_THAN_1024_B
(1024, 64 * 1024 - 1), // BETWEEN_1024_B_AND_64_KB
(64 * 1024, 256 * 1024 - 1), // BETWEEN_64_KB_AND_256_KB
(256 * 1024, 512 * 1024 - 1), // BETWEEN_256_KB_AND_512_KB
@@ -362,7 +360,7 @@ impl SizeHistogram {
// the sub-ranges in [1 KiB, 512 KiB).
const ONE_MIB: u64 = 1024 * 1024;
let intervals = [
(0, 1024), // LESS_THAN_1024_B
(0, 1024 - 1), // LESS_THAN_1024_B
(1024, 64 * 1024 - 1), // BETWEEN_1024_B_AND_64_KB
(64 * 1024, 256 * 1024 - 1), // BETWEEN_64_KB_AND_256_KB
(256 * 1024, 512 * 1024 - 1), // BETWEEN_256_KB_AND_512_KB
@@ -1110,9 +1108,39 @@ fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String
}
}
/// Hash a path for data usage caching
fn clean_data_usage_path(data: &str) -> String {
let rooted = data.starts_with('/');
let mut parts = Vec::new();
for part in data.split('/') {
match part {
"" | "." => {}
".." => {
if parts.last().is_some_and(|last| *last != "..") {
parts.pop();
} else if !rooted {
parts.push(part);
}
}
_ => parts.push(part),
}
}
let clean = parts.join("/");
match (rooted, clean.is_empty()) {
(true, true) => "/".to_string(),
(true, false) => format!("/{clean}"),
(false, true) => ".".to_string(),
(false, false) => clean,
}
}
/// Hash a slash-separated path for data usage caching.
///
/// Cache identifiers are persisted and exchanged across nodes, so their
/// normalization must not depend on the host operating system.
pub fn hash_path(data: &str) -> DataUsageHash {
DataUsageHash(Path::new(&data).clean().to_string_lossy().to_string())
DataUsageHash(clean_data_usage_path(data))
}
impl DataUsageInfo {
@@ -1497,6 +1525,23 @@ mod tests {
buckets_count: u64,
}
#[test]
fn hash_path_uses_portable_slash_semantics() {
for (input, expected) in [
("", "."),
(".", "."),
("/", "/"),
("//bucket///prefix/", "/bucket/prefix"),
("bucket/./prefix//object", "bucket/prefix/object"),
("bucket/a/../b", "bucket/b"),
("../bucket/..", ".."),
("/../../bucket", "/bucket"),
("bucket\\prefix/object", "bucket\\prefix/object"),
] {
assert_eq!(hash_path(input).key(), expected, "unexpected portable cache key for {input:?}");
}
}
#[test]
fn completeness_marker_is_additive_for_legacy_named_readers() {
let current = DataUsageInfo {
@@ -1601,6 +1646,49 @@ mod tests {
assert_eq!(map["BETWEEN_512_KB_AND_1_MB"], 1);
}
#[test]
fn test_size_histogram_classifies_adjacent_boundaries_once() {
let cases = [
(1023, 0),
(1024, 1),
(64 * 1024 - 1, 1),
(64 * 1024, 2),
(256 * 1024 - 1, 2),
(256 * 1024, 3),
(512 * 1024 - 1, 3),
(512 * 1024, 4),
(1024 * 1024 - 1, 4),
(1024 * 1024, 6),
(10 * 1024 * 1024 - 1, 6),
(10 * 1024 * 1024, 7),
(64 * 1024 * 1024 - 1, 7),
(64 * 1024 * 1024, 8),
(128 * 1024 * 1024 - 1, 8),
(128 * 1024 * 1024, 9),
(512 * 1024 * 1024 - 1, 9),
(512 * 1024 * 1024, 10),
];
for (size, expected_bucket) in cases {
let mut hist = SizeHistogram::default();
hist.add(size);
assert_eq!(hist.0.iter().sum::<u64>(), 1, "size {size} must have exactly one physical bucket");
assert_eq!(hist.0[expected_bucket], 1, "size {size} must select the expected bucket");
}
}
#[test]
fn test_size_histogram_1024_bytes_contributes_to_compat_rollup() {
let mut hist = SizeHistogram::default();
hist.add(1024);
let map = hist.to_map();
assert_eq!(map["LESS_THAN_1024_B"], 0);
assert_eq!(map["BETWEEN_1024_B_AND_64_KB"], 1);
assert_eq!(map["BETWEEN_1024B_AND_1_MB"], 1);
}
#[test]
fn test_size_histogram_compat_rollup_saturates_on_corrupt_counts() {
let mut hist = SizeHistogram::default();
+48
View File
@@ -25,10 +25,58 @@ workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-protos/hotpath",
"rustfs-rio/hotpath",
"rustfs-signer/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
ftps = []
sftp = []
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-credentials.workspace = true
rustfs-ecstore.workspace = true
@@ -117,9 +117,14 @@ mod tests {
.key("assets/explicit-copy.js")
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Copy)
.customize()
.mutate_request(|request| {
request.headers_mut().insert("content-type", "application/octet-stream");
request.headers_mut().insert("x-amz-meta-request-only", "ignored");
})
.send()
.await
.expect("explicit COPY directive failed");
.expect("explicit COPY directive with request metadata failed");
let explicit_copy_head = client
.head_object()
.bucket(bucket)
@@ -128,6 +133,18 @@ mod tests {
.await
.expect("HEAD failed after explicit COPY");
assert_eq!(explicit_copy_head.cache_control(), Some("max-age=60"));
assert_eq!(explicit_copy_head.content_type(), Some("text/javascript; charset=utf-8"));
assert_eq!(
explicit_copy_head.metadata().and_then(|metadata| metadata.get("mtime")),
Some(&"1777992333".to_string())
);
assert_eq!(
explicit_copy_head
.metadata()
.and_then(|metadata| metadata.get("request-only")),
None,
"COPY must ignore request metadata"
);
assert_eq!(
explicit_copy_head.website_redirect_location(),
None,
@@ -571,20 +588,6 @@ mod tests {
Some("InvalidArgument")
);
let ignored_replacement = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.content_type("application/ignored")
.send()
.await
.expect_err("Replacement fields without REPLACE should be rejected");
assert_eq!(
ignored_replacement.as_service_error().and_then(|error| error.code()),
Some("InvalidRequest")
);
let unchanged = client
.get_object()
.bucket(bucket)
@@ -56,6 +56,21 @@ mod tests {
);
}
async fn assert_current_list_hides_delete_marker(client: &Client, bucket: &str, key: &str) {
let listed = client
.list_objects_v2()
.bucket(bucket)
.prefix(key)
.send()
.await
.expect("list current objects after delete marker");
assert!(
listed.contents().iter().all(|object| object.key() != Some(key)),
"ListObjectsV2 must hide an object whose latest version is a delete marker"
);
}
#[tokio::test]
#[serial]
async fn test_versioning_only_delete_marker_has_minio_compatible_visibility_for_migration_proof() {
@@ -94,6 +109,7 @@ mod tests {
assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
assert_eq!(markers[0].is_latest(), Some(true));
assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
assert_current_list_hides_delete_marker(&client, bucket, key).await;
}
#[tokio::test]
@@ -118,6 +134,17 @@ mod tests {
.await
.expect("put historical version");
let data_version_id = put.version_id().expect("put should return data version id");
let listed_before_delete = client
.list_objects_v2()
.bucket(bucket)
.prefix(key)
.send()
.await
.expect("list current object before creating delete marker");
assert!(
listed_before_delete.contents().iter().any(|object| object.key() == Some(key)),
"ListObjectsV2 must include the current object before it is deleted"
);
let delete_marker = client
.delete_object()
@@ -145,6 +172,7 @@ mod tests {
assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
assert_eq!(markers[0].is_latest(), Some(true));
assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
assert_current_list_hides_delete_marker(&client, bucket, key).await;
let historical = client
.get_object()
@@ -2136,11 +2136,20 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
assert_eq!(terminal["bucket"].as_str(), Some(bucket.as_str()));
assert_eq!(terminal["prefix"].as_str(), Some(prefix));
assert_eq!(terminal["dry_run"].as_bool(), Some(false));
assert_eq!(
terminal["status"].as_str(),
Some("partial"),
let terminal_status = terminal["status"].as_str();
assert!(
matches!(terminal_status, Some("partial" | "unknown")),
"small transition queue should surface terminal backpressure: {terminal}"
);
if terminal_status == Some("unknown") {
let failure_reason = terminal["failure_reason"]
.as_str()
.ok_or_else(|| format!("unknown terminal status omitted failure_reason: {terminal}"))?;
assert!(
failure_reason.contains("worker result was not persisted before the transition queue drained"),
"unknown terminal status should identify lost worker-result persistence: {terminal}"
);
}
let skipped_queue_full = terminal["report"]["skipped_queue_full"]
.as_u64()
.ok_or_else(|| format!("terminal status omitted report.skipped_queue_full: {terminal}"))?;
@@ -13,8 +13,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Cross-process replay / tamper acceptance for the internode NodeService v2 RPC
//! signature (<https://github.com/rustfs/backlog/issues/1327>).
//! Cross-process replay / tamper acceptance for internode NodeService RPC
//! signatures (<https://github.com/rustfs/backlog/issues/1327>,
//! <https://github.com/rustfs/backlog/issues/1542>).
//!
//! # Why this exists on top of the in-process tests
//!
@@ -78,6 +79,8 @@
//! | mixed version: legacy-only still served, not blocked | [`legacy_only_signature_is_accepted_in_default_posture`] |
//! | strict flip closes the signature downgrade | [`signature_strict_rejects_legacy_only_downgrade`] |
//! | strict flip closes the body-digest downgrade, incl. v1 | [`body_digest_strict_rejects_digestless_mutation`] |
//! | replay scope binds every RPC and rejects restart replay | [`replay_scope_rejects_replay_path_transplant_and_stale_epoch_e2e`] |
//! | strict replay scope allows only Ping bootstrap before v3 | [`replay_scope_strict_requires_v3_after_ping_bootstrap_e2e`] |
//!
//! Two acceptance items are deliberately left to the in-process tests. A stale
//! timestamp cannot be forged from outside — it is inside the HMAC — so
@@ -88,18 +91,20 @@
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::storage_api::internode_rpc_signature::{
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers, node_service_time_out_client_no_auth,
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
node_service_time_out_client_no_auth, verify_tonic_boot_epoch_response,
};
use http::{HeaderMap, Method};
use rustfs_config::{
ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, ENV_INTERNODE_RPC_SIGNATURE_STRICT,
ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT,
ENV_INTERNODE_RPC_SIGNATURE_STRICT,
};
use rustfs_protos::canonical_make_volume_request_body;
use rustfs_protos::proto_gen::node_service::{MakeVolumeRequest, MakeVolumeResponse};
use rustfs_protos::proto_gen::node_service::{MakeVolumeRequest, MakeVolumeResponse, PingRequest, PingResponse};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::error::Error;
use tonic::{Code, Request, Status};
use tonic::{Code, Request, Response, Status};
use uuid::Uuid;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
@@ -115,12 +120,14 @@ const TEST_RPC_SECRET: &str = "rustfs-internode-signature-e2e-secret";
/// clears authentication stops harmlessly at `find_disk`.
const ABSENT_DISK: &str = "/nonexistent/rustfs-signature-e2e-disk";
/// Wire names of the two v2 headers these tests edit. They are `pub(crate)` in
/// ecstore, so they are repeated here rather than imported — [`overwrite_header`]
/// asserts the header it replaces was actually present, which turns a rename
/// into a loud failure instead of silently reducing an attack to a no-op.
/// Wire names of the v2 and replay-scope headers these black-box tests edit. They are
/// `pub(crate)` in ecstore, so they are repeated here rather than imported.
/// [`overwrite_header`] asserts the header it replaces was actually present, which turns a
/// rename into a loud failure instead of silently reducing an attack to a no-op.
const CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
const NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
const BOOT_EPOCH_CHALLENGE_HEADER: &str = "x-rustfs-rpc-boot-epoch-challenge";
/// gRPC service name carried in the signed scope, i.e. `TONIC_RPC_PREFIX`
/// without its leading `/`.
@@ -155,19 +162,28 @@ fn align_rpc_secret_with_server() {
///
/// Uses the no-cleanup spawn so a `pkill` pattern cannot reap servers belonging
/// to other tests running in the same binary.
async fn start_server(extra_env: &[(&str, &str)]) -> Result<RustFSTestEnvironment, Box<dyn Error + Send + Sync>> {
let mut env = RustFSTestEnvironment::new().await?;
fn server_env(extra_env: &[(&'static str, &'static str)]) -> Vec<(&'static str, &'static str)> {
let mut child_env = vec![
("RUSTFS_RPC_SECRET", TEST_RPC_SECRET),
(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "false"),
(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "false"),
(ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT, "false"),
(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "1048576"),
];
child_env.extend_from_slice(extra_env);
env.start_rustfs_server_without_cleanup_with_env(&child_env).await?;
child_env
}
async fn start_server_with_env(child_env: &[(&str, &str)]) -> Result<RustFSTestEnvironment, Box<dyn Error + Send + Sync>> {
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_without_cleanup_with_env(child_env).await?;
Ok(env)
}
async fn start_server(extra_env: &[(&'static str, &'static str)]) -> Result<RustFSTestEnvironment, Box<dyn Error + Send + Sync>> {
start_server_with_env(&server_env(extra_env)).await
}
/// Stop the child and drop the cached gRPC channel for its address.
///
/// `node_service_time_out_client_no_auth` memoises channels in a process-global
@@ -238,12 +254,83 @@ fn overwrite_header(headers: &mut HeaderMap, name: &'static str, value: &str) {
/// and nothing else — no interceptor adds or rewrites auth metadata, so the
/// bytes on the wire are the ones the test chose.
async fn call_make_volume(url: &str, request: MakeVolumeRequest, headers: HeaderMap) -> Result<MakeVolumeResponse, Status> {
call_make_volume_response(url, request, headers)
.await
.map(Response::into_inner)
}
async fn call_make_volume_response(
url: &str,
request: MakeVolumeRequest,
headers: HeaderMap,
) -> Result<Response<MakeVolumeResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(request);
rpc_request.metadata_mut().as_mut().extend(headers);
client.make_volume(rpc_request).await.map(|response| response.into_inner())
client.make_volume(rpc_request).await
}
async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(PingRequest {
version: 1,
body: bytes::Bytes::new(),
});
rpc_request.metadata_mut().as_mut().extend(headers);
client.ping(rpc_request).await
}
fn attach_boot_epoch_challenge(headers: &mut HeaderMap) -> Uuid {
let challenge = Uuid::new_v4();
headers.insert(
BOOT_EPOCH_CHALLENGE_HEADER,
challenge.to_string().parse().expect("UUID must be a valid header value"),
);
challenge
}
fn mint_replay_scope_headers(audience: &str, path: &str, content_sha256: &str, boot_epoch: Uuid) -> HeaderMap {
let mut headers = mint_v2_headers(audience, "MakeVolume", Some(content_sha256));
let timestamp = headers
.get(TIMESTAMP_HEADER)
.and_then(|value| value.to_str().ok())
.expect("v2 headers must carry a timestamp")
.to_string();
headers.extend(
gen_tonic_replay_scope_headers(audience, path, &timestamp, content_sha256, boot_epoch)
.expect("replay-scope headers must mint with the aligned RPC secret"),
);
headers
}
async fn learn_boot_epoch_from_make_volume(url: &str, audience: &str) -> Uuid {
let request = make_volume_request("signature-e2e-epoch-bootstrap");
let mut headers = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&request)));
let challenge = attach_boot_epoch_challenge(&mut headers);
let response = call_make_volume_response(url, request, headers)
.await
.expect("v2 request with epoch challenge must clear default authentication");
let boot_epoch = verify_tonic_boot_epoch_response(audience, challenge, response.metadata().as_ref())
.expect("server must HMAC-authenticate the advertised boot epoch");
assert_authenticated(
Ok(response.into_inner()),
"a v2 epoch-challenge request in the default replay-scope posture",
);
boot_epoch
}
async fn learn_boot_epoch_from_ping(url: &str, audience: &str) -> Uuid {
let mut headers = mint_v2_headers(audience, "Ping", None);
let challenge = attach_boot_epoch_challenge(&mut headers);
let response = call_ping_response(url, headers)
.await
.expect("v2 Ping with an epoch challenge must bootstrap strict replay scope");
verify_tonic_boot_epoch_response(audience, challenge, response.metadata().as_ref())
.expect("strict replay-scope Ping must return a valid boot epoch proof")
}
/// Assert a call cleared authentication.
@@ -332,6 +419,122 @@ async fn internode_rpc_signature_default_posture_e2e() -> TestResult {
Ok(())
}
/// A replay-scoped signature is usable exactly once against the exact gRPC path and the server
/// process epoch that minted it. This crosses the child-process boundary twice: the HMAC-protected
/// epoch is learned from a real response, then the same server is restarted in place to prove its
/// replacement epoch rejects the captured request even though the nonce cache is necessarily new.
#[tokio::test]
#[serial]
async fn replay_scope_rejects_replay_path_transplant_and_stale_epoch_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
let child_env = server_env(&[]);
let mut env = start_server_with_env(&child_env).await?;
let url = env.url.clone();
let audience = audience_of(&env);
let boot_epoch = learn_boot_epoch_from_make_volume(&url, &audience).await;
let request = make_volume_request("replay-scope-e2e-once");
let captured = mint_replay_scope_headers(
&audience,
&format!("{TONIC_RPC_PREFIX}/MakeVolume"),
&canonical_digest(&request),
boot_epoch,
);
assert_authenticated(
call_make_volume(&url, request.clone(), captured.clone()).await,
"the first replay-scoped mutation delivery",
);
assert_rejected(
call_make_volume(&url, request.clone(), captured).await,
Code::Unauthenticated,
None,
"the same replay-scoped mutation delivered twice",
);
let transplanted =
mint_replay_scope_headers(&audience, &format!("{TONIC_RPC_PREFIX}/Ping"), &canonical_digest(&request), boot_epoch);
assert_rejected(
call_make_volume(&url, request.clone(), transplanted).await,
Code::Unauthenticated,
None,
"a replay-scoped Ping signature transplanted onto MakeVolume",
);
let stale_epoch = mint_replay_scope_headers(
&audience,
&format!("{TONIC_RPC_PREFIX}/MakeVolume"),
&canonical_digest(&request),
boot_epoch,
);
env.restart_server_preserving_data(Vec::new(), &child_env).await?;
rustfs_protos::evict_failed_connection(&url).await;
assert_rejected(
call_make_volume(&url, request.clone(), stale_epoch).await,
Code::Unauthenticated,
None,
"a replay-scoped signature captured before the receiving process restart",
);
let restarted_epoch = learn_boot_epoch_from_make_volume(&url, &audience).await;
assert_ne!(boot_epoch, restarted_epoch, "a restarted child process must advertise a new boot epoch");
let fresh_epoch = mint_replay_scope_headers(
&audience,
&format!("{TONIC_RPC_PREFIX}/MakeVolume"),
&canonical_digest(&request),
restarted_epoch,
);
assert_authenticated(
call_make_volume(&url, request, fresh_epoch).await,
"a replay-scoped mutation signed with the replacement process epoch",
);
stop_server(env, &url).await;
Ok(())
}
/// Strict replay scope leaves one authenticated v2 bootstrap: `Ping` carrying a fresh challenge.
/// A mutating v2 request cannot use that lane; once the epoch proof is returned, the first v3
/// mutation succeeds. This protects a server restart without reopening a general downgrade path.
#[tokio::test]
#[serial]
async fn replay_scope_strict_requires_v3_after_ping_bootstrap_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
let env = start_server(&[(ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT, "true")]).await?;
let url = env.url.clone();
let audience = audience_of(&env);
let v2_request = make_volume_request("replay-scope-e2e-strict-v2");
assert_rejected(
call_make_volume(
&url,
v2_request.clone(),
mint_v2_headers(&audience, "MakeVolume", Some(&canonical_digest(&v2_request))),
)
.await,
Code::Unauthenticated,
None,
"a v2 mutation after replay-scope strictness is enabled",
);
let boot_epoch = learn_boot_epoch_from_ping(&url, &audience).await;
let request = make_volume_request("replay-scope-e2e-strict-v3");
let replay_scoped = mint_replay_scope_headers(
&audience,
&format!("{TONIC_RPC_PREFIX}/MakeVolume"),
&canonical_digest(&request),
boot_epoch,
);
assert_authenticated(
call_make_volume(&url, request, replay_scoped).await,
"a replay-scoped mutation after Ping bootstrap under strict replay scope",
);
stop_server(env, &url).await;
Ok(())
}
/// Baseline: correctly signed mutations are accepted, both with and without a
/// body digest.
///
@@ -21,18 +21,22 @@
//! function, never as an S3 event sink.
//!
//! Coverage:
//! * PUT / multipart-complete / DELETE each deliver one event with the correct
//! * PUT / multipart-complete / DeleteObject / DeleteObjects each deliver one event with the correct
//! eventName, bucket, key, versionId and eTag.
//! * prefix/suffix filters drop non-matching keys (rule-engine gate).
//! * an event queued while the target endpoint is unreachable is redelivered
//! from the on-disk store once the endpoint recovers (store-and-forward).
//! * responseElements and the S3 response use the canonical request ID while
//! requestParameters preserve a conflicting client-supplied value.
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::operation::RequestId;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Event, FilterRule, FilterRuleName,
NotificationConfiguration, NotificationConfigurationFilter, QueueConfiguration, S3KeyFilter, VersioningConfiguration,
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Delete, Event, FilterRule, FilterRuleName,
NotificationConfiguration, NotificationConfigurationFilter, ObjectIdentifier, QueueConfiguration, S3KeyFilter,
VersioningConfiguration,
};
use http::header::{CONTENT_TYPE, HOST};
use local_ip_address::local_ip;
@@ -40,10 +44,12 @@ use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER};
use s3s::Body;
use serde_json::Value;
use serial_test::serial;
use std::error::Error;
use std::io::Cursor;
use std::path::Path;
use std::sync::{
Arc, Once,
@@ -63,6 +69,8 @@ type BoxError = Box<dyn Error + Send + Sync>;
/// for target lookup (see `process_queue_configurations`), so the region is
/// nominal, but it must be present for `ARN::parse` to succeed.
const NOTIFY_REGION: &str = "us-east-1";
const CLIENT_REQUEST_ID: &str = "client-supplied-request-id";
const CLIENT_AMZ_REQUEST_ID: &str = "client-supplied-amz-request-id";
/// Webhook targets are registered as `TargetID { id: <name>, name: "webhook" }`,
/// so the ARN a notification rule references is
@@ -579,6 +587,36 @@ fn trimmed_etag(value: Option<&str>) -> Option<String> {
value.map(|e| e.trim_matches('"').to_string())
}
fn assert_conflicting_request_id_correlation(record: &Value, server_request_id: &str) {
assert_eq!(
record["requestParameters"][REQUEST_ID_HEADER].as_str(),
Some(CLIENT_REQUEST_ID),
"notification request parameters should retain the actual client header: {record}"
);
assert_eq!(
record["requestParameters"][AMZ_REQUEST_ID].as_str(),
Some(CLIENT_AMZ_REQUEST_ID),
"notification request parameters should retain the actual client header: {record}"
);
assert_eq!(
record["responseElements"][AMZ_REQUEST_ID].as_str(),
Some(server_request_id),
"notification response elements should use the canonical request ID: {record}"
);
}
fn assert_generated_request_id_correlation(record: &Value, request_id: &str) {
assert!(
record["requestParameters"][AMZ_REQUEST_ID].is_null(),
"notification request parameters must not invent a client request header: {record}"
);
assert_eq!(
record["responseElements"][AMZ_REQUEST_ID].as_str(),
Some(request_id),
"notification response elements should match the S3 response request ID: {record}"
);
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
@@ -671,8 +709,17 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
.bucket(bucket)
.key(put_key)
.body(ByteStream::from_static(b"peri-1 put body"))
.customize()
.mutate_request(|request| {
request.headers_mut().insert(REQUEST_ID_HEADER, CLIENT_REQUEST_ID);
request.headers_mut().insert(AMZ_REQUEST_ID, CLIENT_AMZ_REQUEST_ID);
})
.send()
.await?;
let put_request_id = put.request_id().ok_or("PUT response missing request ID")?.to_owned();
assert!(uuid::Uuid::parse_str(&put_request_id).is_ok());
assert_ne!(put_request_id, CLIENT_REQUEST_ID);
assert_ne!(put_request_id, CLIENT_AMZ_REQUEST_ID);
let put_version = put
.version_id()
.ok_or("PUT response missing versionId (versioning not enabled?)")?;
@@ -692,6 +739,7 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
"record eventName: {record}"
);
assert_eq!(record["s3"]["bucket"]["name"].as_str(), Some(bucket), "bucket in event: {record}");
assert_conflicting_request_id_correlation(record, &put_request_id);
assert_eq!(object["versionId"].as_str(), Some(put_version), "versionId in event: {object}");
assert_eq!(
trimmed_etag(object["eTag"].as_str()),
@@ -744,6 +792,35 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
"multipart eTag in event: {mp_record}"
);
// --- Snowball extract: direct notification path keeps response correlation
let snowball_key = "uploads/snowball.dat";
let snowball_body = b"snowball notification body";
let mut archive_builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
let mut archive_header = tokio_tar::Header::new_gnu();
archive_header.set_size(u64::try_from(snowball_body.len()).expect("snowball fixture length should fit in u64"));
archive_header.set_mode(0o644);
archive_header.set_cksum();
archive_builder
.append_data(&mut archive_header, snowball_key, Cursor::new(snowball_body))
.await?;
let archive = archive_builder.into_inner().await?.into_inner();
let snowball = client
.put_object()
.bucket(bucket)
.key("snowball-fixture.tar")
.body(ByteStream::from(archive))
.customize()
.mutate_request(|request| {
request.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await?;
let snowball_request_id = snowball.request_id().ok_or("Snowball response missing request ID")?;
let snowball_event = wait_for_event(&mut rx, snowball_key, "s3:ObjectCreated:", Duration::from_secs(20)).await?;
let snowball_record = &snowball_event["Records"][0];
assert_generated_request_id_correlation(snowball_record, snowball_request_id);
// --- Filter: non-matching prefix and suffix must never be delivered ------
let wrong_prefix = "logs/report.dat"; // right suffix, wrong prefix
let wrong_suffix = "uploads/report.txt"; // right prefix, wrong suffix
@@ -772,7 +849,32 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
"wrong-suffix key {wrong_suffix} bypassed the filter; saw {seen:?}"
);
// --- DELETE on a versioned bucket: ObjectRemoved:* with delete-marker version
// --- DeleteObjects: direct notification path keeps response correlation --
let delete_many_key = "uploads/delete-many.dat";
client
.put_object()
.bucket(bucket)
.key(delete_many_key)
.body(ByteStream::from_static(b"delete objects notification body"))
.send()
.await?;
wait_for_event(&mut rx, delete_many_key, "s3:ObjectCreated:", Duration::from_secs(20)).await?;
let delete_many = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(delete_many_key).build()?)
.build()?,
)
.send()
.await?;
let delete_many_request_id = delete_many.request_id().ok_or("DeleteObjects response missing request ID")?;
let delete_many_event = wait_for_event(&mut rx, delete_many_key, "s3:ObjectRemoved:", Duration::from_secs(20)).await?;
assert_generated_request_id_correlation(&delete_many_event["Records"][0], delete_many_request_id);
// --- DeleteObject on a versioned bucket: delete-marker version ----------
let delete = client.delete_object().bucket(bucket).key(put_key).send().await?;
let removed = wait_for_event(&mut rx, put_key, "s3:ObjectRemoved:", Duration::from_secs(20)).await?;
let removed_record = &removed["Records"][0];
@@ -26,6 +26,7 @@
use super::common::*;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
use aws_sdk_s3::types::{
CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
@@ -2120,6 +2121,127 @@ async fn test_multipart_default_retention_fixed_at_create() {
// Versioning Auto-Enable Tests
// ============================================================================
#[tokio::test]
#[serial]
async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
init_logging();
info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
let mut env = ObjectLockTestEnvironment::new()
.await
.expect("failed to create Object Lock test environment");
env.start_rustfs().await.expect("failed to start RustFS");
let bucket = "test-object-lock-delete-cleanup";
let key = "unretained-object";
env.create_object_lock_bucket(bucket)
.await
.expect("failed to create Object Lock bucket");
let client = env.s3_client();
let put_response = client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"unretained data"))
.send()
.await
.expect("failed to upload unretained object");
let object_version_id = put_response
.version_id()
.expect("Object Lock buckets must create versioned objects")
.to_string();
let delete_response = client
.delete_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("failed to create delete marker");
assert_eq!(delete_response.delete_marker(), Some(true));
let delete_marker_version_id = delete_response
.version_id()
.expect("Deleting without a version ID must create a delete marker")
.to_string();
let get_error = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect_err("GET must not return an object hidden by a delete marker");
assert_eq!(get_error.raw_response().map(|response| response.status().as_u16()), Some(404));
assert_eq!(get_error.as_service_error().and_then(|error| error.code()), Some("NoSuchKey"));
let listed_objects = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("failed to list current objects");
assert!(
listed_objects.contents().iter().all(|object| object.key() != Some(key)),
"ListObjectsV2 must hide objects whose latest version is a delete marker"
);
let listed_versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("failed to list object versions");
assert!(
listed_versions
.versions()
.iter()
.any(|version| version.key() == Some(key) && version.version_id() == Some(object_version_id.as_str())),
"The data version must remain until it is explicitly deleted"
);
assert!(
listed_versions
.delete_markers()
.iter()
.any(|marker| marker.key() == Some(key) && marker.version_id() == Some(delete_marker_version_id.as_str())),
"ListObjectVersions must expose the delete marker"
);
client
.delete_object()
.bucket(bucket)
.key(key)
.version_id(object_version_id)
.send()
.await
.expect("failed to delete the data version");
client
.delete_object()
.bucket(bucket)
.key(key)
.version_id(delete_marker_version_id)
.send()
.await
.expect("failed to delete the delete marker");
let remaining_versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("failed to list versions after cleanup");
assert!(remaining_versions.versions().is_empty());
assert!(remaining_versions.delete_markers().is_empty());
client
.delete_bucket()
.bucket(bucket)
.send()
.await
.expect("Deleting every version must remove xl.meta so the bucket can be deleted normally");
}
#[tokio::test]
#[serial]
async fn test_versioning_auto_enabled_with_object_lock() {
@@ -24,7 +24,7 @@ use rustfs_protos::proto_gen::node_service::{BatchGenerallyLockRequest, Generall
use tonic::Request;
use tracing::{info, warn};
use crate::storage_api::grpc_lock::{TonicInterceptor, node_service_time_out_client_no_auth};
use crate::storage_api::grpc_lock::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
/// gRPC lock client without authentication for testing
/// Similar to RemoteClient but uses no_auth client
@@ -42,7 +42,7 @@ impl GrpcLockClient {
&self,
) -> Result<
rustfs_protos::proto_gen::node_service::node_service_client::NodeServiceClient<
tonic::service::interceptor::InterceptedService<tonic::transport::Channel, TonicInterceptor>,
tonic::service::interceptor::InterceptedService<AuthenticatedChannel, TonicInterceptor>,
>,
> {
node_service_time_out_client_no_auth(&self.addr)
+8 -4
View File
@@ -16,9 +16,12 @@
pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::disk::{VolumeInfo, WalkDirOptions};
pub(crate) use rustfs_ecstore::api::rpc::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers};
pub(crate) use rustfs_ecstore::api::rpc::{TonicInterceptor, node_service_time_out_client_no_auth};
pub(crate) use rustfs_ecstore::api::rpc::{
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
verify_tonic_boot_epoch_response,
};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{gen_tonic_signature_interceptor, node_service_time_out_client};
@@ -30,7 +33,7 @@ pub(crate) mod node_interact {
}
pub(crate) mod grpc_lock {
pub(crate) use super::{TonicInterceptor, node_service_time_out_client_no_auth};
pub(crate) use super::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
}
/// Signing/transport surface used by the cross-process internode RPC signature
@@ -40,7 +43,8 @@ pub(crate) mod grpc_lock {
#[cfg(test)]
pub(crate) mod internode_rpc_signature {
pub(crate) use super::{
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers, node_service_time_out_client_no_auth,
TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
node_service_time_out_client_no_auth, verify_tonic_boot_epoch_response,
};
}
+94 -5
View File
@@ -33,14 +33,98 @@ workspace = true
[features]
default = []
rio-v2 = ["dep:rustfs-rio-v2"]
hotpath = ["dep:hotpath", "hotpath/hotpath", "rustfs-filemeta/hotpath", "rustfs-rio/hotpath"]
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/async-channel",
"hotpath/parking_lot",
"hotpath/reqwest-0-13",
"rustfs-checksums/hotpath",
"rustfs-common/hotpath",
"rustfs-concurrency/hotpath",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-io-metrics/hotpath",
"rustfs-lifecycle/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-object-capacity/hotpath",
"rustfs-policy/hotpath",
"rustfs-protos/hotpath",
"rustfs-replication/hotpath",
"rustfs-rio/hotpath",
"rustfs-rio-v2?/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-signer/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-tls-runtime/hotpath",
"rustfs-utils/hotpath",
"rustfs-crypto/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-checksums/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-concurrency/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
"rustfs-lifecycle/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-object-capacity/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-replication/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-rio-v2?/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-tls-runtime/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-crypto/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-checksums/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-concurrency/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
"rustfs-lifecycle/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-object-capacity/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-replication/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-rio-v2?/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-tls-runtime/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-crypto/hotpath-cpu",
]
# Exposes shared lifecycle/tier test utilities (MockWarmBackend, fault
# injection, xl.meta transition assertions) via `api::tier::test_util`.
# Enable only from `[dev-dependencies]` (rustfs/backlog#1148 ilm-6).
test-util = []
[dependencies]
hotpath = { workspace = true, optional = true }
hotpath.workspace = true
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
rustfs-rio.workspace = true
@@ -57,7 +141,6 @@ rustfs-policy.workspace = true
rustfs-protos.workspace = true
rustfs-replication.workspace = true
rustfs-lifecycle.workspace = true
rustfs-kms.workspace = true
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-object-capacity.workspace = true
@@ -105,6 +188,7 @@ tempfile.workspace = true
hyper = { workspace = true, features = ["http2", "http1", "server"] }
hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
hostname.workspace = true
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls-pki-types.workspace = true
tokio = { workspace = true, features = ["io-util", "sync", "signal", "fs", "rt-multi-thread"] }
@@ -124,8 +208,6 @@ libc.workspace = true
rustix = { workspace = true, features = ["process", "fs"] }
rustfs-madmin.workspace = true
reqwest = { workspace = true }
aes-gcm = { workspace = true, features = ["rand_core"] }
chacha20poly1305.workspace = true
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
smallvec = { workspace = true, features = ["serde"] }
@@ -153,11 +235,18 @@ metrics = { workspace = true }
[target.'cfg(target_os = "linux")'.dependencies]
rustfs-uring = "0.2.1"
[target.'cfg(windows)'.dependencies]
winapi-util.workspace = true
windows-sys = { workspace = true, features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }
[dev-dependencies]
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
criterion = { workspace = true, features = ["html_reports"] }
temp-env = { workspace = true, features = ["async_closure"] }
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
# Only for `pin_callsite_interest_for_test`, which registers a `NoSubscriber`
# dispatcher to keep tracing's process-global callsite-interest cache honest.
tracing-core = { workspace = true }
serial_test = { workspace = true }
opentelemetry_sdk = { workspace = true, features = ["rt-tokio"] }
proptest = "1"
+35 -21
View File
@@ -67,6 +67,14 @@ pub mod bucket {
};
}
pub mod transition_transaction {
pub use crate::bucket::lifecycle::transition_transaction::{
TransitionOperatorDeleteResult, TransitionOperatorError, TransitionOperatorProbe, TransitionOperatorStatus,
delete_transition_candidate_for_operator, finalize_missing_transition_transaction_for_operator,
inspect_transition_transaction_for_operator,
};
}
pub mod evaluator {
pub use crate::bucket::lifecycle::evaluator::Evaluator;
}
@@ -122,13 +130,13 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BucketMetadataSys, acquire_bucket_targets_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
BucketMetadataSys, acquire_bucket_metadata_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
update, update_bucket_targets_under_transaction_lock, update_config_with,
update, update_bucket_targets_under_transaction_lock, update_config_with, update_under_transaction_lock,
};
}
@@ -176,8 +184,8 @@ pub mod bucket {
get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
should_use_existing_delete_replication_source, unsupported_replication_config_field,
validate_replication_config_target_arns, version_purge_status_to_filemeta,
};
}
@@ -259,12 +267,13 @@ pub mod config {
pub mod com {
pub use crate::config::com::{
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
ServerConfigCorruptError, ServerConfigSnapshot, delete_config, is_server_config_corrupt_error, lookup_configs,
read_config, read_config_no_lock, read_config_with_metadata, read_config_without_migrate,
read_config_without_migrate_no_lock, read_existing_server_config_no_lock, read_server_config_snapshot, save_config,
save_config_no_lock, save_config_with_opts, save_server_config, save_server_config_no_lock,
save_server_config_snapshot, server_config_path, try_migrate_server_config, with_config_object_read_lock,
with_config_object_write_lock, with_server_config_read_lock, with_server_config_write_lock,
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
with_server_config_read_lock, with_server_config_write_lock,
};
}
@@ -377,15 +386,18 @@ pub mod metrics {
pub mod notification {
pub use crate::services::notification_sys::{
NotificationPeerErr, NotificationSys, get_global_notification_sys, new_global_notification_sys,
start_remote_version_state_fleet_probe,
};
}
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
unregister_object_mutation_hook,
};
pub use crate::store::PreparedGetObjectReader;
}
@@ -407,13 +419,15 @@ pub mod rio {
pub mod rpc {
pub use crate::cluster::rpc::{
LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerPeerActivity,
TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, verify_rpc_signature,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature,
AuthenticatedChannel, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient,
PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing,
ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
verify_tonic_rpc_signature_with_bootstrap,
};
}
@@ -23,6 +23,7 @@ use uuid::Uuid;
use crate::bucket::lifecycle::config_boundary;
use crate::bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
use crate::bucket::lifecycle::tier_sweeper::{
delete_confirmed_transition_candidate_exact_with_lease_idempotent,
delete_confirmed_transition_candidate_exact_with_manager_and_identity,
delete_object_from_remote_tier_idempotent_with_manager_and_identity,
};
@@ -616,6 +617,199 @@ pub enum TransitionTransactionRecoveryOutcome {
Retained,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum TransitionOperatorProbe {
Missing,
UnversionedPresent,
VersionedPresent(String),
Ambiguous,
Unsupported,
}
impl From<TransitionCandidateProbe> for TransitionOperatorProbe {
fn from(value: TransitionCandidateProbe) -> Self {
match value {
TransitionCandidateProbe::Missing => Self::Missing,
TransitionCandidateProbe::UnversionedPresent => Self::UnversionedPresent,
TransitionCandidateProbe::VersionedPresent(version_id) => Self::VersionedPresent(version_id),
TransitionCandidateProbe::Ambiguous => Self::Ambiguous,
TransitionCandidateProbe::Unsupported => Self::Unsupported,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct TransitionOperatorStatus {
pub transaction_id: Uuid,
pub state: TransitionTransactionState,
pub tier_name: String,
pub remote_object: String,
pub not_after_unix_nanos: i64,
pub probe: TransitionOperatorProbe,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct TransitionOperatorDeleteResult {
pub status: TransitionOperatorStatus,
pub journal_observed_after_delete: bool,
}
#[derive(Debug, thiserror::Error)]
pub enum TransitionOperatorError {
#[error("transition transaction was not found")]
NotFound,
#[error("transition transaction is still inside its active ownership window")]
NotExpired,
#[error("transition transaction state is not eligible for operator reconciliation: {0:?}")]
InvalidState(TransitionTransactionState),
#[error("an exact non-empty remote version is required")]
RemoteVersionRequired,
#[error("remote candidate is not proven missing: {0:?}")]
CandidateNotMissing(TransitionOperatorProbe),
#[error("remote candidate version does not match requested exact version: expected {expected}, observed {actual:?}")]
CandidateVersionMismatch {
expected: String,
actual: TransitionOperatorProbe,
},
#[error("transition transaction store failed: {0}")]
Store(#[source] Error),
#[error("remote tier reconciliation failed: {0}")]
Remote(#[source] std::io::Error),
}
type TransitionOperatorResult<T> = std::result::Result<T, TransitionOperatorError>;
fn validate_operator_reconcile_transaction(
transaction: &TransitionTransaction,
now_unix_nanos: i128,
) -> TransitionOperatorResult<()> {
transaction
.validate()
.map_err(|err| TransitionOperatorError::Store(Error::other(err)))?;
if transaction.state != TransitionTransactionState::UploadOutcomeUnknown {
return Err(TransitionOperatorError::InvalidState(transaction.state));
}
if now_unix_nanos < i128::from(transaction.not_after_unix_nanos) {
return Err(TransitionOperatorError::NotExpired);
}
Ok(())
}
async fn load_operator_reconcile_transaction(
api: Arc<ECStore>,
transaction_id: Uuid,
) -> TransitionOperatorResult<TransitionTransaction> {
match load_transition_transaction_record(api, transaction_id).await {
Ok(transaction) => Ok(transaction),
Err(Error::ConfigNotFound) => Err(TransitionOperatorError::NotFound),
Err(err) => Err(TransitionOperatorError::Store(err)),
}
}
async fn operator_probe_transition_candidate(
api: Arc<ECStore>,
transaction: &TransitionTransaction,
) -> TransitionOperatorResult<TransitionOperatorProbe> {
let lease = TierConfigMgr::acquire_operation_lease_for_backend_identity(
&api.tier_config_mgr(),
&transaction.tier_name,
transaction.backend_fingerprint,
)
.await
.map_err(|err| TransitionOperatorError::Remote(std::io::Error::other(err)))?;
lease
.probe_transition_candidate_for(&transaction.remote_object, transaction.transaction_id)
.await
.map(TransitionOperatorProbe::from)
.map_err(TransitionOperatorError::Remote)
}
pub async fn inspect_transition_transaction_for_operator(
api: Arc<ECStore>,
transaction_id: Uuid,
) -> TransitionOperatorResult<TransitionOperatorStatus> {
let transaction = load_operator_reconcile_transaction(api.clone(), transaction_id).await?;
validate_operator_reconcile_transaction(&transaction, time::OffsetDateTime::now_utc().unix_timestamp_nanos())?;
let probe = operator_probe_transition_candidate(api, &transaction).await?;
Ok(TransitionOperatorStatus {
transaction_id,
state: transaction.state,
tier_name: transaction.tier_name,
remote_object: transaction.remote_object,
not_after_unix_nanos: transaction.not_after_unix_nanos,
probe,
})
}
pub async fn delete_transition_candidate_for_operator(
api: Arc<ECStore>,
transaction_id: Uuid,
remote_version_id: &str,
) -> TransitionOperatorResult<TransitionOperatorDeleteResult> {
if remote_version_id.is_empty() {
return Err(TransitionOperatorError::RemoteVersionRequired);
}
let transaction = load_operator_reconcile_transaction(api.clone(), transaction_id).await?;
validate_operator_reconcile_transaction(&transaction, time::OffsetDateTime::now_utc().unix_timestamp_nanos())?;
let lease = TierConfigMgr::acquire_operation_lease_for_backend_identity(
&api.tier_config_mgr(),
&transaction.tier_name,
transaction.backend_fingerprint,
)
.await
.map_err(|err| TransitionOperatorError::Remote(std::io::Error::other(err)))?;
lease
.validate_remote_version_id(remote_version_id)
.map_err(TransitionOperatorError::Remote)?;
let before_delete_probe = lease
.probe_transition_candidate_for(&transaction.remote_object, transaction.transaction_id)
.await
.map(TransitionOperatorProbe::from)
.map_err(TransitionOperatorError::Remote)?;
if !matches!(&before_delete_probe, TransitionOperatorProbe::VersionedPresent(version_id) if version_id == remote_version_id) {
return Err(TransitionOperatorError::CandidateVersionMismatch {
expected: remote_version_id.to_string(),
actual: before_delete_probe,
});
}
delete_confirmed_transition_candidate_exact_with_lease_idempotent(&transaction.remote_object, remote_version_id, &lease)
.await
.map_err(TransitionOperatorError::Remote)?;
let probe = operator_probe_transition_candidate(api.clone(), &transaction).await?;
let journal_observed_after_delete = match load_transition_transaction_record(api, transaction_id).await {
Ok(_) => true,
Err(Error::ConfigNotFound) => false,
Err(err) => return Err(TransitionOperatorError::Store(err)),
};
Ok(TransitionOperatorDeleteResult {
status: TransitionOperatorStatus {
transaction_id,
state: transaction.state,
tier_name: transaction.tier_name,
remote_object: transaction.remote_object,
not_after_unix_nanos: transaction.not_after_unix_nanos,
probe,
},
journal_observed_after_delete,
})
}
pub async fn finalize_missing_transition_transaction_for_operator(
api: Arc<ECStore>,
transaction_id: Uuid,
) -> TransitionOperatorResult<()> {
let transaction = load_operator_reconcile_transaction(api.clone(), transaction_id).await?;
validate_operator_reconcile_transaction(&transaction, time::OffsetDateTime::now_utc().unix_timestamp_nanos())?;
let probe = operator_probe_transition_candidate(api.clone(), &transaction).await?;
if probe != TransitionOperatorProbe::Missing {
return Err(TransitionOperatorError::CandidateNotMissing(probe));
}
delete_transition_transaction_record(api, transaction_id)
.await
.map_err(TransitionOperatorError::Store)
}
pub(crate) fn decode_transition_transaction_record(object: &str, data: &[u8]) -> Result<TransitionTransaction> {
let transaction_id = transition_transaction_id_from_record_object_name(object)?;
TransitionTransaction::decode(transaction_id, data)
@@ -700,7 +894,7 @@ async fn recover_unknown_upload_outcome(
.map_err(Error::other)?;
match lease
.probe_transition_candidate(&transaction.remote_object)
.probe_transition_candidate_for(&transaction.remote_object, transaction.transaction_id)
.await
.map_err(Error::other)?
{
@@ -1097,6 +1291,27 @@ mod tests {
.expect("upload state change should succeed")
}
#[test]
fn operator_reconcile_requires_expired_unknown_upload_outcome() {
let mut transaction = new_transaction();
let active_deadline = transaction.not_after_unix_nanos;
assert!(matches!(
validate_operator_reconcile_transaction(&transaction, i128::from(active_deadline) + 1),
Err(TransitionOperatorError::InvalidState(TransitionTransactionState::UploadStarted))
));
transaction
.advance(transaction.fence(), TransitionTransactionState::UploadOutcomeUnknown, None)
.expect("unknown upload outcome should be recorded");
assert!(matches!(
validate_operator_reconcile_transaction(&transaction, i128::from(active_deadline) - 1),
Err(TransitionOperatorError::NotExpired)
));
validate_operator_reconcile_transaction(&transaction, i128::from(active_deadline))
.expect("expired unknown upload outcome should be eligible");
}
fn cleanup_proof(transaction: &TransitionTransaction, decision: TransitionCleanupDecision) -> TransitionCleanupProof {
TransitionCleanupProof {
transaction_id: transaction.transaction_id,
+357 -51
View File
@@ -242,73 +242,138 @@ pub(crate) async fn remove_bucket_metadata_in(ctx: &crate::runtime::instance::In
Ok(lock.remove(bucket).await)
}
/// Rewrite one config file of a bucket's metadata, serialized cluster-wide.
///
/// See [`acquire_bucket_metadata_transaction_lock`] for why every config
/// write — not just the replication-targets one — has to hold that lock.
pub async fn update(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let _targets_guard = if config_file == BUCKET_TARGETS_FILE {
Some(acquire_bucket_targets_transaction_lock(bucket).await?)
} else {
None
};
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
update_with_sys(get_bucket_metadata_sys()?, bucket, config_file, data).await
}
pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
delete_with_sys(get_bucket_metadata_sys()?, bucket, config_file).await
}
/// [`update`] against an explicitly supplied metadata system.
///
/// The free functions resolve the instance's own system; this variant takes
/// it as an argument so a test can drive two independent systems over one
/// backing store — the in-process stand-in for two nodes, which is the only
/// configuration where the transaction lock is what does the serializing.
async fn update_with_sys(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
config_file: &str,
data: Vec<u8>,
) -> Result<OffsetDateTime> {
let (_transaction_guard, mut sys) = acquire_config_write_guards(sys, bucket).await?;
sys.update(bucket, config_file, data).await
}
/// [`delete`] against an explicitly supplied metadata system. See
/// [`update_with_sys`].
async fn delete_with_sys(sys: Arc<RwLock<BucketMetadataSys>>, bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
let (_transaction_guard, mut sys) = acquire_config_write_guards(sys, bucket).await?;
sys.delete(bucket, config_file).await
}
/// Take, in the one order every config write uses, the two guards a
/// read-modify-write of a bucket's metadata needs: the cluster-wide
/// transaction lock first, then this process's metadata-system write guard.
///
/// The order is load-bearing. Acquiring the process-local guard first would
/// park every local reader and writer of *every* bucket behind a lock whose
/// holder may be another node, turning remote contention into a local stall.
async fn acquire_config_write_guards(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
) -> Result<(rustfs_lock::NamespaceLockGuard, tokio::sync::OwnedRwLockWriteGuard<BucketMetadataSys>)> {
let transaction_guard = acquire_transaction_lock_with_sys(&sys, bucket).await?;
let sys_guard = sys.write_owned().await;
Ok((transaction_guard, sys_guard))
}
/// Rewrite one config file while the caller already holds this bucket's
/// transaction lock.
///
/// [`update`] would deadlock here: the lock is not reentrant, so a holder
/// that called it would block until its own guard timed out.
pub async fn update_under_transaction_lock(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update(bucket, config_file, data).await
}
pub async fn update_bucket_targets_under_transaction_lock(bucket: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update(bucket, BUCKET_TARGETS_FILE, data).await
update_under_transaction_lock(bucket, BUCKET_TARGETS_FILE, data).await
}
/// Read-modify-write one bucket config file under the metadata system's
/// outer write guard.
/// Read-modify-write one bucket config file under both guards a config
/// write takes.
///
/// `mutate` sees the freshly loaded on-disk metadata and returns the
/// replacement payload for `config_file` (empty clears it, like
/// [`delete`]). Both the read and the persisted write happen inside the
/// same guard that [`update`] uses, so within this process the rewrite can
/// neither clobber a concurrent update to another config file nor lose a
/// concurrent write to the same one — unlike caching a mutated clone of
/// previously read metadata.
/// same guards [`update`] takes, so the rewrite can neither clobber a
/// concurrent update to another config file nor lose a concurrent write to
/// the same one — unlike caching a mutated clone of previously read
/// metadata.
///
/// This guard is process-local. Writers on other nodes still race, exactly
/// as they do for [`update`]: each rewrites the whole metadata file, so the
/// later save wins. What this narrows is the window — from "as stale as the
/// local cache" down to a single metadata read plus write.
/// That exclusion is cluster-wide, not merely process-local: the transaction
/// lock is now taken for every config file rather than only the replication
/// targets one, so a writer on another node cannot land a whole-file save in
/// the middle of this read-modify-write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let _targets_guard = if config_file == BUCKET_TARGETS_FILE {
Some(acquire_bucket_targets_transaction_lock(bucket).await?)
} else {
None
};
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update_config_with(bucket, config_file, mutate).await
let (_transaction_guard, mut sys) = acquire_config_write_guards(get_bucket_metadata_sys()?, bucket).await?;
sys.update_config_with(bucket, config_file, mutate).await
}
pub async fn acquire_bucket_targets_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let api = bucket_meta_sys_lock.read().await.object_store();
/// Acquire a bucket's metadata transaction lock, held across a whole
/// read-modify-write of its metadata file.
///
/// Every config write loads the entire [`BucketMetadata`] blob, replaces one
/// field, and saves the whole thing back. The namespace locks inside
/// `read_config`/`save_config` are taken and released separately, so they do
/// not span that cycle: two nodes updating *different* config files of one
/// bucket both load the same blob, each set their own field, and the later
/// save drops the other's — with both clients already told 2xx. This is not
/// last-writer-wins on one document; an orthogonal config silently vanishes.
///
/// So the lock is per bucket, not per config file: a per-file key would let
/// exactly that pair run concurrently.
///
/// Callers that hold this guard must use [`update_under_transaction_lock`]
/// rather than [`update`] — see that function.
pub async fn acquire_bucket_metadata_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_transaction_lock_with_sys(&get_bucket_metadata_sys()?, bucket).await
}
async fn acquire_transaction_lock_with_sys(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
// Resolve the store under a short-lived read guard: this runs before the
// write guard in `acquire_config_write_guards`, and must not still hold a
// read guard when the namespace lock is awaited.
let api = sys.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_targets_transaction_lock_key(bucket))
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
Ok(lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout()).await?)
}
fn bucket_targets_transaction_lock_key(bucket: &str) -> String {
/// The lock resource name is deliberately still the `bucket-targets` one it
/// had when only replication-target writes took it. The key is what nodes
/// agree on, so renaming it would leave a mixed-version cluster with two
/// disjoint keys — and old and new nodes would stop excluding each other on
/// the very writes that are serialized today.
fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
format!("bucket-targets/{bucket}/transaction.lock")
}
pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.delete(bucket, config_file).await
}
pub async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -514,8 +579,26 @@ pub struct BucketMetadataSys {
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
/// Deduplicates concurrent lazy loads of one bucket's metadata, so N
/// simultaneous cache misses issue a single disk read instead of N.
///
/// This is the `singleflight` that upstream applies to its own lazy
/// `GetConfig`. Without it the namespace *read* lock the load holds is no
/// help: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers, and every caller still pays a full
/// erasure-set metadata fanout. A separate registry from
/// `metadata_publish_locks`, reusing the same per-bucket lock machinery.
///
/// Lock order: this lock, then the namespace lock, then the publish lock,
/// then the metadata map. It is only ever taken as the first of those, so
/// it cannot invert against a path that already holds one of the others.
lazy_load_locks: Arc<MetadataPublishLockRegistry>,
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool,
/// Counts disk loads taken by the lazy `get_config` path, so a test can
/// prove concurrent misses collapse into one.
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize,
/// Buckets recently observed to have no persisted metadata. Serving the
/// fabricated default from here (instead of re-reading disk) keeps the
/// per-request cost of repeated lookups for such names bounded — without
@@ -534,8 +617,13 @@ impl BucketMetadataSys {
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
lazy_load_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool::new(false),
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize::new(0),
absent_metadata: moka::future::Cache::builder()
.max_capacity(ABSENT_BUCKET_METADATA_MAX_ENTRIES)
.time_to_live(ABSENT_BUCKET_METADATA_TTL)
@@ -550,16 +638,22 @@ impl BucketMetadataSys {
}
fn metadata_publish_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
let mut locks = self
.metadata_publish_locks
.locks
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
Self::bucket_lock_in(&self.metadata_publish_locks, bucket)
}
/// Per-bucket gate for the lazy `get_config` disk load. See
/// [`Self::lazy_load_locks`].
fn lazy_load_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.lazy_load_locks, bucket)
}
fn bucket_lock_in(registry: &Arc<MetadataPublishLockRegistry>, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
let mut locks = registry.locks.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
locks.get(bucket).and_then(Weak::upgrade).unwrap_or_else(|| {
let lock = Arc::new_cyclic(|lock| {
Mutex::new(MetadataPublishLockState {
bucket: bucket.to_string(),
registry: Arc::downgrade(&self.metadata_publish_locks),
registry: Arc::downgrade(registry),
lock: lock.clone(),
})
});
@@ -864,11 +958,11 @@ impl BucketMetadataSys {
}
async fn update_and_parse(&mut self, bucket: &str, config_file: &str, data: Vec<u8>, parse: bool) -> Result<OffsetDateTime> {
let Some(store) = runtime_sources::object_store_handle() else {
return Err(Error::other("errServerNotInitialized"));
};
let mut bm = Self::load_bucket_metadata_for_update(store, bucket, parse).await?;
// Load through this system's own store, the one `save` persists to
// (backlog#1052 S7). Reading from the ambient handle instead made the
// read and the write of a single read-modify-write able to target
// different instances.
let mut bm = Self::load_bucket_metadata_for_update(self.api.clone(), bucket, parse).await?;
let updated = bm.update_config(config_file, data)?;
@@ -1014,6 +1108,27 @@ impl BucketMetadataSys {
return Ok((Arc::new(bm), true));
}
// Collapse concurrent misses for this bucket into one disk load.
// Taken before the namespace lock — see `lazy_load_locks` for the
// ordering rule.
let load_lock = self.lazy_load_lock(bucket);
let _load_guard = load_lock.lock_owned().await;
// Re-check both caches: whoever held the gate before us may have
// already answered this exact question, and repeating the fanout
// is the whole cost this gate exists to avoid.
if let Some(bm) = self.metadata_map.read().await.get(bucket).cloned() {
return Ok((bm, true));
}
if self.absent_metadata.get(bucket).await.is_some() {
let mut bm = BucketMetadata::new(bucket);
bm.default_timestamps();
return Ok((Arc::new(bm), true));
}
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let lock = self.api.new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -1366,6 +1481,43 @@ mod tests {
use serial_test::serial;
use tokio::time::timeout;
/// Concurrent cache misses for one bucket must collapse into a single disk
/// load.
///
/// The namespace read lock the lazy path already holds does not provide
/// this: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers. Without the dedup gate every caller pays its
/// own namespace-lock acquisition plus a full erasure-set metadata fanout —
/// and the paths that reach `get_config` are per-request, so the multiplier
/// is request concurrency.
#[tokio::test]
async fn concurrent_lazy_loads_of_one_bucket_issue_a_single_disk_read() {
use std::sync::atomic::Ordering;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(BucketMetadataSys::new(ecstore));
// A name with no persisted metadata: every caller misses the map, and
// the absent-cache entry does not exist until the first load records it.
let bucket = "singleflight-bucket";
let waiters = 8;
let results = futures::future::join_all((0..waiters).map(|_| {
let sys = Arc::clone(&sys);
async move { sys.get_config(bucket).await.map(|(bm, _)| bm.name.clone()) }
}))
.await;
for result in results {
assert_eq!(result.expect("every caller must get an answer"), bucket);
}
assert_eq!(
sys.lazy_disk_loads.load(Ordering::Relaxed),
1,
"concurrent misses for one bucket must share a single disk load"
);
}
/// Pins the fail-closed caching contract of the lazy `get_config` path
/// and the refresh no-replace rule: fabricated defaults are returned but
/// never served by the map-only `get()`, persisted metadata is cached on
@@ -1815,6 +1967,160 @@ mod tests {
);
}
/// Two metadata systems over one backing store: the in-process stand-in
/// for two nodes. They share no `RwLock`, so nothing but the transaction
/// lock can serialize them — exactly the cross-node case.
async fn two_nodes_over_one_store() -> (Vec<tempfile::TempDir>, Arc<RwLock<BucketMetadataSys>>, Arc<RwLock<BucketMetadataSys>>)
{
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let node_a = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
let node_b = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
(dirs, node_a, node_b)
}
/// Writers on different nodes updating *different* config files of one
/// bucket must both survive. Each rewrites the whole metadata blob, so
/// without a lock spanning the read-modify-write the later save carries
/// the earlier writer's field back to its pre-update value — losing an
/// orthogonal config while both clients were told the write succeeded.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn concurrent_config_writes_from_separate_nodes_do_not_lose_writes() {
use crate::bucket::metadata::{BUCKET_POLICY_CONFIG, BUCKET_TAGGING_CONFIG};
let (_dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "cross-node-config-writes";
node_a
.read()
.await
.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
// Several rounds: a single pass can serialize by luck, but a lost
// update only needs one interleaving to show up.
const ROUNDS: usize = 8;
for round in 0..ROUNDS {
let tagging = format!("<Tagging><Round>{round}</Round></Tagging>").into_bytes();
let policy = format!(r#"{{"Version":"2012-10-17","Round":{round}}}"#).into_bytes();
let start = Arc::new(tokio::sync::Barrier::new(2));
let tagging_writer = {
let (node, start, tagging) = (node_a.clone(), start.clone(), tagging.clone());
tokio::spawn(async move {
start.wait().await;
update_with_sys(node, bucket, BUCKET_TAGGING_CONFIG, tagging).await
})
};
let policy_writer = {
let (node, start, policy) = (node_b.clone(), start.clone(), policy.clone());
tokio::spawn(async move {
start.wait().await;
update_with_sys(node, bucket, BUCKET_POLICY_CONFIG, policy).await
})
};
tagging_writer
.await
.expect("tagging writer should join")
.expect("tagging update should succeed");
policy_writer
.await
.expect("policy writer should join")
.expect("policy update should succeed");
// Disk truth, not either node's cache: the losing write is the one
// that never reached the metadata file.
let persisted = node_a
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("metadata should load from disk");
assert_eq!(
persisted.tagging_config_xml, tagging,
"round {round}: the policy write clobbered the concurrent tagging write"
);
assert_eq!(
persisted.policy_config_json, policy,
"round {round}: the tagging write clobbered the concurrent policy write"
);
}
}
/// The guard has to cover the load as well as the save. If it were taken
/// only around the save, a second node could load between the two and
/// still overwrite with pre-update state.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn bucket_metadata_transaction_lock_blocks_a_concurrent_config_write() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
let (_dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "cross-node-transaction-lock";
node_a
.read()
.await
.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
let held = acquire_transaction_lock_with_sys(&node_a, bucket)
.await
.expect("transaction lock should be acquirable");
let blocked = tokio::spawn({
let node_b = node_b.clone();
async move { update_with_sys(node_b, bucket, BUCKET_TAGGING_CONFIG, b"<Tagging/>".to_vec()).await }
});
// Long enough for the write to have finished had it not waited: the
// whole read-modify-write against temp disks is far quicker than this.
tokio::time::sleep(Duration::from_millis(500)).await;
assert!(
!blocked.is_finished(),
"a config write must not proceed while another node holds the bucket's transaction lock"
);
drop(held);
let updated = timeout(Duration::from_secs(10), blocked)
.await
.expect("the blocked write should proceed once the lock is released")
.expect("blocked writer should join");
updated.expect("the write should succeed after acquiring the lock");
}
/// A holder of the transaction lock must not call the locking entry
/// point: the namespace lock is not reentrant, so it would block on
/// itself until the acquire timeout.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn transaction_lock_is_not_reentrant() {
let (_dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "transaction-lock-reentrancy";
let held = acquire_transaction_lock_with_sys(&node_a, bucket)
.await
.expect("first acquisition should succeed");
// Same store, hence the same locker owner: exclusion must not depend
// on the two acquisitions coming from different owners. Either
// outcome is acceptable — still waiting, or refused — as long as no
// second guard is handed out.
let reacquired = timeout(Duration::from_millis(500), acquire_transaction_lock_with_sys(&node_b, bucket)).await;
assert!(
!matches!(reacquired, Ok(Ok(_))),
"the transaction lock must exclude a second holder even under the same owner"
);
drop(held);
timeout(Duration::from_secs(10), acquire_transaction_lock_with_sys(&node_b, bucket))
.await
.expect("re-acquisition should not time out once released")
.expect("the lock should be acquirable after release");
}
fn target(bucket: &str, id: &str) -> BucketTarget {
BucketTarget {
source_bucket: bucket.to_string(),
+1 -1
View File
@@ -46,7 +46,7 @@ mod runtime_boundary;
pub use datatypes::ResyncStatusType;
pub use replication_config_boundary::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
};
#[cfg(test)]
pub(crate) use replication_filemeta_boundary::ReplicateTargetDecision;
@@ -14,5 +14,5 @@
pub use rustfs_replication::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
};
@@ -14,8 +14,11 @@
use std::{collections::HashMap, sync::Arc};
use super::replication_error_boundary::Result;
use super::replication_filemeta_boundary::{ReplicateDecision, ReplicationStatusType, ReplicationType};
use super::replication_object_config::{check_replicate_delete, get_must_replicate_options, must_replicate};
use super::replication_object_config::{
check_replicate_delete, check_replicate_delete_strict, get_must_replicate_options, must_replicate,
};
use super::replication_object_decision_boundary::MustReplicateOptions;
use super::replication_pool::{schedule_replication, schedule_replication_delete};
use super::replication_queue_boundary::DeletedObjectReplicationInfo;
@@ -50,6 +53,16 @@ impl ReplicationObjectBridge {
check_replicate_delete(bucket, object, source, opts, get_error).await
}
pub async fn check_delete_strict(
bucket: &str,
object: &ObjectToDelete,
source: &ObjectInfo,
opts: &ObjectOptions,
get_error: Option<String>,
) -> Result<ReplicateDecision> {
check_replicate_delete_strict(bucket, object, source, opts, get_error).await
}
pub async fn schedule_object<S: ReplicationStorage>(
object: ObjectInfo,
storage: Arc<S>,
@@ -169,9 +169,8 @@ pub(crate) async fn check_replicate_delete(
del_opts: &ObjectOptions,
gerr: Option<String>,
) -> ReplicateDecision {
let rcfg = match get_replication_config(bucket).await {
Ok(Some(config)) => config,
Ok(None) => return ReplicateDecision::default(),
match check_replicate_delete_strict(bucket, dobj, oi, del_opts, gerr).await {
Ok(decision) => decision,
Err(err) => {
error!(
event = EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED,
@@ -182,16 +181,30 @@ pub(crate) async fn check_replicate_delete(
error = %err,
"Failed to look up replication config for delete replication"
);
return ReplicateDecision::default();
ReplicateDecision::default()
}
}
}
pub(crate) async fn check_replicate_delete_strict(
bucket: &str,
dobj: &ObjectToDelete,
oi: &ObjectInfo,
del_opts: &ObjectOptions,
gerr: Option<String>,
) -> Result<ReplicateDecision> {
let rcfg = match get_replication_config(bucket).await {
Ok(Some(config)) => config,
Ok(None) => return Ok(ReplicateDecision::default()),
Err(err) => return Err(err),
};
if del_opts.replication_request {
return ReplicateDecision::default();
return Ok(ReplicateDecision::default());
}
if !del_opts.versioned {
return ReplicateDecision::default();
if !del_opts.versioned && !del_opts.version_suspended {
return Ok(ReplicateDecision::default());
}
let replication_delete = object_to_delete_for_replication(dobj);
@@ -209,7 +222,7 @@ pub(crate) async fn check_replicate_delete(
let mut dsc = ReplicateDecision::new();
if tgt_arns.is_empty() {
return dsc;
return Ok(dsc);
}
for tgt_arn in tgt_arns {
@@ -239,7 +252,7 @@ pub(crate) async fn check_replicate_delete(
dsc.set(tgt_dsc);
}
dsc
Ok(dsc)
}
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
@@ -1151,60 +1151,6 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
dobj.delete_object.version_id
};
let _rcfg = match get_replication_config(&bucket).await {
Ok(Some(config)) => config,
Ok(None) => {
debug!(
event = EVENT_REPLICATION_DELETE_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
reason = "replication_config_missing",
"Skipping replication delete because replication config is missing"
);
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: ObjectInfo {
bucket: bucket.clone(),
name: dobj.delete_object.object_name.clone(),
version_id,
delete_marker: dobj.delete_object.delete_marker,
..Default::default()
},
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
return;
}
Err(err) => {
debug!(
event = EVENT_REPLICATION_DELETE_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
error = %err,
reason = "replication_config_lookup_failed",
"Skipping replication delete because replication config lookup failed"
);
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: ObjectInfo {
bucket: bucket.clone(),
name: dobj.delete_object.object_name.clone(),
version_id,
delete_marker: dobj.delete_object.delete_marker,
..Default::default()
},
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
return;
}
};
if dobj.delete_object.delete_marker
&& let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id
{
@@ -172,7 +172,7 @@ impl ProviderVersionCapabilities {
}
}
fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
pub(crate) fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
if version_id.is_empty() {
return Err(Error::new(
ErrorKind::InvalidData,
+1 -15
View File
@@ -46,16 +46,6 @@ lazy_static! {
m.insert("x-amz-replication-status".to_string(), true);
m
};
static ref SSE_HEADERS: HashMap<String, bool> = {
let mut m = HashMap::new();
m.insert("x-amz-server-side-encryption".to_string(), true);
m.insert("x-amz-server-side-encryption-aws-kms-key-id".to_string(), true);
m.insert("x-amz-server-side-encryption-context".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-algorithm".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key-md5".to_string(), true);
m
};
}
pub fn is_standard_query_value(qs_key: &str) -> bool {
@@ -70,16 +60,12 @@ pub fn is_standard_header(header_key: &str) -> bool {
*SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_sse_header(header_key: &str) -> bool {
*SSE_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_amz_header(header_key: &str) -> bool {
let key = header_key.to_lowercase();
key.starts_with("x-amz-meta-")
|| key.starts_with("x-amz-grant-")
|| key == "x-amz-acl"
|| is_sse_header(header_key)
|| rustfs_utils::http::is_sse_header(header_key)
|| key.starts_with("x-amz-checksum-")
}
+214 -9
View File
@@ -12,11 +12,20 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::cluster::rpc::http_auth::RPC_CONTENT_SHA256_HEADER;
use crate::cluster::rpc::{gen_tonic_signature_headers, normalize_tonic_rpc_audience};
#[cfg(test)]
use crate::cluster::rpc::http_auth::RPC_REPLAY_SCOPE_VERSION_HEADER;
use crate::cluster::rpc::http_auth::{
RPC_AUTH_VERSION_HEADER, RPC_AUTH_VERSION_V2, RPC_BOOT_EPOCH_CHALLENGE_HEADER, RPC_BOOT_EPOCH_HEADER,
RPC_BOOT_EPOCH_PROOF_HEADER, RPC_CONTENT_SHA256_HEADER, TIMESTAMP_HEADER,
};
use crate::cluster::rpc::{
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience, verify_tonic_boot_epoch_response,
};
#[cfg(test)]
use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
use crate::runtime::sources as runtime_sources;
use http::Uri;
use http::{Request as HttpRequest, Response as HttpResponse, Uri};
use rustfs_protos::{
ChannelClass, create_new_channel, get_channel_for_class,
proto_gen::node_service::{
@@ -24,9 +33,19 @@ use rustfs_protos::{
tier_mutation_control_service_client::TierMutationControlServiceClient,
},
};
use std::{error::Error, io::ErrorKind};
use std::{
collections::HashMap,
error::Error,
future::Future,
io::ErrorKind,
pin::Pin,
sync::{LazyLock, Mutex},
task::{Context, Poll},
};
use tonic::{service::interceptor::InterceptedService, transport::Channel};
use tower::Service;
use tracing::debug;
use uuid::Uuid;
use super::context_propagation::{inject_request_id_into_metadata, inject_trace_context_into_metadata};
@@ -35,7 +54,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
pub async fn node_service_time_out_client(
addr: &String,
interceptor: TonicInterceptor,
) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
// Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
// `_for_class` variant below (grpc-optimization P1).
node_service_time_out_client_for_class(addr, interceptor, ChannelClass::Control).await
@@ -44,13 +63,14 @@ pub async fn node_service_time_out_client(
pub async fn heal_control_time_out_client(
addr: &str,
interceptor: TonicInterceptor,
) -> Result<HealControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
@@ -59,13 +79,14 @@ pub async fn heal_control_time_out_client(
pub async fn tier_mutation_control_time_out_client(
addr: &str,
interceptor: TonicInterceptor,
) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
@@ -81,7 +102,7 @@ pub async fn node_service_time_out_client_for_class(
addr: &String,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match class {
ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
@@ -96,6 +117,7 @@ pub async fn node_service_time_out_client_for_class(
};
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
@@ -103,7 +125,7 @@ pub async fn node_service_time_out_client_for_class(
pub async fn node_service_time_out_client_no_auth(
addr: &String,
) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
}
@@ -199,6 +221,104 @@ pub(crate) fn is_network_like_disk_error(err: &DiskErrorType) -> bool {
}
}
/// The transport service that learns an authenticated peer boot epoch and adds the replay-scoped
/// signature only after one has been observed. The v1/v2 interceptor stays inside this wrapper so
/// old servers continue receiving precisely the metadata they understand.
#[derive(Clone, Debug)]
pub struct ReplayScopeChannel<S> {
inner: S,
audience: Option<String>,
}
/// The channel type used by internode clients after v2 authentication and replay-scope handling.
pub type AuthenticatedChannel = ReplayScopeChannel<Channel>;
static PEER_BOOT_EPOCHS: LazyLock<Mutex<HashMap<String, Uuid>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
impl<S> ReplayScopeChannel<S> {
fn new(inner: S, audience: Option<String>) -> Self {
Self { inner, audience }
}
}
fn cached_peer_boot_epoch(audience: &str) -> Option<Uuid> {
PEER_BOOT_EPOCHS.lock().ok().and_then(|epochs| epochs.get(audience).copied())
}
fn remember_peer_boot_epoch(audience: String, epoch: Uuid) {
if let Ok(mut epochs) = PEER_BOOT_EPOCHS.lock() {
epochs.insert(audience, epoch);
}
}
impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
where
S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
S::Error: Send + 'static,
S::Future: Send + 'static,
ReqBody: Send + 'static,
ResBody: Send + 'static,
{
type Response = HttpResponse<ResBody>;
type Error = S::Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
self.inner.poll_ready(cx)
}
fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
let authenticated = self.audience.as_ref().is_some_and(|_| {
request
.headers()
.get(RPC_AUTH_VERSION_HEADER)
.and_then(|value| value.to_str().ok())
== Some(RPC_AUTH_VERSION_V2)
});
let challenge = authenticated.then(Uuid::new_v4);
if let (Some(audience), Some(challenge)) = (self.audience.as_deref(), challenge) {
// The challenge is independently HMAC-authenticated by the response proof. It is not
// part of v2 so old peers ignore it, while a new peer can safely advertise its epoch.
request.headers_mut().insert(
RPC_BOOT_EPOCH_CHALLENGE_HEADER,
challenge.to_string().parse().expect("UUID must be a valid header value"),
);
if let (Some(boot_epoch), Some(timestamp), Some(content_sha256)) = (
cached_peer_boot_epoch(audience),
request.headers().get(TIMESTAMP_HEADER).and_then(|value| value.to_str().ok()),
request
.headers()
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok()),
) {
match gen_tonic_replay_scope_headers(audience, request.uri().path(), timestamp, content_sha256, boot_epoch) {
Ok(headers) => request.headers_mut().extend(headers),
Err(error) => debug!(error = %error, "could not attach replay-scoped RPC signature"),
}
}
}
let audience = self.audience.clone();
let future = self.inner.call(request);
Box::pin(async move {
let response = future.await?;
if let (Some(audience), Some(challenge)) = (audience, challenge) {
match verify_tonic_boot_epoch_response(&audience, challenge, response.headers()) {
Ok(epoch) => remember_peer_boot_epoch(audience, epoch),
Err(error)
if response.headers().contains_key(RPC_BOOT_EPOCH_HEADER)
|| response.headers().contains_key(RPC_BOOT_EPOCH_PROOF_HEADER) =>
{
debug!(error = %error, "peer boot epoch response proof was rejected")
}
Err(_) => {}
}
}
Ok(response)
})
}
}
pub struct TonicSignatureInterceptor {
audience: Option<String>,
}
@@ -257,6 +377,13 @@ impl TonicInterceptor {
}
Ok(self)
}
fn replay_scope_audience(&self) -> Option<String> {
match self {
Self::Signature(interceptor) => interceptor.audience.clone(),
Self::NoOp(_) => None,
}
}
}
impl tonic::service::Interceptor for TonicInterceptor {
@@ -279,6 +406,38 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[derive(Clone)]
struct EpochProofService {
audience: String,
seen_headers: std::sync::Arc<Mutex<Vec<http::HeaderMap>>>,
}
impl Service<HttpRequest<()>> for EpochProofService {
type Response = HttpResponse<()>;
type Error = std::convert::Infallible;
type Future = std::future::Ready<Result<Self::Response, Self::Error>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, request: HttpRequest<()>) -> Self::Future {
self.seen_headers
.lock()
.expect("test header capture lock must not be poisoned")
.push(request.headers().clone());
let challenge = tonic_boot_epoch_challenge(request.headers())
.expect("client challenge must be syntactically valid")
.expect("authenticated client request must carry a boot epoch challenge");
let mut response = HttpResponse::new(());
response.headers_mut().extend(
tonic_boot_epoch_response_headers(&self.audience, challenge)
.expect("test server must be able to sign an epoch proof"),
);
std::future::ready(Ok(response))
}
}
fn ensure_test_rpc_secret() {
runtime_sources::ensure_test_rpc_secret();
}
@@ -420,6 +579,52 @@ mod tests {
assert_eq!(interceptor.audience.as_deref(), Some("node-a:9000"));
}
#[test]
fn replay_scope_channel_uses_epoch_proof_before_sending_v3() {
ensure_test_rpc_secret();
let audience = "replay-scope-client-test:9000";
PEER_BOOT_EPOCHS
.lock()
.expect("peer epoch cache lock must not be poisoned")
.remove(audience);
let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
let service = EpochProofService {
audience: audience.to_string(),
seen_headers: seen_headers.clone(),
};
let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
let make_request = || {
let mut request = HttpRequest::builder()
.uri("/node_service.NodeService/Ping")
.body(())
.expect("test RPC request must build");
request.headers_mut().extend(
gen_tonic_signature_headers(audience, "node_service.NodeService", "Ping", None)
.expect("v2 test headers must mint"),
);
request
};
futures::executor::block_on(channel.call(make_request())).expect("first request must complete");
futures::executor::block_on(channel.call(make_request())).expect("second request must complete");
let headers = seen_headers.lock().expect("test header capture lock must not be poisoned");
assert_eq!(headers.len(), 2);
assert!(headers[0].contains_key(RPC_BOOT_EPOCH_CHALLENGE_HEADER));
assert!(
!headers[0].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
"the first request must remain v2-compatible until the peer proves its epoch"
);
assert!(
headers[1].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
"the second request must carry the replay-scoped v3 signature"
);
PEER_BOOT_EPOCHS
.lock()
.expect("peer epoch cache lock must not be poisoned")
.remove(audience);
}
#[test]
fn test_signature_interceptor_requires_generated_method_metadata() {
ensure_test_rpc_secret();
+435 -9
View File
@@ -20,8 +20,8 @@
//! rustfs/rustfs#4402) is anchored by the `ghsa_r5qv_*` tests in the module
//! below, plus the broader negative-signature suite. The advisory class is: a
//! node must never accept an RPC whose auth is missing, malformed, or signed
//! with the default/empty shared secret. Body-bound v2 requests additionally
//! receive process-local replay protection. See
//! with the default/empty shared secret. Body-bound v2 requests and all replay-scoped v3
//! requests additionally receive process-local replay protection. See
//! `docs/testing/security-regressions.md` for the full advisory -> test map.
//!
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q>
@@ -50,13 +50,22 @@ use uuid::Uuid;
type HmacSha256 = Hmac<Sha256>;
const SIGNATURE_HEADER: &str = "x-rustfs-signature";
const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
const RPC_AUTH_VERSION_HEADER: &str = "x-rustfs-rpc-auth-version";
pub(crate) const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
pub(crate) const RPC_AUTH_VERSION_HEADER: &str = "x-rustfs-rpc-auth-version";
const RPC_SIGNATURE_V2_HEADER: &str = "x-rustfs-rpc-signature-v2";
const RPC_NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
pub(crate) const RPC_CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
const RPC_AUTH_VERSION_V2: &str = "2";
pub(crate) const RPC_AUTH_VERSION_V2: &str = "2";
pub const RPC_REPLAY_SCOPE_VERSION_HEADER: &str = "x-rustfs-rpc-replay-scope-version";
pub const RPC_REPLAY_SCOPE_SIGNATURE_HEADER: &str = "x-rustfs-rpc-signature-v3";
pub const RPC_REPLAY_SCOPE_NONCE_HEADER: &str = "x-rustfs-rpc-replay-nonce";
pub const RPC_BOOT_EPOCH_HEADER: &str = "x-rustfs-rpc-boot-epoch";
pub const RPC_BOOT_EPOCH_CHALLENGE_HEADER: &str = "x-rustfs-rpc-boot-epoch-challenge";
pub const RPC_BOOT_EPOCH_PROOF_HEADER: &str = "x-rustfs-rpc-boot-epoch-proof";
const RPC_REPLAY_SCOPE_VERSION_V3: &str = "3";
const RPC_RESPONSE_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-response-proof-v1\0";
const RPC_REPLAY_SCOPE_DOMAIN: &[u8] = b"rustfs-rpc-replay-scope-v3\0";
const RPC_BOOT_EPOCH_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-boot-epoch-proof-v1\0";
const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
@@ -75,9 +84,15 @@ static INTERNODE_RPC_BODY_DIGEST_STRICT: LazyLock<bool> = LazyLock::new(|| {
rustfs_config::DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT,
)
});
// Sized for peak legitimate body-bound mutation RPS x the retention window; overflow fails closed
// and increments the replay-cache overflow counter. Clamped to at least 1 so a misconfigured zero
// cannot disable replay protection by rejecting every body-bound request.
static INTERNODE_RPC_REPLAY_SCOPE_STRICT: LazyLock<bool> = LazyLock::new(|| {
get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT,
rustfs_config::DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT,
)
});
// Sized for peak legitimate authenticated RPC RPS x the retention window once replay scope is
// active; overflow fails closed and increments the replay-cache overflow counter. Clamped to at
// least 1 so a misconfigured zero cannot disable replay protection by rejecting every request.
static REPLAY_CACHE_CAPACITY: LazyLock<usize> = LazyLock::new(|| {
rustfs_utils::get_env_usize(
rustfs_config::ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY,
@@ -86,6 +101,7 @@ static REPLAY_CACHE_CAPACITY: LazyLock<usize> = LazyLock::new(|| {
.max(1)
});
static RPC_SECRET_RESOLUTION_LOG_ONCE: Once = Once::new();
static RPC_BOOT_EPOCH: LazyLock<Uuid> = LazyLock::new(Uuid::new_v4);
#[derive(Default)]
struct RpcNonceCache {
@@ -313,6 +329,189 @@ fn verify_signature_v2(secret: &str, scope: SignatureV2Scope<'_>, signature: &st
mac.verify_slice(&signature).is_ok()
}
#[derive(Clone, Copy)]
struct ReplayScope<'a> {
audience: &'a str,
path: &'a str,
timestamp: &'a str,
nonce: Uuid,
content_sha256: &'a str,
boot_epoch: Uuid,
}
fn update_replay_scope(mac: &mut HmacSha256, scope: ReplayScope<'_>) {
mac.update(RPC_REPLAY_SCOPE_DOMAIN);
for part in [
scope.audience.as_bytes(),
b"|",
scope.path.as_bytes(),
b"|POST|",
scope.timestamp.as_bytes(),
b"|",
scope.nonce.as_bytes(),
b"|",
scope.content_sha256.as_bytes(),
b"|",
scope.boot_epoch.as_bytes(),
] {
mac.update(part);
}
}
fn generate_replay_scope_signature(secret: &str, scope: ReplayScope<'_>) -> std::io::Result<String> {
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_replay_scope(&mut mac, scope);
Ok(general_purpose::STANDARD.encode(mac.finalize().into_bytes()))
}
fn verify_replay_scope_signature(secret: &str, scope: ReplayScope<'_>, signature: &str) -> bool {
let Ok(signature) = general_purpose::STANDARD.decode(signature) else {
return false;
};
let Ok(mut mac) = <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()) else {
return false;
};
update_replay_scope(&mut mac, scope);
mac.verify_slice(&signature).is_ok()
}
fn update_boot_epoch_proof(mac: &mut HmacSha256, audience: &str, challenge: Uuid, boot_epoch: Uuid) {
mac.update(RPC_BOOT_EPOCH_PROOF_DOMAIN);
mac.update(audience.as_bytes());
mac.update(b"|");
mac.update(challenge.as_bytes());
mac.update(boot_epoch.as_bytes());
}
fn generate_boot_epoch_proof(secret: &str, audience: &str, challenge: Uuid, boot_epoch: Uuid) -> std::io::Result<String> {
if audience.is_empty() || challenge.is_nil() || boot_epoch.is_nil() {
return Err(std::io::Error::other("Invalid RPC boot epoch proof scope"));
}
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_boot_epoch_proof(&mut mac, audience, challenge, boot_epoch);
Ok(general_purpose::STANDARD.encode(mac.finalize().into_bytes()))
}
fn verify_boot_epoch_proof(secret: &str, audience: &str, challenge: Uuid, boot_epoch: Uuid, proof: &str) -> std::io::Result<()> {
if audience.is_empty() || challenge.is_nil() || boot_epoch.is_nil() {
return Err(std::io::Error::other("Invalid RPC boot epoch proof scope"));
}
let proof = general_purpose::STANDARD
.decode(proof)
.map_err(|_| std::io::Error::other("Invalid RPC boot epoch proof"))?;
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_boot_epoch_proof(&mut mac, audience, challenge, boot_epoch);
mac.verify_slice(&proof)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid RPC boot epoch proof"))
}
fn non_nil_uuid(value: &str, name: &str) -> std::io::Result<Uuid> {
let value = Uuid::parse_str(value).map_err(|_| std::io::Error::other(format!("Invalid {name}")))?;
(!value.is_nil())
.then_some(value)
.ok_or_else(|| std::io::Error::other(format!("Invalid {name}")))
}
fn parse_tonic_rpc_path(path: &str) -> std::io::Result<(&str, &str)> {
path.strip_prefix('/')
.and_then(|path| path.split_once('/'))
.filter(|(service, rpc_method)| !service.is_empty() && !rpc_method.is_empty() && !rpc_method.contains('/'))
.ok_or_else(|| std::io::Error::other("Invalid RPC request path"))
}
/// The process-unique epoch included in every replay-scoped server verification.
///
/// A fresh process gets a fresh value, so a signature captured before a server restart cannot be
/// admitted even though the bounded in-memory nonce cache necessarily starts empty again.
pub fn tonic_rpc_boot_epoch() -> Uuid {
*RPC_BOOT_EPOCH
}
/// Build the additive replay-scope headers for a request that already carries rolling-upgrade-safe
/// v1/v2 metadata. `timestamp` and `content_sha256` are deliberately reused from the v2 scope so
/// old servers can continue validating the same request unchanged.
pub fn gen_tonic_replay_scope_headers(
audience: &str,
path: &str,
timestamp: &str,
content_sha256: &str,
boot_epoch: Uuid,
) -> std::io::Result<HeaderMap> {
if audience.is_empty() || !path.starts_with('/') || !valid_content_sha256(content_sha256) || boot_epoch.is_nil() {
return Err(std::io::Error::other("Invalid replay-scoped RPC signing scope"));
}
parse_tonic_rpc_path(path)?;
timestamp
.parse::<i64>()
.map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
let nonce = Uuid::new_v4();
let signature = generate_replay_scope_signature(
&get_shared_secret()?,
ReplayScope {
audience,
path,
timestamp,
nonce,
content_sha256,
boot_epoch,
},
)?;
let mut headers = HeaderMap::new();
headers.insert(RPC_REPLAY_SCOPE_VERSION_HEADER, HeaderValue::from_static(RPC_REPLAY_SCOPE_VERSION_V3));
headers.insert(
RPC_REPLAY_SCOPE_SIGNATURE_HEADER,
header_value(&signature, RPC_REPLAY_SCOPE_SIGNATURE_HEADER)?,
);
headers.insert(
RPC_REPLAY_SCOPE_NONCE_HEADER,
header_value(&nonce.to_string(), RPC_REPLAY_SCOPE_NONCE_HEADER)?,
);
headers.insert(RPC_BOOT_EPOCH_HEADER, header_value(&boot_epoch.to_string(), RPC_BOOT_EPOCH_HEADER)?);
Ok(headers)
}
/// Parse the optional client challenge used to authenticate a server boot-epoch advertisement.
pub fn tonic_boot_epoch_challenge(headers: &HeaderMap) -> std::io::Result<Option<Uuid>> {
headers
.get(RPC_BOOT_EPOCH_CHALLENGE_HEADER)
.map(|value| {
value
.to_str()
.map_err(|_| std::io::Error::other("Invalid RPC boot epoch challenge"))
.and_then(|value| non_nil_uuid(value, "RPC boot epoch challenge"))
})
.transpose()
}
/// Build the authenticated response headers for a client boot-epoch challenge.
pub fn tonic_boot_epoch_response_headers(audience: &str, challenge: Uuid) -> std::io::Result<HeaderMap> {
let boot_epoch = tonic_rpc_boot_epoch();
let proof = generate_boot_epoch_proof(&get_shared_secret()?, audience, challenge, boot_epoch)?;
let mut headers = HeaderMap::new();
headers.insert(RPC_BOOT_EPOCH_HEADER, header_value(&boot_epoch.to_string(), RPC_BOOT_EPOCH_HEADER)?);
headers.insert(RPC_BOOT_EPOCH_PROOF_HEADER, header_value(&proof, RPC_BOOT_EPOCH_PROOF_HEADER)?);
Ok(headers)
}
/// Verify the server boot-epoch response for a challenge generated by this client.
pub fn verify_tonic_boot_epoch_response(audience: &str, challenge: Uuid, headers: &HeaderMap) -> std::io::Result<Uuid> {
let boot_epoch = headers
.get(RPC_BOOT_EPOCH_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC boot epoch"))
.and_then(|value| non_nil_uuid(value, "RPC boot epoch"))?;
let proof = headers
.get(RPC_BOOT_EPOCH_PROOF_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC boot epoch proof"))?;
verify_boot_epoch_proof(&get_shared_secret()?, audience, challenge, boot_epoch, proof)?;
Ok(boot_epoch)
}
fn valid_content_sha256(value: &str) -> bool {
value == UNSIGNED_PAYLOAD
|| (value.len() == 64
@@ -531,6 +730,17 @@ fn has_v2_auth_headers(headers: &HeaderMap) -> bool {
.any(|name| headers.contains_key(*name))
}
fn has_replay_scope_headers(headers: &HeaderMap) -> bool {
[
RPC_REPLAY_SCOPE_VERSION_HEADER,
RPC_REPLAY_SCOPE_SIGNATURE_HEADER,
RPC_REPLAY_SCOPE_NONCE_HEADER,
RPC_BOOT_EPOCH_HEADER,
]
.iter()
.any(|name| headers.contains_key(*name))
}
/// Whether the server requires target-bound v2 authentication on every internode gRPC request,
/// rejecting the legacy constant-target fallback instead of accepting it. Default-off rollout
/// lever gated on the v1-fallback counter reading zero fleet-wide; see
@@ -540,9 +750,127 @@ fn internode_rpc_signature_strict() -> bool {
*INTERNODE_RPC_SIGNATURE_STRICT
}
fn internode_rpc_replay_scope_strict() -> bool {
*INTERNODE_RPC_REPLAY_SCOPE_STRICT
}
fn verify_tonic_replay_scope_signature(audience: &str, path: &str, headers: &HeaderMap) -> std::io::Result<()> {
if audience.is_empty() {
return Err(std::io::Error::other("Missing RPC audience"));
}
parse_tonic_rpc_path(path)?;
let version = headers
.get(RPC_REPLAY_SCOPE_VERSION_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC replay scope version"))?;
if version != RPC_REPLAY_SCOPE_VERSION_V3 {
return Err(std::io::Error::other("Unsupported RPC replay scope version"));
}
let signature = headers
.get(RPC_REPLAY_SCOPE_SIGNATURE_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC replay scope signature"))?;
let timestamp = headers
.get(TIMESTAMP_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing timestamp header"))?;
let signed_at = timestamp
.parse::<i64>()
.map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
check_timestamp(signed_at)?;
let nonce = headers
.get(RPC_REPLAY_SCOPE_NONCE_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC replay scope nonce"))
.and_then(|value| non_nil_uuid(value, "RPC replay scope nonce"))?;
let content_sha256 = headers
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC content SHA-256"))?;
if !valid_content_sha256(content_sha256) {
return Err(std::io::Error::other("Invalid RPC content SHA-256"));
}
let boot_epoch = headers
.get(RPC_BOOT_EPOCH_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC boot epoch"))
.and_then(|value| non_nil_uuid(value, "RPC boot epoch"))?;
let secret = get_shared_secret()?;
if !verify_replay_scope_signature(
&secret,
ReplayScope {
audience,
path,
timestamp,
nonce,
content_sha256,
boot_epoch,
},
signature,
) {
return Err(std::io::Error::other("Invalid RPC replay scope signature"));
}
if boot_epoch != tonic_rpc_boot_epoch() {
return Err(std::io::Error::other("RPC boot epoch is stale"));
}
check_and_record_nonce(nonce, signed_at)
}
/// Verify gRPC authentication, preferring v2 without downgrade on malformed v2 metadata.
pub fn verify_tonic_rpc_signature(audience: &str, path: &str, headers: &HeaderMap) -> std::io::Result<()> {
verify_tonic_rpc_signature_with_strictness(audience, path, headers, internode_rpc_signature_strict())
verify_tonic_rpc_signature_with_policy(
audience,
path,
headers,
internode_rpc_signature_strict(),
internode_rpc_replay_scope_strict(),
false,
)
}
/// Verify gRPC authentication while allowing the narrowly scoped v2 `Ping` bootstrap used to
/// obtain an authenticated server boot epoch when replay-scope strictness is enabled.
pub fn verify_tonic_rpc_signature_with_bootstrap(
audience: &str,
path: &str,
headers: &HeaderMap,
allow_replay_scope_bootstrap: bool,
) -> std::io::Result<()> {
verify_tonic_rpc_signature_with_policy(
audience,
path,
headers,
internode_rpc_signature_strict(),
internode_rpc_replay_scope_strict(),
allow_replay_scope_bootstrap,
)
}
fn verify_tonic_rpc_signature_with_policy(
audience: &str,
path: &str,
headers: &HeaderMap,
signature_strict: bool,
replay_scope_strict: bool,
allow_replay_scope_bootstrap: bool,
) -> std::io::Result<()> {
if has_replay_scope_headers(headers) {
return verify_tonic_replay_scope_signature(audience, path, headers);
}
// Only a method-bound v2 Ping with a syntactically valid challenge may bootstrap a strict
// client after its peer restarts. Legacy metadata never gets this exception.
let bootstrap = allow_replay_scope_bootstrap
&& has_v2_auth_headers(headers)
&& tonic_boot_epoch_challenge(headers).is_ok_and(|challenge| challenge.is_some());
if replay_scope_strict && !bootstrap {
return Err(std::io::Error::other("RPC replay-scoped authentication required"));
}
verify_tonic_rpc_signature_with_strictness(audience, path, headers, signature_strict)?;
global_internode_metrics().record_replay_scope_fallback();
Ok(())
}
/// [`verify_tonic_rpc_signature`] with the strict gate injected as a parameter, so both rollout
@@ -1273,6 +1601,104 @@ mod tests {
assert_eq!(error.to_string(), "Invalid RPC v2 signature");
}
#[test]
fn replay_scope_binds_path_epoch_and_random_nonce() {
ensure_test_rpc_secret();
let path = "/node_service.NodeService/Ping";
let mut headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
.expect("v2 compatibility headers should build");
let timestamp = headers
.get(TIMESTAMP_HEADER)
.and_then(|value| value.to_str().ok())
.expect("v2 timestamp")
.to_string();
let content_sha256 = headers
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok())
.expect("v2 content digest")
.to_string();
headers.extend(
gen_tonic_replay_scope_headers("node-a:9000", path, &timestamp, &content_sha256, tonic_rpc_boot_epoch())
.expect("replay-scope headers should build"),
);
assert!(
verify_tonic_rpc_signature_with_policy("node-a:9000", path, &headers, false, false, false).is_ok(),
"the first replay-scoped request must be accepted"
);
let replay = verify_tonic_rpc_signature_with_policy("node-a:9000", path, &headers, false, false, false)
.expect_err("the random replay-scope nonce must be single-use");
assert_eq!(replay.to_string(), "RPC request replay detected");
let path_error = verify_tonic_replay_scope_signature("node-a:9000", "/node_service.NodeService/SignalService", &headers)
.expect_err("a replay-scoped signature must not move to another method");
assert_eq!(path_error.to_string(), "Invalid RPC replay scope signature");
}
#[test]
fn replay_scope_rejects_partial_metadata_and_stale_epoch_without_fallback() {
ensure_test_rpc_secret();
let path = "/node_service.NodeService/Ping";
let mut partial = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
.expect("v2 compatibility headers should build");
partial.insert(RPC_REPLAY_SCOPE_VERSION_HEADER, HeaderValue::from_static(RPC_REPLAY_SCOPE_VERSION_V3));
let error = verify_tonic_rpc_signature_with_policy("node-a:9000", path, &partial, false, false, false)
.expect_err("partial replay-scope metadata must never downgrade to v2");
assert_eq!(error.to_string(), "Missing RPC replay scope signature");
let timestamp = partial
.get(TIMESTAMP_HEADER)
.and_then(|value| value.to_str().ok())
.expect("v2 timestamp")
.to_string();
let content_sha256 = partial
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok())
.expect("v2 content digest")
.to_string();
let stale_epoch = Uuid::new_v4();
partial.extend(
gen_tonic_replay_scope_headers("node-a:9000", path, &timestamp, &content_sha256, stale_epoch)
.expect("replay-scope headers should build"),
);
let stale = verify_tonic_rpc_signature_with_policy("node-a:9000", path, &partial, false, false, false)
.expect_err("a signature from a prior server boot epoch must be rejected");
assert_eq!(stale.to_string(), "RPC boot epoch is stale");
}
#[test]
fn replay_scope_strictness_allows_only_authenticated_ping_bootstrap() {
ensure_test_rpc_secret();
let mut headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
.expect("v2 compatibility headers should build");
let rejected =
verify_tonic_rpc_signature_with_policy("node-a:9000", "/node_service.NodeService/Ping", &headers, false, true, false)
.expect_err("strict replay scope must reject stripped new metadata");
assert_eq!(rejected.to_string(), "RPC replay-scoped authentication required");
headers.insert(
RPC_BOOT_EPOCH_CHALLENGE_HEADER,
HeaderValue::from_str(&Uuid::new_v4().to_string()).expect("UUID header"),
);
assert!(
verify_tonic_rpc_signature_with_policy("node-a:9000", "/node_service.NodeService/Ping", &headers, false, true, true,)
.is_ok(),
"only the signed Ping bootstrap may obtain a new server epoch in strict mode"
);
}
#[test]
fn boot_epoch_response_proof_binds_audience_challenge_and_epoch() {
ensure_test_rpc_secret();
let challenge = Uuid::new_v4();
let headers = tonic_boot_epoch_response_headers("node-a:9000", challenge).expect("proof headers should build");
let epoch =
verify_tonic_boot_epoch_response("node-a:9000", challenge, &headers).expect("matching proof headers should verify");
assert_eq!(epoch, tonic_rpc_boot_epoch());
assert!(verify_tonic_boot_epoch_response("node-b:9000", challenge, &headers).is_err());
assert!(verify_tonic_boot_epoch_response("node-a:9000", Uuid::new_v4(), &headers).is_err());
}
#[test]
fn malformed_v2_auth_does_not_downgrade_to_valid_legacy_signature() {
ensure_test_rpc_secret();
+10 -5
View File
@@ -26,13 +26,18 @@ pub(crate) mod runtime_sources;
pub use background_monitor::shutdown_background_monitors;
pub(crate) use background_monitor::spawn_background_monitor;
pub use client::{
TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth,
};
// Re-exported through `api::rpc`; not every item is consumed inside this crate.
#[allow(unused_imports)]
pub use http_auth::{
TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience,
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof,
verify_ns_scanner_capability, verify_rpc_signature, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, sign_ns_scanner_capability,
sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, verify_ns_scanner_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::cluster::rpc::client::{
TonicInterceptor, embedded_tonic_status, gen_tonic_signature_interceptor, heal_control_time_out_client,
AuthenticatedChannel, TonicInterceptor, embedded_tonic_status, gen_tonic_signature_interceptor, heal_control_time_out_client,
is_network_like_status, message_has_network_needle, node_service_time_out_client, tier_mutation_control_time_out_client,
};
use crate::cluster::rpc::{set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, verify_tonic_rpc_response_proof};
@@ -58,7 +58,7 @@ use std::{
collections::HashMap,
io::Cursor,
sync::{
Arc,
Arc, Weak,
atomic::{AtomicBool, Ordering},
},
time::SystemTime,
@@ -66,7 +66,6 @@ use std::{
use tokio::{net::TcpStream, time::Duration};
use tonic::Request;
use tonic::service::interceptor::InterceptedService;
use tonic::transport::Channel;
use tracing::{debug, info, warn};
use uuid::Uuid;
@@ -222,6 +221,21 @@ fn validate_heal_control_response_proof(canonical_response: &[u8], proof: &[u8])
.map_err(|_| Error::other("peer returned an invalid heal control response proof"))
}
fn decode_remote_version_state_capability(expected_member: &str, result: &[u8]) -> Result<Uuid> {
let (topology_member, process_epoch) = rustfs_protos::decode_remote_version_state_capability(result).map_err(Error::other)?;
if topology_member != expected_member {
return Err(Error::other(
"peer returned a remote version state capability for a different topology member",
));
}
let server_epoch =
Uuid::from_slice(process_epoch).map_err(|_| Error::other("peer returned an invalid remote version state epoch"))?;
if server_epoch.is_nil() {
return Err(Error::other("peer returned a nil remote version state epoch"));
}
Ok(server_epoch)
}
#[derive(Clone, Debug)]
pub struct PeerLiveEventsBatch {
pub events: Vec<u8>,
@@ -233,6 +247,7 @@ pub struct PeerLiveEventsBatch {
pub struct PeerRestClient {
pub host: XHost,
pub grid_host: String,
topology_member: String,
offline: Arc<AtomicBool>,
recovery_running: Arc<AtomicBool>,
}
@@ -325,14 +340,54 @@ impl PeerRestClient {
}
pub fn new(host: XHost, grid_host: String) -> Self {
let topology_member = host.to_string();
Self {
host,
grid_host,
topology_member,
offline: Arc::new(AtomicBool::new(false)),
recovery_running: Arc::new(AtomicBool::new(false)),
}
}
fn parse_topology_host(peer_host_port: &str, grid_host: &str) -> Result<XHost> {
let url = url::Url::parse(grid_host).map_err(|_| Error::other("peer grid host is not a valid URL"))?;
if !matches!(url.scheme(), "http" | "https")
|| !url.username().is_empty()
|| url.password().is_some()
|| url.query().is_some()
|| url.fragment().is_some()
|| url.path() != "/"
{
return Err(Error::other("peer grid host has an invalid URL shape"));
}
let url_host = url.host().ok_or_else(|| Error::other("peer grid host is missing a host"))?;
let topology_host = match url.port() {
Some(port) => format!("{url_host}:{port}"),
None => url_host.to_string(),
};
let explicit_port = url.port();
let name = match url_host {
url::Host::Domain(domain) => domain.to_string(),
url::Host::Ipv4(address) => address.to_string(),
url::Host::Ipv6(address) if explicit_port.is_none() => format!("[{address}]"),
url::Host::Ipv6(address) => address.to_string(),
};
let port = url
.port_or_known_default()
.filter(|port| *port > 0)
.ok_or_else(|| Error::other("peer grid host is missing a valid port"))?;
let host = XHost {
name,
port,
is_port_set: explicit_port.is_some(),
};
if topology_host != peer_host_port {
return Err(Error::other("peer topology host does not match its grid URL"));
}
Ok(host)
}
fn build_clients_from_slots(
slots: Vec<(String, Option<String>, bool)>,
) -> (Vec<Option<Self>>, Vec<Option<Self>>, Vec<String>) {
@@ -346,10 +401,14 @@ impl PeerRestClient {
}
let client = match grid_host {
Some(grid_host) => match XHost::try_from(peer_host_port.clone()) {
Ok(host) => Some(PeerRestClient::new(host, grid_host)),
Some(grid_host) => match Self::parse_topology_host(&peer_host_port, &grid_host) {
Ok(host) => {
let mut client = PeerRestClient::new(host, grid_host);
client.topology_member = peer_host_port.clone();
Some(client)
}
Err(err) => {
warn!(peer = %peer_host_port, "Xhost parse failed while constructing peer client: {err:?}");
warn!(peer = %peer_host_port, "peer topology host parse failed while constructing peer client: {err:?}");
None
}
},
@@ -390,7 +449,7 @@ impl PeerRestClient {
(remote, all, remote_topology_hosts)
}
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
if self.offline.load(Ordering::Acquire) {
self.mark_offline_and_spawn_recovery();
return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
@@ -411,7 +470,7 @@ impl PeerRestClient {
&self,
) -> Result<
rustfs_protos::proto_gen::node_service::heal_control_service_client::HealControlServiceClient<
InterceptedService<Channel, TonicInterceptor>,
InterceptedService<AuthenticatedChannel, TonicInterceptor>,
>,
> {
if self.offline.load(Ordering::Acquire) {
@@ -432,7 +491,7 @@ impl PeerRestClient {
async fn get_tier_mutation_control_client(
&self,
) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
if self.offline.load(Ordering::Acquire) {
self.mark_offline_and_spawn_recovery();
return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
@@ -498,9 +557,8 @@ impl PeerRestClient {
}
let grid_host = self.grid_host.clone();
let offline = Arc::clone(&self.offline);
let recovery_running = Arc::clone(&self.recovery_running);
let span = Self::recovery_monitor_span(&grid_host);
let offline = Arc::downgrade(&self.offline);
let recovery_running = Arc::downgrade(&self.recovery_running);
// The offline flag and its recovery are the silent half of
// rustfs/backlog#888: log the monitor's start and its success so an
// "offline then back" episode leaves a trace on the observing node.
@@ -509,13 +567,34 @@ impl PeerRestClient {
grid_host = %self.grid_host,
"peer RPC connection marked offline after a network-like failure; starting background recovery monitor"
);
drop(Self::spawn_recovery_monitor(grid_host, offline, recovery_running));
}
fn spawn_recovery_monitor(
grid_host: String,
offline: Weak<AtomicBool>,
recovery_running: Weak<AtomicBool>,
) -> tokio::task::JoinHandle<()> {
let span = Self::recovery_monitor_span(&grid_host);
super::spawn_background_monitor(span, async move {
let mut delay = get_drive_active_check_interval();
let connect_timeout = get_drive_active_check_timeout();
for attempt in 1..=PEER_REST_RECOVERY_MAX_ATTEMPTS {
if offline.strong_count() == 0 || recovery_running.strong_count() == 0 {
return;
}
tokio::time::sleep(delay).await;
if offline.strong_count() == 0 || recovery_running.strong_count() == 0 {
return;
}
if Self::perform_connectivity_check(&grid_host, connect_timeout).await.is_ok() {
let Some(offline) = offline.upgrade() else {
return;
};
let Some(recovery_running) = recovery_running.upgrade() else {
return;
};
offline.store(false, Ordering::Release);
recovery_running.store(false, Ordering::Release);
info!(
@@ -535,8 +614,10 @@ impl PeerRestClient {
attempts = PEER_REST_RECOVERY_MAX_ATTEMPTS,
"peer recovery monitor reached max attempts; will retry on next request"
);
recovery_running.store(false, Ordering::Release);
});
if let Some(recovery_running) = recovery_running.upgrade() {
recovery_running.store(false, Ordering::Release);
}
})
}
#[cfg(test)]
@@ -1154,6 +1235,15 @@ impl PeerRestClient {
validate_heal_control_capability_proof(&canonical_ack, &proof)
}
pub async fn probe_remote_version_state(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
let probe = rustfs_protos::remote_version_state_capability_probe(Uuid::new_v4().as_bytes());
let result = self
.heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
.await?;
let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
Ok((self.topology_member.clone(), epoch))
}
pub async fn load_bucket_metadata(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
self.finalize_result(
async {
@@ -1777,9 +1867,13 @@ fn tier_config_reload_status_outcome(status: tonic::Status) -> TierConfigReloadO
mod tests {
use super::*;
use crate::config::com::STORAGE_CLASS_SUB_SYS;
use crate::layout::{disks_layout::DisksLayout, endpoints::SetupType};
use rustfs_config::{ENV_KUBERNETES_SERVICE_HOST, ENV_LOCAL_ENDPOINT_HOST, ENV_STARTUP_TOPOLOGY_WAIT_MODE};
use serde_json::Value;
use serial_test::serial;
use std::io::{self, Write};
use std::sync::{Arc, Mutex};
use temp_env::async_with_vars;
use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
#[test]
@@ -1897,30 +1991,115 @@ mod tests {
fn build_clients_from_slots_preserves_missing_remote_topology_slots() {
let slots = vec![
("127.0.0.1:9000".to_string(), None, true),
("127.0.0.1:9001".to_string(), Some("http://127.0.0.1:9001".to_string()), false),
(
"rustfs-1.invalid:9001".to_string(),
Some("http://rustfs-1.invalid:9001".to_string()),
false,
),
("rustfs-2.invalid".to_string(), Some("http://rustfs-2.invalid".to_string()), false),
("127.0.0.1:notaport".to_string(), Some("http://127.0.0.1:notaport".to_string()), false),
("127.0.0.1:9003".to_string(), None, false),
];
let (remote, all, remote_topology_hosts) = PeerRestClient::build_clients_from_slots(slots);
assert_eq!(remote.len(), 3, "local node is excluded but remote slots are not compacted away");
assert_eq!(all.len(), 4, "all slots preserve the sorted cluster topology shape");
assert_eq!(remote.len(), 4, "local node is excluded but remote slots are not compacted away");
assert_eq!(all.len(), 5, "all slots preserve the sorted cluster topology shape");
assert_eq!(
remote_topology_hosts,
vec![
"127.0.0.1:9001".to_string(),
"rustfs-1.invalid:9001".to_string(),
"rustfs-2.invalid".to_string(),
"127.0.0.1:notaport".to_string(),
"127.0.0.1:9003".to_string()
]
);
assert!(remote[0].is_some(), "valid remote peer should get a client");
assert!(remote[1].is_none(), "unparseable remote peer should remain observable as a missing slot");
assert!(remote[2].is_none(), "missing grid host should remain observable as a missing slot");
let unresolved = remote[0]
.as_ref()
.expect("temporarily unresolved remote peer should retain a client");
assert_eq!(unresolved.host.to_string(), "rustfs-1.invalid:9001");
let default_port = remote[1]
.as_ref()
.expect("temporarily unresolved scheme-default remote peer should retain a client");
assert_eq!(default_port.host.to_string(), "rustfs-2.invalid");
assert_eq!(default_port.host.port, 80);
assert!(!default_port.host.is_port_set);
assert!(remote[2].is_none(), "unparseable remote peer should remain observable as a missing slot");
assert!(remote[3].is_none(), "missing grid host should remain observable as a missing slot");
assert!(all[0].is_none(), "local node is represented by the local server_info row");
assert!(all[1].is_some());
assert!(all[2].is_none());
assert!(all[2].is_some());
assert!(all[3].is_none());
assert!(all[4].is_none());
}
#[test]
fn topology_host_parser_preserves_names_and_bracketed_ipv6() {
let domain = PeerRestClient::parse_topology_host("rustfs-1.invalid", "https://rustfs-1.invalid")
.expect("unresolved HTTPS topology host should parse without DNS");
assert_eq!(domain.to_string(), "rustfs-1.invalid");
assert_eq!(domain.port, 443);
assert!(!domain.is_port_set);
let ipv6 = PeerRestClient::parse_topology_host("[2001:db8::1]:9000", "http://[2001:db8::1]:9000")
.expect("bracketed IPv6 topology host should parse without changing its identity");
assert_eq!(ipv6.to_string(), "[2001:db8::1]:9000");
let default_port_ipv6 = PeerRestClient::parse_topology_host("[2001:db8::2]", "http://[2001:db8::2]")
.expect("scheme-default IPv6 topology host should parse without DNS");
assert_eq!(default_port_ipv6.to_string(), "[2001:db8::2]");
assert_eq!(default_port_ipv6.port, 80);
assert!(!default_port_ipv6.is_port_set);
assert!(PeerRestClient::parse_topology_host("peer.invalid:0", "http://peer.invalid:0").is_err());
assert!(PeerRestClient::parse_topology_host("peer-a.invalid:9000", "http://peer-b.invalid:9000").is_err());
assert!(PeerRestClient::parse_topology_host("peer.invalid:9000", "http://peer.invalid:9000/unexpected").is_err());
}
#[tokio::test]
#[serial]
async fn unresolved_default_port_endpoint_topology_retains_all_peer_clients() {
let volumes = (0..4)
.map(|index| format!("http://rustfs-{index}.invalid:80/data{index}"))
.collect::<Vec<_>>();
let layout = DisksLayout::from_volumes(&volumes).expect("distributed default-port topology should parse");
async_with_vars(
[
(ENV_STARTUP_TOPOLOGY_WAIT_MODE, Some("orchestrated")),
(ENV_LOCAL_ENDPOINT_HOST, Some("rustfs-0.invalid")),
(ENV_KUBERNETES_SERVICE_HOST, None),
],
async {
let (server_pools, setup_type) = EndpointServerPools::create_server_endpoints("0.0.0.0:80", &layout)
.await
.expect("explicit local identity should avoid peer DNS during endpoint construction");
assert_eq!(setup_type, SetupType::DistErasure);
let (remote, all, remote_topology_hosts) =
PeerRestClient::build_clients_from_slots(server_pools.peer_grid_host_slots_sorted());
assert_eq!(remote.len(), 3);
assert!(
remote.iter().all(Option::is_some),
"unresolved remote peers must retain reconnectable clients"
);
assert_eq!(all.len(), 4);
assert_eq!(all.iter().filter(|client| client.is_none()).count(), 1);
assert_eq!(remote_topology_hosts.len(), 3);
assert!(
remote_topology_hosts.iter().all(|host| !host.contains(':')),
"scheme-default ports must preserve the legacy topology identity"
);
assert!(
remote
.iter()
.flatten()
.all(|client| client.host.port == 80 && !client.host.is_port_set),
"scheme-default peers must retain the effective dial port"
);
},
)
.await;
}
#[test]
@@ -2470,6 +2649,22 @@ mod tests {
}
}
#[test]
fn remote_version_state_capability_decoder_fails_closed() {
let epoch = Uuid::new_v4();
let result = rustfs_protos::encode_remote_version_state_capability("node-a:9000", epoch.as_bytes())
.expect("small capability response should encode");
assert_eq!(
decode_remote_version_state_capability("node-a:9000", &result).expect("valid epoch should decode"),
epoch
);
assert!(decode_remote_version_state_capability("node-b:9000", &result).is_err());
assert!(decode_remote_version_state_capability("node-a:9000", &result[..result.len() - 1]).is_err());
let nil = rustfs_protos::encode_remote_version_state_capability("node-a:9000", Uuid::nil().as_bytes())
.expect("small capability response should encode");
assert!(decode_remote_version_state_capability("node-a:9000", &nil).is_err());
}
struct TierMutationResponseFixture<'a> {
version: u32,
phase: TierMutationRpcPhase,
@@ -2694,6 +2889,31 @@ mod tests {
assert!(!client.offline.load(Ordering::Acquire));
}
#[tokio::test(start_paused = true)]
async fn dropped_peer_client_releases_and_stops_its_recovery_monitor() {
let client = test_peer_client();
client.offline.store(true, Ordering::Release);
client.recovery_running.store(true, Ordering::Release);
let offline = Arc::downgrade(&client.offline);
let recovery_running = Arc::downgrade(&client.recovery_running);
let handle = PeerRestClient::spawn_recovery_monitor(client.grid_host.clone(), offline.clone(), recovery_running.clone());
let started = tokio::time::Instant::now();
drop(client);
assert!(offline.upgrade().is_none(), "detached recovery must not retain offline state");
assert!(
recovery_running.upgrade().is_none(),
"detached recovery must not retain its running state"
);
handle.await.expect("recovery monitor should not panic");
assert_eq!(
tokio::time::Instant::now(),
started,
"recovery monitor should stop before advancing to its first delayed probe"
);
}
#[tokio::test]
async fn peer_rest_client_finalize_result_keeps_online_for_app_errors_mentioning_unavailable() {
// Regression: application error text containing "unavailable" (a
@@ -2741,6 +2961,11 @@ mod tests {
.with_span_list(true),
);
let _guard = tracing::subscriber::set_default(subscriber);
// The `recovery-monitor` callsite is shared with the production
// `mark_offline_and_spawn_recovery` path that sibling tests exercise from
// subscriber-less threads; without this the span can be cached as
// `Interest::never()` and silently degrade to `Span::none()`.
let _callsite_pin = crate::test_tracing::pin_callsite_interest_for_test();
let client = test_peer_client();
let span = tracing::info_span!("request-span", request_id = "req-peer-rest");
+180 -13
View File
@@ -14,7 +14,8 @@
use crate::bucket::metadata_sys;
use crate::cluster::rpc::client::{
TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error, node_service_time_out_client,
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_time_out_client,
};
use crate::cluster::rpc::set_tonic_mutation_body_digest;
use crate::disk::error::DiskError;
@@ -40,16 +41,84 @@ use rustfs_protos::proto_gen::node_service::node_service_client::NodeServiceClie
use rustfs_protos::proto_gen::node_service::{
DeleteBucketRequest, GetBucketInfoRequest, HealBucketRequest, ListBucketRequest, MakeBucketRequest,
};
#[cfg(test)]
use std::sync::{
Mutex as StdMutex,
atomic::{AtomicBool, Ordering},
};
use std::{collections::HashMap, fmt::Debug, sync::Arc, time::Duration};
#[cfg(test)]
use tokio::sync::Notify;
use tokio::{net::TcpStream, sync::RwLock, time};
use tokio_util::sync::CancellationToken;
use tonic::Request;
use tonic::service::interceptor::InterceptedService;
use tonic::transport::Channel;
use tracing::{debug, info, warn};
type Client = Arc<Box<dyn PeerS3Client>>;
#[cfg(test)]
#[derive(Default)]
pub(crate) struct DeleteBucketEmptyScanBarrier {
arrived: AtomicBool,
arrived_notify: Notify,
released: AtomicBool,
release_notify: Notify,
}
#[cfg(test)]
impl DeleteBucketEmptyScanBarrier {
pub(crate) async fn wait_until_paused(&self) {
loop {
let notified = self.arrived_notify.notified();
if self.arrived.load(Ordering::Acquire) {
return;
}
notified.await;
}
}
pub(crate) fn release(&self) {
self.released.store(true, Ordering::Release);
self.release_notify.notify_waiters();
}
async fn pause(&self) {
self.arrived.store(true, Ordering::Release);
self.arrived_notify.notify_waiters();
loop {
let notified = self.release_notify.notified();
if self.released.load(Ordering::Acquire) {
return;
}
notified.await;
}
}
}
#[cfg(test)]
static DELETE_BUCKET_EMPTY_SCAN_BARRIER: StdMutex<Option<Arc<DeleteBucketEmptyScanBarrier>>> = StdMutex::new(None);
#[cfg(test)]
pub(crate) fn install_delete_bucket_empty_scan_barrier() -> Arc<DeleteBucketEmptyScanBarrier> {
let barrier = Arc::new(DeleteBucketEmptyScanBarrier::default());
*DELETE_BUCKET_EMPTY_SCAN_BARRIER
.lock()
.expect("empty scan barrier lock should not be poisoned") = Some(barrier.clone());
barrier
}
#[cfg(test)]
async fn pause_after_delete_bucket_empty_scan() {
let barrier = DELETE_BUCKET_EMPTY_SCAN_BARRIER
.lock()
.expect("empty scan barrier lock should not be poisoned")
.take();
if let Some(barrier) = barrier {
barrier.pause().await;
}
}
#[derive(Clone, Debug)]
pub struct ScannerBucketListing {
pub buckets: Vec<BucketInfo>,
@@ -650,24 +719,22 @@ impl PeerS3Client for LocalPeerS3Client {
return Err(Error::ErasureWriteQuorum);
}
let force = if opts.force_if_empty && !opts.force {
if opts.force_if_empty && !opts.force {
for disk in local_disks.iter() {
if has_xlmeta_files(&disk.path().join(bucket)).await.map_err(Error::Io)? {
return Err(Error::VolumeNotEmpty);
}
}
true
} else {
opts.force
};
#[cfg(test)]
pause_after_delete_bucket_empty_scan().await;
}
let mut futures = Vec::with_capacity(local_disks.len());
for disk in local_disks.iter() {
// Non-force delete refuses a non-empty bucket (VolumeNotEmpty), which
// the recreate loop below turns into BucketNotEmpty; only an explicit
// force delete removes recursively (backlog#799 B1).
futures.push(disk.delete_volume(bucket, force));
// `force_if_empty` is validation-only. Passing it as force would let
// a PutObject committed after the scan be removed recursively.
futures.push(disk.delete_volume(bucket, opts.force));
}
let results = join_all(futures).await;
@@ -730,6 +797,15 @@ pub struct RemotePeerS3Client {
}
impl RemotePeerS3Client {
fn encode_delete_bucket_options(opts: &DeleteBucketOptions) -> Result<String> {
let mut remote_opts = opts.clone();
// Older peers promote `force_if_empty` to recursive force after their
// metadata scan. Keep this coordinator-only hint off the wire so a
// mixed-version delete fails closed on non-empty directory remnants.
remote_opts.force_if_empty = false;
serde_json::to_string(&remote_opts).map_err(Into::into)
}
fn recovery_monitor_span(addr: &str) -> tracing::Span {
tracing::info_span!(
"recovery-monitor",
@@ -756,7 +832,7 @@ impl RemotePeerS3Client {
client
}
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
node_service_time_out_client(&self.addr, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
.await
.map_err(|err| Error::other(format!("can not get client, err: {err}")))
@@ -1040,7 +1116,7 @@ impl PeerS3Client for RemotePeerS3Client {
async fn delete_bucket(&self, bucket: &str, opts: &DeleteBucketOptions) -> Result<()> {
self.execute_with_timeout(
|| async {
let options = serde_json::to_string(opts)?;
let options = Self::encode_delete_bucket_options(opts)?;
let mut client = self.get_client().await?;
let mut request = Request::new(DeleteBucketRequest {
@@ -1414,6 +1490,49 @@ mod tests {
}
}
#[test]
fn remote_delete_bucket_options_fail_closed_for_legacy_peers() {
let encoded = RemotePeerS3Client::encode_delete_bucket_options(&DeleteBucketOptions {
no_lock: true,
no_recreate: true,
force_if_empty: true,
..Default::default()
})
.expect("remote delete options should serialize");
let legacy_opts: DeleteBucketOptions =
serde_json::from_str(&encoded).expect("legacy peer should decode remote delete options");
assert!(legacy_opts.no_lock);
assert!(legacy_opts.no_recreate);
assert!(!legacy_opts.force);
assert!(!legacy_opts.force_if_empty);
let legacy_recursive_force = if legacy_opts.force_if_empty && !legacy_opts.force {
true
} else {
legacy_opts.force
};
assert!(
!legacy_recursive_force,
"legacy peer must not upgrade empty-only delete to recursive force"
);
}
#[test]
fn remote_delete_bucket_options_preserve_explicit_force() {
let encoded = RemotePeerS3Client::encode_delete_bucket_options(&DeleteBucketOptions {
force: true,
force_if_empty: true,
..Default::default()
})
.expect("remote force-delete options should serialize");
let remote_opts: DeleteBucketOptions =
serde_json::from_str(&encoded).expect("remote peer should decode force-delete options");
assert!(remote_opts.force);
assert!(!remote_opts.force_if_empty);
}
#[tokio::test]
async fn test_execute_with_timeout_marks_remote_peer_faulty_on_network_like_error() {
let client = test_remote_peer("http://peer-network-error:9000");
@@ -1571,6 +1690,54 @@ mod tests {
reset_local_disk_test_state().await;
}
#[tokio::test]
#[serial]
async fn local_peer_force_if_empty_preserves_unclassified_file_in_selected_pool() {
reset_local_disk_test_state().await;
let temp_dir = TempDir::new().expect("create temp dir for empty-only delete regression");
let disks = init_test_local_disks_for_pools(
&temp_dir,
&[(0, 1), (1, 1)],
"local-peer-force-if-empty-preserves-unclassified-file",
)
.await;
let bucket = "empty-only-delete-bucket";
let marker = "object/commit-marker";
let data = bytes::Bytes::from_static(b"committed object data");
disks[1]
.make_volume(bucket)
.await
.expect("bucket should be created in the selected pool");
disks[1]
.write_all(bucket, marker, data.clone())
.await
.expect("unclassified committed file should be written");
let err = LocalPeerS3Client::new_with_local_disks(None, Some(vec![1]), disks.clone())
.delete_bucket(
bucket,
&DeleteBucketOptions {
force_if_empty: true,
..Default::default()
},
)
.await
.expect_err("empty-only delete must not recursively remove an unclassified file");
assert_eq!(err, Error::VolumeNotEmpty);
assert_eq!(
disks[1]
.read_all(bucket, marker)
.await
.expect("unclassified committed file should be preserved"),
data
);
reset_local_disk_test_state().await;
}
#[tokio::test]
#[serial]
async fn heal_bucket_local_recreates_missing_bucket_volumes() {
+45 -7
View File
@@ -13,8 +13,8 @@
// limitations under the License.
use crate::cluster::rpc::client::{
TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error, node_service_time_out_client,
node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
};
use crate::cluster::rpc::http_auth::set_tonic_canonical_body_digest;
use crate::cluster::rpc::internode_data_transport::{
@@ -71,7 +71,7 @@ use tokio::{
time::timeout,
};
use tokio_util::sync::CancellationToken;
use tonic::{Code, Request, service::interceptor::InterceptedService, transport::Channel};
use tonic::{Code, Request, service::interceptor::InterceptedService};
use tracing::{debug, trace, warn};
use uuid::Uuid;
@@ -1083,7 +1083,7 @@ impl RemoteDisk {
internode_offline_bypass_reason(&self.addr).map(Error::other)
}
async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
@@ -1096,7 +1096,7 @@ impl RemoteDisk {
/// Routes onto the isolated bulk channel pool so large transfers cannot head-of-line block
/// lock/health RPCs (grpc-optimization P1). Falls back to the control channel when isolation
/// is disabled.
async fn get_bulk_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
async fn get_bulk_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
@@ -3005,6 +3005,7 @@ mod tests {
use crate::cluster::rpc::internode_data_transport::{InternodeDataTransportCapabilities, TcpHttpInternodeDataTransport};
use crate::runtime::sources as runtime_sources;
use serde_json::Value;
use serial_test::serial;
use std::io::{self as std_io, Write};
use std::pin::Pin;
use std::sync::{Arc, Mutex, Mutex as StdMutex, Once};
@@ -3018,6 +3019,20 @@ mod tests {
static INIT: Once = Once::new();
// `#[serial(internode_metrics)]` marks every test that observes
// `global_internode_metrics()`. Those counters are a process-wide singleton:
// some of these tests snapshot a counter, run one decode, and assert on the
// delta, while others deliberately record decode errors or call
// `reset_internode_metrics_for_test()`. Run concurrently in one process they
// corrupt each other's deltas — a sibling's error bumps the "no decode error"
// assertion off zero, and a sibling's reset can drive an `after > before`
// assertion backwards.
//
// The marker only takes effect under the `cargo test` fallback; nextest
// already isolates each test in its own process, so every test there gets its
// own copy of the counters (see `docs/testing/README.md`). Any new test that
// reads or mutates the global internode metrics belongs in this group.
#[test]
fn snapshot_lease_response_requires_current_protocol_and_valid_token() {
let token = SnapshotLeaseToken::new();
@@ -3306,6 +3321,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn read_multiple_response_decode_prefers_msgpack_payloads() {
let endpoint = sample_remote_endpoint();
let msgpack_resp = sample_read_multiple_resp("msgpack", b"binary");
@@ -3328,6 +3344,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn read_multiple_response_decode_falls_back_to_json_payloads() {
let endpoint = sample_remote_endpoint();
let json_resp = sample_read_multiple_resp("json", b"fallback");
@@ -3349,6 +3366,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn rename_data_response_accepts_legacy_json_without_decode_error() {
crate::cluster::rpc::runtime_sources::reset_internode_metrics_for_test();
let response = RenameDataResp {
@@ -3500,6 +3518,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn read_multiple_response_decode_reports_corrupt_msgpack_item() {
let endpoint = sample_remote_endpoint();
let response = ReadMultipleResponse {
@@ -3525,6 +3544,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn read_multiple_response_decode_reports_corrupt_json_item() {
let endpoint = sample_remote_endpoint();
let response = ReadMultipleResponse {
@@ -3565,6 +3585,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn batch_read_version_response_decode_prefers_msgpack_payloads() {
let endpoint = sample_remote_endpoint();
let msgpack_resp = sample_batch_read_version_resp(7, "msgpack-object", true);
@@ -3585,6 +3606,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn batch_read_version_response_rejects_invalid_success_metadata() {
let endpoint = sample_remote_endpoint();
let mut response_item = sample_batch_read_version_resp(0, "invalid-object", true);
@@ -3604,6 +3626,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn batch_read_version_response_decode_reports_corrupt_msgpack_item() {
let endpoint = sample_remote_endpoint();
let response = BatchReadVersionResponse {
@@ -3630,6 +3653,7 @@ mod tests {
}
#[test]
#[serial(internode_metrics)]
fn batch_read_version_response_decode_reports_corrupt_json_item() {
let endpoint = sample_remote_endpoint();
let response = BatchReadVersionResponse {
@@ -4419,6 +4443,7 @@ mod tests {
}
#[tokio::test]
#[serial(internode_metrics)]
async fn test_remote_disk_create_file_retries_once_on_retryable_open_write_error() {
let transport = RetryingOpenWriteInternodeDataTransport::with_steps(vec![
OpenWriteTestStep::Error(DiskError::from(rustfs_rio::new_test_internode_http_io_error(
@@ -4457,6 +4482,7 @@ mod tests {
}
#[tokio::test]
#[serial(internode_metrics)]
async fn test_remote_disk_read_file_stream_retries_once_on_retryable_open_read_error() {
// A transient reset-by-peer on a shard read during the read-after-write window must be
// absorbed by one re-dial rather than eroding read quorum (issue #2761).
@@ -4788,9 +4814,11 @@ mod tests {
async fn test_remote_disk_endpoints_with_different_schemes() {
let test_cases = vec![
("http://server:9000", "server:9000"),
("https://secure-server:443", "secure-server"), // Default HTTPS port is omitted
("http://plain-server:80", "plain-server"),
("http://plain-server", "plain-server"),
("https://secure-server:443", "secure-server"),
("http://192.168.1.100:8080", "192.168.1.100:8080"),
("https://secure-server", "secure-server"), // No port specified
("https://secure-server", "secure-server"),
];
for (url_str, expected_hostname) in test_cases {
@@ -5306,6 +5334,11 @@ mod tests {
.with_span_list(true),
);
let _guard = tracing::subscriber::set_default(subscriber);
// The `recovery-monitor` span and the monitor's own log events are
// production callsites that sibling tests exercise from subscriber-less
// threads; without this they can be cached as `Interest::never()` and go
// silently missing here.
let _callsite_pin = crate::test_tracing::pin_callsite_interest_for_test();
let endpoint = Endpoint {
url: url::Url::parse("http://127.0.0.1:59996/data").expect("endpoint URL should parse"),
@@ -5360,6 +5393,11 @@ mod tests {
.with_span_list(true),
);
let _guard = tracing::subscriber::set_default(subscriber);
// The `recovery-monitor` span and the monitor's own log events are
// production callsites that sibling tests exercise from subscriber-less
// threads; without this they can be cached as `Interest::never()` and go
// silently missing here.
let _callsite_pin = crate::test_tracing::pin_callsite_interest_for_test();
let addr = "http://127.0.0.1:59997".to_string();
let endpoint = Endpoint {
@@ -12,7 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::cluster::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
};
use crate::cluster::rpc::set_tonic_mutation_body_digest;
use async_trait::async_trait;
use bytes::Bytes;
@@ -29,7 +31,6 @@ use std::time::Duration;
use tokio::time::timeout;
use tonic::Request;
use tonic::service::interceptor::InterceptedService;
use tonic::transport::Channel;
use tracing::{debug, info, warn};
/// Remote lock client implementation
@@ -78,7 +79,7 @@ impl RemoteClient {
}
}
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
// P3-2 offline bypass (now covering the lock path too): fast-fail a peer already marked
// offline instead of paying the connect timeout, so dsync reaches quorum sooner. Does not
// change quorum; the self-healing re-probe keeps the peer recoverable.
+642 -53
View File
@@ -53,12 +53,14 @@ use std::sync::LazyLock;
use std::sync::{Arc, RwLock};
use tokio::sync::{OwnedRwLockWriteGuard, RwLock as AsyncRwLock};
use tracing::{debug, error, info, instrument, warn};
use uuid::Uuid;
pub const CONFIG_PREFIX: &str = "config";
const SERVER_CONFIG_OBJECT: &str = "config/config.json";
const CONFIG_TRANSACTION_LOCK_SUFFIX: &str = ".transaction.lock";
// Server-config lock order: SERVER_CONFIG_LOCK -> distributed namespace lock
// for SERVER_CONFIG_OBJECT. Readers and writers must never reverse this order.
// Server-config lock order: SERVER_CONFIG_LOCK -> transaction lock ->
// SERVER_CONFIG_OBJECT. Readers and writers must never reverse this order.
static SERVER_CONFIG_LOCK: LazyLock<Arc<AsyncRwLock<()>>> = LazyLock::new(|| Arc::new(AsyncRwLock::new(())));
fn config_task_join_error(operation: &'static str, error: tokio::task::JoinError) -> Error {
@@ -76,8 +78,11 @@ where
T: Send + 'static,
{
tokio::spawn(async move {
// Lock order: SERVER_CONFIG_LOCK -> namespace write lock.
// Lock order: SERVER_CONFIG_LOCK -> transaction lock -> object lock.
let _local_guard = SERVER_CONFIG_LOCK.write().await;
let transaction_lock = server_config_transaction_lock_path();
let transaction_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let _transaction_guard = transaction_lock.get_write_lock(get_lock_acquire_timeout()).await?;
let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
Ok(operation().await)
@@ -96,8 +101,11 @@ where
T: Send + 'static,
{
tokio::spawn(async move {
// Lock order: SERVER_CONFIG_LOCK -> namespace read lock.
// Lock order: SERVER_CONFIG_LOCK -> transaction lock -> object lock.
let _local_guard = SERVER_CONFIG_LOCK.read().await;
let transaction_lock = server_config_transaction_lock_path();
let transaction_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let _transaction_guard = transaction_lock.get_read_lock(get_lock_acquire_timeout()).await?;
let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
let _read_guard = namespace_lock.get_read_lock(get_lock_acquire_timeout()).await?;
Ok(operation().await)
@@ -567,6 +575,21 @@ where
}
pub async fn save_config_with_opts<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
save_config_with_opts_and_metadata(api, file, data, opts).await.map(|_| ())
}
async fn save_config_with_opts_and_metadata<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<ObjectInfo>
where
S: ObjectIO<
Error = Error,
@@ -579,11 +602,13 @@ where
>,
{
let mut put_data = PutObjReader::from_vec(data);
if let Err(err) = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
return Err(err);
match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
Ok(object_info) => Ok(object_info),
Err(err) => {
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
Err(err)
}
}
Ok(())
}
fn new_server_config() -> Config {
@@ -594,8 +619,12 @@ async fn new_and_save_server_config<S>(api: Arc<S>) -> Result<Config>
where
S: EcstoreObjectIO + StorageAdminApi + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
let snapshot = read_server_config_snapshot(api.clone()).await?;
if snapshot.object_exists() {
return Ok(snapshot.config.clone());
}
let cfg = new_server_config();
save_server_config(api, &cfg).await?;
save_server_config_snapshot(api, &cfg, &snapshot).await?;
Ok(cfg)
}
@@ -617,6 +646,10 @@ pub fn server_config_path() -> String {
SERVER_CONFIG_OBJECT.to_string()
}
fn server_config_transaction_lock_path() -> String {
format!("{}{CONFIG_TRANSACTION_LOCK_SUFFIX}", server_config_path())
}
fn storage_class_kvs_mut(cfg: &mut Config) -> &mut KVS {
let sub_cfg = cfg.0.entry(STORAGE_CLASS_SUB_SYS.to_string()).or_insert_with(|| {
let mut section = HashMap::new();
@@ -819,6 +852,9 @@ fn apply_external_scalar_config_map(
let Some(config_value) = root.get(descriptor.subsystem_key) else {
return Ok(false);
};
if descriptor.subsystem_key == HEAL_SUB_SYS && config_value.is_null() {
return Ok(false);
}
let overrides = decode_scalar_config_value(config_value, descriptor)?;
if overrides.is_empty() {
@@ -1463,21 +1499,142 @@ fn build_audit_object(cfg: &Config) -> Map<String, Value> {
build_target_object(cfg, &audit_target_descriptors())
}
fn sync_rendered_target_instance(existing: Value, rendered: Option<&Value>, valid_keys: &[&str]) -> Option<Value> {
match existing {
Value::Object(mut instance) => {
for key in valid_keys {
instance.remove(*key);
}
if let Some(Value::Object(rendered)) = rendered {
instance.extend(rendered.clone());
}
(!instance.is_empty()).then_some(Value::Object(instance))
}
Value::Array(entries) => {
let mut pending = rendered
.and_then(Value::as_object)
.map(|rendered| {
rendered
.iter()
.filter_map(|(key, value)| parse_target_scalar_value(key, value).map(|value| (key.clone(), value)))
.collect::<HashMap<_, _>>()
})
.unwrap_or_default();
let mut updated = Vec::with_capacity(entries.len().saturating_add(pending.len()));
for entry in entries {
let Some(entry_obj) = entry.as_object() else {
updated.push(entry);
continue;
};
let Some(key) = entry_obj.get("key").and_then(Value::as_str) else {
updated.push(entry);
continue;
};
if !valid_keys.contains(&key) {
updated.push(entry);
continue;
}
let Some(value) = pending.remove(key) else {
continue;
};
let mut entry_obj = entry_obj.clone();
entry_obj.insert("value".to_string(), Value::String(value));
updated.push(Value::Object(entry_obj));
}
updated.extend(rendered_scalar_config_kvs_entries(&pending));
(!updated.is_empty()).then_some(Value::Array(updated))
}
value if rendered.is_none() => Some(value),
_ => rendered.cloned(),
}
}
fn sync_rendered_target_object(
target_obj: &mut Map<String, Value>,
rendered_target: &Map<String, Value>,
descriptors: &[TargetConfigDescriptor],
) {
for descriptor in descriptors {
match rendered_target.get(descriptor.external_key) {
Some(Value::Object(v)) => {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(v.clone()));
target_obj.remove(descriptor.subsystem_key);
let existing = target_obj.remove(descriptor.external_key);
let alias = target_obj.remove(descriptor.subsystem_key);
let mut section = existing
.or(alias)
.and_then(|value| value.as_object().cloned())
.unwrap_or_default();
let rendered = rendered_target.get(descriptor.external_key).and_then(Value::as_object);
if is_target_instance_shorthand(&section, descriptor.valid_keys) {
let has_named_instances = rendered.is_some_and(|instances| instances.keys().any(|name| name != "default"));
if !has_named_instances {
if let Some(section) = sync_rendered_target_instance(
Value::Object(section),
rendered.and_then(|instances| instances.get("default")),
descriptor.valid_keys,
) {
target_obj.insert(descriptor.external_key.to_string(), section);
}
continue;
}
_ => {
target_obj.remove(descriptor.external_key);
target_obj.remove(descriptor.subsystem_key);
let mut nested = Map::new();
if let Some(default) = sync_rendered_target_instance(
Value::Object(section),
rendered.and_then(|instances| instances.get("default")),
descriptor.valid_keys,
) {
nested.insert("default".to_string(), default);
}
if let Some(rendered) = rendered {
for (instance_name, instance) in rendered {
if instance_name != "default" {
nested.insert(instance_name.clone(), instance.clone());
}
}
}
if !nested.is_empty() {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(nested));
}
continue;
}
if let Some(default_alias) = section.remove(DEFAULT_DELIMITER) {
if let Some(default) = section.get_mut("default") {
if let Some(alias) = sync_rendered_target_instance(default_alias, None, descriptor.valid_keys) {
match (default, alias) {
(Value::Object(default), Value::Object(alias)) => {
for (key, value) in alias {
default.entry(key).or_insert(value);
}
}
(Value::Array(default), Value::Array(alias)) => default.extend(alias),
_ => {}
}
}
} else {
section.insert("default".to_string(), default_alias);
}
}
let mut merged = Map::new();
for (instance_name, instance) in section {
if let Some(instance) = sync_rendered_target_instance(
instance,
rendered.and_then(|instances| instances.get(&instance_name)),
descriptor.valid_keys,
) {
merged.insert(instance_name, instance);
}
}
if let Some(rendered) = rendered {
for (instance_name, instance) in rendered {
if !merged.contains_key(instance_name) {
merged.insert(instance_name.clone(), instance.clone());
}
}
}
if !merged.is_empty() {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(merged));
}
}
}
@@ -1496,6 +1653,14 @@ fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8
Some(Value::Object(v)) => v,
_ => Map::new(),
};
for key in [
storageclass::CLASS_STANDARD,
storageclass::CLASS_RRS,
storageclass::OPTIMIZE,
storageclass::INLINE_BLOCK,
] {
sc_obj.remove(key);
}
for (k, v) in build_storageclass_object(cfg) {
sc_obj.insert(k, v);
}
@@ -1503,7 +1668,10 @@ fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8
root.remove("storage_class");
for descriptor in [scanner_config_descriptor(), heal_config_descriptor()] {
let existing = root.remove(descriptor.subsystem_key);
let mut existing = root.remove(descriptor.subsystem_key);
if descriptor.subsystem_key == HEAL_SUB_SYS && existing.as_ref().is_some_and(Value::is_null) {
existing = None;
}
let rendered = build_scalar_config_object(cfg, descriptor);
if let Some(config_value) = sync_rendered_scalar_config_value(existing, &rendered, descriptor)? {
root.insert(descriptor.subsystem_key.to_string(), config_value);
@@ -1560,6 +1728,7 @@ fn is_standard_object_server_config(data: &[u8]) -> bool {
matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty())
&& matches!(root.get("storageclass"), Some(Value::Object(_)))
&& !root.contains_key("storage_class")
&& !matches!(root.get(HEAL_SUB_SYS), Some(Value::Null))
}
fn configs_semantically_equal(lhs: &Config, rhs: &Config) -> bool {
@@ -1593,7 +1762,7 @@ where
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
> + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
if let Some(decrypt) = &decrypt_fn {
register_server_config_decrypt_fn(decrypt.clone());
@@ -1601,14 +1770,7 @@ where
let config_file = server_config_path();
match api
.get_object_info(
RUSTFS_META_BUCKET,
&config_file,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.get_object_info(RUSTFS_META_BUCKET, &config_file, &ObjectOptions::default())
.await
{
Ok(_) => {
@@ -1624,7 +1786,6 @@ where
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
@@ -1677,7 +1838,33 @@ where
}
};
match save_config(api, &config_file, normalized).await {
let snapshot = match read_server_config_snapshot(api.clone()).await {
Ok(snapshot) => snapshot,
Err(err) => {
warn!("recheck target server config failed, skip migration: {:?}", err);
return;
}
};
if snapshot.object_exists() {
debug!("server config was created while legacy migration was preparing, skip migration");
return;
}
match save_config_with_opts(
api,
&config_file,
normalized,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
{
Ok(()) => {
info!("Migrated compatible server config from legacy metadata bucket");
}
@@ -1769,8 +1956,13 @@ where
{
let config_file = server_config_path();
// Try to read the configuration file
match read_config_no_lock(api.clone(), &config_file).await {
// Try to read the configuration file.
let data = if namespace_lock_held {
read_config_no_lock(api.clone(), &config_file).await
} else {
read_config(api.clone(), &config_file).await
};
match data {
Ok(data) => read_server_config(api, &data, namespace_lock_held).await,
Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration", namespace_lock_held).await,
Err(err) => handle_config_read_error(err, &config_file),
@@ -1787,7 +1979,12 @@ where
warn!("Received empty configuration data, try to reread from '{}'", config_file);
// Try to read the configuration again
match read_config_no_lock(api.clone(), &config_file).await {
let data = if namespace_lock_held {
read_config_no_lock(api.clone(), &config_file).await
} else {
read_config(api.clone(), &config_file).await
};
match data {
Ok(cfg_data) => {
let cfg = decode_persisted_server_config(&cfg_data)?;
return Ok(cfg.merge());
@@ -2036,11 +2233,16 @@ pub struct ServerConfigSnapshot {
raw: Option<Vec<u8>>,
seed: Option<Vec<u8>>,
etag: Option<String>,
generation: Option<Uuid>,
_local_guard: OwnedRwLockWriteGuard<()>,
_guard: rustfs_lock::NamespaceLockGuard,
}
impl ServerConfigSnapshot {
pub fn object_exists(&self) -> bool {
self.raw.is_some()
}
pub fn ensure_lock_held(&self) -> Result<()> {
if self._guard.is_lock_lost() {
return Err(Error::other("server config transaction lock was lost"));
@@ -2051,12 +2253,34 @@ impl ServerConfigSnapshot {
pub fn is_lock_lost(&self) -> bool {
self._guard.is_lock_lost()
}
pub fn generation(&self) -> Option<Uuid> {
self.generation
}
}
/// Read a server config transaction snapshot while holding the same local and
/// distributed write locks used by every other server-config writer. Internal
/// reads and the later conditional write use no-lock object I/O; the guards
/// remain live until the snapshot is dropped.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ServerConfigSaveResult {
persisted: bool,
generation: Option<Uuid>,
}
impl ServerConfigSaveResult {
pub fn persisted(&self) -> bool {
self.persisted
}
pub fn generation(&self) -> Option<Uuid> {
self.generation
}
}
/// Read a server config transaction snapshot while holding a dedicated
/// transaction lock. The config object's normal namespace lock remains
/// available to fence reads and the conditional write at commit time.
/// The transaction guard remains live until the snapshot is dropped,
/// serializing persistence and history ordering across admin nodes. Runtime
/// state is reloaded from the durable object after this guard is released.
pub async fn read_server_config_snapshot<S>(api: Arc<S>) -> Result<ServerConfigSnapshot>
where
S: ObjectIO<
@@ -2071,12 +2295,10 @@ where
{
let config_file = server_config_path();
let local_guard = SERVER_CONFIG_LOCK.clone().write_owned().await;
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &config_file).await?;
let transaction_lock = server_config_transaction_lock_path();
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let guard = lock.get_write_lock(get_lock_acquire_timeout()).await?;
let read_options = ObjectOptions {
no_lock: true,
..Default::default()
};
let read_options = ObjectOptions::default();
match read_config_with_metadata_inner(api, &config_file, &read_options, true).await {
Ok((raw, object_info)) => {
let (config, seed) = decode_persisted_server_config_with_seed(&raw)?;
@@ -2085,6 +2307,7 @@ where
raw: Some(raw),
seed: Some(seed),
etag: object_info.etag,
generation: object_info.data_dir.filter(|generation| !generation.is_nil()),
_local_guard: local_guard,
_guard: guard,
})
@@ -2094,6 +2317,7 @@ where
raw: None,
seed: None,
etag: None,
generation: None,
_local_guard: local_guard,
_guard: guard,
}),
@@ -2108,6 +2332,27 @@ where
/// lock, so a concurrent update or transaction lease loss cannot commit an
/// unfenced overwrite.
pub async fn save_server_config_snapshot<S>(api: Arc<S>, cfg: &Config, snapshot: &ServerConfigSnapshot) -> Result<bool>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
save_server_config_snapshot_with_generation(api, cfg, snapshot)
.await
.map(|result| result.persisted())
}
pub async fn save_server_config_snapshot_with_generation<S>(
api: Arc<S>,
cfg: &Config,
snapshot: &ServerConfigSnapshot,
) -> Result<ServerConfigSaveResult>
where
S: ObjectIO<
Error = Error,
@@ -2126,13 +2371,19 @@ where
&& configs_semantically_equal(&snapshot.config, cfg)
{
debug!("server config unchanged and already in standard object shape, skip write");
return Ok(false);
return Ok(ServerConfigSaveResult {
persisted: false,
generation: snapshot.generation(),
});
}
let data = encode_server_config_blob(cfg, snapshot.seed.as_deref())?;
if snapshot.raw.as_deref().is_some_and(|current| current == data.as_slice()) {
debug!("server config bytes unchanged after encode, skip write");
return Ok(false);
return Ok(ServerConfigSaveResult {
persisted: false,
generation: snapshot.generation(),
});
}
let http_preconditions = if snapshot.raw.is_some() {
@@ -2152,19 +2403,22 @@ where
}
};
save_config_with_opts(
snapshot.ensure_lock_held()?;
let object_info = save_config_with_opts_and_metadata(
api,
&config_file,
data,
&ObjectOptions {
max_parity: true,
no_lock: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await?;
Ok(true)
Ok(ServerConfigSaveResult {
persisted: true,
generation: object_info.data_dir.filter(|generation| !generation.is_nil()),
})
}
/// Saves the server config while an upper layer holds the namespace write
@@ -2301,8 +2555,9 @@ mod tests {
use super::{
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
lookup_configs, read_config, read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate,
read_server_config_snapshot, save_server_config, save_server_config_snapshot, server_config_path, storage_class_kvs_mut,
lookup_configs, new_and_save_server_config, read_config, read_config_preserve_empty, read_config_with_metadata,
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, storage_class_kvs_mut,
};
use crate::config::{audit, heal, notify, oidc, scanner};
use crate::disk::endpoint::Endpoint;
@@ -2311,7 +2566,9 @@ mod tests {
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
use crate::runtime::sources as runtime_sources;
use crate::set_disk::SetDisks;
use crate::storage_api_contracts::{admin::StorageAdminApi, namespace::NamespaceLocking as _, range::HTTPRangeSpec};
use crate::storage_api_contracts::{
admin::StorageAdminApi, namespace::NamespaceLocking as _, object::HTTPPreconditions, range::HTTPRangeSpec,
};
use http::HeaderMap;
use rustfs_config::audit::{AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
use rustfs_config::notify::{
@@ -3104,6 +3361,85 @@ mod tests {
);
}
#[test]
fn root_heal_null_decodes_as_no_override_and_is_canonicalized_on_save() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"heal":null,
"future_root":{"mode":"keep"},
"openid":{"default":{
"config_url":"https://issuer.example/.well-known/openid-configuration",
"client_id":"console",
"client_secret":"oidc-secret",
"future_provider_control":"keep"
}},
"notify":{"webhook":{"primary":{
"enable":true,
"endpoint":"https://notify.example/hook",
"auth_token":"notify-secret",
"future_notify_control":"keep"
}}},
"logger":{"webhook":{"primary":{
"enable":true,
"endpoint":"https://audit.example/hook",
"auth_token":"audit-secret",
"future_audit_control":"keep"
}}}
}"#;
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
assert!(!is_standard_object_server_config(seed));
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
let value: Value = serde_json::from_slice(&encoded).expect("canonical config should be valid JSON");
assert!(value.get(HEAL_SUB_SYS).is_none());
assert_eq!(value["future_root"]["mode"].as_str(), Some("keep"));
assert_eq!(value["openid"]["default"]["client_secret"].as_str(), Some("oidc-secret"));
assert_eq!(value["openid"]["default"]["future_provider_control"].as_str(), Some("keep"));
assert_eq!(value["notify"]["webhook"]["primary"]["auth_token"].as_str(), Some("notify-secret"));
assert_eq!(value["notify"]["webhook"]["primary"]["future_notify_control"].as_str(), Some("keep"));
assert_eq!(value["logger"]["webhook"]["primary"]["auth_token"].as_str(), Some("audit-secret"));
assert_eq!(value["logger"]["webhook"]["primary"]["future_audit_control"].as_str(), Some("keep"));
assert!(is_standard_object_server_config(&encoded));
}
#[test]
fn invalid_scalar_and_nested_null_config_shapes_remain_rejected() {
let invalid_sections = [
r#""scanner":null"#,
r#""heal":"""#,
r#""heal":false"#,
r#""heal":0"#,
r#""heal":{"default":null}"#,
r#""heal":{"_":null}"#,
r#""heal":{"bitrot_cycle":null}"#,
r#""heal":[{"key":"bitrot_cycle","value":null}]"#,
];
for section in invalid_sections {
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
let err = decode_server_config_blob(input.as_bytes()).expect_err("invalid scalar shape must remain rejected");
assert!(
err.to_string().contains("expected"),
"invalid section {section} returned an unrelated error: {err}"
);
}
}
#[test]
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
let kvs_array =
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
}
#[test]
fn scanner_update_preserves_unknown_root_and_oidc_provider_fields() {
let seed = br#"{
@@ -3131,6 +3467,171 @@ mod tests {
assert_eq!(value["openid"]["default"]["client_id"].as_str(), Some("console"));
}
#[test]
fn storageclass_reset_removes_stale_inline_block_from_seed() {
let seed = br#"{
"version":"33",
"storageclass":{
"standard":"EC:2",
"rrs":"EC:1",
"optimize":"availability",
"inline_block":"64KiB",
"future_storage_control":"keep"
}
}"#;
let encoded = encode_server_config_blob(&Config::new(), Some(seed)).expect("storageclass reset should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let storageclass = value["storageclass"].as_object().expect("storageclass object");
assert!(storageclass.get(crate::config::storageclass::INLINE_BLOCK).is_none());
assert_eq!(storageclass["future_storage_control"].as_str(), Some("keep"));
}
#[test]
fn target_update_preserves_unknown_fields_without_restoring_removed_instances() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"primary":{
"enable":true,
"endpoint":"https://notify.example/old",
"auth_token":"notify-secret",
"future_control":"keep"
},
"removed":{"enable":true,"endpoint":"https://notify.example/removed"},
"retained":{"enable":true,"endpoint":"https://notify.example/retained","future_control":"keep"},
"enable":{"enable":true,"endpoint":"https://notify.example/named-enable","future_control":"keep"}
}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("target seed should decode");
let webhook = cfg
.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.expect("notify webhook subsystem should exist");
webhook
.get_mut("primary")
.expect("primary target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
webhook.remove("removed");
webhook.remove("retained");
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("target update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook section");
assert_eq!(webhook["primary"]["endpoint"].as_str(), Some("https://notify.example/new"));
assert_eq!(webhook["primary"]["auth_token"].as_str(), Some("notify-secret"));
assert_eq!(webhook["primary"]["future_control"].as_str(), Some("keep"));
assert!(webhook.get("removed").is_none());
assert_eq!(webhook["retained"]["future_control"].as_str(), Some("keep"));
assert!(webhook["retained"].get("enable").is_none());
assert!(webhook["retained"].get("endpoint").is_none());
assert_eq!(webhook["enable"]["endpoint"].as_str(), Some("https://notify.example/named-enable"));
assert_eq!(webhook["enable"]["future_control"].as_str(), Some("keep"));
}
#[test]
fn shorthand_target_update_preserves_shape_and_unknown_nested_fields() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"enable":true,
"endpoint":"https://notify.example/old",
"future_control":{"endpoint":"leave-untouched","mode":"keep"}
}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("shorthand target should decode");
cfg.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.and_then(|targets| targets.get_mut(DEFAULT_DELIMITER))
.expect("default webhook target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("shorthand target update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook shorthand object");
assert_eq!(webhook["endpoint"].as_str(), Some("https://notify.example/new"));
assert!(webhook.get("default").is_none());
assert_eq!(webhook["future_control"]["endpoint"].as_str(), Some("leave-untouched"));
assert_eq!(webhook["future_control"]["mode"].as_str(), Some("keep"));
let decoded = decode_server_config_blob(&encoded).expect("updated shorthand target should remain decodable");
assert_eq!(
decoded
.get_value(NOTIFY_WEBHOOK_SUB_SYS, DEFAULT_DELIMITER)
.expect("updated default webhook target")
.get(rustfs_config::WEBHOOK_ENDPOINT),
"https://notify.example/new"
);
}
#[test]
fn target_kvs_update_preserves_unknown_entries_and_attributes() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{"primary":[
{"key":"enable","value":"on","hidden_if_empty":false},
{"key":"endpoint","value":"https://notify.example/old","future_attribute":"keep-endpoint"},
{"key":"future_control","value":"keep","future_attribute":"keep-control"}
]}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("target KVS seed should decode");
cfg.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.and_then(|targets| targets.get_mut("primary"))
.expect("primary webhook target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("target KVS update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let entries = value["notify"]["webhook"]["primary"]
.as_array()
.expect("target KVS shape should be preserved");
let endpoint = entries
.iter()
.find(|entry| entry["key"].as_str() == Some(rustfs_config::WEBHOOK_ENDPOINT))
.expect("endpoint entry should remain");
let future = entries
.iter()
.find(|entry| entry["key"].as_str() == Some("future_control"))
.expect("unknown target entry should remain");
assert_eq!(endpoint["value"].as_str(), Some("https://notify.example/new"));
assert_eq!(endpoint["future_attribute"].as_str(), Some("keep-endpoint"));
assert_eq!(future["value"].as_str(), Some("keep"));
assert_eq!(future["future_attribute"].as_str(), Some("keep-control"));
}
#[test]
fn target_default_alias_is_canonicalized_without_losing_unknown_fields() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"_":{"enable":false,"endpoint":"https://notify.example/alias","future_alias":"keep"},
"default":{"enable":true,"endpoint":"https://notify.example/default","future_default":"keep"}
}}
}"#;
let cfg = decode_server_config_blob(seed).expect("dual default aliases should decode");
let expected_endpoint = cfg
.get_value(NOTIFY_WEBHOOK_SUB_SYS, DEFAULT_DELIMITER)
.expect("default webhook target should exist")
.get(rustfs_config::WEBHOOK_ENDPOINT);
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("default alias should canonicalize");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook section");
assert!(webhook.get(DEFAULT_DELIMITER).is_none());
assert_eq!(webhook["default"]["endpoint"].as_str(), Some(expected_endpoint.as_str()));
assert_eq!(webhook["default"]["future_alias"].as_str(), Some("keep"));
assert_eq!(webhook["default"]["future_default"].as_str(), Some("keep"));
}
#[test]
fn test_scanner_config_changes_are_semantically_significant() {
let baseline = Config::new();
@@ -4124,6 +4625,7 @@ mod tests {
/// What reads of the config object currently return.
enum RecoveryReadState {
Missing,
Blob(Vec<u8>),
QuorumError,
}
@@ -4136,6 +4638,7 @@ mod tests {
heal_calls: AtomicUsize,
write_calls: AtomicUsize,
last_put_no_lock: AtomicBool,
last_put_preconditions: Mutex<Option<HTTPPreconditions>>,
revision: AtomicUsize,
drive_counts: Vec<usize>,
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
@@ -4150,6 +4653,7 @@ mod tests {
heal_calls: AtomicUsize::new(0),
write_calls: AtomicUsize::new(0),
last_put_no_lock: AtomicBool::new(false),
last_put_preconditions: Mutex::new(None),
revision: AtomicUsize::new(1),
drive_counts: vec![2],
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
@@ -4206,6 +4710,7 @@ mod tests {
_opts: &ObjectOptions,
) -> Result<GetObjectReader> {
let data = match &*self.state.lock().expect("state lock poisoned") {
RecoveryReadState::Missing => return Err(Error::ConfigNotFound),
RecoveryReadState::Blob(data) => data.clone(),
RecoveryReadState::QuorumError => return Err(Error::ErasureReadQuorum),
};
@@ -4213,6 +4718,9 @@ mod tests {
size: data.len() as i64,
actual_size: data.len() as i64,
etag: Some(format!("config-{}", self.revision.load(Ordering::SeqCst))),
data_dir: Some(uuid::Uuid::from_u128(
u128::try_from(self.revision.load(Ordering::SeqCst)).expect("test revision should fit in u128"),
)),
..Default::default()
};
Ok(GetObjectReader {
@@ -4231,15 +4739,19 @@ mod tests {
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
let current_etag = format!("config-{}", self.revision.load(Ordering::SeqCst));
let object_exists = matches!(&*self.state.lock().expect("state lock poisoned"), RecoveryReadState::Blob(_));
if let Some(preconditions) = &opts.http_preconditions
&& (preconditions.if_match_value().is_some_and(|etag| etag != current_etag)
|| preconditions.if_none_match_value() == Some("*"))
&& (preconditions
.if_match_value()
.is_some_and(|etag| !object_exists || etag != current_etag)
|| (object_exists && preconditions.if_none_match_value() == Some("*")))
{
return Err(Error::PreconditionFailed);
}
let mut body = Vec::new();
data.stream.read_to_end(&mut body).await?;
self.last_put_no_lock.store(opts.no_lock, Ordering::SeqCst);
*self.last_put_preconditions.lock().expect("preconditions lock poisoned") = opts.http_preconditions.clone();
self.write_calls.fetch_add(1, Ordering::SeqCst);
*self.state.lock().expect("state lock poisoned") = RecoveryReadState::Blob(body.clone());
let revision = self.revision.fetch_add(1, Ordering::SeqCst) + 1;
@@ -4247,6 +4759,7 @@ mod tests {
size: i64::try_from(body.len()).expect("test config should fit in i64"),
actual_size: i64::try_from(body.len()).expect("test config should fit in i64"),
etag: Some(format!("config-{revision}")),
data_dir: Some(uuid::Uuid::from_u128(u128::try_from(revision).expect("test revision should fit in u128"))),
..Default::default()
})
}
@@ -4284,10 +4797,18 @@ mod tests {
.expect("scanner-only config change should be persisted");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 1);
assert!(store.last_put_no_lock.load(Ordering::SeqCst));
assert!(!store.last_put_no_lock.load(Ordering::SeqCst));
let preconditions = store
.last_put_preconditions
.lock()
.expect("preconditions lock poisoned")
.clone()
.expect("existing config update must be conditional");
assert_eq!(preconditions.if_match_value(), Some("config-1"));
assert_eq!(preconditions.if_none_match_value(), None);
assert_eq!(
store.lock_resources.lock().expect("lock resources mutex poisoned").as_slice(),
&[server_config_path()]
&[server_config_transaction_lock_path()]
);
let decoded = read_config_without_migrate(store)
.await
@@ -4301,6 +4822,73 @@ mod tests {
);
}
#[tokio::test]
async fn server_config_snapshot_save_returns_committed_generation() {
let baseline = encode_server_config_blob(&Config::new(), None).expect("baseline config should encode");
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline), None));
let snapshot = read_server_config_snapshot(store.clone())
.await
.expect("server config snapshot");
let result = save_server_config_snapshot_with_generation(store, &config_with_scanner_cycle("61"), &snapshot)
.await
.expect("conditional config save");
assert!(result.persisted());
assert_eq!(result.generation(), Some(uuid::Uuid::from_u128(2)));
}
#[tokio::test]
async fn missing_server_config_is_created_with_if_none_match() {
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Missing, None));
let cfg = config_with_scanner_cycle("61");
save_server_config(store.clone(), &cfg)
.await
.expect("missing config should be created conditionally");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 1);
assert!(!store.last_put_no_lock.load(Ordering::SeqCst));
let preconditions = store
.last_put_preconditions
.lock()
.expect("preconditions lock poisoned")
.clone()
.expect("missing config create must be conditional");
assert_eq!(preconditions.if_match_value(), None);
assert_eq!(preconditions.if_none_match_value(), Some("*"));
let persisted = read_config_without_migrate(store)
.await
.expect("created config should reload");
assert_eq!(
persisted
.get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
.expect("persisted scanner config")
.get(SCANNER_CYCLE),
"61"
);
}
#[tokio::test]
async fn missing_config_initialization_recheck_preserves_concurrent_config() {
let existing = config_with_scanner_cycle("73");
let baseline = encode_server_config_blob(&existing, None).expect("existing config should encode");
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline), None));
let observed = new_and_save_server_config(store.clone())
.await
.expect("initialization recheck should return the config created by another writer");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 0);
assert_eq!(
observed
.get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
.expect("concurrent scanner config")
.get(SCANNER_CYCLE),
"73"
);
}
#[tokio::test]
async fn stale_server_config_snapshot_cannot_overwrite_newer_update() {
let baseline = encode_server_config_blob(&Config::new(), None).expect("baseline config should encode");
@@ -4357,7 +4945,7 @@ mod tests {
let lock = rustfs_lock::NamespaceLock::new("server-config-lease-loss".to_string(), client.clone());
let guard = lock
.lock_guard(
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, server_config_path()),
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, server_config_transaction_lock_path()),
"server-config-lease-loss",
std::time::Duration::from_secs(1),
std::time::Duration::from_millis(120),
@@ -4372,6 +4960,7 @@ mod tests {
raw: Some(baseline.clone()),
seed: None,
etag: Some("config-0".to_string()),
generation: Some(uuid::Uuid::from_u128(1)),
_local_guard: local_guard,
_guard: guard,
};
+199 -20
View File
@@ -37,7 +37,9 @@ use crate::{
runtime::instance::{InstanceContext, bootstrap_ctx},
runtime::sources as runtime_sources,
set_disk::{PreparedGetObjectMetadata, SetDisks},
store::init_format::{check_format_erasure_values, get_format_erasure_in_quorum, load_format_erasure_all, save_format_file},
store::init_format::{
check_format_erasure_values, load_format_erasure_all, save_format_file, select_format_erasure_in_quorum,
},
};
use futures::{
future::join_all,
@@ -947,7 +949,7 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
#[tracing::instrument(skip(self))]
async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
let (disks, _) = init_storage_disks_with_errors(
let (disks, init_errs) = init_storage_disks_with_errors(
&self.endpoints.endpoints,
&DiskOption {
cleanup: false,
@@ -955,15 +957,36 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
},
)
.await;
let (formats, errs) = load_format_erasure_all(&disks, true).await;
let (formats, mut errs) = load_format_erasure_all(&disks, true).await;
for (err, init_err) in errs.iter_mut().zip(init_errs) {
if init_err.is_some() {
*err = init_err;
}
}
if errs.iter().any(|err| {
matches!(
err,
Some(DiskError::InconsistentDisk | DiskError::CorruptedFormat | DiskError::CorruptedBackend)
)
}) {
return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
}
if let Err(err) = check_format_erasure_values(&formats, self.set_drive_count) {
info!("failed to check formats erasure values: {}", err);
return Ok((HealResultItem::default(), Some(err)));
}
let ref_format = match get_format_erasure_in_quorum(&formats) {
Ok(format) => format,
let (ref_format, quorum_members) = match select_format_erasure_in_quorum(&formats, 0) {
Ok((format, members)) if format.shared_identity() == self.format.shared_identity() => (format, members),
Ok(_) => return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat))),
Err(err) => return Ok((HealResultItem::default(), Some(err))),
};
if formats
.iter()
.zip(quorum_members)
.any(|(format, member)| format.is_some() && !member)
{
return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
}
let mut res = HealResultItem {
heal_item_type: HealItemType::Metadata.to_string(),
detail: "disk-format".to_string(),
@@ -985,11 +1008,6 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
return Ok((res, Some(StorageError::NoHealRequired)));
}
// if !self.format.eq(&ref_format) {
// info!("format ({:?}) not eq ref_format ({:?})", self.format, ref_format);
// return Ok((res, Some(Error::new(DiskError::CorruptedFormat))));
// }
let (new_format_sets, _) = new_heal_format_sets(&ref_format, self.set_count, self.set_drive_count, &formats, &errs);
if !dry_run {
let mut tmp_new_formats = vec![None; self.set_count * self.set_drive_count];
@@ -1298,7 +1316,7 @@ mod tests {
assert_eq!(result, (Some(3), Some(1), Some(0)));
}
async fn multipart_listing_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
async fn two_set_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
let format = FormatV3::new(2, 2);
let mut temp_dirs = Vec::new();
let mut all_endpoints = Vec::new();
@@ -1339,8 +1357,8 @@ mod tests {
Arc::new(RwLock::new(disks)),
2,
1,
0,
set_index,
0,
endpoints,
format.clone(),
vec![Arc::new(LocalClient::new()), Arc::new(LocalClient::new())],
@@ -1373,11 +1391,114 @@ mod tests {
(temp_dirs, sets)
}
#[tokio::test]
async fn set_format_heal_accepts_quorum_from_a_nonzero_set() {
let (_temp_dirs, sets) = two_set_test_sets().await;
let (result, err) = sets.disk_set[1]
.heal_format(false)
.await
.expect("the second erasure set should load its own format quorum");
assert!(matches!(err, Some(StorageError::NoHealRequired)), "unexpected heal result: {err:?}");
assert_eq!(result.disk_count, 2);
assert_eq!(result.set_count, 1);
}
#[tokio::test]
async fn format_heal_rejects_foreign_majorities_at_set_and_pool_scopes() {
let (_temp_dirs, _canonical_format, sets) = setup_heal_format_sets(2, true).await;
let set_disks = set_level_heal_view(&sets).await;
let (_, set_err) = set_disks
.heal_format(false)
.await
.expect("set format heal should report a typed mismatch");
assert!(
matches!(set_err, Some(StorageError::CorruptedFormat)),
"foreign set majority must not replace the cached format: {set_err:?}"
);
let (_, pool_err) = sets
.heal_format(false)
.await
.expect("pool format heal should report a typed mismatch");
assert!(
matches!(pool_err, Some(StorageError::CorruptedFormat)),
"foreign pool majority must not replace the cached format: {pool_err:?}"
);
}
#[tokio::test]
async fn pool_format_heal_rejects_a_wrong_slot_minority() {
let (_temp_dirs, canonical_format, sets) = setup_heal_format_sets(3, false).await;
let mut poisoned_format = canonical_format.clone();
poisoned_format.erasure.this = canonical_format.erasure.sets[0][0];
replace_heal_test_format(&sets, 2, &poisoned_format).await;
let probe_err = new_disk(
&sets.endpoints.endpoints.as_ref()[2],
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect_err("a wrong-slot local format must fail disk initialization");
assert_eq!(probe_err, DiskError::InconsistentDisk);
let (_, pool_err) = sets
.heal_format(false)
.await
.expect("pool format heal should report a typed slot mismatch");
assert!(
matches!(pool_err, Some(StorageError::CorruptedFormat)),
"a wrong-slot minority must not be reported as no-heal-required: {pool_err:?}"
);
assert_eq!(
read_heal_test_format(&sets, 2).await,
poisoned_format,
"format heal must not overwrite a wrong-slot disk"
);
}
#[tokio::test]
async fn format_heal_rejects_a_foreign_minority_at_set_and_pool_scopes() {
let (_temp_dirs, canonical_format, sets) = setup_heal_format_sets(3, false).await;
let mut poisoned_format = canonical_format.clone();
poisoned_format.id = Uuid::new_v4();
poisoned_format.erasure.this = poisoned_format.erasure.sets[0][2];
replace_heal_test_format(&sets, 2, &poisoned_format).await;
let set_disks = set_level_heal_view(&sets).await;
let (_, set_err) = set_disks
.heal_format(false)
.await
.expect("set format heal should report a typed identity mismatch");
assert!(
matches!(set_err, Some(StorageError::CorruptedFormat)),
"a foreign minority must not be reported as no-heal-required: {set_err:?}"
);
let (_, pool_err) = sets
.heal_format(false)
.await
.expect("pool format heal should report a typed identity mismatch");
assert!(
matches!(pool_err, Some(StorageError::CorruptedFormat)),
"a foreign minority must not be reported as no-heal-required: {pool_err:?}"
);
assert_eq!(
read_heal_test_format(&sets, 2).await,
poisoned_format,
"format heal must not overwrite a foreign disk"
);
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn list_multipart_uploads_merges_all_sets_without_pagination_loss() {
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::Erasure).await;
let (_temp_dirs, sets) = multipart_listing_test_sets().await;
let (_temp_dirs, sets) = two_set_test_sets().await;
let bucket = format!("multipart-list-{}", Uuid::new_v4().simple());
sets.make_bucket(&bucket, &MakeBucketOptions::default())
.await
@@ -1616,11 +1737,15 @@ mod tests {
// formatting the first `num_formatted` of them against a shared reference
// format and leaving the rest unformatted. Returns the live TempDir handles
// (must be kept alive), the reference format, and the assembled `Sets`.
// `disk_set` is intentionally empty: these tests only drive `heal_format`
// with `dry_run == true`, which never touches `disk_set`.
async fn setup_heal_format_sets(num_formatted: usize) -> (Vec<tempfile::TempDir>, FormatV3, Sets) {
// `disk_set` is intentionally empty: these tests only exercise paths that
// return before pool-level healing delegates into a set.
async fn setup_heal_format_sets(num_formatted: usize, foreign_identity: bool) -> (Vec<tempfile::TempDir>, FormatV3, Sets) {
const SET_DRIVE_COUNT: usize = 3;
let ref_format = FormatV3::new(1, SET_DRIVE_COUNT);
let mut stored_format = ref_format.clone();
if foreign_identity {
stored_format.id = Uuid::new_v4();
}
let mut dirs = Vec::with_capacity(SET_DRIVE_COUNT);
let mut endpoints = Vec::with_capacity(SET_DRIVE_COUNT);
@@ -1645,8 +1770,8 @@ mod tests {
)
.await
.expect("disk should be created");
let mut disk_format = ref_format.clone();
disk_format.erasure.this = ref_format.erasure.sets[0][i];
let mut disk_format = stored_format.clone();
disk_format.erasure.this = stored_format.erasure.sets[0][i];
save_format_file(&Some(disk), &Some(disk_format))
.await
.expect("format should be saved");
@@ -1677,6 +1802,60 @@ mod tests {
(dirs, ref_format, sets)
}
async fn set_level_heal_view(sets: &Sets) -> Arc<SetDisks> {
let endpoints = sets.endpoints.endpoints.as_ref().clone();
let mut disks = Vec::with_capacity(endpoints.len());
for endpoint in &endpoints {
disks.push(Some(
new_disk(
endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("fresh set-level disk handle should open"),
));
}
SetDisks::new(
"test-owner".to_string(),
Arc::new(RwLock::new(disks)),
endpoints.len(),
1,
0,
0,
endpoints,
sets.format.clone(),
Vec::new(),
)
.await
}
async fn replace_heal_test_format(sets: &Sets, disk_index: usize, format: &FormatV3) {
let disk = new_disk(
&sets.endpoints.endpoints.as_ref()[disk_index],
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("heal test disk should open");
save_format_file(&Some(disk.clone()), &Some(format.clone()))
.await
.expect("poisoned test format should be written");
}
async fn read_heal_test_format(sets: &Sets, disk_index: usize) -> FormatV3 {
let path = std::path::Path::new(&sets.endpoints.endpoints.as_ref()[disk_index].get_file_path())
.join(crate::disk::RUSTFS_META_BUCKET)
.join(crate::disk::FORMAT_CONFIG_FILE);
let data = tokio::fs::read(path).await.expect("test format should be readable");
FormatV3::try_from(data.as_slice()).expect("test format should parse")
}
// Regression for #956 (NoHealRequired path): with every disk already
// formatted, `heal_format` reports exactly one drive record per disk
// (N = set_count * set_drive_count), each carrying a real endpoint. Before
@@ -1685,7 +1864,7 @@ mod tests {
#[tokio::test]
#[serial]
async fn heal_format_no_heal_required_reports_one_record_per_disk() {
let (_dirs, _ref_format, sets) = setup_heal_format_sets(3).await;
let (_dirs, _ref_format, sets) = setup_heal_format_sets(3, false).await;
let (res, err) = sets.heal_format(true).await.expect("heal_format should succeed");
// All disks formatted -> NoHealRequired early return, still returns `res`.
@@ -1715,7 +1894,7 @@ mod tests {
#[serial]
async fn heal_format_heal_path_reports_one_record_per_disk_aligned() {
// Disks 0 and 1 formatted (quorum), disk 2 unformatted.
let (_dirs, _ref_format, sets) = setup_heal_format_sets(2).await;
let (_dirs, _ref_format, sets) = setup_heal_format_sets(2, false).await;
let (res, err) = sets.heal_format(true).await.expect("heal_format should succeed");
// Unformatted disk present -> heal path, not NoHealRequired.
+4
View File
@@ -1049,6 +1049,10 @@ impl LocalDiskWrapper {
Ok(())
}
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) {
*self.disk_id.write().await = id;
}
/// Get the current disk ID
pub async fn get_current_disk_id(&self) -> Option<Uuid> {
*self.disk_id.read().await
+600 -17
View File
@@ -381,6 +381,291 @@ fn classify_delete_volume_error(err: std::io::Error) -> DiskError {
}
}
#[cfg(unix)]
struct EmptyDirectoryFrame {
path: PathBuf,
name_in_parent: std::ffi::CString,
entries: rustix::fs::Dir,
}
#[cfg(unix)]
fn empty_tree_io_error(err: rustix::io::Errno) -> std::io::Error {
match err {
rustix::io::Errno::NOTDIR | rustix::io::Errno::LOOP => std::io::Error::from(ErrorKind::DirectoryNotEmpty),
_ => err.into(),
}
}
#[cfg(unix)]
fn remove_empty_directory_tree_unix_with(
root: &Path,
mut before_descend: impl FnMut(&Path) -> std::io::Result<()>,
mut before_remove: impl FnMut(&Path) -> std::io::Result<()>,
) -> std::io::Result<()> {
use rustix::{
fs::{AtFlags, Dir, Mode, OFlags, fstat, open, openat, statat, unlinkat},
io::Errno,
};
use std::os::fd::AsFd;
use std::os::unix::ffi::OsStrExt;
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
let root_parent_path = root
.parent()
.ok_or_else(|| std::io::Error::from(ErrorKind::DirectoryNotEmpty))?;
let root_name = root
.file_name()
.ok_or_else(|| std::io::Error::from(ErrorKind::DirectoryNotEmpty))?
.as_bytes();
let root_name = std::ffi::CString::new(root_name).map_err(|_| std::io::Error::from(ErrorKind::DirectoryNotEmpty))?;
let root_parent = open(root_parent_path, flags, Mode::empty()).map_err(empty_tree_io_error)?;
let root_fd = openat(&root_parent, root_name.as_c_str(), flags, Mode::empty()).map_err(empty_tree_io_error)?;
// Each frame owns one directory iterator/FD, so memory and descriptors are
// bounded by path depth rather than by the number of empty remnants.
let mut stack = vec![EmptyDirectoryFrame {
path: root.to_path_buf(),
name_in_parent: root_name,
entries: Dir::new(root_fd).map_err(empty_tree_io_error)?,
}];
while let Some(mut frame) = stack.pop() {
let next_child = loop {
let Some(entry) = frame.entries.next() else {
break None;
};
let entry = entry.map_err(std::io::Error::from)?;
let name = entry.file_name();
if name.to_bytes() == b"." || name.to_bytes() == b".." {
continue;
}
let name = name.to_owned();
let child_path = frame.path.join(std::ffi::OsStr::from_bytes(name.as_bytes()));
before_descend(&child_path)?;
break Some((child_path, name));
};
if let Some((child_path, name)) = next_child {
let parent = frame.entries.fd().map_err(std::io::Error::from)?;
let child = match openat(parent, name.as_c_str(), flags, Mode::empty()) {
Ok(child) => child,
// A concurrent cleanup may remove an empty child after readdir
// returns it. Resume the parent instead of treating the whole
// bucket as missing and leaving its root behind.
Err(Errno::NOENT) => {
stack.push(frame);
continue;
}
Err(err) => return Err(empty_tree_io_error(err)),
};
stack.push(frame);
stack.push(EmptyDirectoryFrame {
path: child_path,
name_in_parent: name,
entries: Dir::new(child).map_err(empty_tree_io_error)?,
});
continue;
}
before_remove(&frame.path)?;
let parent = if let Some(parent) = stack.last() {
parent.entries.fd().map_err(std::io::Error::from)?
} else {
root_parent.as_fd()
};
let expected = fstat(frame.entries.fd().map_err(std::io::Error::from)?).map_err(empty_tree_io_error)?;
let current = statat(parent, frame.name_in_parent.as_c_str(), AtFlags::SYMLINK_NOFOLLOW).map_err(empty_tree_io_error)?;
if current.st_dev != expected.st_dev || current.st_ino != expected.st_ino {
return Err(std::io::Error::from(ErrorKind::DirectoryNotEmpty));
}
match unlinkat(parent, frame.name_in_parent.as_c_str(), AtFlags::REMOVEDIR).map_err(empty_tree_io_error) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
Ok(())
}
#[cfg(unix)]
async fn remove_empty_directory_tree_with(
root: &Path,
before_descend: impl FnMut(&Path) -> std::io::Result<()>,
before_remove: impl FnMut(&Path) -> std::io::Result<()>,
) -> std::io::Result<()> {
remove_empty_directory_tree_unix_with(root, before_descend, before_remove)
}
#[cfg(unix)]
async fn remove_empty_directory_tree(root: &Path) -> std::io::Result<()> {
let root = root.to_path_buf();
tokio::task::spawn_blocking(move || remove_empty_directory_tree_unix_with(&root, |_| Ok(()), |_| Ok(()))).await?
}
#[cfg(windows)]
#[derive(Debug)]
struct LockedEmptyDirectory {
handle: winapi_util::Handle,
}
#[cfg(windows)]
fn validate_windows_empty_directory(file_attributes: u64) -> std::io::Result<()> {
const FILE_ATTRIBUTE_DIRECTORY: u64 = 0x10;
const FILE_ATTRIBUTE_REPARSE_POINT: u64 = 0x400;
if file_attributes & FILE_ATTRIBUTE_DIRECTORY == 0 || file_attributes & FILE_ATTRIBUTE_REPARSE_POINT != 0 {
return Err(std::io::Error::from(ErrorKind::DirectoryNotEmpty));
}
Ok(())
}
#[cfg(windows)]
async fn lock_windows_empty_directory(path: &Path, canonical_root: Option<&Path>) -> std::io::Result<LockedEmptyDirectory> {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::GENERIC_READ,
Storage::FileSystem::{DELETE, FILE_SHARE_READ},
};
const FILE_FLAG_BACKUP_SEMANTICS: u32 = 0x0200_0000;
const FILE_FLAG_OPEN_REPARSE_POINT: u32 = 0x0020_0000;
let path = path.to_path_buf();
let canonical_root = canonical_root.map(Path::to_path_buf);
tokio::task::spawn_blocking(move || {
let file = std::fs::OpenOptions::new()
.access_mode(GENERIC_READ | DELETE)
.share_mode(FILE_SHARE_READ)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&path)?;
let handle = winapi_util::Handle::from_file(file);
let info = winapi_util::file::information(&handle)?;
validate_windows_empty_directory(info.file_attributes())?;
if let Some(canonical_root) = canonical_root {
let canonical_path = std::fs::canonicalize(path)?;
if !canonical_path.starts_with(canonical_root) {
return Err(std::io::Error::from(ErrorKind::DirectoryNotEmpty));
}
}
Ok::<_, std::io::Error>(LockedEmptyDirectory { handle })
})
.await?
}
#[cfg(windows)]
// SAFETY: This helper only passes an owned live handle and one initialized
// FILE_DISPOSITION_INFO to the synchronous Windows deletion API.
#[allow(unsafe_code)]
async fn remove_windows_empty_directory(directory: LockedEmptyDirectory) -> std::io::Result<()> {
tokio::task::spawn_blocking(move || {
use std::os::windows::io::AsRawHandle;
use windows_sys::Win32::Storage::FileSystem::{FILE_DISPOSITION_INFO, FileDispositionInfo, SetFileInformationByHandle};
let disposition = FILE_DISPOSITION_INFO { DeleteFile: true };
let disposition_size = u32::try_from(std::mem::size_of_val(&disposition))
.map_err(|_| std::io::Error::other("FILE_DISPOSITION_INFO size exceeds the Win32 API limit"))?;
let handle = directory.handle.as_raw_handle();
// SAFETY: `handle` is owned by `directory` and stays live for this synchronous
// call. `disposition` is initialized with the exact structure and byte size
// required by `FileDispositionInfo`; Windows does not retain the pointer.
let deleted = unsafe {
SetFileInformationByHandle(handle, FileDispositionInfo, std::ptr::from_ref(&disposition).cast(), disposition_size)
};
if deleted == 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
})
.await?
}
#[cfg(windows)]
struct WindowsEmptyDirectoryFrame {
path: PathBuf,
directory: LockedEmptyDirectory,
entries: fs::ReadDir,
}
#[cfg(windows)]
async fn remove_empty_directory_tree_with(
root: &Path,
mut before_descend: impl FnMut(&Path) -> std::io::Result<()>,
mut before_remove: impl FnMut(&Path) -> std::io::Result<()>,
) -> std::io::Result<()> {
let root_directory = lock_windows_empty_directory(root, None).await?;
let canonical_root = fs::canonicalize(root).await?;
let root_entries = fs::read_dir(root).await?;
// Holding each validated directory without delete sharing keeps its path
// generation stable until handle-relative deletion. State is O(depth).
let mut stack = vec![WindowsEmptyDirectoryFrame {
path: root.to_path_buf(),
directory: root_directory,
entries: root_entries,
}];
while let Some(mut frame) = stack.pop() {
match frame.entries.next_entry().await {
Ok(Some(entry)) => {
let child = entry.path();
before_descend(&child)?;
let child_directory = match lock_windows_empty_directory(&child, Some(&canonical_root)).await {
Ok(directory) => directory,
Err(err) if err.kind() == ErrorKind::NotFound => {
stack.push(frame);
continue;
}
Err(err) => return Err(err),
};
let child_entries = match fs::read_dir(&child).await {
Ok(entries) => entries,
Err(err) if err.kind() == ErrorKind::NotFound => {
stack.push(frame);
continue;
}
Err(err) if err.kind() == ErrorKind::NotADirectory => {
return Err(std::io::Error::from(ErrorKind::DirectoryNotEmpty));
}
Err(err) => return Err(err),
};
stack.push(frame);
stack.push(WindowsEmptyDirectoryFrame {
path: child,
directory: child_directory,
entries: child_entries,
});
}
Ok(None) => {
before_remove(&frame.path)?;
drop(frame.entries);
match remove_windows_empty_directory(frame.directory).await {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if err.kind() == ErrorKind::NotADirectory => {
return Err(std::io::Error::from(ErrorKind::DirectoryNotEmpty));
}
Err(err) => return Err(err),
}
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
Ok(())
}
#[cfg(windows)]
async fn remove_empty_directory_tree(root: &Path) -> std::io::Result<()> {
remove_empty_directory_tree_with(root, |_| Ok(()), |_| Ok(())).await
}
#[cfg(all(not(unix), not(windows)))]
async fn remove_empty_directory_tree(root: &Path) -> std::io::Result<()> {
fs::remove_dir(root).await
}
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_DISK_LOCAL: &str = "disk_local";
const EVENT_DISK_LOCAL_STARTUP_CLEANUP: &str = "disk_local_startup_cleanup";
@@ -1556,6 +1841,8 @@ static RENAME_DATA_FAIL_COMMIT_RENAME: std::sync::Mutex<Option<String>> = std::s
#[cfg(test)]
static LOCAL_INLINE_ROLLBACK_HARDLINK_FAILURE: std::sync::Mutex<Option<PathBuf>> = std::sync::Mutex::new(None);
#[cfg(test)]
static RENAME_DATA_REMOVE_DST_BASE_BEFORE_COMMIT: std::sync::Mutex<Option<(String, PathBuf)>> = std::sync::Mutex::new(None);
#[cfg(test)]
static DELETE_VERSION_FAIL_AFTER_DATA_STAGED: std::sync::Mutex<Vec<String>> = std::sync::Mutex::new(Vec::new());
#[cfg(test)]
static DELETE_VERSION_FAIL_AFTER_COMMIT: std::sync::Mutex<Vec<(PathBuf, String)>> = std::sync::Mutex::new(Vec::new());
@@ -1588,6 +1875,13 @@ fn set_local_inline_rollback_hardlink_failure(dst_path: &Path) {
.expect("test failpoint lock should not be poisoned") = Some(dst_path.to_path_buf());
}
#[cfg(test)]
fn set_rename_data_remove_dst_base_before_commit(dst_path: &str, dst_base: &Path) {
*RENAME_DATA_REMOVE_DST_BASE_BEFORE_COMMIT
.lock()
.expect("test failpoint lock should not be poisoned") = Some((dst_path.to_string(), dst_base.to_path_buf()));
}
#[cfg(test)]
fn set_delete_version_fail_after_data_staged(path: &str) {
DELETE_VERSION_FAIL_AFTER_DATA_STAGED
@@ -1656,6 +1950,19 @@ fn should_fail_local_inline_rollback_hardlink(dst_path: &Path) -> bool {
}
}
#[cfg(test)]
fn remove_dst_base_before_commit(dst_path: &str) -> std::io::Result<()> {
let mut target = RENAME_DATA_REMOVE_DST_BASE_BEFORE_COMMIT
.lock()
.expect("test failpoint lock should not be poisoned");
let Some((_, base)) = target.as_ref().filter(|(target_path, _)| target_path == dst_path) else {
return Ok(());
};
std::fs::remove_dir_all(base)?;
target.take();
Ok(())
}
#[cfg(test)]
fn should_fail_after_delete_data_staged(path: &str) -> bool {
let mut targets = DELETE_VERSION_FAIL_AFTER_DATA_STAGED
@@ -1705,6 +2012,11 @@ fn should_fail_local_inline_rollback_hardlink(_dst_path: &Path) -> bool {
false
}
#[cfg(not(test))]
fn remove_dst_base_before_commit(_dst_path: &str) -> std::io::Result<()> {
Ok(())
}
#[cfg(not(test))]
fn should_fail_after_delete_data_staged(_path: &str) -> bool {
false
@@ -4243,9 +4555,11 @@ impl LocalDisk {
let tmp_path = Self::meta_path(root, RUSTFS_META_TMP_BUCKET);
let tmp_old_path = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET).join(Uuid::new_v4().to_string());
rename_all(&tmp_path, &tmp_old_path, root).await.inspect_err(|err| {
log_startup_disk_error("cleanup_tmp_rename_all", &tmp_path, err);
})?;
rename_all_ignore_missing_source(&tmp_path, &tmp_old_path, root)
.await
.inspect_err(|err| {
log_startup_disk_error("cleanup_tmp_rename_all", &tmp_path, err);
})?;
let tmp_deleted_path = Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET);
tokio::fs::create_dir_all(&tmp_deleted_path).await.inspect_err(|err| {
@@ -4764,7 +5078,7 @@ impl LocalDisk {
Ok((buf, mtime))
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
async fn read_metadata_with_dmtime(&self, file_path: impl AsRef<Path>) -> Result<(Vec<u8>, Option<OffsetDateTime>)> {
check_path_length(file_path.as_ref().to_string_lossy().as_ref())?;
@@ -4807,7 +5121,7 @@ impl LocalDisk {
Ok((data, modtime))
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
async fn read_all_data(&self, volume: &str, volume_dir: impl AsRef<Path>, file_path: impl AsRef<Path>) -> Result<Vec<u8>> {
// TODO: timeout support
let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?;
@@ -7359,6 +7673,10 @@ impl DiskAPI for LocalDisk {
dst_path: &str,
) -> Result<RenameDataResp> {
crate::hp_guard!("LocalDisk::rename_data");
// A non-force DeleteBucket must not remove a directory while a local
// object commit is publishing into it. The peer's empty scan remains
// optimistic; this guard establishes the local commit/delete order.
let _volume_mutation_guard = os::disk_volume_mutation_lock(&self.root, dst_volume).read_owned().await;
if fi.is_legacy_indexed_delete_marker() {
fi.erasure.index = 0;
}
@@ -7555,6 +7873,7 @@ impl DiskAPI for LocalDisk {
// sequential version did.
tmp_meta_res?;
shard_sync_res?;
remove_dst_base_before_commit(dst_path).map_err(to_file_error)?;
if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref()
&& let Err(err) = rename_all(src_data_path, dst_data_path, &skip_parent).await
@@ -7814,6 +8133,7 @@ impl DiskAPI for LocalDisk {
xlmeta.add_version(fi)?;
let version_signature = rename_data_versions_signature(&xlmeta);
let new_buf = xlmeta.marshal_msg()?;
remove_dst_base_before_commit(&dst_path_for_failpoint).map_err(to_file_error)?;
// Write new xl.meta + rename. Inline objects carry their data
// inside xl.meta, so this whole sequence is a metadata commit:
@@ -7838,9 +8158,11 @@ impl DiskAPI for LocalDisk {
{
let old_path = dst_parent.join(old_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
let old_parent = old_path.parent().map(|p| p.to_path_buf());
if let Some(ref old_parent) = old_parent {
std::fs::create_dir_all(old_parent)?;
}
let _old_parent_guard = old_parent
.as_deref()
.map(|parent| os::mkdir_all_below_existing_base_std(parent, &bucket_dir))
.transpose()
.map_err(to_file_error)?;
// This rollback backup is the sole restore source for a later
// undo_write when the set-level write quorum fails. Persist it as
// durably as the new xl.meta written above (and as the non-inline
@@ -7866,6 +8188,12 @@ impl DiskAPI for LocalDisk {
local_rollback_path = Some(create_local_inline_rollback_backup(&dst, &src, old_metadata)?);
}
#[cfg(windows)]
let _commit_parent_guard = if let Some(parent) = dst.parent() {
Some(os::mkdir_all_below_existing_base_std(parent, &bucket_dir).map_err(to_file_error)?)
} else {
None
};
let commit_result = if should_fail_commit_rename(&dst_path_for_failpoint) {
Err(std::io::Error::other("test fail during metadata commit rename"))
} else {
@@ -7874,7 +8202,8 @@ impl DiskAPI for LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound && !src.exists() => Ok(()),
Err(err) if err.kind() == ErrorKind::NotFound => {
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent)?;
let _parent_guard =
os::mkdir_all_below_existing_base_std(parent, &bucket_dir).map_err(to_file_error)?;
}
std::fs::rename(&src, &dst).map_err(to_file_error)?;
Ok(())
@@ -8791,18 +9120,16 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn delete_volume(&self, volume: &str, force_delete: bool) -> Result<()> {
let p = self.get_bucket_path(volume)?;
let _volume_mutation_guard = os::disk_volume_mutation_lock(&self.root, volume).write_owned().await;
// Non-force is non-recursive: `remove_dir` (rmdir) fails atomically with
// `DirectoryNotEmpty` -> VolumeNotEmpty if the bucket still holds any
// object data, so a misclassified "dangling" bucket on the heal path
// (or a non-force S3 DeleteBucket on a populated bucket) can never be
// recursively wiped. Only an explicit `force_delete` (e.g. S3 force
// bucket delete) removes recursively. Mirrors MinIO's
// xlStorage.DeleteVol (Remove vs RemoveAll). (backlog#799 B1)
// Non-force removes empty directory remnants children-first with
// non-recursive rmdir calls. A file that exists during the scan, or
// appears before its parent is removed, fails closed with
// VolumeNotEmpty. Only an explicit force delete removes recursively.
let res = if force_delete {
fs::remove_dir_all(&p).await
} else {
fs::remove_dir(&p).await
remove_empty_directory_tree(&p).await
};
if let Err(err) = res {
@@ -9061,6 +9388,35 @@ mod test {
}
}
#[cfg(windows)]
#[test]
fn windows_empty_tree_requires_non_reparse_directory() {
validate_windows_empty_directory(0x10).expect("ordinary directories should be accepted");
assert!(validate_windows_empty_directory(0).is_err());
assert!(validate_windows_empty_directory(0x10 | 0x400).is_err());
}
#[cfg(windows)]
#[tokio::test]
async fn windows_empty_tree_blocks_replacement_at_final_delete_boundary() {
let root = tempfile::tempdir().expect("temporary root should be created");
let bucket_path = root.path().join("bucket");
let child_path = bucket_path.join("child");
fs::create_dir_all(&child_path).await.expect("bucket child should be created");
let canonical_root = fs::canonicalize(&bucket_path).await.expect("bucket path should canonicalize");
let directory = lock_windows_empty_directory(&child_path, Some(&canonical_root))
.await
.expect("child directory should be locked");
std::fs::rename(&child_path, bucket_path.join("replacement"))
.expect_err("the locked directory must not be replaceable at the final deletion boundary");
remove_windows_empty_directory(directory)
.await
.expect("handle-relative deletion should remove the locked directory");
assert!(!child_path.exists(), "the exact locked directory should be removed");
}
async fn ensure_test_volume(disk: &LocalDisk, volume: &str) {
match disk.make_volume(volume).await {
Ok(()) | Err(DiskError::VolumeExists) => {}
@@ -10787,6 +11143,72 @@ mod test {
);
}
#[tokio::test]
#[serial_test::serial(rename_data_deleted_bucket)]
async fn rename_data_non_inline_does_not_recreate_bucket_deleted_before_commit() {
let dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "deleted-before-non-inline-commit";
let object = "prefix/object";
let tmp_object = "tmp-non-inline-delete-race";
let data_dir = Uuid::parse_str("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee").expect("data dir should parse");
ensure_test_volume(&disk, bucket).await;
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
let staged_data = disk
.get_object_path(RUSTFS_META_TMP_BUCKET, &format!("{tmp_object}/{data_dir}/part.1"))
.expect("staged data path should resolve");
fs::create_dir_all(staged_data.parent().expect("staged data should have a parent"))
.await
.expect("staged data parent should be created");
fs::write(&staged_data, b"payload")
.await
.expect("staged shard should be written");
let bucket_path = disk.get_bucket_path(bucket).expect("bucket path should resolve");
set_rename_data_remove_dst_base_before_commit(object, &bucket_path);
let fi = test_file_info(object, Uuid::new_v4(), Some(data_dir), None);
let err = disk
.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, fi, bucket, object)
.await
.expect_err("commit must fail after the destination bucket is deleted");
assert_eq!(err, DiskError::FileNotFound);
assert!(!bucket_path.exists(), "rename_data must not recreate the deleted bucket");
assert!(staged_data.exists(), "failed commit must preserve the staged shard");
}
#[tokio::test]
#[serial_test::serial(rename_data_deleted_bucket)]
async fn rename_data_inline_does_not_recreate_bucket_deleted_before_commit() {
let dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "deleted-before-inline-commit";
let object = "prefix/object";
let tmp_object = "tmp-inline-delete-race";
ensure_test_volume(&disk, bucket).await;
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
let bucket_path = disk.get_bucket_path(bucket).expect("bucket path should resolve");
set_rename_data_remove_dst_base_before_commit(object, &bucket_path);
let fi = test_file_info(object, Uuid::new_v4(), None, Some(Bytes::from_static(b"inline payload")));
let err = disk
.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, fi, bucket, object)
.await
.expect_err("inline commit must fail after the destination bucket is deleted");
assert_eq!(err, DiskError::FileNotFound);
assert!(!bucket_path.exists(), "inline rename_data must not recreate the deleted bucket");
assert!(
disk.get_object_path(RUSTFS_META_TMP_BUCKET, &format!("{tmp_object}/{STORAGE_FORMAT_FILE}"))
.expect("staged metadata path should resolve")
.exists(),
"failed inline commit must preserve staged metadata"
);
}
#[test]
fn test_resolve_durability_mode_mapping() {
// Default: nothing set -> strict (current main behavior).
@@ -12645,6 +13067,19 @@ mod test {
assert!(format_info.last_check.is_none(), "cached format timestamp should be cleared");
}
#[tokio::test]
async fn cleanup_tmp_on_startup_allows_missing_tmp_directory() {
use tempfile::tempdir;
let dir = tempdir().expect("operation should succeed");
LocalDisk::cleanup_tmp_on_startup(dir.path(), Arc::new(AtomicU32::new(0)), Arc::new(Notify::new()))
.await
.expect("missing temporary directory should already be clean");
assert!(LocalDisk::meta_path(dir.path(), RUSTFS_META_TMP_DELETED_BUCKET).exists());
}
#[tokio::test]
async fn cleanup_tmp_on_startup_moves_existing_tmp_and_recreates_trash() {
use tempfile::tempdir;
@@ -14517,6 +14952,154 @@ mod test {
let _ = fs::remove_dir_all(&test_dir).await;
}
#[tokio::test]
async fn delete_volume_non_force_removes_nested_empty_directories() {
let root = tempfile::tempdir().expect("temporary disk root should be created");
let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "nested-empty-bucket";
disk.make_volume(bucket).await.expect("bucket should be created");
fs::create_dir_all(disk.path().join(bucket).join("a/b/c"))
.await
.expect("nested empty directories should be created");
disk.delete_volume(bucket, false)
.await
.expect("non-force delete should remove an empty directory tree");
assert!(matches!(disk.stat_volume(bucket).await, Err(DiskError::VolumeNotFound)));
}
#[tokio::test]
async fn empty_tree_delete_preserves_xlmeta_published_after_scan() {
let root = tempfile::tempdir().expect("temporary disk root should be created");
let bucket_path = root.path().join("bucket");
let object_path = bucket_path.join("object").join(STORAGE_FORMAT_FILE);
fs::create_dir_all(object_path.parent().expect("object path should have a parent"))
.await
.expect("empty object directory should be created");
let data = b"committed object metadata";
let mut published = false;
let err = remove_empty_directory_tree_with(
&bucket_path,
|_| Ok(()),
|directory| {
if !published && directory == object_path.parent().expect("object path should have a parent") {
std::fs::write(&object_path, data)?;
published = true;
}
Ok(())
},
)
.await
.expect_err("rmdir should refuse metadata published after the directory scan");
assert!(published, "test barrier should publish metadata before rmdir");
assert!(matches!(classify_delete_volume_error(err), DiskError::VolumeNotEmpty));
assert_eq!(std::fs::read(&object_path).expect("object metadata should be preserved"), data);
}
#[cfg(unix)]
#[tokio::test]
async fn empty_tree_delete_rejects_child_replaced_with_external_symlink() {
use std::os::unix::fs::symlink;
let root = tempfile::tempdir().expect("temporary root should be created");
let bucket_path = root.path().join("bucket");
let child_path = bucket_path.join("child");
let outside_path = root.path().join("outside");
let outside_empty = outside_path.join("must-remain");
fs::create_dir_all(&child_path).await.expect("bucket child should be created");
fs::create_dir_all(&outside_empty)
.await
.expect("outside directory should be created");
let mut replaced = false;
let err = remove_empty_directory_tree_with(
&bucket_path,
|child| {
if !replaced && child == child_path {
std::fs::remove_dir(&child_path)?;
symlink(&outside_path, &child_path)?;
replaced = true;
}
Ok(())
},
|_| Ok(()),
)
.await
.expect_err("a replaced child must fail closed");
assert!(replaced, "test barrier should replace the child before it is opened");
assert!(matches!(classify_delete_volume_error(err), DiskError::VolumeNotEmpty));
assert!(outside_empty.exists(), "bucket deletion must not remove directories outside the bucket");
assert!(
std::fs::symlink_metadata(&child_path)
.expect("replacement symlink should remain")
.file_type()
.is_symlink()
);
}
#[cfg(unix)]
#[tokio::test]
async fn empty_tree_delete_rechecks_parent_after_child_disappears() {
let root = tempfile::tempdir().expect("temporary root should be created");
let bucket_path = root.path().join("bucket");
let child_path = bucket_path.join("child");
fs::create_dir_all(&child_path).await.expect("bucket child should be created");
let mut removed = false;
remove_empty_directory_tree_with(
&bucket_path,
|child| {
if !removed && child == child_path {
std::fs::remove_dir(&child_path)?;
removed = true;
}
Ok(())
},
|_| Ok(()),
)
.await
.expect("a vanished empty child should not leave the bucket root behind");
assert!(removed, "test hook should remove the child before openat");
assert!(!bucket_path.exists(), "parent should be rechecked and removed after the child disappears");
}
#[cfg(unix)]
#[tokio::test]
async fn empty_tree_delete_rejects_root_generation_replacement() {
let root = tempfile::tempdir().expect("temporary root should be created");
let bucket_path = root.path().join("bucket");
let moved_path = root.path().join("bucket-before-replacement");
fs::create_dir(&bucket_path).await.expect("bucket should be created");
let mut replaced = false;
let err = remove_empty_directory_tree_with(
&bucket_path,
|_| Ok(()),
|directory| {
if !replaced && directory == bucket_path {
std::fs::rename(&bucket_path, &moved_path)?;
std::fs::create_dir(&bucket_path)?;
replaced = true;
}
Ok(())
},
)
.await
.expect_err("a replacement root directory must fail closed");
assert!(replaced, "test hook should replace the root generation before rmdir");
assert!(matches!(classify_delete_volume_error(err), DiskError::VolumeNotEmpty));
assert!(moved_path.exists(), "the originally scanned root should not be removed by its old name");
assert!(bucket_path.exists(), "the replacement root should remain after identity validation fails");
}
#[tokio::test]
async fn test_local_disk_volume_operations() {
let test_dir = "./test_local_disk_volumes";
+78
View File
@@ -132,6 +132,18 @@ pub enum Disk {
Remote(Box<RemoteDisk>),
}
impl Disk {
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
match self {
Disk::Local(local_disk) => {
local_disk.set_disk_id_state(id).await;
Ok(())
}
Disk::Remote(remote_disk) => remote_disk.set_disk_id(id).await,
}
}
}
#[async_trait::async_trait]
impl DiskAPI for Disk {
fn to_string(&self) -> String {
@@ -1552,6 +1564,72 @@ mod tests {
let _ = fs::remove_dir_all(&test_dir).await;
}
#[tokio::test]
#[serial_test::serial]
async fn local_disk_id_state_does_not_publish_to_the_process_registry() {
let local_dir = tempfile::tempdir().expect("local disk tempdir should be created");
let mut endpoint =
Endpoint::try_from(local_dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let local_disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
let disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("local wrapper state should accept a disk ID");
let Disk::Local(local_disk) = &disk else {
panic!("test disk should remain local");
};
assert_eq!(local_disk.get_current_disk_id().await, Some(disk_id));
assert!(
!crate::runtime::global::current_ctx()
.local_disk_id_map()
.read()
.await
.contains_key(&disk_id),
"state-only startup publication must not update the process disk-ID registry"
);
disk.set_disk_id_state(None)
.await
.expect("local wrapper state should clear a disk ID");
assert_eq!(local_disk.get_current_disk_id().await, None);
}
#[tokio::test]
async fn remote_disk_id_state_delegates_some_and_none() {
let mut endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let remote_disk = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
Arc::new(crate::cluster::rpc::TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk should initialize");
let disk = Disk::Remote(Box::new(remote_disk));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("remote state should accept a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), Some(disk_id));
disk.set_disk_id_state(None)
.await
.expect("remote state should clear a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), None);
}
#[tokio::test]
async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
let local_dir = tempfile::tempdir().unwrap();
+321 -9
View File
@@ -25,7 +25,7 @@ use std::{
sync::{Arc, LazyLock, Weak},
};
use tokio::fs;
use tokio::sync::{OwnedSemaphorePermit, Semaphore, SemaphorePermit};
use tokio::sync::{OwnedSemaphorePermit, RwLock, Semaphore, SemaphorePermit};
use tracing::warn;
/// Check path length according to OS limits.
@@ -123,6 +123,8 @@ const TEST_GLOBAL_FILE_SYNCS: usize = 64;
static FILE_SYNC_PERMITS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(global_file_sync_limit()));
static DISK_FILE_SYNC_LIMITERS: LazyLock<Mutex<HashMap<PathBuf, Weak<Semaphore>>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
static DISK_VOLUME_MUTATION_LOCKS: LazyLock<Mutex<HashMap<PathBuf, Weak<RwLock<()>>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
fn default_global_file_sync_limit(cpu_count: usize, max_blocking_threads: usize) -> usize {
let cpu_scaled = cpu_count
@@ -157,6 +159,24 @@ pub(crate) fn disk_file_sync_limiter(root: &Path) -> Arc<Semaphore> {
limiter
}
/// Serialize a bucket's local metadata commits with physical bucket removal.
///
/// The key includes the canonical disk root, so independently reconnected
/// [`LocalDisk`](super::local::LocalDisk) instances share the same lock while
/// disconnected disks do not keep the registry alive.
pub(crate) fn disk_volume_mutation_lock(root: &Path, volume: &str) -> Arc<RwLock<()>> {
let key = root.join(volume);
let mut locks = DISK_VOLUME_MUTATION_LOCKS.lock();
locks.retain(|_, lock| lock.strong_count() > 0);
if let Some(lock) = locks.get(&key).and_then(Weak::upgrade) {
return lock;
}
let lock = Arc::new(RwLock::new(()));
locks.insert(key, Arc::downgrade(&lock));
lock
}
/// Always acquire the per-disk permit before the process-wide permit. Keeping
/// this order uniform prevents one slow disk from reserving global capacity
/// while it waits for its own concurrency slot.
@@ -572,15 +592,14 @@ async fn reliable_rename_inner(
base_dir: impl AsRef<Path>,
warn_on_missing_source: bool,
) -> io::Result<()> {
if let Some(parent) = dst_file_path.as_ref().parent()
&& !file_exists(parent)
{
reliable_mkdir_all(parent, base_dir.as_ref()).await?;
}
let parent_guard = match dst_file_path.as_ref().parent() {
Some(parent) => Some(mkdir_all_below_existing_base(parent, base_dir.as_ref()).await?),
None => None,
};
let mut i = 0;
loop {
if let Err(e) = super::fs::rename_std(src_file_path.as_ref(), dst_file_path.as_ref()) {
if let Err(e) = rename_into_existing_parent(src_file_path.as_ref(), dst_file_path.as_ref(), parent_guard.as_ref()) {
if should_retry_rename(&e, i) {
i += 1;
continue;
@@ -597,6 +616,159 @@ async fn reliable_rename_inner(
Ok(())
}
#[cfg(unix)]
fn rename_into_existing_parent(
src_file_path: &Path,
dst_file_path: &Path,
parent_guard: Option<&ExistingBaseDirectoryGuard>,
) -> io::Result<()> {
use rustix::fs::{Mode, OFlags, open, renameat};
let Some(parent_guard) = parent_guard else {
return super::fs::rename_std(src_file_path, dst_file_path);
};
let src_parent = src_file_path
.parent()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename source must have a parent directory"))?;
let src_name = src_file_path
.file_name()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename source must have a file name"))?;
let dst_name = dst_file_path
.file_name()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must have a file name"))?;
let src_parent = open(
src_parent,
OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)
.map_err(io::Error::from)?;
let dst_parent = parent_guard
.last()
.ok_or_else(|| io::Error::other("rename destination parent guard is empty"))?;
renameat(&src_parent, src_name, dst_parent, dst_name).map_err(io::Error::from)
}
#[cfg(not(unix))]
fn rename_into_existing_parent(
src_file_path: &Path,
dst_file_path: &Path,
_parent_guard: Option<&ExistingBaseDirectoryGuard>,
) -> io::Result<()> {
super::fs::rename_std(src_file_path, dst_file_path)
}
async fn mkdir_all_below_existing_base(dir_path: &Path, base_dir: &Path) -> io::Result<ExistingBaseDirectoryGuard> {
let dir_path = dir_path.to_path_buf();
let base_dir = base_dir.to_path_buf();
tokio::task::spawn_blocking(move || mkdir_all_below_existing_base_std(&dir_path, &base_dir)).await?
}
#[cfg(windows)]
pub(crate) type ExistingBaseDirectoryGuard = Vec<winapi_util::Handle>;
#[cfg(unix)]
pub(crate) type ExistingBaseDirectoryGuard = Vec<std::os::fd::OwnedFd>;
#[cfg(all(not(unix), not(windows)))]
pub(crate) type ExistingBaseDirectoryGuard = ();
#[cfg(windows)]
fn lock_windows_directory(path: &Path) -> io::Result<winapi_util::Handle> {
use std::os::windows::fs::OpenOptionsExt;
const FILE_ATTRIBUTE_DIRECTORY: u64 = 0x10;
const FILE_ATTRIBUTE_REPARSE_POINT: u32 = 0x400;
const FILE_FLAG_BACKUP_SEMANTICS: u32 = 0x0200_0000;
const FILE_FLAG_OPEN_REPARSE_POINT: u32 = 0x0020_0000;
const FILE_SHARE_READ: u32 = 0x1;
let file = std::fs::OpenOptions::new()
.read(true)
.share_mode(FILE_SHARE_READ)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(path)?;
let handle = winapi_util::Handle::from_file(file);
let info = winapi_util::file::information(&handle)?;
if info.file_attributes() & FILE_ATTRIBUTE_DIRECTORY == 0
|| info.file_attributes() & u64::from(FILE_ATTRIBUTE_REPARSE_POINT) != 0
{
return Err(io::Error::from(io::ErrorKind::NotADirectory));
}
Ok(handle)
}
pub(crate) fn mkdir_all_below_existing_base_std(dir_path: &Path, base_dir: &Path) -> io::Result<ExistingBaseDirectoryGuard> {
let relative = dir_path
.strip_prefix(base_dir)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must remain below its base directory"))?;
for component in relative.components() {
if !matches!(component, Component::Normal(_) | Component::CurDir) {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"rename destination contains an invalid path component",
));
}
}
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, mkdirat, open, openat};
use rustix::io::Errno;
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
let mode = Mode::RWXU | Mode::RWXG | Mode::RWXO;
let mut parents = vec![open(base_dir, flags, Mode::empty()).map_err(io::Error::from)?];
for component in relative.components() {
let Component::Normal(component) = component else {
continue;
};
let parent = parents
.last()
.expect("base directory guard should contain the base directory");
match mkdirat(parent, component, mode) {
Ok(()) => {}
Err(Errno::EXIST) => {}
Err(err) => return Err(err.into()),
}
parents.push(openat(parent, component, flags, Mode::empty()).map_err(io::Error::from)?);
}
Ok(parents)
}
#[cfg(windows)]
{
let mut handles = vec![lock_windows_directory(base_dir)?];
let mut current = base_dir.to_path_buf();
for component in relative.components() {
let Component::Normal(component) = component else {
continue;
};
current.push(component);
match std::fs::create_dir(&current) {
Ok(()) => {}
Err(err) if err.kind() == io::ErrorKind::AlreadyExists => {}
Err(err) => return Err(err),
}
handles.push(lock_windows_directory(&current)?);
}
Ok(handles)
}
#[cfg(all(not(unix), not(windows)))]
{
let _ = relative;
Err(io::Error::new(
io::ErrorKind::Unsupported,
"safe recursive directory creation is unavailable on this platform",
))
}
}
fn warn_reliable_rename_failure(src_file_path: &Path, dst_file_path: &Path, base_dir: &Path, err: &io::Error) {
warn!(
"reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
@@ -727,6 +899,20 @@ mod tests {
Arc::new(Semaphore::new(MAX_PARALLEL_FILE_SYNCS))
}
#[tokio::test]
async fn disk_volume_mutation_lock_is_shared_per_root_and_volume() {
let temp_dir = tempdir().expect("create temp dir");
let first = disk_volume_mutation_lock(temp_dir.path(), "bucket");
let second = disk_volume_mutation_lock(temp_dir.path(), "bucket");
let other = disk_volume_mutation_lock(temp_dir.path(), "other-bucket");
assert!(Arc::ptr_eq(&first, &second), "reconnected disks must share a bucket mutation lock");
assert!(!Arc::ptr_eq(&first, &other), "different buckets must not serialize each other");
let _write_guard = first.write().await;
assert!(second.try_read().is_err(), "a bucket delete lock must exclude local commits");
}
#[derive(Clone, Default)]
struct CapturedLogs {
buffer: Arc<Mutex<Vec<u8>>>,
@@ -771,9 +957,26 @@ mod tests {
}
}
/// Holds a `warn_capture()` capture alive: the thread-local subscriber, plus
/// the pin that keeps tracing's process-global callsite-interest cache from
/// being decided by some other test's thread.
struct WarnCaptureGuard {
_subscriber: tracing::subscriber::DefaultGuard,
_callsite_pin: tracing::Dispatch,
}
/// Capture WARN-level output on the current thread; tokio tests here run on
/// the current-thread runtime, so the guard covers the whole test body.
fn warn_capture() -> (CapturedLogs, tracing::subscriber::DefaultGuard) {
///
/// The callsite pin matters because `warn_reliable_rename_failure` is a
/// single production callsite shared with tests that call `rename_all`
/// *without* installing a subscriber — `rename_all_missing_source_returns_file_not_found`
/// is one. Whichever thread reaches it first fixes its `Interest`
/// process-wide, so without the pin that sibling can cache
/// `Interest::never()` and the WARN never fires here at all, leaving the
/// "must keep the WARN" assertions staring at empty output. See
/// [`crate::test_tracing::pin_callsite_interest_for_test`].
fn warn_capture() -> (CapturedLogs, WarnCaptureGuard) {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
.with_max_level(tracing::Level::WARN)
@@ -781,7 +984,10 @@ mod tests {
.with_ansi(false)
.without_time()
.finish();
let guard = tracing::subscriber::set_default(subscriber);
let guard = WarnCaptureGuard {
_subscriber: tracing::subscriber::set_default(subscriber),
_callsite_pin: crate::test_tracing::pin_callsite_interest_for_test(),
};
(logs, guard)
}
@@ -968,6 +1174,112 @@ mod tests {
assert_eq!(std::fs::read(dst.join("nested").join("part.1")).expect("read moved part"), b"payload");
}
#[tokio::test]
async fn rename_all_does_not_recreate_missing_base_directory() {
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
std::fs::create_dir(&base).expect("create destination base");
let src = temp_dir.path().join("staged-object");
std::fs::write(&src, b"payload").expect("write staged object");
let dst = base.join("object").join("xl.meta");
std::fs::remove_dir(&base).expect("delete destination base before commit");
let err = rename_all(&src, &dst, &base)
.await
.expect_err("rename must not recreate a deleted destination base");
assert!(matches!(err, DiskError::FileNotFound));
assert!(src.exists(), "failed commit must preserve the staged source");
assert!(!base.exists(), "failed commit must not recreate the deleted bucket");
}
#[cfg(unix)]
#[tokio::test]
async fn rename_all_rejects_a_replaced_base_with_an_existing_parent() {
use std::os::unix::fs::symlink;
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let outside = temp_dir.path().join("outside");
std::fs::create_dir(&base).expect("create destination base");
std::fs::create_dir_all(outside.join("object")).expect("create outside destination parent");
let src = temp_dir.path().join("staged-object");
std::fs::write(&src, b"payload").expect("write staged object");
let dst = base.join("object").join("xl.meta");
std::fs::remove_dir(&base).expect("remove destination base before replacement");
symlink(&outside, &base).expect("replace destination base with a symlink");
rename_all(&src, &dst, &base)
.await
.expect_err("rename must reject an existing destination parent below a replaced base");
assert!(src.exists(), "rejected rename must preserve the staged source");
assert!(
!outside.join("object/xl.meta").exists(),
"rename must not publish through the replacement symlink"
);
}
#[cfg(windows)]
#[test]
fn windows_parent_guard_blocks_base_and_intermediate_replacement() {
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
std::fs::create_dir(&base).expect("create destination base");
let parent = base.join("object").join("nested");
let guard = mkdir_all_below_existing_base_std(&parent, &base).expect("create and lock destination parents");
std::fs::rename(&base, temp_dir.path().join("replacement-base"))
.expect_err("the locked base must not be replaceable before commit");
std::fs::rename(base.join("object"), base.join("replacement-object"))
.expect_err("a locked intermediate directory must not be replaceable before commit");
drop(guard);
std::fs::rename(base.join("object"), base.join("replacement-object"))
.expect("replacement should succeed after the commit guard is released");
}
#[cfg(unix)]
#[tokio::test]
async fn rename_parent_creation_rejects_symlinked_base() {
use std::os::unix::fs::symlink;
let temp_dir = tempdir().expect("create temp dir");
let outside = temp_dir.path().join("outside");
std::fs::create_dir(&outside).expect("create outside directory");
let base = temp_dir.path().join("bucket");
symlink(&outside, &base).expect("create symlinked base");
mkdir_all_below_existing_base(&base.join("object"), &base)
.await
.expect_err("symlinked base must be rejected");
assert!(!outside.join("object").exists(), "parent creation must remain confined to the base");
}
#[cfg(unix)]
#[tokio::test]
async fn rename_parent_creation_rejects_symlink_below_base() {
use std::os::unix::fs::symlink;
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let outside = temp_dir.path().join("outside");
std::fs::create_dir(&base).expect("create destination base");
std::fs::create_dir(&outside).expect("create outside directory");
symlink(&outside, base.join("linked")).expect("create symlink below base");
mkdir_all_below_existing_base(&base.join("linked/object"), &base)
.await
.expect_err("symlink below base must be rejected");
assert!(
!outside.join("object").exists(),
"parent creation must not follow a symlink outside the base"
);
}
#[tokio::test]
async fn fsync_dir_succeeds_on_directory() {
let temp_dir = tempdir().expect("create temp dir");
+3 -3
View File
@@ -103,7 +103,7 @@ where
/// or `out` is larger than one shard. On error `out`'s contents are
/// unspecified but never contain bytes that failed the hash check — the copy
/// happens only after verification.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
let want = out.len();
self.begin_read(want)?;
@@ -303,7 +303,7 @@ where
/// Write a (hash+data) block. Returns the number of data bytes written.
/// Returns an error if called after a short write or if data exceeds shard_size.
#[cfg_attr(feature = "hotpath", hotpath::measure(label = "BitrotWriter::write"))]
#[hotpath::measure(label = "BitrotWriter::write")]
pub async fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
if buf.is_empty() {
return Ok(0);
@@ -455,7 +455,7 @@ pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorith
/// stores those as whole-file bitrot with no interleaved hash, so the size guard
/// on the next line would reject a genuinely healthy part. Reading legacy V1
/// whole-file-bitrot objects would need a separate verification path.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
mut r: R,
want_size: usize,
+2 -2
View File
@@ -691,7 +691,7 @@ impl<R> ParallelReader<R>
where
R: crate::erasure::coding::ShardSource,
{
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn read(&mut self) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>) {
// On the reconstruction-verifying GET path, read every live shard reader
// in lockstep so all readers advance one block per stripe and stay
@@ -1505,7 +1505,7 @@ where
}
impl Erasure {
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn decode<W, R>(
&self,
writer: &mut W,
+4 -4
View File
@@ -504,7 +504,7 @@ impl Erasure {
Ok((reader, total))
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn encode<R>(
self: Arc<Self>,
reader: R,
@@ -670,7 +670,7 @@ impl Erasure {
Ok((reader, total))
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn encode_batched<R>(
self: Arc<Self>,
mut reader: R,
@@ -798,7 +798,7 @@ impl Erasure {
/// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
/// Reads all data, encodes directly, writes shards sequentially.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn encode_inline_small<R>(
self: Arc<Self>,
reader: R,
@@ -813,7 +813,7 @@ impl Erasure {
/// Fast path for single-block non-inline objects: avoids the producer/consumer
/// pipeline in `encode()` while keeping the same writer/quorum/shutdown semantics.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn encode_single_block_non_inline<R>(
self: Arc<Self>,
reader: R,
+5 -5
View File
@@ -640,7 +640,7 @@ impl Erasure {
/// # Returns
/// A vector of encoded shards as `Bytes`.
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -688,7 +688,7 @@ impl Erasure {
/// Encode owned data, avoiding a copy when the caller already has a heap buffer.
/// Falls back to copying into a new buffer if zero-copy conversion fails.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub fn encode_data_owned(&self, data: Vec<u8>) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -752,7 +752,7 @@ impl Erasure {
/// block), the `resize(need_total_size)` below stays within capacity for every
/// `data_len <= block_size` — both shard-size formulas are monotone in
/// `data_len` — so this function never reallocates the buffer.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub fn encode_data_bytes_mut(&self, mut data_buffer: BytesMut, data_len: usize) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -805,7 +805,7 @@ impl Erasure {
///
/// # Returns
/// Ok if reconstruction succeeds, error otherwise.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
@@ -825,7 +825,7 @@ impl Erasure {
}
/// Decode and reconstruct missing data shards, then regenerate parity shards.
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub fn decode_data_and_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
+12 -1
View File
@@ -203,7 +203,7 @@ fn get_all_sets<T: AsRef<str>>(set_drive_count: usize, is_ellipses: bool, args:
for args in set_args.iter() {
for arg in args {
if unique_args.contains(arg) {
return Err(Error::other(format!("Input args {arg} has duplicate ellipses")));
return Err(Error::other("input arguments contain a duplicate endpoint after ellipsis expansion"));
}
unique_args.insert(arg);
}
@@ -924,4 +924,15 @@ mod test {
}
}
}
#[test]
fn layout_errors_do_not_echo_url_credentials() {
for volumes in [
vec!["http://:duplicate-secret@server/path", "http://:duplicate-secret@server/path"],
vec!["http://:ellipsis...secret@server/path"],
] {
let err = DisksLayout::from_volumes(&volumes).unwrap_err();
assert!(!err.to_string().contains("secret"), "layout error leaked endpoint credentials: {err}");
}
}
}
+19 -2
View File
@@ -88,6 +88,7 @@ impl TryFrom<&str> for Endpoint {
// - All field should be empty except Host and Path.
if !((url.scheme() == "http" || url.scheme() == "https")
&& url.username().is_empty()
&& url.password().is_none()
&& url.fragment().is_none()
&& url.query().is_none())
{
@@ -366,6 +367,12 @@ mod test {
expected_type: None,
expected_err: Some(Error::other("invalid URL endpoint format")),
},
TestCase {
arg: "http://:topsecret@server/path",
expected_endpoint: None,
expected_type: None,
expected_err: Some(Error::other("invalid URL endpoint format")),
},
TestCase {
arg: "http://:/path",
expected_endpoint: None,
@@ -505,8 +512,18 @@ mod test {
let endpoint = Endpoint::try_from("http://example.com:9000/path").unwrap();
assert_eq!(endpoint.host_port(), "example.com:9000");
let endpoint_no_port = Endpoint::try_from("https://example.com/path").unwrap();
assert_eq!(endpoint_no_port.host_port(), "example.com");
for endpoint in [
Endpoint::try_from("http://example.com/path").unwrap(),
Endpoint::try_from("http://example.com:80/path").unwrap(),
] {
assert_eq!(endpoint.host_port(), "example.com");
}
for endpoint in [
Endpoint::try_from("https://example.com/path").unwrap(),
Endpoint::try_from("https://example.com:443/path").unwrap(),
] {
assert_eq!(endpoint.host_port(), "example.com");
}
let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
assert_eq!(file_endpoint.host_port(), "");
File diff suppressed because it is too large Load Diff
+52 -47
View File
@@ -18,7 +18,7 @@ use serde::{Deserialize, Serialize};
use serde_json::Error as JsonError;
use uuid::Uuid;
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
pub enum FormatMetaVersion {
#[serde(rename = "1")]
V1,
@@ -27,7 +27,7 @@ pub enum FormatMetaVersion {
Unknown,
}
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
pub enum FormatBackend {
#[serde(rename = "xl")]
Erasure,
@@ -64,7 +64,7 @@ pub struct FormatErasureV3 {
pub distribution_algo: DistributionAlgoVersion,
}
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
pub enum FormatErasureVersion {
#[serde(rename = "1")]
V1,
@@ -77,7 +77,7 @@ pub enum FormatErasureVersion {
Unknown,
}
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
pub enum DistributionAlgoVersion {
#[serde(rename = "CRCMOD")]
V1,
@@ -121,6 +121,15 @@ pub struct FormatV3 {
pub disk_info: Option<DiskInfo>,
}
pub(crate) type SharedFormatIdentity<'a> = (
&'a FormatMetaVersion,
&'a FormatBackend,
&'a Uuid,
&'a FormatErasureVersion,
&'a [Vec<Uuid>],
&'a DistributionAlgoVersion,
);
impl TryFrom<&[u8]> for FormatV3 {
type Error = JsonError;
@@ -198,52 +207,24 @@ impl FormatV3 {
}
pub fn check_other(&self, other: &FormatV3) -> Result<()> {
let mut tmp = other.clone();
let this = tmp.erasure.this;
tmp.erasure.this = Uuid::nil();
if self.erasure.sets.len() != other.erasure.sets.len() {
return Err(Error::other(format!(
"Expected number of sets {}, got {}",
self.erasure.sets.len(),
other.erasure.sets.len()
)));
if self.shared_identity() != other.shared_identity() {
return Err(Error::other("storage formats do not match"));
}
for i in 0..self.erasure.sets.len() {
if self.erasure.sets[i].len() != other.erasure.sets[i].len() {
return Err(Error::other(format!(
"Each set should be of same size, expected {}, got {}",
self.erasure.sets[i].len(),
other.erasure.sets[i].len()
)));
}
self.find_disk_index_by_disk_id(other.erasure.this).map(|_| ())
}
for j in 0..self.erasure.sets[i].len() {
if self.erasure.sets[i][j] != other.erasure.sets[i][j] {
return Err(Error::other(format!(
"UUID on positions {}:{} do not match with, expected {:?} got {:?}: (%w)",
i,
j,
self.erasure.sets[i][j].to_string(),
other.erasure.sets[i][j].to_string(),
)));
}
}
}
for i in 0..tmp.erasure.sets.len() {
for j in 0..tmp.erasure.sets[i].len() {
if this == tmp.erasure.sets[i][j] {
return Ok(());
}
}
}
Err(Error::other(format!(
"DriveID {:?} not found in any drive sets {:?}",
this, other.erasure.sets
)))
/// Fields that must agree across every disk in one erasure format,
/// excluding the disk-specific `this` UUID and runtime-only `disk_info`.
pub(crate) fn shared_identity(&self) -> SharedFormatIdentity<'_> {
(
&self.version,
&self.format,
&self.id,
&self.erasure.version,
&self.erasure.sets,
&self.erasure.distribution_algo,
)
}
}
@@ -437,6 +418,30 @@ mod test {
assert!(result.is_ok());
}
#[test]
fn test_check_other_rejects_shared_identity_mismatches() {
type FormatMutation = (&'static str, fn(&mut FormatV3));
let format = FormatV3::new(1, 2);
let mutations: [FormatMutation; 5] = [
("meta version", |other| other.version = FormatMetaVersion::Unknown),
("backend", |other| other.format = FormatBackend::ErasureSingle),
("deployment id", |other| other.id = Uuid::new_v4()),
("erasure version", |other| other.erasure.version = FormatErasureVersion::V2),
("distribution algorithm", |other| {
other.erasure.distribution_algo = DistributionAlgoVersion::V2
}),
];
for (field, mutate) in mutations {
let mut other = format.clone();
other.erasure.this = format.erasure.sets[0][0];
mutate(&mut other);
assert!(format.check_other(&other).is_err(), "{field} mismatch must be rejected");
}
}
#[test]
fn test_check_other_different_set_count() {
let format1 = FormatV3::new(2, 4);
+6 -1
View File
@@ -12,8 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#![recursion_limit = "256"]
/// Scope-based hotpath measurement for `#[async_trait]` methods, where
/// `#[cfg_attr(feature = "hotpath", hotpath::measure)]` would only time the boxed-future construction.
/// `#[hotpath::measure]` would only time the boxed-future construction.
/// The guard records wall time from this statement until the enclosing
/// (desugared) async block completes, including early returns via `?`.
#[cfg(feature = "hotpath")]
@@ -93,3 +95,6 @@ pub(crate) mod ecstore_validation_blackbox;
#[cfg(test)]
pub(crate) mod test_metrics;
#[cfg(test)]
pub(crate) mod test_tracing;
@@ -0,0 +1,80 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use async_trait::async_trait;
use http::{HeaderMap, HeaderValue};
use std::collections::HashMap;
use std::error::Error;
use std::fmt::{Display, Formatter};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadEncryptionMode {
Direct { base_nonce: [u8; 12] },
Object,
}
pub struct ReadEncryptionMaterial {
pub key_bytes: [u8; 32],
pub mode: ReadEncryptionMode,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EncryptionResolutionErrorKind {
InvalidRequest,
InvalidMetadata,
ServiceUnavailable,
DecryptionFailed,
}
#[derive(Debug)]
pub struct EncryptionResolutionError {
kind: EncryptionResolutionErrorKind,
message: String,
}
impl EncryptionResolutionError {
pub fn new(kind: EncryptionResolutionErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
pub fn kind(&self) -> EncryptionResolutionErrorKind {
self.kind
}
}
impl Display for EncryptionResolutionError {
fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.message)
}
}
impl Error for EncryptionResolutionError {}
pub struct ReadEncryptionRequest<'a> {
pub bucket: &'a str,
pub object: &'a str,
pub metadata: &'a HashMap<String, String>,
pub headers: &'a HeaderMap<HeaderValue>,
}
#[async_trait]
pub trait ObjectEncryptionResolver: Send + Sync {
async fn resolve_read_material(
&self,
request: ReadEncryptionRequest<'_>,
) -> Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError>;
}
+5
View File
@@ -84,6 +84,7 @@ pub(crate) fn legacy_encrypted_range_seek_enabled() -> bool {
}
mod body_cache_hook;
mod encryption;
mod hook_slot;
mod object_mutation_hook;
mod readers;
@@ -98,6 +99,10 @@ pub use body_cache_hook::{
pub(crate) use body_cache_hook::{
get_object_body_cache_hook, get_object_body_cache_hook_suppressed, without_get_object_body_cache_hook,
};
pub use encryption::{
EncryptionResolutionError, EncryptionResolutionErrorKind, ObjectEncryptionResolver, ReadEncryptionMaterial,
ReadEncryptionMode, ReadEncryptionRequest,
};
pub(crate) use object_mutation_hook::notify_object_mutation;
pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
pub use readers::*;
File diff suppressed because it is too large Load Diff
+17 -46
View File
@@ -273,29 +273,9 @@ impl ObjectInfo {
}
pub fn is_encrypted(&self) -> bool {
// Corresponding to the logic in rustfs/src/sse.rs/encryption_material_to_metadata function
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
self.user_defined.keys().any(|key| {
let lower = key.to_ascii_lowercase();
lower.starts_with("x-minio-encryption-")
|| lower.starts_with("x-minio-internal-server-side-encryption-")
|| matches!(
lower.as_str(),
"x-minio-internal-encrypted-multipart"
| "x-rustfs-encryption-key"
| "x-rustfs-encryption-algorithm"
| "x-rustfs-encryption-iv"
| "x-rustfs-encryption-key-id"
| "x-rustfs-encryption-context"
| "x-rustfs-encryption-tag"
| "x-amz-server-side-encryption-aws-kms-key-id"
| SSEC_ALGORITHM_HEADER
| SSEC_KEY_HEADER
| SSEC_KEY_MD5_HEADER
| "x-amz-server-side-encryption"
)
})
self.user_defined
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
}
/// Maximum inline size for non-versioned objects (128 KiB).
@@ -339,26 +319,7 @@ impl ObjectInfo {
}
pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
let actual_size = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE);
if let Some(size_str) = self
.user_defined
.get("x-rustfs-encryption-original-size")
.map(String::as_str)
.or_else(|| {
self.user_defined
.get("x-amz-server-side-encryption-customer-original-size")
.map(String::as_str)
})
.or(actual_size.as_deref())
&& !size_str.is_empty()
{
let size = size_str
.parse::<i64>()
.map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
return Ok(Some(size));
}
Ok(None)
rustfs_utils::http::get_object_encryption_original_size(&self.user_defined)
}
pub fn decrypted_size(&self) -> std::io::Result<i64> {
@@ -388,9 +349,6 @@ impl ObjectInfo {
return Ok(actual_size);
}
// Check if object is encrypted
// Managed SSE stores original size in x-rustfs-encryption-original-size metadata
// SSE-C stores original size in x-amz-server-side-encryption-customer-original-size
if let Some(size) = self.encryption_original_size()? {
return Ok(size);
}
@@ -881,6 +839,19 @@ mod tests {
assert!(!object.is_inline_fast_path_eligible(), "transitioned objects must fall back");
}
#[test]
fn minio_internal_encryption_metadata_is_not_treated_as_plaintext() {
let object = ObjectInfo {
user_defined: Arc::new(HashMap::from([(
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key".to_string(),
"sealed".to_string(),
)])),
..Default::default()
};
assert!(object.is_encrypted());
}
#[test]
fn versions_after_marker_handles_null_version_marker() {
let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
+17
View File
@@ -46,6 +46,7 @@ use crate::bucket::metadata_sys::BucketMetadataSys;
use crate::bucket::replication::{DynReplicationPool, ReplicationStats};
use crate::disk::DiskStore;
use crate::layout::endpoints::{EndpointServerPools, SetupType};
use crate::object_api::ObjectEncryptionResolver;
use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
@@ -159,6 +160,8 @@ pub struct InstanceContext {
/// workers (scanner/heal/tier/lifecycle) without touching another instance.
/// Replaces the process-global cancel-token static.
background_cancel_token: OnceLock<CancellationToken>,
/// Resolves object-encryption material at the application boundary.
object_encryption_resolver: OnceLock<Arc<dyn ObjectEncryptionResolver>>,
tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
#[cfg(test)]
@@ -197,6 +200,7 @@ impl InstanceContext {
local_disk_set_drives: Arc::new(RwLock::new(Vec::new())),
bucket_metadata_sys: std::sync::Mutex::new(None),
background_cancel_token: OnceLock::new(),
object_encryption_resolver: OnceLock::new(),
tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
#[cfg(test)]
@@ -209,6 +213,19 @@ impl InstanceContext {
self.lock_manager.clone()
}
/// Install the application-owned object-encryption resolver once.
pub fn set_object_encryption_resolver(
&self,
resolver: Arc<dyn ObjectEncryptionResolver>,
) -> Result<(), Arc<dyn ObjectEncryptionResolver>> {
self.object_encryption_resolver.set(resolver)
}
/// Return the configured object-encryption resolver, if startup installed one.
pub fn object_encryption_resolver(&self) -> Option<&dyn ObjectEncryptionResolver> {
self.object_encryption_resolver.get().map(Arc::as_ref)
}
/// Set this instance's S3 region.
///
/// Write-once: panics on a second write, preserving the startup fail-fast
+132 -12
View File
@@ -27,7 +27,7 @@ use crate::{
bucket::replication::{DynReplicationPool, ReplicationStats},
config::{get_global_storage_class, get_global_storage_class_snapshot, set_global_storage_class, storageclass},
disk::{DiskAPI, DiskOption, DiskStore, new_disk},
error::Result,
error::{Error, Result},
layout::endpoints::{EndpointServerPools, SetupType},
runtime::global::{
GLOBAL_BOOT_TIME, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
@@ -46,7 +46,6 @@ use crate::{
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_kms::{ObjectEncryptionService, get_global_encryption_service};
use rustfs_lock::client::LockClient;
use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
@@ -105,10 +104,6 @@ pub(crate) fn record_erasure_write_quorum_failure(stage: &'static str, dominant_
global_internode_metrics().record_erasure_write_quorum_failure(stage, dominant_error);
}
pub(crate) async fn object_encryption_service() -> Option<Arc<ObjectEncryptionService>> {
get_global_encryption_service().await
}
pub fn object_store_handle() -> Option<Arc<ECStore>> {
resolve_object_store_handle()
}
@@ -417,14 +412,14 @@ pub(crate) async fn clear_local_disk_id_map_for_test() {
local_disk_id_map_handle().write().await.clear();
}
pub(crate) async fn record_local_disk_id(instance_ctx: &Arc<InstanceContext>, disk_id: Uuid, endpoint: String) {
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
}
pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Option<Uuid>, endpoint: String) {
let id_map = local_disk_id_map_handle();
let mut disk_id_map = id_map.write().await;
if let Some(previous_id) = previous {
if let Some(previous_id) = previous
&& disk_id_map
.get(&previous_id)
.is_some_and(|registered_endpoint| registered_endpoint == &endpoint)
{
disk_id_map.remove(&previous_id);
}
if let Some(current_id) = current {
@@ -432,6 +427,53 @@ pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Optio
}
}
pub(crate) async fn reconcile_local_disk_ids(
instance_ctx: &InstanceContext,
pool_endpoints: &[String],
selected: &[(Uuid, String)],
) {
let pool_endpoints = pool_endpoints.iter().map(String::as_str).collect::<HashSet<_>>();
let disk_id_map = instance_ctx.local_disk_id_map();
let mut disk_ids = disk_id_map.write().await;
disk_ids.retain(|_, registered_endpoint| !pool_endpoints.contains(registered_endpoint.as_str()));
disk_ids.extend(selected.iter().cloned());
}
pub(crate) async fn quarantine_local_disks(instance_ctx: &InstanceContext, endpoints: &[Endpoint]) -> Result<()> {
let slots = endpoints
.iter()
.map(|endpoint| {
Ok((
usize::try_from(endpoint.pool_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.set_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.disk_idx).map_err(|_| Error::CorruptedFormat)?,
))
})
.collect::<Result<Vec<_>>>()?;
let local_disk_map = instance_ctx.local_disk_map();
let mut local_disks = local_disk_map.write().await;
for endpoint in endpoints {
local_disks.insert(endpoint.to_string(), None);
}
drop(local_disks);
let set_drives = instance_ctx.local_disk_set_drives();
let mut local_set_drives = set_drives.write().await;
if local_set_drives.is_empty() {
return Ok(());
}
for (pool_idx, set_idx, disk_idx) in slots {
let disk = local_set_drives
.get_mut(pool_idx)
.and_then(|sets| sets.get_mut(set_idx))
.and_then(|disks| disks.get_mut(disk_idx))
.ok_or(Error::CorruptedFormat)?;
*disk = None;
}
Ok(())
}
pub(crate) async fn record_local_disks(instance_ctx: &Arc<InstanceContext>, disks: Vec<DiskStore>) {
let map = instance_ctx.local_disk_map();
let mut global_local_disk_map = map.write().await;
@@ -558,10 +600,14 @@ pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
#[cfg(test)]
mod tests {
use super::{LockRegistry, local_node_name, set_local_node_name};
use super::{
LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, reconcile_local_disk_ids,
replace_local_disk_id, set_local_node_name,
};
use crate::disk::endpoint::Endpoint;
use rustfs_lock::{LocalClient, LockClient};
use std::{collections::HashMap, sync::Arc};
use uuid::Uuid;
fn url_endpoint(raw: &str) -> Endpoint {
Endpoint {
@@ -607,4 +653,78 @@ mod tests {
assert_eq!(observed, next);
}
#[tokio::test]
#[serial_test::serial]
async fn clearing_a_stale_disk_id_does_not_remove_another_endpoint() {
clear_local_disk_id_map_for_test().await;
let disk_id = Uuid::new_v4();
replace_local_disk_id(None, Some(disk_id), "endpoint-a".to_string()).await;
replace_local_disk_id(Some(disk_id), None, "endpoint-b".to_string()).await;
assert_eq!(local_disk_path_by_id(&disk_id).await, Some("endpoint-a".to_string()));
clear_local_disk_id_map_for_test().await;
}
#[tokio::test]
#[serial_test::serial]
async fn reconciling_pool_disk_ids_preserves_other_endpoints() {
let instance_ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let process_ctx = crate::runtime::global::current_ctx();
let bootstrap_ctx = crate::runtime::instance::bootstrap_ctx();
let retained_id = Uuid::new_v4();
let removed_id = Uuid::new_v4();
let selected_id = Uuid::new_v4();
let process_sentinel = Uuid::new_v4();
let bootstrap_sentinel = Uuid::new_v4();
instance_ctx.local_disk_id_map().write().await.extend([
(retained_id, "endpoint-a".to_string()),
(removed_id, "endpoint-b".to_string()),
]);
process_ctx
.local_disk_id_map()
.write()
.await
.insert(process_sentinel, "endpoint-b".to_string());
bootstrap_ctx
.local_disk_id_map()
.write()
.await
.insert(bootstrap_sentinel, "endpoint-b".to_string());
reconcile_local_disk_ids(
&instance_ctx,
&["endpoint-b".to_string(), "endpoint-c".to_string()],
&[(selected_id, "endpoint-c".to_string())],
)
.await;
let disk_ids = instance_ctx.local_disk_id_map();
let disk_ids = disk_ids.read().await;
assert_eq!(disk_ids.get(&retained_id).map(String::as_str), Some("endpoint-a"));
assert_eq!(disk_ids.get(&removed_id), None);
assert_eq!(disk_ids.get(&selected_id).map(String::as_str), Some("endpoint-c"));
drop(disk_ids);
assert_eq!(
process_ctx
.local_disk_id_map()
.read()
.await
.get(&process_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
assert_eq!(
bootstrap_ctx
.local_disk_id_map()
.read()
.await
.get(&bootstrap_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
process_ctx.local_disk_id_map().write().await.remove(&process_sentinel);
bootstrap_ctx.local_disk_id_map().write().await.remove(&bootstrap_sentinel);
}
}
+388 -11
View File
@@ -29,11 +29,11 @@ use rustfs_madmin::metrics::RealtimeMetrics;
use rustfs_madmin::net::NetInfo;
use rustfs_madmin::{ItemState, ServerProperties, StorageInfo};
use rustfs_utils::XHost;
use std::collections::{HashMap, hash_map::DefaultHasher};
use std::collections::{BTreeMap, HashMap, hash_map::DefaultHasher};
use std::future::Future;
use std::hash::{Hash, Hasher};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::{Duration, SystemTime};
use std::time::{Duration, Instant, SystemTime};
use tokio::time::{sleep, timeout};
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, info, warn};
@@ -47,6 +47,9 @@ const EVENT_NOTIFICATION_PEER_PROPAGATION: &str = "notification_peer_propagation
const SCANNER_ACTIVITY_PROBE_TIMEOUT: Duration = Duration::from_secs(5);
const TIER_CONFIG_RELOAD_RETRY_BASE: Duration = Duration::from_millis(100);
const TIER_CONFIG_RELOAD_RETRY_CAP: Duration = Duration::from_secs(5);
const REMOTE_VERSION_STATE_PROBE_INTERVAL: Duration = Duration::from_secs(10);
const REMOTE_VERSION_STATE_PROBE_TIMEOUT: Duration = Duration::from_secs(5);
const REMOTE_VERSION_STATE_PROOF_TTL: Duration = Duration::from_secs(30);
/// Cached result from the last successful admin call to a peer.
struct PeerAdminCache {
@@ -91,6 +94,180 @@ lazy_static! {
pub static ref GLOBAL_NOTIFICATION_SYS: OnceLock<Arc<NotificationSys>> = OnceLock::new();
}
#[derive(Clone)]
struct RemoteVersionStateFleetProof {
topology_fingerprint: String,
peer_epochs: Arc<BTreeMap<String, Uuid>>,
expires_at: Instant,
}
impl RemoteVersionStateFleetProof {
fn token(&self) -> RemoteVersionStateFleetProofToken {
RemoteVersionStateFleetProofToken {
topology_fingerprint: self.topology_fingerprint.clone(),
peer_epochs: self.peer_epochs.clone(),
}
}
}
#[derive(Clone, PartialEq, Eq)]
pub(crate) struct RemoteVersionStateFleetProofToken {
topology_fingerprint: String,
peer_epochs: Arc<BTreeMap<String, Uuid>>,
}
#[derive(Default)]
struct RemoteVersionStateFleetProofState {
proof: Option<RemoteVersionStateFleetProof>,
topology_conflict: bool,
}
static REMOTE_VERSION_STATE_FLEET_PROOF: OnceLock<std::sync::RwLock<RemoteVersionStateFleetProofState>> = OnceLock::new();
static REMOTE_VERSION_STATE_PROBE_TOPOLOGY: OnceLock<String> = OnceLock::new();
fn remote_version_state_fleet_proof_slot() -> &'static std::sync::RwLock<RemoteVersionStateFleetProofState> {
REMOTE_VERSION_STATE_FLEET_PROOF.get_or_init(|| std::sync::RwLock::new(RemoteVersionStateFleetProofState::default()))
}
fn replace_remote_version_state_fleet_proof(proof: Option<RemoteVersionStateFleetProof>) {
replace_remote_version_state_fleet_proof_in(remote_version_state_fleet_proof_slot(), proof);
}
fn replace_remote_version_state_fleet_proof_in(
slot: &std::sync::RwLock<RemoteVersionStateFleetProofState>,
proof: Option<RemoteVersionStateFleetProof>,
) {
slot.write().unwrap_or_else(std::sync::PoisonError::into_inner).proof = proof;
}
fn publish_remote_version_state_probe_result(
slot: &std::sync::RwLock<RemoteVersionStateFleetProofState>,
topology_fingerprint: &str,
result: Result<BTreeMap<String, Uuid>>,
observed_at: Instant,
) -> Option<Error> {
match result {
Ok(peer_epochs) => {
let mut state = slot.write().unwrap_or_else(std::sync::PoisonError::into_inner);
let peer_epochs = state
.proof
.as_ref()
.filter(|proof| proof.topology_fingerprint == topology_fingerprint && proof.peer_epochs.as_ref() == &peer_epochs)
.map(|proof| Arc::clone(&proof.peer_epochs))
.unwrap_or_else(|| Arc::new(peer_epochs));
state.proof = Some(RemoteVersionStateFleetProof {
topology_fingerprint: topology_fingerprint.to_string(),
peer_epochs,
expires_at: observed_at + REMOTE_VERSION_STATE_PROOF_TTL,
});
None
}
Err(err) => {
replace_remote_version_state_fleet_proof_in(slot, None);
Some(err)
}
}
}
pub(crate) fn acquire_remote_version_state_fleet_proof() -> Option<RemoteVersionStateFleetProofToken> {
let expected_topology = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get()?;
let state = remote_version_state_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
acquire_remote_version_state_fleet_proof_from(&state, expected_topology, Instant::now())
}
fn acquire_remote_version_state_fleet_proof_from(
state: &RemoteVersionStateFleetProofState,
expected_topology: &str,
now: Instant,
) -> Option<RemoteVersionStateFleetProofToken> {
if state.topology_conflict || !remote_version_state_fleet_proof_valid_at(state.proof.as_ref(), expected_topology, now) {
return None;
}
state.proof.as_ref().map(RemoteVersionStateFleetProof::token)
}
pub(crate) fn remote_version_state_fleet_proof_matches(proof: &RemoteVersionStateFleetProofToken) -> bool {
let Some(expected_topology) = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() else {
return false;
};
let state = remote_version_state_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if state.topology_conflict {
return false;
}
state.proof.as_ref().is_some_and(|current| {
current.topology_fingerprint == *expected_topology
&& current.topology_fingerprint == proof.topology_fingerprint
&& Arc::ptr_eq(&current.peer_epochs, &proof.peer_epochs)
&& Instant::now() < current.expires_at
})
}
fn remote_version_state_fleet_proof_valid_at(
proof: Option<&RemoteVersionStateFleetProof>,
expected_topology: &str,
now: Instant,
) -> bool {
proof.is_some_and(|proof| proof.topology_fingerprint == expected_topology && now < proof.expires_at)
}
fn insert_remote_version_state_peer(peer_epochs: &mut BTreeMap<String, Uuid>, peer: String, epoch: Uuid) -> Result<()> {
if epoch.is_nil() || peer_epochs.values().any(|existing| *existing == epoch) || peer_epochs.insert(peer, epoch).is_some() {
return Err(Error::other("remote version state capability peer identity is invalid"));
}
Ok(())
}
pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY.set(topology_fingerprint.clone()).is_err() {
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() != Some(&topology_fingerprint) {
let mut state = remote_version_state_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.topology_conflict = true;
state.proof = None;
}
return;
}
tokio::spawn(async move {
loop {
let result = match get_global_notification_sys() {
Some(notification_sys) => {
match timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_remote_version_state_fleet(&topology_fingerprint),
)
.await
{
Ok(result) => result,
Err(_) => Err(Error::other("remote version state fleet capability probe timed out")),
}
}
None => Err(Error::other("remote version state fleet capability notification system is unavailable")),
};
let topology_conflict = remote_version_state_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.topology_conflict;
if topology_conflict {
replace_remote_version_state_fleet_proof(None);
} else if let Some(err) = publish_remote_version_state_probe_result(
remote_version_state_fleet_proof_slot(),
&topology_fingerprint,
result,
Instant::now(),
) {
debug!(error = %err, "remote version state fleet capability probe failed closed");
}
sleep(REMOTE_VERSION_STATE_PROBE_INTERVAL).await;
}
});
}
pub async fn new_global_notification_sys(eps: EndpointServerPools) -> Result<()> {
let _ = GLOBAL_NOTIFICATION_SYS
.set(Arc::new(NotificationSys::new(eps).await))
@@ -115,7 +292,17 @@ pub struct NotificationSys {
impl NotificationSys {
pub async fn new(eps: EndpointServerPools) -> Self {
let expected_remote_hosts = eps
.peer_grid_host_slots_sorted()
.into_iter()
.filter_map(|(peer, _, is_local)| (!is_local).then_some(peer))
.collect::<Vec<_>>();
let (peer_clients, all_peer_clients, peer_topology_hosts) = PeerRestClient::new_clients_with_topology(eps).await;
let peer_topology_hosts = if peer_topology_hosts.is_empty() {
expected_remote_hosts
} else {
peer_topology_hosts
};
let peer_admin_caches = (0..peer_clients.len()).map(|_| Mutex::new(PeerAdminCache::new())).collect();
Self {
peer_clients,
@@ -125,6 +312,24 @@ impl NotificationSys {
tier_config_reload_workers: Default::default(),
}
}
async fn probe_remote_version_state_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
if self.peer_clients.len() != self.peer_topology_hosts.len() {
return Err(Error::other("remote version state capability fleet membership is incomplete"));
}
let probes = self.peer_clients.iter().map(|client| async {
let client = client
.as_ref()
.ok_or_else(|| Error::other("remote version state capability peer is unreachable"))?;
client.probe_remote_version_state(topology_fingerprint.to_string()).await
});
let mut peer_epochs = BTreeMap::new();
for result in join_all(probes).await {
let (peer, epoch) = result?;
insert_remote_version_state_peer(&mut peer_epochs, peer, epoch)?;
}
Ok(peer_epochs)
}
}
pub struct NotificationPeerErr {
@@ -1839,15 +2044,10 @@ fn synthesized_disks(host: &str, endpoints: &EndpointServerPools, state: ItemSta
/// Whether `peer_host` refers to the same node as an endpoint whose
/// `host_port()` is `ep_host_port`.
///
/// `PeerRestClient::host` is an `XHost`, which resolves names to an address on
/// construction (`hosts_sorted` -> `XHost::try_from` -> `to_socket_addrs`), so
/// `peer_host` is the resolved `IP:port`. An endpoint's `host_port()`, however,
/// is `url.host():port` — still the raw `hostname:port` on hostname-based
/// deployments. A plain string compare therefore misses on hostname clusters,
/// leaving the synthesized/degraded drive list empty and `unknownDisks` at 0
/// (rustfs/rustfs#4607 follow-up). Compare directly first (fast path / IP
/// deployments), then canonicalize the endpoint side through the same `XHost`
/// resolution and compare again.
/// Current topology clients preserve the endpoint `hostname:port`, so the
/// direct comparison is the normal path. The resolution fallback keeps
/// compatibility with older or manually constructed clients whose `XHost`
/// contains a resolved `IP:port` (rustfs/rustfs#4607 follow-up).
fn endpoint_host_matches(peer_host: &str, ep_host_port: &str) -> bool {
if peer_host == ep_host_port {
return true;
@@ -1890,6 +2090,183 @@ fn aggregate_scanner_dirty_usage_acknowledgement_results(
mod tests {
use super::*;
#[test]
fn remote_version_state_fleet_proof_rejects_stale_or_mismatched_membership() {
let now = Instant::now();
let mut peer_epochs = BTreeMap::new();
peer_epochs.insert("peer-a".to_string(), Uuid::new_v4());
let proof = RemoteVersionStateFleetProof {
topology_fingerprint: "topology-a".to_string(),
peer_epochs: Arc::new(peer_epochs),
expires_at: now + Duration::from_secs(1),
};
assert!(remote_version_state_fleet_proof_valid_at(Some(&proof), "topology-a", now));
assert!(!remote_version_state_fleet_proof_valid_at(Some(&proof), "topology-b", now));
assert!(!remote_version_state_fleet_proof_valid_at(Some(&proof), "topology-a", proof.expires_at));
assert!(!remote_version_state_fleet_proof_valid_at(None, "topology-a", now));
}
#[test]
fn remote_version_state_fleet_proof_rejects_nil_process_epoch() {
let mut peer_epochs = BTreeMap::new();
assert!(insert_remote_version_state_peer(&mut peer_epochs, "peer-a".to_string(), Uuid::nil()).is_err());
assert!(peer_epochs.is_empty());
}
#[test]
fn remote_version_state_fleet_proof_accepts_single_node_membership() {
let now = Instant::now();
let proof = RemoteVersionStateFleetProof {
topology_fingerprint: "topology-a".to_string(),
peer_epochs: Arc::new(BTreeMap::new()),
expires_at: now + Duration::from_secs(1),
};
assert!(remote_version_state_fleet_proof_valid_at(Some(&proof), "topology-a", now));
}
#[test]
fn remote_version_state_fleet_proof_token_changes_with_process_epoch() {
let now = Instant::now();
let proof = RemoteVersionStateFleetProof {
topology_fingerprint: "topology-a".to_string(),
peer_epochs: Arc::new(BTreeMap::from([("peer-a".to_string(), Uuid::new_v4())])),
expires_at: now + Duration::from_secs(1),
};
let captured = proof.token();
let restarted = RemoteVersionStateFleetProof {
topology_fingerprint: proof.topology_fingerprint.clone(),
peer_epochs: Arc::new(BTreeMap::from([("peer-a".to_string(), Uuid::new_v4())])),
expires_at: proof.expires_at,
};
assert!(captured != restarted.token());
}
#[test]
fn remote_version_state_fleet_proof_renewal_preserves_only_same_epoch_token() {
let slot = std::sync::RwLock::new(RemoteVersionStateFleetProofState::default());
let now = Instant::now();
let epoch = Uuid::new_v4();
let peers = BTreeMap::from([("peer-a".to_string(), epoch)]);
assert!(publish_remote_version_state_probe_result(&slot, "topology-a", Ok(peers.clone()), now).is_none());
let original = slot
.read()
.expect("proof slot should not poison")
.proof
.as_ref()
.expect("successful probe should publish proof")
.token();
assert!(
publish_remote_version_state_probe_result(&slot, "topology-a", Ok(peers), now + Duration::from_millis(1)).is_none()
);
let renewed = slot
.read()
.expect("proof slot should not poison")
.proof
.as_ref()
.expect("renewal should retain proof")
.token();
assert!(Arc::ptr_eq(&original.peer_epochs, &renewed.peer_epochs));
let restarted = BTreeMap::from([("peer-a".to_string(), Uuid::new_v4())]);
assert!(
publish_remote_version_state_probe_result(&slot, "topology-a", Ok(restarted), now + Duration::from_millis(2))
.is_none()
);
let replaced = slot
.read()
.expect("proof slot should not poison")
.proof
.as_ref()
.expect("restarted peer should publish a new proof")
.token();
assert!(!Arc::ptr_eq(&original.peer_epochs, &replaced.peer_epochs));
}
#[test]
fn remote_version_state_fleet_proof_conflict_revokes_atomic_snapshot() {
let now = Instant::now();
let mut state = RemoteVersionStateFleetProofState {
proof: Some(RemoteVersionStateFleetProof {
topology_fingerprint: "topology-a".to_string(),
peer_epochs: Arc::new(BTreeMap::new()),
expires_at: now + Duration::from_secs(1),
}),
topology_conflict: false,
};
assert!(acquire_remote_version_state_fleet_proof_from(&state, "topology-a", now).is_some());
state.topology_conflict = true;
assert!(acquire_remote_version_state_fleet_proof_from(&state, "topology-a", now).is_none());
}
#[test]
fn remote_version_state_fleet_probe_rejects_duplicate_member_or_process_epoch() {
let epoch = Uuid::new_v4();
let mut peer_epochs = BTreeMap::new();
insert_remote_version_state_peer(&mut peer_epochs, "node-a:9000".to_string(), epoch)
.expect("first member should be admitted");
assert!(insert_remote_version_state_peer(&mut peer_epochs, "node-b:9000".to_string(), epoch).is_err());
assert!(insert_remote_version_state_peer(&mut peer_epochs, "node-a:9000".to_string(), Uuid::new_v4()).is_err());
assert!(insert_remote_version_state_peer(&mut peer_epochs, "node-c:9000".to_string(), Uuid::nil()).is_err());
}
#[test]
fn remote_version_state_fleet_probe_failure_revokes_previous_proof() {
let slot = std::sync::RwLock::new(RemoteVersionStateFleetProofState::default());
let now = Instant::now();
let peer_epochs = BTreeMap::from([("node-a:9000".to_string(), Uuid::new_v4())]);
assert!(publish_remote_version_state_probe_result(&slot, "topology-a", Ok(peer_epochs), now).is_none());
assert!(slot.read().expect("proof slot should not poison").proof.is_some());
assert!(
publish_remote_version_state_probe_result(&slot, "topology-a", Err(Error::other("peer unavailable")), now,).is_some()
);
assert!(slot.read().expect("proof slot should not poison").proof.is_none());
let peer_epochs = BTreeMap::from([("node-a:9000".to_string(), Uuid::new_v4())]);
assert!(publish_remote_version_state_probe_result(&slot, "topology-a", Ok(peer_epochs), now).is_none());
assert!(slot.read().expect("proof slot should not poison").proof.is_some());
}
#[tokio::test]
async fn remote_version_state_fleet_probe_rejects_unreachable_member() {
let notification_sys = NotificationSys {
peer_clients: vec![None],
all_peer_clients: vec![None, None],
peer_topology_hosts: vec!["peer-a".to_string()],
peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
tier_config_reload_workers: Default::default(),
};
let err = notification_sys
.probe_remote_version_state_fleet("topology-a")
.await
.expect_err("an unreachable configured member must fail the fleet proof");
assert!(err.to_string().contains("unreachable"));
}
#[tokio::test]
async fn remote_version_state_fleet_probe_rejects_missing_member_slot() {
let notification_sys = NotificationSys {
peer_clients: Vec::new(),
all_peer_clients: vec![None],
peer_topology_hosts: vec!["peer-a".to_string()],
peer_admin_caches: Vec::new(),
tier_config_reload_workers: Default::default(),
};
let err = notification_sys
.probe_remote_version_state_fleet("topology-a")
.await
.expect_err("a missing configured member slot must fail the fleet proof");
assert!(err.to_string().contains("incomplete"));
}
fn build_props(endpoint: &str) -> ServerProperties {
ServerProperties {
endpoint: endpoint.to_string(),
+110 -3
View File
@@ -72,6 +72,7 @@ use crate::{
cluster::rpc::peer_rest_client::{PeerRestClient, PeerTierMutationState},
config::com::{CONFIG_PREFIX, read_config, read_config_with_metadata},
disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET},
layout::endpoints::EndpointServerPools,
object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader},
runtime::sources as runtime_sources,
set_disk::get_lock_acquire_timeout,
@@ -236,6 +237,7 @@ struct TierPublishTransition {
struct PreparedTierDriver {
tier_name: String,
tier_config: TierConfig,
config_fingerprint: TierDriverFingerprint,
backend_identity: TierDestinationId,
exact_get_delete: bool,
@@ -903,14 +905,17 @@ async fn remote_tier_mutation_peers() -> io::Result<Vec<Arc<dyn TierMutationPeer
let Some(endpoints) = runtime_sources::endpoint_pools() else {
return Err(tier_mutation_replay_error("cluster endpoint topology is not initialized"));
};
let remote_host_count = endpoints.hosts_sorted().iter().flatten().count();
let (peers, _) = PeerRestClient::new_clients(endpoints).await;
remote_tier_mutation_peers_from_topology(endpoints).await
}
async fn remote_tier_mutation_peers_from_topology(endpoints: EndpointServerPools) -> io::Result<Vec<Arc<dyn TierMutationPeer>>> {
let (peers, _, remote_topology_hosts) = PeerRestClient::new_clients_with_topology(endpoints).await;
let peers = peers
.into_iter()
.flatten()
.map(|peer| Arc::new(peer) as Arc<dyn TierMutationPeer>)
.collect::<Vec<_>>();
ensure_complete_tier_mutation_commit_peer_set(peers.len(), remote_host_count)?;
ensure_complete_tier_mutation_commit_peer_set(peers.len(), remote_topology_hosts.len())?;
Ok(peers)
}
@@ -1342,6 +1347,7 @@ fn tier_exact_get_delete(config: &TierConfig) -> bool {
struct TierDriverGeneration {
tier_name: Arc<str>,
tier_config: TierConfig,
generation: DriverRevision,
// Process-local only: this may reflect credential changes and must never be persisted or logged.
config_fingerprint: TierDriverFingerprint,
@@ -1349,6 +1355,9 @@ struct TierDriverGeneration {
backend_identity: TierDestinationId,
exact_get_delete: bool,
driver: SharedWarmBackend,
reconciler: tokio::sync::OnceCell<
Option<Arc<dyn crate::services::tier::warm_backend::TransitionCandidateReconciler + Send + Sync + 'static>>,
>,
accepting: AtomicBool,
active_leases: AtomicUsize,
drained: tokio::sync::Notify,
@@ -1473,6 +1482,35 @@ impl TierOperationLease {
self.inner.backend_identity
}
pub(crate) async fn probe_transition_candidate_for(
&self,
object: &str,
transaction_id: uuid::Uuid,
) -> io::Result<TransitionCandidateProbe> {
let Some(reconciler) = self
.inner
.reconciler
.get_or_try_init(|| async {
crate::services::tier::warm_backend::new_transition_candidate_reconciler(&self.inner.tier_config)
.await
.map(|reconciler| reconciler.map(Arc::from))
})
.await
.map_err(|err| io::Error::other(err.message))?
else {
return self.inner.driver.probe_transition_candidate(object).await;
};
reconciler
.probe_transition_candidate_for(
object,
crate::services::tier::warm_backend::TransitionCandidateIdentity {
transaction_id,
destination_id: self.backend_identity(),
},
)
.await
}
pub(crate) fn validate_remote_version_id(&self, remote_version_id: &str) -> io::Result<()> {
self.inner.driver.validate_remote_version_id(remote_version_id)?;
if !remote_version_id.is_empty() && !self.inner.exact_get_delete {
@@ -2833,6 +2871,7 @@ impl TierConfigMgr {
let exact_get_delete = tier_exact_get_delete(config);
Some(PreparedTierDriver {
tier_name: tier_name.to_string(),
tier_config: config.clone(),
config_fingerprint,
backend_identity,
exact_get_delete,
@@ -2861,11 +2900,13 @@ impl TierConfigMgr {
})?;
let entry = Arc::new(TierDriverGeneration {
tier_name: Arc::from(prepared.tier_name.as_str()),
tier_config: prepared.tier_config.clone(),
generation,
config_fingerprint: prepared.config_fingerprint,
backend_identity: prepared.backend_identity,
exact_get_delete: prepared.exact_get_delete,
driver: prepared.driver.clone(),
reconciler: tokio::sync::OnceCell::new(),
accepting: AtomicBool::new(true),
active_leases: AtomicUsize::new(0),
drained: tokio::sync::Notify::new(),
@@ -3609,11 +3650,13 @@ impl TierConfigMgr {
let driver: SharedWarmBackend = Arc::from(driver);
let entry = Arc::new(TierDriverGeneration {
tier_name: Arc::from(tier_name),
tier_config: config.clone(),
generation,
config_fingerprint,
backend_identity,
exact_get_delete,
driver: driver.clone(),
reconciler: tokio::sync::OnceCell::new(),
accepting: AtomicBool::new(true),
active_leases: AtomicUsize::new(0),
drained: tokio::sync::Notify::new(),
@@ -4268,6 +4311,34 @@ fn tier_config_not_initialized_error(operation: &str) -> std::io::Error {
#[cfg(test)]
mod tests {
use super::*;
use crate::layout::{
endpoint::Endpoint,
endpoints::{Endpoints, PoolEndpoints, SetupType},
};
struct SetupTypeGuard {
previous: SetupType,
}
impl SetupTypeGuard {
async fn switch_to(next: SetupType) -> Self {
let previous = runtime_sources::current_setup_type().await;
runtime_sources::set_setup_type(next).await;
Self { previous }
}
}
impl Drop for SetupTypeGuard {
fn drop(&mut self) {
let previous = self.previous.clone();
let handle = tokio::runtime::Handle::current();
tokio::task::block_in_place(|| {
handle.block_on(async move {
runtime_sources::set_setup_type(previous).await;
});
});
}
}
fn build_s3_tier(name: &str) -> TierConfig {
TierConfig {
@@ -6315,6 +6386,42 @@ mod tests {
assert!(err.to_string().contains("without peer commit clients"), "{err}");
}
#[tokio::test(flavor = "multi_thread")]
#[serial_test::serial]
async fn tier_mutation_peer_composition_preserves_unresolved_topology_slots() {
let mut endpoints = Vec::new();
for disk_index in 0..4 {
let mut endpoint = Endpoint::try_from(format!("http://rustfs-{disk_index}.invalid:9000/data{disk_index}").as_str())
.expect("unresolved topology endpoint should parse without DNS");
endpoint.is_local = disk_index == 0;
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_index);
endpoints.push(endpoint);
}
let topology = EndpointServerPools::from(vec![PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 4,
endpoints: Endpoints::from(endpoints),
cmd_line: "unresolved-tier-mutation-topology".to_string(),
platform: "test".to_string(),
}]);
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let peers = remote_tier_mutation_peers_from_topology(topology)
.await
.expect("every unresolved remote topology slot should retain a tier mutation client");
assert_eq!(
peers.iter().map(|peer| peer.peer_label()).collect::<Vec<_>>(),
vec![
"http://rustfs-1.invalid:9000".to_string(),
"http://rustfs-2.invalid:9000".to_string(),
"http://rustfs-3.invalid:9000".to_string(),
]
);
}
#[tokio::test]
async fn coordinator_fanout_prepare_failure_aborts_prepared_peers_without_cas() {
let manager = TierConfigMgr::new();
@@ -73,6 +73,21 @@ pub enum TransitionCandidateProbe {
Unsupported,
}
#[derive(Clone, Copy)]
pub(crate) struct TransitionCandidateIdentity {
pub transaction_id: uuid::Uuid,
pub destination_id: [u8; 32],
}
#[async_trait::async_trait]
pub(crate) trait TransitionCandidateReconciler {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: TransitionCandidateIdentity,
) -> Result<TransitionCandidateProbe, std::io::Error>;
}
#[async_trait::async_trait]
pub trait WarmBackend {
async fn validate(&self) -> Result<(), std::io::Error> {
@@ -189,6 +204,20 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
metadata.remove(key);
}
for suffix in [
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
] {
for key in [
rustfs_utils::http::metadata_compat::internal_key_rustfs(suffix),
format!("{}{}", rustfs_utils::http::metadata_compat::MINIO_INTERNAL_PREFIX, suffix),
] {
if let Some(value) = metadata.remove(&key) {
metadata.insert(format!("x-amz-meta-{key}"), value);
}
}
}
opts.user_metadata = metadata;
opts
}
@@ -446,6 +475,51 @@ pub async fn new_warm_backend(tier: &TierConfig, probe: bool) -> Result<WarmBack
Ok(d)
}
pub(crate) async fn new_transition_candidate_reconciler(
tier: &TierConfig,
) -> Result<Option<Box<dyn TransitionCandidateReconciler + Send + Sync + 'static>>, AdminError> {
let reconciler: Box<dyn TransitionCandidateReconciler + Send + Sync + 'static> = match tier.tier_type {
TierType::S3 => Box::new(
WarmBackendS3::new(tier.s3.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
.map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
TierType::MinIO => Box::new(
WarmBackendMinIO::new(tier.minio.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
.map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
TierType::RustFS => Box::new(
WarmBackendRustFS::new(tier.rustfs.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
.map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
TierType::R2 => Box::new(
WarmBackendR2::new(tier.r2.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
.map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
_ => return Ok(None),
};
Ok(Some(reconciler))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -775,6 +849,37 @@ mod tests {
assert!(!opts.user_metadata.contains_key(X_AMZ_REPLICATION_STATUS.as_str()));
}
#[test]
fn build_transition_put_options_persists_both_candidate_identity_keys_as_s3_metadata() {
let mut metadata = HashMap::new();
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa".to_string(),
);
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
"5a".repeat(32),
);
let opts = build_transition_put_options("COLD".to_string(), metadata);
for suffix in [
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
] {
assert!(opts.user_metadata.contains_key(&format!(
"x-amz-meta-{}",
rustfs_utils::http::metadata_compat::internal_key_rustfs(suffix)
)));
assert!(opts.user_metadata.contains_key(&format!(
"x-amz-meta-{}{suffix}",
rustfs_utils::http::metadata_compat::MINIO_INTERNAL_PREFIX
)));
}
}
#[test]
fn build_transition_put_options_requests_no_checksum_and_content_md5() {
// Regression for rustfs/rustfs#4811: transition uploads must leave the
@@ -135,6 +135,20 @@ impl WarmBackend for WarmBackendMinIO {
}
}
#[async_trait::async_trait]
impl crate::services::tier::warm_backend::TransitionCandidateReconciler for WarmBackendMinIO {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: crate::services::tier::warm_backend::TransitionCandidateIdentity,
) -> Result<TransitionCandidateProbe, std::io::Error> {
crate::services::tier::warm_backend::TransitionCandidateReconciler::probe_transition_candidate_for(
&self.0, object, identity,
)
.await
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
@@ -135,6 +135,20 @@ impl WarmBackend for WarmBackendR2 {
}
}
#[async_trait::async_trait]
impl crate::services::tier::warm_backend::TransitionCandidateReconciler for WarmBackendR2 {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: crate::services::tier::warm_backend::TransitionCandidateIdentity,
) -> Result<TransitionCandidateProbe, std::io::Error> {
crate::services::tier::warm_backend::TransitionCandidateReconciler::probe_transition_candidate_for(
&self.0, object, identity,
)
.await
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
@@ -132,6 +132,20 @@ impl WarmBackend for WarmBackendRustFS {
}
}
#[async_trait::async_trait]
impl crate::services::tier::warm_backend::TransitionCandidateReconciler for WarmBackendRustFS {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: crate::services::tier::warm_backend::TransitionCandidateIdentity,
) -> Result<TransitionCandidateProbe, std::io::Error> {
crate::services::tier::warm_backend::TransitionCandidateReconciler::probe_transition_candidate_for(
&self.0, object, identity,
)
.await
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
@@ -29,6 +29,7 @@ use crate::client::{
api_remove::{RemoveObjectOptions, RemoveObjectResult},
api_s3_datatypes::ListVersionsResult,
credentials::{Credentials, SignatureType, Static, Value},
provider_versions::validate_remote_version_id,
transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore},
transition_api::{ReadCloser, ReaderImpl},
};
@@ -36,7 +37,10 @@ use crate::error::ErrorResponse;
use crate::error::error_resp_to_object_err;
use crate::services::tier::{
tier_config::TierS3,
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
TransitionCandidateIdentity, TransitionCandidateProbe, TransitionCandidateReconciler, WarmBackend, WarmBackendGetOpts,
build_transition_put_options,
},
};
use http::HeaderMap;
use rustfs_utils::egress::validate_outbound_url;
@@ -189,44 +193,188 @@ impl WarmBackendS3 {
remote_bucket_versioning_from_status(config.status.as_ref().map(|status| status.as_str()))
}
async fn list_transition_candidate_versions(&self, object: &str) -> Result<ListVersionsResult, std::io::Error> {
async fn probe_transition_candidate_versions(
&self,
object: &str,
bucket_versioning: RemoteBucketVersioning,
) -> Result<TransitionCandidateProbe, std::io::Error> {
let remote_object = self.get_dest(object);
let mut opts = ListObjectsOptions::default();
opts.set("prefix", &self.get_dest(object));
opts.set("max-keys", "2");
self.client.list_object_versions_query(&self.bucket, &opts, "", "", "").await
opts.set("prefix", &remote_object);
opts.set("max-keys", "1000");
let mut key_marker = String::new();
let mut version_id_marker = String::new();
let mut candidates = TransitionCandidateVersions::default();
loop {
let versions = self
.client
.list_object_versions_query(&self.bucket, &opts, &key_marker, &version_id_marker, "")
.await?;
candidates.extend(&remote_object, &versions);
if candidates.is_ambiguous() {
return Ok(TransitionCandidateProbe::Ambiguous);
}
if !versions.is_truncated {
return classify_transition_candidates(candidates, bucket_versioning);
}
advance_version_markers(&mut key_marker, &mut version_id_marker, &versions)?;
}
}
async fn probe_transition_candidate_identity(
&self,
object: &str,
identity: TransitionCandidateIdentity,
bucket_versioning: RemoteBucketVersioning,
) -> Result<TransitionCandidateProbe, std::io::Error> {
let remote_object = self.get_dest(object);
let mut opts = ListObjectsOptions::default();
opts.set("prefix", &remote_object);
opts.set("max-keys", "1000");
let mut key_marker = String::new();
let mut version_id_marker = String::new();
let mut matched_version = None;
let mut saw_unproven_candidate = false;
loop {
let versions = self
.client
.list_object_versions_query(&self.bucket, &opts, &key_marker, &version_id_marker, "")
.await?;
for version in versions.versions.iter().filter(|version| version.key == remote_object) {
let mut stat_opts = GetObjectOptions::default();
stat_opts.version_id.clone_from(&version.version_id);
let info = self.client.stat_object(&self.bucket, &remote_object, &stat_opts).await?;
let mut metadata = info.user_metadata;
for (name, value) in &info.metadata {
if (name
.as_str()
.starts_with(rustfs_utils::http::metadata_compat::RUSTFS_INTERNAL_PREFIX)
|| name
.as_str()
.starts_with(rustfs_utils::http::metadata_compat::MINIO_INTERNAL_PREFIX))
&& let Ok(value) = value.to_str()
{
metadata.insert(name.as_str().to_string(), value.to_string());
}
}
if transition_candidate_metadata_matches(&metadata, identity)? {
if matched_version.is_some() {
return Ok(TransitionCandidateProbe::Ambiguous);
}
matched_version = Some(version.version_id.clone());
} else {
saw_unproven_candidate = true;
}
}
if !versions.is_truncated {
if matched_version.is_none() && saw_unproven_candidate {
return Ok(TransitionCandidateProbe::Unsupported);
}
let candidates = TransitionCandidateVersions {
version_id: matched_version,
ambiguous: false,
};
return classify_transition_candidates(candidates, bucket_versioning);
}
advance_version_markers(&mut key_marker, &mut version_id_marker, &versions)?;
}
}
}
fn classify_transition_candidate_versions(
remote_object: &str,
bucket_versioning: RemoteBucketVersioning,
versions: &ListVersionsResult,
) -> TransitionCandidateProbe {
if versions.is_truncated {
return TransitionCandidateProbe::Ambiguous;
}
if versions.delete_markers.iter().any(|marker| marker.key == remote_object) {
return TransitionCandidateProbe::Ambiguous;
}
let mut exact_versions = versions.versions.iter().filter(|version| version.key == remote_object);
let Some(version) = exact_versions.next() else {
return TransitionCandidateProbe::Missing;
fn transition_candidate_metadata_matches(
metadata: &HashMap<String, String>,
identity: TransitionCandidateIdentity,
) -> Result<bool, std::io::Error> {
use rustfs_utils::http::metadata_compat::{
SUFFIX_TRANSITION_TIER_DESTINATION_ID, SUFFIX_TRANSITION_TRANSACTION_ID, contains_key_str, get_consistent_str,
};
if exact_versions.next().is_some() {
return TransitionCandidateProbe::Ambiguous;
let transaction_id = get_consistent_str(metadata, SUFFIX_TRANSITION_TRANSACTION_ID);
let destination_id = get_consistent_str(metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID);
if transaction_id.is_none() || destination_id.is_none() {
if contains_key_str(metadata, SUFFIX_TRANSITION_TRANSACTION_ID)
|| contains_key_str(metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID)
{
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"transition candidate identity metadata is empty or conflicting",
));
}
return Ok(false);
}
let expected_transaction_id = identity.transaction_id.to_string();
let expected_destination_id = rustfs_utils::crypto::hex(identity.destination_id);
Ok(transaction_id == Some(expected_transaction_id.as_str()) && destination_id == Some(expected_destination_id.as_str()))
}
fn classify_transition_candidates(
candidates: TransitionCandidateVersions,
bucket_versioning: RemoteBucketVersioning,
) -> Result<TransitionCandidateProbe, std::io::Error> {
let probe = candidates.classify(bucket_versioning);
if let TransitionCandidateProbe::VersionedPresent(version_id) = &probe {
validate_remote_version_id(version_id)?;
}
Ok(probe)
}
fn advance_version_markers(
key_marker: &mut String,
version_id_marker: &mut String,
versions: &ListVersionsResult,
) -> Result<(), std::io::Error> {
let next_markers = (&versions.next_key_marker, &versions.next_version_id_marker);
if next_markers == (&*key_marker, &*version_id_marker) {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"ListObjectVersions pagination markers did not advance",
));
}
key_marker.clone_from(&versions.next_key_marker);
version_id_marker.clone_from(&versions.next_version_id_marker);
Ok(())
}
#[derive(Default)]
struct TransitionCandidateVersions {
version_id: Option<String>,
ambiguous: bool,
}
impl TransitionCandidateVersions {
fn extend(&mut self, remote_object: &str, versions: &ListVersionsResult) {
for version in versions.versions.iter().filter(|version| version.key == remote_object) {
if self.version_id.is_some() {
self.ambiguous = true;
return;
}
self.version_id = Some(version.version_id.clone());
}
}
match bucket_versioning {
RemoteBucketVersioning::Disabled => TransitionCandidateProbe::UnversionedPresent,
RemoteBucketVersioning::Suspended if version.version_id == "null" => {
TransitionCandidateProbe::VersionedPresent(version.version_id.clone())
fn is_ambiguous(&self) -> bool {
self.ambiguous
}
fn classify(self, bucket_versioning: RemoteBucketVersioning) -> TransitionCandidateProbe {
if self.ambiguous {
return TransitionCandidateProbe::Ambiguous;
}
RemoteBucketVersioning::Suspended | RemoteBucketVersioning::Enabled if !version.version_id.is_empty() => {
TransitionCandidateProbe::VersionedPresent(version.version_id.clone())
let Some(version_id) = self.version_id else {
return TransitionCandidateProbe::Missing;
};
match bucket_versioning {
RemoteBucketVersioning::Disabled => TransitionCandidateProbe::UnversionedPresent,
RemoteBucketVersioning::Suspended if version_id == "null" => TransitionCandidateProbe::VersionedPresent(version_id),
RemoteBucketVersioning::Suspended | RemoteBucketVersioning::Enabled if !version_id.is_empty() => {
TransitionCandidateProbe::VersionedPresent(version_id)
}
RemoteBucketVersioning::Suspended | RemoteBucketVersioning::Enabled => TransitionCandidateProbe::Ambiguous,
}
RemoteBucketVersioning::Suspended | RemoteBucketVersioning::Enabled => TransitionCandidateProbe::Ambiguous,
}
}
@@ -275,74 +423,163 @@ mod tests {
}
}
fn classify_pages(bucket_versioning: RemoteBucketVersioning, pages: &[ListVersionsResult]) -> TransitionCandidateProbe {
let mut candidates = TransitionCandidateVersions::default();
for page in pages {
candidates.extend("archive/object", page);
}
candidates.classify(bucket_versioning)
}
fn candidate_identity() -> TransitionCandidateIdentity {
TransitionCandidateIdentity {
transaction_id: uuid::Uuid::parse_str("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa").unwrap(),
destination_id: [0x5a; 32],
}
}
fn candidate_metadata(identity: TransitionCandidateIdentity) -> HashMap<String, String> {
let mut metadata = HashMap::new();
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
identity.transaction_id.to_string(),
);
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
rustfs_utils::crypto::hex(identity.destination_id),
);
metadata
}
#[test]
fn transition_candidate_identity_requires_exact_compatible_metadata() {
let identity = candidate_identity();
let metadata = candidate_metadata(identity);
assert!(transition_candidate_metadata_matches(&metadata, identity).unwrap());
let mut adjacent = metadata;
rustfs_utils::http::metadata_compat::insert_str(
&mut adjacent,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
uuid::Uuid::parse_str("bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb")
.unwrap()
.to_string(),
);
assert!(!transition_candidate_metadata_matches(&adjacent, identity).unwrap());
assert!(!transition_candidate_metadata_matches(&HashMap::new(), identity).unwrap());
}
#[test]
fn transition_candidate_identity_rejects_conflicting_compatibility_keys() {
let identity = candidate_identity();
let mut metadata = candidate_metadata(identity);
metadata.insert(
rustfs_utils::http::metadata_compat::internal_key_rustfs(
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
),
uuid::Uuid::new_v4().to_string(),
);
assert!(transition_candidate_metadata_matches(&metadata, identity).is_err());
}
#[test]
fn transition_candidate_probe_classifier_is_fail_closed() {
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
RemoteBucketVersioning::Disabled,
&list_versions(&[], &[], false),
),
classify_pages(RemoteBucketVersioning::Disabled, &[list_versions(&[], &[], false)],),
TransitionCandidateProbe::Missing
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
RemoteBucketVersioning::Disabled,
&list_versions(&[("archive/object", "")], &[], false),
),
classify_pages(RemoteBucketVersioning::Disabled, &[list_versions(&[("archive/object", "")], &[], false)],),
TransitionCandidateProbe::UnversionedPresent
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
classify_pages(
RemoteBucketVersioning::Enabled,
&list_versions(&[("archive/object", "version-a")], &[], false),
&[list_versions(&[("archive/object", "version-a")], &[], false)],
),
TransitionCandidateProbe::VersionedPresent("version-a".to_string())
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
classify_pages(
RemoteBucketVersioning::Suspended,
&list_versions(&[("archive/object", "null")], &[], false),
&[list_versions(&[("archive/object", "null")], &[], false)],
),
TransitionCandidateProbe::VersionedPresent("null".to_string())
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
RemoteBucketVersioning::Enabled,
&list_versions(&[("archive/object", "")], &[], false),
),
classify_pages(RemoteBucketVersioning::Enabled, &[list_versions(&[("archive/object", "")], &[], false)],),
TransitionCandidateProbe::Ambiguous
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
classify_pages(
RemoteBucketVersioning::Enabled,
&list_versions(&[("archive/object", "version-a"), ("archive/object", "version-b")], &[], false),
&[list_versions(
&[("archive/object", "version-a"), ("archive/object", "version-b")],
&[],
false,
)],
),
TransitionCandidateProbe::Ambiguous
);
}
#[test]
fn transition_candidate_probe_reconciles_all_pages_and_ignores_delete_markers() {
assert_eq!(
classify_pages(
RemoteBucketVersioning::Enabled,
&[
list_versions(&[], &[("archive/object", "marker-a")], true),
list_versions(&[("archive/object", "version-a"), ("archive/object-adjacent", "unrelated"),], &[], false,),
],
),
TransitionCandidateProbe::VersionedPresent("version-a".to_string())
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
classify_pages(
RemoteBucketVersioning::Enabled,
&list_versions(&[("archive/object", "version-a")], &[("archive/object", "marker-a")], false),
),
TransitionCandidateProbe::Ambiguous
);
assert_eq!(
classify_transition_candidate_versions(
"archive/object",
RemoteBucketVersioning::Enabled,
&list_versions(&[("archive/object", "version-a")], &[], true),
&[
list_versions(&[("archive/object", "version-a")], &[], true),
list_versions(&[("archive/object", "version-b")], &[], false),
],
),
TransitionCandidateProbe::Ambiguous
);
}
#[test]
fn transition_candidate_pagination_advances_both_markers() {
let mut key_marker = "old-key".to_string();
let mut version_id_marker = "old-version".to_string();
let page = ListVersionsResult {
next_key_marker: "next-key".to_string(),
next_version_id_marker: "next-version".to_string(),
..Default::default()
};
advance_version_markers(&mut key_marker, &mut version_id_marker, &page)
.expect("new ListObjectVersions markers should advance pagination");
assert_eq!(key_marker, "next-key");
assert_eq!(version_id_marker, "next-version");
let err = advance_version_markers(&mut key_marker, &mut version_id_marker, &page)
.expect_err("repeated ListObjectVersions markers must fail closed");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
}
#[test]
fn transition_candidate_probe_rejects_untrusted_version_ids() {
let mut candidates = TransitionCandidateVersions::default();
candidates.extend("archive/object", &list_versions(&[("archive/object", "version\ninjection")], &[], false));
let err = classify_transition_candidates(candidates, RemoteBucketVersioning::Enabled)
.expect_err("control characters in listed version IDs must fail closed");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
}
#[test]
fn remote_bucket_versioning_status_parser_fails_closed() {
assert_eq!(
@@ -397,12 +634,7 @@ impl WarmBackend for WarmBackendS3 {
async fn probe_transition_candidate(&self, object: &str) -> Result<TransitionCandidateProbe, std::io::Error> {
let bucket_versioning = self.remote_bucket_versioning().await?;
let versions = self.list_transition_candidate_versions(object).await?;
Ok(classify_transition_candidate_versions(
&self.get_dest(object),
bucket_versioning,
&versions,
))
self.probe_transition_candidate_versions(object, bucket_versioning).await
}
async fn in_use(&self) -> Result<bool, std::io::Error> {
@@ -414,3 +646,16 @@ impl WarmBackend for WarmBackendS3 {
Ok(result.common_prefixes.len() > 0 || result.contents.len() > 0)
}
}
#[async_trait::async_trait]
impl TransitionCandidateReconciler for WarmBackendS3 {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: TransitionCandidateIdentity,
) -> Result<TransitionCandidateProbe, std::io::Error> {
let bucket_versioning = self.remote_bucket_versioning().await?;
self.probe_transition_candidate_identity(object, identity, bucket_versioning)
.await
}
}
+1 -3
View File
@@ -505,9 +505,7 @@ impl SetDisks {
}
fn file_info_has_encryption_metadata(meta: &FileInfo) -> bool {
meta.metadata
.keys()
.any(|name| http::is_encryption_metadata_key(name) || http::is_sse_header(name))
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
}
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
+255 -26
View File
@@ -110,7 +110,10 @@ use crate::{
object_api::{GetObjectReader, ObjectInfo, PutObjReader},
// event::name::EventName,
services::event_notification::{EventArgs, send_event},
store::init_format::{get_format_erasure_in_quorum, load_format_erasure, load_format_erasure_all, save_format_file},
store::init_format::{
formats_match_reference_slots, get_format_erasure_in_quorum, load_format_erasure, load_format_erasure_all,
save_format_file,
},
};
use bytes::Bytes;
use bytesize::ByteSize;
@@ -144,15 +147,17 @@ use rustfs_object_capacity::capacity_scope::{
CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
};
use rustfs_s3_types::EventName;
#[cfg(test)]
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
use rustfs_utils::http::headers::{
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
};
use rustfs_utils::http::{
SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, contains_key_str,
get_header_map, get_str, insert_str, is_encryption_metadata_key, remove_header_map,
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_RESTORE_OPERATION_ID, contains_key_str, get_header_map, get_str, insert_str, is_object_encryption_marker,
remove_header_map,
};
use rustfs_utils::{
HashAlgorithm,
@@ -667,10 +672,7 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
}
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
metadata.keys().any(|key| is_encryption_metadata_key(key))
|| metadata.contains_key(SSEC_ALGORITHM_HEADER)
|| metadata.contains_key(SSEC_KEY_HEADER)
|| metadata.contains_key(SSEC_KEY_MD5_HEADER)
metadata.keys().any(|key| is_object_encryption_marker(key))
}
/// Per-set memoized capacity dirty scope.
@@ -4712,6 +4714,7 @@ mod tests {
use crate::layout::endpoints::SetupType;
use crate::object_api::BLOCK_SIZE_V2;
use crate::object_api::ObjectInfo;
use crate::set_disk::core::io_primitives::rename_fanout_barrier;
use crate::storage_api_contracts::{
heal::HealOperations as _, lifecycle::TransitionedObject, list::ListOperations as _, multipart::CompletePart,
namespace::NamespaceLocking as _, object::ObjectIO as _, object::ObjectOperations as _,
@@ -5849,23 +5852,27 @@ mod tests {
crate::disk::DataDirDeleteStatus::Deleted
);
release_slow_candidate.notify_one();
for _ in 0..10 {
tokio::task::yield_now().await;
}
assert_eq!(
disk2
.delete_data_dir(
bucket,
data_dir,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
.expect("a token acquired after the deadline must be released"),
crate::disk::DataDirDeleteStatus::Deleted
);
timeout(Duration::from_secs(1), async {
loop {
match disk2
.delete_data_dir(
bucket,
data_dir,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
.expect("a token acquired after the deadline must be released")
{
crate::disk::DataDirDeleteStatus::Deleted => return,
crate::disk::DataDirDeleteStatus::Deferred => tokio::time::sleep(Duration::from_millis(1)).await,
}
}
})
.await
.expect("late snapshot lease cleanup must finish within the bounded wait");
}
#[tokio::test(start_paused = true)]
@@ -9559,6 +9566,15 @@ mod tests {
make_local_bucket_test_set_disks_with_drive_count(2).await
}
fn assert_exclusive_object_lock_held(set_disks: &SetDisks, bucket: &str, object: &str) {
let lock = set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.expect("object lock should be visible while rename is paused");
assert!(matches!(lock.mode, rustfs_lock::LockMode::Exclusive));
assert_eq!(lock.owner.as_ref(), set_disks.locker_owner.as_str());
}
async fn make_local_bucket_test_set_disks_with_drive_count(drive_count: usize) -> Arc<SetDisks> {
let format = FormatV3::new(1, drive_count);
let mut endpoints = Vec::new();
@@ -9593,7 +9609,9 @@ mod tests {
disks.push(Some(disk));
}
let set_disks = SetDisks::new(
let instance_ctx = Arc::new(InstanceContext::new());
instance_ctx.update_erasure_type(SetupType::Erasure).await;
let set_disks = SetDisks::new_with_instance_ctx(
"test-owner".to_string(),
Arc::new(RwLock::new(disks)),
drive_count,
@@ -9603,6 +9621,7 @@ mod tests {
endpoints,
format,
Vec::new(),
instance_ctx,
)
.await;
set_disks.set_test_storage_class_config(
@@ -10256,6 +10275,216 @@ mod tests {
));
}
#[tokio::test]
async fn conditional_replace_holds_object_lock_through_rename() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-conditional-replace-fence";
let object = "config/conditional-replace.json";
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut initial_reader = PutObjReader::from_vec(b"initial config".to_vec());
let initial = set_disks
.put_object(
bucket,
object,
&mut initial_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("initial config should be written");
let initial_etag = initial.etag.expect("initial config should have an ETag");
let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
let writer_store = set_disks.clone();
let expected_etag = initial_etag.clone();
let writer = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(b"replacement config".to_vec());
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
preserve_etag: Some("replacement-etag".to_string()),
http_preconditions: Some(HTTPPreconditions {
if_match: Some(expected_etag),
..Default::default()
}),
..Default::default()
},
)
.await
});
tokio::time::timeout(std::time::Duration::from_secs(30), barrier.wait_until_paused())
.await
.expect("conditional replace should reach the rename barrier");
assert_exclusive_object_lock_held(&set_disks, bucket, object);
barrier.release();
writer
.await
.expect("conditional writer task should finish")
.expect("matching conditional replace should commit");
assert!(
set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.is_none(),
"conditional replace should release the object lock after commit"
);
let contender = set_disks
.new_ns_lock(bucket, object)
.await
.expect("contender namespace lock should be created");
let contender_guard = contender
.get_write_lock(std::time::Duration::from_secs(30))
.await
.expect("contender should acquire after conditional replace commits");
drop(contender_guard);
let mut stale_reader = PutObjReader::from_vec(b"stale config".to_vec());
let err = set_disks
.put_object(
bucket,
object,
&mut stale_reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(initial_etag),
..Default::default()
}),
..Default::default()
},
)
.await
.expect_err("the old ETag must fail after the fenced replacement commits");
assert_eq!(err, StorageError::PreconditionFailed);
}
#[tokio::test]
async fn repeated_body_write_keeps_etag_but_changes_data_dir_generation() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-write-generation";
let object = "config/write-generation.json";
let body = b"identical config body".to_vec();
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut first_reader = PutObjReader::from_vec(body.clone());
let first = set_disks
.put_object(
bucket,
object,
&mut first_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("first config body should be written");
let mut second_reader = PutObjReader::from_vec(body);
let second = set_disks
.put_object(
bucket,
object,
&mut second_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("identical config body should be rewritten");
assert_eq!(first.etag, second.etag, "content ETag should expose the ABA collision");
assert_ne!(first.data_dir, second.data_dir, "each committed body write needs a unique generation");
assert!(first.data_dir.is_some() && second.data_dir.is_some());
}
#[tokio::test]
async fn conditional_create_holds_object_lock_through_rename() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-conditional-create-fence";
let object = "config/conditional-create.json";
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
let writer_store = set_disks.clone();
let writer = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(b"created config".to_vec());
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
});
tokio::time::timeout(std::time::Duration::from_secs(30), barrier.wait_until_paused())
.await
.expect("conditional create should reach the rename barrier");
assert_exclusive_object_lock_held(&set_disks, bucket, object);
barrier.release();
writer
.await
.expect("conditional writer task should finish")
.expect("first conditional create should commit");
assert!(
set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.is_none(),
"conditional create should release the object lock after commit"
);
let contender = set_disks
.new_ns_lock(bucket, object)
.await
.expect("contender namespace lock should be created");
let contender_guard = contender
.get_write_lock(std::time::Duration::from_secs(30))
.await
.expect("contender should acquire after conditional create commits");
drop(contender_guard);
let mut duplicate_reader = PutObjReader::from_vec(b"duplicate config".to_vec());
let err = set_disks
.put_object(
bucket,
object,
&mut duplicate_reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
.expect_err("a second create-only write must not replace the committed config");
assert_eq!(err, StorageError::PreconditionFailed);
}
#[tokio::test]
async fn set_level_if_none_match_fails_closed_without_read_quorum() {
let set_disks = make_local_bucket_test_set_disks_with_drive_count(4).await;
+63 -4
View File
@@ -1373,11 +1373,24 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
let disks = self.disks.read().await.clone();
let (formats, errs) = load_format_erasure_all(&disks, true).await;
let ref_format = match get_format_erasure_in_quorum(&formats) {
Ok(format) => format,
if errs.iter().any(|err| {
matches!(
err,
Some(DiskError::InconsistentDisk | DiskError::CorruptedFormat | DiskError::CorruptedBackend)
)
}) {
return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
}
let slot_offset = self
.set_index
.checked_mul(self.set_drive_count)
.ok_or_else(|| Error::other("erasure set slot offset overflow"))?;
let ref_format = match get_format_erasure_in_quorum(&formats, slot_offset) {
Ok(format) if format.shared_identity() == self.format.shared_identity() => format,
Ok(_) => return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat))),
Err(err) => {
let can_use_cached_layout = count_errs(&errs, &DiskError::UnformattedDisk) > 0
&& formats.iter().flatten().all(|format| self.format.check_other(format).is_ok())
&& formats_match_reference_slots(&formats, &self.format, slot_offset)
&& errs
.iter()
.all(|err| err.is_none() || matches!(err, Some(DiskError::UnformattedDisk)));
@@ -1388,6 +1401,9 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
}
}
};
if !formats_match_reference_slots(&formats, &ref_format, slot_offset) {
return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
}
let endpoints = crate::layout::endpoints::Endpoints::from(self.set_endpoints.clone());
let before_drives = crate::layout::set_heal::formats_to_drives_info(&endpoints, &formats, &errs);
@@ -1543,11 +1559,16 @@ mod heal_result_report_tests {
use crate::disk::error::DiskError;
use crate::disk::format::FormatV3;
use crate::disk::{DiskAPI as _, DiskOption, DiskStore, RUSTFS_META_TMP_BUCKET, ReadOptions, new_disk};
use crate::error::Error;
use crate::object_api::{ObjectOptions, PutObjReader};
use crate::set_disk::ops::object::hermetic_set_disks_support::hermetic_set_disks_isolated;
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
use crate::{config::storageclass, store::init_format::save_format_file};
use crate::{
config::storageclass,
store::init_format::{load_format_erasure, save_format_file},
};
use rustfs_common::heal_channel::{DriveState, HealOpts, HealScanMode};
use rustfs_filemeta::{BLOCK_SIZE_V2, FileInfo, ObjectPartInfo, TRANSITION_COMPLETE};
use std::sync::Arc;
@@ -1863,6 +1884,44 @@ mod heal_result_report_tests {
}
}
#[tokio::test]
async fn format_heal_cached_layout_rejects_a_disk_from_another_slot() {
let mut _temp_dirs = Vec::new();
let mut endpoints = Vec::new();
let mut disks = Vec::new();
for disk_index in 0..3 {
let (temp_dir, mut endpoint, disk) = real_disk().await;
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_index);
_temp_dirs.push(temp_dir);
endpoints.push(endpoint);
disks.push(Some(disk));
}
let set = set_disks_with(disks.clone(), endpoints, 1).await;
let mut wrong_slot = set.format.clone();
wrong_slot.erasure.this = set.format.erasure.sets[0][1];
save_format_file(&disks[0], &Some(wrong_slot))
.await
.expect("wrong-slot format fixture should be saved");
let mut correct_slot = set.format.clone();
correct_slot.erasure.this = set.format.erasure.sets[0][2];
save_format_file(&disks[2], &Some(correct_slot))
.await
.expect("correct format fixture should be saved");
let (_, heal_err) = set
.heal_format(false)
.await
.expect("format heal should report the quorum failure in its result");
assert!(matches!(heal_err, Some(Error::CorruptedFormat)));
let unformatted = load_format_erasure(disks[1].as_ref().expect("second disk should be online"), true)
.await
.expect_err("a rejected fallback must not format the missing slot");
assert_eq!(unformatted, DiskError::UnformattedDisk);
}
// Regression for #955: an offline disk must contribute exactly one drive
// record. Before the fix the offline branch fell through and pushed a second
// (Corrupt) record for the same disk, so `before/after.drives` grew to
+114 -9
View File
@@ -343,20 +343,23 @@ impl SetDisks {
}
};
// The drive's format may place it in a different erasure set than this
// one. Claiming a misplaced drive into `self.disks` would let two sets
// manage the same drive and degrade together, so reject it here
// (backlog#799 B19).
if set_idx != self.set_index {
// Claiming a misplaced drive into `self.disks` would let two slots or
// sets manage the same drive and degrade together (backlog#799 B19).
if set_idx != self.set_index || self.set_endpoints.get(disk_idx) != Some(ep) {
warn!(
"renew_disk: drive {:?} belongs to set {} but is being renewed on set {}; skipping",
ep, set_idx, self.set_index
endpoint = %ep,
format_set_index = set_idx,
format_disk_index = disk_idx,
endpoint_pool_index = ep.pool_idx,
endpoint_set_index = ep.set_idx,
endpoint_disk_index = ep.disk_idx,
expected_pool_index = self.pool_index,
expected_set_index = self.set_index,
"renew_disk rejected a drive whose endpoint and format do not identify the same topology slot"
);
return;
}
// Check that the endpoint matches
let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
new_disk.enable_health_check();
@@ -715,6 +718,108 @@ mod tests {
drop(temp_dirs);
}
#[tokio::test]
async fn renew_disk_rejects_a_format_from_another_slot_or_cluster() {
let disk_count = 3;
let format = FormatV3::new(1, disk_count);
let mut temp_dirs = Vec::with_capacity(disk_count);
let mut endpoints = Vec::with_capacity(disk_count);
let mut fixture_disks = Vec::with_capacity(disk_count);
for disk_idx in 0..disk_count {
let (temp_dir, endpoint, disk) = make_formatted_local_disk(disk_idx, &format).await;
temp_dirs.push(temp_dir);
endpoints.push(endpoint);
fixture_disks.push(disk);
}
let set_disks = SetDisks::new(
"test-owner".to_string(),
Arc::new(RwLock::new(vec![Some(fixture_disks[0].clone()), None, None])),
disk_count,
disk_count / 2,
0,
0,
endpoints.clone(),
format.clone(),
Vec::new(),
)
.await;
let mut other_cluster_format = format.clone();
other_cluster_format.id = Uuid::new_v4();
other_cluster_format.erasure.this = format.erasure.sets[0][2];
save_format_file(&Some(fixture_disks[2].clone()), &Some(other_cluster_format))
.await
.expect("other-cluster format should be written for the rejection test");
set_disks.renew_disk(&endpoints[2]).await;
let disks = set_disks.get_disks_internal().await;
assert_eq!(
disks[0]
.as_ref()
.expect("the canonical first slot must remain attached")
.endpoint(),
endpoints[0]
);
assert!(
disks[2].is_none(),
"a disk from another deployment must remain detached even when its slot UUID matches"
);
let mut correct_format = format.clone();
correct_format.erasure.this = format.erasure.sets[0][2];
let replacement_disk = new_disk(
&endpoints[2],
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("third endpoint should reopen after other-cluster rejection");
save_format_file(&Some(replacement_disk), &Some(correct_format))
.await
.expect("correct slot format should be restored");
set_disks.renew_disk(&endpoints[2]).await;
let disks = set_disks.get_disks_internal().await;
assert_eq!(
disks[0]
.as_ref()
.expect("the canonical first slot must remain attached")
.endpoint(),
endpoints[0]
);
assert_eq!(disks[2].as_ref().expect("the restored third slot should attach").endpoint(), endpoints[2]);
let third_disk = disks[2].clone();
let mut wrong_slot_format = format.clone();
wrong_slot_format.erasure.this = format.erasure.sets[0][0];
save_format_file(&third_disk, &Some(wrong_slot_format))
.await
.expect("wrong-slot format should be written for the rejection test");
set_disks.disks.write().await[2] = None;
let mut misplaced_endpoint = endpoints[2].clone();
misplaced_endpoint.set_disk_index(0);
set_disks.renew_disk(&misplaced_endpoint).await;
let disks = set_disks.get_disks_internal().await;
assert_eq!(
disks[0]
.as_ref()
.expect("the canonical first slot must remain attached")
.endpoint(),
endpoints[0]
);
assert!(disks[2].is_none(), "a disk claiming another endpoint's slot must remain detached");
drop(temp_dirs);
}
// SetDisks split P0 (#816): the borrow handle must mirror the core state and
// the List operation family must run identically through it.
#[tokio::test]
+88 -75
View File
@@ -3379,85 +3379,98 @@ mod tests {
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn complete_holds_object_then_upload_lock_through_commit() {
temp_env::async_with_vars([(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("true"))], async {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
let bucket = "multipart-layout-lock-order-bucket";
let object = "object";
make_bucket_on_all(&disk_stores, bucket).await;
let create_opts = ObjectOptions {
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]),
..Default::default()
};
let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x43; 4096], &create_opts).await;
let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
let upload_resource = rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path);
let object_resource = rustfs_lock::ObjectKey::new(bucket, object);
signaling.set_target(upload_resource.clone());
signaling.clear_observed();
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::AfterRename);
temp_env::async_with_vars(
[
(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("true")),
(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("60")),
],
async {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
let bucket = "multipart-layout-lock-order-bucket";
let object = "object";
make_bucket_on_all(&disk_stores, bucket).await;
let create_opts = ObjectOptions {
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]),
..Default::default()
};
let (upload_id, parts) =
stage_upload_with_create_opts(&set_disks, bucket, object, &[0x43; 4096], &create_opts).await;
let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
let upload_resource = rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path);
let object_resource = rustfs_lock::ObjectKey::new(bucket, object);
signaling.set_target(upload_resource.clone());
signaling.clear_observed();
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::AfterRename);
let complete_store = set_disks.clone();
let complete_upload_id = upload_id.clone();
let complete = tokio::spawn(async move {
complete_store
.complete_multipart_upload(bucket, object, &complete_upload_id, parts, &ObjectOptions::default())
let complete_store = set_disks.clone();
let complete_upload_id = upload_id.clone();
let complete = tokio::spawn(async move {
complete_store
.complete_multipart_upload(bucket, object, &complete_upload_id, parts, &ObjectOptions::default())
.await
});
barrier.wait_until_paused().await;
let observed = signaling.observed();
let object_position = observed
.iter()
.position(|resource| resource == &object_resource)
.expect("completion should acquire the object lock");
let upload_position = observed
.iter()
.position(|resource| resource == &upload_resource)
.expect("completion should acquire the upload lock");
assert!(object_position < upload_position, "completion must acquire object before upload");
let abort_store = set_disks.clone();
let abort_upload_id = upload_id.clone();
let abort = tokio::spawn(async move {
abort_store
.abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(2).await;
tokio::task::yield_now().await;
assert!(!abort.is_finished(), "abort must wait until completion releases the upload lock");
let list_store = set_disks.clone();
let list_upload_id = upload_id.clone();
let list = tokio::spawn(async move {
list_store
.list_object_parts(bucket, object, &list_upload_id, None, MAX_PARTS_COUNT, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(3).await;
tokio::task::yield_now().await;
assert!(!list.is_finished(), "ListParts must wait until completion releases the upload lock");
barrier.release();
complete
.await
});
barrier.wait_until_paused().await;
let observed = signaling.observed();
let object_position = observed
.iter()
.position(|resource| resource == &object_resource)
.expect("completion should acquire the object lock");
let upload_position = observed
.iter()
.position(|resource| resource == &upload_resource)
.expect("completion should acquire the upload lock");
assert!(object_position < upload_position, "completion must acquire object before upload");
let abort_store = set_disks.clone();
let abort_upload_id = upload_id.clone();
let abort = tokio::spawn(async move {
abort_store
.abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default())
.expect("completion task should not panic")
.expect("completion should commit after the barrier is released");
let abort_err = abort
.await
});
signaling.wait_for_attempts(2).await;
tokio::task::yield_now().await;
assert!(!abort.is_finished(), "abort must wait until completion releases the upload lock");
let list_store = set_disks.clone();
let list_upload_id = upload_id.clone();
let list = tokio::spawn(async move {
list_store
.list_object_parts(bucket, object, &list_upload_id, None, MAX_PARTS_COUNT, &ObjectOptions::default())
.expect("abort task should not panic")
.expect_err("the committed upload should no longer exist when abort acquires the lock");
assert!(
matches!(abort_err, StorageError::InvalidUploadID(..)),
"abort should return InvalidUploadID after completion, got {abort_err:?}"
);
let list_err = list
.await
});
signaling.wait_for_attempts(3).await;
tokio::task::yield_now().await;
assert!(!list.is_finished(), "ListParts must wait until completion releases the upload lock");
barrier.release();
complete
.await
.expect("completion task should not panic")
.expect("completion should commit after the barrier is released");
let abort_err = abort
.await
.expect("abort task should not panic")
.expect_err("the committed upload should no longer exist when abort acquires the lock");
assert!(matches!(abort_err, StorageError::InvalidUploadID(..)));
let list_err = list
.await
.expect("ListParts task should not panic")
.expect_err("the committed upload should no longer exist when ListParts acquires the lock");
assert!(matches!(list_err, StorageError::InvalidUploadID(..)));
})
.expect("ListParts task should not panic")
.expect_err("the committed upload should no longer exist when ListParts acquires the lock");
assert!(
matches!(list_err, StorageError::InvalidUploadID(..)),
"ListParts should return InvalidUploadID after completion, got {list_err:?}"
);
},
)
.await;
}
+171 -27
View File
@@ -42,6 +42,7 @@ use crate::object_api::{GetObjectBodySource, get_object_body_cache_hook_suppress
use crate::services::tier::tier::{TierConfigMgr, TierOperationLease};
use crate::store::ECStore;
use futures::FutureExt as _;
use http::HeaderValue;
use std::future::Future;
fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Erasure> {
@@ -49,6 +50,17 @@ fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Er
.map_err(Error::from)
}
async fn get_object_reader_with_context(
ctx: &InstanceContext,
reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
range: Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
opts: &ObjectOptions,
headers: &HeaderMap<HeaderValue>,
) -> Result<(GetObjectReader, usize, i64)> {
GetObjectReader::new_with_resolver(reader, range, object_info, opts, headers, ctx.object_encryption_resolver()).await
}
/// Length of the full plaintext body when — and only when — this read's output
/// is exactly the object's complete plaintext, so the app-layer body cache may
/// serve it in place of the erasure read.
@@ -713,7 +725,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
size_bucket,
);
record_get_object_reader_path_observation(GET_OBJECT_PATH_CODEC_STREAMING, object_class, size_bucket);
let (mut reader, _offset, _length) = GetObjectReader::new(stream, range, &object_info, opts, &h).await?;
let (mut reader, _offset, _length) =
get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its
// now-redundant lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -745,7 +758,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
let (mut reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h).await?;
let (mut reader, offset, length) =
get_object_reader_with_context(&self.ctx, Box::new(rd), range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its now-redundant
// lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -2320,13 +2334,25 @@ async fn pause_transition_commit(bucket: &str, object: &str, pause: TransitionCo
}
}
#[cfg(test)]
fn persisted_transition_version(
remote_version: &str,
) -> std::io::Result<(Option<String>, rustfs_filemeta::TransitionVersionState)> {
persisted_transition_version_with_gate(remote_version, remote_version_state_writer_enabled())
}
#[cfg(test)]
fn remote_version_state_writer_enabled() -> bool {
remote_version_state_writer_fleet_proof().is_some()
}
fn remote_version_state_writer_fleet_proof() -> Option<crate::services::notification_sys::RemoteVersionStateFleetProofToken> {
remote_version_state_writer_requested()
.then(crate::services::notification_sys::acquire_remote_version_state_fleet_proof)
.flatten()
}
fn remote_version_state_writer_requested() -> bool {
remote_version_state_writer_enabled_for(
rustfs_utils::get_env_bool(
rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_WRITE,
@@ -2336,11 +2362,25 @@ fn remote_version_state_writer_enabled() -> bool {
rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
rustfs_config::DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
),
true,
)
}
fn remote_version_state_writer_enabled_for(requested: bool, fleet_confirmed: bool) -> bool {
requested && fleet_confirmed
fn remote_version_state_writer_fleet_proof_matches(
proof: &crate::services::notification_sys::RemoteVersionStateFleetProofToken,
) -> bool {
remote_version_state_writer_fleet_proof_matches_for(
remote_version_state_writer_requested(),
crate::services::notification_sys::remote_version_state_fleet_proof_matches(proof),
)
}
fn remote_version_state_writer_fleet_proof_matches_for(requested: bool, fleet_proof_matches: bool) -> bool {
requested && fleet_proof_matches
}
fn remote_version_state_writer_enabled_for(requested: bool, fleet_confirmed: bool, fleet_proof_valid: bool) -> bool {
requested && fleet_confirmed && fleet_proof_valid
}
fn persisted_transition_version_with_gate(
@@ -2359,7 +2399,7 @@ fn persisted_transition_version_with_gate(
Ok(_) => Ok((Some(remote_version.to_string()), rustfs_filemeta::TransitionVersionState::Exact)),
Err(_) if !remote_version_state_writer_enabled => Err(std::io::Error::new(
std::io::ErrorKind::Unsupported,
"opaque remote tier versions require the operator-attested fleet gate",
"opaque remote tier versions require the operator-attested live fleet capability gate",
)),
Err(_) if remote_version == "null" => {
Ok((Some(remote_version.to_string()), rustfs_filemeta::TransitionVersionState::SuspendedNull))
@@ -2605,7 +2645,7 @@ mod transition_upload_completion_tests {
mod transition_version_id_tests {
use super::{
TransitionUploadCandidate, persisted_transition_version, persisted_transition_version_with_gate,
remote_version_state_writer_enabled_for,
remote_version_state_writer_enabled_for, remote_version_state_writer_fleet_proof_matches_for,
};
use rustfs_filemeta::TransitionVersionState;
use uuid::Uuid;
@@ -2648,15 +2688,35 @@ mod transition_version_id_tests {
#[test]
fn remote_version_state_writer_requires_request_and_fleet_confirmation() {
for (case, requested, fleet_confirmed, expected) in [
("old defaults", false, false, false),
("missing fleet confirmation", true, false, false),
("missing local opt-in", false, true, false),
("explicitly unconfirmed fleet", true, false, false),
("rolled-back writer", false, true, false),
("fully upgraded fleet", true, true, true),
for (case, requested, fleet_confirmed, fleet_proof_valid, expected) in [
("old defaults", false, false, false, false),
("missing fleet confirmation", true, false, true, false),
("missing local opt-in", false, true, true, false),
("missing fleet proof", true, true, false, false),
("explicitly unconfirmed fleet", true, false, true, false),
("rolled-back writer", false, true, true, false),
("fully upgraded fleet", true, true, true, true),
] {
assert_eq!(remote_version_state_writer_enabled_for(requested, fleet_confirmed), expected, "{case}");
assert_eq!(
remote_version_state_writer_enabled_for(requested, fleet_confirmed, fleet_proof_valid),
expected,
"{case}"
);
}
}
#[test]
fn remote_version_state_commit_rechecks_operator_gate_and_live_proof() {
for (case, requested, fleet_proof_matches, expected) in [
("operator gate closed", false, true, false),
("fleet proof changed", true, false, false),
("current authorization", true, true, true),
] {
assert_eq!(
remote_version_state_writer_fleet_proof_matches_for(requested, fleet_proof_matches),
expected,
"{case}"
);
}
}
@@ -3841,6 +3901,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let dest_obj = transaction.remote_object.clone();
let mut transition_meta = (*oi.user_defined).clone();
transition_meta.insert("name".to_string(), object.to_string());
rustfs_utils::http::metadata_compat::insert_str(
&mut transition_meta,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID,
transaction.transaction_id.to_string(),
);
rustfs_utils::http::metadata_compat::insert_str(
&mut transition_meta,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
rustfs_utils::crypto::hex(transaction.backend_fingerprint),
);
if let Some(content_type) = oi.content_type.as_ref().filter(|value| !value.is_empty()) {
transition_meta.insert(CONTENT_TYPE.to_ascii_lowercase(), content_type.clone());
@@ -3951,20 +4021,24 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
delete_transition_transaction_after_remote_cleanup(transaction_api.as_ref(), transaction_id, bucket, object).await;
return Err(err.into());
}
let (transition_version_id, transition_version_state) = match persisted_transition_version(candidate.remote_version()) {
Ok(version) => version,
Err(err) => {
let cleanup_api = transition_cleanup_store(&self.ctx).await;
if let Err(cleanup_err) = upload_cleanup.cleanup_rejected_upload(cleanup_api).await {
return Err(StorageError::Io(std::io::Error::other(format!(
"{err}; rejected remote upload cleanup failed: {cleanup_err}"
))));
let fleet_proof = remote_version_state_writer_fleet_proof();
let remote_version_requires_fleet_proof =
!candidate.remote_version().is_empty() && Uuid::parse_str(candidate.remote_version()).is_err();
let (transition_version_id, transition_version_state) =
match persisted_transition_version_with_gate(candidate.remote_version(), fleet_proof.is_some()) {
Ok(version) => version,
Err(err) => {
let cleanup_api = transition_cleanup_store(&self.ctx).await;
if let Err(cleanup_err) = upload_cleanup.cleanup_rejected_upload(cleanup_api).await {
return Err(StorageError::Io(std::io::Error::other(format!(
"{err}; rejected remote upload cleanup failed: {cleanup_err}"
))));
}
delete_transition_transaction_after_remote_cleanup(transaction_api.as_ref(), transaction_id, bucket, object)
.await;
return Err(err.into());
}
delete_transition_transaction_after_remote_cleanup(transaction_api.as_ref(), transaction_id, bucket, object)
.await;
return Err(err.into());
}
};
};
if let Err(err) = advance_and_save_transition_transaction(
transaction_api.as_ref(),
&mut transaction,
@@ -4080,6 +4154,21 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
}
#[cfg(test)]
pause_transition_commit(bucket, object, TransitionCommitPause::AfterLeaseValidation).await;
// This check is the fleet-proof lease linearization point. Revocation
// blocks later commits; an already-authorized local quorum commit is
// allowed to finish without holding a synchronous lock across I/O.
if remote_version_requires_fleet_proof
&& !fleet_proof
.as_ref()
.is_some_and(remote_version_state_writer_fleet_proof_matches)
{
drop(transition_lock_guard);
if upload_cleanup.cleanup().await.is_ok() {
delete_transition_transaction_after_remote_cleanup(transaction_api.as_ref(), transaction_id, bucket, object)
.await;
}
return Err(Error::other("remote version state fleet capability changed during transition"));
}
if let Err(err) = advance_and_save_transition_transaction(
transaction_api.as_ref(),
&mut transaction,
@@ -4461,6 +4550,61 @@ mod erasure_construction_tests {
}
}
#[cfg(test)]
mod object_encryption_resolver_wiring_tests {
use super::*;
use crate::object_api::{EncryptionResolutionError, ObjectEncryptionResolver, ReadEncryptionMaterial, ReadEncryptionRequest};
use std::io::Cursor;
use std::sync::atomic::{AtomicUsize, Ordering};
struct CountingResolver {
calls: AtomicUsize,
}
#[async_trait::async_trait]
impl ObjectEncryptionResolver for CountingResolver {
async fn resolve_read_material(
&self,
_request: ReadEncryptionRequest<'_>,
) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
self.calls.fetch_add(1, Ordering::Relaxed);
Ok(None)
}
}
#[tokio::test]
async fn get_object_reader_forwards_instance_resolver() {
let resolver = Arc::new(CountingResolver {
calls: AtomicUsize::new(0),
});
let ctx = InstanceContext::new();
assert!(
ctx.set_object_encryption_resolver(resolver.clone()).is_ok(),
"fresh context should accept resolver"
);
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: 1,
user_defined: Arc::new(HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())])),
..Default::default()
};
let result = get_object_reader_with_context(
&ctx,
Box::new(Cursor::new(Vec::<u8>::new())),
None,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await;
assert!(result.is_err(), "resolver returning no material must fail closed");
assert_eq!(resolver.calls.load(Ordering::Relaxed), 1);
}
}
#[cfg(test)]
pub(in crate::set_disk::ops) mod hermetic_set_disks_support {
//! Shared hermetic `SetDisks` construction for the ops tests below: the
+7 -7
View File
@@ -199,7 +199,7 @@ impl SetDisks {
);
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub async fn read_version_optimized(
&self,
bucket: &str,
@@ -238,7 +238,7 @@ impl SetDisks {
}
#[tracing::instrument(level = "debug", skip(self))]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn get_object_fileinfo(
&self,
bucket: &str,
@@ -410,7 +410,7 @@ impl SetDisks {
Ok((fi, parts_metadata, op_online_disks))
}
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn get_object_info_and_quorum(
&self,
bucket: &str,
@@ -605,7 +605,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn get_object_with_fileinfo<W>(
// &self,
bucket: &str,
@@ -1140,7 +1140,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn get_object_decode_reader_with_fileinfo(
bucket: &str,
object: &str,
@@ -1296,7 +1296,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
async fn build_codec_streaming_part_reader(
bucket: &str,
object: &str,
@@ -1469,7 +1469,7 @@ fn multipart_part_checksum_algo(fi: &FileInfo, part_number: usize) -> HashAlgori
/// `get_object_with_fileinfo` (backlog#870) so both report the same
/// stage-duration semantics.
#[allow(clippy::too_many_arguments)]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
async fn setup_multipart_part_readers(
files: &[FileInfo],
disks: &[Option<DiskStore>],
+56 -4
View File
@@ -310,12 +310,13 @@ impl ECStore {
meta.set_created(opts.created_at);
if opts.lock_enabled {
meta.object_lock_config_xml = crate::bucket::utils::serialize::<ObjectLockConfiguration>(&enableObjcetLockConfig)?;
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&enableVersioningConfig)?;
meta.object_lock_config_xml =
crate::bucket::utils::serialize::<ObjectLockConfiguration>(&ENABLED_OBJECT_LOCK_CONFIG)?;
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
}
if opts.versioning_enabled {
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&enableVersioningConfig)?;
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
}
await_bucket_namespace_operation(
@@ -568,6 +569,7 @@ mod tests {
};
use crate::bucket::metadata::table_bucket_catalog_metadata_prefix;
use crate::bucket::metadata_sys;
use crate::cluster::rpc::peer_s3_client::install_delete_bucket_empty_scan_barrier;
use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
use crate::error::StorageError;
use crate::object_api::{ObjectOptions, PutObjReader};
@@ -589,6 +591,7 @@ mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
use time::OffsetDateTime;
use tokio::io::AsyncReadExt;
use tokio::sync::{Notify, OnceCell};
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
@@ -1048,7 +1051,7 @@ mod tests {
.await
.expect("bucket should be created");
for disk_path in &disk_paths {
tokio::fs::create_dir_all(disk_path.join(&bucket).join("empty-directory"))
tokio::fs::create_dir_all(disk_path.join(&bucket).join("empty-directory/nested/leaf"))
.await
.expect("empty directory remnant should be created");
}
@@ -1257,6 +1260,55 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn bucket_delete_preserves_put_committed_after_empty_scan() {
let (_, ecstore) = setup_bucket_delete_test_env().await;
let bucket = format!("bucket-delete-empty-scan-race-{}", Uuid::new_v4().simple());
let object = "committed-after-empty-scan";
let payload = b"object committed after DeleteBucket empty scan".to_vec();
ecstore
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let barrier = install_delete_bucket_empty_scan_barrier();
let delete_store = ecstore.clone();
let delete_bucket = bucket.clone();
let delete = tokio::spawn(async move {
delete_store
.delete_bucket(&delete_bucket, &DeleteBucketOptions::default())
.await
});
barrier.wait_until_paused().await;
let mut put_reader = PutObjReader::from_vec(payload.clone());
ecstore
.put_object(&bucket, object, &mut put_reader, &ObjectOptions::default())
.await
.expect("PUT should commit while DeleteBucket is paused after its empty scan");
barrier.release();
let err = delete
.await
.expect("DeleteBucket task should join")
.expect_err("DeleteBucket must reject a PUT committed after its empty scan");
assert!(matches!(err, StorageError::BucketNotEmpty(name) if name == bucket));
let mut reader = ecstore
.get_object_reader(&bucket, object, None, http::HeaderMap::new(), &ObjectOptions::default())
.await
.expect("committed object should remain readable after DeleteBucket fails");
let mut restored = Vec::new();
reader
.stream
.read_to_end(&mut restored)
.await
.expect("object body should remain readable");
assert_eq!(restored, payload);
}
#[tokio::test]
#[serial]
async fn bucket_recreation_does_not_publish_unverified_usage() {
+137 -2
View File
@@ -30,6 +30,7 @@ impl ECStore {
};
let mut count_no_heal = 0;
let mut first_error = None;
for pool in self.pools.iter() {
let (mut result, err) = pool.heal_format(dry_run).await?;
if let Some(err) = err {
@@ -37,8 +38,8 @@ impl ECStore {
StorageError::NoHealRequired => {
count_no_heal += 1;
}
_ => {
continue;
err => {
first_error.get_or_insert(err);
}
}
}
@@ -47,6 +48,9 @@ impl ECStore {
r.before.drives.append(&mut result.before.drives);
r.after.drives.append(&mut result.after.drives);
}
if let Some(err) = first_error {
return Ok((r, Some(err)));
}
if count_no_heal == self.pools.len() {
info!(
event = EVENT_HEAL_FORMAT_COMPLETED,
@@ -165,3 +169,134 @@ impl ECStore {
Err(StorageError::NotImplemented)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disk::{DiskOption, format::FormatV3, new_disk};
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
use crate::store::init_format::{load_format_erasure, save_format_file};
#[tokio::test]
async fn handle_heal_format_continues_after_a_pool_error() {
let canonical_format = FormatV3::new(1, 3);
let mut foreign_format = canonical_format.clone();
foreign_format.id = Uuid::new_v4();
let mut temp_dirs = Vec::new();
let mut endpoints = Vec::new();
let mut disks = Vec::new();
for disk_index in 0..3 {
let temp_dir = tempfile::tempdir().expect("temporary disk root should be created");
let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8"))
.expect("temporary endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_index);
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("temporary disk should open");
let mut disk_format = foreign_format.clone();
disk_format.erasure.this = foreign_format.erasure.sets[0][disk_index];
save_format_file(&Some(disk.clone()), &Some(disk_format))
.await
.expect("foreign format should be written");
temp_dirs.push(temp_dir);
endpoints.push(endpoint);
disks.push(Some(disk));
}
let pool_endpoints = PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 3,
endpoints: Endpoints::from(endpoints),
cmd_line: "foreign-format-majority-test".to_string(),
platform: "test".to_string(),
};
let pool = Sets::new(disks, &pool_endpoints, &canonical_format, 0, 1)
.await
.expect("test pool should build around the cached canonical format");
let mut recoverable_format = FormatV3::new(1, 3);
recoverable_format.id = canonical_format.id;
let mut recoverable_temp_dirs = Vec::new();
let mut recoverable_endpoints = Vec::new();
let mut recoverable_disks = Vec::new();
let mut unformatted_disk = None;
for disk_index in 0..3 {
let temp_dir = tempfile::tempdir().expect("temporary disk root should be created");
let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8"))
.expect("temporary endpoint should parse");
endpoint.set_pool_index(1);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_index);
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("temporary disk should open");
if disk_index < 2 {
let mut disk_format = recoverable_format.clone();
disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index];
save_format_file(&Some(disk.clone()), &Some(disk_format))
.await
.expect("recoverable format should be written");
} else {
unformatted_disk = Some(disk.clone());
}
recoverable_temp_dirs.push(temp_dir);
recoverable_endpoints.push(endpoint);
recoverable_disks.push(Some(disk));
}
let recoverable_pool_endpoints = PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 3,
endpoints: Endpoints::from(recoverable_endpoints),
cmd_line: "recoverable-format-test".to_string(),
platform: "test".to_string(),
};
let recoverable_pool = Sets::new(recoverable_disks, &recoverable_pool_endpoints, &recoverable_format, 1, 1)
.await
.expect("recoverable test pool should build");
let endpoint_pools = EndpointServerPools::from(vec![pool_endpoints.clone(), recoverable_pool_endpoints.clone()]);
let store = ECStore {
id: canonical_format.id,
disk_map: HashMap::new(),
pools: vec![pool, recoverable_pool],
peer_sys: S3PeerSys::new(&endpoint_pools),
pool_meta: RwLock::new(PoolMeta::default()),
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::new(()),
ctx: crate::runtime::instance::bootstrap_ctx(),
};
let (result, err) = store
.handle_heal_format(false)
.await
.expect("format heal should return the typed pool error");
assert!(
matches!(err, Some(StorageError::CorruptedFormat)),
"foreign format majority must not be downgraded to a successful heal: {err:?}"
);
assert_eq!(result.disk_count, 3, "the recoverable pool should still be inspected");
let healed = load_format_erasure(&unformatted_disk.expect("the unformatted disk handle should be retained"), true)
.await
.expect("the later pool should be healed despite the first pool error");
assert_eq!(healed.erasure.this, recoverable_format.erasure.sets[0][2]);
}
}
+227 -10
View File
@@ -101,6 +101,10 @@ fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
}
fn should_retry_format_load(err: &Error) -> bool {
!matches!(err, Error::CorruptedFormat)
}
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
rebalance_meta_loaded && !decommission_running
}
@@ -294,7 +298,7 @@ impl ECStore {
// periodic monitoring until format loading succeeds. Startup RPC
// failures can still spawn recovery probes for peers that come up
// after this node.
let (disks, errs) = init_format::init_disks(
let (mut disks, errs) = init_format::init_disks(
&pool_eps.endpoints,
&DiskOption {
cleanup: true,
@@ -309,9 +313,10 @@ impl ECStore {
let mut times = 0;
let mut interval = 1;
loop {
match init_format::connect_load_init_formats(
match init_format::connect_load_init_formats_with_instance_ctx(
&instance_ctx,
pool_first_is_local,
&disks,
&mut disks,
pool_eps.set_count,
pool_eps.drives_per_set,
deployment_id,
@@ -319,6 +324,7 @@ impl ECStore {
.await
{
Ok(fm) => break Ok(fm),
Err(e) if !should_retry_format_load(&e) => break Err(e),
// Wrap the final error if we are giving up
Err(e) if times >= 10 => {
break Err(Error::other(format!("store init failed to load formats after {times} retries: {e}")));
@@ -551,7 +557,7 @@ mod tests {
LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES, load_pool_meta_for_startup, pool_first_endpoint_is_local,
pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
};
#[cfg(feature = "test-util")]
use crate::{
@@ -562,10 +568,12 @@ mod tests {
},
tier_sweeper::Jentry,
transition_transaction::{
TRANSITION_TRANSACTION_RECORD_PREFIX, TransitionCleanupDecision, TransitionCleanupProof, TransitionRemoteVersion,
TransitionSourceIdentity, TransitionSourceVersionMode, TransitionTransaction, TransitionTransactionInit,
TransitionTransactionState, load_transition_transaction_record, recover_transition_transaction_records,
save_transition_transaction_record,
TRANSITION_TRANSACTION_RECORD_PREFIX, TransitionCleanupDecision, TransitionCleanupProof, TransitionOperatorError,
TransitionOperatorProbe, TransitionRemoteVersion, TransitionSourceIdentity, TransitionSourceVersionMode,
TransitionTransaction, TransitionTransactionInit, TransitionTransactionState,
delete_transition_candidate_for_operator, finalize_missing_transition_transaction_for_operator,
inspect_transition_transaction_for_operator, load_transition_transaction_record,
recover_transition_transaction_records, save_transition_transaction_record,
},
},
client::transition_api::ReaderImpl,
@@ -575,6 +583,7 @@ mod tests {
services::tier::{
test_util::{MockWarmBackend, MockWarmOp, TransitionCleanupStoreBarrier, register_mock_tier},
tier::{TIER_CONFIG_FILE, TierConfigMgr},
tier_config::{TierConfig, TierType, TierWasabi},
tier_mutation_intent::{
TIER_MUTATION_INTENT_RECORD_PREFIX, TierMutationIntent, TierMutationIntentKind, TierMutationIntentState,
TierMutationIntentTarget, advance_tier_mutation_intent_record_idempotent, delete_tier_mutation_intent_record,
@@ -770,6 +779,13 @@ mod tests {
assert!(!should_retry_local_decommission_resume(&StorageError::SlowDown, 0));
}
#[test]
fn test_should_retry_format_load_rejects_permanent_corruption() {
assert!(!should_retry_format_load(&StorageError::CorruptedFormat));
assert!(should_retry_format_load(&StorageError::ErasureReadQuorum));
assert!(should_retry_format_load(&StorageError::FirstDiskWait));
}
#[test]
fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
assert!(should_auto_start_rebalance_after_init(false, true));
@@ -1163,6 +1179,37 @@ mod tests {
.len()
}
#[cfg(feature = "test-util")]
async fn register_transition_reconcile_test_tier(
handle: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
tier_name: &str,
) -> MockWarmBackend {
let backend = MockWarmBackend::new();
let mut manager = handle.write().await;
manager.tiers.insert(
tier_name.to_string(),
TierConfig {
version: "v1".to_string(),
tier_type: TierType::Wasabi,
name: tier_name.to_string(),
wasabi: Some(TierWasabi {
name: tier_name.to_string(),
endpoint: "https://s3.wasabisys.com".to_string(),
access_key: "test-access-key".to_string(),
secret_key: "test-secret-key".to_string(),
bucket: "mock-tier".to_string(),
prefix: format!("mock/{}/", uuid::Uuid::new_v4()),
region: "us-east-1".to_string(),
}),
..Default::default()
},
);
manager
.install_test_driver(tier_name, Box::new(backend.clone()))
.expect("mock fallback tier driver should install");
backend
}
#[cfg(feature = "test-util")]
async fn wait_for_tier_delete_journal_recovery(
store: Arc<crate::store::ECStore>,
@@ -1213,6 +1260,16 @@ mod tests {
let registered: Vec<String> = instance_ctx.local_disk_map().read().await.keys().cloned().collect();
assert_eq!(registered.len(), 4, "the passed context must register all four local disks");
let registered_disk_ids = instance_ctx.local_disk_id_map();
let registered_disk_ids = registered_disk_ids.read().await;
assert_eq!(registered_disk_ids.len(), 4, "the passed context must publish all four disk IDs");
for endpoint in registered_disk_ids.values() {
assert!(
registered.contains(endpoint),
"every disk ID in the passed context must resolve to one of its registered endpoints"
);
}
drop(registered_disk_ids);
let bootstrap = crate::runtime::instance::bootstrap_ctx();
assert_ne!(
bootstrap.deployment_id(),
@@ -1227,6 +1284,15 @@ mod tests {
"the bootstrap context must not absorb the fresh store's disks"
);
}
drop(bootstrap_map);
let bootstrap_disk_ids = bootstrap.local_disk_id_map();
let bootstrap_disk_ids = bootstrap_disk_ids.read().await;
for endpoint in bootstrap_disk_ids.values() {
assert!(
!registered.contains(endpoint),
"the bootstrap context must not absorb the fresh store's disk IDs"
);
}
}
#[tokio::test]
@@ -2738,7 +2804,7 @@ mod tests {
.await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await;
let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await;
let backend_identity = TierConfigMgr::acquire_operation_lease(&ctx.tier_config_mgr(), tier_name)
.await
.expect("tier lease should resolve")
@@ -2809,6 +2875,157 @@ mod tests {
}
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn operator_reconcile_deletes_exact_candidate_before_finalizing_record() {
let temp_dir = tempfile::tempdir().expect("create temp store dir");
let (ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "transition-operator-reconcile", &[4])).await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let tier_name = "TXOPERATOR";
let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await;
let backend_identity = TierConfigMgr::acquire_operation_lease(&ctx.tier_config_mgr(), tier_name)
.await
.expect("tier lease should resolve")
.backend_identity();
let mut transaction = TransitionTransaction::new(TransitionTransactionInit {
deployment_id: ctx.deployment_id().expect("test store should initialize deployment id"),
transaction_id: uuid::Uuid::new_v4(),
owner_epoch: uuid::Uuid::new_v4(),
write_id: uuid::Uuid::new_v4(),
source: TransitionSourceIdentity {
bucket: "source-bucket".to_string(),
object: "source-object".to_string(),
version_id: None,
data_dir: uuid::Uuid::new_v4(),
mod_time_unix_nanos: 1_770_000_000_000_000_000,
size: 42,
etag: "source-etag".to_string(),
version_mode: TransitionSourceVersionMode::Unversioned,
},
tier_name: tier_name.to_string(),
backend_fingerprint: backend_identity,
not_after_unix_nanos: 1,
})
.expect("transaction should build");
transaction
.advance(transaction.fence(), TransitionTransactionState::UploadOutcomeUnknown, None)
.expect("transaction should enter unknown upload outcome state");
let remote_version = uuid::Uuid::new_v4().to_string();
backend.set_put_remote_version(Some(remote_version.clone())).await;
let candidate = bytes::Bytes::from_static(b"operator-confirmed transition candidate");
backend
.put(
&transaction.remote_object,
ReaderImpl::Body(candidate.clone()),
i64::try_from(candidate.len()).expect("test candidate length should fit i64"),
)
.await
.expect("mock backend should accept candidate");
save_transition_transaction_record(store.clone(), &transaction)
.await
.expect("transaction record should persist");
let status = inspect_transition_transaction_for_operator(store.clone(), transaction.transaction_id)
.await
.expect("operator inspection should probe the candidate");
assert_eq!(status.probe, TransitionOperatorProbe::VersionedPresent(remote_version.clone()));
let wrong_version = uuid::Uuid::new_v4().to_string();
let err = delete_transition_candidate_for_operator(store.clone(), transaction.transaction_id, &wrong_version)
.await
.expect_err("a mismatched exact version must fail before deleting a candidate");
assert!(matches!(
err,
TransitionOperatorError::CandidateVersionMismatch {
expected,
actual: TransitionOperatorProbe::VersionedPresent(ref observed),
} if expected == wrong_version && observed == &remote_version
));
assert!(backend.contains(&transaction.remote_object).await);
assert_eq!(backend.exact_remove_count(), 0);
load_transition_transaction_record(store.clone(), transaction.transaction_id)
.await
.expect("an incorrect exact version must retain the transaction journal");
let result = delete_transition_candidate_for_operator(store.clone(), transaction.transaction_id, &remote_version)
.await
.expect("operator-confirmed exact candidate should be deleted");
assert_eq!(result.status.probe, TransitionOperatorProbe::Missing);
assert!(result.journal_observed_after_delete);
assert_eq!(backend.exact_remove_count(), 1);
assert_eq!(backend.remove_versions().await, vec![(transaction.remote_object.clone(), remote_version)]);
load_transition_transaction_record(store.clone(), transaction.transaction_id)
.await
.expect("candidate deletion must retain the transaction journal");
finalize_missing_transition_transaction_for_operator(store.clone(), transaction.transaction_id)
.await
.expect("a separately confirmed missing candidate should permit finalization");
assert!(matches!(
load_transition_transaction_record(store, transaction.transaction_id).await,
Err(Error::ConfigNotFound)
));
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn operator_finalize_retains_record_without_missing_proof() {
let temp_dir = tempfile::tempdir().expect("create temp store dir");
let (ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "transition-operator-fail-closed", &[4])).await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let tier_name = "TXOPERATORFAIL";
let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await;
let backend_identity = TierConfigMgr::acquire_operation_lease(&ctx.tier_config_mgr(), tier_name)
.await
.expect("tier lease should resolve")
.backend_identity();
let mut transaction = TransitionTransaction::new(TransitionTransactionInit {
deployment_id: ctx.deployment_id().expect("test store should initialize deployment id"),
transaction_id: uuid::Uuid::new_v4(),
owner_epoch: uuid::Uuid::new_v4(),
write_id: uuid::Uuid::new_v4(),
source: TransitionSourceIdentity {
bucket: "source-bucket".to_string(),
object: "source-object".to_string(),
version_id: None,
data_dir: uuid::Uuid::new_v4(),
mod_time_unix_nanos: 1_770_000_000_000_000_000,
size: 42,
etag: "source-etag".to_string(),
version_mode: TransitionSourceVersionMode::Unversioned,
},
tier_name: tier_name.to_string(),
backend_fingerprint: backend_identity,
not_after_unix_nanos: 1,
})
.expect("transaction should build");
transaction
.advance(transaction.fence(), TransitionTransactionState::UploadOutcomeUnknown, None)
.expect("transaction should enter unknown upload outcome state");
save_transition_transaction_record(store.clone(), &transaction)
.await
.expect("transaction record should persist");
backend
.set_transition_candidate_probe_override(Some(TransitionCandidateProbe::Unsupported))
.await;
assert!(matches!(
finalize_missing_transition_transaction_for_operator(store.clone(), transaction.transaction_id).await,
Err(TransitionOperatorError::CandidateNotMissing(TransitionOperatorProbe::Unsupported))
));
load_transition_transaction_record(store, transaction.transaction_id)
.await
.expect("an unsupported probe must retain the transaction journal");
assert_eq!(backend.remove_count().await, 0);
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
@@ -2819,7 +3036,7 @@ mod tests {
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let tier_name = "TXRESPONSELOSS";
let backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await;
let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await;
let bucket = "transition-response-loss-bucket";
let object = "source.bin";
store
File diff suppressed because it is too large Load Diff
+1
View File
@@ -6895,6 +6895,7 @@ mod test {
)
.await
.expect("local disk should be created");
disk.make_volume("bucket").await.expect("test bucket should be created");
let mut fi = FileInfo::new("object", 1, 1);
fi.volume = "bucket".to_owned();
fi.name = "object".to_owned();
+4 -4
View File
@@ -428,11 +428,11 @@ impl ECStore {
}
lazy_static! {
static ref enableObjcetLockConfig: ObjectLockConfiguration = ObjectLockConfiguration {
static ref ENABLED_OBJECT_LOCK_CONFIG: ObjectLockConfiguration = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
..Default::default()
};
static ref enableVersioningConfig: VersioningConfiguration = VersioningConfiguration {
static ref ENABLED_VERSIONING_CONFIG: VersioningConfiguration = VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
..Default::default()
};
@@ -989,7 +989,7 @@ mod tests {
init_local_disks(endpoint_pools.clone()).await.expect("init local disks");
let (disks, errs) = init_disks(
let (mut disks, errs) = init_disks(
&endpoint_pools.as_ref().first().expect("pool endpoints").endpoints,
&DiskOption {
cleanup: true,
@@ -999,7 +999,7 @@ mod tests {
.await;
assert!(errs.iter().all(|err| err.is_none()), "disk init should succeed: {errs:?}");
connect_load_init_formats(true, &disks, 1, 4, None)
connect_load_init_formats(true, &mut disks, 1, 4, None)
.await
.expect("initialize format metadata");
+1 -1
View File
@@ -238,7 +238,7 @@ impl ECStore {
}
#[instrument(skip(self, data))]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn handle_put_object_part(
&self,
bucket: &str,
+2 -2
View File
@@ -815,7 +815,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self))]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn handle_get_object_reader(
&self,
bucket: &str,
@@ -849,7 +849,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self, data))]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
#[hotpath::measure]
pub(super) async fn handle_put_object(
&self,
bucket: &str,
+73 -2
View File
@@ -28,8 +28,26 @@ async fn remember_local_disk_id(disk: &DiskStore) -> Option<Uuid> {
async fn remember_local_disk_id_with_instance_ctx(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore) -> Option<Uuid> {
let disk_id = disk.get_disk_id().await.ok().flatten()?;
runtime_sources::record_local_disk_id(instance_ctx, disk_id, disk.endpoint().to_string()).await;
Some(disk_id)
record_local_disk_id_if_active(instance_ctx, disk, disk_id)
.await
.then_some(disk_id)
}
async fn record_local_disk_id_if_active(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore, disk_id: Uuid) -> bool {
let endpoint = disk.endpoint().to_string();
let local_disk_map = instance_ctx.local_disk_map();
let local_disks = local_disk_map.read().await;
let Some(active_disk) = local_disks.get(&endpoint).and_then(Option::as_ref) else {
return false;
};
if !Arc::ptr_eq(active_disk, disk) {
return false;
}
// Lock order is local_disk_map -> local_disk_id_map so quarantine is the
// linearization point for rejecting an in-flight stale disk snapshot.
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
true
}
pub async fn find_local_disk(disk_path: &str) -> Option<DiskStore> {
@@ -228,6 +246,7 @@ pub async fn get_disk_infos(disks: &[Option<DiskStore>]) -> Vec<Option<DiskInfo>
#[cfg(test)]
mod tests {
use super::*;
use crate::disk::new_disk;
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
fn single_local_disk_pools(dir: &std::path::Path) -> EndpointServerPools {
@@ -314,4 +333,56 @@ mod tests {
);
}
}
#[tokio::test]
async fn stale_local_disk_snapshot_cannot_repopulate_the_id_registry() {
let temp_dir = tempfile::tempdir().expect("create temp disk dir");
let endpoint_pools = single_local_disk_pools(temp_dir.path());
let instance_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools)
.await
.expect("local disk should be registered");
let disk = instance_ctx
.local_disk_map()
.read()
.await
.values()
.find_map(|disk| disk.clone())
.expect("registered local disk");
let endpoint = disk.endpoint().to_string();
let disk_id = Uuid::new_v4();
let local_disk_map = instance_ctx.local_disk_map();
let mut quarantine = local_disk_map.write().await;
let replacement = new_disk(
&disk.endpoint(),
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("replacement disk should initialize");
assert!(!Arc::ptr_eq(&disk, &replacement));
let task_ctx = instance_ctx.clone();
let task_disk = disk.clone();
let remember = tokio::spawn(async move { record_local_disk_id_if_active(&task_ctx, &task_disk, disk_id).await });
tokio::task::yield_now().await;
quarantine.insert(endpoint.clone(), Some(replacement.clone()));
drop(quarantine);
assert!(!remember.await.expect("stale lookup task should complete"));
assert!(!instance_ctx.local_disk_id_map().read().await.contains_key(&disk_id));
let active = instance_ctx
.local_disk_map()
.read()
.await
.get(&endpoint)
.cloned()
.flatten()
.expect("replacement disk should remain registered");
assert!(Arc::ptr_eq(&active, &replacement));
assert!(record_local_disk_id_if_active(&instance_ctx, &replacement, disk_id).await);
assert_eq!(instance_ctx.local_disk_id_map().read().await.get(&disk_id), Some(&endpoint));
}
}
+43
View File
@@ -0,0 +1,43 @@
//! Test-only guard against tracing's process-global callsite-interest cache.
//!
//! Any test that installs its own subscriber and then asserts on span or event
//! context is asserting on process-global state. See
//! [`pin_callsite_interest_for_test`] for what goes wrong and why holding its
//! guard fixes it.
/// Stop a sibling test thread from poisoning tracing's process-global callsite
/// interest cache while this test asserts on span or event context.
///
/// `tracing` caches every callsite's `Interest` in **process-global** state, and
/// the first thread to reach a callsite fixes that value. While at most one
/// dispatcher is registered, tracing-core takes a fast path
/// (`Dispatchers::rebuilder` -> `Rebuilder::JustOne`) that derives a
/// newly-registered callsite's interest from whichever subscriber is current
/// *on the registering thread*, and registration happens exactly once (guarded
/// by a CAS in `DefaultCallsite::register`).
///
/// In a multi-threaded `cargo test` binary a sibling test therefore routinely
/// reaches a production callsite first, from a thread with no subscriber
/// installed: the interest is derived from that thread's `NoSubscriber` and
/// cached as `Interest::never()` for the whole process. From then on the
/// callsite is dead for *every* later caller, including a test that installed
/// its own subscriber — an `info_span!` silently evaluates to `Span::none()`, and
/// a `warn!`/`info!` event never fires at all.
///
/// Registering a second, inert dispatcher closes both halves of that race:
///
/// * constructing it rebuilds every *already-registered* callsite's interest
/// against the live dispatcher set — which includes the caller's subscriber —
/// repairing whatever a sibling may already have poisoned; and
/// * holding it alive keeps tracing-core off the single-dispatcher fast path, so
/// a callsite registered *later* by any thread is resolved against that live
/// set instead of the registering thread's `NoSubscriber`.
///
/// Call this **after** installing the test's subscriber, and hold the returned
/// guard for the rest of the test. Only the `cargo test` fallback needs it:
/// nextest runs each test in its own process, where there is no sibling thread
/// to lose the race to (see `docs/testing/README.md`).
#[must_use = "callsite interest is only pinned while the returned guard is held"]
pub(crate) fn pin_callsite_interest_for_test() -> tracing::Dispatch {
tracing::Dispatch::new(tracing_core::subscriber::NoSubscriber::default())
}

Some files were not shown because too many files have changed in this diff Show More