Compare commits

...

24 Commits

Author SHA1 Message Date
overtrue 1558c2ed3f Merge branch 'fix/ant-request-layer-batch' into fix/ant-table-catalog-batch 2026-11-05 01:17:10 +08:00
overtrue 524c774e2f fix(table-catalog): bound maintenance work and isolate table namespaces
Scan findings in the Iceberg table catalog:

- Metadata maintenance deleted every unreferenced *.metadata.json in a
  table's directory without checking whether a renamed table in another
  namespace still owned those files (backlog#3060); it now resolves the
  owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
  writer schema and no recursion depth limit; a self-referencing record
  schema overflowed the stack (backlog#3080). Schemas are validated for
  depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
  memory; a decompression bomb OOM-killed the server (backlog#3085). Row
  groups are streamed through a byte budget and oversized inputs are
  refused.
- An S3 request for a key outside an indexed table prefix listed the
  bucket's whole namespaces/ catalog prefix and failed once it held more
  than 4096 objects (backlog#3086); a per-bucket directory index answers
  index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
  modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
  left the bucket's catalog entry locked at 503 (backlog#3088); the rename
  runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
  starving every object write (backlog#3111); commits now hold it under a
  deadline and object writes are admitted between commits.

The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.

The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.
2026-11-05 01:17:10 +08:00
overtrue faf636e1c7 fix(api): drop imports left unused by the base merge 2026-11-05 00:53:14 +08:00
overtrue 2401a8d8ac Merge branch 'fix/ant-storage-layer-batch' into fix/ant-request-layer-batch
# Conflicts:
#	rustfs/src/app/object/put.rs
2026-11-05 00:52:11 +08:00
overtrue 0aa9ba496b Merge remote-tracking branch 'upstream-https/main' into fix/ant-storage-layer-batch
# Conflicts:
#	crates/ecstore/src/disk/local/delete_recovery.rs
2026-11-05 00:47:38 +08:00
overtrue d15dde7bb5 fix(s3): authorize what handlers store and bound request-driven work
Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.

Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
  s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
  the tag set it copies (backlog#3034); HeadObject with the replication
  check header still authorizes s3:GetObject (backlog#3047); the
  x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
  recursive force-delete with a REPLICA header authorizes every
  descendant as a delete (backlog#3070); include-deleted listings require
  s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
  table isolation and uses the same prefix it authorizes (backlog#3053,
  backlog#3054); If-Match probing needs read permission (backlog#3097);
  Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
  proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
  returns lock and tag headers only with permission (backlog#3113);
  IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
  (backlog#3049); bucket policies are parsed once and cached
  (backlog#3032); keystone: access keys no longer verify with an empty
  secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
  (backlog#3023); PutObject takes buffer and foreground admission only
  after body bytes arrive and bounds the wait (backlog#3031,
  backlog#3084); xl.meta size is bounded on the write path
  (backlog#3100); XML compat layers stream instead of buffering whole
  bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
  unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
  completion (backlog#3130); MinIO admin decryption runs Argon2id off the
  async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
  (backlog#3092, backlog#3093); control characters in keys and lifecycle
  rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
  mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
  Content-MD5 and admin/STS reject SigV2 replays with altered query or
  body (backlog#3037, backlog#3026); request logs redact presigned
  queries and handler spans skip credentials (backlog#3068,
  backlog#3107); SSE-C listings and completions withhold plaintext
  metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
  bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
  keys (backlog#3126); notifications fire after a client disconnect
  (backlog#3127); ListenNotification streams re-check access
  (backlog#3119).
2026-11-04 23:27:28 +08:00
overtrue 0c56ea85db test(e2e): seed legacy duplicate bucket tags offline for the OPA contract 2026-11-04 23:25:00 +08:00
Hauser Zhi 4d46bd25f8 perf(ecstore): parallelize bulk-delete lock and metadata fan-out (#9155)
* perf(ecstore): probe bulk-delete write locks in parallel waves

Bulk delete acquired one distributed write lock per key in serialized
order, paying a full quorum round-trip per key. Probe the unique keys in
bounded parallel waves with a short timeout instead; any contention drops
the whole wave and the canonical serial order resolves it as before, so
two overlapping batches can never deadlock.

* perf(ecstore): probe bulk-delete transaction locks in parallel

The set-layer delete-transaction locks were also acquired one quorum
round-trip per key after the raw write locks. Apply the same optimistic
all-or-nothing probe wave with a canonical serial fallback, and keep the
per-key error reporting the fallback already provided.

* perf(ecstore): pre-fetch bulk-delete source metadata in parallel

The admission loop ran one metadata quorum fan-out per key in sequence
on top of the per-key lock round-trips. Pre-fetch the source reads the
loop needs with bounded parallelism under the locks already held, and
derive the per-key read inputs from one shared helper so the pre-fetch
and the loop cannot disagree.

* test(ecstore): drop unused import in batch-delete probe test

* refactor(ecstore,lock): await wave releases and unify probe waves

Adversarial review of the bulk-delete lock waves found three real
issues. Distributed guard release is fire-and-forget on drop, so the
serial fallback could re-acquire keys while unlock RPCs were still in
flight and block on locks it already owned: guards are now released
with an awaited unlock before the fallback runs. The wave also probed
every key after the first failure, adding up to one probe timeout per
chunk of pure overhead under contention: probing now stops at the
first failing chunk. Finally the two copy-pasted wave implementations
are replaced by one shared probe_lock_wave, and the per-key delete
read inputs are resolved once instead of being recomputed.

Tests now assert the wave/fallback counters so they prove which path
ran, and an overlapping-batch test pins the no-deadlock property.

* refactor(ecstore,lock): batch bulk-delete lock rounds per key

Replace the optimistic probe waves with the bulk lock primitive the lock
layer already supports on the wire: one lock_batch RPC per lock client
for a whole DeleteObjects key set, per-key quorum guards with the usual
TTL heartbeat, and per-key outcomes. The batched round is deadlock-free
by construction (every lock server orders concurrent batches by the
same global (shard, key) order), so the probe timeout, the wave
release-and-wait dance, and the canonical serial fallback all go away.

Per-key lock outcomes are now reported per key end to end: a locked-out
key fails alone in the response while the rest of the batch proceeds,
matching the S3 DeleteObjects contract and the set layer's existing
behavior. The set layer's raw-write-lock and delete-transaction-lock
waves fold into the same primitive via one shared resource-folding
helper, and the fast-lock batch fast path now counts acquisitions so
shard metrics keep their meaning.

* perf(ecstore): move journal publish off the one-worker namespace lane

A bulk delete publishes one recovery transaction per key, and every
journal inspection and settle-plan closure ran on the per-disk
foreground namespace lane (one worker), serializing the whole publish
phase per drive behind every other mutation. Route those closures onto
the durable-mutation lane the fused delete prepare already uses (eight
workers per disk, fsyncs coalesce in the filesystem journal), and keep
32 journal publications in flight so a 16-drive set can feed those
lanes.

* test(ecstore): flatten lock-error iteration in overlap test

* fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock

Review findings on the batched lock rounds. A batch entry whose recovery
rolls back an interrupted prepared delete can bring a retained version
back; a later duplicate of the same key still held its pre-fetched
source-missing read, skipped the object-lock check, and deleted the
restored version. Recovery now evicts the pre-fetched reads of every
other entry naming the same object, so those entries take the loop's
live re-read and see the restored version.

The batch rollback release awaited the full unlock retry schedule,
including the deferred background pass, parking the request behind an
unreachable peer. The synchronous phase is now a single release pass
with the deferred retries spawned in the background.

* fix(ecstore): release local batch locks before deferred recovery

A set-layer batch delete that runs deferred key recovery re-acquires a
write lock on the same keys through recover_delete_object, but the raw
local batch guards were only ever released at function exit — the
distributed path releases its batch ids before the deferred loop and the
local path missed the mirror release, so the recovery deadlocked against
the batch's own locks and surfaced as a spurious Lock(Timeout) for every
deferred key.

A regression test plants the interrupted prepared delete this needs and
pins the end-to-end contract: a batch naming a GOVERNANCE-retained
version twice must deny both entries after the first entry's recovery
rolls the prepared delete back, and the restored version must survive.

* test(lock): cover batch acquire partial-grant rollback

The distributed acquire_guard_batch rollback path — a key granted by
fewer than quorum clients must fail alone and release its partial grants
before the call returns — was only reachable in distributed mode and had
no coverage. A scripted per-object LockClient pins both halves: the
below-quorum key reports QuorumNotReached with its partial vote, and the
orphaned grant is released synchronously while the quorate key's guard
stays held.
2026-11-04 13:20:13 +00:00
Ramakrishna Chilaka ff5f491c5a perf(ecstore): skip unmatched FdCache scans and maintenance wakeups (#9165)
Track active descriptor volumes, exact paths, and component prefixes in FdCacheTracker so unmatched invalidate_under, invalidate_volume, and invalidate_exact calls return without scanning Moka or waking the background maintenance task.
2026-11-04 13:12:11 +00:00
overtrue 3483a48fcb fix(ecstore,heal,select): bound storage-layer work driven by client input
Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:

- Multipart: upload ids are canonicalized before they name a staging path
  or a lock key (backlog#3027); trailing-slash keys are encoded like every
  other handler (backlog#3055); unlisted parts are removed at completion
  (backlog#3106); per-bucket uploads are found through a signed per-disk
  index instead of a scan of the shared staging area (backlog#3121);
  encrypted part ETags carry a per-upload secret so a write-only user
  cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
  (backlog#3038); a delete marker on a key whose xl.meta path is a
  directory, and sidecar names used as child keys, no longer wedge the
  journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
  bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
  counts delete markers against max-keys (backlog#3108), and rejects a
  marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
  DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
  bucket is skipped instead of failing the rebuild (backlog#3040), and
  directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
  modification time (backlog#3073, backlog#3074); Snowball transport
  headers are not forwarded to the tier (backlog#3090); an epoch
  modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
  (backlog#3131); SelectObjectContent releases its namespace locks once
  the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
  (backlog#3129); a bucket-metadata reload timeout no longer marks the
  peer offline (backlog#3124); replication target requests carry an
  operation deadline (backlog#3102); the scanner resumes its interrupted
  wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
  batches, Parquet dictionary bounds and pre-planner projection checks
  (backlog#3075, backlog#3076, backlog#3077, backlog#3078).
2026-11-04 21:02:58 +08:00
Chris An 7f5fa4b8f6 fix(protocols,kms,keystone): enforce client CA, protect the KMS probe (#9168)
Scan findings in the alternate-protocol front ends and the KMS and
Keystone integrations:

- WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE,
  checked that the file existed, and then built their TLS listeners
  with with_no_client_auth(), so mTLS was never enforced. A shared
  protocols::common::tls::server_config builds the rustls config with a
  client-certificate verifier from the configured CA, startup fails on
  an invalid CA, and a CA cannot be combined with cleartext
  (backlog#3069).
- The KMS readiness probe used a fixed key id that the key-lifecycle
  admin routes accepted like any customer key, so kms:DisableKey on it
  failed readiness on every node. The probe key is reserved from key
  management and a disabled probe key no longer counts as a backend
  failure (backlog#3115).
- The Keystone middleware answered anonymous requests carrying a junk
  X-Auth-Token with the configured Keystone URL in the body while
  Keystone was unreachable; failures no longer disclose upstream
  locations, and the login cache is bypassed when a subscription token
  refresh is required (backlog#3116).
2026-11-04 20:25:55 +08:00
Chris An a97c65895c fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and
that stand alone from the storage core:

- trusted-proxies: the simple implementation takes the client address
  from the right-most untrusted X-Forwarded-For hop instead of the
  left-most (backlog#3082); the legacy validator never falls back to the
  proxy identity when the chain is rejected and reads X-Forwarded-For
  unless RFC 7239 is explicitly selected (backlog#3051).
- targets: webhook and similar outbound clients drain responses up to a
  byte limit instead of buffering arbitrary bodies (backlog#3062); the
  on-disk notification queue has a shared byte budget in addition to the
  entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES
  (backlog#3122); AMQP and Redis manifests declare the broker url as a
  secret and bind credentials to their connection fields, and the
  plugin-instance PUT refuses to restore a redacted secret when the
  connection fields changed (backlog#3065, backlog#3066).
- obs: the debug-level s3s response dump is filtered so STS and
  service-account credentials never reach the log (backlog#3029).
2026-11-04 19:49:53 +08:00
Chris An 73e7afa01a fix(iam): close identity, session and admin-authorization gaps (#9163)
Scan findings in the IAM crate and the admin identity handlers, grouped
because they share the IamCache mutation lock, the STS session sweep and
the admin client-address plumbing:

- Admin and STS authorization used the socket peer as aws:SourceIp
  behind a trusted proxy; they now read the verified client address the
  S3 path already uses (backlog#3025), and the admin usage route passes
  the bucket named in the path to the s3:ListBucket check (backlog#3033).
- AssumeRoleWithWebIdentity accepted HS256 ID tokens MACed with the OIDC
  client secret; only provider-signed tokens verify (backlog#3028), and
  JWKS refresh is single-flight with a failure cool-down (backlog#3103).
- A disabled group emptied the member's whole policy list, dropping
  every Deny; direct and other-group policies are preserved (backlog#3042).
- Identity rewrites (secret change, add-user, status) ran without a lock
  and without re-reading the record, so a user's password change could
  undo an admin disable or reset (backlog#3048). add-user on an existing
  user and in-flight AssumeRole both left sessions minted under the old
  secret valid (backlog#3071, backlog#3079); rotation now revokes under
  the per-key mutation lock and derived-credential publication is ordered
  with it.
- aws:groups never carried a built-in user's IAM group membership and
  jwt:iss was always the local marker, so group- and issuer-scoped
  Denies never matched; both now come from live IAM state
  (backlog#3095, backlog#3096).
- OPA mode skipped the account-status gates for service accounts and STS
  sessions of a disabled user (backlog#3057).
- The periodic IAM reload published nothing when any cache write raced
  it and kept stale partitions when a prefix listed empty, so a missed
  revocation never applied (backlog#3064, backlog#3114).
- update-group-members never evaluated admin:RemoveUserFromGroup
  (backlog#3098); ExportIAM compared the site-replicator account against
  a name it never has (backlog#3045); add-user accepted the access key of
  a live STS session or service account (backlog#3134); a service account
  could create or extend a lifetime beyond its own (backlog#3112);
  sts:DurationSeconds and svc:DurationSeconds were never populated
  (backlog#3136); temporary-account-info returned service accounts
  (backlog#3137); an access key the object layer refuses no longer holds
  the service-account lock through the write retries (backlog#3133).
2026-11-04 18:44:50 +08:00
Hauser Zhi a8be876fd9 fix(s3): honor presigned x-amz-meta-* query parameters (#9146)
* fix(s3): honor presigned x-amz-meta-* query parameters

The AWS SDK presigners hoist signable x-amz-* headers into the query
string (smithy moveHeadersToQuery), and AWS S3 accepts the hoisted form
as the header it came from. RustFS read user metadata from headers only,
so a presigned PutObject carrying ?x-amz-meta-user-id=u1 returned 200 and
stored an object with empty metadata (rustfs/rustfs#9142).

Add one shared extractor for user-metadata query parameters and wire it
into every metadata-accepting write path (PutObject including snowball
members, CreateMultipartUpload, CopyObject REPLACE) plus the snowball
extract trigger, instead of normalizing per operation the way the
presigned UploadPart checksum fix did. The stored key form is produced by
the same helper as the header path, so the hoisted and header forms land
on disk identically.

Only a signature-covered query is honored: SigV4 signs the canonical
query string, while SigV2 signs headers plus a fixed subresource list
only, so honoring its query would let a presigned-URL holder add object
properties the presigner never authorized. Keys and values that could not
have arrived as a header are rejected rather than stored, since the read
path turns them into header fields and a control character would fail
every Head/Get of that object.

* refactor(s3): hoist signed x-amz query params at the request boundary

Replace the per-write-path query metadata extractor with one boundary
mechanism: restore the `x-amz-*` request properties the SigV4 presigners
hoist into the query (`@smithy/signature-v4` `moveHeadersToQuery`) as
request headers before s3s routes and deserializes the request.

One hook now covers every reader at once — s3s input parsing,
`extract_metadata_from_mime`, the snowball extract trigger and its
transport-key filter, tagging, storage class, object lock and copy
source — so PutObject, CreateMultipartUpload and CopyObject need no
changes at all. It also replaces repeated query parsing along the write
path with a single pass at the boundary.

An existing header always wins, so the hoist is purely additive and the
unsigned-header guard keeps rejecting anything the signer did not sign;
that guard now also accepts a header whose single value equals a
same-named query parameter, which is what the hoist produces and what
the signature covers.

`query_string_is_signed` rules out SigV2 before it trusts any SigV4
marker. SigV2's string-to-sign covers headers plus a fixed subresource
list rather than arbitrary query parameters, so a SigV2 presigned URL
stays valid after `X-Amz-Algorithm`/`X-Amz-Signature` markers are
appended to it; without this rule a URL holder could turn on
`x-amz-meta-snowball-auto-extract` and have a plain upload untarred
(review of #9146).

* fix(s3): hoist signed query params after s3s signature verification

The request-boundary hoist ran before s3s, so the restored `x-amz-*`
headers reached s3s's own verifier, which rejects any `x-amz-*` header
outside `X-Amz-SignedHeaders` (s3s `ops/signature.rs`,
`reject_unsigned_amz_headers`). A valid presigned PUT with
`SignedHeaders=host` and `?x-amz-meta-user-id=u1` therefore came back
with 403 AccessDenied before any handler ran (review of #9146).

Move the hoist to `authorize_request`, RustFS's shared authorization
entry: s3s has already verified the signature there, and policy
evaluation plus the handler read request properties after it, so the
restored headers reach every operation without ever appearing unsigned
to the verifier. The mechanism is unchanged — one gated, validated pass
over the signature-covered query.

CopyObject's replacement metadata now also reads the headers, because
s3s parsed `input.metadata` before the hoist ran.

The hoist itself and its tests stay in `server::layer`; the tower layer
is gone.

* perf(s3): single-pass signature-aware query hoist in auth

Move the query hoist into `auth.rs`, next to the unsigned-header rules it
is part of, so `storage` no longer reaches into `server` for it. Its
documented contract — which request properties the signature covers, and
why SigV2 must never be widened — now lives with the rule it protects.

Make the hot path allocation-free and single-pass. The caller no longer
clones the query just to satisfy the borrow checker, the function no
longer clones it again, and one `form_urlencoded::parse` pass both
recognizes the presigned-SigV4 marker set and stages the hoistable
parameters. Nothing is validated or applied until the gate passes, so a
request whose query is not signature-covered is never rejected here. A
single ASCII scan skips the whole thing when the query names no `x-amz-`
parameter at all, which is every ordinary `?versionId=`/`?uploadId=`
request and every queryless one.

The duplicate-parameter and header-value checks stay strict; only when
they moved, not how they behave.

* fix(s3): keep verifying a query-carried payload hash after the hoist

The hoist restores `?x-amz-content-sha256=<digest>` as a header, but it
runs after s3s has already built the request body. s3s reads its
payload-hash input from the headers only, so it installs no hash
validator for a digest that arrived in the query, and the restored header
then made `get_content_sha256_with_query` conclude the auth layer was
enforcing it. Neither layer verified the body, so a presigned PUT whose
digest exists only in the query accepted a modified payload (review of
#9146).

The question is not whether a digest header is present but whether the
digest reached the request as a query parameter: that one is always the
storage layer's to verify. A digest sent as a header is still left to the
auth layer, which wraps the body in a hash-validating reader for it, so
no payload is hashed twice.

* fix(auth): limit late query hoisting to user metadata

* refactor(s3): drop the stale payload-hash compensation, scope the pre-filter

Limiting the hoist to `x-amz-meta-*` user metadata means it never
manufactures an `x-amz-content-sha256` header, so the payload-hash
predicate's original signal — a digest header present at handler time is
one s3s is enforcing — is exact again. Restore it and drop the
query-aware form: that change existed only to undo a header the hoist is
no longer allowed to create, and it would have hashed a payload twice
for a presigned request that declared a real digest both in the query and
as a header.

Match the pre-filter to the same scope so a query carrying only a typed
property (`x-amz-tagging`, `x-amz-storage-class`, `versionId`) skips the
parse entirely instead of scanning past it.

* test(access): use explicit imports in the signed-query lock regression

The architecture migration rule requires RustFS storage-owner modules to
name their imports instead of pulling the parent module in wholesale, and
`signed_query_lock_permission_regression` was still on `use super::*`.
Import the two symbols it actually uses and drop the unused one.

* test(access): reference FS through the parent import

The layer dependency guard tracks `use crate::...` edges, and naming
`crate::storage::ecfs::FS` from the signed-query regression looked like a
new `infra -> interface` reverse dependency. `access.rs` already reaches
the same type through `use super::ecfs::FS`, so reuse that import instead
of introducing a second, tracked spelling of an edge the module has
always had. No baseline entry, no new edge.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-04 10:41:52 +00:00
Chris An e0f26a540d fix(ecstore): gate null-version overwrites under Object Lock (#9157)
A put-like write without a version id to a key that is not
versioning-Enabled replaces the key's null version in place. The
commit-time Object Lock gate in put_object_with_old_current_size_inner
ran only when the write carried an explicit version id, and the
request-layer pre-check reads the key's latest version. When the
retained null version was not the latest entry (a UUID version or a
delete marker sat above it), a user holding only s3:PutObject could
overwrite it together with its COMPLIANCE retention or legal hold, and
CompleteMultipartUpload had no commit-time gate at all.

Name the version a commit replaces (the explicit id, or the nil id when
the write is not versioned and the bucket's Object Lock snapshot is not
ConfirmedAbsent) and run the existing WORM gate against it, tolerating
a null delete marker. complete_multipart_upload gets the same gate at
its commit point, and ECStore::handle_complete_multipart_upload attaches
the bucket's Object Lock snapshot the way a single PUT already does.

Refs rustfs/backlog#3019 (ANT-2026-EH17ZP59).
2026-11-04 17:26:54 +08:00
Chris An 3b3cc018aa fix(policy): close condition, action, principal and resource gaps (#9160)
Eight scan findings in the shared policy engine, each a case where a
Deny did not apply or an Allow was broader than written:

- A bare `*` Action was parsed as `s3:*`, so `Deny *` never matched
  admin:, kms: or sts: actions. `Action::All` now denies every family
  while the historical S3-only Allow is preserved (backlog#3041).
- A bare IPv6 address in aws:SourceIp got `/32` appended, matching a
  2^96-address network; a bare address now means one host, and IPv4
  clients on a dual-stack listener are matched through their mapped
  form (backlog#3030).
- Date conditions on aws:EpochTime parsed the epoch seconds as RFC 3339
  and never matched, so a time-bounded Deny failed open (backlog#3036).
- NumericNotEquals and DateNotEquals evaluated to false when the key was
  absent; negated operators now follow the AWS rule and match
  (backlog#3123).
- Every trailing IfExists on a condition operator added a level of
  recursion without bound; the chain is rejected before parsing
  (backlog#3091).
- Policy variable substitution re-expanded its own output for up to ten
  passes, so an access key holding k placeholders grew to k^10; values
  are now substituted once as literal data (backlog#3067).
- A bucket-policy Service principal was matched against the signing
  access key, so a service account named after the service received its
  grants; Service principals round-trip but never match (backlog#3081).
- Resource matching cleaned the trailing slash off a directory object,
  so a Deny on `prefix/*` missed `prefix/` itself (backlog#3125).

Also deduplicate large Resource sets through a HashSet instead of a
quadratic scan.
2026-11-04 17:16:17 +08:00
Hauser Zhi fe3ada0689 chore(deps): refresh Cargo dependencies (#9156)
* chore(deps): refresh Cargo dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update rustfs kafka to 1.4.1

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-04 09:03:37 +00:00
Chris An f4f5635aab fix(auth): build policy conditions from the value a write stores (#9159)
PutObjectLegalHold and PutObjectTagging apply the value in their XML
body, and a POST upload applies its form fields, while the
s3:object-lock-legal-hold, s3:RequestObjectTag/<key> and
s3:RequestObjectTagKeys condition keys (and every other header-derived
key for POST) were built from the HTTP headers. The policy engine
therefore judged a value the handler did not store: absent when the
client sent no header, and whatever the caller wrote when it did. A
user limited by a Deny on the requested legal hold or tag value could
release a hold and delete the held version, or retag an object so a
tag-based grant exposed it.

Record the body value in request extensions before authorization, the
way PutObjectRetention already does, and let authorization_conditions
replace the header-derived keys with it. For POST uploads project the
parsed form fields over the headers that feed the condition map, removing
a header whose field is absent.

Refs rustfs/backlog#3018 (ANT-2026-KONBE376).
2026-11-04 17:00:42 +08:00
Hauser Zhi 926355ae1f fix(api): keep SlowDownRead off write-path quorum failures (#9149)
* fix(api): keep SlowDownRead off write-path quorum failures

* refactor(api): write-safe quorum default, GET opts into SlowDownRead

* fix(api): keep object-read mapping behind error.rs architecture boundary

* refactor(api): drop redundant object-read mapper wrapper
2026-11-04 08:08:23 +00:00
Chris An 36ad5a6b87 fix(ecstore): judge the null version a versionless delete removes (#9151)
DeleteObjects ran its Object Lock check on one version of a key and then
removed a different one. For an entry without a version id on a key that
is not versioning-Enabled, delete_objects_with_accounting judged the
key's latest entry and then had the disks remove the null version. When
a UUID version or a delete marker sat above a null version under
COMPLIANCE retention or legal hold, that retention was never read and a
user holding only s3:DeleteObject could remove the retained version. The
single-object DELETE on a Suspended key had the same gap.

Add SetDisks::check_null_version_object_lock_delete, which looks up the
null version under the lock the caller already holds and runs the
existing check_object_lock_delete against it when the latest entry
carries a UUID version id. The batch path is not gated on
source_missing because a delete marker on top makes the latest-version
lookup report the key as missing. Plain unversioned keys and
versioning-Enabled keys keep a single lookup and unchanged behavior.

Refs rustfs/backlog#3017 (ANT-2026-6ED2F7II).
2026-11-04 15:59:48 +08:00
Chris An 6c14128072 fix(s3): reserve the __XLDIR__ directory-marker suffix in client keys (#9153)
Storage encodes a directory object `photos/` as `photos__XLDIR__/xl.meta`
and LocalDisk::scan_dir decodes every directory entry ending in the
suffix back into a marker. The S3 write path never refused the suffix in
client keys, so a 1-byte PutObject to the key `__XLDIR__` made the walk
schedule the directory being scanned as its own child and recurse until
the stack overflowed, aborting the server on every restart until the
directory was removed by hand. Keys below a `*__XLDIR__/` segment were
stored but hidden from every listing and from quota accounting.

validate_object_key now rejects any client key containing the suffix,
which covers PutObject, CopyObject, Snowball extraction and the admin
internal put; CreateMultipartUpload and CompleteMultipartUpload now run
the same validation so uploads created before the key was reserved
cannot publish it. scan_dir additionally ignores a marker whose decoded
name is empty, so malformed markers left by older writers or corrupt
disks can no longer recurse into their parent.

Refs rustfs/backlog#3020 (ANT-2026-SBJ1JLC2).
2026-11-04 15:39:48 +08:00
Chris An addc258a24 fix(admin): keep Vault local-file reads on the operator connection (#9152)
POST /rustfs/admin/v3/kms/reconfigure and configure accepted
server-local file paths in the Vault request body (Kubernetes jwt_path,
AppRole secret_id_file, TokenFile path, TLS ca/client cert and key
paths), read the named file with the server's own authority and sent
its content to the Vault address given in the same body. A user holding
only kms:Configure could read any UTF-8 file the process can read,
including /proc/self/environ with the root credentials, and exfiltrate
it to an address of their choosing.

normalize_configure_request_secrets now runs
validate_vault_local_file_access for both handlers: a Vault request
that names a server-local file, or inherits the stored credentials by
sending a blank token, must keep the operator-configured address and
namespace, the same file-based auth method (role, path, mount) and the
same TLS paths and verification setting. Requests with explicit inline
credentials and no file paths are unchanged.

Refs rustfs/backlog#3021 (ANT-2026-F8MRFKCC).
2026-11-04 15:39:43 +08:00
Chris An 73ea527f55 fix(auth): canonicalize s3:VersionId before policy evaluation (#9150)
The object handlers accept several spellings of one version id (upper
case, unhyphenated, braced, urn:uuid:) and the storage layer resolves
all of them to the same version, but the s3:VersionId condition value
was the client string verbatim. A Deny with StringEquals on one version
could therefore be bypassed by re-spelling the id in ?versionId=.

Canonicalize the id once where the condition value is produced so every
hook and policy type (IAM, bucket, session) sees the spelling the server
emits: hyphenated lower-case UUID, with null in any case and the nil UUID
mapped to null. Non-UUID strings pass through unchanged and are rejected
by the handlers as before.

Refs rustfs/backlog#3022 (ANT-2026-O5MHY8VJ).
2026-11-04 15:39:38 +08:00
hector 55da14ad82 ci(nightly): embed the console UI in the GNU nightly build (#9158) 2026-11-04 15:21:41 +08:00
241 changed files with 33790 additions and 3425 deletions
+191 -3
View File
@@ -87,18 +87,206 @@ jobs:
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
# Packaging tools are off on this lane, so setup does not install unzip.
if ! command -v unzip >/dev/null 2>&1; then
sudo apt-get update -qq
sudo apt-get install -y -qq unzip
fi
if ! command -v unzip >/dev/null 2>&1; then
echo "unzip is required to extract console assets" >&2
exit 1
fi
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
chmod +w ./rustfs/static/LICENSE || true
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
shell: bash
run: |
# Force rebuild by touching build.rs so a restored cache cannot
# reuse a rustfs artifact compiled before these console assets existed.
touch rustfs/build.rs
cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bin rustfs
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Verify console
shell: bash
run: |
set -euo pipefail
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
binary="target/x86_64-unknown-linux-gnu/release/rustfs"
data_dir="$(mktemp -d)"
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$data_dir"
' EXIT
if [[ ! -x "$binary" ]]; then
echo "rustfs binary not found: ${binary}" >&2
exit 1
fi
api_port="$(find_free_port)"
console_port="$(find_free_port)"
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$binary" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
response=""
for _ in {1..40}; do
response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
echo "Console endpoint did not return 200 text/html: $response" >&2
cat "$log_file"
exit 1
- name: Build DEB package
id: deb
shell: bash
Generated
+19 -16
View File
@@ -4731,7 +4731,6 @@ version = "1.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "337d46834ee672ab3e48caca2cb0c78cc174fb12b3a68d0d88f99a0519a5e36e"
dependencies = [
"generic-array 0.14.7",
"rustversion",
"typenum",
]
@@ -9349,9 +9348,9 @@ dependencies = [
[[package]]
name = "russh"
version = "0.65.0"
version = "0.65.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3bb77a4b259312b2abbc541bca1c13cb4e9b61c560a7fd894068c3b92e0c9dc3"
checksum = "94be3d86c71b9ed220a675c405e35c7a02565a009bf8da2e9cc13a700320a5c5"
dependencies = [
"aes 0.9.4",
"aws-lc-rs",
@@ -10115,9 +10114,9 @@ dependencies = [
[[package]]
name = "rustfs-kafka"
version = "1.4.0"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bd2f8b46ec4dd462c2235948d1efa2103957230c171e118c6ff8e5d1a845ab48"
checksum = "82dc39ba1a5a9f6b2e49decb0b54c23ebaebdfb08a899307c266694bfd269bf9"
dependencies = [
"base64 0.23.1",
"bytes",
@@ -10140,9 +10139,9 @@ dependencies = [
[[package]]
name = "rustfs-kafka-async"
version = "1.4.0"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b3b98705f6f9f5071b8793d29935de764c31668c6e9877553b229eb67127af6"
checksum = "684f5e78ea1421bbf915e4cd0515bdefd587cfa281da000e41ae4a434f90195a"
dependencies = [
"base64 0.23.1",
"bytes",
@@ -10277,6 +10276,7 @@ dependencies = [
"hotpath",
"parking_lot 0.13.0",
"rand 0.10.3",
"rustfs-common",
"rustfs-io-metrics",
"rustfs-utils",
"serde",
@@ -10535,6 +10535,7 @@ dependencies = [
"md-5 0.11.0",
"percent-encoding",
"proptest",
"rcgen",
"regex",
"russh",
"russh-sftp",
@@ -10788,8 +10789,10 @@ version = "1.0.2"
dependencies = [
"async-trait",
"bytes",
"csv-core",
"datafusion",
"derive_builder",
"flate2",
"futures",
"hotpath",
"parking_lot 0.13.0",
@@ -11024,6 +11027,7 @@ version = "1.0.2"
dependencies = [
"async-trait",
"axum",
"futures",
"hotpath",
"http 1.5.0",
"ipnetwork",
@@ -11199,15 +11203,14 @@ dependencies = [
[[package]]
name = "rustls-connector"
version = "0.23.9"
version = "0.23.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "524994fd6957827c775abc941f41a146ce351bbd0a9b8cf632ee37f43a13c205"
checksum = "f799ef81c26fda57609b9456ae1e6750e2ca3dafc2dc5478232c0705f1458fbf"
dependencies = [
"futures-io",
"futures-rustls",
"log",
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
"rustls-webpki",
]
@@ -12858,9 +12861,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.1.8+spec-1.1.0"
version = "1.2.0+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20489e00e4d8741d6be680764cc12e270655e375a20d1011e844a9c3379e678d"
checksum = "c5ffcdf8f50436dbbfd675c1fe1b8e0854f42b7defea8584bdebe1aee900e561"
dependencies = [
"indexmap 2.14.2",
"serde_core",
@@ -12882,9 +12885,9 @@ dependencies = [
[[package]]
name = "toml_edit"
version = "0.25.17+spec-1.1.0"
version = "0.25.18+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c"
checksum = "1ec6e78bd70db9420d8d212fa9dc97f116448a1ceff5d00eabb991d407e21396"
dependencies = [
"indexmap 2.14.2",
"toml_datetime",
@@ -12903,9 +12906,9 @@ dependencies = [
[[package]]
name = "toml_writer"
version = "1.1.3+spec-1.1.0"
version = "1.2.0+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06bdbd8cfc056b8d2e2e85f29b56a3bdbecb527cef81eb39e3e7b98af4652770"
checksum = "009dbb7700f11c55ef83aebbe4c0e9cf1f1ac0844bd8d6f00dc38f302026e13a"
[[package]]
name = "tonic"
+4 -3
View File
@@ -168,7 +168,7 @@ http-body-util = "0.1.5"
memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.0" }
rustfs-kafka-async = { version = "1.4.1" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
@@ -191,6 +191,7 @@ quinn = { version = "0.11.12", default-features = false, features = [
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
csv-core = "0.1.13"
bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
@@ -212,7 +213,7 @@ serde_urlencoded = "0.7.1"
# releases.
aes-gcm = { version = "0.11.1" }
argon2 = { version = "0.6.0" }
aws-lc-rs = { version = "1.18.0", default-features = false }
aws-lc-rs = { version = "1.19.0", default-features = false }
blake2 = "0.11.0"
chacha20poly1305 = { version = "0.11.0" }
crc-fast = "1.10.0"
@@ -378,7 +379,7 @@ libunftp = { version = "0.24.0" }
unftp-core = "0.1.0"
suppaftp = { version = "12.2.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.0" }
russh = { version = "0.65.1" }
russh-sftp = "3.0.1"
# WebDAV
+3 -3
View File
@@ -30,10 +30,10 @@ pub const ENV_TRUSTED_PROXY_VALIDATION_MODE: &str = "RUSTFS_TRUSTED_PROXY_VALIDA
/// Default validation mode is "hop_by_hop".
pub const DEFAULT_TRUSTED_PROXY_VALIDATION_MODE: &str = "hop_by_hop";
/// Environment variable to enable RFC 7239 "Forwarded" header support.
/// Select RFC 7239 "Forwarded" instead of "X-Forwarded-For" in legacy mode.
pub const ENV_TRUSTED_PROXY_ENABLE_RFC7239: &str = "RUSTFS_TRUSTED_PROXY_ENABLE_RFC7239";
/// RFC 7239 support is enabled by default.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = true;
/// Legacy mode uses X-Forwarded-For unless RFC 7239 is explicitly selected.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = false;
/// Environment variable for the maximum allowed proxy hops.
pub const ENV_TRUSTED_PROXY_MAX_HOPS: &str = "RUSTFS_TRUSTED_PROXY_MAX_HOPS";
+6
View File
@@ -146,3 +146,9 @@ pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";
/// Queue-store compression is enabled by default to reduce disk footprint.
pub const DEFAULT_TARGET_STORE_COMPRESS: bool = true;
/// Maximum on-disk bytes in each target queue, including failed entries and pending writes.
pub const ENV_TARGET_STORE_MAX_BYTES: &str = "RUSTFS_TARGET_STORE_MAX_BYTES";
/// Queue writes fail at one GiB unless the operator sets another positive byte budget.
pub const DEFAULT_TARGET_STORE_MAX_BYTES: u64 = 1024 * 1024 * 1024;
+9
View File
@@ -94,3 +94,12 @@ pub(crate) mod scanner_peer_bucket {
pub(crate) use ecstore_erasure::{ReedSolomonEncoder, calc_shard_size};
pub(crate) use ecstore_rpc::PeerRestClient;
}
/// Storage handles used to seed a pre-validation bucket record while the fixture server is stopped.
#[cfg(test)]
pub(crate) mod bucket_tag_fixture {
pub(crate) use rustfs_ecstore::api::bucket::metadata::load_bucket_metadata;
pub(crate) use rustfs_ecstore::api::layout::{DisksLayout, EndpointServerPools};
pub(crate) use rustfs_ecstore::api::runtime::InstanceContext;
pub(crate) use rustfs_ecstore::api::storage::{ECStore, init_local_disks_with_instance_ctx};
}
+147 -29
View File
@@ -360,12 +360,26 @@ impl OpaMock {
pub(crate) async fn expect_bucket_tags(&mut self, action: &str, bucket: &str, department: Option<&str>) -> TestResult {
let payload = self.next_request().await?;
Self::assert_bucket_tags(&payload, action, bucket, department);
Ok(())
}
async fn expect_versioned_tag_permission(&mut self, bucket: &str, object: &str, version: &str) -> TestResult {
let payload = self.next_request().await?;
Self::assert_bucket_tags(&payload, "s3:GetObjectVersionTagging", bucket, Some("finance"));
assert_eq!(payload["input"]["resource"]["object"], object);
assert_eq!(payload["input"]["context"]["conditions"]["versionid"], serde_json::json!([version]));
Ok(())
}
fn assert_bucket_tags(payload: &Value, action: &str, bucket: &str, department: Option<&str>) {
let input = &payload["input"];
assert_eq!(input["identity"]["account"], "opabuckettags");
assert_eq!(input["identity"]["is_owner"], false);
assert_eq!(input["action"], action);
assert_eq!(input["resource"]["bucket"], bucket);
let expected = department.map(|value| serde_json::json!([value]));
assert_eq!(input["context"]["conditions"].get("ExistingBucketTag/department"), expected.as_ref());
Ok(())
}
async fn wait_for_validation(&mut self) -> TestResult {
@@ -678,22 +692,61 @@ async fn test_list_buckets_opa_contract() -> TestResult {
Ok(())
}
/// Stop and reap the directory's owning server before opening the fixture store.
async fn seed_legacy_duplicate_bucket_tags(env: &mut RustFSTestEnvironment, bucket: &str) -> TestResult {
use crate::storage_api::bucket_tag_fixture::{
DisksLayout, ECStore, EndpointServerPools, InstanceContext, init_local_disks_with_instance_ctx, load_bucket_metadata,
};
let server = env
.process
.as_mut()
.ok_or("missing server process before offline fixture write")?;
server.kill()?;
server.wait()?;
env.process = None;
let layout = DisksLayout::from_volumes(std::slice::from_ref(&env.temp_dir))?;
let (pools, _) = EndpointServerPools::create_server_endpoints(&env.address, &layout).await?;
let context = std::sync::Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&context, pools.clone()).await?;
let store = ECStore::new_with_instance_ctx(env.address.parse()?, pools, Default::default(), context.clone()).await?;
let _cancel = context
.background_cancel_token()
.expect("store binds its shutdown token")
.drop_guard();
let mut metadata = load_bucket_metadata(store.clone(), bucket).await?;
let incarnation = metadata.bucket_incarnation_id;
let policy = metadata.policy_config_json.clone();
assert!(!incarnation.is_nil());
assert!(!policy.is_empty(), "the bucket-policy Allow must survive legacy seeding");
// Encode the duplicate tag set with the same bucket record and erasure writer used by the store.
let xml = b"<Tagging><TagSet><Tag><Key>department</Key><Value>finance</Value></Tag><Tag><Key>department</Key><Value>engineering</Value></Tag></TagSet></Tagging>";
metadata.tagging_config_xml = xml.to_vec();
metadata.save_with_store(store.clone()).await?;
let persisted = load_bucket_metadata(store, bucket).await?;
assert_eq!(persisted.bucket_incarnation_id, incarnation);
assert_eq!(persisted.policy_config_json, policy);
assert_eq!(persisted.tagging_config_xml.as_slice(), xml);
Ok(())
}
#[tokio::test]
async fn test_bucket_tags_opa_contract() -> TestResult {
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Tag, Tagging, VersioningConfiguration,
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Tag, Tagging, TaggingDirective, VersioningConfiguration,
};
init_logging();
let mut opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_without_cleanup_with_env(&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
let opa_url = opa.url.clone();
let server_env = [
("RUSTFS_POLICY_PLUGIN_URL", opa_url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
("NO_PROXY", "127.0.0.1,localhost"),
])
.await?;
];
env.start_rustfs_server_without_cleanup_with_env(&server_env).await?;
let secret = uuid::Uuid::new_v4().to_string();
create_user(&env, "opabuckettags", &secret).await?;
let client = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, "opabuckettags", &secret, None, "e2e-bucket-tags"));
@@ -733,7 +786,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.await?;
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
client
let head = client
.head_object()
.bucket(source)
.key("report")
@@ -741,6 +794,8 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
assert_eq!(head.tag_count(), Some(1));
opa.expect_versioned_tag_permission(source, "report", version).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
@@ -750,28 +805,65 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
admin.create_bucket().bucket(destination).send().await?;
set_department(&admin, destination, "engineering").await?;
let copy_source = format!("{source}/report?versionId={version}");
let error = client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.send()
.await
.expect_err("source tags must not authorize a different destination");
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:PutObject", destination, Some("engineering"))
.await?;
for directive in [TaggingDirective::Copy, TaggingDirective::Replace] {
let error = client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.set_tagging_directive((directive == TaggingDirective::Replace).then_some(TaggingDirective::Replace))
.set_tagging((directive == TaggingDirective::Replace).then(|| "department=legal".to_owned()))
.send()
.await
.expect_err("source tags must not authorize a different destination");
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
if directive == TaggingDirective::Copy {
opa.expect_versioned_tag_permission(source, "report", version).await?;
}
opa.expect_bucket_tags("s3:PutObject", destination, Some("engineering"))
.await?;
let missing = admin
.head_object()
.bucket(destination)
.key("copied")
.send()
.await
.expect_err("denied COPY and REPLACE must leave no destination object");
assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
}
set_department(&admin, destination, "finance").await?;
client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
for directive in [TaggingDirective::Copy, TaggingDirective::Replace] {
client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.set_tagging_directive((directive == TaggingDirective::Replace).then_some(TaggingDirective::Replace))
.set_tagging((directive == TaggingDirective::Replace).then(|| "department=legal".to_owned()))
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
// The access hook checks preliminary tags, then the copy use case checks
// the exact source reader's tags before publishing the destination.
for _ in 0..2 {
if directive == TaggingDirective::Copy {
opa.expect_versioned_tag_permission(source, "report", version).await?;
}
opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
}
let tags = admin.get_object_tagging().bucket(destination).key("copied").send().await?;
assert_eq!(tags.tag_set().len(), 1);
assert_eq!(tags.tag_set()[0].key(), "department");
assert_eq!(
tags.tag_set()[0].value(),
if directive == TaggingDirective::Copy {
"engineering"
} else {
"legal"
}
);
}
let upload = client
.create_multipart_upload()
@@ -847,6 +939,14 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("engineering"))
.await?;
let missing = admin
.head_object()
.bucket(destination)
.key("denied-source")
.send()
.await
.expect_err("a denied source must leave no destination object");
assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
set_department(&client, source, "finance")
.await
.expect_err("requested tags must not authorize their own mutation");
@@ -885,7 +985,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObject", destination, Some("engineering"))
.await?;
admin
let invalid_write = admin
.put_bucket_tagging()
.bucket(destination)
.tagging(
@@ -895,7 +995,25 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.build()?,
)
.send()
.await
.expect_err("new duplicate tags must be rejected before publication");
assert_eq!(invalid_write.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidTag"));
assert_eq!(invalid_write.raw_response().map(|response| response.status().as_u16()), Some(400));
set_department(&admin, source, "finance").await?;
seed_legacy_duplicate_bucket_tags(&mut env, destination).await?;
env.start_rustfs_server_without_cleanup_with_env(&server_env).await?;
// Prove this process can authorize the same caller and read a healthy bucket's persisted tags.
let read = client
.get_object()
.bucket(source)
.key("report")
.version_id(version)
.send()
.await?;
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
let error = client
.get_object()
.bucket(destination)
+16 -5
View File
@@ -192,6 +192,7 @@ pub mod bucket {
table_catalog_path_hash, unreadable_config_refusal,
};
pub use crate::bucket::metadata::{BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, validate_tag_set_entries};
}
pub mod config_parse_mode {
@@ -331,8 +332,8 @@ pub mod bucket {
pub mod utils {
pub use crate::bucket::utils::{
check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
serialize,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, has_xml_invalid_control_characters,
is_meta_bucketname, is_valid_object_prefix, serialize,
};
}
@@ -451,9 +452,15 @@ pub mod disk {
pub use crate::cluster::rpc::background_reader_stream::BackgroundReaderStream;
pub use crate::disk::background_memory::BackgroundEncodeBuffer;
pub use crate::disk::disk_store::get_object_disk_read_timeout;
#[cfg(feature = "test-util")]
pub use crate::disk::disk_store::wait_for_multipart_index_bootstrap;
pub use crate::disk::local::{ReplacementExecutionLease, ScanGuard};
#[cfg(feature = "test-util")]
pub use crate::disk::local::{local_durability_modes, set_local_rename_data_fail_after_metadata_commit};
pub use crate::disk::multipart_index::{
MULTIPART_INDEX_RPC_MAX_BYTES, MULTIPART_INDEX_RPC_MAX_REQUEST_BYTES, MultipartIndexPage, MultipartIndexRpcReply,
MultipartIndexRpcRequest,
};
pub use crate::disk::multipart_source::{
MULTIPART_SOURCE_PROTOCOL_VERSION, MultipartAbortCertificate, MultipartAbortProof, MultipartAbortProofKind,
MultipartAbortRound, MultipartSourceOperation, MultipartSourcePhase, MultipartSourceRequest, MultipartSourceResponse,
@@ -466,7 +473,9 @@ pub mod disk {
pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, CurrentDeleteTransition,
BoundedDeleteRecoveryDiscovery, BoundedDeleteRecoveryInspection, BoundedDeleteRecoverySettlement,
BoundedDeleteRecoverySettlementReply, CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration,
CurrentDeleteTransition, DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES, DELETE_RECOVERY_MAX_INSPECTION_ENTRIES,
DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DeleteRecoveryState,
Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, FileReader, FileWriter,
@@ -617,7 +626,7 @@ pub mod rebalance {
pub mod rio {
pub use crate::io_support::rio::{
DecryptReader, DynReader, EncryptReader, HardLimitReader, HashReader, ReadStream, Reader, WriteEncryption, WritePlan,
boxed_reader, compression_metadata_value, wrap_reader,
boxed_reader, compression_metadata_value, validate_compression_scheme, wrap_reader,
};
}
@@ -654,7 +663,9 @@ pub mod set_disk {
#[cfg(feature = "test-util")]
pub mod test_util {
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
pub use crate::bucket::quota::reservation::{
fail_next_quota_ledger_save_for_test, initialize_quota_ledger_for_test, retry_quota_reconciliation_for_test,
};
pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
+325 -5
View File
@@ -16,7 +16,8 @@ use crate::bucket::metadata::BucketMetadata;
use crate::bucket::metadata_sys::get_bucket_targets_config;
use crate::bucket::metadata_sys::get_replication_config;
use crate::bucket::remote_s3_client::{
PathStyle, REPLICATION_TARGET_RETRY_POLICY, RemoteCredentials, RemoteS3EndpointSpec, build_remote_s3_client,
PathStyle, REPLICATION_TARGET_RETRY_POLICY, RemoteCredentials, RemoteS3EndpointSpec, RemoteS3RetryPolicy,
UploadResponseDeadline, build_remote_s3_client,
};
use crate::bucket::replication::{ObjectLockIntegrity, object_lock_put_integrity, replication_etags_match};
use crate::bucket::replication::{ReplicationStatusType, ReplicationTargetConfigBridge};
@@ -119,6 +120,7 @@ impl From<&BucketTarget> for RemoteS3EndpointSpec {
ca_cert_pem: (!target.ca_cert_pem.trim().is_empty()).then(|| target.ca_cert_pem.clone()),
connect_timeout: None,
read_timeout: None,
operation_timeout: Some(TargetRequestTimeouts::from_env().request),
// Replication has no retry budget of its own on the request path,
// so it keeps the SDK's standard three attempts; stating it here
// pins the behaviour to this line instead of an SDK default.
@@ -1191,6 +1193,7 @@ impl BucketTargetSys {
health_check_duration: target.health_check_duration,
replicate_sync: target.replication_sync,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::from_env(),
})
}
@@ -1922,6 +1925,83 @@ impl std::fmt::Display for S3ClientError {
}
}
/// Replication request budgets, snapshotted when a target client is built.
/// Streamed GET bodies are not subject to the upload-response budget.
#[derive(Debug, Clone, Copy)]
pub struct TargetRequestTimeouts {
request: Duration,
complete: Duration,
minimum_transfer_rate: u64,
}
impl Default for TargetRequestTimeouts {
fn default() -> Self {
Self {
request: Duration::from_secs(60),
complete: Duration::from_secs(15 * 60),
minimum_transfer_rate: 1024 * 1024,
}
}
}
impl TargetRequestTimeouts {
fn from_env() -> Self {
let defaults = Self::default();
let seconds = |name: &str, fallback: Duration| {
let value = rustfs_utils::get_env_u64(name, fallback.as_secs());
Duration::from_secs(if (1..=86400).contains(&value) {
value
} else {
fallback.as_secs()
})
};
let minimum_transfer_rate =
rustfs_utils::get_env_u64("RUSTFS_REPLICATION_MIN_TRANSFER_RATE_BYTES_PER_SEC", defaults.minimum_transfer_rate);
Self {
request: seconds("RUSTFS_REPLICATION_REQUEST_TIMEOUT_SECS", defaults.request),
complete: seconds("RUSTFS_REPLICATION_COMPLETE_TIMEOUT_SECS", defaults.complete),
minimum_transfer_rate: if minimum_transfer_rate == 0 {
defaults.minimum_transfer_rate
} else {
minimum_transfer_rate
},
}
}
fn upload_timeout(self, size: i64) -> Result<Duration, S3ClientError> {
let size = u64::try_from(size).map_err(|_| S3ClientError::new("replication upload size must be non-negative"))?;
// The SDK can make three attempts. Leave transmission headroom for
// all three; the separate response deadline still bounds a trickling
// target after each upload finishes.
let attempts = match REPLICATION_TARGET_RETRY_POLICY {
RemoteS3RetryPolicy::Disabled => 1,
RemoteS3RetryPolicy::Standard { max_attempts } => u64::from(max_attempts.max(1)),
};
let seconds = size
.div_ceil(self.minimum_transfer_rate)
.checked_mul(attempts)
.ok_or_else(|| S3ClientError::new("replication upload timeout exceeds supported duration"))?;
let timeout = self
.request
.checked_add(Duration::from_secs(seconds))
.ok_or_else(|| S3ClientError::new("replication upload timeout exceeds supported duration"))?;
if Instant::now().checked_add(timeout).is_none() {
return Err(S3ClientError::new("replication upload timeout exceeds supported duration"));
}
Ok(timeout)
}
fn upload_override(self, size: i64) -> Result<aws_sdk_s3::config::Builder, S3ClientError> {
Ok(aws_sdk_s3::config::Builder::new()
.timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(self.upload_timeout(size)?)
.build(),
)
.interceptor(UploadResponseDeadline(self.request)))
}
}
#[derive(Debug)]
pub struct TargetClient {
pub endpoint: String,
@@ -1935,6 +2015,7 @@ pub struct TargetClient {
pub health_check_duration: Duration,
pub replicate_sync: bool,
pub client: Arc<S3Client>,
pub timeouts: TargetRequestTimeouts,
}
impl TargetClient {
@@ -2360,12 +2441,11 @@ impl TargetClient {
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
});
let mut config = self.timeouts.upload_override(size)?;
if forwards_source_checksum {
operation = operation.config_override(
aws_sdk_s3::config::Builder::new()
.request_checksum_calculation(aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired),
);
config = config.request_checksum_calculation(aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired);
}
operation = operation.config_override(config);
match operation.send().await {
Ok(output) => {
// Under SSE-KMS/DSSE or SSE-C the target's ETag is not the MD5
@@ -2501,6 +2581,7 @@ impl TargetClient {
}
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
})
.config_override(self.timeouts.upload_override(size)?)
.send()
.await
{
@@ -2538,6 +2619,13 @@ impl TargetClient {
}
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
})
.config_override(
aws_sdk_s3::config::Builder::new().timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(self.timeouts.complete)
.build(),
),
)
.send()
.await
{
@@ -2966,11 +3054,241 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
},
request_headers,
)
}
#[derive(Debug, Clone)]
struct TimedResponseConnector {
upload_delay: Duration,
response_delay: Duration,
prefix_bytes: usize,
response: bytes::Bytes,
frames: Arc<std::sync::atomic::AtomicUsize>,
dropped: Arc<std::sync::atomic::AtomicUsize>,
}
struct ResponseDrop(Arc<std::sync::atomic::AtomicUsize>);
impl Drop for ResponseDrop {
fn drop(&mut self) {
self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
}
}
impl SmithyHttpConnector for TimedResponseConnector {
fn call(&self, request: HttpRequest) -> HttpConnectorFuture {
let settings = self.clone();
HttpConnectorFuture::new(async move {
http_body_util::BodyExt::collect(request.into_body())
.await
.map_err(aws_smithy_runtime_api::client::result::ConnectorError::io)?;
tokio::time::sleep(settings.upload_delay).await;
let guard = ResponseDrop(Arc::clone(&settings.dropped));
let stream = futures::stream::unfold(
(settings.prefix_bytes, Some(settings.response), true, guard),
move |(remaining, mut body, first, guard)| {
let frames = Arc::clone(&settings.frames);
async move {
if remaining == 0 && body.is_none() {
return None;
}
if !first {
tokio::time::sleep(settings.response_delay).await;
}
let bytes = if remaining > 0 {
bytes::Bytes::from_static(b" ")
} else {
body.take()?
};
frames.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
Some((
Ok::<_, std::io::Error>(http_body::Frame::data(bytes)),
(remaining.saturating_sub(1), body, false, guard),
))
}
},
)
.fuse();
let body = SdkBody::from_body_1_x(http_body_util::StreamBody::new(stream));
let mut response =
HttpResponse::new(aws_smithy_runtime_api::http::StatusCode::try_from(200u16).expect("status"), body);
response.headers_mut().insert("etag", "\"test-etag\"");
Ok(response)
})
}
}
fn timed_response_target(
upload_delay: Duration,
response_delay: Duration,
prefix_bytes: usize,
response: &'static [u8],
) -> (TargetClient, Arc<std::sync::atomic::AtomicUsize>, Arc<std::sync::atomic::AtomicUsize>) {
let frames = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let dropped = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let connector = SharedHttpConnector::new(TimedResponseConnector {
upload_delay,
response_delay,
prefix_bytes,
response: bytes::Bytes::from_static(response),
frames: Arc::clone(&frames),
dropped: Arc::clone(&dropped),
});
let http_client = http_client_fn(move |_settings, _components| connector.clone());
let (mut target, _) = header_recording_target_client(Vec::new());
target.timeouts = TargetRequestTimeouts {
request: Duration::from_secs(2),
complete: Duration::from_secs(10),
minimum_transfer_rate: 1024 * 1024,
};
target.client = Arc::new(S3Client::from_conf(
target
.client
.config()
.to_builder()
.http_client(http_client)
.retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled())
.timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(target.timeouts.request)
.build(),
)
.build(),
));
(target, frames, dropped)
}
#[test]
fn replication_upload_budgets_preserve_large_transfers_and_reject_invalid_sizes() {
let policy = TargetRequestTimeouts::default();
assert_eq!(policy.upload_timeout(0).expect("empty upload"), Duration::from_secs(60));
assert_eq!(
policy.upload_timeout(5 * 1024 * 1024 * 1024).expect("maximum single PUT"),
Duration::from_secs(15420)
);
assert!(policy.upload_timeout(-1).is_err());
let slow = TargetRequestTimeouts {
minimum_transfer_rate: 1,
..policy
};
assert!(slow.upload_timeout(i64::MAX).is_err());
}
#[tokio::test(start_paused = true)]
async fn replication_upload_response_deadline_is_independent_of_size_and_checksum_override() {
for operation in ["put", "forwarded-checksum", "part"] {
let (target, frames, dropped) = timed_response_target(Duration::from_secs(4), Duration::from_millis(500), 100, b"");
let body = vec![b'x'; 2 * 1024 * 1024];
let size = i64::try_from(body.len()).expect("size");
let started = tokio::time::Instant::now();
let error = if operation == "part" {
target
.put_object_part(
"bucket",
"key",
"upload",
1,
size,
ByteStream::from(body),
&PutObjectPartOptions::default(),
)
.await
.expect_err("response deadline")
} else {
let mut opts = PutObjectOptions::default();
if operation == "forwarded-checksum" {
opts.user_metadata.insert(
"x-amz-checksum-sha256".to_owned(),
"OoJ3yNhRwv3wwtZoGqEIPrPX9xwTnfLl+ka0wStN1g0=".to_owned(),
);
}
target
.put_object("bucket", "key", size, ByteStream::from(body), &opts)
.await
.expect_err("response deadline")
};
assert!(
started.elapsed() >= Duration::from_secs(6),
"upload gets its full independent window: {error}"
);
assert!(
started.elapsed() < target.timeouts.upload_timeout(size).expect("budget"),
"response must end before the larger total deadline"
);
assert!(
frames.load(std::sync::atomic::Ordering::SeqCst) >= 2,
"trickle delivered progress before failure"
);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1, "transport body released");
}
}
#[tokio::test(start_paused = true)]
async fn replication_slow_upload_preserves_transfer_window_and_streaming_get_body() {
let (target, _, _) = timed_response_target(Duration::from_secs(4), Duration::from_millis(100), 5, b"");
let body = vec![b'x'; 2 * 1024 * 1024];
let size = i64::try_from(body.len()).expect("size");
target
.put_object("bucket", "key", size, ByteStream::from(body), &PutObjectOptions::default())
.await
.expect("slow upload, prompt response");
let (target, _, dropped) = timed_response_target(Duration::ZERO, Duration::from_secs(1), 5, b"payload");
let response = target
.get_object("bucket", "key", None, None, None, HeaderMap::new())
.await
.expect("GET headers");
let started = tokio::time::Instant::now();
let body = response
.body
.collect()
.await
.expect("streamed GET remains outside upload deadline")
.into_bytes();
assert!(started.elapsed() > target.timeouts.request);
assert_eq!(&body[..], b" payload");
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
#[tokio::test(start_paused = true)]
async fn replication_complete_processing_whitespace_has_a_separate_bounded_window() {
const RESULT: &[u8] = b"<CompleteMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><ETag>\"etag\"</ETag></CompleteMultipartUploadResult>";
for prefix in [5, 100] {
let (target, frames, dropped) = timed_response_target(Duration::ZERO, Duration::from_secs(1), prefix, RESULT);
let started = tokio::time::Instant::now();
let result = target
.complete_multipart_upload(
"bucket",
"key",
"upload",
vec![CompletedPart::builder().part_number(1).e_tag("etag").build()],
&PutObjectOptions::default(),
)
.await;
if prefix == 5 {
assert!(result.is_ok(), "legal processing whitespace: {result:?}");
} else {
assert!(result.is_err());
assert!(started.elapsed() <= Duration::from_secs(11));
}
assert!(started.elapsed() > target.timeouts.request);
assert!(frames.load(std::sync::atomic::Ordering::SeqCst) >= 2);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
}
#[tokio::test(start_paused = true)]
async fn replication_control_operation_deadline_covers_response_trickle() {
let (target, frames, dropped) =
timed_response_target(Duration::ZERO, Duration::from_millis(500), 100, b"<VersioningConfiguration/>");
let started = tokio::time::Instant::now();
assert!(target.get_bucket_versioning("bucket").await.is_err());
assert!(started.elapsed() <= Duration::from_secs(3));
assert!(frames.load(std::sync::atomic::Ordering::SeqCst) >= 2);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
fn streaming_test_body(payload: &'static [u8]) -> ByteStream {
let stream = tokio_util::io::ReaderStream::new(std::io::Cursor::new(payload));
let body = http_body_util::StreamBody::new(futures::StreamExt::map(stream, |r| r.map(http_body::Frame::data)));
@@ -3279,6 +3597,7 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
},
request_uris,
)
@@ -3591,6 +3910,7 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
})
}
@@ -5487,6 +5487,14 @@ pub async fn put_restore_opts(
if !oi.user_tags.is_empty() {
meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
}
if let Some(expires) = oi.expires {
meta.insert(
"expires".to_string(),
expires
.format(&time::format_description::well_known::Rfc3339)
.map_err(std::io::Error::other)?,
);
}
let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
meta.insert(
metadata_keys::RESTORE.to_string(),
@@ -5506,7 +5514,6 @@ pub async fn put_restore_opts(
// computed MD5 is not the object's public plaintext ETag.
preserve_etag: oi.etag.clone(),
shard_integrity_write_mode: Some(oi.shard_integrity_write_mode()),
//expires: oi.expires,
..Default::default()
})
}
@@ -7144,6 +7151,25 @@ mod tests {
0,
"tier generation lease should release after EOF"
);
let unread = get_transitioned_object_reader_with_tier_manager(
&object_info.bucket,
&object_info.name,
&None,
&HeaderMap::new(),
&object_info,
&ObjectOptions::default(),
&manager,
None,
)
.await
.expect("open an unread transitioned response");
assert_eq!(TierConfigMgr::active_operation_lease_count(&manager, &tier).await, 1);
drop(unread);
assert_eq!(
TierConfigMgr::active_operation_lease_count(&manager, &tier).await,
0,
"canceling an unread response must release the tier lease"
);
}
#[cfg(feature = "test-util")]
+71
View File
@@ -256,6 +256,38 @@ pub const BUCKET_POLICY_CONFIG: &str = "policy.json";
pub const BUCKET_NOTIFICATION_CONFIG: &str = "notification.xml";
pub const BUCKET_LIFECYCLE_CONFIG: &str = "lifecycle.xml";
pub const BUCKET_SSECONFIG: &str = "bucket-encryption.xml";
/// Shared tag-entry validation for object requests and persisted bucket tags.
pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
let mut keys = std::collections::HashSet::with_capacity(tag_set.len());
for tag in tag_set {
let key = tag
.key
.as_deref()
.filter(|key| !key.is_empty())
.ok_or("Tag key cannot be empty")?;
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
let value = tag.value.as_deref().ok_or("Tag value cannot be null")?;
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
if !keys.insert(key) {
return Err("Cannot provide multiple Tags with the same key");
}
}
Ok(())
}
pub const MAX_BUCKET_TAGS: usize = 50;
pub const BUCKET_TAGGING_CONFIG: &str = "tagging.xml";
pub const BUCKET_QUOTA_CONFIG_FILE: &str = "quota.json";
pub const OBJECT_LOCK_CONFIG: &str = "object-lock.xml";
@@ -990,6 +1022,20 @@ impl BucketMetadata {
self.encryption_config_updated_at = updated;
}
BUCKET_TAGGING_CONFIG => {
if !data.is_empty() {
let tagging = deserialize::<Tagging>(&data).map_err(|_| {
Error::InvalidArgument(self.name.clone(), String::new(), "Invalid bucket tagging XML".to_owned())
})?;
if tagging.tag_set.len() > MAX_BUCKET_TAGS {
return Err(Error::InvalidArgument(
self.name.clone(),
String::new(),
"Cannot have more than 50 tags per bucket".to_owned(),
));
}
validate_tag_set_entries(&tagging.tag_set)
.map_err(|message| Error::InvalidArgument(self.name.clone(), String::new(), message.to_owned()))?;
}
self.tagging_config_xml = data;
// Drop the parsed form (like lifecycle above) so clearing the
// payload can't leave stale parsed tags to be cached.
@@ -1784,6 +1830,31 @@ mod test {
/// backlog#2292: a replicated config is stamped with the source
/// `updated_at` it was given, not the local clock, while the plain
/// `update_config` entry keeps stamping the local clock.
#[test]
fn bucket_tag_updates_reject_unreadable_shapes_without_replacing_metadata() {
let mut metadata = BucketMetadata::new("validated-tags");
let valid = br#"<Tagging><TagSet><Tag><Key>team</Key><Value></Value></Tag></TagSet></Tagging>"#.to_vec();
metadata
.update_config(BUCKET_TAGGING_CONFIG, valid.clone())
.expect("empty value is valid");
let updated = metadata.tagging_config_updated_at;
for invalid in [
"<Tagging><TagSet><Tag><Key>team</Key></Tag></TagSet></Tagging>".to_owned(),
"<Tagging><TagSet><Tag><Key></Key><Value>x</Value></Tag></TagSet></Tagging>".to_owned(),
"<Tagging><TagSet><Tag><Key>team</Key><Value>x</Value></Tag><Tag><Key>team</Key><Value>y</Value></Tag></TagSet></Tagging>".to_owned(),
format!("<Tagging><TagSet>{}</TagSet></Tagging>", (0..51).map(|i| format!("<Tag><Key>k{i}</Key><Value>v</Value></Tag>")).collect::<String>()),
] {
assert!(metadata.update_config_at(BUCKET_TAGGING_CONFIG, invalid.into_bytes(), updated + time::Duration::hours(1)).is_err());
assert_eq!(metadata.tagging_config_xml, valid);
assert_eq!(metadata.tagging_config_updated_at, updated);
}
metadata.tagging_config_xml = b"legacy corrupt tags".to_vec();
metadata
.update_config(BUCKET_TAGGING_CONFIG, Vec::new())
.expect("deletion repairs legacy corruption");
assert!(metadata.tagging_config_xml.is_empty());
}
#[test]
fn update_config_at_stamps_the_given_time_and_update_config_stamps_now() {
let source_time = OffsetDateTime::now_utc() - time::Duration::hours(3);
+11 -5
View File
@@ -3428,7 +3428,7 @@ mod tests {
),
(
BUCKET_TAGGING_CONFIG,
r#"<Tagging><TagSet><Tag><Key>environment</Key><Value>测试-🦀</Value></Tag></TagSet></Tagging>"#.as_bytes(),
r#"<Tagging><TagSet><Tag><Key>environment</Key><Value>测试-rust</Value></Tag></TagSet></Tagging>"#.as_bytes(),
),
(BUCKET_REPLICATION_CONFIG, NEW_WRITER_REPLICATION_XML),
(
@@ -4556,8 +4556,13 @@ mod tests {
std::fs::create_dir_all(dir.path().join(bucket)).expect("bucket volume should be created");
}
let mut bm = BucketMetadata::new(bucket);
bm.update_config(config_file, raw.to_vec())
.expect("raw config should be stored");
if config_file == crate::bucket::metadata::BUCKET_TAGGING_CONFIG {
// Model legacy persisted corruption without passing the write-time validator.
bm.tagging_config_xml = raw.to_vec();
} else {
bm.update_config(config_file, raw.to_vec())
.expect("raw config should be stored");
}
sys.persist_new_and_set(bm).await.expect("initial metadata should persist");
}
@@ -5738,7 +5743,8 @@ mod tests {
// update only needs one interleaving to show up.
const ROUNDS: usize = 8;
for round in 0..ROUNDS {
let tagging = format!("<Tagging><Round>{round}</Round></Tagging>").into_bytes();
let tagging =
format!("<Tagging><TagSet><Tag><Key>round</Key><Value>{round}</Value></Tag></TagSet></Tagging>").into_bytes();
let policy = format!(r#"{{"Version":"2012-10-17","Round":{round}}}"#).into_bytes();
let start = Arc::new(tokio::sync::Barrier::new(2));
@@ -5811,7 +5817,7 @@ mod tests {
let blocked = tokio::spawn({
let node_b = node_b.clone();
async move { update_with_sys(node_b, bucket, BUCKET_TAGGING_CONFIG, b"<Tagging/>".to_vec()).await }
async move { update_with_sys(node_b, bucket, BUCKET_TAGGING_CONFIG, b"<Tagging><TagSet/></Tagging>".to_vec()).await }
});
// Long enough for the write to have finished had it not waited: the
@@ -119,6 +119,21 @@ async fn durable_quota_reporting_retires_degraded_floors_without_changing_ledger
// Exercise the real storage entry point before testing the new
// reporting semantics: the same ledger must admit and refuse
// these writes even when a scanner floor claims a million bytes.
if protocol == 1 {
let mut bootstrap = PutObjReader::from_vec(vec![0x5a; 512]);
assert!(
matches!(
store
.put_object(&bucket, "bootstrap", &mut bootstrap, &ObjectOptions::default())
.await,
Err(StorageError::SlowDown)
),
"a missing authoritative ledger must not admit a write from scanner reporting"
);
crate::bucket::quota::reservation::retry_quota_reconciliation_for_test(store.clone(), &bucket)
.await
.expect("bootstrap authoritative usage through the production worker");
}
let mut too_large = PutObjReader::from_vec(vec![0x5a; 8192]);
assert!(
matches!(
+739 -49
View File
@@ -19,7 +19,7 @@ use crate::data_usage::compute_bucket_usage;
use crate::disk::RUSTFS_META_BUCKET;
use crate::disk::{DiskAPI, error::DiskError};
use crate::error::{Result, StorageError, is_err_object_not_found, is_err_version_not_found};
use crate::object_api::{ObjectInfo, ObjectOptions, QuotaAdmission};
use crate::object_api::{ObjectInfo, ObjectLockConfigSnapshot, ObjectOptions, QuotaAdmission};
use crate::set_disk::{SetDisks, get_lock_acquire_timeout};
use crate::storage_api_contracts::namespace::NamespaceLocking;
use crate::storage_api_contracts::{list::ListOperations as _, object::ObjectOperations};
@@ -28,7 +28,7 @@ use futures::{StreamExt, stream};
use rustfs_lock::NamespaceLockGuard;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
use std::collections::{BTreeMap, HashSet};
use std::sync::Arc;
use std::time::Duration;
use time::OffsetDateTime;
@@ -36,6 +36,8 @@ use tracing::warn;
use uuid::Uuid;
const QUOTA_LEDGER_FORMAT_VERSION: u8 = 1;
const RECONCILE_RETRY_SECONDS: i64 = 30;
const EVENT_QUOTA_RECONCILIATION: &str = "quota_reconciliation";
const MAX_ORPHANS_REAPED_PER_WRITE: usize = 64;
const MAX_ORPHAN_PROBES_PER_WRITE: usize = 128;
const ORPHAN_PROBE_CONCURRENCY: usize = 32;
@@ -96,6 +98,10 @@ struct QuotaLedger {
version: u8,
bucket_incarnation: Uuid,
quota_revision_unix_nanos: i128,
/// Changes on every durable write, including zero-growth reservations.
/// Missing on an older writer's output, which invalidates a concurrent scan.
#[serde(default)]
writer_revision: Uuid,
accounted_usage: u64,
reservations: BTreeMap<Uuid, PersistedReservation>,
#[serde(default)]
@@ -110,6 +116,7 @@ impl QuotaLedger {
version: QUOTA_LEDGER_FORMAT_VERSION,
bucket_incarnation,
quota_revision_unix_nanos: quota_revision.unix_timestamp_nanos(),
writer_revision: Uuid::nil(),
accounted_usage,
reservations: BTreeMap::new(),
reconcile_required: false,
@@ -485,22 +492,23 @@ impl QuotaContext {
let ledger_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &ledger_data.ledger_object).await?;
let ledger_guard = Arc::new(ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?);
fence_namespace_mutations(&store, RUSTFS_META_BUCKET, &ledger_data.ledger_object, None).await?;
let mut ledger = load_current_ledger_locked(
Arc::clone(&store),
&ledger_data.bucket,
&ledger_data.ledger_object,
bucket_incarnation,
quota_revision,
)
.await?;
let mut ledger =
load_current_ledger_locked(Arc::clone(&store), &ledger_data.ledger_object, bucket_incarnation, quota_revision)
.await?;
let job = ReconcileJob::new(Arc::clone(&store), ledger_data.bucket.clone(), bucket_incarnation, quota_revision);
let Some(mut ledger) = ledger.take() else {
// Bootstrap has no authoritative usage yet. Release every
// request-owned fence by returning; the worker owns no request.
job.schedule();
return Err(StorageError::SlowDown);
};
let growth = reservation.growth();
let mut current_usage = ledger.admitted_usage()?;
let mut expected_usage = current_usage.checked_add(growth).ok_or(StorageError::PartMissingOrCorrupt)?;
if growth > 0 && expected_usage > quota_limit && growth <= quota_limit && ledger.should_reconcile_after_denial() {
reconcile_exact(&store, &ledger_data.bucket, &mut ledger).await?;
current_usage = ledger.admitted_usage()?;
expected_usage = current_usage.checked_add(growth).ok_or(StorageError::PartMissingOrCorrupt)?;
let current_usage = ledger.admitted_usage()?;
let expected_usage = current_usage.checked_add(growth).ok_or(StorageError::PartMissingOrCorrupt)?;
if ledger.reconcile_required
|| (growth > 0 && expected_usage > quota_limit && growth <= quota_limit && ledger.should_reconcile_after_denial())
{
job.schedule();
}
if growth > 0 && expected_usage > quota_limit {
return Err(StorageError::QuotaExceeded {
@@ -518,7 +526,7 @@ impl QuotaContext {
});
}
ledger.reserve(operation_id, reservation)?;
save_ledger_locked(Arc::clone(&store), &ledger_data.ledger_object, &ledger, &ledger_guard).await?;
save_ledger_locked(Arc::clone(&store), &ledger_data.ledger_object, &mut ledger, &ledger_guard).await?;
Ok(QuotaReservation {
ledger: Some(ReservationLedgerData::Single(ledger_data)),
@@ -1082,7 +1090,7 @@ async fn mark_commit_started(data: &ReservationLedgerData) -> Result<()> {
fence_namespace_mutations(&data.store, RUSTFS_META_BUCKET, &data.ledger_object, None).await?;
let mut ledger = load_ledger_locked(Arc::clone(&data.store), &data.ledger_object).await?;
ledger.mark_commit_started(data.operation_id, &data.reservation)?;
save_ledger_locked(Arc::clone(&data.store), &data.ledger_object, &ledger, &ledger_guard).await
save_ledger_locked(Arc::clone(&data.store), &data.ledger_object, &mut ledger, &ledger_guard).await
})
.await
.map_err(|err| StorageError::other_with_context("quota commit marker task failed", err))?
@@ -1113,7 +1121,7 @@ async fn settle(data: &ReservationLedgerData, committed: bool) -> Result<()> {
} else {
ledger.abort(operation_id, &reservation)?;
}
save_ledger_locked(Arc::clone(&store), &ledger_object, &ledger, &ledger_guard).await
save_ledger_locked(Arc::clone(&store), &ledger_object, &mut ledger, &ledger_guard).await
})
.await
.map_err(|err| StorageError::other_with_context("quota ledger settlement task failed", err))?
@@ -1121,22 +1129,15 @@ async fn settle(data: &ReservationLedgerData, committed: bool) -> Result<()> {
async fn load_current_ledger_locked(
store: Arc<ECStore>,
bucket: &str,
ledger_object: &str,
bucket_incarnation: Uuid,
quota_revision: OffsetDateTime,
) -> Result<QuotaLedger> {
match load_ledger_locked(Arc::clone(&store), ledger_object).await {
Ok(ledger) if ledger.matches(bucket_incarnation, quota_revision) => Ok(ledger),
Ok(ledger) if ledger.reservations.is_empty() && !ledger.reconcile_required => {
let usage = exact_bucket_usage(&store, bucket).await?;
Ok(QuotaLedger::new(bucket_incarnation, quota_revision, usage))
}
) -> Result<Option<QuotaLedger>> {
match load_ledger_locked(store, ledger_object).await {
Ok(ledger) if ledger.matches(bucket_incarnation, quota_revision) => Ok(Some(ledger)),
Ok(ledger) if ledger.reservations.is_empty() => Ok(None),
Ok(_) => Err(StorageError::PartMissingOrCorrupt),
Err(StorageError::ConfigNotFound) => {
let usage = exact_bucket_usage(&store, bucket).await?;
Ok(QuotaLedger::new(bucket_incarnation, quota_revision, usage))
}
Err(StorageError::ConfigNotFound) => Ok(None),
Err(err) => Err(err),
}
}
@@ -1245,15 +1246,11 @@ async fn reap_stale_reservations(store: Arc<ECStore>, bucket: &str, ledger_objec
for (object, target) in orphan_commit_targets {
fence_namespace_mutations(&store, bucket, &object, target).await?;
}
let mut removed = remove_orphan_reservations(&mut ledger, &orphan_ids)?;
if ledger.reservations.is_empty() && ledger.reconcile_required {
reconcile_exact(&store, bucket, &mut ledger).await?;
removed = true;
}
let removed = remove_orphan_reservations(&mut ledger, &orphan_ids)?;
if !removed && !cursor_changed {
return Ok(());
}
save_ledger_locked(store, ledger_object, &ledger, &ledger_guard).await
save_ledger_locked(store, ledger_object, &mut ledger, &ledger_guard).await
}
fn remove_orphan_reservations(ledger: &mut QuotaLedger, operation_ids: &[Uuid]) -> Result<bool> {
@@ -1273,13 +1270,217 @@ fn remove_orphan_reservations(ledger: &mut QuotaLedger, operation_ids: &[Uuid])
Ok(removed)
}
async fn reconcile_exact(store: &Arc<ECStore>, bucket: &str, ledger: &mut QuotaLedger) -> Result<()> {
if !ledger.reservations.is_empty() {
return Err(StorageError::PartMissingOrCorrupt);
#[derive(Debug, Serialize, Deserialize)]
struct ReconcileControl {
bucket_incarnation: Uuid,
quota_revision_unix_nanos: i128,
next_attempt_at: i64,
}
#[derive(Clone)]
struct ReconcileJob {
store: Arc<ECStore>,
bucket: String,
bucket_incarnation: Uuid,
quota_revision: OffsetDateTime,
}
type ReconcileKey = (Uuid, String, Uuid, i128);
fn active_reconciliations() -> &'static std::sync::Mutex<HashSet<ReconcileKey>> {
static ACTIVE: std::sync::OnceLock<std::sync::Mutex<HashSet<ReconcileKey>>> = std::sync::OnceLock::new();
ACTIVE.get_or_init(Default::default)
}
struct ActiveReconciliation(ReconcileKey);
impl Drop for ActiveReconciliation {
fn drop(&mut self) {
active_reconciliations()
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.remove(&self.0);
}
}
impl ReconcileJob {
fn new(store: Arc<ECStore>, bucket: String, bucket_incarnation: Uuid, quota_revision: OffsetDateTime) -> Self {
Self {
store,
bucket,
bucket_incarnation,
quota_revision,
}
}
fn schedule(self) {
let key = (
self.store.id,
self.bucket.clone(),
self.bucket_incarnation,
self.quota_revision.unix_timestamp_nanos(),
);
if !active_reconciliations()
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert(key.clone())
{
return;
}
let owner = ActiveReconciliation(key);
// The owner carries no object, metadata, lifecycle or reservation guard.
// Cancellation of the rejected request cannot cancel its reconciliation.
tokio::spawn(async move {
let _owner = owner;
if let Err(error) = self.run().await {
warn!(event = EVENT_QUOTA_RECONCILIATION, component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_QUOTA, state = "deferred", bucket = %self.bucket,
error = %error, "Quota reconciliation deferred");
}
});
}
fn control_object(&self) -> String {
format!(
"{}.reconcile/{}/{}",
ledger_object(&self.bucket),
self.bucket_incarnation,
self.quota_revision.unix_timestamp_nanos()
)
}
async fn save_control(&self, object: &str, control: &ReconcileControl, guard: &Arc<NamespaceLockGuard>) -> Result<()> {
if guard.is_lock_lost() {
return Err(StorageError::SlowDown);
}
let mut opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let _ = opts.set_quota_admission(0, u64::MAX);
opts.add_owned_write_lock(Arc::clone(guard), RUSTFS_META_BUCKET, object);
opts.write_completion = crate::object_api::WriteCompletion::TailDrained;
save_config_with_opts(Arc::clone(&self.store), object, serde_json::to_vec(control)?, &opts).await
}
async fn run(&self) -> Result<()> {
let control_object = self.control_object();
let lock = self.store.new_ns_lock(RUSTFS_META_BUCKET, &control_object).await?;
let Ok(guard) = lock.get_write_lock_quiet(Duration::from_millis(50)).await else {
return Ok(());
};
let guard = Arc::new(guard);
let now = now_unix();
match read_config_no_lock(Arc::clone(&self.store), &control_object).await {
Ok(bytes) => {
let control: ReconcileControl = serde_json::from_slice(&bytes)?;
if control.bucket_incarnation != self.bucket_incarnation
|| control.quota_revision_unix_nanos != self.quota_revision.unix_timestamp_nanos()
{
return Err(StorageError::PartMissingOrCorrupt);
}
if control.next_attempt_at > now {
return Ok(());
}
}
Err(StorageError::ConfigNotFound) => {}
Err(error) => return Err(error),
}
fence_namespace_mutations(&self.store, RUSTFS_META_BUCKET, &control_object, None).await?;
let mut control = ReconcileControl {
bucket_incarnation: self.bucket_incarnation,
quota_revision_unix_nanos: self.quota_revision.unix_timestamp_nanos(),
next_attempt_at: now.saturating_add(RECONCILE_RETRY_SECONDS),
};
// Persist the throttle before work, including bootstrap and failed scans.
self.save_control(&control_object, &control, &guard).await?;
let result = self.scan_and_publish(&guard).await;
control.next_attempt_at = now_unix().saturating_add(RECONCILE_RETRY_SECONDS);
self.save_control(&control_object, &control, &guard).await?;
result
}
/// Take a short authoritative configuration snapshot without recursively
/// acquiring a reader behind a queued metadata/lifecycle writer.
async fn configuration_snapshot(&self) -> Result<Arc<ObjectLockConfigSnapshot>> {
let snapshot = self.store.object_lock_config_snapshot_for_single_put(&self.bucket).await?;
let guard = snapshot
.metadata_transaction_guard_for(self.store.id, &self.bucket, Some(self.bucket_incarnation))
.ok_or(StorageError::PreconditionFailed)?;
let opts = ObjectOptions {
expected_bucket_incarnation_id: Some(self.bucket_incarnation),
object_lock_config_snapshot: Some(Arc::clone(&snapshot)),
..Default::default()
};
let (quota, incarnation, revision) =
metadata_sys::get_quota_config_and_incarnation_from_disk_for_options_in(&self.store.ctx, &self.bucket, &opts, &guard)
.await?;
if incarnation != self.bucket_incarnation
|| revision != self.quota_revision
|| !quota.as_ref().is_some_and(|quota| {
quota.uses_durable_reservations() && quota.reservation_protocol != Some(QUOTA_RESERVATION_PROTOCOL_V2)
})
{
return Err(StorageError::PreconditionFailed);
}
Ok(snapshot)
}
async fn scan_and_publish(&self, owner: &NamespaceLockGuard) -> Result<()> {
let ledger_object = ledger_object(&self.bucket);
let expected_revision = {
// Lock order: reconcile owner, lifecycle, metadata, ledger.
// Request paths never wait for the reconcile owner.
let _configuration = self.configuration_snapshot().await?;
let lock = self.store.new_ns_lock(RUSTFS_META_BUCKET, &ledger_object).await?;
let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await?);
match load_ledger_locked(Arc::clone(&self.store), &ledger_object).await {
Ok(mut ledger) => {
if !ledger.reservations.is_empty() {
return Ok(());
}
// Seed a non-nil token even for a legacy ledger. An old
// node drops this field on write and cannot pass the CAS.
fence_namespace_mutations(&self.store, RUSTFS_META_BUCKET, &ledger_object, None).await?;
save_ledger_locked(Arc::clone(&self.store), &ledger_object, &mut ledger, &guard).await?;
Some(ledger.writer_revision)
}
Err(StorageError::ConfigNotFound) => None,
Err(error) => return Err(error),
}
};
#[cfg(test)]
tests::pause_reconciliation(&self.bucket, tests::ReconcilePauseStage::BeforeScan).await?;
// Only the independent reconcile owner is held during this whole-bucket
// walk. Concurrent reserve/settle/reap writes invalidate its result.
let usage = exact_bucket_usage(&self.store, &self.bucket).await?;
#[cfg(test)]
tests::pause_reconciliation(&self.bucket, tests::ReconcilePauseStage::BeforePublish).await?;
let configuration = self.configuration_snapshot().await?;
let lock = self.store.new_ns_lock(RUSTFS_META_BUCKET, &ledger_object).await?;
let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await?);
let unchanged = match load_ledger_locked(Arc::clone(&self.store), &ledger_object).await {
Ok(ledger) => {
expected_revision == Some(ledger.writer_revision)
&& !ledger.writer_revision.is_nil()
&& ledger.reservations.is_empty()
}
Err(StorageError::ConfigNotFound) => expected_revision.is_none(),
Err(error) => return Err(error),
};
if !unchanged {
return Ok(());
}
if owner.is_lock_lost()
|| !configuration.is_valid_for_destructive_put(self.store.id, &self.bucket, self.bucket_incarnation)
{
return Err(StorageError::SlowDown);
}
fence_namespace_mutations(&self.store, RUSTFS_META_BUCKET, &ledger_object, None).await?;
let mut ledger = QuotaLedger::new(self.bucket_incarnation, self.quota_revision, usage);
save_ledger_locked(Arc::clone(&self.store), &ledger_object, &mut ledger, &guard).await
}
ledger.accounted_usage = exact_bucket_usage(store, bucket).await?;
ledger.reconcile_required = false;
Ok(())
}
async fn exact_bucket_usage(store: &Arc<ECStore>, bucket: &str) -> Result<u64> {
@@ -1935,7 +2136,7 @@ async fn load_ledger_locked(store: Arc<ECStore>, ledger_object: &str) -> Result<
async fn save_ledger_locked(
store: Arc<ECStore>,
ledger_object: &str,
ledger: &QuotaLedger,
ledger: &mut QuotaLedger,
ledger_guard: &Arc<NamespaceLockGuard>,
) -> Result<()> {
if ledger_guard.is_lock_lost() {
@@ -1959,9 +2160,71 @@ async fn save_ledger_locked(
let _ = opts.set_quota_admission(0, u64::MAX);
opts.add_owned_write_lock(Arc::clone(ledger_guard), RUSTFS_META_BUCKET, ledger_object);
opts.write_completion = crate::object_api::WriteCompletion::TailDrained;
ledger.writer_revision = Uuid::new_v4();
save_config_with_opts(store, ledger_object, serde_json::to_vec(ledger)?, &opts).await
}
/// Advance the persisted reconciliation retry deadline for deterministic fixtures.
/// The production reaper, ownership fence, scan and compare-and-swap still run.
#[cfg(any(test, feature = "test-util"))]
pub async fn retry_quota_reconciliation_for_test(store: Arc<ECStore>, bucket: &str) -> Result<()> {
let (_, incarnation, revision) = store.bucket_quota_and_incarnation_from_disk(bucket).await?;
let job = ReconcileJob::new(Arc::clone(&store), bucket.to_owned(), incarnation, revision);
let key = (store.id, bucket.to_owned(), incarnation, revision.unix_timestamp_nanos());
tokio::time::timeout(Duration::from_secs(10), async {
loop {
let active = active_reconciliations()
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains(&key);
if !active {
break;
}
tokio::task::yield_now().await;
}
})
.await
.map_err(|_| StorageError::SlowDown)?;
reap_stale_reservations(Arc::clone(&store), bucket, &ledger_object(bucket)).await?;
let object = job.control_object();
{
let lock = store.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await?);
match read_config_no_lock(Arc::clone(&store), &object).await {
Ok(bytes) => {
let mut control: ReconcileControl = serde_json::from_slice(&bytes)?;
control.next_attempt_at = 0;
fence_namespace_mutations(&store, RUSTFS_META_BUCKET, &object, None).await?;
job.save_control(&object, &control, &guard).await?;
}
Err(StorageError::ConfigNotFound) => {}
Err(error) => return Err(error),
}
}
job.run().await
}
/// Bootstrap a test bucket through the real background reconciliation path.
/// Fixtures call this after enabling V1 quota, before issuing writes whose
/// contract is independent of the initial retryable admission response.
#[cfg(any(test, feature = "test-util"))]
pub async fn initialize_quota_ledger_for_test(store: Arc<ECStore>, bucket: &str) -> Result<()> {
let (quota, incarnation, revision) = store.bucket_quota_and_incarnation_from_disk(bucket).await?;
if !quota
.as_ref()
.is_some_and(|quota| quota.uses_durable_reservations() && !quota.uses_sharded_reservations())
{
return Ok(());
}
ReconcileJob::new(Arc::clone(&store), bucket.to_owned(), incarnation, revision)
.run()
.await?;
match load_current_ledger_locked(store, &ledger_object(bucket), incarnation, revision).await? {
Some(_) => Ok(()),
None => Err(StorageError::SlowDown),
}
}
#[cfg(any(test, feature = "test-util"))]
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn fail_next_quota_ledger_save_for_test() {
@@ -2003,6 +2266,434 @@ mod tests {
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use serial_test::serial;
#[derive(Clone, Copy, PartialEq, Eq)]
pub(super) enum ReconcilePauseStage {
BeforeScan,
BeforePublish,
}
struct ReconcilePause {
bucket: String,
stage: ReconcilePauseStage,
armed: std::sync::atomic::AtomicBool,
scans: std::sync::atomic::AtomicUsize,
fail_scan: std::sync::atomic::AtomicBool,
reached: tokio::sync::Notify,
release: tokio::sync::Notify,
}
static RECONCILE_PAUSE: std::sync::Mutex<Option<Arc<ReconcilePause>>> = std::sync::Mutex::new(None);
struct ReconcilePauseGuard(Arc<ReconcilePause>);
impl ReconcilePauseGuard {
fn install(bucket: &str, stage: ReconcilePauseStage) -> Self {
let pause = Arc::new(ReconcilePause {
bucket: bucket.to_owned(),
stage,
armed: true.into(),
scans: 0.into(),
fail_scan: false.into(),
reached: tokio::sync::Notify::new(),
release: tokio::sync::Notify::new(),
});
*RECONCILE_PAUSE.lock().expect("reconcile hooks") = Some(Arc::clone(&pause));
Self(pause)
}
async fn reached(&self) {
tokio::time::timeout(Duration::from_secs(10), self.0.reached.notified())
.await
.expect("reconciliation reached barrier");
}
}
impl Drop for ReconcilePauseGuard {
fn drop(&mut self) {
self.0.release.notify_one();
let mut hook = RECONCILE_PAUSE.lock().expect("reconcile hooks");
if hook.as_ref().is_some_and(|hook| Arc::ptr_eq(hook, &self.0)) {
*hook = None;
}
}
}
pub(super) async fn pause_reconciliation(bucket: &str, stage: ReconcilePauseStage) -> Result<()> {
use std::sync::atomic::Ordering;
let hook = RECONCILE_PAUSE
.lock()
.expect("reconcile hooks")
.as_ref()
.filter(|hook| hook.bucket == bucket)
.cloned();
if let Some(hook) = hook {
if stage == ReconcilePauseStage::BeforeScan {
hook.scans.fetch_add(1, Ordering::SeqCst);
}
if stage == hook.stage && hook.armed.swap(false, Ordering::SeqCst) {
hook.reached.notify_one();
hook.release.notified().await;
}
if stage == ReconcilePauseStage::BeforeScan && hook.fail_scan.load(Ordering::SeqCst) {
return Err(StorageError::other("injected quota usage failure"));
}
}
Ok(())
}
async fn reconciliation_fixture(limit: u64) -> (Vec<tempfile::TempDir>, ReconcileJob) {
use crate::object_api::PutObjReader;
use crate::storage_api_contracts::object::ObjectIO;
let (directories, store) = metadata_sys::test_support::isolated_store_over_temp_disks().await;
metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test();
let bucket = format!("quota-reconcile-{}", Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("bucket");
let mut seed = PutObjReader::from_vec(vec![1; 10]);
store
.put_object(&bucket, "seed", &mut seed, &ObjectOptions::default())
.await
.expect("existing usage");
store
.update_bucket_metadata_config(
&bucket,
"quota.json",
serde_json::to_vec(&super::super::BucketQuota::new(Some(limit))).expect("quota JSON"),
)
.await
.expect("enable quota");
let (_, incarnation, revision) = store
.bucket_quota_and_incarnation_from_disk(&bucket)
.await
.expect("quota identity");
(directories, ReconcileJob::new(store, bucket, incarnation, revision))
}
async fn quota_reservation(job: &ReconcileJob, old: u64, new: u64) -> Result<QuotaReservation> {
begin(&job.store.ctx, &job.bucket, "candidate", &ObjectOptions::default(), 0, 0)
.await?
.reserve(old, new)
.await
}
async fn read_test_ledger(job: &ReconcileJob) -> QuotaLedger {
load_ledger_locked(Arc::clone(&job.store), &ledger_object(&job.bucket))
.await
.expect("ledger")
}
async fn drain_reconciliation_scheduler(job: &ReconcileJob) {
let key = (
job.store.id,
job.bucket.clone(),
job.bucket_incarnation,
job.quota_revision.unix_timestamp_nanos(),
);
tokio::time::timeout(Duration::from_secs(10), async {
while active_reconciliations().lock().expect("active workers").contains(&key) {
tokio::task::yield_now().await;
}
})
.await
.expect("owned reconciliation finished");
}
async fn expire_reconciliation_backoff(job: &ReconcileJob) {
drain_reconciliation_scheduler(job).await;
let object = job.control_object();
let lock = job.store.new_ns_lock(RUSTFS_META_BUCKET, &object).await.expect("owner lock");
let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await.expect("owner guard"));
let mut control: ReconcileControl =
serde_json::from_slice(&read_config_no_lock(Arc::clone(&job.store), &object).await.expect("control"))
.expect("control JSON");
control.next_attempt_at = 0;
fence_namespace_mutations(&job.store, RUSTFS_META_BUCKET, &object, None)
.await
.expect("control fence");
job.save_control(&object, &control, &guard).await.expect("advance test clock");
}
async fn wait_for_reconcile_owner(job: &ReconcileJob) {
let object = job.control_object();
let lock = job.store.new_ns_lock(RUSTFS_META_BUCKET, &object).await.expect("owner lock");
let _guard = tokio::time::timeout(Duration::from_secs(10), lock.get_write_lock(get_lock_acquire_timeout()))
.await
.expect("worker finished")
.expect("owner guard");
}
#[tokio::test]
#[serial]
async fn quota_reconciliation_bootstrap_is_single_flight_without_request_guards() {
use std::sync::atomic::Ordering;
let (_directories, job) = reconciliation_fixture(10).await;
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforeScan);
assert!(
matches!(quota_reservation(&job, 0, 1).await, Err(StorageError::SlowDown)),
"unknown usage cannot admit a reservation"
);
pause.reached().await;
assert!(
matches!(
load_ledger_locked(Arc::clone(&job.store), &ledger_object(&job.bucket)).await,
Err(StorageError::ConfigNotFound)
),
"bootstrap must not publish zero usage"
);
assert!(matches!(quota_reservation(&job, 0, 1).await, Err(StorageError::SlowDown)));
// Bypass the local scheduler, exercising the cross-node ownership boundary.
tokio::time::timeout(Duration::from_secs(2), job.run())
.await
.expect("contender must not wait for a scan")
.expect("another owner is active");
let lifecycle = tokio::time::timeout(Duration::from_secs(2), job.store.acquire_bucket_lifecycle_write_lock(&job.bucket))
.await
.expect("scan does not retain lifecycle readers")
.expect("lifecycle writer");
let metadata = tokio::time::timeout(
Duration::from_secs(2),
metadata_sys::acquire_bucket_metadata_transaction_lock_in(&job.store.ctx, &job.bucket),
)
.await
.expect("scan does not retain metadata readers")
.expect("metadata writer");
drop(metadata);
drop(lifecycle);
pause.0.release.notify_one();
wait_for_reconcile_owner(&job).await;
let ledger = read_test_ledger(&job).await;
assert_eq!(ledger.accounted_usage, 10);
assert!(ledger.reservations.is_empty());
assert!(!ledger.writer_revision.is_nil());
for _ in 0..3 {
assert!(matches!(quota_reservation(&job, 0, 1).await, Err(StorageError::QuotaExceeded { .. })));
job.run().await.expect("persisted cooldown");
}
drain_reconciliation_scheduler(&job).await;
assert_eq!(
pause.0.scans.load(Ordering::SeqCst),
1,
"denied requests cannot repeatedly scan a full bucket"
);
}
#[tokio::test]
#[serial]
async fn quota_reconciliation_rejects_concurrent_zero_growth_settle_reap_and_legacy_writes() {
let (_directories, job) = reconciliation_fixture(100).await;
job.run().await.expect("bootstrap");
for mutation in ["abort", "commit", "reap", "legacy"] {
expire_reconciliation_backoff(&job).await;
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforePublish);
let worker = tokio::spawn({
let job = job.clone();
async move { job.run().await }
});
pause.reached().await;
let initial = read_test_ledger(&job).await.writer_revision;
if mutation == "legacy" {
let object = ledger_object(&job.bucket);
let lock = job.store.new_ns_lock(RUSTFS_META_BUCKET, &object).await.expect("ledger lock");
let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await.expect("ledger writer"));
let mut document = serde_json::to_value(read_test_ledger(&job).await).expect("ledger JSON");
document.as_object_mut().expect("ledger map").remove("writer_revision");
fence_namespace_mutations(&job.store, RUSTFS_META_BUCKET, &object, None)
.await
.expect("ledger fence");
let mut opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let _ = opts.set_quota_admission(0, u64::MAX);
opts.add_owned_write_lock(Arc::clone(&guard), RUSTFS_META_BUCKET, &object);
opts.write_completion = crate::object_api::WriteCompletion::TailDrained;
save_config_with_opts(
Arc::clone(&job.store),
&object,
serde_json::to_vec(&document).expect("old node JSON"),
&opts,
)
.await
.expect("old node drops new field");
} else {
let mut reservation = quota_reservation(&job, 10, 10)
.await
.expect("zero-growth admission while scanning");
let reserved = read_test_ledger(&job).await.writer_revision;
assert_ne!(reserved, initial, "even zero-growth reserve must invalidate a scan");
reservation.mark_commit_started().await.expect("commit marker");
assert_ne!(read_test_ledger(&job).await.writer_revision, reserved);
match mutation {
"abort" => reservation.abort().await,
"commit" => reservation.commit().await,
"reap" => {
reservation.defer_after_fence();
reap_stale_reservations(Arc::clone(&job.store), &job.bucket, &ledger_object(&job.bucket))
.await
.expect("reap orphan");
}
_ => unreachable!(),
}
}
let written = read_test_ledger(&job).await;
assert!(written.reservations.is_empty());
assert_ne!(written.writer_revision, initial);
pause.0.release.notify_one();
worker.await.expect("worker join").expect("stale scan discarded");
let current = read_test_ledger(&job).await;
assert_eq!(
current.writer_revision, written.writer_revision,
"{mutation}: stale usage must not replace a concurrent writer"
);
assert_eq!(current.accounted_usage, written.accounted_usage);
}
}
#[tokio::test]
#[serial]
async fn quota_reconciliation_failure_is_throttled_and_deletions_eventually_reclaim_usage() {
use std::sync::atomic::Ordering;
let (_directories, job) = reconciliation_fixture(10).await;
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforeScan);
pause.0.fail_scan.store(true, Ordering::SeqCst);
pause.0.release.notify_one();
assert!(job.run().await.is_err());
assert!(matches!(
load_ledger_locked(Arc::clone(&job.store), &ledger_object(&job.bucket)).await,
Err(StorageError::ConfigNotFound)
));
job.run().await.expect("failure cooldown");
assert_eq!(pause.0.scans.load(Ordering::SeqCst), 1);
pause.0.fail_scan.store(false, Ordering::SeqCst);
expire_reconciliation_backoff(&job).await;
job.run().await.expect("recover after failure");
assert_eq!(read_test_ledger(&job).await.accounted_usage, 10);
job.store
.delete_object(&job.bucket, "seed", ObjectOptions::default())
.await
.expect("delete stored usage");
assert!(
matches!(quota_reservation(&job, 0, 1).await, Err(StorageError::QuotaExceeded { .. })),
"stale usage remains conservative during cooldown"
);
expire_reconciliation_backoff(&job).await;
job.run().await.expect("refresh deleted usage");
assert_eq!(read_test_ledger(&job).await.accounted_usage, 0);
quota_reservation(&job, 0, 1)
.await
.expect("delete must not leave bucket permanently full")
.abort()
.await;
}
#[tokio::test]
#[serial]
async fn quota_reconciliation_survives_waiter_cancellation_and_rechecks_configuration() {
let (_directories, job) = reconciliation_fixture(100).await;
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforePublish);
let caller = tokio::spawn({
let job = job.clone();
async move {
job.schedule();
std::future::pending::<()>().await
}
});
pause.reached().await;
caller.abort();
assert!(caller.await.expect_err("cancel caller").is_cancelled());
job.store
.update_bucket_metadata_config(
&job.bucket,
"quota.json",
serde_json::to_vec(&super::super::BucketQuota::new(Some(200))).expect("updated quota"),
)
.await
.expect("quota changes while scanning");
pause.0.release.notify_one();
wait_for_reconcile_owner(&job).await;
assert!(
matches!(
load_ledger_locked(Arc::clone(&job.store), &ledger_object(&job.bucket)).await,
Err(StorageError::ConfigNotFound)
),
"old config cannot publish bootstrap usage"
);
drop(pause);
let (_, incarnation, revision) = job
.store
.bucket_quota_and_incarnation_from_disk(&job.bucket)
.await
.expect("current configuration");
let current = ReconcileJob::new(Arc::clone(&job.store), job.bucket.clone(), incarnation, revision);
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforeScan);
let caller = tokio::spawn({
let job = current.clone();
async move {
job.schedule();
std::future::pending::<()>().await
}
});
pause.reached().await;
caller.abort();
assert!(caller.await.expect_err("cancel new caller").is_cancelled());
pause.0.release.notify_one();
wait_for_reconcile_owner(&current).await;
assert!(
read_test_ledger(&current).await.matches(incarnation, revision),
"owned scan publishes after its waiter is canceled"
);
}
#[tokio::test]
#[serial]
async fn quota_reconciliation_cannot_publish_into_recreated_bucket() {
use crate::storage_api_contracts::bucket::DeleteBucketOptions;
let (_directories, job) = reconciliation_fixture(100).await;
let pause = ReconcilePauseGuard::install(&job.bucket, ReconcilePauseStage::BeforePublish);
let worker = tokio::spawn({
let job = job.clone();
async move { job.run().await }
});
pause.reached().await;
job.store
.delete_object(&job.bucket, "seed", ObjectOptions::default())
.await
.expect("empty old bucket");
job.store
.delete_bucket(&job.bucket, &DeleteBucketOptions::default())
.await
.expect("retire bucket while scan is paused");
job.store
.make_bucket(&job.bucket, &MakeBucketOptions::default())
.await
.expect("replacement bucket");
job.store
.update_bucket_metadata_config(
&job.bucket,
"quota.json",
serde_json::to_vec(&super::super::BucketQuota::new(Some(100))).expect("new quota"),
)
.await
.expect("replacement quota");
pause.0.release.notify_one();
assert!(worker.await.expect("old worker join").is_err(), "old incarnation must not publish");
assert!(matches!(
load_ledger_locked(Arc::clone(&job.store), &ledger_object(&job.bucket)).await,
Err(StorageError::ConfigNotFound)
));
drop(pause);
initialize_quota_ledger_for_test(Arc::clone(&job.store), &job.bucket)
.await
.expect("replacement can bootstrap independently");
let replacement = read_test_ledger(&job).await;
assert_ne!(replacement.bucket_incarnation, job.bucket_incarnation);
assert_eq!(replacement.accounted_usage, 0);
}
fn ledger(accounted_usage: u64) -> QuotaLedger {
QuotaLedger::new(Uuid::new_v4(), OffsetDateTime::now_utc(), accounted_usage)
}
@@ -2090,8 +2781,7 @@ mod tests {
.expect("persist crash-left reservation fixture");
assert!(matches!(
load_current_ledger_locked(Arc::clone(&store), &bucket, &ledger_path, new_incarnation, OffsetDateTime::now_utc())
.await,
load_current_ledger_locked(Arc::clone(&store), &ledger_path, new_incarnation, OffsetDateTime::now_utc()).await,
Err(StorageError::PartMissingOrCorrupt)
));
assert!(matches!(
@@ -2109,10 +2799,10 @@ mod tests {
Err(StorageError::ConfigNotFound)
));
let replacement =
load_current_ledger_locked(Arc::clone(&store), &bucket, &ledger_path, new_incarnation, OffsetDateTime::now_utc())
load_current_ledger_locked(Arc::clone(&store), &ledger_path, new_incarnation, OffsetDateTime::now_utc())
.await
.expect("new quota admission must not inherit a fresh stale reservation");
assert!(replacement.reservations.is_empty());
assert!(replacement.is_none(), "replacement must await authoritative bootstrap before admission");
}
#[test]
+87 -2
View File
@@ -40,7 +40,9 @@ use aws_smithy_runtime_api::client::http::{
HttpConnector as SmithyHttpConnector, HttpConnectorFuture, SharedHttpConnector, http_client_fn,
};
use aws_smithy_runtime_api::client::interceptors::Intercept;
use aws_smithy_runtime_api::client::interceptors::context::BeforeTransmitInterceptorContextMut;
use aws_smithy_runtime_api::client::interceptors::context::{
BeforeDeserializationInterceptorContextMut, BeforeTransmitInterceptorContextMut,
};
use aws_smithy_runtime_api::client::orchestrator::{HttpRequest, HttpResponse};
use aws_smithy_runtime_api::client::result::ConnectorError;
use aws_smithy_runtime_api::client::runtime_components::RuntimeComponents;
@@ -155,6 +157,8 @@ pub struct RemoteS3EndpointSpec {
pub ca_cert_pem: Option<String>,
pub connect_timeout: Option<Duration>,
pub read_timeout: Option<Duration>,
/// Entire SDK call, including non-streaming response bodies and retries.
pub operation_timeout: Option<Duration>,
/// How many wire requests one logical call may cost. Every consumer
/// declares it; see [`RemoteS3RetryPolicy`].
pub retry: RemoteS3RetryPolicy,
@@ -252,6 +256,78 @@ pub(crate) fn remote_sdk_credentials(credentials: &RemoteCredentials, now: Syste
Ok(builder.build())
}
/// Only installed on non-streaming upload operations. The budget begins at
/// response headers, independently of the potentially long request upload.
#[derive(Debug)]
pub(crate) struct UploadResponseDeadline(pub Duration);
impl Intercept for UploadResponseDeadline {
fn name(&self) -> &'static str {
"RustfsUploadResponseDeadline"
}
fn modify_before_deserialization(
&self,
context: &mut BeforeDeserializationInterceptorContextMut<'_>,
_runtime_components: &RuntimeComponents,
_cfg: &mut ConfigBag,
) -> Result<(), BoxError> {
let response = context.response_mut();
let inner = std::mem::replace(response.body_mut(), SdkBody::taken());
*response.body_mut() = SdkBody::from_body_1_x(UploadResponseBody {
inner: Some(inner),
deadline: Box::pin(tokio::time::sleep(self.0)),
});
Ok(())
}
}
struct UploadResponseBody {
inner: Option<SdkBody>,
deadline: std::pin::Pin<Box<tokio::time::Sleep>>,
}
impl http_body::Body for UploadResponseBody {
type Data = bytes::Bytes;
type Error = BoxError;
fn poll_frame(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
use std::future::Future as _;
use std::task::Poll;
if self.inner.is_none() {
return Poll::Ready(None);
}
// The absolute response budget never renews on data, empty frames,
// or trailers. A retry gets its own bounded response window.
if self.deadline.as_mut().poll(cx).is_ready() {
self.inner.take();
return Poll::Ready(Some(Err(std::io::Error::new(
std::io::ErrorKind::TimedOut,
"replication response body deadline exceeded",
)
.into())));
}
let Some(inner) = self.inner.as_mut() else {
return Poll::Ready(None);
};
let result = std::pin::Pin::new(inner).poll_frame(cx);
if matches!(result, Poll::Ready(None) | Poll::Ready(Some(Err(_)))) {
self.inner.take();
}
result
}
fn is_end_stream(&self) -> bool {
self.inner.as_ref().is_none_or(http_body::Body::is_end_stream)
}
fn size_hint(&self) -> http_body::SizeHint {
self.inner.as_ref().map(http_body::Body::size_hint).unwrap_or_default()
}
}
/// Appends a caller-identifying token to the SDK `User-Agent`. Runs after
/// signing: SigV4 excludes `user-agent` from the canonical request, so the
/// signature stays valid.
@@ -309,7 +385,7 @@ pub async fn build_remote_s3_config(spec: &RemoteS3EndpointSpec) -> Result<aws_s
config_builder = config_builder.http_client(http_client);
}
if spec.connect_timeout.is_some() || spec.read_timeout.is_some() {
if spec.connect_timeout.is_some() || spec.read_timeout.is_some() || spec.operation_timeout.is_some() {
let mut timeouts = TimeoutConfig::builder();
if let Some(connect_timeout) = spec.connect_timeout {
timeouts = timeouts.connect_timeout(connect_timeout);
@@ -317,6 +393,9 @@ pub async fn build_remote_s3_config(spec: &RemoteS3EndpointSpec) -> Result<aws_s
if let Some(read_timeout) = spec.read_timeout {
timeouts = timeouts.read_timeout(read_timeout);
}
if let Some(operation_timeout) = spec.operation_timeout {
timeouts = timeouts.operation_timeout(operation_timeout);
}
config_builder = config_builder.timeout_config(timeouts.build());
}
@@ -674,6 +753,7 @@ mod tests {
ca_cert_pem: None,
connect_timeout: None,
read_timeout: None,
operation_timeout: None,
retry: RemoteS3RetryPolicy::Disabled,
user_agent_suffix: "",
}
@@ -1027,11 +1107,16 @@ mod tests {
spec.user_agent_suffix = "RustFS-Test/0.0";
spec.connect_timeout = Some(Duration::from_secs(5));
spec.read_timeout = Some(Duration::from_secs(5));
spec.operation_timeout = Some(Duration::from_secs(12));
let config = build_remote_s3_config(&spec)
.await
.expect("spec should build")
.http_client(http_client)
.build();
assert_eq!(
config.timeout_config().and_then(TimeoutConfig::operation_timeout),
Some(Duration::from_secs(12))
);
S3Client::from_conf(config)
.head_bucket()
.bucket("bucket")
@@ -16,8 +16,8 @@ use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Error as EcstoreError, is_err_object_not_found, is_err_version_not_found};
use super::replication_filemeta_boundary::{
MrfOpKind, MrfReplicateEntry, REPLICATE_HEAL_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicatedTargetInfo,
ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision, replicate_decision_for_admitted_targets,
replication_statuses_map, version_purge_statuses_map,
ReplicationGenerationSnapshot, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision,
replicate_decision_for_admitted_targets, replication_statuses_map, version_purge_statuses_map,
};
use super::replication_lock_boundary::ReplicationLockTiming;
use super::replication_logging::{EVENT_REPLICATION_CONFIG_LOOKUP_SKIPPED, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REPLICATION};
@@ -1581,7 +1581,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
/// Queues a replica task
pub async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission {
// A version that is already queued or being uploaded is not driven a
// A source generation already queued or being uploaded is not driven a
// second time: the scanner heal pass sees it as PENDING until the
// first upload lands and would otherwise re-queue it every cycle
// (backlog#2362). The key is released when the worker finishes, or
@@ -2404,28 +2404,47 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
}
/// Object versions currently queued or being uploaded, keyed by bucket,
/// object name and version. `queue_replica_task` admits a version only once
/// object name, version and source generation. `queue_replica_task` admits a generation only once
/// while it is in flight; the scanner heal pass and MRF replays that arrive
/// in the meantime are `Skipped` instead of driving a second complete upload
/// (backlog#2362). Entries are removed when the worker finishes the task or
/// when no worker accepted it.
#[derive(Debug, Default)]
pub(crate) struct ReplicationInFlight {
keys: std::sync::Mutex<std::collections::HashSet<(String, String, Option<uuid::Uuid>)>>,
keys: std::sync::Mutex<std::collections::HashSet<ReplicationInFlightKey>>,
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
struct ReplicationInFlightKey {
bucket: String,
object: String,
version_id: Option<uuid::Uuid>,
generation: ReplicationGenerationSnapshot,
}
impl From<&ReplicateObjectInfo> for ReplicationInFlightKey {
fn from(info: &ReplicateObjectInfo) -> Self {
Self {
bucket: info.bucket.clone(),
object: info.name.clone(),
version_id: info.version_id,
generation: info.replication_generation.clone(),
}
}
}
impl ReplicationInFlight {
fn lock(&self) -> std::sync::MutexGuard<'_, std::collections::HashSet<(String, String, Option<uuid::Uuid>)>> {
fn lock(&self) -> std::sync::MutexGuard<'_, std::collections::HashSet<ReplicationInFlightKey>> {
self.keys.lock().unwrap_or_else(|poisoned| poisoned.into_inner())
}
/// Claim `ri`; `false` when the same version is already in flight.
/// Claim `ri`; a metadata mutation is independent of its earlier generation.
fn try_begin(&self, ri: &ReplicateObjectInfo) -> bool {
self.lock().insert((ri.bucket.clone(), ri.name.clone(), ri.version_id))
self.lock().insert(ReplicationInFlightKey::from(ri))
}
fn finish(&self, ri: &ReplicateObjectInfo) {
self.lock().remove(&(ri.bucket.clone(), ri.name.clone(), ri.version_id));
self.lock().remove(&ReplicationInFlightKey::from(ri));
}
#[cfg(test)]
@@ -2438,26 +2457,21 @@ impl ReplicationInFlight {
/// including when replication panics.
struct ReplicationInFlightGuard {
in_flight: Arc<ReplicationInFlight>,
key: ReplicateObjectInfo,
key: ReplicationInFlightKey,
}
impl ReplicationInFlightGuard {
fn new(in_flight: Arc<ReplicationInFlight>, ri: &ReplicateObjectInfo) -> Self {
Self {
in_flight,
key: ReplicateObjectInfo {
bucket: ri.bucket.clone(),
name: ri.name.clone(),
version_id: ri.version_id,
..Default::default()
},
key: ReplicationInFlightKey::from(ri),
}
}
}
impl Drop for ReplicationInFlightGuard {
fn drop(&mut self) {
self.in_flight.finish(&self.key);
self.in_flight.lock().remove(&self.key);
}
}
@@ -4054,6 +4068,48 @@ mod tests {
assert_eq!(current_queue(&pool, "in-flight-bucket").await, (3, 3 * 4096));
}
#[tokio::test]
async fn queue_replica_task_admits_metadata_changes_during_an_initial_upload() {
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", empty_resync_shared_state()))).await;
let (tx, mut rx) = mpsc::channel(4);
pool.workers.write().await.push(tx);
let initial = ReplicateObjectInfo {
bucket: "pending-tags-bucket".to_string(),
name: "object".to_string(),
version_id: Some(uuid::Uuid::new_v4()),
size: 4096,
replication_generation: ReplicationGenerationSnapshot {
mutation_id: Some(uuid::Uuid::new_v4().to_string()),
..Default::default()
},
..Default::default()
};
let metadata = ReplicateObjectInfo {
op_type: ReplicationType::Metadata,
user_tags: String::new(),
replication_generation: ReplicationGenerationSnapshot {
mutation_id: Some(uuid::Uuid::new_v4().to_string()),
..Default::default()
},
..initial.clone()
};
assert_eq!(pool.queue_replica_task(initial.clone()).await, ReplicationQueueAdmission::Queued);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Queued);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Skipped);
assert_eq!(pool.in_flight.len(), 2);
drop(ReplicationInFlightGuard::new(pool.in_flight.clone(), &initial));
assert_eq!(pool.in_flight.len(), 1);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Skipped);
for expected in [&initial, &metadata] {
let ReplicationOperation::Object(queued) = rx.recv().await.expect("queued operation") else {
panic!("expected object replication");
};
assert_eq!(queued.replication_generation, expected.replication_generation);
}
drop(ReplicationInFlightGuard::new(pool.in_flight.clone(), &metadata));
assert_eq!(pool.in_flight.len(), 0);
}
#[tokio::test]
async fn queue_replica_task_releases_the_version_when_no_worker_accepts_it() {
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", empty_resync_shared_state()))).await;
@@ -6609,6 +6609,7 @@ mod tests {
health_check_duration: std::time::Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(aws_sdk_s3::Client::from_conf(config)),
timeouts: Default::default(),
})
}
+12
View File
@@ -259,6 +259,10 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res
pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) -> Result<()> {
check_put_object_args(bucket, object)?;
if !data_movement && !is_meta_bucketname(bucket) && has_xml_invalid_control_characters(object) {
return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
}
if cfg!(target_os = "windows") && !data_movement && !is_meta_bucketname(bucket) && object.contains('\\') {
return Err(StorageError::InvalidArgument(
bucket.to_owned(),
@@ -270,6 +274,14 @@ pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) ->
Ok(())
}
/// New public keys must remain representable in an unencoded XML listing.
/// Existing keys still use the read/delete validators so they can be removed.
pub fn has_xml_invalid_control_characters(object: &str) -> bool {
object
.chars()
.any(|character| character < ' ' && !matches!(character, '\t' | '\n' | '\r'))
}
pub fn check_copy_obj_args(bucket: &str, object: &str) -> Result<()> {
check_bucket_and_object_names(bucket, object)
}
@@ -224,6 +224,8 @@ pub struct ListPathRawOptions {
pub recursive: bool,
pub incl_deleted: bool,
pub skip_hidden_prefix_check: bool,
pub hidden_entry_budget: usize,
pub pending_prefix: Option<String>,
pub filter_prefix: Option<String>,
pub forward_to: Option<String>,
pub min_disks: usize,
@@ -260,6 +262,8 @@ impl Clone for ListPathRawOptions {
recursive: self.recursive,
incl_deleted: self.incl_deleted,
skip_hidden_prefix_check: self.skip_hidden_prefix_check,
hidden_entry_budget: self.hidden_entry_budget,
pending_prefix: self.pending_prefix.clone(),
filter_prefix: self.filter_prefix.clone(),
forward_to: self.forward_to.clone(),
min_disks: self.min_disks,
@@ -289,6 +293,8 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
recursive: opts.recursive,
incl_deleted: opts.incl_deleted,
skip_hidden_prefix_check: opts.skip_hidden_prefix_check,
hidden_entry_budget: opts.hidden_entry_budget,
pending_prefix: opts.pending_prefix.clone(),
report_notfound: opts.report_not_found,
filter_prefix: opts.filter_prefix.clone(),
forward_to: opts.forward_to.clone(),
@@ -301,13 +307,15 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
}
/// How a merged raw listing stream ended.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct ListPathRawOutcome {
/// At least one producer stopped at its per-disk limit instead of at the
/// end of its namespace, so the closed stream is not proof of exhaustion.
/// Carried in-band by the metacache stream trailer, so remote producers
/// report it as well.
pub limit_reached: bool,
/// Inclusive sorted boundary proved by every reader before a bounded stop.
pub scan_cut: Option<String>,
}
pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> disk::error::Result<()> {
@@ -722,6 +730,8 @@ async fn list_path_raw_inner(
// Sticky across rounds: once any reader's trailer says its producer
// stopped at the per-disk limit, the merged stream may hide more keys.
let mut limit_reached = false;
let mut scan_cut = None;
let mut last_scanned_key: Option<String> = None;
loop {
let mut current = MetaCacheEntry::default();
@@ -733,7 +743,7 @@ async fn list_path_raw_inner(
// );
if revjob_rx.is_cancelled() {
return Ok(ListPathRawOutcome { limit_reached });
return Ok(ListPathRawOutcome { limit_reached, scan_cut });
}
let mut top_entries: Vec<Option<MetaCacheEntry>> = vec![None; readers.len()];
@@ -793,7 +803,12 @@ async fn list_path_raw_inner(
continue;
}
// eof
limit_reached |= r.limit_reached() == Some(true);
if r.limit_reached() == Some(true) {
limit_reached = true;
scan_cut = Some(last_scanned_key.clone().ok_or_else(|| {
DiskError::other("bounded listing reader ended without a scan boundary")
})?);
}
at_eof += 1;
// warn!("list_path_raw: peek eof, disk: {}", i);
continue;
@@ -867,7 +882,7 @@ async fn list_path_raw_inner(
*r = MetacacheReader::new(detached_rd);
continue;
}
PeekOutcome::Cancelled => return Ok(ListPathRawOutcome { limit_reached }),
PeekOutcome::Cancelled => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
}
};
@@ -926,7 +941,7 @@ async fn list_path_raw_inner(
if let Some(finished_fn) = opts.finished.as_ref() {
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = finished_fn(&errs) => {}
}
}
@@ -976,6 +991,14 @@ async fn list_path_raw_inner(
return Err(classify_listing_quorum_failure(&failures));
}
// The next head from a bounded reader is unknown. The previous
// globally consumed group is a safe cut; consuming another reader's
// later head would turn this absence of evidence into a skipped key.
// Check quorum failures above before accepting the bounded result.
if limit_reached {
break;
}
// Break if all at EOF or error.
if at_eof + has_err == readers.len() {
if has_err > 0
@@ -983,7 +1006,7 @@ async fn list_path_raw_inner(
{
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = finished_fn(&errs) => {}
}
}
@@ -994,6 +1017,7 @@ async fn list_path_raw_inner(
break;
}
last_scanned_key = Some(current.name.clone());
if agree == readers.len() && !opts.preserve_replica_metadata {
for r in readers.iter_mut() {
if !r.skip_current() {
@@ -1005,7 +1029,7 @@ async fn list_path_raw_inner(
// warn!("list_path_raw: agreed_fn start, current: {:?}", &current.name);
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
result = agreed_fn(current) => result?
}
// warn!("list_path_raw: agreed_fn done");
@@ -1029,12 +1053,12 @@ async fn list_path_raw_inner(
if let Some(partial_fn) = opts.partial.as_ref() {
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = partial_fn(MetaCacheEntries(top_entries), &errs) => {}
}
}
}
Ok(ListPathRawOutcome { limit_reached })
Ok(ListPathRawOutcome { limit_reached, scan_cut })
});
let merge_started = std::time::Instant::now();
@@ -1199,6 +1223,24 @@ mod tests {
assert!(!options.skip_total_timeout);
}
#[test]
fn cloned_reader_options_preserve_hidden_scan_budget_and_pending_prefix() {
let request = ListPathRawOptions {
bucket: "versioned".to_owned(),
hidden_entry_budget: 100,
pending_prefix: Some("pending/".to_owned()),
forward_to: Some("pending/0099".to_owned()),
..Default::default()
};
// Primary and fallback readers both construct their walk from the
// per-job clone, rather than directly from the request options.
let job = request.clone();
let walk = walk_dir_options(&job);
assert_eq!(walk.hidden_entry_budget, 100);
assert_eq!(walk.pending_prefix.as_deref(), request.pending_prefix.as_deref());
assert_eq!(walk.forward_to.as_deref(), Some("pending/0099"));
}
#[tokio::test]
async fn list_path_raw_empty_disks_returns_read_quorum() {
let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
@@ -1316,6 +1358,7 @@ mod tests {
outcome.limit_reached,
"a reader whose trailer reports a limit stop must mark the merged stream as possibly incomplete"
);
assert_eq!(outcome.scan_cut.as_deref(), Some("bucket/object"));
}
#[tokio::test]
+44 -1
View File
@@ -1588,7 +1588,13 @@ fn verify_tonic_replay_scope_signature(audience: &str, path: &str, headers: &Hea
if boot_epoch != tonic_rpc_boot_epoch() {
return Err(std::io::Error::other("RPC boot epoch is stale"));
}
check_and_record_tonic_nonce(nonce, signed_at, path)
check_and_record_tonic_nonce(nonce, signed_at, path)?;
// A body-bound request also carries a usable v2 signature. Consume both
// nonces so stripping the replay-scope headers cannot authorize it again.
if content_sha256 != UNSIGNED_PAYLOAD {
verify_tonic_rpc_signature_with_strictness(audience, path, headers, true)?;
}
Ok(())
}
/// Verify gRPC authentication, preferring v2 without downgrade on malformed v2 metadata.
@@ -2579,6 +2585,43 @@ mod tests {
assert_eq!(path_error.to_string(), "Invalid RPC replay scope signature");
}
#[test]
fn replay_scope_consumes_the_body_bound_v2_nonce() {
ensure_test_rpc_secret();
let audience = "node-a:9000";
let path = "/node_service.NodeService/RecoverDeleteTransaction";
let digest = "a".repeat(64);
for v3_first in [true, false] {
let mut headers =
gen_tonic_signature_headers(audience, "node_service.NodeService", "RecoverDeleteTransaction", Some(&digest))
.expect("body-bound v2 headers should build");
let timestamp = headers[TIMESTAMP_HEADER].to_str().expect("timestamp").to_string();
headers.extend(
gen_tonic_replay_scope_headers(audience, path, &timestamp, &digest, tonic_rpc_boot_epoch())
.expect("replay-scope headers should build"),
);
let mut downgraded = headers.clone();
for header in [
RPC_REPLAY_SCOPE_VERSION_HEADER,
RPC_REPLAY_SCOPE_SIGNATURE_HEADER,
RPC_REPLAY_SCOPE_NONCE_HEADER,
RPC_BOOT_EPOCH_HEADER,
] {
downgraded.remove(header);
}
let (first, replay) = if v3_first {
(&headers, &downgraded)
} else {
(&downgraded, &headers)
};
verify_tonic_rpc_signature_with_policy(audience, path, first, true, false, false)
.expect("the first delivery must be accepted");
let error = verify_tonic_rpc_signature_with_policy(audience, path, replay, true, false, false)
.expect_err("a mutation cannot replay through the other signature generation");
assert_eq!(error.to_string(), "RPC request replay detected");
}
}
#[test]
fn replay_scope_rejects_partial_metadata_and_stale_epoch_without_fallback() {
ensure_test_rpc_secret();
@@ -2014,9 +2014,21 @@ impl PeerRestClient {
}
result
})
.await
.unwrap_or_else(|_| Err(Error::other(format!("load_bucket_metadata({bucket}) timed out"))));
self.finalize_result(result).await
.await;
self.settle_bucket_metadata_reload(bucket, result).await
}
async fn settle_bucket_metadata_reload(
&self,
bucket: &str,
result: std::result::Result<Result<()>, tokio::time::error::Elapsed>,
) -> Result<()> {
match result {
Ok(result) => self.finalize_result(result).await,
// A local operation deadline does not establish that the peer is
// unreachable. In particular, it must not gate IAM revocations.
Err(_) => Err(Error::other(format!("load_bucket_metadata({bucket}) timed out"))),
}
}
async fn load_bucket_metadata_once(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
@@ -4753,6 +4765,31 @@ mod tests {
assert!(client.offline.load(Ordering::Acquire));
}
#[tokio::test]
async fn bucket_metadata_deadline_does_not_gate_healthy_peer_notifications() {
let client = test_peer_client();
let elapsed = tokio::time::timeout(std::time::Duration::ZERO, std::future::pending::<Result<()>>()).await;
let error = client
.settle_bucket_metadata_reload("slow-policy", elapsed)
.await
.expect_err("local budget expires");
assert!(error.to_string().contains("timed out"));
assert!(
!client.offline.load(Ordering::Acquire),
"local work must not suppress subsequent IAM RPCs"
);
client
.settle_bucket_metadata_reload("healthy", Ok(Ok(())))
.await
.expect("peer reply");
let error = client
.settle_bucket_metadata_reload("unreachable", Ok(Err(Error::from(tonic::Status::unavailable("connection refused")))))
.await
.expect_err("real transport failure");
assert!(error.to_string().contains("connection refused"));
assert!(client.offline.load(Ordering::Acquire), "transport failures must still enter recovery");
}
#[tokio::test]
async fn peer_rest_client_finalize_result_keeps_online_for_business_errors() {
let client = test_peer_client();
+298 -1
View File
@@ -1082,7 +1082,10 @@ impl RemoteDisk {
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
}
let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
let settle_timeout = if matches!(
&operation,
DeleteRecoveryRpcOperation::Settle { .. } | DeleteRecoveryRpcOperation::SettleBounded(_)
) {
Some(crate::disk::get_delete_recovery_timeout()?)
} else {
None
@@ -2716,6 +2719,54 @@ impl RemoteDisk {
#[async_trait::async_trait]
impl DiskAPI for RemoteDisk {
async fn list_multipart_index_roots(
&self,
bucket: &str,
marker: Option<&str>,
) -> Result<crate::disk::multipart_index::MultipartIndexPage> {
use crate::disk::multipart_index::{MULTIPART_INDEX_RPC_MAX_BYTES, MultipartIndexRpcReply, MultipartIndexRpcRequest};
let expected_disk = self
.id
.lock()
.await
.filter(|id| !id.is_nil())
.ok_or(DiskError::DiskNotFound)?;
let query = MultipartIndexRpcRequest::new(bucket, marker);
let endpoint = self.endpoint.to_string();
let canonical = query.canonical_request(&endpoint)?;
let opts = serde_json::to_string(&query)?;
self.execute_with_timeout(
|| async {
let mut client = self.get_client().await?;
let mut request = Request::new(DiskInfoRequest {
disk: endpoint.clone(),
opts: opts.clone(),
});
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical).map_err(Error::other)?;
let response = client.disk_info(request).await?.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
}
if response.disk_info.len() > MULTIPART_INDEX_RPC_MAX_BYTES {
return Err(DiskError::FileCorrupt);
}
// An old server ignores the additive query and returns ordinary DiskInfo.
// It cannot satisfy the dedicated reply schema or the fresh challenge.
let reply: MultipartIndexRpcReply =
serde_json::from_str(&response.disk_info).map_err(|_| DiskError::MethodNotAllowed)?;
reply.validate(&query, expected_disk)?;
crate::cluster::rpc::verify_tonic_rpc_response_proof(
&query.canonical_response(&endpoint, &reply.page)?,
&reply.proof,
)
.map_err(Error::other)?;
Ok(reply.page)
},
get_drive_disk_info_timeout(),
)
.await
}
async fn supports_guarded_delete_versions(&self) -> Result<()> {
let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?;
if !response.errors.is_empty() || !response.item_errors.is_empty() {
@@ -4299,6 +4350,35 @@ impl DiskAPI for RemoteDisk {
}
}
async fn discover_delete_transactions_bounded(
&self,
volume: &str,
object: &str,
max_entries: u32,
) -> Result<crate::disk::BoundedDeleteRecoveryDiscovery> {
// Older peers reject this variant. Do not retry through the unbounded protocol.
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::DiscoverBounded {
object: object.to_owned(),
max_entries,
},
)
.await
{
Ok(DeleteRecoveryRpcReply::DiscoveredBounded(page)) => Ok(page),
Ok(_) => Err(Error::FileCorrupt),
Err(error)
if matches!(error, Error::MethodNotAllowed | Error::UnsupportedDisk)
|| matches!(&error, Error::Io(source) if crate::cluster::rpc::client::embedded_tonic_status(source)
.is_some_and(|status| matches!(status.code(), tonic::Code::Unimplemented | tonic::Code::InvalidArgument))) =>
{
Err(Error::DeleteRecoveryWorkLimit)
}
Err(error) => Err(error),
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
@@ -4334,6 +4414,58 @@ impl DiskAPI for RemoteDisk {
_ => Err(Error::other("unexpected delete discovery reply")),
}
}
async fn settle_delete_transaction_bounded(
&self,
volume: &str,
request: crate::disk::BoundedDeleteRecoverySettlement,
_namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<crate::disk::BoundedDeleteRecoverySettlementReply> {
match self
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::SettleBounded(request))
.await
{
Ok(DeleteRecoveryRpcReply::SettledBounded(reply)) => Ok(reply),
Ok(_) => Err(Error::FileCorrupt),
Err(error)
if matches!(error, Error::MethodNotAllowed | Error::UnsupportedDisk)
|| matches!(&error, Error::Io(source) if crate::cluster::rpc::client::embedded_tonic_status(source)
.is_some_and(|status| matches!(status.code(), tonic::Code::Unimplemented | tonic::Code::InvalidArgument))) =>
{
Err(Error::DeleteRecoveryWorkLimit)
}
Err(error) => Err(error),
}
}
async fn inspect_delete_transaction_bounded(
&self,
volume: &str,
object: &str,
transaction: Uuid,
max_entries: u32,
) -> Result<crate::disk::BoundedDeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::InspectBounded {
object: object.to_owned(),
transaction,
max_entries,
},
)
.await
{
Ok(DeleteRecoveryRpcReply::InspectedBounded(page)) => Ok(page),
Ok(_) => Err(Error::FileCorrupt),
Err(error)
if matches!(error, Error::MethodNotAllowed | Error::UnsupportedDisk)
|| matches!(&error, Error::Io(source) if crate::cluster::rpc::client::embedded_tonic_status(source)
.is_some_and(|status| matches!(status.code(), tonic::Code::Unimplemented | tonic::Code::InvalidArgument))) =>
{
Err(Error::DeleteRecoveryWorkLimit)
}
Err(error) => Err(error),
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
@@ -4854,6 +4986,113 @@ mod tests {
}
}
#[tokio::test]
async fn bounded_delete_discovery_lost_remote_reply_leaves_no_cursor() {
runtime_sources::ensure_test_rpc_secret();
let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new())
.await
.expect("test peer must bind");
let directory = tempfile::tempdir().expect("temporary disk");
let endpoint = Endpoint::try_from(directory.path().to_str().expect("disk path")).expect("endpoint");
let disk = Arc::new(
crate::disk::local::LocalDisk::new(&endpoint, false)
.await
.expect("local disk"),
);
disk.make_volume("bucket").await.expect("bucket");
let prefix = directory.path().join("bucket/prefix");
std::fs::create_dir(&prefix).expect("prefix");
for index in 0..64 {
std::fs::create_dir(prefix.join(format!("child-{index}"))).expect("legitimate child");
}
let fixture = Arc::new(TestBoundedDiscovery {
disk: Arc::clone(&disk),
response_gate: TestFormatReadGate::default(),
});
*peer.peer.bounded_discovery.lock().expect("bounded fixture") = Some(Arc::clone(&fixture));
let remote = remote_disk_for_addr(&peer.addr).await;
let task = tokio::spawn(async move { remote.discover_delete_transactions_bounded("bucket", "prefix", 8).await });
tokio::time::timeout(Duration::from_secs(5), fixture.response_gate.entered.notified())
.await
.expect("disk completed bounded scan");
task.abort();
assert!(task.await.expect_err("caller cancelled").is_cancelled());
assert_eq!(disk.public_delete_discovery_sessions_for_test(), 0);
fixture.response_gate.release.cancel();
peer.stop().await;
assert_eq!(disk.public_delete_discovery_sessions_for_test(), 0);
assert!(prefix.join("child-63").is_dir());
}
#[tokio::test]
async fn multipart_index_old_disk_info_handler_is_not_a_ready_empty_index() {
runtime_sources::ensure_test_rpc_secret();
let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new())
.await
.expect("old peer must bind");
let remote = remote_disk_for_addr(&peer.addr).await;
remote
.set_disk_id(Some(Uuid::new_v4()))
.await
.expect("expected disk identity");
assert!(matches!(
remote.list_multipart_index_roots("bucket", None).await,
Err(DiskError::MethodNotAllowed)
));
assert_eq!(
peer.peer.disk_info_calls.load(Ordering::Acquire),
1,
"exercise the actual legacy JSON handler"
);
peer.stop().await;
}
#[tokio::test]
async fn bounded_delete_discovery_old_peer_rejects_without_paginated_fallback() {
runtime_sources::ensure_test_rpc_secret();
let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new())
.await
.expect("old peer must bind");
let remote = remote_disk_for_addr(&peer.addr).await;
assert_eq!(
remote
.discover_delete_transactions_bounded("bucket", "prefix", 4096)
.await
.expect_err("old peer cannot prove a bounded inventory"),
DiskError::DeleteRecoveryWorkLimit
);
let transaction = Uuid::new_v4();
assert_eq!(
remote
.inspect_delete_transaction_bounded("bucket", "prefix", transaction, 65536)
.await
.expect_err("old peer cannot inspect with a budget"),
DiskError::DeleteRecoveryWorkLimit
);
assert_eq!(
remote
.settle_delete_transaction_bounded(
"bucket",
crate::disk::BoundedDeleteRecoverySettlement {
object: "prefix".to_owned(),
transaction,
expected_fingerprint: [0; 32],
action: DeleteRecoveryAction::PublishCommit,
max_entries: 65536,
},
None
)
.await
.expect_err("old peer cannot settle with a budget"),
DiskError::DeleteRecoveryWorkLimit
);
assert_eq!(
*peer.peer.unsupported_methods.lock().expect("observed methods"),
vec!["/node_service.NodeService/RecoverDeleteTransaction"; 3]
);
peer.stop().await;
}
#[tokio::test]
async fn multipart_source_recovery_old_peer_rejects_without_generic_fallback() {
runtime_sources::ensure_test_rpc_secret();
@@ -5746,9 +5985,16 @@ mod tests {
format_data: Bytes,
read_all_data: Bytes,
next_format_read_gate: Arc<StdMutex<Option<Arc<TestFormatReadGate>>>>,
bounded_discovery: Arc<StdMutex<Option<Arc<TestBoundedDiscovery>>>>,
unsupported_methods: Arc<StdMutex<Vec<String>>>,
}
#[derive(Debug)]
struct TestBoundedDiscovery {
disk: Arc<crate::disk::local::LocalDisk>,
response_gate: TestFormatReadGate,
}
#[derive(Debug, Default)]
struct TestFormatReadGate {
entered: tokio::sync::Notify,
@@ -5765,6 +6011,7 @@ mod tests {
format_data,
read_all_data,
next_format_read_gate: Arc::default(),
bounded_discovery: Arc::default(),
unsupported_methods: Arc::default(),
}
}
@@ -5864,6 +6111,56 @@ mod tests {
Ok(grpc.unary(method, request).await)
})
}
"/node_service.NodeService/RecoverDeleteTransaction"
if self.peer.bounded_discovery.lock().expect("bounded fixture").is_some() =>
{
#[derive(Clone)]
struct BoundedSvc(Arc<TestBoundedDiscovery>);
impl tonic::server::UnaryService<RecoverDeleteTransactionRequest> for BoundedSvc {
type Response = rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse;
type Future = Pin<Box<dyn Future<Output = std::result::Result<Response<Self::Response>, Status>> + Send>>;
fn call(&mut self, request: Request<RecoverDeleteTransactionRequest>) -> Self::Future {
let fixture = Arc::clone(&self.0);
Box::pin(async move {
let request = request.into_inner();
let operation: DeleteRecoveryRpcOperation =
rustfs_protos::decode_delete_recovery_payload(&request.operation)
.map_err(|_| Status::invalid_argument("invalid bounded request"))?;
let DeleteRecoveryRpcOperation::DiscoverBounded { object, max_entries } = operation else {
return Err(Status::invalid_argument("unexpected fallback"));
};
let page = fixture
.disk
.discover_delete_transactions_bounded(&request.volume, &object, max_entries)
.await
.map_err(|_| Status::internal("bounded discovery failed"))?;
// Lose the response only after the real disk operation has completed.
fixture.response_gate.entered.notify_one();
fixture.response_gate.release.cancelled().await;
let result = rmp_serde::to_vec_named(&DeleteRecoveryRpcReply::DiscoveredBounded(page))
.map_err(|_| Status::internal("encode bounded reply"))?;
Ok(Response::new(rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse {
success: true,
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
result: result.into(),
error: None,
}))
})
}
}
let fixture = self
.peer
.bounded_discovery
.lock()
.expect("bounded fixture")
.clone()
.expect("configured fixture");
Box::pin(async move {
let codec = tonic_prost::ProstCodec::default();
let mut grpc = tonic::server::Grpc::new(codec);
Ok(grpc.unary(BoundedSvc(fixture), request).await)
})
}
"/node_service.NodeService/ReadAll" => {
#[derive(Clone)]
struct ReadAllSvc(AuthenticatedReadPeer);
+3 -2
View File
@@ -317,8 +317,9 @@ impl Sets {
&self.ctx
}
#[cfg(test)]
pub(crate) async fn stop_endpoint_monitor_for_test(&self) {
/// Stop endpoint reconnection and wait for any in-flight pass before a test owns disk probes.
#[cfg(any(test, feature = "test-util"))]
pub async fn stop_endpoint_monitor_for_test(&self) {
if let Some(exit_signal) = &self.exit_signal {
let _ = exit_signal.send(());
// Wait for an in-flight reconnect pass as well as its receiver to exit.
+98 -3
View File
@@ -55,6 +55,20 @@ use tokio_util::sync::CancellationToken;
use tracing::{info, warn};
use uuid::Uuid;
/// Join the local disk's actual bootstrap worker for deterministic coverage tests.
/// Completion may mean legacy fallback after a failed scan; callers must assert
/// the expected index result separately. Remote disks have no local worker.
#[cfg(any(test, feature = "test-util"))]
pub async fn wait_for_multipart_index_bootstrap(disk: &crate::disk::DiskStore) -> Result<()> {
match disk.as_ref() {
crate::disk::Disk::Local(local) => {
local.disk.wait_for_multipart_index_bootstrap().await;
Ok(())
}
crate::disk::Disk::Remote(_) => Err(DiskError::MethodNotAllowed),
}
}
/// Disk health status constants
const DISK_HEALTH_OK: u32 = 0;
const DISK_HEALTH_FAULTY: u32 = 1;
@@ -2443,6 +2457,7 @@ impl LocalDiskWrapper {
object: &str,
transaction: uuid::Uuid,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(crate::disk::DeleteRecoveryJournal, bool, bool)>> {
self.track_disk_health_mutation(
"settle_delete_transaction",
@@ -2450,7 +2465,7 @@ impl LocalDiskWrapper {
|| {
Box::pin(async {
self.disk
.try_publish_prepared_delete_local(volume, object, transaction, namespace_owner)
.try_publish_prepared_delete_local_with_budget(volume, object, transaction, namespace_owner, work_budget)
.await
})
},
@@ -2466,6 +2481,7 @@ impl LocalDiskWrapper {
transaction: uuid::Uuid,
journal: &crate::disk::DeleteRecoveryJournal,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<bool> {
self.track_disk_health_mutation(
"settle_delete_transaction",
@@ -2473,7 +2489,14 @@ impl LocalDiskWrapper {
|| {
Box::pin(async {
self.disk
.try_finish_and_finalize_prepared_delete_local(volume, object, transaction, journal, namespace_owner)
.try_finish_and_finalize_prepared_delete_local_with_budget(
volume,
object,
transaction,
journal,
namespace_owner,
work_budget,
)
.await
})
},
@@ -2728,6 +2751,18 @@ impl LocalDiskWrapper {
#[async_trait::async_trait]
impl DiskAPI for LocalDiskWrapper {
async fn list_multipart_index_roots(
&self,
bucket: &str,
marker: Option<&str>,
) -> Result<crate::disk::multipart_index::MultipartIndexPage> {
if self.health.is_faulty() {
return Err(DiskError::FaultyDisk);
}
self.check_disk_stale().await?;
self.disk.list_multipart_index_roots(bucket, marker).await
}
async fn supports_guarded_delete_versions(&self) -> Result<()> {
// The local implementation shares this binary's capability. Availability
// and disk identity remain mutation checks and write-quorum inputs.
@@ -2781,6 +2816,62 @@ impl DiskAPI for LocalDiskWrapper {
)
.await
}
async fn discover_delete_transactions_bounded(
&self,
volume: &str,
object: &str,
max_entries: u32,
) -> Result<crate::disk::BoundedDeleteRecoveryDiscovery> {
self.track_disk_health_with_op(
"discover_delete_transactions",
|| async {
self.disk
.discover_delete_transactions_bounded(volume, object, max_entries)
.await
},
get_max_timeout_duration(),
)
.await
}
async fn inspect_delete_transaction_bounded(
&self,
volume: &str,
object: &str,
transaction: uuid::Uuid,
max_entries: u32,
) -> Result<crate::disk::BoundedDeleteRecoveryInspection> {
self.track_disk_health_with_op(
"inspect_delete_transaction",
|| async {
self.disk
.inspect_delete_transaction_bounded(volume, object, transaction, max_entries)
.await
},
get_max_timeout_duration(),
)
.await
}
async fn settle_delete_transaction_bounded(
&self,
volume: &str,
request: crate::disk::BoundedDeleteRecoverySettlement,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<crate::disk::BoundedDeleteRecoverySettlementReply> {
self.track_disk_health_mutation(
"settle_delete_transaction",
DiskMetricMutation::Delete,
|| async {
self.disk
.settle_delete_transaction_bounded(volume, request, namespace_owner)
.await
},
crate::disk::get_delete_recovery_timeout()?,
)
.await
}
async fn discover_delete_transactions(
&self,
volume: &str,
@@ -3749,7 +3840,11 @@ mod tests {
assert!(futures::poll!(&mut operation).is_pending());
time::advance(SLOW_DISK_OPERATION_THRESHOLD - Duration::from_millis(1)).await;
assert!(futures::poll!(&mut operation).is_pending());
assert!(logs.contents().is_empty(), "normal disk latency should not warn");
assert!(
!logs.contents().contains("disk_operation_slow"),
"normal disk latency should not warn: {}",
logs.contents()
);
time::advance(Duration::from_millis(1)).await;
assert!(futures::poll!(&mut operation).is_pending());
let warning = logs.contents();
+6
View File
@@ -172,6 +172,8 @@ pub enum DiskError {
#[error("guarded batch deletion is not supported by this disk")]
GuardedDeleteUnsupported,
#[error("foreground delete recovery work limit reached")]
DeleteRecoveryWorkLimit,
#[error("erasure write quorum")]
ErasureWriteQuorum,
@@ -770,6 +772,7 @@ impl Clone for DiskError {
DiskError::MethodNotAllowed => DiskError::MethodNotAllowed,
DiskError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict,
DiskError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported,
DiskError::DeleteRecoveryWorkLimit => DiskError::DeleteRecoveryWorkLimit,
DiskError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum,
DiskError::ErasureReadQuorum => DiskError::ErasureReadQuorum,
DiskError::ShortWrite => DiskError::ShortWrite,
@@ -829,6 +832,7 @@ impl DiskError {
DiskError::RemoteClientUnavailable(_) => 0x2B,
DiskError::ConditionalDeleteConflict => 0x2C,
DiskError::GuardedDeleteUnsupported => 0x2D,
DiskError::DeleteRecoveryWorkLimit => 0x2E,
}
}
@@ -879,6 +883,7 @@ impl DiskError {
0x2B => Some(DiskError::RemoteClientUnavailable(String::new())),
0x2C => Some(DiskError::ConditionalDeleteConflict),
0x2D => Some(DiskError::GuardedDeleteUnsupported),
0x2E => Some(DiskError::DeleteRecoveryWorkLimit),
_ => None,
}
}
@@ -1662,6 +1667,7 @@ mod tests {
for (error, code) in [
(DiskError::ConditionalDeleteConflict, 0x2C),
(DiskError::GuardedDeleteUnsupported, 0x2D),
(DiskError::DeleteRecoveryWorkLimit, 0x2E),
] {
assert_eq!(error.to_u32(), code);
assert_eq!(DiskError::from_u32(code), Some(error.clone()));
+456
View File
@@ -183,3 +183,459 @@ async fn uring_fd_cache_invalidation_keeps_borrowed_old_inode_alive_and_allows_r
assert_eq!(fresh.len, 11);
}
}
#[tokio::test]
async fn fd_cache_tracker_skips_unmatched_invalidations_and_handles_deep_prefixes() {
let (_directory, file) = fixture();
let mut cache = FdCache::new();
// Unmatched invalidations on an empty cache must not spawn the maintenance task.
cache.invalidate_under("v", "missing/prefix");
cache.invalidate_volume("v");
cache.invalidate_exact("v", "missing/prefix/part.1").await;
cache.clear();
assert!(
cache.maintenance_task.get().is_none(),
"empty-cache invalidations must not spawn maintenance"
);
// Populate a normal entry and verify matched vs unmatched filter gates directly,
// and ensure unmatched invalidations do not enqueue pending removals.
let cached_key = key("v", "obj-1/0198a3f0/part.1", false);
cache.insert(cached_key.clone(), Arc::clone(&file)).await;
assert_eq!(cache.tracker.live_entries(), 1);
assert!(cache.tracker.may_match_volume("v"));
assert!(!cache.tracker.may_match_volume("other-volume"));
assert!(cache.tracker.may_match_prefix("v", "obj-1"));
assert!(cache.tracker.may_match_prefix("v", "obj-1/0198a3f0"));
assert!(!cache.tracker.may_match_prefix("v", "obj-2/0198a3f0"));
assert!(cache.tracker.may_match_exact("v", "obj-1/0198a3f0/part.1"));
assert!(!cache.tracker.may_match_exact("v", "obj-1/0198a3f0/xl.meta"));
cache.invalidate_volume("other-volume");
cache.invalidate_under("v", "obj-2/0198a3f0");
cache.invalidate_exact("v", "obj-1/0198a3f0/xl.meta").await;
assert!(
!cache
.tracker
.has_pending_invalidations
.load(std::sync::atomic::Ordering::SeqCst),
"unmatched invalidations must not enqueue maintenance removals"
);
assert!(cache.get(&cached_key).await.is_some());
// Deeply nested path (> FD_CACHE_MAX_INDEXED_PREFIXES slashes) must fall back
// to per-volume unindexed_counts without degrading other volumes.
let other_vol_key = key("v2", "shallow/part.1", false);
cache.insert(other_vol_key.clone(), Arc::clone(&file)).await;
let deep_path = "c1/c2/c3/c4/c5/c6/c7/c8/c9/c10/c11/c12/c13/c14/part.1";
let deep_key = key("v", deep_path, false);
cache.insert(deep_key.clone(), Arc::clone(&file)).await;
assert_eq!(
cache
.tracker
.unindexed_counts
.iter()
.map(|c| c.load(std::sync::atomic::Ordering::Acquire))
.sum::<u32>(),
1
);
assert!(
!cache.tracker.may_match_prefix("v2", "unmatched/prefix"),
"deep key in volume v must not disable prefix filtering in volume v2"
);
cache.invalidate_under("v", "c1/c2/c3/c4/c5/c6/c7/c8/c9/c10/c11/c12/c13");
assert!(cache.get(&deep_key).await.is_none());
cache.invalidate_volume("v2");
assert!(cache.get(&other_vol_key).await.is_none());
// Replacing an invalidated key before maintenance drains pending_invalidations
// must release the old descriptor immediately and keep the replacement live
// even when a strong reference to the old CachedFdEntry shell survives until
// maintenance runs its Arc::ptr_eq check.
let old_file = Arc::new(tempfile::tempfile().expect("create replaced descriptor"));
let old_weak = Arc::downgrade(&old_file);
cache.insert(cached_key.clone(), old_file).await;
let held_old_shell = cache.cache.get(&cached_key).expect("cached shell must exist");
cache.invalidate_under("v", "obj-1/0198a3f0");
assert!(cache.get(&cached_key).await.is_none());
cache.insert(cached_key.clone(), Arc::clone(&file)).await;
assert!(
old_weak.upgrade().is_none(),
"replacing an entry must retire and drop the previous descriptor immediately"
);
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while cache
.tracker
.has_pending_invalidations
.load(std::sync::atomic::Ordering::SeqCst)
{
tokio::task::yield_now().await;
}
})
.await
.expect("maintenance should drain pending invalidations");
drop(held_old_shell);
assert!(
cache.get(&cached_key).await.is_some(),
"stale pending invalidation must not evict a newer replacement"
);
assert_eq!(
cache
.tracker
.unindexed_counts
.iter()
.map(|c| c.load(std::sync::atomic::Ordering::Acquire))
.sum::<u32>(),
0
);
// Dropping a cache while a live non-invalidated entry is present (even if a
// cloned Moka handle outlives FdCache and the maintenance task is aborted)
// must retire the descriptor and release FdCacheTracker without cycles.
let live_drop_file = Arc::new(tempfile::tempfile().expect("create live drop descriptor"));
let live_drop_weak = Arc::downgrade(&live_drop_file);
let shutdown_file = Arc::new(tempfile::tempfile().expect("create shutdown descriptor"));
let shutdown_weak = Arc::downgrade(&shutdown_file);
let tracker_weak = Arc::downgrade(&cache.tracker);
let cloned_moka = cache.cache.clone();
cache.insert(key("v", "live-on-drop/part.1", false), live_drop_file).await;
cache.insert(key("v", "shutdown/part.1", false), shutdown_file).await;
cache.invalidate_under("v", "shutdown");
if let Some(handle) = cache.maintenance_task.take() {
handle.abort();
let _ = handle.await;
}
drop(cache);
assert!(
live_drop_weak.upgrade().is_none(),
"Drop for FdCache must retire live descriptors even if a cloned Moka handle survives"
);
assert!(
shutdown_weak.upgrade().is_none(),
"dropping cache when maintenance is aborted must still release descriptors"
);
assert!(
tracker_weak.upgrade().is_none(),
"dropping cache must release FdCacheTracker without reference cycles"
);
drop(cloned_moka);
// Short-TTL expired entries (including mixed expired and fresh entries) must
// be retired by invalidate_under and clear() even though Moka's iterator skips
// expired entries.
let ttl_cache = FdCache::with_ttl(std::time::Duration::from_millis(15));
let expired_file = Arc::new(tempfile::tempfile().expect("create short-ttl descriptor"));
let expired_weak = Arc::downgrade(&expired_file);
ttl_cache.insert(key("v", "ttl-obj/part.1", false), expired_file).await;
tokio::time::sleep(std::time::Duration::from_millis(30)).await;
let fresh_file = Arc::new(tempfile::tempfile().expect("create fresh descriptor"));
let fresh_weak = Arc::downgrade(&fresh_file);
ttl_cache.insert(key("v", "ttl-obj/part.2", false), fresh_file).await;
ttl_cache.invalidate_under("v", "ttl-obj");
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while expired_weak.upgrade().is_some() || fresh_weak.upgrade().is_some() {
tokio::task::yield_now().await;
}
})
.await
.expect("invalidate_under must retire both expired and fresh matching descriptors");
assert_eq!(ttl_cache.tracker.live_entries(), 0);
let clear_expired_file = Arc::new(tempfile::tempfile().expect("create clear short-ttl descriptor"));
let clear_expired_weak = Arc::downgrade(&clear_expired_file);
ttl_cache
.insert(key("v", "ttl-clear/part.1", false), clear_expired_file)
.await;
tokio::time::sleep(std::time::Duration::from_millis(30)).await;
let clear_live_file = Arc::new(tempfile::tempfile().expect("create clear live descriptor"));
let clear_live_weak = Arc::downgrade(&clear_live_file);
ttl_cache.insert(key("v", "ttl-clear/part.2", false), clear_live_file).await;
ttl_cache.clear();
assert!(
clear_expired_weak.upgrade().is_none() && clear_live_weak.upgrade().is_none(),
"clear() must synchronously retire both TTL-expired and live descriptors"
);
assert_eq!(ttl_cache.tracker.live_entries(), 0);
}
#[tokio::test]
async fn fd_cache_lockstep_oracle_equivalence_and_counter_conservation() {
use std::collections::HashMap;
use std::sync::atomic::Ordering;
let (_directory, file) = fixture();
let cache = FdCache::with_ttl(std::time::Duration::from_secs(3600));
let mut oracle: HashMap<FdKey, Arc<FdCacheEntry>> = HashMap::new();
let volumes = ["vol-a", "vol-b", "vol-c", ".rustfs.sys/multipart"];
let paths = [
"a/b",
"a/b/part.1",
"a/b/dir/part.2",
"a/bc",
"a/bc/part.1",
"prefix/obj-0001.bin/0198a3f0/part.1",
"prefix/obj-0002.bin/0198a3f0/part.1",
"d1/d2/d3/d4/d5/d6/d7/d8/d9/d10/d11/d12/d13/part.1",
];
let prefixes = [
"a",
"a/",
"a/b",
"a/b/",
"a/b/dir",
"a/bc",
"prefix",
"prefix/obj-0001.bin",
"prefix/obj-0001.bin/0198a3f0",
"d1/d2/d3/d4/d5/c_missing",
"d1/d2/d3/d4/d5/d6/d7/d8/d9/d10/d11/d12/d13",
"nonexistent/prefix",
];
let mut candidate_keys = Vec::new();
for v in volumes {
for p in paths {
for direct in [false, true] {
candidate_keys.push(key(v, p, direct));
}
}
}
let mut rng = 0x9e37_79b9_7f4a_7c15u64;
let mut next_u64 = || {
rng ^= rng << 13;
rng ^= rng >> 7;
rng ^= rng << 17;
rng
};
for step in 0..1000 {
let op = next_u64() % 100;
if op < 45 {
// Insert or replace (either fresh or stale generation)
let k = candidate_keys[(next_u64() as usize) % candidate_keys.len()].clone();
let entry = Arc::new(FdCacheEntry {
file: Arc::clone(&file),
len: step,
invalidated: AtomicBool::new(false),
});
let cur_gen = cache.generation();
let stale = (next_u64() % 5) == 0 && cur_gen > 0;
let gen_to_use = if stale { cur_gen - 1 } else { cur_gen };
cache.insert_if_fresh(k.clone(), Arc::clone(&entry), gen_to_use).await;
if !stale {
oracle.insert(k, entry);
}
} else if op < 65 {
// Exact invalidation
let v = volumes[(next_u64() as usize) % volumes.len()];
let p = paths[(next_u64() as usize) % paths.len()];
cache.invalidate_exact(v, p).await;
oracle.remove(&key(v, p, false));
oracle.remove(&key(v, p, true));
} else if op < 88 {
// Prefix invalidation
let v = volumes[(next_u64() as usize) % volumes.len()];
let raw_prefix = prefixes[(next_u64() as usize) % prefixes.len()];
cache.invalidate_under(v, raw_prefix);
let trimmed = raw_prefix.trim_end_matches('/');
oracle.retain(|k, _| {
!(k.volume == v && (k.path == trimmed || k.path.strip_prefix(trimmed).is_some_and(|r| r.starts_with('/'))))
});
} else if op < 97 {
// Volume invalidation
let v = volumes[(next_u64() as usize) % volumes.len()];
cache.invalidate_volume(v);
oracle.retain(|k, _| k.volume != v);
} else {
// Full clear
cache.clear();
oracle.clear();
}
// Verify lockstep equivalence against the reference oracle after every operation.
for k in &candidate_keys {
let got = cache.get(k).await;
let expected = oracle.get(k);
match (got, expected) {
(Some(g), Some(e)) => assert!(
Arc::ptr_eq(&g, e),
"step {step}: pointer mismatch for key ({}, {}, {})",
k.volume,
k.path,
k.direct
),
(None, None) => {}
(g, e) => panic!(
"step {step}: oracle mismatch for key ({}, {}, {}): got={}, expected={}",
k.volume,
k.path,
k.direct,
g.is_some(),
e.is_some()
),
}
}
}
// Clear and wait for maintenance to drain all entries, then verify exact counter conservation.
cache.clear();
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while cache.tracker.live_entries() != 0 || cache.tracker.has_pending_invalidations.load(Ordering::SeqCst) {
tokio::task::yield_now().await;
}
})
.await
.expect("clear must drain all live entries and pending invalidations");
assert_eq!(cache.tracker.live_entries(), 0);
assert!(cache.tracker.unindexed_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.volume_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.exact_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.prefix_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.prefix_shards.iter().all(|s| s.lock().is_empty()));
assert!(cache.tracker.entry_shards.iter().all(|s| s.lock().is_empty()));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn fd_cache_concurrent_insert_and_invalidation_stress() {
use std::sync::atomic::{AtomicU64, Ordering};
let (_directory, file) = fixture();
let cache = Arc::new(FdCache::new());
// Deterministic Dekker-partner interleaving check: pause `CachedFdEntry::new`
// at `entry_shards[shard].lock()` after it has passed the initial generation
// check and incremented `volume_counts`, bump generation via an unmatched
// invalidation, and verify the post-`new()` recheck refuses publication and
// preserves the existing entry.
let seam_key = key("v-seam", "obj-1/part.1", false);
cache.insert(seam_key.clone(), Arc::clone(&file)).await;
let initial_entry = cache.get(&seam_key).await.expect("initial entry must exist");
let (vol_hash, vol_slot) = super::fd_cache_volume_hash_and_slot("v-seam");
let entry_shard = (vol_hash as usize) & (super::FD_CACHE_PREFIX_SHARDS - 1);
let stale_gen = cache.generation();
let stale_entry = Arc::new(FdCacheEntry {
file: Arc::clone(&file),
len: 999_999,
invalidated: AtomicBool::new(false),
});
let blocked_insert = {
let hold = cache.tracker.entry_shards[entry_shard].lock();
let cache_clone = Arc::clone(&cache);
let seam_key_clone = seam_key.clone();
let task = tokio::spawn(async move {
cache_clone.insert_if_fresh(seam_key_clone, stale_entry, stale_gen).await;
});
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
while cache.tracker.volume_counts[usize::from(vol_slot)].load(Ordering::SeqCst) < 2 {
assert!(
std::time::Instant::now() < deadline,
"insert never reached CachedFdEntry::new with the previous entry still live"
);
std::thread::yield_now();
}
cache.invalidate_under("v-seam", "unmatched-prefix");
drop(hold);
task
};
blocked_insert.await.expect("blocked insert task should complete");
let preserved = cache
.get(&seam_key)
.await
.expect("refused stale insert must not retire a valid previous entry");
assert!(Arc::ptr_eq(&preserved, &initial_entry));
cache.invalidate_volume("v-seam");
let floor_v1 = Arc::new(AtomicU64::new(0));
let floor_v2 = Arc::new(AtomicU64::new(0));
let mut handles = Vec::new();
for t in 0..8usize {
let cache = Arc::clone(&cache);
let file = Arc::clone(&file);
let floor_v1 = Arc::clone(&floor_v1);
let floor_v2 = Arc::clone(&floor_v2);
handles.push(tokio::spawn(async move {
for i in 0..250u64 {
let is_v1 = (t + i as usize).is_multiple_of(2);
let vol = if is_v1 { "v1" } else { "v2" };
let path = format!("dir-{}/obj-{}/part.1", t % 4, i % 16);
let k = key(vol, &path, i.is_multiple_of(2));
let g = cache.generation();
let entry = Arc::new(FdCacheEntry {
file: Arc::clone(&file),
len: g,
invalidated: AtomicBool::new(false),
});
cache.insert_if_fresh(k.clone(), entry, g).await;
let min_gen = if is_v1 {
floor_v1.load(Ordering::SeqCst)
} else {
floor_v2.load(Ordering::SeqCst)
};
if let Some(hit) = cache.get(&k).await {
assert!(
hit.len >= min_gen,
"stale descriptor served after volume invalidation: opened_gen={}, floor_gen={min_gen}",
hit.len
);
}
}
}));
}
for t in 0..4usize {
let cache = Arc::clone(&cache);
let floor_v1 = Arc::clone(&floor_v1);
let floor_v2 = Arc::clone(&floor_v2);
handles.push(tokio::spawn(async move {
for i in 0..250u64 {
let is_v1 = (t + i as usize).is_multiple_of(2);
let vol = if is_v1 { "v1" } else { "v2" };
match i % 4 {
0 => cache.invalidate_under(vol, &format!("dir-{}/obj-{}", t % 4, i % 16)),
1 => {
let p = format!("dir-{}/obj-{}/part.1", t % 4, i % 16);
cache.invalidate_exact(vol, &p).await;
}
2 => {
let before = cache.generation();
cache.invalidate_volume(vol);
let floor = before + 1;
if is_v1 {
floor_v1.fetch_max(floor, Ordering::SeqCst);
} else {
floor_v2.fetch_max(floor, Ordering::SeqCst);
}
}
_ => cache.invalidate_under("uncached-vol", "uncached/prefix"),
}
tokio::task::yield_now().await;
}
}));
}
for h in handles {
h.await.expect("worker task should succeed");
}
cache.clear();
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while cache.tracker.live_entries() != 0 || cache.tracker.has_pending_invalidations.load(Ordering::SeqCst) {
tokio::task::yield_now().await;
}
})
.await
.expect("clear after concurrent stress must drain all live entries");
assert_eq!(cache.tracker.live_entries(), 0);
assert!(cache.tracker.unindexed_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.volume_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.exact_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.prefix_counts.iter().all(|c| c.load(Ordering::SeqCst) == 0));
assert!(cache.tracker.prefix_shards.iter().all(|s| s.lock().is_empty()));
assert!(cache.tracker.entry_shards.iter().all(|s| s.lock().is_empty()));
}
File diff suppressed because it is too large Load Diff
+3
View File
@@ -486,6 +486,9 @@ impl LocalDisk {
state.preflight_rejection = Some(LocalRenamePreflightRejection(()));
})?;
crate::hp_guard!("LocalDisk::rename_data");
if dst_volume == super::super::RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_info(dst_path, &fi).await?;
}
#[cfg(feature = "e2e-test-hooks")]
if fi.is_healing() {
wait_for_heal_commit_test_barrier(&self.root, dst_volume, dst_path).await?;
+530 -26
View File
@@ -5,6 +5,28 @@ use crate::disk::{
};
use sha2::{Digest, Sha256};
pub(super) struct RecoveryOperationScope {
pub(super) budget: Arc<crate::object_api::DeleteRecoveryWorkBudget>,
may_publish_receipt: bool,
receipt_published: std::sync::atomic::AtomicBool,
}
tokio::task_local! {
static RECOVERY_OPERATION: Arc<RecoveryOperationScope>;
}
pub(super) fn recovery_operation_scope() -> Option<Arc<RecoveryOperationScope>> {
RECOVERY_OPERATION.try_with(Arc::clone).ok()
}
impl RecoveryOperationScope {
pub(super) fn receipt_published(&self) {
if self.may_publish_receipt {
self.receipt_published.store(true, std::sync::atomic::Ordering::Release);
}
}
}
pub(super) const PREPARED: &str = "delete-prepared";
pub(super) const TARGET: &str = "delete-target";
pub(super) const FINISHED: &str = "delete-commit-finished";
@@ -21,22 +43,27 @@ struct DiscoverySession {
impl DiscoverySession {
/// Consume one bounded step, returning ownership only when traversal remains.
fn advance(mut self, limit: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>)> {
fn advance(self, limit: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>)> {
self.advance_bounded(limit, super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)
.map(|(transactions, next, _)| (transactions, next))
}
fn advance_bounded(mut self, limit: u32, max_entries: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>, u32)> {
if self.frames.is_empty() {
match std::fs::read_dir(&self.root) {
Ok(entries) => self.frames.push((String::new(), entries)),
Err(err) if err.kind() == ErrorKind::NotFound => {
return Ok((Vec::new(), None));
return Ok((Vec::new(), None, 0));
}
Err(err) => return Err(err),
}
}
let mut transactions = BTreeSet::new();
let mut seen_candidates = HashSet::new();
let mut visited = 0usize;
let mut visited = 0u32;
while !self.frames.is_empty()
&& transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))?
&& visited < 4096
&& visited < max_entries
{
let (parent, entries) = self
.frames
@@ -156,7 +183,7 @@ impl DiscoverySession {
.map(|(object, transaction)| DeleteTransactionId { object, transaction })
.collect();
let next = (!self.frames.is_empty()).then_some(self);
Ok((transactions, next))
Ok((transactions, next, visited))
}
async fn has_evidence(mut self, disk: &LocalDisk, path: &Path) -> Result<bool> {
@@ -580,12 +607,41 @@ fn explicit_delete_referenced_data_dirs(
Ok(Some(referenced))
}
fn explicit_delete_plan(
pub(super) async fn next_recovery_entry(entries: &mut tokio::fs::ReadDir) -> std::io::Result<Option<tokio::fs::DirEntry>> {
if let Some(scope) = recovery_operation_scope() {
scope.budget.reserve(1).map_err(std::io::Error::from)?;
}
entries.next_entry().await
}
pub(super) fn recovery_entries(
mut entries: std::fs::ReadDir,
budget: Option<&crate::object_api::DeleteRecoveryWorkBudget>,
) -> impl Iterator<Item = std::io::Result<std::fs::DirEntry>> + '_ {
let mut finished = false;
std::iter::from_fn(move || {
if finished {
return None;
}
if let Some(budget) = budget
&& let Err(error) = budget.reserve(1)
{
finished = true;
return Some(Err(error.into()));
}
let entry = entries.next();
finished = entry.is_none();
entry
})
}
fn explicit_delete_plan_bounded(
root: &Path,
transaction: Uuid,
original: Option<&FileMeta>,
journal: Option<&DeleteRecoveryJournal>,
current_missing: bool,
budget: Option<&crate::object_api::DeleteRecoveryWorkBudget>,
) -> Result<Option<ExplicitDeletePlan>> {
let Some(referenced) = explicit_delete_referenced_data_dirs(original, journal, current_missing)? else {
return Ok(None);
@@ -602,7 +658,7 @@ fn explicit_delete_plan(
let mut candidates = Vec::new();
match std::fs::read_dir(&rollback) {
Ok(entries) => {
for entry in entries {
for entry in recovery_entries(entries, budget) {
let entry = entry.map_err(to_file_error)?;
let kind = entry.file_type().map_err(to_file_error)?;
let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?;
@@ -634,7 +690,7 @@ fn explicit_delete_plan(
}
match std::fs::read_dir(root) {
Ok(entries) => {
for entry in entries {
for entry in recovery_entries(entries, budget) {
let entry = entry.map_err(to_file_error)?;
let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?;
let Some(id) = canonical_uuid(&name).filter(|id| !id.is_nil()) else {
@@ -658,7 +714,7 @@ fn explicit_delete_plan(
candidates.dedup();
for dir in candidates {
let mut has_parts = false;
for entry in std::fs::read_dir(&dir).map_err(to_file_error)? {
for entry in recovery_entries(std::fs::read_dir(&dir).map_err(to_file_error)?, budget) {
let entry = entry.map_err(to_file_error)?;
if !entry.file_type().map_err(to_file_error)?.is_file() {
return Ok(None);
@@ -697,15 +753,27 @@ fn canonical_uuid(name: &str) -> Option<Uuid> {
Uuid::parse_str(name).ok().filter(|id| name == id.hyphenated().to_string())
}
pub(super) fn inspection_with_markers(
pub(super) fn missing_marker_preserves_child_prefix(journal: &DeleteRecoveryJournal) -> bool {
journal.original_digest.is_none()
&& journal.original_versions.is_empty()
&& journal.data_dirs.is_empty()
&& journal.target_digest.is_some()
&& !journal.requests.is_empty()
&& journal
.requests
.iter()
.all(|request| request.deleted && request.version_id.is_some_and(|id| !id.is_nil()))
}
pub(super) fn inspection_with_markers_bounded(
root: &Path,
volume: &str,
object: &str,
transaction: Uuid,
budget: Option<&crate::object_api::DeleteRecoveryWorkBudget>,
) -> Result<(DeleteRecoveryInspection, bool)> {
let rollback = root.join(transaction.to_string());
let before = regular_metadata_bytes(&rollback.join(STORAGE_FORMAT_FILE_BACKUP))?;
let current = regular_metadata_bytes(&root.join(STORAGE_FORMAT_FILE))?;
let prepared = regular_bytes(&rollback.join(PREPARED))?;
let target = regular_metadata_bytes(&rollback.join(TARGET))?;
let receipt_bytes = regular_bytes(&rollback.join(DELETE_QUORUM_COMMITTED_FILE))?;
@@ -739,6 +807,20 @@ pub(super) fn inspection_with_markers(
{
return Err(DiskError::FileCorrupt);
}
// A failed creation of a marker over a child named `xl.meta` did not
// replace that directory. Older releases nevertheless left a journal.
// Only this bound, originally absent generation may treat it as absence.
let current_path = root.join(STORAGE_FORMAT_FILE);
let current = if !committed
&& before.is_none()
&& (journal.as_ref().is_some_and(missing_marker_preserves_child_prefix)
|| (journal.is_none() && target.is_none() && receipt_bytes.is_none()))
&& std::fs::symlink_metadata(&current_path).is_ok_and(|metadata| metadata.is_dir())
{
None
} else {
regular_metadata_bytes(&current_path)?
};
let backup_digest = before.as_deref().map(digest);
let current_digest = current.as_deref().map(digest);
let prepared_digest = prepared.as_deref().map(digest);
@@ -781,7 +863,7 @@ pub(super) fn inspection_with_markers(
Err(err) => return Err(to_file_error(err).into()),
};
if let Some(entries) = entries {
for entry in entries {
for entry in recovery_entries(entries, budget) {
let entry = entry.map_err(to_file_error)?;
let kind = entry.file_type().map_err(to_file_error)?;
if !kind.is_file() && !kind.is_dir() {
@@ -798,7 +880,7 @@ pub(super) fn inspection_with_markers(
descriptions.push((name.clone(), kind.is_dir(), entry.metadata().map_err(to_file_error)?.len()));
if kind.is_dir() && id.is_some_and(|id| !id.is_nil()) {
has_other_uuid_dir |= id != Some(transaction);
for child in std::fs::read_dir(entry.path()).map_err(to_file_error)? {
for child in recovery_entries(std::fs::read_dir(entry.path()).map_err(to_file_error)?, budget) {
let child = child.map_err(to_file_error)?;
let kind = child.file_type().map_err(to_file_error)?;
if !kind.is_dir() && !kind.is_file() {
@@ -844,7 +926,7 @@ pub(super) fn inspection_with_markers(
}
if name == transaction.to_string() && kind.is_dir() {
has_staged_dir = true;
for file in std::fs::read_dir(child.path()).map_err(to_file_error)? {
for file in recovery_entries(std::fs::read_dir(child.path()).map_err(to_file_error)?, budget) {
let file = file.map_err(to_file_error)?;
let kind = file.file_type().map_err(to_file_error)?;
let file_name = file.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?;
@@ -968,7 +1050,8 @@ pub(super) fn inspection_with_markers(
.is_some_and(|referenced| !referenced.contains(&transaction))
}
DeleteRecoveryState::LegacyAmbiguous | DeleteRecoveryState::NewPrepared => {
explicit_delete_plan(root, transaction, before_meta.as_ref(), journal.as_ref(), current.is_none())?.is_some()
explicit_delete_plan_bounded(root, transaction, before_meta.as_ref(), journal.as_ref(), current.is_none(), budget)?
.is_some()
}
_ => false,
};
@@ -1039,6 +1122,7 @@ impl LocalDisk {
let volume = volume.to_owned();
let object = object.to_owned();
let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner).await;
let recovery_scope = recovery_operation_scope();
os::run_blocking_namespace_operation(lease, move || {
Ok((|| {
let entries = match std::fs::read_dir(&root) {
@@ -1048,7 +1132,7 @@ impl LocalDisk {
};
let mut predecessors = BTreeSet::new();
let mut visited = 0usize;
for entry in entries {
for entry in recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())) {
visited += 1;
if visited > 65536 {
return Err(DiskError::DiskOngoingReq);
@@ -1201,6 +1285,42 @@ impl LocalDisk {
.map_err(|error| to_file_error(error).into())
}
#[cfg(test)]
pub(crate) fn public_delete_discovery_sessions_for_test(&self) -> usize {
DISCOVERY_SESSIONS
.lock()
.values()
.filter(|session| session.disk_root == self.io_root())
.count()
}
pub(super) async fn delete_recovery_discover_bounded(
&self,
volume: &str,
object: &str,
max_entries: u32,
) -> Result<super::super::BoundedDeleteRecoveryDiscovery> {
if !(1..=super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES).contains(&max_entries) {
return Err(DiskError::InvalidPath);
}
let path = self.delete_discovery_path(volume, Some(object))?;
let session = self.new_delete_discovery_session(volume, Some(object), &path)?;
let lease = os::acquire_namespace_mutation_lease_with_owner(&path, None).await;
self.validate_delete_discovery_owner(&session)?;
// The blocking owner drops every directory handle even if the RPC waiter disappears.
// A partial inventory is never evidence of absence and never enters DISCOVERY_SESSIONS.
os::run_blocking_namespace_operation(lease, move || {
let (transactions, next, visited) = session.advance_bounded(256, max_entries)?;
Ok(super::super::BoundedDeleteRecoveryDiscovery {
transactions: if next.is_none() { transactions } else { Vec::new() },
visited,
complete: next.is_none(),
})
})
.await
.map_err(|error| to_file_error(error).into())
}
pub(super) async fn delete_recovery_discover(
&self,
volume: &str,
@@ -1248,6 +1368,39 @@ impl LocalDisk {
.await
}
pub(super) async fn delete_recovery_inspect_bounded(
&self,
volume: &str,
object: &str,
transaction: Uuid,
max_entries: u32,
) -> Result<super::super::BoundedDeleteRecoveryInspection> {
if !safe_path(object, false)
|| transaction.is_nil()
|| !(1..=super::super::DELETE_RECOVERY_MAX_INSPECTION_ENTRIES).contains(&max_entries)
{
return Err(DiskError::InvalidPath);
}
let root = self.get_object_path(volume, object)?;
let owner = os::acquire_metadata_mutation_lease(&root, None).await;
let lease = os::acquire_namespace_mutation_lease_with_owner(&root, Some(owner)).await;
let volume = volume.to_owned();
let object = object.to_owned();
os::run_blocking_namespace_operation(lease, move || {
let budget = crate::object_api::DeleteRecoveryWorkBudget::new(max_entries);
Ok(
inspection_with_markers_bounded(&root, &volume, &object, transaction, Some(&budget)).map(|(inspection, _)| {
super::super::BoundedDeleteRecoveryInspection {
inspection,
visited: max_entries - budget.remaining(),
}
}),
)
})
.await
.map_err(to_file_error)?
}
pub(super) async fn delete_recovery_inspect(
&self,
volume: &str,
@@ -1287,9 +1440,22 @@ impl LocalDisk {
let volume = volume.to_owned();
let object = object.to_owned();
let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner).await;
os::run_blocking_namespace_operation(lease, move || Ok(inspection_with_markers(&root, &volume, &object, transaction)))
.await
.map_err(to_file_error)?
let scope = recovery_operation_scope();
// Journal inspections ride the durable-mutation lane (the fused delete
// prepare already does): a bulk delete publishes one transaction per
// key, and the one-worker foreground namespace lane would serialize
// those per-drive publishes behind every other mutation.
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
Ok(inspection_with_markers_bounded(
&root,
&volume,
&object,
transaction,
scope.as_ref().map(|scope| scope.budget.as_ref()),
))
})
.await
.map_err(to_file_error)?
}
async fn activate_delete_preparation(
@@ -1487,6 +1653,49 @@ impl LocalDisk {
self.activate_delete_preparation(&journal, original, target, owner).await
}
pub(super) async fn delete_recovery_settle_bounded(
&self,
volume: &str,
request: super::super::BoundedDeleteRecoverySettlement,
owner: Option<Arc<dyn Send + Sync>>,
) -> Result<super::super::BoundedDeleteRecoverySettlementReply> {
if !(1..=super::super::DELETE_RECOVERY_MAX_INSPECTION_ENTRIES).contains(&request.max_entries) {
return Err(DiskError::InvalidPath);
}
let scope = Arc::new(RecoveryOperationScope {
budget: Arc::new(crate::object_api::DeleteRecoveryWorkBudget::new(request.max_entries)),
may_publish_receipt: matches!(
&request.action,
DeleteRecoveryAction::PublishCommit | DeleteRecoveryAction::AdoptCommit { .. }
),
receipt_published: std::sync::atomic::AtomicBool::new(false),
});
let result = RECOVERY_OPERATION
.scope(
Arc::clone(&scope),
self.delete_recovery_settle(
volume,
&request.object,
request.transaction,
request.expected_fingerprint,
request.action,
owner,
),
)
.await;
let outcome = match result {
Ok(inspection) => super::super::DeleteRecoverySettlementOutcome::Inspected(Box::new(inspection)),
Err(DiskError::DeleteRecoveryWorkLimit) if scope.receipt_published.load(std::sync::atomic::Ordering::Acquire) => {
super::super::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred
}
Err(error) => return Err(error),
};
Ok(super::super::BoundedDeleteRecoverySettlementReply {
outcome,
visited: request.max_entries - scope.budget.remaining(),
})
}
pub(crate) async fn delete_recovery_settle(
&self,
volume: &str,
@@ -1685,14 +1894,26 @@ impl LocalDisk {
let plan_root = root.clone();
let missing = observed.current_digest.is_none();
let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner.clone()).await;
let plan = os::run_blocking_namespace_operation(lease, move || {
let scope = recovery_operation_scope();
// Same durable-mutation lane as the journal inspection above: this
// plan closure runs inside every settle on the bulk-delete publish
// path.
let plan = os::run_blocking_namespace_operation_with_sync(lease, true, move || {
Ok((|| {
let before =
regular_metadata_bytes(&plan_root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP))?
.as_deref()
.map(FileMeta::load)
.transpose()?;
explicit_delete_plan(&plan_root, transaction, before.as_ref(), None, missing)?.ok_or(DiskError::FileCorrupt)
explicit_delete_plan_bounded(
&plan_root,
transaction,
before.as_ref(),
None,
missing,
scope.as_ref().map(|scope| scope.budget.as_ref()),
)?
.ok_or(DiskError::FileCorrupt)
})())
})
.await
@@ -1957,6 +2178,7 @@ impl LocalDisk {
let (rollback_buf, root_buf, volume_str, object_str) =
(rollback.clone(), root.clone(), volume.to_owned(), object.to_owned());
let publication_root = self.publication_root.clone();
let scope = recovery_operation_scope();
return os::run_blocking_namespace_operation_with_sync(receipt_lease, sync, move || {
let _publication_root = publication_root;
for (path, _) in &file_leases {
@@ -1989,8 +2211,14 @@ impl LocalDisk {
}
}
}
let inspected =
inspection_with_markers(&root_buf, &volume_str, &object_str, transaction).map(|(state, _)| state);
let inspected = inspection_with_markers_bounded(
&root_buf,
&volume_str,
&object_str,
transaction,
scope.as_ref().map(|scope| scope.budget.as_ref()),
)
.map(|(state, _)| state);
drop((file_leases, prune_dirs));
Ok(inspected)
})
@@ -2147,6 +2375,7 @@ impl LocalDisk {
let volume_str = volume.to_owned();
let object_str = object.to_owned();
let recovery_scope = recovery_operation_scope();
let res = os::run_blocking_namespace_operation_with_sync(
lease,
sync,
@@ -2193,7 +2422,9 @@ impl LocalDisk {
let mut predecessors = BTreeSet::new();
let mut sole_transaction = true;
if let Ok(entries) = std::fs::read_dir(&file_path) {
for (visited, entry) in entries.enumerate() {
for (visited, entry) in
recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())).enumerate()
{
if visited >= 65536 {
return Err(DiskError::DiskOngoingReq);
}
@@ -2406,12 +2637,25 @@ impl LocalDisk {
Ok(Some(()))
}
#[cfg(test)]
pub(crate) async fn try_publish_prepared_delete_local(
&self,
volume: &str,
object: &str,
transaction: Uuid,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
self.try_publish_prepared_delete_local_with_budget(volume, object, transaction, namespace_owner, None)
.await
}
pub(crate) async fn try_publish_prepared_delete_local_with_budget(
&self,
volume: &str,
object: &str,
transaction: Uuid,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) || transaction.is_nil() {
@@ -2490,7 +2734,7 @@ impl LocalDisk {
if sole_transaction {
match std::fs::read_dir(&root) {
Ok(entries) => {
for entry in entries {
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let Ok(name) = entry.file_name().into_string() else {
sole_transaction = false;
@@ -2538,6 +2782,7 @@ impl LocalDisk {
.map_err(to_file_error)?
}
#[cfg(test)]
pub(crate) async fn try_finish_and_finalize_prepared_delete_local(
&self,
volume: &str,
@@ -2545,6 +2790,26 @@ impl LocalDisk {
transaction: Uuid,
journal: &DeleteRecoveryJournal,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<bool> {
self.try_finish_and_finalize_prepared_delete_local_with_budget(
volume,
object,
transaction,
journal,
namespace_owner,
None,
)
.await
}
pub(crate) async fn try_finish_and_finalize_prepared_delete_local_with_budget(
&self,
volume: &str,
object: &str,
transaction: Uuid,
journal: &DeleteRecoveryJournal,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<bool> {
fn ignore_not_found(res: std::io::Result<()>) -> std::io::Result<()> {
match res {
@@ -2611,7 +2876,7 @@ impl LocalDisk {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in entries {
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
@@ -4115,6 +4380,100 @@ mod tests {
);
}
#[tokio::test]
async fn delete_marker_collision_rejects_new_journals_and_recovers_legacy_proof() {
let (temp, disk, _info, original) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object path");
let staged_child = temp.path().join("child-fixture");
fs::rename(&root, &staged_child).await.expect("move intact object");
fs::create_dir(&root).await.expect("create parent prefix");
fs::rename(staged_child, root.join(STORAGE_FORMAT_FILE))
.await
.expect("seed an independent object named object/xl.meta");
let transaction = Uuid::new_v4();
let marker = FileInfo {
volume: "recovery".to_owned(),
name: "object".to_owned(),
version_id: Some(Uuid::new_v4()),
deleted: true,
mod_time: Some(OffsetDateTime::now_utc()),
..Default::default()
};
disk.delete_version_recoverable(
"recovery",
"object",
marker.clone(),
true,
DeleteOptions {
old_data_dir: Some(transaction),
..Default::default()
},
)
.await
.expect_err("a child xl.meta directory cannot be replaced by a marker");
assert!(!root.join(transaction.to_string()).exists(), "rejection must precede journal publication");
// Reproduce the durable evidence written by releases that prepared a
// missing-object marker before attempting to replace the directory.
disk.prepare_delete_recovery(
"recovery",
"object",
None,
&[marker],
None,
DeletePreparationContext {
transaction,
bucket_incarnation: None,
namespace_owner: None,
},
)
.await
.expect("seed legacy prepared marker");
let journal_path = root.join(transaction.to_string()).join(PREPARED);
let journal_bytes = fs::read(&journal_path).await.expect("prepared journal");
let mut incompatible: DeleteRecoveryJournal = serde_json::from_slice(&journal_bytes).expect("journal");
incompatible.original_digest = Some([7; 32]);
fs::write(&journal_path, serde_json::to_vec(&incompatible).expect("serialize journal"))
.await
.expect("write incompatible evidence");
assert!(
disk.inspect_delete_transaction("recovery", "object", transaction)
.await
.is_err(),
"a journal that claims existing metadata cannot ignore a directory collision"
);
fs::write(&journal_path, journal_bytes)
.await
.expect("restore exact legacy proof");
let observed = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("inspect legacy proof");
assert_eq!(observed.state, DeleteRecoveryState::NewPrepared);
let recovered = disk
.settle_delete_transaction("recovery", "object", transaction, observed.fingerprint, DeleteRecoveryAction::Rollback)
.await
.expect("rollback legacy marker without unlinking the child directory");
assert_eq!(recovered.state, DeleteRecoveryState::Absent);
assert!(!root.join(transaction.to_string()).exists());
assert_eq!(
disk.read_all("recovery", "object/xl.meta/xl.meta")
.await
.expect("child metadata")
.as_ref(),
original
);
assert!(
disk.read_version("", "recovery", "object/xl.meta", "", &Default::default())
.await
.is_ok()
);
assert!(
discover_all_keys(&disk, Some("object")).await.is_empty(),
"no proof may keep replacement heal pending"
);
}
// Data movement copies a delete marker onto a target that holds no
// metadata yet. Rolling back that sub-quorum write removes the marker it
// published, and must not report the backup it never wrote as missing:
@@ -5758,6 +6117,151 @@ mod tests {
);
}
#[tokio::test]
async fn delete_recovery_bounded_inspection_charges_root_and_uuid_children() {
let (_temp, disk, _, _) = fixture().await;
let transaction = Uuid::new_v4();
apply(&disk, transaction).await;
let full = disk
.inspect_delete_transaction_bounded("recovery", "object", transaction, 65536)
.await
.expect("full bounded inspection");
assert!(full.visited > 1, "both root and transaction directory entries are charged");
let error = disk
.inspect_delete_transaction_bounded("recovery", "object", transaction, full.visited - 1)
.await
.expect_err("nested entry exceeds allowance");
assert_eq!(error, DiskError::DeleteRecoveryWorkLimit);
let unchanged = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("unchanged proof");
assert_eq!(full.inspection.fingerprint, unchanged.fingerprint);
}
#[tokio::test]
async fn delete_recovery_receipt_publication_distinguishes_deferred_inspection_from_write_failure() {
for write_fails in [false, true] {
let (_temp, disk, _, _) = fixture().await;
let transaction = Uuid::new_v4();
apply(&disk, transaction).await;
let full = disk
.inspect_delete_transaction_bounded("recovery", "object", transaction, 65536)
.await
.expect("measure preparation work");
let receipt = disk
.io_get_object_path("recovery", &format!("object/{transaction}/{DELETE_QUORUM_COMMITTED_FILE}"))
.expect("receipt path");
if write_fails {
let destination = receipt.clone();
set_owned_file_before_open(&receipt, move || std::fs::create_dir(&destination).expect("reject receipt rename"));
}
let result = disk
.settle_delete_transaction_bounded(
"recovery",
super::super::super::BoundedDeleteRecoverySettlement {
object: "object".to_owned(),
transaction,
expected_fingerprint: full.inspection.fingerprint,
action: DeleteRecoveryAction::PublishCommit,
max_entries: full.visited,
},
None,
)
.await;
if write_fails {
assert!(result.is_err(), "a failed receipt write never counts as publication");
assert!(receipt.is_dir());
} else {
let result = result.expect("persisted receipt is acknowledged independently of post-write inspection");
assert!(matches!(
result.outcome,
super::super::super::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred
));
assert_eq!(result.visited, full.visited);
let state = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("durable receipt");
assert_eq!(state.state, DeleteRecoveryState::QuorumCommitted);
}
}
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_refuses_partial_inventory_without_public_cursor() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("prefix")).expect("prefix path");
fs::create_dir_all(&path).await.expect("physical prefix");
for index in 0..64 {
fs::create_dir(path.join(format!("child-{index}")))
.await
.expect("legitimate child");
}
for _ in 0..140 {
let page = disk
.discover_delete_transactions_bounded("recovery", "prefix", 8)
.await
.expect("bounded probe");
assert_eq!(page.visited, 8);
assert!(!page.complete, "an empty partial page cannot establish absence");
assert!(page.transactions.is_empty());
}
let page = disk
.discover_delete_transactions_bounded("recovery", "prefix", 128)
.await
.expect("complete probe");
assert!(page.complete);
assert_eq!(page.visited, 64);
assert!(page.transactions.is_empty());
let transaction = Uuid::new_v4();
apply(&disk, transaction).await;
let before = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("before probe");
let page = disk
.discover_delete_transactions_bounded("recovery", "object", 4096)
.await
.expect("complete evidence");
assert!(page.complete);
assert!(page.transactions.iter().any(|id| id.transaction == transaction));
let after = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("after probe");
assert_eq!(before.fingerprint, after.fingerprint);
assert!(
DISCOVERY_SESSIONS
.lock()
.values()
.all(|session| session.disk_root != disk.io_root())
);
}
#[tokio::test]
async fn delete_recovery_cancelled_bounded_discovery_drops_private_session() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("object")).expect("probe scope");
let lease = os::acquire_namespace_mutation_lease_with_owner(&path, None).await;
let mut probe = Box::pin(disk.discover_delete_transactions_bounded("recovery", "object", 4096));
assert!(futures::poll!(probe.as_mut()).is_pending());
drop(probe);
assert!(
DISCOVERY_SESSIONS
.lock()
.values()
.all(|session| session.disk_root != disk.io_root())
);
drop(lease);
assert!(
disk.discover_delete_transactions_bounded("recovery", "object", 4096)
.await
.expect("subsequent probe")
.complete
);
}
#[tokio::test]
async fn delete_recovery_evidence_probes_do_not_register_public_cursors() {
let (_temp, disk, _, _) = fixture().await;
File diff suppressed because it is too large Load Diff
@@ -117,6 +117,7 @@ pub(super) struct LoadedSource {
}
pub(super) struct SourceStore {
multipart_index: Arc<super::multipart_index::IndexOwner>,
pub disk_id: Uuid,
pub physical_slot: usize,
pub disk_ids: Vec<Uuid>,
@@ -186,6 +187,7 @@ impl LocalDisk {
let logical = self.get_object_path(RUSTFS_META_MULTIPART_BUCKET, path)?;
let lease = os::acquire_namespace_mutation_lease_with_owner(&logical, owner.clone()).await;
Ok(SourceStore {
multipart_index: self.multipart_index.clone(),
disk_id,
physical_slot,
disk_ids,
@@ -755,6 +757,9 @@ impl SourceStore {
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
self.multipart_index
.record_head(&format!("{}/xl.meta", self.path), bytes)
.await?;
invalidate_active_multipart_source_head(&self.source);
let path = self.source.join(STORAGE_FORMAT_FILE);
let base = self.base.clone();
+170 -4
View File
@@ -27,6 +27,7 @@ mod io_admission;
#[cfg(test)]
pub(crate) mod lifecycle_test_io;
pub mod local;
pub mod multipart_index;
pub mod multipart_source;
pub mod os;
@@ -414,10 +415,11 @@ impl Disk {
object: &str,
transaction: Uuid,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
match self {
Self::Local(disk) => {
disk.try_publish_prepared_delete_local(volume, object, transaction, namespace_owner)
disk.try_publish_prepared_delete_local(volume, object, transaction, namespace_owner, work_budget)
.await
}
Self::Remote(_) => Ok(None),
@@ -431,11 +433,19 @@ impl Disk {
transaction: Uuid,
journal: &DeleteRecoveryJournal,
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<bool> {
match self {
Self::Local(disk) => {
disk.try_finish_and_finalize_prepared_delete_local(volume, object, transaction, journal, namespace_owner)
.await
disk.try_finish_and_finalize_prepared_delete_local(
volume,
object,
transaction,
journal,
namespace_owner,
work_budget,
)
.await
}
Self::Remote(_) => Ok(false),
}
@@ -512,6 +522,17 @@ impl Disk {
#[async_trait::async_trait]
impl DiskAPI for Disk {
async fn list_multipart_index_roots(
&self,
bucket: &str,
marker: Option<&str>,
) -> Result<multipart_index::MultipartIndexPage> {
match self {
Self::Local(disk) => disk.list_multipart_index_roots(bucket, marker).await,
Self::Remote(disk) => disk.list_multipart_index_roots(bucket, marker).await,
}
}
async fn supports_guarded_delete_versions(&self) -> Result<()> {
match self {
Self::Local(disk) => disk.supports_guarded_delete_versions().await,
@@ -555,6 +576,17 @@ impl DiskAPI for Disk {
Self::Remote(disk) => disk.has_delete_transaction_evidence(volume, object).await,
}
}
async fn discover_delete_transactions_bounded(
&self,
volume: &str,
object: &str,
max_entries: u32,
) -> Result<BoundedDeleteRecoveryDiscovery> {
match self {
Self::Local(disk) => disk.discover_delete_transactions_bounded(volume, object, max_entries).await,
Self::Remote(disk) => disk.discover_delete_transactions_bounded(volume, object, max_entries).await,
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
@@ -567,6 +599,35 @@ impl DiskAPI for Disk {
Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
}
}
async fn settle_delete_transaction_bounded(
&self,
volume: &str,
request: BoundedDeleteRecoverySettlement,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<BoundedDeleteRecoverySettlementReply> {
match self {
Self::Local(disk) => disk.settle_delete_transaction_bounded(volume, request, namespace_owner).await,
Self::Remote(disk) => disk.settle_delete_transaction_bounded(volume, request, namespace_owner).await,
}
}
async fn inspect_delete_transaction_bounded(
&self,
volume: &str,
object: &str,
transaction: Uuid,
max_entries: u32,
) -> Result<BoundedDeleteRecoveryInspection> {
match self {
Self::Local(disk) => {
disk.inspect_delete_transaction_bounded(volume, object, transaction, max_entries)
.await
}
Self::Remote(disk) => {
disk.inspect_delete_transaction_bounded(volume, object, transaction, max_entries)
.await
}
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
@@ -1513,6 +1574,15 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
fn host_name(&self) -> String;
fn endpoint(&self) -> Endpoint;
async fn close(&self) -> Result<()>;
/// Return candidates only while this disk maintains complete boot-local coverage.
/// Unsupported or unready disks must never return an authoritative empty page.
async fn list_multipart_index_roots(
&self,
_bucket: &str,
_marker: Option<&str>,
) -> Result<multipart_index::MultipartIndexPage> {
Err(DiskError::MethodNotAllowed)
}
async fn get_disk_id(&self) -> Result<Option<Uuid>>;
async fn set_disk_id(&self, id: Option<Uuid>) -> Result<()>;
@@ -1690,6 +1760,33 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
) -> Result<DeleteRecoveryPage> {
Err(DiskError::MethodNotAllowed)
}
/// Bounded foreground inventory owns all traversal state until return or cancellation.
/// Unsupported peers must fail closed; callers must never fall back to paginated discovery.
async fn discover_delete_transactions_bounded(
&self,
_volume: &str,
_object: &str,
_max_entries: u32,
) -> Result<BoundedDeleteRecoveryDiscovery> {
Err(DiskError::DeleteRecoveryWorkLimit)
}
async fn inspect_delete_transaction_bounded(
&self,
_volume: &str,
_object: &str,
_transaction: Uuid,
_max_entries: u32,
) -> Result<BoundedDeleteRecoveryInspection> {
Err(DiskError::DeleteRecoveryWorkLimit)
}
async fn settle_delete_transaction_bounded(
&self,
_volume: &str,
_request: BoundedDeleteRecoverySettlement,
_namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<BoundedDeleteRecoverySettlementReply> {
Err(DiskError::DeleteRecoveryWorkLimit)
}
/// A read-only capability check; unsupported implementations must fail closed.
async fn supports_guarded_delete_versions(&self) -> Result<()> {
Err(DiskError::GuardedDeleteUnsupported)
@@ -2075,6 +2172,13 @@ pub struct WalkDirOptions {
#[serde(default)]
pub skip_hidden_prefix_check: bool,
/// Bound hidden metadata read while proving a non-recursive prefix visible.
#[serde(default)]
pub hidden_entry_budget: usize,
/// A prefix whose visibility was not established before the preceding cut.
#[serde(default)]
pub pending_prefix: Option<String>,
// ReportNotFound will return errFileNotFound if all disks reports the BaseDir cannot be found.
pub report_notfound: bool,
@@ -2407,9 +2511,62 @@ pub enum DeleteRecoveryAction {
AdoptCommit { authority: Box<DeleteRecoveryJournal> },
}
/// A complete inventory or a work-limit refusal. No public discovery cursor is allocated.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BoundedDeleteRecoveryDiscovery {
pub transactions: Vec<DeleteTransactionId>,
pub visited: u32,
pub complete: bool,
}
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 4096;
pub const DELETE_RECOVERY_MAX_INSPECTION_ENTRIES: u32 = 65536;
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BoundedDeleteRecoverySettlement {
pub object: String,
pub transaction: Uuid,
pub expected_fingerprint: DeleteRecoveryDigest,
pub action: DeleteRecoveryAction,
pub max_entries: u32,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoverySettlementOutcome {
Inspected(Box<DeleteRecoveryInspection>),
ReceiptPublishedInspectionDeferred,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BoundedDeleteRecoverySettlementReply {
pub outcome: DeleteRecoverySettlementOutcome,
pub visited: u32,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BoundedDeleteRecoveryInspection {
pub inspection: DeleteRecoveryInspection,
pub visited: u32,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoveryRpcOperation {
SettleBounded(BoundedDeleteRecoverySettlement),
InspectBounded {
object: String,
transaction: Uuid,
max_entries: u32,
},
DiscoverBounded {
object: String,
max_entries: u32,
},
InspectOrphanDataDir {
path: String,
},
@@ -2447,7 +2604,11 @@ impl DeleteRecoveryRpcOperation {
/// Full metadata application and adoption share the existing mutation envelope.
pub fn max_wire_payload_bytes(&self) -> usize {
match self {
Self::ApplyVersion { .. }
Self::SettleBounded(BoundedDeleteRecoverySettlement {
action: DeleteRecoveryAction::AdoptCommit { .. },
..
})
| Self::ApplyVersion { .. }
| Self::ApplyVersions { .. }
| Self::Settle {
action: DeleteRecoveryAction::AdoptCommit { .. },
@@ -2463,8 +2624,11 @@ impl DeleteRecoveryRpcOperation {
pub enum DeleteRecoveryRpcReply {
OrphanDataDir { plain: bool },
Discovered(DeleteRecoveryPage),
DiscoveredBounded(BoundedDeleteRecoveryDiscovery),
Inspected(DeleteRecoveryInspection),
InspectedBounded(BoundedDeleteRecoveryInspection),
Settled(DeleteRecoveryInspection),
SettledBounded(BoundedDeleteRecoverySettlementReply),
AppliedVersion,
AppliedVersions(Vec<Option<u32>>),
Evidence { present: bool },
@@ -2942,6 +3106,8 @@ mod tests {
recursive: true,
incl_deleted: false,
skip_hidden_prefix_check: false,
hidden_entry_budget: 0,
pending_prefix: None,
report_notfound: false,
filter_prefix: Some("prefix_".to_string()),
forward_to: Some("object/path".to_string()),
+275
View File
@@ -0,0 +1,275 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! A candidate index never replaces multipart metadata or quorum validation.
use serde::{Deserialize, Serialize};
use uuid::Uuid;
pub(crate) const RUSTFS_MULTIPART_BUCKET_KEY: &str = "x-rustfs-internal-multipart-bucket";
pub(crate) const RUSTFS_MULTIPART_OBJECT_KEY: &str = "x-rustfs-internal-multipart-object";
pub const MULTIPART_INDEX_PAGE_SIZE: usize = 256;
/// A bounded page from one disk's continuously maintained, boot-local index.
/// `epoch` changes whenever coverage must be rebuilt; callers must not combine
/// pages from different epochs or physical disk identities.
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MultipartIndexPage {
pub bucket: String,
pub disk_id: Uuid,
pub epoch: Uuid,
pub roots: Vec<String>,
pub next_marker: Option<String>,
}
pub const MULTIPART_INDEX_RPC_MAX_BYTES: usize = 64 * 1024;
pub const MULTIPART_INDEX_RPC_MAX_REQUEST_BYTES: usize = 4096;
/// Additive operation inside the legacy read-only DiskInfo JSON envelope.
/// Old peers may ignore this field, but their ordinary response cannot decode
/// as the required signed, challenge-bound MultipartIndexRpcReply.
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MultipartIndexRpcRequest {
pub disk_id: String,
pub metrics: bool,
pub noop: bool,
pub multipart_index: MultipartIndexQuery,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MultipartIndexQuery {
pub operation: String,
pub version: u32,
pub challenge: Uuid,
pub bucket: String,
pub marker: Option<String>,
pub page_cap: usize,
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MultipartIndexRpcReply {
pub operation: String,
pub version: u32,
pub challenge: Uuid,
pub page: MultipartIndexPage,
pub proof: Vec<u8>,
}
impl MultipartIndexRpcRequest {
pub fn new(bucket: &str, marker: Option<&str>) -> Self {
Self {
disk_id: String::new(),
metrics: false,
noop: true,
multipart_index: MultipartIndexQuery {
operation: "list_multipart_index_roots".into(),
version: 1,
challenge: Uuid::new_v4(),
bucket: bucket.to_owned(),
marker: marker.map(str::to_owned),
page_cap: MULTIPART_INDEX_PAGE_SIZE,
},
}
}
pub fn validate(&self) -> crate::disk::error::Result<()> {
let query = &self.multipart_index;
if !self.disk_id.is_empty()
|| self.metrics
|| !self.noop
|| query.operation != "list_multipart_index_roots"
|| query.version != 1
|| query.challenge.is_nil()
|| query.bucket.is_empty()
|| query.bucket.len() > 255
|| query.bucket.contains('/')
|| matches!(query.bucket.as_str(), "." | "..")
|| query.page_cap != MULTIPART_INDEX_PAGE_SIZE
|| query.marker.as_ref().is_some_and(|marker| !valid_root(marker))
{
return Err(crate::disk::error::DiskError::MethodNotAllowed);
}
Ok(())
}
pub fn canonical_request(&self, disk: &str) -> crate::disk::error::Result<Vec<u8>> {
self.validate()?;
if disk.len() > MULTIPART_INDEX_RPC_MAX_REQUEST_BYTES {
return Err(crate::disk::error::DiskError::FileCorrupt);
}
Ok(serde_json::to_vec(&("rustfs-multipart-index-request-v1", disk, self))?)
}
pub fn canonical_response(&self, disk: &str, page: &MultipartIndexPage) -> crate::disk::error::Result<Vec<u8>> {
self.validate()?;
Ok(serde_json::to_vec(&("rustfs-multipart-index-response-v1", disk, self, page))?)
}
}
fn valid_root(root: &str) -> bool {
root.len() == 64 && root.bytes().all(|byte| byte.is_ascii_hexdigit())
}
impl MultipartIndexPage {
pub fn validate(&self, bucket: &str, marker: Option<&str>, expected_disk: Uuid) -> crate::disk::error::Result<()> {
if expected_disk.is_nil()
|| self.disk_id != expected_disk
|| self.epoch.is_nil()
|| self.bucket != bucket
|| self.roots.len() > MULTIPART_INDEX_PAGE_SIZE
|| self
.roots
.iter()
.any(|root| !valid_root(root) || marker.is_some_and(|marker| root.as_str() <= marker))
|| self.roots.windows(2).any(|pair| pair[0] >= pair[1])
|| self
.next_marker
.as_ref()
.is_some_and(|marker| self.roots.last() != Some(marker) || self.roots.len() != MULTIPART_INDEX_PAGE_SIZE)
{
return Err(crate::disk::error::DiskError::FileCorrupt);
}
Ok(())
}
}
impl MultipartIndexRpcReply {
/// This is structural validation only; the caller must also verify `proof`.
pub fn validate(&self, request: &MultipartIndexRpcRequest, expected_disk: Uuid) -> crate::disk::error::Result<()> {
let query = &request.multipart_index;
request.validate()?;
if self.operation != query.operation
|| self.version != query.version
|| self.challenge != query.challenge
|| self.proof.len() != 32
{
return Err(crate::disk::error::DiskError::FileCorrupt);
}
self.page.validate(&query.bucket, query.marker.as_deref(), expected_disk)
}
}
/// A disk can attest coverage only after every page belongs to the same physical
/// identity and index generation. An interrupted/changed stream contributes no
/// partial candidates to the erasure set's coverage count.
pub(crate) async fn collect_pages<F, Fut>(bucket: &str, identity: Uuid, mut fetch: F) -> crate::disk::error::Result<Vec<String>>
where
F: FnMut(Option<String>) -> Fut,
Fut: std::future::Future<Output = crate::disk::error::Result<MultipartIndexPage>>,
{
let mut epoch = None;
let mut marker = None;
let mut roots = Vec::new();
loop {
let page = fetch(marker.clone()).await?;
page.validate(bucket, marker.as_deref(), identity)?;
if epoch.is_some_and(|epoch| epoch != page.epoch) {
return Err(crate::disk::error::DiskError::DiskNotFound);
}
epoch = Some(page.epoch);
roots.extend(page.roots);
marker = page.next_marker;
if marker.is_none() {
return Ok(roots);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn multipart_index_reply_rejects_challenge_disk_cursor_and_unbounded_pages() {
let request = MultipartIndexRpcRequest::new("bucket", None);
let id = Uuid::new_v4();
let mut reply = MultipartIndexRpcReply {
operation: request.multipart_index.operation.clone(),
version: 1,
challenge: request.multipart_index.challenge,
page: MultipartIndexPage {
bucket: "bucket".into(),
disk_id: id,
epoch: Uuid::new_v4(),
roots: vec!["a".repeat(64)],
next_marker: None,
},
proof: vec![0; 32],
};
reply.validate(&request, id).expect("valid structural envelope");
assert!(
reply.validate(&MultipartIndexRpcRequest::new("bucket", None), id).is_err(),
"old challenge cannot replay"
);
assert!(reply.validate(&request, Uuid::new_v4()).is_err(), "physical disk identity cannot change");
reply.page.next_marker = Some("a".repeat(64));
assert!(reply.validate(&request, id).is_err(), "short page cannot claim a continuation");
reply.page.next_marker = None;
reply.page.roots = vec!["a".repeat(64); MULTIPART_INDEX_PAGE_SIZE + 1];
assert!(reply.validate(&request, id).is_err(), "peer cannot enlarge the fixed page cap");
}
#[test]
fn multipart_index_response_proof_binds_the_complete_query_and_page() {
crate::runtime::sources::ensure_test_rpc_secret();
let mut request = MultipartIndexRpcRequest::new("bucket", None);
let mut page = MultipartIndexPage {
bucket: "bucket".into(),
disk_id: Uuid::new_v4(),
epoch: Uuid::new_v4(),
roots: vec![],
next_marker: None,
};
let canonical = request.canonical_response("disk", &page).expect("canonical response");
let proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(&canonical).expect("signed response");
let verify = |request: &MultipartIndexRpcRequest, page: &MultipartIndexPage| {
crate::cluster::rpc::verify_tonic_rpc_response_proof(
&request.canonical_response("disk", page).expect("canonical"),
&proof,
)
};
verify(&request, &page).expect("exact response");
request.multipart_index.marker = Some("0".repeat(64));
assert!(verify(&request, &page).is_err(), "cursor is signed");
request.multipart_index.marker = None;
page.epoch = Uuid::new_v4();
assert!(verify(&request, &page).is_err(), "boot-local coverage epoch is signed");
}
#[tokio::test]
async fn multipart_index_page_collection_rejects_replaced_disk_or_restarted_epoch() {
let id = Uuid::new_v4();
let epoch = Uuid::new_v4();
for replace_disk in [false, true] {
let first = MultipartIndexPage {
bucket: "bucket".into(),
disk_id: id,
epoch,
roots: (0..MULTIPART_INDEX_PAGE_SIZE).map(|index| format!("{index:064x}")).collect(),
next_marker: Some(format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE - 1)),
};
let second = MultipartIndexPage {
bucket: "bucket".into(),
disk_id: if replace_disk { Uuid::new_v4() } else { id },
epoch: if replace_disk { epoch } else { Uuid::new_v4() },
roots: vec![],
next_marker: None,
};
let mut replies = std::collections::VecDeque::from([first, second]);
let result = collect_pages("bucket", id, |marker| {
let page = replies.pop_front().expect("bounded fixture pages");
if marker.is_some() {
assert!(page.roots.is_empty());
}
std::future::ready(Ok(page))
})
.await;
assert!(result.is_err(), "changed physical source cannot contribute partial coverage");
assert!(replies.is_empty(), "exercise the continuation response");
}
}
}
+28
View File
@@ -2731,6 +2731,11 @@ pub(crate) struct NamespaceMutationLease {
}
impl NamespaceMutationLease {
#[cfg(test)]
pub(crate) fn blocking_queue_for_test(&self) -> Arc<DiskBlockingQueue> {
self.blocking_queue.clone()
}
pub(crate) fn attach_external_guard(&self, guard: Arc<dyn Send + Sync>) {
*self.external_guard.lock() = Some(guard);
}
@@ -5382,6 +5387,29 @@ impl PublicationRoot {
})
}
/// Distinguish reconnects from replacement mounts before reusing disk-local state.
pub(crate) fn physical_identity(&self) -> io::Result<(u64, [u8; 16])> {
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
#[cfg(target_os = "linux")]
let metadata = self.directory.metadata()?;
#[cfg(not(target_os = "linux"))]
let metadata = std::fs::metadata(&*self.path)?;
let mut inode = [0; 16];
inode[..8].copy_from_slice(&metadata.ino().to_le_bytes());
Ok((metadata.dev(), inode))
}
#[cfg(windows)]
{
windows_file_identity(&self.directory.handle)
}
#[cfg(not(any(unix, windows)))]
{
Err(io::Error::new(io::ErrorKind::Unsupported, "physical root identity unavailable"))
}
}
pub(crate) fn path(&self) -> &Path {
&self.path
}
+10
View File
@@ -269,6 +269,11 @@ pub enum StorageError {
/// retained for operator inspection.
#[error("delete recovery found an ambiguous legacy residue; operator inspection is required")]
DeleteRecoveryLegacyAmbiguous,
/// Delete recovery exhausted the per-request work budget before it could
/// classify every residue. Retrying later makes progress, so callers treat
/// it like `SlowDown`; it stays distinct so the disk error round-trips.
#[error("delete recovery work limit reached; please retry")]
DeleteRecoveryWorkLimit,
#[error("Decommission not started")]
DecommissionNotStarted,
#[error("Decommission already running")]
@@ -480,6 +485,7 @@ impl From<DiskError> for StorageError {
DiskError::InvalidPath => StorageError::InvalidPath,
DiskError::ConditionalDeleteConflict => StorageError::ConditionalDeleteConflict,
DiskError::GuardedDeleteUnsupported => StorageError::GuardedDeleteUnsupported,
DiskError::DeleteRecoveryWorkLimit => StorageError::DeleteRecoveryWorkLimit,
DiskError::RemoteClientUnavailable(detail) => StorageError::RemoteClientUnavailable(detail),
}
}
@@ -507,6 +513,7 @@ impl StorageError {
StorageError::MethodNotAllowed => DiskError::MethodNotAllowed,
StorageError::StorageFull => DiskError::DiskFull,
StorageError::SlowDown => DiskError::TooManyOpenFiles,
StorageError::DeleteRecoveryWorkLimit => DiskError::DeleteRecoveryWorkLimit,
StorageError::MaxVersionsExceeded => DiskError::MaxVersionsExceeded,
StorageError::InconsistentDisk => DiskError::InconsistentDisk,
StorageError::UnsupportedDisk => DiskError::UnsupportedDisk,
@@ -710,6 +717,7 @@ impl Clone for StorageError {
StorageError::StorageFull => StorageError::StorageFull,
StorageError::SlowDown => StorageError::SlowDown,
StorageError::DeleteRecoveryLegacyAmbiguous => StorageError::DeleteRecoveryLegacyAmbiguous,
StorageError::DeleteRecoveryWorkLimit => StorageError::DeleteRecoveryWorkLimit,
StorageError::PrefixAccessDenied(a, b) => StorageError::PrefixAccessDenied(a.clone(), b.clone()),
StorageError::InvalidUploadIDKeyCombination(a, b) => {
StorageError::InvalidUploadIDKeyCombination(a.clone(), b.clone())
@@ -852,6 +860,7 @@ impl StorageError {
StorageError::StorageFull => StorageErrorCode::StorageFull,
StorageError::SlowDown => StorageErrorCode::SlowDown,
StorageError::DeleteRecoveryLegacyAmbiguous => StorageErrorCode::DeleteRecoveryLegacyAmbiguous,
StorageError::DeleteRecoveryWorkLimit => StorageErrorCode::DeleteRecoveryWorkLimit,
StorageError::PrefixAccessDenied(_, _) => StorageErrorCode::PrefixAccessDenied,
StorageError::InvalidUploadIDKeyCombination(_, _) => StorageErrorCode::InvalidUploadIDKeyCombination,
StorageError::MalformedUploadID(_) => StorageErrorCode::MalformedUploadID,
@@ -958,6 +967,7 @@ impl StorageError {
StorageErrorCode::StorageFull => Some(StorageError::StorageFull),
StorageErrorCode::SlowDown => Some(StorageError::SlowDown),
StorageErrorCode::DeleteRecoveryLegacyAmbiguous => Some(StorageError::DeleteRecoveryLegacyAmbiguous),
StorageErrorCode::DeleteRecoveryWorkLimit => Some(StorageError::DeleteRecoveryWorkLimit),
StorageErrorCode::PrefixAccessDenied => {
Some(StorageError::PrefixAccessDenied(Default::default(), Default::default()))
}
+62 -4
View File
@@ -79,6 +79,14 @@ pub fn compression_scheme_to_read_plan(scheme: &str) -> std::io::Result<(Compres
Ok((CompressionAlgorithm::from_str(scheme)?, ReadCompressionBackend::Legacy))
}
/// Validate that replicated compression metadata can be read by this build.
///
/// # Errors
/// Returns an error for an unsupported compression scheme.
pub fn validate_compression_scheme(scheme: &str) -> std::io::Result<()> {
compression_scheme_to_read_plan(scheme).map(|_| ())
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadEncryptionBackend {
Legacy,
@@ -444,10 +452,12 @@ impl WritePlan {
if let Some(encryption) = self.encryption {
let etag_nonce = match encryption.mode {
WriteEncryptionMode::Singlepart { base_nonce } | WriteEncryptionMode::MultipartLegacy { base_nonce, .. } => {
base_nonce
}
WriteEncryptionMode::SinglepartObjectKey | WriteEncryptionMode::MultipartObjectKey { .. } => [0u8; 12],
WriteEncryptionMode::Singlepart { base_nonce } => base_nonce,
WriteEncryptionMode::SinglepartObjectKey => [0u8; 12],
// Each part write needs an independent fingerprint, including a
// retry at the same part number. This salt affects only the ETag
// MAC; encryption framing and its data nonce stay unchanged.
WriteEncryptionMode::MultipartLegacy { .. } | WriteEncryptionMode::MultipartObjectKey { .. } => rand::random(),
};
let protected_reader = rustfs_rio::EtagReader::new_encrypted(reader, encryption.key_bytes, etag_nonce)?
.with_inner_etag_delegation(delegate_inner_etag);
@@ -558,6 +568,54 @@ mod tests {
}
}
#[tokio::test]
async fn multipart_etags_are_unique_per_write_without_changing_data_encryption() {
let plaintext = b"same confidential part";
let size = i64::try_from(plaintext.len()).expect("test length");
for object_key_mode in [false, true] {
let mut tags = std::collections::HashSet::new();
for part_number in [1, 2, 1] {
let encryption = if object_key_mode {
WriteEncryption::multipart_object_key([7; 32], part_number)
} else {
WriteEncryption::multipart([7; 32], [9; 12], part_number as usize)
};
let source = HashReader::from_stream(Cursor::new(plaintext), size, size, None, None, false).expect("plaintext");
let mut reader = WritePlan::new()
.with_encryption(encryption)
.apply(source, size)
.expect("part plan");
let mut ciphertext = Vec::new();
reader.read_to_end(&mut ciphertext).await.expect("encrypt part");
let tag = reader.try_resolve_etag().expect("part fingerprint");
assert_eq!(reader.try_resolve_etag().as_ref(), Some(&tag), "reading a stored result stays stable");
assert!(
tags.insert(tag),
"different part numbers and retries must not identify matching plaintext"
);
#[cfg(feature = "rio-v2")]
let mut decrypt = if object_key_mode {
DecryptReader::new_multipart_with_object_key(Cursor::new(ciphertext), [7; 32], vec![part_number as usize])
} else {
DecryptReader::new_multipart(Cursor::new(ciphertext), [7; 32], [9; 12], vec![part_number as usize])
};
#[cfg(not(feature = "rio-v2"))]
let mut decrypt = DecryptReader::new_multipart(
Cursor::new(ciphertext),
[7; 32],
if object_key_mode { [0; 12] } else { [9; 12] },
vec![part_number as usize],
);
let mut decoded = Vec::new();
decrypt
.read_to_end(&mut decoded)
.await
.expect("existing data nonce still decrypts");
assert_eq!(decoded, plaintext);
}
}
}
async fn assert_non_trailing_checksum_survives(plan: WritePlan) {
let plaintext = b"checksum-context-through-write-plan".repeat(256);
let actual_size = plaintext.len() as i64;
+72
View File
@@ -949,8 +949,65 @@ pub enum ShardIntegrityWriteMode {
Protected,
}
/// Shared foreground recovery allowance. Clones of ObjectOptions retain one request budget.
/// Credits cover directory entries (including a minimum charge for an empty probe) and
/// transaction inspections. Failed/cancelled probes keep their worst-case reservation.
#[derive(Debug)]
pub struct DeleteRecoveryWorkBudget {
remaining: std::sync::atomic::AtomicU32,
}
/// Entries one disk may reserve up front per recovery step (discovery plus
/// inspection). Every disk of every set a request fans out to reserves this
/// much before any refund arrives, so a budget must cover that many disks.
const DELETE_RECOVERY_RESERVATION_PER_DISK: u32 = 4096 + 65_536;
impl Default for DeleteRecoveryWorkBudget {
/// Covers a full 16-drive set with headroom; `for_disks` scales it when a
/// request spans more drives than that.
fn default() -> Self {
Self::new(16 * 1024 * 1024)
}
}
impl DeleteRecoveryWorkBudget {
/// A per-request budget sized for `disks` concurrent up-front reservations.
pub(crate) fn for_disks(disks: usize) -> Self {
let per_disk = u64::from(DELETE_RECOVERY_RESERVATION_PER_DISK) * 2;
let scaled = u64::try_from(disks).unwrap_or(u64::MAX).saturating_mul(per_disk);
let default = u64::from(Self::default().remaining());
Self::new(u32::try_from(scaled.max(default)).unwrap_or(u32::MAX))
}
pub(crate) fn new(entries: u32) -> Self {
Self {
remaining: std::sync::atomic::AtomicU32::new(entries),
}
}
pub(crate) fn reserve(&self, maximum: u32) -> crate::disk::error::Result<u32> {
use std::sync::atomic::Ordering;
self.remaining
.try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| {
(remaining > 0).then(|| remaining.saturating_sub(maximum))
})
.map(|remaining| remaining.min(maximum))
.map_err(|_| crate::disk::error::DiskError::DeleteRecoveryWorkLimit)
}
pub(crate) fn remaining(&self) -> u32 {
self.remaining.load(std::sync::atomic::Ordering::Acquire)
}
pub(crate) fn refund(&self, unused: u32) {
self.remaining.fetch_add(unused, std::sync::atomic::Ordering::Release);
}
}
#[derive(Default, Clone)]
pub struct ObjectOptions {
#[doc(hidden)]
pub delete_recovery_work_budget: Option<Arc<DeleteRecoveryWorkBudget>>,
// Use the maximum parity (N/2), used when saving server configuration files
pub max_parity: bool,
pub mod_time: Option<OffsetDateTime>,
@@ -1102,6 +1159,12 @@ pub struct ObjectOptions {
pub expiration: ExpirationOptions,
pub lifecycle_audit_event: LcAuditEvent,
/// In-process request gate evaluated on authoritative completion metadata.
/// Not persisted or sent over RPC. The trait deliberately has no Debug bound:
/// implementations may own customer encryption keys.
#[doc(hidden)]
pub multipart_completion_authorization:
Option<Arc<dyn crate::storage_api_contracts::multipart::MultipartCompletionAuthorization<Error = Error>>>,
pub eval_metadata: Option<HashMap<String, String>>,
/// Internal compare-and-set condition for replication workers publishing
/// terminal status after remote I/O. Storage validates it while holding
@@ -1115,6 +1178,10 @@ pub struct ObjectOptions {
/// Authoritative bucket Object Lock snapshot installed inside `ECStore`
/// before a destructive commit reaches the set layer.
pub object_lock_config_snapshot: Option<Arc<ObjectLockConfigSnapshot>>,
/// ECStore single PUTs acquire their destructive-write configuration guards
/// only after staging the client body. External lock owners supply a snapshot.
#[doc(hidden)]
pub defer_put_object_lock_snapshot: bool,
pub want_checksum: Option<Checksum>,
pub skip_verify_bitrot: bool,
@@ -1146,6 +1213,11 @@ pub enum ReplicationStatusWritebackMode {
}
impl ObjectOptions {
pub(crate) fn ensure_delete_recovery_work_budget(&mut self) {
self.delete_recovery_work_budget
.get_or_insert_with(|| Arc::new(DeleteRecoveryWorkBudget::default()));
}
pub(crate) fn shard_integrity_write_enabled(&self) -> bool {
match self.shard_integrity_write_mode {
Some(ShardIntegrityWriteMode::Protected) => true,
@@ -786,7 +786,7 @@ impl WarmBackend for MockWarmBackend {
bytes.len()
};
Ok(tokio::io::BufReader::new(Cursor::new(bytes[start.min(bytes.len())..end].to_vec())))
Ok(ReadCloser::new(Cursor::new(bytes[start.min(bytes.len())..end].to_vec())))
}
async fn remove(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
+3 -4
View File
@@ -37,7 +37,6 @@ use std::{
time::Duration,
};
use time::OffsetDateTime;
use tokio::io::BufReader;
use tokio::{
select,
sync::{Notify, RwLock},
@@ -8183,7 +8182,7 @@ mod tests {
}
async fn get(&self, _object: &str, _rv: &str, _opts: WarmBackendGetOpts) -> io::Result<ReadCloser> {
Ok(BufReader::new(Cursor::new(b"RustFS".to_vec())))
Ok(ReadCloser::new(Cursor::new(b"RustFS".to_vec())))
}
async fn remove(&self, _object: &str, _rv: &str) -> io::Result<()> {
@@ -8248,7 +8247,7 @@ mod tests {
_opts: WarmBackendGetOpts,
) -> std::result::Result<ReadCloser, std::io::Error> {
if self.healthy {
Ok(BufReader::new(Cursor::new(b"RustFS".to_vec())))
Ok(ReadCloser::new(Cursor::new(b"RustFS".to_vec())))
} else {
Err(std::io::Error::other("mock get failed"))
}
@@ -9895,7 +9894,7 @@ mod tests {
}
async fn get(&self, _object: &str, _rv: &str, _opts: WarmBackendGetOpts) -> io::Result<ReadCloser> {
Ok(BufReader::new(Cursor::new(b"RustFS".to_vec())))
Ok(ReadCloser::new(Cursor::new(b"RustFS".to_vec())))
}
async fn remove(&self, _object: &str, _rv: &str) -> io::Result<()> {
+8 -1
View File
@@ -195,7 +195,14 @@ impl SetDisks {
metadata.add_version(selected)?;
}
if metadata.versions.is_empty() {
return Ok(None);
// With no decodable version, the per-version quorum loop cannot
// reject corrupt or unavailable replicas. Only complete, readable
// evidence may establish that this key has no authoritative version.
return if complete {
Ok(None)
} else {
Err(DiskError::ErasureReadQuorum)
};
}
Ok(Some(MetaCacheEntry {
name,
+1 -2
View File
@@ -357,8 +357,7 @@ use crate::io_support::rio::HashReader;
pub const DEFAULT_READ_BUFFER_SIZE: usize = MI_B; // 1 MiB = 1024 * 1024;
pub const MAX_PARTS_COUNT: usize = 10000;
pub(crate) const RUSTFS_MULTIPART_BUCKET_KEY: &str = "x-rustfs-internal-multipart-bucket";
pub(crate) const RUSTFS_MULTIPART_OBJECT_KEY: &str = "x-rustfs-internal-multipart-object";
pub(crate) use crate::disk::multipart_index::{RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY};
pub(crate) const DATA_MOVEMENT_MULTIPART_PREFIX: &str = "data-movement";
const ENV_ISSUE3031_DIAG_ENABLE: &str = "RUSTFS_ISSUE3031_DIAG_ENABLE";
@@ -141,6 +141,7 @@ pub(crate) struct DeleteRecoverySummary {
#[derive(Clone)]
pub(crate) struct DeleteRecoveryContext<'a> {
pub incarnation: Option<Uuid>,
pub work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
pub dry_run: bool,
pub explicit_delete: bool,
/// Only the locked heal path may repair a live marker-creation source.
@@ -156,6 +157,7 @@ impl<'a> DeleteRecoveryContext<'a> {
pub(crate) fn new(incarnation: Option<Uuid>, can_commit: &'a (dyn Fn() -> bool + Sync)) -> Self {
Self {
incarnation,
work_budget: None,
dry_run: false,
explicit_delete: false,
repair_marker_source: false,
@@ -230,19 +232,54 @@ async fn inspect(
bucket: &str,
object: &str,
transaction: Uuid,
budget: Option<&crate::object_api::DeleteRecoveryWorkBudget>,
) -> Vec<Result<DeleteRecoveryInspection>> {
join_all(disks.iter().map(|disk| async move {
let state = disk
.as_ref()
.ok_or(DiskError::DiskNotFound)?
.inspect_delete_transaction(bucket, object, transaction)
.await?;
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
let state = if let Some(budget) = budget {
let reserved = budget.reserve(crate::disk::DELETE_RECOVERY_MAX_INSPECTION_ENTRIES)?;
let reply = disk
.inspect_delete_transaction_bounded(bucket, object, transaction, reserved)
.await?;
if reply.visited > reserved {
return Err(DiskError::FileCorrupt);
}
budget.refund(reserved - reply.visited.max(1));
reply.inspection
} else {
disk.inspect_delete_transaction(bucket, object, transaction).await?
};
validate(&state, object, transaction)?;
Ok(state)
}))
.await
}
async fn discover_bounded(
disk: &DiskStore,
bucket: &str,
object: &str,
budget: &crate::object_api::DeleteRecoveryWorkBudget,
) -> Result<Vec<DeleteTransactionId>> {
let reserved = budget.reserve(crate::disk::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)?;
let page = disk.discover_delete_transactions_bounded(bucket, object, reserved).await?;
if page.visited > reserved
|| page.transactions.len() > 256
|| page
.transactions
.iter()
.any(|id| id.object != object || id.transaction.is_nil())
|| (!page.complete && !page.transactions.is_empty())
{
return Err(DiskError::FileCorrupt);
}
budget.refund(reserved - page.visited.max(1));
if !page.complete {
return Err(DiskError::DeleteRecoveryWorkLimit);
}
Ok(page.transactions)
}
async fn discover(disk: &DiskStore, bucket: &str, object: Option<&str>) -> Result<Vec<DeleteTransactionId>> {
let mut cursor = None;
let mut ids = BTreeSet::new();
@@ -273,30 +310,92 @@ async fn discover(disk: &DiskStore, bucket: &str, object: Option<&str>) -> Resul
.collect())
}
#[derive(Clone, Copy)]
struct RecoveryWorkScope<'a> {
namespace_owner: Option<&'a Arc<dyn Send + Sync>>,
budget: Option<&'a crate::object_api::DeleteRecoveryWorkBudget>,
}
async fn settle_outcome(
disk: &DiskStore,
bucket: &str,
state: &DeleteRecoveryInspection,
action: DeleteRecoveryAction,
can_commit: &(impl Fn() -> bool + Sync + ?Sized),
scope: RecoveryWorkScope<'_>,
) -> Result<crate::disk::DeleteRecoverySettlementOutcome> {
use crate::disk::DeleteRecoverySettlementOutcome;
if !can_commit() {
return Err(DiskError::other("delete recovery namespace fence was lost"));
}
let outcome = if let Some(budget) = scope.budget {
let reserved = budget.reserve(crate::disk::DELETE_RECOVERY_MAX_INSPECTION_ENTRIES)?;
let may_publish = matches!(&action, DeleteRecoveryAction::PublishCommit | DeleteRecoveryAction::AdoptCommit { .. });
let reply = disk
.settle_delete_transaction_bounded(
bucket,
crate::disk::BoundedDeleteRecoverySettlement {
object: state.id.object.clone(),
transaction: state.id.transaction,
expected_fingerprint: state.fingerprint,
action,
max_entries: reserved,
},
scope.namespace_owner.cloned(),
)
.await?;
if reply.visited > reserved
|| (!may_publish && matches!(&reply.outcome, DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred))
{
return Err(DiskError::FileCorrupt);
}
budget.refund(reserved - reply.visited.max(1));
reply.outcome
} else {
DeleteRecoverySettlementOutcome::Inspected(Box::new(
disk.settle_delete_transaction_with_namespace_owner(
bucket,
&state.id.object,
state.id.transaction,
state.fingerprint,
action,
scope.namespace_owner.cloned(),
)
.await?,
))
};
if let DeleteRecoverySettlementOutcome::Inspected(next) = &outcome {
validate(next, &state.id.object, state.id.transaction)?;
}
Ok(outcome)
}
async fn settle(
disk: &DiskStore,
bucket: &str,
state: &DeleteRecoveryInspection,
action: DeleteRecoveryAction,
can_commit: &(impl Fn() -> bool + Sync + ?Sized),
namespace_owner: Option<&Arc<dyn Send + Sync>>,
scope: RecoveryWorkScope<'_>,
) -> Result<DeleteRecoveryInspection> {
if !can_commit() {
return Err(DiskError::other("delete recovery namespace fence was lost"));
match settle_outcome(disk, bucket, state, action, can_commit, scope).await? {
crate::disk::DeleteRecoverySettlementOutcome::Inspected(inspection) => Ok(*inspection),
crate::disk::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred => {
Err(DiskError::DeleteRecoveryWorkLimit)
}
}
}
fn has_receipt_quorum(outcomes: &[Result<crate::disk::DeleteRecoverySettlementOutcome>], quorum: usize) -> bool {
outcomes.iter().filter(|outcome| published_receipt(outcome)).count() >= quorum
}
fn published_receipt(outcome: &Result<crate::disk::DeleteRecoverySettlementOutcome>) -> bool {
match outcome {
Ok(crate::disk::DeleteRecoverySettlementOutcome::Inspected(state)) => receipt(state.state),
Ok(crate::disk::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred) => true,
Err(_) => false,
}
let result = disk
.settle_delete_transaction_with_namespace_owner(
bucket,
&state.id.object,
state.id.transaction,
state.fingerprint,
action,
namespace_owner.cloned(),
)
.await;
let next = result?;
validate(&next, &state.id.object, state.id.transaction)?;
Ok(next)
}
/// Each phase fans out across independent drives, and awaits all started work
@@ -307,23 +406,16 @@ async fn settle_group(
copies: &mut [DeleteRecoveryInspection],
action: impl Fn(&DeleteRecoveryInspection) -> Option<DeleteRecoveryAction> + Sync,
can_commit: &(impl Fn() -> bool + Sync + ?Sized),
namespace_owner: Option<&Arc<dyn Send + Sync>>,
scope: RecoveryWorkScope<'_>,
removed_versions: &mut HashSet<Uuid>,
) -> Result<bool> {
let results = join_all(disks.iter().zip(copies.iter()).map(|(disk, copy)| {
let action = action(copy);
async move {
match action {
Some(action) => settle(
disk.as_ref().ok_or(DiskError::DiskNotFound)?,
bucket,
copy,
action,
can_commit,
namespace_owner,
)
.await
.map(Some),
Some(action) => settle(disk.as_ref().ok_or(DiskError::DiskNotFound)?, bucket, copy, action, can_commit, scope)
.await
.map(Some),
None => Ok(None),
}
}
@@ -439,16 +531,31 @@ impl SetDisks {
true
}
};
let mut fast_receipt_quorum = false;
if can_try_local_fast_path && can_commit() {
let fast_published = join_all(disks.iter().map(|disk| async {
if !can_commit() {
return Err(DiskError::other("delete recovery namespace fence was lost"));
}
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
disk.try_publish_prepared_delete_local(bucket, object, transaction, context.namespace_owner.clone())
.await
disk.try_publish_prepared_delete_local(
bucket,
object,
transaction,
context.namespace_owner.clone(),
context.work_budget.clone(),
)
.await
}))
.await;
fast_receipt_quorum = fast_published.iter().filter(|result| matches!(result, Ok(Some(_)))).count() >= write_quorum;
if fast_receipt_quorum
&& fast_published
.iter()
.any(|result| matches!(result, Err(DiskError::DeleteRecoveryWorkLimit)))
{
return Ok(());
}
let all_fast_published = fast_published.len() == self.set_drive_count
&& fast_published.iter().all(|res| matches!(res, Ok(Some((_, true, true)))));
if all_fast_published && can_commit() {
@@ -456,6 +563,7 @@ impl SetDisks {
test_hooks::record_cleanup_attempt(bucket, object);
let finalized = join_all(disks.iter().zip(fast_published.iter()).map(|(disk, pub_res)| {
let ns_owner = context.namespace_owner.clone();
let work_budget = context.work_budget.clone();
async move {
if !can_commit() {
return Ok(false);
@@ -466,11 +574,24 @@ impl SetDisks {
let Ok(Some((journal, _, _))) = pub_res else {
return Ok(false);
};
disk.try_finish_and_finalize_prepared_delete_local(bucket, object, transaction, journal, ns_owner)
.await
disk.try_finish_and_finalize_prepared_delete_local(
bucket,
object,
transaction,
journal,
ns_owner,
work_budget,
)
.await
}
}))
.await;
if finalized
.iter()
.any(|result| matches!(result, Err(DiskError::DeleteRecoveryWorkLimit)))
{
return Ok(());
}
if finalized.iter().all(|res| matches!(res, Ok(true))) {
if let Some(ticket) = context.recovery_ticket.as_ref() {
self.ctx.delete_recovery_hints().ack(ticket);
@@ -479,7 +600,17 @@ impl SetDisks {
}
}
}
let states = inspect(disks, bucket, object, transaction).await;
let states = inspect(disks, bucket, object, transaction, context.work_budget.as_deref()).await;
if states
.iter()
.any(|result| matches!(result, Err(DiskError::DeleteRecoveryWorkLimit)))
{
return if fast_receipt_quorum {
Ok(())
} else {
Err(DiskError::DeleteRecoveryWorkLimit)
};
}
let untouched_absence = |state: &Result<DeleteRecoveryInspection>| matches!(state, Ok(state) if state.state == DeleteRecoveryState::Absent && state.current_digest.is_none());
if states.iter().filter(|state| untouched_absence(state)).count() >= write_quorum
&& states.iter().all(|state| {
@@ -499,25 +630,50 @@ impl SetDisks {
if state.state == DeleteRecoveryState::LegacyAmbiguous {
return Err(DiskError::FileCorrupt);
}
settle(
settle_outcome(
disk,
bucket,
state,
DeleteRecoveryAction::PublishCommit,
can_commit,
context.namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: context.namespace_owner.as_ref(),
budget: context.work_budget.as_deref(),
},
)
.await
}))
.await;
if published
.iter()
.filter(|state| matches!(state, Ok(state) if receipt(state.state)))
.count()
< write_quorum
{
return Err(DiskError::ErasureWriteQuorum);
if !has_receipt_quorum(&published, write_quorum) {
return Err(
if published
.iter()
.any(|result| matches!(result, Err(DiskError::DeleteRecoveryWorkLimit)))
{
DiskError::DeleteRecoveryWorkLimit
} else {
DiskError::ErasureWriteQuorum
},
);
}
if published.iter().any(|outcome| {
matches!(
outcome,
Ok(crate::disk::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred)
)
}) {
return Ok(());
}
let published = published
.into_iter()
.map(|outcome| match outcome {
Ok(crate::disk::DeleteRecoverySettlementOutcome::Inspected(state)) => Ok(*state),
Ok(crate::disk::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred) => {
Err(DiskError::DeleteRecoveryWorkLimit)
}
Err(error) => Err(error),
})
.collect::<Vec<_>>();
// Physical cleanup is retryable after the certified decision. An
// unavailable participant retains the decision for its eventual return.
let ticket = context.recovery_ticket.clone();
@@ -670,6 +826,7 @@ impl SetDisks {
}
let DeleteRecoveryContext {
incarnation,
work_budget,
dry_run,
explicit_delete,
repair_marker_source,
@@ -681,7 +838,13 @@ impl SetDisks {
} = context;
let disks = self.disk_inventory().await;
let complete_inventory = disks.len() == self.set_drive_count && disks.iter().all(Option::is_some);
let pages = join_all(disks.iter().flatten().map(|disk| discover(disk, bucket, Some(object)))).await;
let pages = join_all(disks.iter().flatten().map(|disk| async {
match work_budget.as_deref() {
Some(budget) => discover_bounded(disk, bucket, object, budget).await,
None => discover(disk, bucket, Some(object)).await,
}
}))
.await;
let mut transactions = BTreeSet::new();
let mut inspected = 0;
let mut offline = !complete_inventory;
@@ -712,6 +875,7 @@ impl SetDisks {
..Default::default()
};
let mut all_states = BTreeMap::new();
let mut registered_evidence_hint = false;
for transaction in transactions {
let mut complete = Vec::with_capacity(disks.len());
let inspections = if let Some((published_tx, published_states)) = published_hint
@@ -723,8 +887,14 @@ impl SetDisks {
{
published_states.to_vec()
} else {
inspect(&disks, bucket, object, transaction).await
inspect(&disks, bucket, object, transaction, work_budget.as_deref()).await
};
if inspections
.iter()
.any(|result| matches!(result, Err(DiskError::DeleteRecoveryWorkLimit)))
{
return Err(DiskError::DeleteRecoveryWorkLimit);
}
for state in inspections {
match state {
Ok(state) => {
@@ -736,6 +906,33 @@ impl SetDisks {
summary.blocked += 1;
return Ok(summary);
}
// Candidate UUID directories are not evidence. Only a bound, inspected
// journal can create a foreground hint; empty/partial scans rely on the
// existing raw anti-entropy obligation instead of queuing arbitrary prefixes.
if work_budget.is_some()
&& !registered_evidence_hint
&& journal.protocol_version == DELETE_RECOVERY_PROTOCOL_VERSION
&& journal.volume == bucket
&& journal.object == object
&& journal.transaction == transaction
&& matches!(
state.state,
DeleteRecoveryState::NewPrepared
| DeleteRecoveryState::QuorumCommitted
| DeleteRecoveryState::CommitFinished
)
{
self.ctx.delete_recovery_hints().register(
crate::store::delete_recovery_runtime::DeleteRecoveryKey {
pool: self.pool_index,
set: self.set_index,
bucket: bucket.to_owned(),
object: object.to_owned(),
incarnation,
},
);
registered_evidence_hint = true;
}
}
complete.push(state);
}
@@ -767,7 +964,7 @@ impl SetDisks {
};
if let Some((transaction, source)) = prerequisite {
Box::pin(self.heal_delete_recovery_source(bucket, object, &source, can_commit, namespace_owner.as_ref())).await?;
let refreshed = inspect(&disks, bucket, object, transaction)
let refreshed = inspect(&disks, bucket, object, transaction, work_budget.as_deref())
.await
.into_iter()
.collect::<Result<Vec<_>>>()?;
@@ -853,7 +1050,7 @@ impl SetDisks {
if mutated {
// A sibling may have restored metadata or reclaimed shared
// evidence. Refresh fingerprints before acting on this group.
*copies = inspect(&disks, bucket, object, transaction)
*copies = inspect(&disks, bucket, object, transaction, work_budget.as_deref())
.await
.into_iter()
.collect::<Result<Vec<_>>>()?;
@@ -876,7 +1073,10 @@ impl SetDisks {
(copy.state == DeleteRecoveryState::LegacyAmbiguous).then_some(DeleteRecoveryAction::DiscardLegacy)
},
can_commit,
namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: namespace_owner.as_ref(),
budget: work_budget.as_deref(),
},
&mut summary.removed_versions,
)
.await?
@@ -987,7 +1187,10 @@ impl SetDisks {
)
},
can_commit,
namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: namespace_owner.as_ref(),
budget: work_budget.as_deref(),
},
&mut summary.removed_versions,
)
.await?;
@@ -1001,7 +1204,10 @@ impl SetDisks {
.then_some(DeleteRecoveryAction::FinishCommit)
},
can_commit,
namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: namespace_owner.as_ref(),
budget: work_budget.as_deref(),
},
&mut summary.removed_versions,
)
.await?;
@@ -1048,7 +1254,10 @@ impl SetDisks {
copies,
|copy| (copy.state == DeleteRecoveryState::NewPrepared).then_some(DeleteRecoveryAction::Rollback),
can_commit,
namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: namespace_owner.as_ref(),
budget: work_budget.as_deref(),
},
&mut summary.removed_versions,
)
.await?
@@ -1076,7 +1285,7 @@ impl SetDisks {
settled.push(copies);
} else {
for transaction in all_states.keys() {
let fresh = inspect(&disks, bucket, object, *transaction).await;
let fresh = inspect(&disks, bucket, object, *transaction, work_budget.as_deref()).await;
if fresh.iter().any(|state| !matches!(state, Ok(state) if matches!(state.state, DeleteRecoveryState::Absent | DeleteRecoveryState::CommitFinished))) {
summary.pending += 1;
return Ok(summary);
@@ -1086,7 +1295,7 @@ impl SetDisks {
}
for (index, mut copies) in settled.into_iter().enumerate() {
if index > 0 {
copies = inspect(&disks, bucket, object, copies[0].id.transaction)
copies = inspect(&disks, bucket, object, copies[0].id.transaction, work_budget.as_deref())
.await
.into_iter()
.collect::<Result<Vec<_>>>()?;
@@ -1106,7 +1315,10 @@ impl SetDisks {
&mut copies,
|copy| (copy.state == DeleteRecoveryState::CommitFinished).then_some(DeleteRecoveryAction::FinalizeCommit),
can_commit,
namespace_owner.as_ref(),
RecoveryWorkScope {
namespace_owner: namespace_owner.as_ref(),
budget: work_budget.as_deref(),
},
&mut summary.removed_versions,
)
.await?
@@ -1358,6 +1570,457 @@ mod tests {
body
}
#[tokio::test]
async fn delete_recovery_mixed_inspected_and_deferred_receipts_count_only_real_quorum() {
let bucket = "recover-bounded-mixed-quorum";
let (_dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![73; 4096], false).await;
let transaction = applied(&store, &disks, bucket, None, disks.len()).await;
let mut outcomes = Vec::new();
for (index, disk) in disks.iter().enumerate() {
let state = disk
.inspect_delete_transaction_bounded(bucket, "object", transaction, 65536)
.await
.expect("preparation inspection");
let max_entries = match index {
0 => 65536,
1 | 2 => state.visited,
_ => 1,
};
outcomes.push(
disk.settle_delete_transaction_bounded(
bucket,
crate::disk::BoundedDeleteRecoverySettlement {
object: "object".to_owned(),
transaction,
expected_fingerprint: state.inspection.fingerprint,
action: DeleteRecoveryAction::PublishCommit,
max_entries,
},
None,
)
.await
.map(|reply| reply.outcome),
);
}
assert!(
matches!(&outcomes[0], Ok(crate::disk::DeleteRecoverySettlementOutcome::Inspected(state)) if receipt(state.state))
);
assert!(matches!(
&outcomes[1],
Ok(crate::disk::DeleteRecoverySettlementOutcome::ReceiptPublishedInspectionDeferred)
));
assert!(outcomes[3].is_err());
assert!(has_receipt_quorum(&outcomes, 3));
assert!(!has_receipt_quorum(&outcomes, 4));
outcomes[2] = Err(DiskError::FileAccessDenied);
assert!(
!has_receipt_quorum(&outcomes, 3),
"a write failure cannot substitute for a durable receipt"
);
}
#[tokio::test]
async fn delete_recovery_single_and_batch_share_bounded_work_across_versions_and_retries() {
use crate::object_api::DeleteRecoveryWorkBudget;
use crate::storage_api_contracts::object::ObjectToDelete;
let bucket = "recover-bounded-request";
let (dirs, store, _) = fixture(bucket).await;
for dir in &dirs {
let prefix = dir.path().join(bucket).join("prefix");
std::fs::create_dir_all(&prefix).expect("physical prefix");
for index in 0..64 {
std::fs::create_dir(prefix.join(format!("child-{index}"))).expect("legitimate child");
}
}
let error = store
.delete_object(
bucket,
"prefix",
ObjectOptions {
delete_recovery_work_budget: Some(Arc::new(DeleteRecoveryWorkBudget::new(8))),
..Default::default()
},
)
.await
.expect_err("incomplete discovery is not absence");
assert!(matches!(error, StorageError::DeleteRecoveryWorkLimit), "{error:?}");
for incarnation in [
None,
Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation"),
),
] {
assert!(
!store
.ctx
.delete_recovery_hints()
.is_pending(&crate::store::delete_recovery_runtime::DeleteRecoveryKey {
pool: 0,
set: 0,
bucket: bucket.to_owned(),
object: "prefix".to_owned(),
incarnation,
}),
"an unproven prefix must not move its scan into background recovery"
);
}
for distinct_versions in [false, true] {
let budget = Arc::new(DeleteRecoveryWorkBudget::new(32_768));
let objects = (0..1000)
.map(|_| ObjectToDelete {
object_name: "prefix".to_owned(),
version_id: distinct_versions.then(Uuid::new_v4),
..Default::default()
})
.collect();
let (_, errors) = store
.delete_objects(
bucket,
objects,
ObjectOptions {
delete_recovery_work_budget: Some(Arc::clone(&budget)),
..Default::default()
},
)
.await;
assert!(
errors
.iter()
.any(|error| matches!(error, Some(StorageError::DeleteRecoveryWorkLimit))),
"repeated actions must not each obtain a new allowance: {distinct_versions}"
);
assert!(
dirs.iter()
.all(|dir| dir.path().join(bucket).join("prefix/child-63").is_dir())
);
}
let error = store
.delete_object(bucket, "prefix", ObjectOptions::default())
.await
.expect_err("ordinary missing key");
assert!(
crate::error::is_err_object_not_found(&error),
"small complete inventory retains no-op semantics: {error:?}"
);
}
#[tokio::test]
async fn delete_recovery_batch_budget_survives_pool_and_set_fanout() {
use crate::layout::endpoint::Endpoint;
use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
use crate::runtime::instance::InstanceContext;
let mut dirs = Vec::new();
let mut pools = Vec::new();
for pool_index in 0..2 {
let mut endpoints = Vec::new();
for set_index in 0..2 {
for disk_index in 0..4 {
let dir = tempfile::tempdir().expect("temporary disk");
let mut endpoint = Endpoint::try_from(dir.path().to_str().expect("disk path")).expect("endpoint");
endpoint.set_pool_index(pool_index);
endpoint.set_set_index(set_index);
endpoint.set_disk_index(disk_index);
endpoints.push(endpoint);
dirs.push(dir);
}
}
pools.push(PoolEndpoints {
legacy: false,
set_count: 2,
drives_per_set: 4,
endpoints: Endpoints::from(endpoints),
cmd_line: format!("bounded-recovery-{pool_index}"),
platform: "test".to_owned(),
});
}
let endpoints = EndpointServerPools(pools);
let ctx = Arc::new(InstanceContext::new());
crate::store::init_local_disks_with_instance_ctx(&ctx, endpoints.clone())
.await
.expect("local disks");
let cancel = tokio_util::sync::CancellationToken::new();
let store = ECStore::new_with_instance_ctx("127.0.0.1:0".parse().expect("address"), endpoints, cancel.clone(), ctx)
.await
.expect("two pools with two sets");
cancel.cancel();
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "recover-bounded-fanout";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket");
let candidates = (0..100).map(|index| format!("prefix-{index}")).collect::<Vec<_>>();
let mut keys = BTreeSet::new();
for pool in &store.pools {
for set in &pool.disk_set {
keys.insert(
candidates
.iter()
.find(|key| Arc::ptr_eq(&pool.get_disks_by_key(key), set))
.expect("fixture key for set")
.clone(),
);
}
}
let keys = keys.into_iter().collect::<Vec<_>>();
for pool in &store.pools {
for set in &pool.disk_set {
assert!(
keys.iter().any(|key| Arc::ptr_eq(&pool.get_disks_by_key(key), set)),
"fixture must cover every set"
);
}
}
for dir in &dirs {
for key in &keys {
let prefix = dir.path().join(bucket).join(key);
std::fs::create_dir_all(&prefix).expect("physical prefix");
for child in 0..8 {
std::fs::create_dir(prefix.join(format!("child-{child}"))).expect("child");
}
}
}
let budget = Arc::new(crate::object_api::DeleteRecoveryWorkBudget::new(32_768));
let objects = (0..1000)
.map(|index| crate::storage_api_contracts::object::ObjectToDelete {
object_name: keys[index % keys.len()].clone(),
version_id: Some(Uuid::new_v4()),
..Default::default()
})
.collect();
let (_, errors) = store
.delete_objects(
bucket,
objects,
ObjectOptions {
delete_recovery_work_budget: Some(Arc::clone(&budget)),
..Default::default()
},
)
.await;
assert!(
errors
.iter()
.any(|error| matches!(error, Some(StorageError::DeleteRecoveryWorkLimit))),
"all pools and sets must share the finite allowance"
);
assert!(budget.remaining() < 32_768, "fanout must debit the original request allowance");
assert!(
dirs.iter()
.all(|dir| dir.path().join(bucket).join(&keys[0]).join("child-7").is_dir())
);
}
#[tokio::test]
async fn delete_recovery_fast_commit_keeps_success_when_cleanup_budget_is_empty() {
let bucket = "recover-bounded-fast-commit";
let (_dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![74; 2 * 1024 * 1024], false).await;
// Inline cleanup needs no directory enumeration and may legitimately
// finish with zero scan credits. Require real part files here.
for disk in &disks {
let info = disk
.read_version("", bucket, "object", "", &ReadOptions::default())
.await
.expect("read non-inline source");
assert!(!info.inline_data(), "the fixture must require physical part cleanup");
let data_dir = info.data_dir.expect("physical data directory");
let part = std::path::PathBuf::from(disk.endpoint().to_string())
.join(bucket)
.join("object")
.join(data_dir.to_string())
.join("part.1");
assert!(part.is_file(), "the cleanup scan must encounter a real part");
}
store
.delete_object(
bucket,
"object",
ObjectOptions {
delete_recovery_work_budget: Some(Arc::new(crate::object_api::DeleteRecoveryWorkBudget::new(0))),
..Default::default()
},
)
.await
.expect("a cached prepared receipt commits independently of cleanup budget");
assert!(
store
.ctx
.delete_recovery_hints()
.is_pending(&crate::store::delete_recovery_runtime::DeleteRecoveryKey {
pool: 0,
set: 0,
bucket: bucket.to_owned(),
object: "object".to_owned(),
incarnation: Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation")
),
})
);
assert!(
!disks[0]
.discover_delete_transactions(bucket, Some("object"), None, 128)
.await
.expect("retained recovery proof")
.transactions
.is_empty()
);
store
.recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default())
.await
.expect("background finishes committed cleanup");
}
#[tokio::test]
async fn delete_recovery_unverified_uuid_candidate_does_not_enqueue_background_work() {
let bucket = "recover-bounded-candidate";
let (dirs, store, _) = fixture(bucket).await;
let transaction = Uuid::new_v4();
for directory in &dirs {
let candidate = directory.path().join(bucket).join("candidate").join(transaction.to_string());
std::fs::create_dir_all(&candidate).expect("candidate path");
std::fs::write(candidate.join("delete-prepared"), b"not a bound journal").expect("unverified candidate");
}
assert!(
store
.delete_object(bucket, "candidate", ObjectOptions::default())
.await
.is_err()
);
for incarnation in [
None,
Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation"),
),
] {
assert!(
!store
.ctx
.delete_recovery_hints()
.is_pending(&crate::store::delete_recovery_runtime::DeleteRecoveryKey {
pool: 0,
set: 0,
bucket: bucket.to_owned(),
object: "candidate".to_owned(),
incarnation,
})
);
}
}
#[tokio::test]
async fn delete_recovery_budget_refusal_preserves_evidence_for_background_replay() {
let bucket = "recover-bounded-evidence";
let (_dirs, store, disks) = fixture(bucket).await;
let payload = vec![71; 4096];
put(&store, bucket, &payload, false).await;
let transaction = applied(&store, &disks, bucket, None, disks.len()).await;
let before = disks[0]
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("before refusal");
let error = store
.delete_object(
bucket,
"object",
ObjectOptions {
delete_recovery_work_budget: Some(Arc::new(crate::object_api::DeleteRecoveryWorkBudget::new(1))),
..Default::default()
},
)
.await
.expect_err("no incomplete recovery admission");
assert!(matches!(error, StorageError::DeleteRecoveryWorkLimit), "{error:?}");
let after = disks[0]
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("after refusal");
assert_eq!(before.fingerprint, after.fingerprint);
store
.recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default())
.await
.expect("background-compatible recovery");
assert_eq!(body(&store, bucket, None).await, payload);
}
#[tokio::test]
async fn delete_recovery_committed_delete_defers_exhausted_cleanup_without_rollback() {
let bucket = "recover-bounded-commit";
let (_dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![72; 4096], false).await;
let transaction = applied(&store, &disks, bucket, None, disks.len()).await;
let set = &store.pools[0].disk_set[0];
let key = crate::store::delete_recovery_runtime::DeleteRecoveryKey {
pool: 0,
set: 0,
bucket: bucket.to_owned(),
object: "object".to_owned(),
incarnation: None,
};
let ticket = set.ctx.delete_recovery_hints().register(key.clone());
let budget = Arc::new(crate::object_api::DeleteRecoveryWorkBudget::default());
let exhausted_after_commit = Arc::new(std::sync::atomic::AtomicBool::new(false));
// This hook disables local fast cleanup and runs only after receipt
// publication and the shared recovery path's key-wide GC inspection.
// Exhausting preparation instead would correctly reject the delete.
test_hooks::before_gc_finalize(bucket, "object", {
let budget = Arc::clone(&budget);
let exhausted_after_commit = Arc::clone(&exhausted_after_commit);
Arc::new(move || {
let budget = Arc::clone(&budget);
let exhausted_after_commit = Arc::clone(&exhausted_after_commit);
Box::pin(async move {
assert!(budget.reserve(u32::MAX).expect("credits remain until committed cleanup") > 0);
exhausted_after_commit.store(true, std::sync::atomic::Ordering::Release);
})
})
});
set.publish_delete_commit(
bucket,
"object",
transaction,
&set.disk_inventory().await,
set.set_drive_count / 2 + 1,
DeleteRecoveryContext {
work_budget: Some(Arc::clone(&budget)),
recovery_ticket: ticket,
..DeleteRecoveryContext::new(None, &|| true)
},
)
.await
.expect("the durable decision still succeeds");
assert!(exhausted_after_commit.load(std::sync::atomic::Ordering::Acquire));
assert_eq!(budget.remaining(), 0);
assert!(set.ctx.delete_recovery_hints().is_pending(&key));
let state = disks[0]
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("retained decision");
assert!(receipt(state.state));
store
.recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default())
.await
.expect("eventual cleanup");
assert!(
disks[0]
.discover_delete_transactions(bucket, Some("object"), None, 128)
.await
.expect("cleaned proofs")
.transactions
.is_empty()
);
}
#[tokio::test]
async fn delete_recovery_set_rolls_back_undecided_inline_bytes() {
let bucket = "recover-undecided";
+89 -6
View File
@@ -95,13 +95,8 @@ impl SetDisks {
bucket: &str,
object: &str,
) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
let resource = ObjectKey::with_version(
bucket,
object_name_disk_identity(object).as_ref(),
format!("rustfs:delete-transaction:{}:{}", self.pool_index, self.set_index),
);
let guard = self
.namespace_lock_for_key(resource)
.namespace_lock_for_key(self.delete_transaction_resource(bucket, object))
.await
.get_write_lock(super::super::get_lock_acquire_timeout())
.await
@@ -111,6 +106,94 @@ impl SetDisks {
Ok(Arc::new(guard))
}
fn delete_transaction_resource(&self, bucket: &str, object: &str) -> ObjectKey {
ObjectKey::with_version(
bucket,
object_name_disk_identity(object).as_ref(),
format!("rustfs:delete-transaction:{}:{}", self.pool_index, self.set_index),
)
}
/// Batched write-lock acquisition for the whole key set of one bulk
/// mutation: one lock round instead of one quorum round-trip per key.
///
/// Bulk delete is the only multi-key write-lock holder. The batched round
/// is deadlock-free by construction (a single `lock_batch` RPC per lock
/// client, and every lock server orders concurrent batches by the same
/// global (shard, key) order), so unlike an optimistic probe wave it
/// needs no canonical serial fallback. Outcomes are per key and use the
/// same error shape as the single-key paths.
async fn namespace_write_locks_batch(
&self,
bucket: &str,
resources: &[ObjectKey],
timeout: std::time::Duration,
mode: &'static str,
) -> Vec<Result<rustfs_lock::NamespaceLockGuard>> {
if resources.is_empty() {
return Vec::new();
}
let lock = self.namespace_lock_for_key(resources[0].clone()).await;
lock.get_write_locks_batch(resources, timeout)
.await
.into_iter()
.zip(resources)
.map(|(result, resource)| {
result.map_err(|error| self.map_namespace_lock_error(bucket, resource.object.as_ref(), mode, error))
})
.collect()
}
/// Batched raw namespace write locks for a bulk delete's key set. The
/// resource keys use the same disk-identity folding as `new_ns_lock`, so
/// aliases (`a\b` vs `a/b`) still share one lock.
pub(crate) async fn acquire_object_write_locks_batch(
&self,
bucket: &str,
objects: &[&str],
timeout: std::time::Duration,
) -> Vec<Result<rustfs_lock::NamespaceLockGuard>> {
let resources: Vec<ObjectKey> = objects
.iter()
.map(|object| ObjectKey {
bucket: Arc::from(bucket),
object: Arc::from(object_name_disk_identity(object).as_ref()),
version: None,
})
.collect();
self.namespace_write_locks_batch(bucket, &resources, timeout, "write").await
}
/// Batched delete-transaction locks for a bulk delete's key set; see
/// [`Self::acquire_delete_transaction_lock`] for the resource shape.
pub(in crate::set_disk) async fn acquire_delete_transaction_locks_batch(
&self,
bucket: &str,
objects: &[&str],
timeout: std::time::Duration,
) -> Vec<Result<Arc<rustfs_lock::NamespaceLockGuard>>> {
let resources: Vec<ObjectKey> = objects
.iter()
.map(|object| self.delete_transaction_resource(bucket, object))
.collect();
let mut guards = Vec::with_capacity(resources.len());
for (object, result) in objects.iter().zip(
self.namespace_write_locks_batch(bucket, &resources, timeout, "delete_transaction")
.await,
) {
match result {
Ok(guard) => {
#[cfg(test)]
super::delete_recovery::test_hooks::run_after_transaction_lock_acquired(bucket, object, &guard);
let _ = object;
guards.push(Ok(Arc::new(guard)));
}
Err(error) => guards.push(Err(error)),
}
}
guards
}
#[allow(dead_code, reason = "lock diagnostics formatter with no caller in this port (backlog#1823)")]
pub(in crate::set_disk) fn format_lock_error(&self, bucket: &str, object: &str, mode: &str, err: &LockResult) -> String {
match err {
File diff suppressed because it is too large Load Diff
+627 -45
View File
@@ -3914,6 +3914,7 @@ impl SetDisks {
bucket,
object,
DeleteRecoveryContext {
work_budget: opts.delete_recovery_work_budget.clone(),
explicit_delete: true,
transaction_guard: transaction_guard.cloned(),
namespace_owner: Some(Arc::new((
@@ -4789,6 +4790,27 @@ impl SetDisks {
.await?;
}
// Lock order: lifecycle, metadata transaction, then object namespace.
// No body reads occur below this point. The snapshot must describe the
// configuration at commit, not the pre-body advisory object lookup.
let mut commit_opts;
let opts = if opts.defer_put_object_lock_snapshot {
commit_opts = opts.clone();
let expected = source_bucket_incarnation_id
.ok_or_else(|| Error::other("destructive PUT is missing its bucket incarnation"))?;
let store = metadata_sys::object_store_in(&self.ctx).await?;
let snapshot = store.object_lock_config_snapshot_for_single_put(bucket).await?;
if !snapshot.is_valid_for_destructive_put(store.id, bucket, expected) {
return Err(Error::other("PUT bucket generation or configuration changed while staging"));
}
snapshot.add_lock_fences(&mut commit_opts);
commit_opts.object_lock_config_snapshot = Some(snapshot);
commit_opts.defer_put_object_lock_snapshot = false;
&commit_opts
} else {
opts
};
if let Some(store) = opts.decommission_capacity_admission.as_ref() {
#[cfg(test)]
{
@@ -4958,7 +4980,20 @@ impl SetDisks {
);
}
if let Some(version_id) = opts.version_id.as_deref()
// A write without a version id to a key that is not versioning-Enabled
// replaces the key's null version in place, so the WORM gate must judge
// that version too. The request-layer pre-check reads the latest
// version, which is not the one replaced when a UUID version or a
// delete marker sits above a retained null version.
let worm_gate_version_id = opts.version_id.clone().or_else(|| {
(!opts.versioned
&& opts
.object_lock_config_snapshot
.as_ref()
.is_some_and(|snapshot| !matches!(snapshot.state(), ObjectLockConfigState::ConfirmedAbsent)))
.then(|| Uuid::nil().to_string())
});
if let Some(version_id) = worm_gate_version_id.as_deref()
&& !is_meta_bucketname(bucket)
{
let explicit_version_lookup_started = rustfs_io_metrics::put_stage_timer();
@@ -4978,9 +5013,10 @@ impl SetDisks {
.await;
match current {
Ok(existing) => {
let object_lock_config = opts.object_lock_config_snapshot.as_deref().ok_or_else(|| {
Error::other("explicit-version PUT is missing its Object Lock configuration snapshot")
})?;
let object_lock_config = opts
.object_lock_config_snapshot
.as_deref()
.ok_or_else(|| Error::other("destructive PUT is missing its Object Lock configuration snapshot"))?;
// The WORM gate protects the locked version from local
// overwrites; an authorized replication write passes it
// only when the LWW merge below will judge every
@@ -5011,6 +5047,8 @@ impl SetDisks {
}
}
}
// A null delete marker has no protected payload to replace.
Err(StorageError::MethodNotAllowed) if version_id == Uuid::nil().to_string() => {}
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {}
Err(err) => return Err(err),
}
@@ -5266,6 +5304,7 @@ impl SetDisks {
let commit_decommission_object_lock_guard = decommission_object_lock_guard.take();
let commit_publication_guard = publication_commit_guard.take();
let commit_bucket_lifecycle_guard = bucket_lifecycle_guard.take();
let commit_object_lock_config_snapshot = opts.object_lock_config_snapshot.clone();
let commit_decommission_capacity_guard = decommission_capacity_guard.take();
let commit_scanner_publication_scope = opts.scanner_publication_commit_scope.clone();
// A scanner publication scope owns the movement permit until the
@@ -5328,6 +5367,7 @@ impl SetDisks {
let mut _decommission_object_lock_guard = commit_decommission_object_lock_guard;
let mut _publication_guard = commit_publication_guard;
let mut _bucket_lifecycle_guard = commit_bucket_lifecycle_guard;
let mut _object_lock_config_snapshot = commit_object_lock_config_snapshot;
let mut _decommission_capacity_guard = commit_decommission_capacity_guard;
let mut quota_reservation = quota_reservation;
let rename_stage_start = Instant::now();
@@ -5661,6 +5701,7 @@ impl SetDisks {
let object_lock_guard = _object_lock_guard.take();
let publication_guard = _publication_guard.take();
let bucket_lifecycle_guard = _bucket_lifecycle_guard.take();
let object_lock_config_snapshot = _object_lock_config_snapshot.take();
let decommission_object_lock_guard = _decommission_object_lock_guard.take();
let decommission_capacity_guard = _decommission_capacity_guard.take();
let cleanup_bucket = commit_bucket.clone();
@@ -5683,6 +5724,7 @@ impl SetDisks {
write_context,
publication_guard,
bucket_lifecycle_guard,
object_lock_config_snapshot,
decommission_object_lock_guard,
decommission_capacity_guard,
),
@@ -5704,6 +5746,7 @@ impl SetDisks {
write_context,
publication_guard,
bucket_lifecycle_guard,
object_lock_config_snapshot,
decommission_object_lock_guard,
decommission_capacity_guard,
),
@@ -5717,6 +5760,7 @@ impl SetDisks {
drop(write_context);
drop(publication_guard);
drop(bucket_lifecycle_guard);
drop(object_lock_config_snapshot);
cleanup_runtime::run(cleanup_reservation, async move {
let zero_target_cleanup = targets.is_empty();
rustfs_io_metrics::record_put_object_stage_duration(
@@ -9131,6 +9175,9 @@ impl SetDisks {
&disks,
write_quorum,
DeleteRecoveryContext {
work_budget: commit_context
.options
.and_then(|options| options.delete_recovery_work_budget.clone()),
recovery_ticket,
transaction_guard: commit_context.transaction_guard.clone(),
namespace_owner: namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>),
@@ -9304,6 +9351,60 @@ impl SetDisks {
}
impl SetDisks {
/// Judge the version a versionless destructive delete removes.
///
/// For a key that is not versioning-`Enabled`, a delete without a version
/// id removes the `null` version, while the preceding Object Lock check
/// judged the key's latest entry. When a UUID version or delete marker
/// sits above a retained `null` version, the two differ and the retained
/// version must be judged on its own. The caller holds the object's write
/// lock. Plain unversioned keys and versioning-`Enabled` keys keep a single
/// lookup: their latest entry is the version removed.
///
/// The lookup future is boxed from a synchronous wrapper so it leaves the
/// delete futures' frames: `delete_object` already nests the boxed batch
/// path, and inlining a second `get_object_info` overflowed the 4 MiB test
/// stack on Linux.
fn check_null_version_object_lock_delete<'a>(
&'a self,
bucket: &'a str,
object: &'a str,
latest: &'a ObjectInfo,
opts: &'a ObjectOptions,
) -> impl Future<Output = Result<()>> + Send + 'a {
Box::pin(self.check_null_version_object_lock_delete_inner(bucket, object, latest, opts))
}
async fn check_null_version_object_lock_delete_inner(
&self,
bucket: &str,
object: &str,
latest: &ObjectInfo,
opts: &ObjectOptions,
) -> Result<()> {
if is_meta_bucketname(bucket)
|| opts.version_id.is_some()
|| set_disk_delete_creates_delete_marker(opts)
|| opts.transition.expire_restored
|| latest.version_id.is_none_or(|version| version.is_nil())
{
return Ok(());
}
let null_opts = ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
no_lock: true,
metadata_cache_safe: false,
..opts.clone()
};
match self.get_object_info(bucket, object, &null_opts).await {
Ok(existing) => check_object_lock_delete(&self.ctx, bucket, object, &existing, &null_opts).await,
// A null delete marker has no protected payload to remove.
Err(StorageError::MethodNotAllowed) => Ok(()),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => Ok(()),
Err(err) => Err(err),
}
}
/// Read current metadata without cache admission or read-side repair. The
/// outer batch owner already holds the sorted namespace write-lock set.
pub(crate) async fn conditional_delete_current_snapshot(
@@ -9338,6 +9439,76 @@ where
futures::future::try_join_all(probes).await.map(|_| ()).map_err(Error::from)
}
/// Per-key source-read inputs for the bulk-delete admission loop. The bounded
/// metadata pre-fetch and the loop itself must resolve these identically, so
/// both sides derive them here.
struct BatchDeleteSourceInputs {
replication_object_name: String,
explicit_null_version: bool,
version_id: Option<Uuid>,
versioned: bool,
version_suspended: bool,
check_opts: ObjectOptions,
object_lock_check_required: bool,
replicate_delete: bool,
needs_source_read: bool,
}
fn batch_delete_source_inputs(
bucket: &str,
dobj: &ObjectToDelete,
opts: &ObjectOptions,
delete_config_snapshot: &DeleteReplicationConfigSnapshot,
object_lock_config: &Arc<ObjectLockConfigSnapshot>,
guarded: bool,
) -> BatchDeleteSourceInputs {
let replication_object_name = decode_dir_object(&dobj.object_name);
let explicit_null_version = is_explicit_null_version(dobj.version_id);
let version_id = delete_file_info_version_id(dobj.version_id);
let (versioned, version_suspended) = delete_config_snapshot
.versioning_config()
.delete_state(replication_object_name.as_str());
let check_opts = ObjectOptions {
version_id: dobj.version_id.map(|version_id| version_id.to_string()),
versioned,
version_suspended,
object_lock_delete: opts.object_lock_delete.clone(),
object_lock_config_snapshot: Some(Arc::clone(object_lock_config)),
no_lock: true,
suppress_read_repair: guarded,
..Default::default()
};
// A missing bucket Object Lock configuration does not prove that
// persisted object metadata is clean: migrated or corrupt data may
// still carry an explicit retention or legal hold. Only a real
// delete-marker creation can skip the object-level WORM check.
let object_lock_check_required = !is_meta_bucketname(bucket) && !set_disk_delete_creates_delete_marker(&check_opts);
let replicate_delete = delete_config_snapshot.has_active_rule(&replication_object_name);
let marker_delete = dobj.version_id.is_none() || dobj.synthetic_version_id;
let replication_needs_source = replicate_delete
&& (!marker_delete || delete_config_snapshot.active_delete_marker_rules_require_tags(&replication_object_name));
// Guarded explicit-version deletes are rejected before any read and must
// not join the pre-fetch wave.
let admission_error = guarded && dobj.version_id.is_some();
let needs_source_read = !admission_error
&& (object_lock_check_required
|| replication_needs_source
|| opts.tier_delete_journal_api.is_some()
|| dobj.expected_identity.is_some()
|| dobj.version_id.is_some());
BatchDeleteSourceInputs {
replication_object_name,
explicit_null_version,
version_id,
versioned,
version_suspended,
check_opts,
object_lock_check_required,
replicate_delete,
needs_source_read,
}
}
#[async_trait::async_trait]
impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
type Error = Error;
@@ -9570,9 +9741,26 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
if let Some(etag) = &src_info.etag {
replacement_metadata.insert("etag".to_owned(), etag.clone());
}
// ObjectInfo externalizes these fields; copy callers supply the effective
// values, including empty tags/absent expiry for explicit replacement.
if !src_info.user_tags.is_empty() {
replacement_metadata
.entry(rustfs_utils::http::headers::AMZ_OBJECT_TAGGING.to_string())
.or_insert_with(|| (*src_info.user_tags).clone());
}
if let Some(expires) = src_info.expires
&& !replacement_metadata.contains_key("expires")
{
replacement_metadata.insert(
"expires".to_string(),
expires
.format(&time::format_description::well_known::Rfc3339)
.map_err(Error::other)?,
);
}
fi.metadata = replacement_metadata.clone();
let mod_time = OffsetDateTime::now_utc();
let mod_time = dst_opts.mod_time.unwrap_or_else(OffsetDateTime::now_utc);
fi.mod_time = Some(mod_time);
fi.version_id = version_id;
fi.versioned = src_opts.versioned || src_opts.version_suspended;
@@ -9695,6 +9883,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
objects: Vec<ObjectToDelete>,
mut opts: ObjectOptions,
) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
opts.ensure_delete_recovery_work_budget();
let mut del_objects = vec![DeletedObject::default(); objects.len()];
let mut accounting = vec![None; objects.len()];
let delete_config_snapshot = opts
@@ -9736,7 +9925,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let unique_lock_count = batch.requests.len();
let mut failed_map = HashMap::new();
let mut _local_batch_guards: Vec<FastLockGuard> = Vec::with_capacity(batch.requests.len());
let mut local_batch_guards: Vec<FastLockGuard> = Vec::with_capacity(batch.requests.len());
let mut locked_objects = HashSet::new();
// Instance-scoped, not the ambient facade (backlog#1052) — see
@@ -9753,7 +9942,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
(failed_map, locked_objects, dist_batch_lock_ids) = self.acquire_dist_delete_object_locks_batch(&batch).await;
} else {
let batch_result = self.local_lock_manager.acquire_locks_batch(batch).await;
_local_batch_guards = batch_result.guards;
local_batch_guards = batch_result.guards;
for key in batch_result.successful_locks {
locked_objects.insert(key.object.as_ref().to_string());
@@ -9795,8 +9984,17 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
// Every raw batch lock is acquired first. Canonical sorting also
// gives aliases of one disk key exactly one transaction owner.
let keys: std::collections::BTreeSet<_> = locked_objects.iter().cloned().collect();
for key in keys {
match self.acquire_delete_transaction_lock(bucket, &key).await {
// Bulk delete is the only multi-key transaction-lock holder; one
// batched lock round replaces one quorum round-trip per key and is
// deadlock-free by construction (a single `lock_batch` RPC per lock
// client, global (shard, key) order at each server). Per-key
// failures keep the set layer's per-key error reporting.
let key_names: Vec<&str> = keys.iter().map(|key| key.as_str()).collect();
let acquired = self
.acquire_delete_transaction_locks_batch(bucket, &key_names, super::super::get_lock_acquire_timeout())
.await;
for (key, result) in keys.into_iter().zip(acquired) {
match result {
Ok(guard) => {
opts.add_namespace_lock_guard(&guard);
transaction_guards.insert(key, guard);
@@ -9832,6 +10030,54 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
},
}
};
// Per-key read inputs are resolved exactly once; the pre-fetch and the
// admission loop below consume this table so the two cannot disagree
// on eligibility or read options.
let mut source_inputs: Vec<Option<BatchDeleteSourceInputs>> = Vec::with_capacity(objects.len());
for dobj in &objects {
let guarded = current_generations.is_some_and(|generations| generations.contains_key(&dobj.object_name));
source_inputs.push(Some(batch_delete_source_inputs(
bucket,
dobj,
&opts,
&delete_config_snapshot,
&object_lock_config,
guarded,
)));
}
// Bounded-parallel pre-fetch of the source metadata each admitted key
// needs below. These reads are pure quorum lookups under the locks
// already held; sequencing one metadata fan-out per key made a
// 1000-key batch pay a full round of drive reads at a time.
const DELETE_OBJECTS_SOURCE_READ_CONCURRENCY: usize = 16;
let mut source_reads: HashMap<usize, (ObjectInfo, FileInfo, Option<Error>)> = HashMap::with_capacity(objects.len());
{
let mut pending = Vec::new();
for (i, source) in source_inputs.iter().enumerate() {
if del_errs[i].is_some() {
continue;
}
let Some(source) = source else {
continue;
};
if !source.needs_source_read {
continue;
}
let object_name = objects[i].object_name.clone();
let check_opts = &source.check_opts;
pending.push(async move {
let (goi, file_info, _write_quorum, gerr) = self
.get_object_info_fileinfo_and_quorum(bucket, &object_name, check_opts)
.await;
(i, (goi, file_info, gerr))
});
}
let reads = futures::stream::iter(pending)
.buffer_unordered(DELETE_OBJECTS_SOURCE_READ_CONCURRENCY)
.collect::<Vec<_>>()
.await;
source_reads.extend(reads);
}
let mut vers_map: HashMap<&String, FileInfoVersions> = HashMap::new();
let mut tier_reference_leases: Vec<(usize, String, Option<TierDestinationId>)> = Vec::new();
let mut tier_free_version_receipt_candidates: HashMap<usize, TierFreeVersionReceiptCandidate> = HashMap::new();
@@ -9853,41 +10099,30 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
));
continue;
}
let replication_object_name = decode_dir_object(&dobj.object_name);
let explicit_null_version = is_explicit_null_version(dobj.version_id);
let version_id = delete_file_info_version_id(dobj.version_id);
let (versioned, version_suspended) = delete_config_snapshot
.versioning_config()
.delete_state(replication_object_name.as_str());
let check_opts = ObjectOptions {
version_id: dobj.version_id.map(|version_id| version_id.to_string()),
let Some(source) = source_inputs[i].take() else {
continue;
};
let BatchDeleteSourceInputs {
replication_object_name,
explicit_null_version,
version_id,
versioned,
version_suspended,
object_lock_delete: opts.object_lock_delete.clone(),
object_lock_config_snapshot: Some(Arc::clone(&object_lock_config)),
no_lock: true,
suppress_read_repair: guarded,
..Default::default()
};
// A missing bucket Object Lock configuration does not prove that
// persisted object metadata is clean: migrated or corrupt data may
// still carry an explicit retention or legal hold. Only a real
// delete-marker creation can skip the object-level WORM check.
let object_lock_check_required = !is_meta_bucketname(bucket) && !set_disk_delete_creates_delete_marker(&check_opts);
let replicate_delete = delete_config_snapshot.has_active_rule(&replication_object_name);
let marker_delete = dobj.version_id.is_none() || dobj.synthetic_version_id;
let replication_needs_source = replicate_delete
&& (!marker_delete || delete_config_snapshot.active_delete_marker_rules_require_tags(&replication_object_name));
let (mut goi, mut authoritative_file_info, mut gerr) = if object_lock_check_required
|| replication_needs_source
|| opts.tier_delete_journal_api.is_some()
|| dobj.expected_identity.is_some()
|| dobj.version_id.is_some()
{
let (goi, file_info, _write_quorum, gerr) = self
.get_object_info_fileinfo_and_quorum(bucket, &dobj.object_name, &check_opts)
.await;
(goi, file_info, gerr)
check_opts,
object_lock_check_required,
replicate_delete,
needs_source_read,
} = source;
let (mut goi, mut authoritative_file_info, mut gerr) = if needs_source_read {
match source_reads.remove(&i) {
Some(read) => read,
None => {
let (goi, file_info, _write_quorum, gerr) = self
.get_object_info_fileinfo_and_quorum(bucket, &dobj.object_name, &check_opts)
.await;
(goi, file_info, gerr)
}
}
} else {
(ObjectInfo::default(), FileInfo::default(), None)
};
@@ -9904,11 +10139,23 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
&& !guarded
&& goi.name.is_empty()
{
// Recovery can restore or retire versions of this key
// (rolling back an interrupted prepared delete brings a
// retained version back). A pre-fetched read for a later
// duplicate of the same name would still report the stale
// miss and skip its object-lock check — bypassing retention —
// so evict those entries and let the loop re-read them live.
for (other_index, other) in objects.iter().enumerate() {
if other_index != i && other.object_name == dobj.object_name {
source_reads.remove(&other_index);
}
}
let recovery = self
.recover_delete_key(
bucket,
&dobj.object_name,
DeleteRecoveryContext {
work_budget: opts.delete_recovery_work_budget.clone(),
dry_run: !opts.no_lock,
transaction_guard: transaction_guards.get(lock_names[i].as_ref()).cloned(),
namespace_owner: opts.delete_mutation_owner.clone(),
@@ -10022,6 +10269,17 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
del_errs[i] = Some(err);
continue;
}
// Not gated on `source_missing`: with a delete marker on top the
// latest-version lookup reports the key as missing while the null
// version below it may still be retained.
if object_lock_check_required
&& let Err(err) = self
.check_null_version_object_lock_delete(bucket, &dobj.object_name, &goi, &check_opts)
.await
{
del_errs[i] = Some(err);
continue;
}
if opts.tier_delete_journal_api.is_some()
&& !source_missing
@@ -10477,6 +10735,10 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
// response can hide a persisted decision and must never roll back.
let groups = groups.collect::<Vec<_>>();
published_groups.extend(groups.iter().map(|index| vers[*index].name.clone()));
// Journal publish fans out per object per drive; keep enough
// publications in flight to feed the drives' durable-mutation
// lanes (8 workers each) across a 16-drive set.
const DELETE_JOURNAL_PUBLISH_CONCURRENCY: usize = 32;
let publications = futures::stream::iter(groups.into_iter().map(|index: usize| {
let group = &vers[index];
let recovery_ticket = recovery_tickets[index].clone();
@@ -10502,6 +10764,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
disks,
self.set_drive_count / 2 + 1,
DeleteRecoveryContext {
work_budget: opts.delete_recovery_work_budget.clone(),
recovery_ticket,
namespace_owner,
transaction_guard,
@@ -10512,7 +10775,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
(index, result)
}
}))
.buffer_unordered(16)
.buffer_unordered(DELETE_JOURNAL_PUBLISH_CONCURRENCY)
.collect::<Vec<_>>()
.await;
for (index, result) in publications {
@@ -10652,9 +10915,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
// Deferred recovery takes fresh raw and transaction locks. Discard the
// completed leases' deadlines while retaining every caller-owned fence.
opts.namespace_lock_fence = outer_namespace_lock_fence;
// Release inner owners before reacquiring any raw key lock.
// Release inner owners before reacquiring any raw key lock. The raw
// batch guards must go too: deferred recovery takes a fresh write lock
// on the same keys below, and a still-held local batch guard would
// deadlock it against itself (the distributed path releases its batch
// ids in the same region).
drop(namespace_owner);
drop(transaction_guards);
drop(local_batch_guards);
// TODO(backlog): support partial object deletion for multi-part objects
@@ -10680,6 +10948,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
bucket,
&object.object_name,
DeleteRecoveryContext {
work_budget: opts.delete_recovery_work_budget.clone(),
explicit_delete,
namespace_owner: Some(Arc::new((
self.ctx.begin_namespace_commit(),
@@ -10738,6 +11007,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
#[tracing::instrument(skip(self, opts))]
async fn delete_object(&self, bucket: &str, object: &str, mut opts: ObjectOptions) -> Result<ObjectInfo> {
opts.ensure_delete_recovery_work_budget();
let _scope_outcome_guard = opts
.scanner_publication_commit_scope
.clone()
@@ -11034,6 +11304,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
bucket,
object,
DeleteRecoveryContext {
work_budget: opts.delete_recovery_work_budget.clone(),
transaction_guard: transaction_guard.clone(),
namespace_owner: Some(Arc::new((
self.ctx.begin_namespace_commit(),
@@ -11094,6 +11365,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
opts.precondition_check(&goi)?;
check_object_lock_delete(&self.ctx, bucket, object, &goi, &opts).await?;
}
self.check_null_version_object_lock_delete(bucket, object, &goi, &opts)
.await?;
let delete_marker_purge_candidate = opts
.version_id
.as_deref()
@@ -11643,7 +11916,10 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
// as a second object header set: forwarding SSE intent or wrapped DEKs
// would request a second encryption pass and disclose local envelope
// material to the remote provider.
transition_meta.retain(|key, _| !rustfs_utils::http::is_replication_stripped_encryption_key(key));
transition_meta.retain(|key, _| {
!rustfs_utils::http::is_replication_stripped_encryption_key(key)
&& !rustfs_utils::http::is_snowball_transport_key(key)
});
transition_meta.insert("name".to_string(), object.to_string());
rustfs_utils::http::metadata_compat::insert_str(
&mut transition_meta,
@@ -16119,6 +16395,170 @@ mod metadata_mutation_generation_tests {
assert_retired(&set_disks, &metadata_key).await;
}
#[tokio::test]
async fn restore_metadata_copy_preserves_tags_expiry_and_version_order() {
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use s3s::dto::RestoreRequest;
use time::format_description::well_known::Rfc3339;
let (_temp_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "restore-metadata-preservation";
for disk in &disks {
disk.make_volume(bucket).await.expect("create bucket");
}
let old_time = OffsetDateTime::UNIX_EPOCH + time::Duration::days(1);
let latest_time = old_time + time::Duration::days(1);
let expires = latest_time + time::Duration::days(365);
for deleted in [false, true] {
let object = if deleted { "deleted" } else { "live" };
let tags = "classification=secret&owner=alice";
let old = set
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(b"old protected content".to_vec()),
&ObjectOptions {
versioned: true,
mod_time: Some(old_time),
write_completion: crate::object_api::WriteCompletion::TailDrained,
user_defined: HashMap::from([
(AMZ_OBJECT_TAGGING.to_string(), tags.to_string()),
("x-amz-tagging".to_string(), tags.to_string()),
("expires".to_string(), expires.format(&Rfc3339).expect("expiry")),
]),
..Default::default()
},
)
.await
.expect("write historical version");
let latest = if deleted {
let marker = FileInfo {
version_id: Some(Uuid::new_v4()),
mod_time: Some(latest_time),
deleted: true,
versioned: true,
..Default::default()
};
for disk in &disks {
disk.write_metadata(bucket, bucket, object, marker.clone())
.await
.expect("publish current delete marker");
}
marker.version_id
} else {
set.put_object(
bucket,
object,
&mut PutObjReader::from_vec(b"latest content".to_vec()),
&ObjectOptions {
versioned: true,
mod_time: Some(latest_time),
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("write latest version")
.version_id
};
let opts = ObjectOptions {
version_id: old.version_id.map(|id| id.to_string()),
versioned: true,
mod_time: Some(old_time),
..Default::default()
};
let mut source = set.get_object_info(bucket, object, &opts).await.expect("read old metadata");
assert_eq!(source.user_tags.as_str(), tags);
assert_eq!(source.expires, Some(expires));
assert!(!source.user_defined.contains_key(AMZ_OBJECT_TAGGING));
assert!(source.user_defined.contains_key("x-amz-tagging"));
source.metadata_only = true;
Arc::make_mut(&mut source.user_defined)
.insert(metadata_keys::RESTORE.to_string(), "ongoing-request=\"true\"".to_string());
set.copy_object(bucket, object, bucket, object, &mut source, &opts, &opts)
.await
.expect("restore metadata update");
let restored = set
.get_object_info(bucket, object, &opts)
.await
.expect("read restored metadata");
assert_eq!(restored.user_tags.as_str(), tags);
assert_eq!(restored.expires, Some(expires));
assert_eq!(restored.mod_time, Some(old_time));
for disk in &disks {
let bytes = disk
.read_all(bucket, &format!("{object}/xl.meta"))
.await
.expect("persisted metadata");
let meta = FileMeta::load(&bytes).expect("decode versions");
let versions = meta.list_versions(bucket, object, false).expect("list versions");
assert_eq!(versions.len(), 2);
assert_eq!(versions[0].version_id, latest);
assert_eq!(versions[0].mod_time, Some(latest_time));
assert_eq!(versions[0].deleted, deleted);
assert_eq!(versions[1].version_id, old.version_id);
assert_eq!(versions[1].mod_time, Some(old_time));
}
let restore_opts = put_restore_opts(bucket, object, &RestoreRequest::default(), &restored)
.await
.expect("background restore options");
assert_eq!(restore_opts.user_defined.get(AMZ_OBJECT_TAGGING).map(String::as_str), Some(tags));
assert_eq!(restore_opts.user_defined.get("expires"), Some(&expires.format(&Rfc3339).expect("expiry")));
assert_eq!(restore_opts.mod_time, Some(old_time));
}
}
#[tokio::test]
async fn metadata_copy_honors_explicit_tag_and_expiry_replacement() {
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use time::format_description::well_known::Rfc3339;
let (_temp_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "copy-metadata-replacement";
for disk in &disks {
disk.make_volume(bucket).await.expect("create bucket");
}
let old_time = OffsetDateTime::UNIX_EPOCH + time::Duration::days(1);
let expires = old_time + time::Duration::days(365);
set.put_object(
bucket,
"object",
&mut PutObjReader::from_vec(b"content".to_vec()),
&ObjectOptions {
mod_time: Some(old_time),
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("write original");
let opts = ObjectOptions::default();
let mut source = set.get_object_info(bucket, "object", &opts).await.expect("original metadata");
source.metadata_only = true;
source.user_tags = Arc::new("old=value".to_string());
source.expires = Some(expires - time::Duration::days(1));
let metadata = Arc::make_mut(&mut source.user_defined);
metadata.insert(AMZ_OBJECT_TAGGING.to_string(), "new=value".to_string());
metadata.insert("expires".to_string(), expires.format(&Rfc3339).expect("expiry"));
set.copy_object(bucket, "object", bucket, "object", &mut source, &opts, &opts)
.await
.expect("explicit metadata wins over externalized fields");
let mut copied = set.get_object_info(bucket, "object", &opts).await.expect("copied metadata");
assert_eq!(copied.user_tags.as_str(), "new=value");
assert_eq!(copied.expires, Some(expires));
assert!(copied.mod_time.is_some_and(|time| time > old_time));
copied.metadata_only = true;
copied.user_tags = Arc::new(String::new());
copied.expires = None;
set.copy_object(bucket, "object", bucket, "object", &mut copied, &opts, &opts)
.await
.expect("explicit empty replacement");
let cleared = set.get_object_info(bucket, "object", &opts).await.expect("cleared metadata");
assert!(cleared.user_tags.is_empty());
assert!(cleared.expires.is_none());
}
#[tokio::test]
async fn metadata_only_copy_preserves_valid_part_checksums_and_rejects_conflicting_aliases() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -16571,6 +17011,19 @@ mod transition_commit_failure_tests {
.collect();
user_defined.insert("x-amz-meta-owner".to_string(), "finance".to_string());
let snowball_keys = [
"snowball-auto-extract",
"x-amz-meta-snowball-auto-extract",
"x-amz-snowball-auto-extract",
"snowball-prefix",
"x-amz-meta-rustfs-snowball-prefix",
"x-amz-meta-minio-snowball-ignore-dirs",
"other-vendor-snowball-ignore-errors",
];
for key in snowball_keys {
user_defined.insert(key.to_string(), "true".to_string());
}
let mut reader = PutObjReader::from_vec(b"stored bytes the tier keeps opaque ".repeat(64));
set_disks
.put_object(
@@ -16620,6 +17073,10 @@ mod transition_commit_failure_tests {
.await
.expect("the tier must have received the object");
for key in snowball_keys {
assert!(!remote_metadata.contains_key(key), "tier must receive opaque bytes, not {key}");
assert_eq!(transitioned.user_defined.get(key).map(String::as_str), Some("true"));
}
for (key, _) in encryption_metadata {
assert!(
!remote_metadata.keys().any(|stored| stored.eq_ignore_ascii_case(key)),
@@ -24992,6 +25449,131 @@ mod delete_objects_lock_gating_tests {
.expect("plain object should be written");
}
/// Regression for the PR9155 review finding: a pre-fetched source-missing
/// read of a later duplicate must not skip the object-lock check after an
/// earlier entry's recovery rolls back an interrupted prepared delete and
/// restores a GOVERNANCE-retained version. Without the eviction the later
/// duplicate deletes the restored version, bypassing retention.
#[tokio::test]
async fn duplicate_entries_recheck_object_lock_after_recovery_restores_retained_version() {
let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "recovery-retention-dup";
let object = "retained.bin";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create bucket");
}
// A versioned put, so the batch can address the version explicitly.
let mut reader = PutObjReader::from_vec(vec![7u8; 256]);
set_disks
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
versioned: true,
no_lock: true,
..Default::default()
},
)
.await
.expect("versioned object should be written");
let version = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("read back the version identity")
.version_id
.expect("versioned put assigns a version id");
// GOVERNANCE retention on that version, persisted the way
// PutObjectRetention persists it.
let retain_until = OffsetDateTime::now_utc() + time::Duration::days(60);
set_disks
.put_object_metadata(
bucket,
object,
&ObjectOptions {
no_lock: true,
version_id: Some(version.to_string()),
eval_metadata: Some(HashMap::from([
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string()),
(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until
.format(&time::format_description::well_known::Rfc3339)
.expect("retention date formats"),
),
])),
..Default::default()
},
)
.await
.expect("governance retention should persist");
// Interrupted authorized delete: the recoverable per-disk delete
// applies the version removal and leaves the prepared journal
// undecided — exactly a crash between apply and publish.
let transaction = Uuid::new_v4();
for disk in &disk_stores {
let fi = disk
.read_version("", bucket, object, &version.to_string(), &crate::disk::ReadOptions::default())
.await
.expect("version exists before the interrupted delete");
let errors = disk
.delete_versions_recoverable(
bucket,
vec![FileInfoVersions {
name: object.to_string(),
versions: vec![fi],
..Default::default()
}],
DeleteOptions {
recoverable_delete: true,
old_data_dir: Some(transaction),
..Default::default()
},
)
.await;
assert!(errors.iter().all(Option::is_none), "the plant must apply: {errors:?}");
}
// A no-bypass batch naming the retained version twice: the first
// entry rolls the prepared delete back (restoring the retained
// version) and must be denied; the second must re-read and stay
// denied too instead of reusing the stale pre-fetched miss.
let victim = ObjectToDelete {
object_name: object.to_string(),
version_id: Some(version),
..Default::default()
};
let (_deleted, errors) = set_disks
.delete_objects(
bucket,
vec![victim.clone(), victim],
ObjectOptions {
versioned: true,
object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(
ObjectLockConfigState::ConfirmedAbsent,
))),
..Default::default()
},
)
.await;
assert!(
errors.iter().all(|error| error
.as_ref()
.is_some_and(|error| matches!(error, StorageError::PrefixAccessDenied(_, _)))),
"both duplicates must be denied by the restored retention: {errors:?}"
);
// The retained version survives the batch.
for disk in &disk_stores {
disk.read_version("", bucket, object, &version.to_string(), &crate::disk::ReadOptions::default())
.await
.expect("the retained version must survive the batch");
}
}
#[tokio::test]
#[serial_test::serial(shard_integrity_rollout)]
async fn conditional_delete_snapshot_preserves_multipart_etag() {
@@ -52,8 +52,8 @@ pub(crate) mod list {
pub(crate) mod multipart {
pub(crate) use rustfs_storage_api::{
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations, MultipartUploadInspection,
MultipartUploadResult, PartInfo,
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartCompletionAuthorization, MultipartInfo, MultipartOperations,
MultipartUploadInspection, MultipartUploadResult, PartInfo,
};
}
+291 -36
View File
@@ -14,7 +14,7 @@
use super::*;
use crate::bucket::{
metadata::{BUCKET_TABLE_RESERVED_PREFIX, BUCKET_TAGGING_CONFIG, ConfigState, table_bucket_catalog_metadata_prefix},
metadata::{BUCKET_TABLE_RESERVED_PREFIX, BUCKET_TAGGING_CONFIG, ConfigState},
utils::is_meta_bucketname,
};
use crate::error::is_err_bucket_not_found;
@@ -145,11 +145,41 @@ async fn table_bucket_enabled_in(ctx: &crate::runtime::instance::InstanceContext
}
}
fn bucket_delete_metadata_cleanup_prefixes(bucket: &str) -> [String; 2] {
[
table_bucket_catalog_metadata_prefix(bucket),
format!("{BUCKET_META_PREFIX}/{bucket}"),
]
fn bucket_delete_metadata_cleanup_prefixes(bucket: &str) -> Vec<String> {
use crate::bucket::metadata::*;
let mut prefixes = vec![table_bucket_catalog_metadata_prefix(bucket)];
let root = format!("{BUCKET_META_PREFIX}/{bucket}");
if bucket != "ahm-replacement" {
prefixes.push(root);
return prefixes;
}
// This valid user bucket shares its metadata directory with raw drive
// replacement intents, seals and recovery proofs. Delete only bucket-owned
// paths, including legacy standalone configuration, never the shared root.
// S3 objects live in the separate user volume; configuration APIs write the
// aggregate metadata file, so clients cannot populate arbitrary siblings.
prefixes.extend(
[
BUCKET_METADATA_FILE,
BUCKET_INCARNATION_FILE,
BUCKET_POLICY_CONFIG,
BUCKET_QUOTA_CONFIG_FILE,
BUCKET_TARGETS_FILE,
BUCKET_ACL_CONFIG,
BUCKET_TABLE_CONFIG,
BUCKET_DURABILITY_CONFIG,
BUCKET_ON_DEMAND_MIGRATION_CONFIG,
".replication",
"integrity-jobs",
crate::data_usage::DATA_USAGE_CACHE_NAME,
]
.into_iter()
.chain(XML_BUCKET_CONFIG_FILES)
.map(|file| format!("{root}/{file}")),
);
prefixes
}
fn bucket_deleted_marker_prefix(bucket: &str) -> String {
@@ -160,41 +190,48 @@ fn bucket_deleted_marker_volume(bucket: &str) -> String {
format!("{RUSTFS_META_BUCKET}/{}", bucket_deleted_marker_prefix(bucket))
}
pub(crate) async fn await_bucket_namespace_operation<T, F>(
guard: Option<&rustfs_lock::NamespaceLockGuard>,
bucket: &str,
// Box the child before constructing either lease wrapper. Keeping F inline in
// nested select futures multiplies poll-frame temporaries along DELETE -> quota
// cleanup -> metadata GET. This boundary must not inline back into that chain.
#[inline(never)]
pub(crate) fn await_bucket_namespace_operation<'a, T: 'a, F>(
guard: Option<&'a rustfs_lock::NamespaceLockGuard>,
bucket: &'a str,
operation: &'static str,
future: F,
) -> Result<T>
) -> impl Future<Output = Result<T>> + 'a
where
F: Future<Output = Result<T>>,
F: Future<Output = Result<T>> + 'a,
{
let Some(guard) = guard else {
return future.await;
};
if guard.is_lock_lost() {
return Err(StorageError::other(format!(
"bucket namespace lock was lost before {operation}: {bucket}"
)));
}
tokio::select! {
biased;
_ = guard.lock_lost_notified() => Err(StorageError::other(format!(
"bucket namespace lock was lost during {operation}: {bucket}"
))),
result = future => result,
let future = Box::pin(future);
async move {
let Some(guard) = guard else {
return future.await;
};
if guard.is_lock_lost() {
return Err(StorageError::other(format!(
"bucket namespace lock was lost before {operation}: {bucket}"
)));
}
tokio::select! {
biased;
_ = guard.lock_lost_notified() => Err(StorageError::other(format!(
"bucket namespace lock was lost during {operation}: {bucket}"
))),
result = future => result,
}
}
}
async fn await_bucket_lifecycle_operation<T, F>(
lifecycle_guard: Option<&rustfs_lock::NamespaceLockGuard>,
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
bucket: &str,
fn await_bucket_lifecycle_operation<'a, T: 'a, F>(
lifecycle_guard: Option<&'a rustfs_lock::NamespaceLockGuard>,
namespace_guard: Option<&'a rustfs_lock::NamespaceLockGuard>,
bucket: &'a str,
operation: &'static str,
future: F,
) -> Result<T>
) -> impl Future<Output = Result<T>> + 'a
where
F: Future<Output = Result<T>>,
F: Future<Output = Result<T>> + 'a,
{
await_bucket_namespace_operation(
lifecycle_guard,
@@ -202,7 +239,6 @@ where
operation,
await_bucket_namespace_operation(namespace_guard, bucket, operation, future),
)
.await
}
fn bucket_recovery_lock_unavailable(bucket: &str, object: &str) -> StorageError {
@@ -478,10 +514,18 @@ impl ECStore {
let mut tags = HashMap::with_capacity(tagging.tag_set.len());
for tag in &tagging.tag_set {
let (Some(key), Some(value)) = (&tag.key, &tag.value) else {
return Err(Error::other("stored bucket tag is missing a key or value"));
return Err(crate::bucket::metadata::unreadable_config_error(
bucket,
BUCKET_TAGGING_CONFIG,
metadata.tagging_config_xml.len(),
));
};
if key.is_empty() || tags.insert(key.clone(), value.clone()).is_some() {
return Err(Error::other("stored bucket tags contain empty or duplicate keys"));
return Err(crate::bucket::metadata::unreadable_config_error(
bucket,
BUCKET_TAGGING_CONFIG,
metadata.tagging_config_xml.len(),
));
}
}
Ok(tags)
@@ -1673,7 +1717,17 @@ impl ECStore {
ns_guard.as_ref(),
bucket,
"bucket retirement publication",
crate::bucket::retirement::commit_retirement(store, bucket, incarnation, &record_opts),
crate::bucket::retirement::commit_retirement(store.clone(), bucket, incarnation, &record_opts),
)
.await?;
// The lifecycle fence excludes live reservation owners. A crash-left
// V1 ledger must not block quota admission after same-name recreation.
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
bucket,
"retired bucket quota ledger cleanup",
crate::bucket::quota::reservation::cleanup_retired_bucket_ledger(store, bucket, incarnation),
)
.await?;
}
@@ -1824,6 +1878,20 @@ mod tests {
assert!(!xlmeta.diagnostic_truncated);
}
#[tokio::test]
async fn bucket_namespace_operation_keeps_nested_future_state_off_the_poll_stack() {
let payload = std::hint::black_box([7u8; 128 * 1024]);
let operation = super::await_bucket_lifecycle_operation(None, None, "bucket", "large metadata operation", async move {
tokio::task::yield_now().await;
Ok(std::hint::black_box(payload)[0])
});
assert!(
std::mem::size_of_val(&operation) < 1024,
"nested lease wrappers must not retain the concrete storage future inline"
);
assert_eq!(operation.await.expect("the boxed operation must still execute"), 7);
}
#[tokio::test(start_paused = true)]
async fn bucket_namespace_operation_fails_closed_after_lease_expiry() {
let ttl = Duration::from_millis(20);
@@ -2240,7 +2308,13 @@ mod tests {
let result = store.get_bucket_tags_for_policy(bucket).await;
match expected {
Some(tags) => assert_eq!(&result.expect("valid tag configuration"), tags),
None => assert!(result.is_err(), "corrupt tags must not become an untagged bucket: {xml:?}"),
None => assert!(
crate::bucket::metadata::unreadable_config_refusal(
&result.expect_err("corrupt tags must not become an untagged bucket")
)
.is_some_and(|failure| failure.config_file == super::BUCKET_TAGGING_CONFIG),
"corrupt tags must carry a typed tagging refusal: {xml:?}"
),
}
}
}
@@ -2442,6 +2516,187 @@ mod tests {
assert!(prefixes.contains(&"buckets/analytics".to_string()));
}
#[tokio::test]
#[serial]
async fn replacement_named_bucket_delete_preserves_recovery_and_recreates_clean_metadata() {
use crate::bucket::metadata::{BUCKET_QUOTA_CONFIG_FILE, BUCKET_TAGGING_CONFIG, BUCKET_VERSIONING_CONFIG};
let (_temp, store) = setup_bucket_quorum_test_env(&[4], None).await;
metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "ahm-replacement";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create legal bucket");
let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
for (config, bytes) in [
(BUCKET_QUOTA_CONFIG_FILE, br#"{"quota":1024,"quota_type":"Hard"}"#.as_slice()),
(
BUCKET_TAGGING_CONFIG,
b"<Tagging><TagSet><Tag><Key>old</Key><Value>value</Value></Tag></TagSet></Tagging>".as_slice(),
),
(
BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".as_slice(),
),
] {
metadata_sys::update_in(&store.ctx, bucket, config, bytes.to_vec())
.await
.expect("persist old configuration");
}
let ledger_path = format!("config/quota-ledger/{bucket}.json");
// V1 precharges outstanding growth: four live bytes plus the
// crash-left reservation's 100 bytes must already be accounted.
let ledger = serde_json::to_vec(&serde_json::json!({
"version": 1, "bucket_incarnation": old_incarnation,
"quota_revision_unix_nanos": 0, "accounted_usage": 104,
"reservations": {Uuid::new_v4().to_string(): {
"object": "crashed-put", "old_size": 0, "new_size": 100,
"created_at": 0, "pool_index": 0, "set_index": 0, "commit_started": false
}}, "reconcile_required": true
}))
.expect("encode crash-left ledger");
// The storage API requires the same legacy-quota admission supplied
// by the S3 handler, even for this small fixture object.
let mut live_options = ObjectOptions {
versioned: true,
..Default::default()
};
assert!(live_options.set_quota_admission(0, 1024));
// Failed deletion must leave both the live incarnation and its ledger alone.
store
.put_object(bucket, "live", &mut PutObjReader::from_vec(b"live".to_vec()), &live_options)
.await
.expect("write live object");
crate::config::com::save_config(store.clone(), &ledger_path, ledger.clone())
.await
.expect("persist old ledger");
assert!(matches!(
store.delete_bucket(bucket, &DeleteBucketOptions::default()).await,
Err(StorageError::BucketNotEmpty(_))
));
assert_eq!(
crate::config::com::read_config(store.clone(), &ledger_path)
.await
.expect("live ledger remains"),
ledger
);
// Remove every version; the configured bucket is otherwise still empty.
let versions = store
.clone()
.list_object_versions(bucket, "", None, None, None, 100)
.await
.expect("list live versions");
for object in versions.objects {
store
.delete_object(
bucket,
&object.name,
ObjectOptions {
version_id: object.version_id.map(|id| id.to_string()),
..Default::default()
},
)
.await
.expect("remove live version");
}
let root = format!("{BUCKET_META_PREFIX}/{bucket}");
let task = Uuid::new_v4();
let recovery_names = [
"ahm_replacement_intent.json",
"ahm_replacement_intent_seal",
"ahm_replacement_completion_proof.json",
]
.map(|suffix| format!("{task}_{suffix}"));
let disk_paths = store.pools[0].disk_set[0]
.disks
.read()
.await
.iter()
.flatten()
.map(|disk| PathBuf::from(disk.endpoint().to_string()))
.collect::<Vec<_>>();
for disk in &disk_paths {
let directory = disk.join(RUSTFS_META_BUCKET).join(&root);
for name in &recovery_names {
tokio::fs::write(directory.join(name), b"durable heal record")
.await
.expect("seed raw recovery record");
}
// Legacy standalone settings and bucket-owned subtrees must not survive.
for name in [
"quota.json",
"versioning.xml",
"policy.json",
"bucket-targets.json",
"integrity-jobs/job.json",
".replication/resync.bin",
".usage-cache.bin",
] {
let path = directory.join(name);
tokio::fs::create_dir_all(path.parent().expect("fixture parent"))
.await
.expect("legacy parent");
tokio::fs::write(path, b"old bucket configuration")
.await
.expect("seed legacy configuration");
}
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete legal same-name bucket");
assert!(matches!(
crate::config::com::read_config(store.clone(), &ledger_path).await,
Err(StorageError::ConfigNotFound)
));
for disk in &disk_paths {
let directory = disk.join(RUSTFS_META_BUCKET).join(&root);
for name in &recovery_names {
assert_eq!(
tokio::fs::read(directory.join(name)).await.expect("recovery record survives"),
b"durable heal record"
);
}
for prefix in bucket_delete_metadata_cleanup_prefixes(bucket)
.into_iter()
.filter(|p| p.starts_with(&root))
{
assert!(
!disk.join(RUSTFS_META_BUCKET).join(prefix).exists(),
"bucket-owned metadata must be removed"
);
}
}
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate legal bucket");
let fresh = metadata_sys::get_in(&store.ctx, bucket).await.expect("fresh metadata");
assert_ne!(fresh.bucket_incarnation_id, old_incarnation);
assert!(fresh.quota_config_json.is_empty());
assert!(fresh.tagging_config_xml.is_empty());
assert!(fresh.versioning_config_xml.is_empty());
assert!(fresh.policy_config_json.is_empty());
// The strict purge/recovery cleanup must share the same ownership boundary.
store
.cleanup_deleted_bucket_metadata_strict(bucket, true)
.await
.expect("strict metadata cleanup");
for disk in &disk_paths {
for name in &recovery_names {
assert_eq!(
tokio::fs::read(disk.join(RUSTFS_META_BUCKET).join(&root).join(name))
.await
.expect("strict cleanup preserves recovery"),
b"durable heal record"
);
}
}
}
#[test]
fn bucket_delete_marker_path_uses_internal_deleted_bucket_metadata_prefix() {
assert_eq!(bucket_deleted_marker_prefix("analytics"), "buckets/.deleted/analytics");
+140 -5
View File
@@ -368,7 +368,7 @@ impl ECStore {
Some(expected) => expected,
None => self.bucket_incarnation_id_from_disk(bucket).await?,
};
let object = object.map(str::to_owned);
let object = object.map(encode_dir_object);
let prefix = prefix.map(str::to_owned);
self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
@@ -430,7 +430,7 @@ impl ECStore {
HealItemType::Bucket
}
.to_string(),
object: object.or(prefix).unwrap_or_default(),
object: object.map(|object| decode_dir_object(&object)).or(prefix).unwrap_or_default(),
bucket,
detail: format!(
"delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}, pending={}, failed={}",
@@ -656,12 +656,48 @@ impl ECStore {
Ok(Arc::ptr_eq(&target_set, &pool.get_disks_by_key(POOL_META_NAME)))
}
/// Read the published bucket identity without fabricating or migrating metadata.
/// Absence is distinct from quorum, corruption and transaction-fence failures.
pub async fn replacement_bucket_incarnation_from_disk(self: &Arc<Self>, bucket: &str) -> Result<Option<uuid::Uuid>> {
let transaction =
crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let incarnation = crate::bucket::metadata::load_bucket_incarnation(self.clone(), bucket).await?;
if transaction.is_lock_lost() {
return Err(Error::PreconditionFailed);
}
Ok(incarnation)
}
/// Restore the bucket configuration and incarnation before retiring a replacement marker.
pub async fn heal_replacement_bucket_metadata(
self: &Arc<Self>,
bucket: &str,
opts: &HealOpts,
targets: &[String],
) -> Result<()> {
// Reject an invalid request before touching the drives so callers keep
// the PreconditionFailed contract for bad buckets and options.
if opts.pool.is_none()
|| opts.set.is_none()
|| targets.is_empty()
|| opts.dry_run
|| opts.no_lock
|| check_valid_bucket_name_strict(bucket).is_err()
{
return Err(Error::PreconditionFailed);
}
let expected = self.bucket_incarnation_id_from_disk(bucket).await?;
self.heal_replacement_bucket_metadata_at_incarnation(bucket, expected, opts, targets)
.await
}
/// Keep a replacement pass bound to its admitted bucket through metadata commit.
pub async fn heal_replacement_bucket_metadata_at_incarnation(
self: &Arc<Self>,
bucket: &str,
expected: uuid::Uuid,
opts: &HealOpts,
targets: &[String],
) -> Result<()> {
use crate::bucket::metadata::{BUCKET_INCARNATION_FILE, BUCKET_METADATA_FILE};
use crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in;
@@ -698,9 +734,8 @@ impl ECStore {
// Read the persisted identity without lazy migration: a replacement
// must never turn lost configuration into newly fabricated defaults.
let incarnation = self.bucket_incarnation_id_from_disk(bucket).await?;
let targets = targets.to_vec();
self.run_bucket_heal_at_incarnation(bucket, incarnation, opts, move |store, bucket, opts| async move {
self.run_bucket_heal_at_incarnation(bucket, expected, opts, move |store, bucket, opts| async move {
// Match config writers: lifecycle, metadata transaction, object locks.
// Keep the transaction stable through target readback, including when
// the caller is cancelled and the storage owner finishes its write.
@@ -1022,6 +1057,7 @@ impl ECStore {
set_index: usize,
targets: &[String],
) -> Result<bool> {
let object = encode_dir_object(object);
let pool = self
.pools
.get(pool_index)
@@ -1034,7 +1070,7 @@ impl ECStore {
)
})?;
set.replacement_targets_have_version(bucket, object, version_id, targets)
set.replacement_targets_have_version(bucket, &object, version_id, targets)
.await
.map_err(Into::into)
}
@@ -2859,6 +2895,105 @@ mod tests {
}
}
#[tokio::test]
#[serial_test::serial]
async fn replacement_directory_object_readback_and_exact_recovery_use_encoded_key() {
let (_root, store, shutdown) = heal_store_with_pool_count(1).await;
let bucket = "directory-replacement-readback";
let object = "folder/";
let encoded = encode_dir_object(object);
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
store
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(Vec::new()),
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("write directory object through the public store entry");
let set = &store.pools[0].disk_set[0];
let targets = vec![set.set_endpoints[0].to_string()];
assert!(
store
.replacement_targets_have_version(bucket, object, "", 0, 0, &targets)
.await
.expect("readback")
);
let disk = set.disks.read().await[0].clone().expect("target disk");
disk.delete(
bucket,
&encoded,
DeleteOptions {
recursive: true,
immediate: true,
..Default::default()
},
)
.await
.expect("remove target directory shard");
assert!(
!store
.replacement_targets_have_version(bucket, object, "", 0, 0, &targets)
.await
.expect("missing target")
);
let opts = HealOpts {
pool: Some(0),
set: Some(0),
..Default::default()
};
let (result, error) = store
.heal_object(bucket, object, "", &opts)
.await
.expect("heal directory object");
assert!(error.is_none(), "heal failed: {error:?}");
assert_eq!(result.object, object);
assert!(
store
.replacement_targets_have_version(bucket, object, "", 0, 0, &targets)
.await
.expect("healed readback")
);
assert!(
store
.replacement_targets_have_version(bucket, &encoded, "", 0, 0, &targets)
.await
.expect("encoded readback")
);
// Exact-object recovery must inspect the same physical namespace and
// reject corrupt proof rather than silently report no transactions.
let metadata = disk
.read_all(bucket, &format!("{encoded}/xl.meta"))
.await
.expect("healed metadata");
disk.write_all(
bucket,
&format!("{encoded}/{}/delete-prepared", Uuid::new_v4()),
b"invalid proof".to_vec().into(),
)
.await
.expect("seed corrupt recovery proof");
store
.recover_delete_transactions_at_incarnation(bucket, Some(object), None, &HealOpts { dry_run: true, ..opts })
.await
.expect_err("directory recovery must discover and reject the corrupt proof");
assert_eq!(
disk.read_all(bucket, &format!("{encoded}/xl.meta"))
.await
.expect("unchanged metadata"),
metadata
);
shutdown.cancel();
}
#[tokio::test]
async fn replacement_bucket_metadata_respects_pool_ownership_and_required_records() {
let (root, store, shutdown) = multi_pool_heal_store().await;
+298
View File
@@ -5155,6 +5155,11 @@ mod tests {
let offline_disk = force_set_disk_range_offline_for_test(set, 0..1).await;
let _without_proof = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none());
if !sharded {
crate::bucket::quota::reservation::initialize_quota_ledger_for_test(Arc::clone(&store), &bucket)
.await
.expect("bootstrap quota with one disk offline");
}
let payload = vec![0x51; 4096];
let mut reader = PutObjReader::from_vec(payload.clone());
@@ -21007,6 +21012,299 @@ mod tests {
);
}
async fn seed_retained_null_version(
store: &Arc<crate::store::ECStore>,
bucket: &str,
object: &str,
protection: &str,
latest_is_marker: bool,
) {
let metadata = if protection == "legal-hold" {
HashMap::from([("x-amz-object-lock-legal-hold".to_string(), "ON".to_string())])
} else {
HashMap::from([
("x-amz-object-lock-mode".to_string(), protection.to_string()),
("x-amz-object-lock-retain-until-date".to_string(), "2099-01-01T00:00:00Z".to_string()),
])
};
let mut original = PutObjReader::from_vec(b"protected null version".to_vec());
store
.put_object(
bucket,
object,
&mut original,
&ObjectOptions {
user_defined: metadata,
mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(1)),
..Default::default()
},
)
.await
.expect("seed retained null version");
let mut latest = PutObjReader::from_vec(b"unprotected latest version".to_vec());
store
.put_object(
bucket,
object,
&mut latest,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("seed newer unprotected UUID version");
if latest_is_marker {
store
.delete_object(
bucket,
object,
ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("create a UUID delete marker above the retained null version");
}
}
async fn assert_retained_null_version_body(store: &Arc<crate::store::ECStore>, bucket: &str, object: &str) {
let mut reader = store
.get_object_reader(
bucket,
object,
None,
HeaderMap::new(),
&ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
..Default::default()
},
)
.await
.expect("retained null version must remain readable");
let mut body = Vec::new();
reader
.stream
.read_to_end(&mut body)
.await
.expect("read retained null version");
assert_eq!(body, b"protected null version");
}
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn null_version_object_lock_delete_checks_actual_target() {
use crate::bucket::replication::DeleteReplicationConfigSnapshot;
use s3s::dto::{BucketVersioningStatus, ExcludedPrefix, VersioningConfiguration};
let temp = tempfile::tempdir().expect("create null-version delete store");
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp.path(), "null-version-delete-lock", &[4])).await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("null-delete-lock-{}", Uuid::new_v4());
store
.make_bucket(
&bucket,
&MakeBucketOptions {
lock_enabled: true,
..Default::default()
},
)
.await
.expect("create Object Lock bucket");
let configs = [
VersioningConfiguration::default(),
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)),
..Default::default()
},
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
..Default::default()
},
];
for (case, config) in configs.into_iter().enumerate() {
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(config, None));
for protection in ["COMPLIANCE", "GOVERNANCE", "legal-hold"] {
for latest_is_marker in [false, true] {
let object = format!("archive/{case}-{protection}-{latest_is_marker}");
seed_retained_null_version(&store, &bucket, &object, protection, latest_is_marker).await;
let (_, errors) = store
.delete_objects(
&bucket,
vec![ObjectToDelete {
object_name: object.clone(),
..Default::default()
}],
ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&snapshot)),
..Default::default()
},
)
.await;
assert!(
matches!(errors.as_slice(), [Some(StorageError::PrefixAccessDenied(_, _))]),
"batch delete must judge the retained null version: {object}: {errors:?}"
);
let error = store
.delete_object(
&bucket,
&object,
ObjectOptions {
version_suspended: true,
..Default::default()
},
)
.await
.expect_err("suspended single delete must also judge the retained null version");
assert!(matches!(error, StorageError::PrefixAccessDenied(_, _)), "{object}: {error:?}");
assert_retained_null_version_body(&store, &bucket, &object).await;
if protection == "GOVERNANCE" {
let (_, errors) = store
.delete_objects(
&bucket,
vec![ObjectToDelete {
object_name: object.clone(),
..Default::default()
}],
ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&snapshot)),
object_lock_delete: Some(crate::storage_api_contracts::object::ObjectLockDeleteOptions {
bypass_governance: true,
}),
..Default::default()
},
)
.await;
assert!(errors.iter().all(Option::is_none), "authorized governance bypass: {errors:?}");
let removed = store
.get_object_info(
&bucket,
&object,
&ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
..Default::default()
},
)
.await;
match removed {
Ok(marker) => {
assert!(marker.delete_marker, "only a null delete marker may replace the payload");
assert_eq!(marker.size, 0);
assert!(marker.data_dir.is_none());
}
Err(error) => assert!(
matches!(
error,
StorageError::MethodNotAllowed
| StorageError::ObjectNotFound(_, _)
| StorageError::VersionNotFound(_, _, _)
),
"unexpected bypass result: {error:?}"
),
}
assert!(
store
.get_object_reader(
&bucket,
&object,
None,
HeaderMap::new(),
&ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
..Default::default()
}
)
.await
.is_err(),
"authorized deletion must make the retained null payload unreadable"
);
}
}
}
}
}
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn null_version_object_lock_put_and_multipart_preserve_retained_target() {
let temp = tempfile::tempdir().expect("create null-version overwrite store");
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp.path(), "null-version-overwrite-lock", &[4])).await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("null-overwrite-lock-{}", Uuid::new_v4());
store
.make_bucket(
&bucket,
&MakeBucketOptions {
lock_enabled: true,
..Default::default()
},
)
.await
.expect("create Object Lock bucket");
for protection in ["COMPLIANCE", "GOVERNANCE", "legal-hold"] {
for latest_is_marker in [false, true] {
let object = format!("{protection}-{latest_is_marker}");
seed_retained_null_version(&store, &bucket, &object, protection, latest_is_marker).await;
let mut replacement = PutObjReader::from_vec(b"replacement".to_vec());
let error = store
.put_object(&bucket, &object, &mut replacement, &ObjectOptions::default())
.await
.expect_err("PUT must reject replacement of a retained null version");
assert!(matches!(error, StorageError::PrefixAccessDenied(_, _)), "{object}: {error:?}");
let upload = store
.clone()
.new_multipart_upload(&bucket, &object, &ObjectOptions::default())
.await
.expect("initiate multipart upload");
let mut part_reader = PutObjReader::from_vec(b"multipart replacement".to_vec());
let part = store
.clone()
.put_object_part(&bucket, &object, &upload.upload_id, 1, &mut part_reader, &ObjectOptions::default())
.await
.expect("stage multipart replacement");
let error = store
.clone()
.complete_multipart_upload(
&bucket,
&object,
&upload.upload_id,
vec![crate::storage_api_contracts::multipart::CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
}],
&ObjectOptions::default(),
)
.await
.expect_err("multipart completion must reject replacement of a retained null version");
assert!(matches!(error, StorageError::PrefixAccessDenied(_, _)), "{object}: {error:?}");
assert_retained_null_version_body(&store, &bucket, &object).await;
let mut next_version = PutObjReader::from_vec(b"new UUID version".to_vec());
store
.put_object(
&bucket,
&object,
&mut next_version,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("a new UUID version must remain allowed");
assert_retained_null_version_body(&store, &bucket, &object).await;
}
}
}
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn exact_version_put_uses_only_its_own_bucket_object_lock_snapshot() {
File diff suppressed because it is too large Load Diff
-2
View File
@@ -423,8 +423,6 @@ fn should_enqueue_transition_immediately(oi: &ObjectInfo) -> bool {
!is_meta_bucketname(&oi.bucket)
}
const MAX_UPLOADS_LIST: usize = 10000;
mod bucket;
mod bucket_fence;
pub(crate) use bucket::await_bucket_namespace_operation;
+422 -15
View File
@@ -273,6 +273,28 @@ async fn pause_data_movement_multipart_before_selected_completion(bucket: &str)
}
}
async fn pool_has_multipart_upload_for_key(
pool: &crate::core::sets::Sets,
bucket: &str,
object: &str,
expected_incarnation_id: Option<Uuid>,
) -> Result<bool> {
let results = stream::iter(pool.disk_set.iter().cloned())
.map(|set| async move {
set.has_multipart_upload_for_key(bucket, object, expected_incarnation_id)
.await
})
.buffer_unordered(MULTIPART_LIST_SET_CONCURRENCY)
.collect::<Vec<_>>()
.await;
let mut found = false;
for result in results {
// Preserve errors from every set instead of hiding them behind a hit.
found |= result?;
}
Ok(found)
}
async fn list_pool_multipart_uploads_for_incarnation(
pool: &crate::core::sets::Sets,
bucket: &str,
@@ -490,6 +512,8 @@ impl ECStore {
opts: &ObjectOptions,
) -> Result<ListPartsInfo> {
check_list_parts_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
let opts = &opts;
@@ -620,6 +644,8 @@ impl ECStore {
mutation_fence: Option<&ObjectLockDiagGuard>,
) -> Result<(MultipartUploadResult, usize, Option<Uuid>)> {
check_new_object_args(bucket, object, opts.data_movement)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
if self.single_pool() {
@@ -675,21 +701,7 @@ impl ECStore {
if self.is_suspended(idx).await || self.is_pool_rebalancing(idx).await {
continue;
}
let res = list_pool_multipart_uploads_for_incarnation(
pool,
bucket,
&MultipartUploadListRequest {
prefix: object.to_string(),
key_marker: None,
upload_id_marker: None,
delimiter: None,
max_uploads: MAX_UPLOADS_LIST,
expected_incarnation_id: opts.expected_bucket_incarnation_id,
},
)
.await?;
if !res.uploads.is_empty() {
if pool_has_multipart_upload_for_key(pool, bucket, object, opts.expected_bucket_incarnation_id).await? {
self.apply_decommission_target_mutation_fence(idx, object, &mut opts, mutation_fence)
.await;
let expected_bucket_incarnation_id = opts.expected_bucket_incarnation_id;
@@ -772,6 +784,8 @@ impl ECStore {
opts: &ObjectOptions,
) -> Result<PartInfo> {
check_put_object_part_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (mut opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
if let Some(guard) = bucket_lifecycle_guard {
opts.bucket_lifecycle_lock_owner = Some(Arc::new(guard));
@@ -806,6 +820,8 @@ impl ECStore {
.part_number
.ok_or_else(|| Error::other("targeted multipart upload requires a part number"))?;
check_put_object_part_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
if !opts.data_movement {
return Err(Error::other("targeted multipart upload requires data_movement options"));
}
@@ -838,6 +854,8 @@ impl ECStore {
opts: &ObjectOptions,
) -> Result<MultipartInfo> {
check_list_parts_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
let opts = &opts;
if self.single_pool() {
@@ -871,6 +889,8 @@ impl ECStore {
opts: &ObjectOptions,
) -> Result<()> {
check_abort_multipart_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
for pool_idx in self.existing_multipart_pool_order().await {
@@ -1052,6 +1072,8 @@ impl ECStore {
for upload_id in upload_ids {
check_abort_multipart_args(bucket, object, upload_id)?;
}
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
if !opts.data_movement {
return Err(Error::other("targeted multipart abort requires data_movement options"));
}
@@ -1210,7 +1232,34 @@ impl ECStore {
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
check_complete_multipart_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
// The set-level commit judges the null version this completion
// replaces; give it the same authoritative snapshot a PUT carries.
let expected_incarnation_id = opts
.expected_bucket_incarnation_id
.ok_or_else(|| Error::other("multipart completion is missing its bucket incarnation"))?;
let lifecycle_fence = opts
.bucket_lifecycle_lock_fence
.as_ref()
.ok_or_else(|| Error::other("multipart completion is missing its bucket lifecycle fence"))?;
let snapshot = match opts.object_lock_config_snapshot.as_ref() {
Some(snapshot) => Arc::clone(snapshot),
None => {
self.object_lock_config_snapshot_under_lifecycle_fence(bucket, lifecycle_fence)
.await?
}
};
if !snapshot.is_valid_for_destructive_put(self.id, bucket, expected_incarnation_id) {
return Err(Error::other(
"Object Lock snapshot does not hold valid target bucket generation and configuration fences",
));
}
snapshot.add_lock_fences(&mut opts);
opts.object_lock_config_snapshot = Some(snapshot);
}
if !self.single_pool() {
opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
}
@@ -1319,6 +1368,8 @@ impl ECStore {
let request_opts = opts;
let (target_pool_idx, mutation_fence) = target;
check_complete_multipart_args(bucket, object, upload_id)?;
let encoded_object = encode_dir_object_ref(object);
let object = encoded_object.as_ref();
if !opts.data_movement {
return Err(Error::other("targeted multipart completion requires data_movement options"));
}
@@ -1440,6 +1491,362 @@ mod tests {
};
use std::time::Duration;
#[tokio::test]
async fn create_multipart_pool_selection_uses_only_an_exact_key_upload() {
use crate::bucket::metadata_sys::init_bucket_metadata_sys;
use crate::runtime::instance::InstanceContext;
let mut dirs = Vec::new();
let mut pools = Vec::new();
for pool_index in 0..2 {
let mut endpoints = Vec::new();
for disk_index in 0..4 {
let directory = tempfile::tempdir().expect("temporary disk");
let mut endpoint = Endpoint::try_from(directory.path().to_str().expect("disk path")).expect("local endpoint");
endpoint.set_pool_index(pool_index);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_index);
dirs.push(directory);
endpoints.push(endpoint);
}
pools.push(PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 4,
endpoints: Endpoints::from(endpoints),
cmd_line: format!("exact-multipart-pool-{pool_index}"),
platform: "test".to_owned(),
});
}
let endpoints = EndpointServerPools(pools);
let ctx = Arc::new(InstanceContext::new());
crate::store::init_local_disks_with_instance_ctx(&ctx, endpoints.clone())
.await
.expect("initialize local disks");
let cancel = tokio_util::sync::CancellationToken::new();
let store = ECStore::new_with_instance_ctx("127.0.0.1:0".parse().expect("test address"), endpoints, cancel.clone(), ctx)
.await
.expect("initialize two pools");
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "exact-pool-selection";
let other_bucket = "other-pool-selection";
for bucket in [bucket, other_bucket] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
}
for (upload_bucket, key) in [(bucket, "report-long"), (bucket, "report/"), (other_bucket, "report")] {
let (opts, _guard) = store
.guard_multipart_bucket_incarnation(upload_bucket, &ObjectOptions::default())
.await
.expect("pin upload incarnation");
store.pools[0]
.new_multipart_upload(upload_bucket, &encode_dir_object(key), &opts)
.await
.expect("seed nonmatching upload in first pool");
}
let (opts, _guard) = store
.guard_multipart_bucket_incarnation(bucket, &ObjectOptions::default())
.await
.expect("pin target incarnation");
assert!(
!pool_has_multipart_upload_for_key(&store.pools[0], bucket, "report", opts.expected_bucket_incarnation_id)
.await
.expect("first pool contains only other keys")
);
store.pools[1]
.new_multipart_upload(bucket, "report", &opts)
.await
.expect("seed exact upload in second pool");
let (created, pool_index, incarnation) = store
.handle_new_multipart_upload_with_pool_idx(bucket, "report", &ObjectOptions::default(), None)
.await
.expect("create another upload for the exact key");
assert_eq!(pool_index, 1, "prefix matches in the first pool must not change placement");
assert_eq!(incarnation, opts.expected_bucket_incarnation_id);
assert!(!created.upload_id.is_empty());
assert!(
!pool_has_multipart_upload_for_key(&store.pools[0], bucket, "report", incarnation)
.await
.expect("new upload must not appear in first pool")
);
assert!(
!pool_has_multipart_upload_for_key(&store.pools[1], bucket, "report", Some(Uuid::new_v4()))
.await
.expect("old bucket incarnations must not affect placement")
);
cancel.cancel();
}
#[tokio::test]
async fn multipart_directory_keys_preserve_the_bare_object_and_round_trip() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::object::ObjectIO as _;
use sha2::Digest;
use tokio::io::AsyncReadExt;
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
for versioned in [false, true] {
let bucket = if versioned {
"directory-mpu-versioned"
} else {
"directory-mpu-unversioned"
};
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
if versioned {
store
.update_bucket_metadata_config(
bucket,
"versioning.xml",
br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#.to_vec(),
)
.await
.expect("enable versioning");
}
let opts = ObjectOptions {
versioned,
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
};
let original = store
.put_object(
bucket,
"protected",
&mut PutObjReader::from_vec(b"private original".to_vec()),
&ObjectOptions {
user_defined: HashMap::from([(
rustfs_utils::http::headers::AMZ_OBJECT_TAGGING.to_string(),
"classification=secret".to_string(),
)]),
..opts.clone()
},
)
.await
.expect("write protected bare object");
let upload = store
.new_multipart_upload(bucket, "protected/", &opts)
.await
.expect("create directory upload");
let part = store
.put_object_part(
bucket,
"protected/",
&upload.upload_id,
1,
&mut PutObjReader::from_vec(b"directory payload".to_vec()),
&opts,
)
.await
.expect("write directory part");
store
.put_object_part(
bucket,
"protected/",
&upload.upload_id,
2,
&mut PutObjReader::from_vec(b"unselected private bytes".to_vec()),
&opts,
)
.await
.expect("write an unselected part");
let upload_sha = hex_simd::encode_to_string(
sha2::Sha256::digest(format!("{bucket}/protected__XLDIR__").as_bytes()),
hex_simd::AsciiCase::Lower,
);
let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id);
for dir in &dirs {
let staging = dir
.path()
.join(crate::disk::RUSTFS_META_MULTIPART_BUCKET)
.join(&upload_sha)
.join(&upload_uuid);
let metadata = rustfs_filemeta::FileMeta::load(&std::fs::read(staging.join("xl.meta")).expect("upload metadata"))
.expect("decode upload metadata");
let versions = metadata
.get_file_info_versions(bucket, "protected__XLDIR__", true)
.expect("upload file info");
let data_dir = versions
.versions
.first()
.expect("upload version")
.data_dir
.expect("staging data directory");
assert!(
staging.join(data_dir.to_string()).join("part.2").is_file(),
"unselected part exists before completion"
);
}
assert_eq!(
store
.get_multipart_info(bucket, "protected/", &upload.upload_id, &opts)
.await
.expect("upload info")
.object,
"protected/"
);
assert_eq!(
store
.list_object_parts(bucket, "protected/", &upload.upload_id, None, 10, &opts)
.await
.expect("list parts")
.object,
"protected/"
);
let uploads = store
.list_multipart_uploads(bucket, "protected/", None, None, None, 10)
.await
.expect("list logical prefix");
assert_eq!(uploads.uploads.len(), 1);
assert_eq!(uploads.uploads[0].object, "protected/");
assert_eq!(crate::runtime::sources::upload_uuid_suffix(&uploads.uploads[0].upload_id), upload_uuid,);
let completed = store
.clone()
.complete_multipart_upload(
bucket,
"protected/",
&upload.upload_id,
vec![CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
}],
&opts,
)
.await
.expect("complete directory upload");
assert_eq!(completed.name, "protected/");
for (key, expected) in [
("protected", b"private original".as_slice()),
("protected/", b"directory payload".as_slice()),
] {
let mut reader = store
.get_object_reader(bucket, key, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("read distinct object");
let mut bytes = Vec::new();
reader.stream.read_to_end(&mut bytes).await.expect("read body");
assert_eq!(bytes, expected);
}
let preserved = store
.get_object_info(bucket, "protected", &ObjectOptions::default())
.await
.expect("bare metadata");
assert_eq!(preserved.version_id, original.version_id);
assert_eq!(preserved.etag, original.etag);
assert_eq!(preserved.user_tags.as_str(), "classification=secret");
for dir in &dirs {
assert!(dir.path().join(bucket).join("protected/xl.meta").is_file());
assert!(dir.path().join(bucket).join("protected__XLDIR__/xl.meta").is_file());
let data = dir
.path()
.join(bucket)
.join("protected__XLDIR__")
.join(completed.data_dir.expect("multipart data directory").to_string());
assert!(data.join("part.1").is_file());
assert!(!data.join("part.2").exists(), "unselected part must never be published");
assert!(
!dir.path()
.join(crate::disk::RUSTFS_META_MULTIPART_BUCKET)
.join(&upload_sha)
.join(&upload_uuid)
.exists(),
"original staging must be cleaned"
);
}
let abandoned = store
.new_multipart_upload(bucket, "protected/", &opts)
.await
.expect("create abort control");
store
.abort_multipart_upload(bucket, "protected/", &abandoned.upload_id, &opts)
.await
.expect("abort encoded upload");
assert!(
store
.get_multipart_info(bucket, "protected/", &abandoned.upload_id, &opts)
.await
.is_err()
);
assert!(
store
.get_object_info(bucket, "protected/", &ObjectOptions::default())
.await
.is_ok()
);
}
}
#[cfg(not(windows))]
#[tokio::test]
async fn new_control_character_keys_are_rejected_without_trapping_legacy_objects() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::object::ObjectIO as _;
use tokio::io::AsyncReadExt;
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "xml-key-admission";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
for byte in (1u8..32).filter(|byte| !matches!(byte, 9 | 10 | 13)) {
let key = format!("bad-{}", char::from(byte));
assert!(
store
.put_object(bucket, &key, &mut PutObjReader::from_vec(b"body".to_vec()), &ObjectOptions::default())
.await
.is_err()
);
assert!(
store
.new_multipart_upload(bucket, &key, &ObjectOptions::default())
.await
.is_err()
);
for dir in &dirs {
assert!(!dir.path().join(bucket).join(&key).exists());
}
}
let legacy_key = "legacy-\u{1}";
store.pools[0]
.get_disks_by_key(legacy_key)
.put_object(
bucket,
legacy_key,
&mut PutObjReader::from_vec(b"legacy payload".to_vec()),
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("seed a pre-upgrade object below new-key admission");
let mut reader = store
.get_object_reader(bucket, legacy_key, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("legacy GET stays usable");
let mut bytes = Vec::new();
reader.stream.read_to_end(&mut bytes).await.expect("legacy bytes");
assert_eq!(bytes, b"legacy payload");
drop(reader);
store
.delete_object(bucket, legacy_key, ObjectOptions::default())
.await
.expect("legacy DELETE stays usable");
assert!(
store
.get_object_info(bucket, legacy_key, &ObjectOptions::default())
.await
.is_err()
);
}
fn mp(object: &str, upload_id: &str) -> MultipartInfo {
MultipartInfo {
bucket: "bucket".to_string(),
File diff suppressed because it is too large Load Diff
@@ -20,14 +20,230 @@ use std::sync::Arc;
use std::time::Duration;
use storage_api::contract_compat::ECStore;
use storage_api::metadata_lock::{
BucketOperations, CompletePart, Error, MakeBucketOptions, MultipartOperations, NamespaceLocking, ObjectIO, ObjectOperations,
ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause, init_bucket_metadata_sys,
isolated_store_over_temp_disks,
BucketOperations, CompletePart, DeleteBucketOptions, Error, HashReader, MakeBucketOptions, MultipartOperations,
NamespaceLocking, ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause,
init_bucket_metadata_sys, isolated_store_over_temp_disks,
};
use tokio::io::AsyncReadExt;
use tokio::time::timeout;
use uuid::Uuid;
struct ObservedPutBody {
inner: tokio::io::DuplexStream,
polled: Arc<tokio::sync::Notify>,
}
impl tokio::io::AsyncRead for ObservedPutBody {
fn poll_read(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
self.polled.notify_one();
std::pin::Pin::new(&mut self.inner).poll_read(cx, buf)
}
}
#[tokio::test]
async fn destructive_put_takes_bucket_locks_only_after_its_body_is_staged() {
use tokio::io::AsyncWriteExt;
for (object, version_id) in [("object", None), ("directory/", Some(Uuid::nil().to_string()))] {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "deferred-put-locks";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket");
let encoded_object = if object.ends_with('/') { "directory__XLDIR__" } else { object };
let barrier = PutObjectCommitBarrier::install(bucket, encoded_object, PutObjectCommitPause::BeforeQuotaRename);
let (mut sender, inner) = tokio::io::duplex(16);
let polled = Arc::new(tokio::sync::Notify::new());
let body = ObservedPutBody {
inner,
polled: polled.clone(),
};
let put_store = store.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::new(HashReader::from_stream(body, 4, 4, None, None, false).expect("body"));
put_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
version_id,
..Default::default()
},
)
.await
});
timeout(Duration::from_secs(5), polled.notified())
.await
.expect("PUT must be waiting for client data");
let lock = store
.new_ns_lock(".rustfs.sys", &format!("bucket-targets/{bucket}/transaction.lock"))
.await
.expect("metadata lock");
drop(
timeout(Duration::from_secs(2), lock.get_write_lock(Duration::from_secs(5)))
.await
.expect("body must not block metadata writers")
.expect("metadata write guard"),
);
drop(
timeout(Duration::from_secs(2), store.lock_bucket_for_recursive_delete(bucket))
.await
.expect("body must not retain lifecycle guard")
.expect("lifecycle writer"),
);
sender.write_all(b"body").await.expect("finish body");
sender.shutdown().await.expect("EOF");
timeout(Duration::from_secs(5), barrier.wait_until_paused())
.await
.expect("staged PUT reaches commit");
let writer = lock.get_write_lock(Duration::from_secs(5));
tokio::pin!(writer);
assert!(futures::poll!(&mut writer).is_pending(), "commit must fence metadata updates");
let lifecycle = store.lock_bucket_for_recursive_delete(bucket);
tokio::pin!(lifecycle);
assert!(futures::poll!(&mut lifecycle).is_pending(), "commit must fence bucket deletion");
barrier.release();
let release_metadata = async {
drop(writer.await.expect("metadata writer"));
};
let release_lifecycle = async {
drop(lifecycle.await.expect("lifecycle writer"));
};
timeout(Duration::from_secs(5), async {
let (put, (), ()) = tokio::join!(put, release_metadata, release_lifecycle);
put.expect("owner task").expect("commit");
})
.await
.expect("PUT and queued writers finish");
}
}
#[tokio::test]
async fn staged_put_cannot_publish_into_a_recreated_bucket() {
use tokio::io::AsyncWriteExt;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "deferred-put-generation";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket");
let (mut sender, inner) = tokio::io::duplex(16);
let polled = Arc::new(tokio::sync::Notify::new());
let body = ObservedPutBody {
inner,
polled: polled.clone(),
};
let put_store = store.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::new(HashReader::from_stream(body, 4, 4, None, None, false).expect("body"));
put_store
.put_object(bucket, "object", &mut reader, &ObjectOptions::default())
.await
});
timeout(Duration::from_secs(5), polled.notified()).await.expect("body paused");
store
.delete_bucket(
bucket,
&DeleteBucketOptions {
force: true,
..Default::default()
},
)
.await
.expect("remove old bucket while PUT stalls");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
sender.write_all(b"body").await.expect("finish body");
sender.shutdown().await.expect("EOF");
let error = timeout(Duration::from_secs(5), put)
.await
.expect("commit gate")
.expect("owner")
.expect_err("old request generation must not publish");
assert!(
error.to_string().contains("generation") || matches!(error, Error::BucketNotFound(..)),
"{error:?}"
);
assert!(
store
.get_object_info(bucket, "object", &ObjectOptions::default())
.await
.is_err()
);
}
#[tokio::test]
async fn staged_put_rechecks_current_object_lock_at_commit() {
use tokio::io::AsyncWriteExt;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "deferred-put-retention";
store
.make_bucket(
bucket,
&MakeBucketOptions {
lock_enabled: true,
..Default::default()
},
)
.await
.expect("locked bucket");
let mut old = PutObjReader::from_vec(b"old".to_vec());
store
.put_object(bucket, "object", &mut old, &ObjectOptions::default())
.await
.expect("unlocked predecessor");
let (mut sender, inner) = tokio::io::duplex(16);
let polled = Arc::new(tokio::sync::Notify::new());
let body = ObservedPutBody {
inner,
polled: polled.clone(),
};
let put_store = store.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::new(HashReader::from_stream(body, 4, 4, None, None, false).expect("body"));
put_store
.put_object(bucket, "object", &mut reader, &ObjectOptions::default())
.await
});
timeout(Duration::from_secs(5), polled.notified()).await.expect("body paused");
let mut protected = PutObjReader::from_vec(b"protected".to_vec());
store
.put_object(
bucket,
"object",
&mut protected,
&ObjectOptions {
user_defined: std::collections::HashMap::from([("x-amz-object-lock-legal-hold".to_string(), "ON".to_string())]),
..Default::default()
},
)
.await
.expect("install a protected predecessor while the client stalls");
sender.write_all(b"body").await.expect("finish stale body");
sender.shutdown().await.expect("EOF");
let error = timeout(Duration::from_secs(5), put)
.await
.expect("commit gate")
.expect("task")
.expect_err("pre-body state cannot admit an overwrite of the newly protected version");
assert!(matches!(error, Error::PrefixAccessDenied(..)), "{error:?}");
let info = store
.get_object_info(bucket, "object", &ObjectOptions::default())
.await
.expect("protected object remains");
assert_eq!(info.size, 9);
}
#[tokio::test]
async fn replica_put_completes_while_metadata_writer_waits_for_its_snapshot() {
replica_write_with_waiting_metadata_writer(false).await;
+3 -1
View File
@@ -9,12 +9,14 @@ use rustfs_storage_api as storage_contracts;
#[cfg(feature = "test-util")]
pub(crate) mod metadata_lock {
pub(crate) use super::storage_contracts::{
BucketOperations, CompletePart, MakeBucketOptions, MultipartOperations, NamespaceLocking, ObjectIO, ObjectOperations,
BucketOperations, CompletePart, DeleteBucketOptions, MakeBucketOptions, MultipartOperations, NamespaceLocking, ObjectIO,
ObjectOperations,
};
pub(crate) use super::{Error, ObjectOptions, PutObjReader};
pub(crate) use rustfs_ecstore::api::bucket::metadata_sys::{
init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
pub(crate) use rustfs_ecstore::api::rio::HashReader;
pub(crate) use rustfs_ecstore::api::set_disk::test_util::{PutObjectCommitBarrier, PutObjectCommitPause};
}
+28
View File
@@ -3930,6 +3930,34 @@ mod test {
assert!(versions_with_free.versions.iter().all(|version| version.num_versions == 2));
}
#[test]
fn add_version_rejects_mod_times_that_cannot_round_trip() {
for nanos in [0, i128::from(i64::MAX) + 1, i128::from(i64::MIN) - 1, 1_i128 << 64] {
for deleted in [false, true] {
let mut meta = FileMeta::new();
let mut info = crate::fileinfo::FileInfo::new("object", 2, 1);
info.deleted = deleted;
info.mod_time = Some(OffsetDateTime::from_unix_timestamp_nanos(nanos).expect("time within supported date range"));
assert!(
meta.add_version(info).is_err(),
"unrepresentable time {nanos} must be rejected before publication"
);
}
}
for nanos in [-1, 1, i64::MIN, i64::MAX] {
let mut meta = FileMeta::new();
let mut info = crate::fileinfo::FileInfo::new("object", 2, 1);
let time = OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).expect("valid codec time");
info.mod_time = Some(time);
meta.add_version(info).expect("representable times remain supported");
let reopened = FileMeta::load(&meta.marshal_msg().expect("serialize version")).expect("reopen version");
let versions = reopened
.get_file_info_versions("bucket", "object", false)
.expect("read version history");
assert_eq!(versions.versions[0].mod_time, Some(time));
}
}
#[test]
fn test_data_integrity_validation() {
// Test data integrity checks
+13 -2
View File
@@ -858,13 +858,24 @@ impl FileMetaVersion {
VersionType::Object => self
.object
.as_ref()
.map(|v| v.erasure_algorithm.valid() && v.bitrot_checksum_algo.valid() && v.mod_time.is_some())
.map(|v| {
v.erasure_algorithm.valid()
&& v.bitrot_checksum_algo.valid()
&& v.mod_time.is_some_and(|time| {
let nanos = time.unix_timestamp_nanos();
nanos != 0 && i64::try_from(nanos).is_ok()
})
})
.unwrap_or_default(),
VersionType::Legacy => self.legacy_object.as_ref().map(MetaObjectV1::valid).unwrap_or_default(),
VersionType::Delete => self
.delete_marker
.as_ref()
.map(|v| v.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH) > OffsetDateTime::UNIX_EPOCH)
.map(|v| {
v.mod_time.is_some_and(|time| {
time > OffsetDateTime::UNIX_EPOCH && i64::try_from(time.unix_timestamp_nanos()).is_ok()
})
})
.unwrap_or_default(),
_ => false,
}
+12 -4
View File
@@ -77,6 +77,9 @@ pub enum Error {
#[error("Replacement target is not ready: {0}")]
ReplacementTargetNotReady(String),
#[error("Replacement bucket namespace changed; a fresh identity-bound pass is required")]
ReplacementNamespaceChanged,
#[error("replacement recovery retry budget exhausted")]
ReplacementRetryBudgetExhausted,
@@ -132,7 +135,10 @@ impl Error {
/// Callers must cut their own concurrency instead of retrying at full speed.
pub(crate) fn is_slow_down(&self) -> bool {
match self {
Error::Storage(error) => matches!(error, EcstoreError::SlowDown | EcstoreError::TooManyOpenFiles),
Error::Storage(error) => matches!(
error,
EcstoreError::SlowDown | EcstoreError::TooManyOpenFiles | EcstoreError::DeleteRecoveryWorkLimit
),
Error::Disk(error) => matches!(error, DiskError::TooManyOpenFiles),
Error::TransientSkip { message } | Error::Other(message) | Error::TaskExecutionFailed { message } => {
message.contains("Please reduce your request rate")
@@ -229,9 +235,10 @@ pub(crate) fn classify_heal_error(error: &Error) -> HealErrorClass {
Error::DanglingDeleteDeferred { retry_not_before } => HealErrorClass::DanglingGrace {
retry_not_before: Some(*retry_not_before),
},
Error::ReplacementTargetNotReady(_) | Error::TransientSkip { .. } | Error::BucketDiscoveryIncomplete => {
HealErrorClass::Transient
}
Error::ReplacementTargetNotReady(_)
| Error::ReplacementNamespaceChanged
| Error::TransientSkip { .. }
| Error::BucketDiscoveryIncomplete => HealErrorClass::Transient,
Error::LegacyMrfUnverified { .. } => HealErrorClass::Permanent,
Error::Storage(error) => classify_storage_error(error),
Error::Disk(error) => classify_disk_error(error),
@@ -285,6 +292,7 @@ pub(crate) fn classify_storage_error(error: &EcstoreError) -> HealErrorClass {
| EcstoreError::FaultyDisk
| EcstoreError::FaultyRemoteDisk
| EcstoreError::SlowDown
| EcstoreError::DeleteRecoveryWorkLimit
| EcstoreError::OperationCanceled
| EcstoreError::RemoteClientUnavailable(_) => HealErrorClass::Transient,
_ if error.is_quorum_error() => HealErrorClass::Transient,
+148 -27
View File
@@ -468,7 +468,16 @@ impl ErasureSetHealer {
let task_id = self.get_or_create_task_id(set_disk_id).await?;
// 2. initialize or resume resume state
let (resume_manager, checkpoint_manager) = self.initialize_resume_state(&task_id, set_disk_id, buckets).await?;
let (resume_manager, mut checkpoint_manager) = self.initialize_resume_state(&task_id, set_disk_id, buckets).await?;
let replacement_buckets = if self.replacement_task_id.is_some() {
let current = resume_manager.prepare_replacement_bucket_pass(self.storage.as_ref()).await?;
// Preparation may have durably rewound the companion checkpoint.
checkpoint_manager = CheckpointManager::load_from_disk(self.disk.clone(), &task_id).await?;
Some(current)
} else {
None
};
let buckets = replacement_buckets.as_deref().unwrap_or(buckets);
let admin_buckets = if self.admin_task.is_some() {
Some(
@@ -489,6 +498,9 @@ impl ErasureSetHealer {
.execute_heal_with_resume(buckets, set_disk_id, &resume_manager, &checkpoint_manager)
.await;
if self.replacement_task_id.is_some() {
resume_manager.verify_replacement_bucket_pass(self.storage.as_ref()).await?;
}
result?;
self.verify_replacement_identity_fence("completion").await?;
@@ -967,9 +979,16 @@ impl ErasureSetHealer {
// Recheck even resumed buckets: their user-object cursor does not
// prove that the replacement holds the internal bucket records.
for bucket in buckets {
self.storage
.heal_replacement_bucket_metadata(bucket, &self.heal_opts, targets)
.await?;
if self.replacement_task_id.is_some() {
let expected = resume_manager.replacement_bucket_incarnation(bucket).await?;
self.storage
.heal_replacement_bucket_metadata_at_incarnation(bucket, expected, &self.heal_opts, targets)
.await?;
} else {
self.storage
.heal_replacement_bucket_metadata(bucket, &self.heal_opts, targets)
.await?;
}
}
}
self.heal_replacement_pool_metadata(
@@ -1019,6 +1038,7 @@ impl ErasureSetHealer {
// never hot-retried in place here.
if failed_objects > 0 || skipped_objects > 0 || failed_buckets > 0 {
if self.replacement_task_id.is_some() {
resume_manager.verify_replacement_bucket_pass(self.storage.as_ref()).await?;
let state = resume_manager.get_state().await;
let (grace_objects, retry_not_before, counter_unknown) = checkpoint_manager.dangling_delete_grace().await;
if state.can_retry()
@@ -1452,7 +1472,11 @@ impl ErasureSetHealer {
"Erasure set bucket started"
);
let incarnation = self.admin_bucket_incarnation(checkpoint_manager, bucket).await?;
let incarnation = if self.replacement_task_id.is_some() {
Some(resume_manager.replacement_bucket_incarnation(bucket).await?)
} else {
self.admin_bucket_incarnation(checkpoint_manager, bucket).await?
};
let unresolved_delete_recovery = self.heal_admin_bucket_delete_recovery(bucket, incarnation).await?;
if unresolved_delete_recovery && !self.heal_opts.dry_run {
resume_manager.set_delete_recovery_rescan_pending(true).await?;
@@ -1744,15 +1768,20 @@ impl ErasureSetHealer {
(0, Err(Error::TaskCancelled))
} else {
let result = if replacement_object {
storage
.heal_object_with_task_cancel(
&bucket_name,
&object_name,
version_id.as_deref(),
&heal_opts,
&cancel_token,
)
.await
match incarnation {
Some(expected) => storage
.heal_replacement_object_at_incarnation(
&bucket_name,
&object_name,
version_id.as_deref(),
expected,
&heal_opts,
&cancel_token,
)
.await
.map(|result| (result.item, result.error)),
None => Err(Error::ReplacementNamespaceChanged),
}
} else {
storage
.heal_object(&bucket_name, &object_name, version_id.as_deref(), &heal_opts)
@@ -2638,6 +2667,7 @@ mod resume_loop_tests {
struct FakeStorage {
admin_disk: Mutex<Option<DiskStore>>,
bucket_discovery_incomplete: AtomicBool,
bucket_discovery_error: Mutex<Option<EcstoreError>>,
bucket_discovery_scopes: Mutex<Vec<Option<crate::heal::storage_api::storage::DiskSetSelector>>>,
bucket_pages: Mutex<HashMap<String, Page>>,
incarnations: Mutex<HashMap<String, uuid::Uuid>>,
@@ -2729,6 +2759,9 @@ mod resume_loop_tests {
if self.bucket_discovery_incomplete.load(Ordering::SeqCst) {
return Err(Error::BucketDiscoveryIncomplete);
}
if let Some(error) = self.bucket_discovery_error.lock().expect("discovery failure").clone() {
return Err(Error::Storage(error));
}
self.list_buckets().await
}
@@ -2905,6 +2938,16 @@ mod resume_loop_tests {
}
Ok(())
}
async fn heal_replacement_bucket_metadata_at_incarnation(
&self,
bucket: &str,
expected: uuid::Uuid,
opts: &HealOpts,
targets: &[String],
) -> Result<()> {
self.validate_bucket_incarnation(bucket, Some(expected)).await?;
self.heal_replacement_bucket_metadata(bucket, opts, targets).await
}
async fn replacement_targets_have_version(
&self,
_bucket: &str,
@@ -3021,6 +3064,16 @@ mod resume_loop_tests {
let temp = TempDir::new().unwrap();
let disk = make_disk(&temp).await;
let storage = Arc::new(FakeStorage::default());
if !target_endpoints.is_empty() {
storage.bucket_pages.lock().expect("replacement inventory").insert(
"b".to_string(),
Page {
items: Vec::new(),
next: None,
truncated: false,
},
);
}
let task_id = ResumeUtils::generate_task_id();
let healer = ErasureSetHealer::new(
storage.clone(),
@@ -3040,7 +3093,17 @@ mod resume_loop_tests {
)
.await
.unwrap();
let checkpoint = CheckpointManager::new(disk, task_id.clone()).await.unwrap();
if !healer.target_endpoints.is_empty() {
resume
.prepare_replacement_bucket_pass(storage.as_ref())
.await
.expect("replacement fixture identity admission");
}
let checkpoint = if CheckpointManager::has_checkpoint(&disk, &task_id).await {
CheckpointManager::load_from_disk(disk, &task_id).await.unwrap()
} else {
CheckpointManager::new(disk, task_id.clone()).await.unwrap()
};
Env {
healer,
storage,
@@ -3080,6 +3143,60 @@ mod resume_loop_tests {
(processed, successful, failed, skipped, result.map(|_| ()))
}
#[tokio::test]
async fn replacement_bucket_pass_rejects_incomplete_inventory_without_reusing_or_clearing_progress() {
let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.resume
.set_resume_cursor(Some("held-cursor".to_string()))
.await
.expect("cursor before discovery failure");
env.checkpoint
.add_processed_object("held-key".to_string())
.await
.expect("dedup before discovery failure");
// Model a page whose dedup identities must survive until cursor
// publication. complete_page deliberately prunes that identity set.
env.checkpoint
.advance_page(0, 7)
.await
.expect("persist the retained page identities");
let before = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
.await
.expect("durable checkpoint before discovery failure")
.get_checkpoint()
.await;
assert_eq!(before.current_object_index, 7);
assert!(before.processed_objects.contains("held-key"));
let state = env.resume.get_state().await;
env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst);
assert!(matches!(
env.resume.prepare_replacement_bucket_pass(env.storage.as_ref()).await,
Err(Error::BucketDiscoveryIncomplete)
));
assert_eq!(env.resume.get_state().await, state, "partial discovery must never publish a fresh plan");
let checkpoint = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
.await
.expect("durable checkpoint")
.get_checkpoint()
.await;
assert_eq!(checkpoint.current_object_index, 7);
assert!(checkpoint.processed_objects.contains("held-key"));
env.storage.bucket_discovery_incomplete.store(false, Ordering::SeqCst);
*env.storage.bucket_discovery_error.lock().expect("inject quorum failure") =
Some(EcstoreError::InsufficientReadQuorum("bucket".to_string(), "inventory".to_string()));
assert!(matches!(
env.resume.prepare_replacement_bucket_pass(env.storage.as_ref()).await,
Err(Error::Storage(EcstoreError::InsufficientReadQuorum(..)))
));
assert_eq!(env.resume.get_state().await, state, "quorum failure is not an empty namespace");
*env.storage.bucket_discovery_error.lock().expect("restore inventory") = None;
env.resume
.prepare_replacement_bucket_pass(env.storage.as_ref())
.await
.expect("unchanged complete inventory resumes existing pass");
assert_eq!(env.resume.get_state().await.resume_cursor.as_deref(), Some("held-cursor"));
}
#[tokio::test]
async fn test_empty_bucket_no_panic() {
let env = make_env().await;
@@ -4055,7 +4172,7 @@ mod resume_loop_tests {
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
@@ -4110,7 +4227,7 @@ mod resume_loop_tests {
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer.heal_opts.no_lock = true;
env.healer = env
@@ -4167,7 +4284,7 @@ mod resume_loop_tests {
#[tokio::test]
async fn auto_replacement_retries_a_transient_object_before_rewinding_the_scan() {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
@@ -4199,7 +4316,7 @@ mod resume_loop_tests {
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("1")),
],
async {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
@@ -4231,7 +4348,7 @@ mod resume_loop_tests {
#[tokio::test]
async fn auto_replacement_cancellation_interrupts_object_retry() {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
@@ -4247,14 +4364,18 @@ mod resume_loop_tests {
env.storage.set_outcome("retry", None, HealOutcome::Transient);
let cancel = env.healer.cancel_token.clone();
let observe = async {
loop {
let notified = env.storage.heal_started.notified();
if !env.storage.calls().is_empty() {
break;
let started = tokio::time::timeout(std::time::Duration::from_secs(5), async {
loop {
let notified = env.storage.heal_started.notified();
if !env.storage.calls().is_empty() {
break;
}
notified.await;
}
notified.await;
}
})
.await;
cancel.cancel();
started.expect("replacement object retry must start");
};
let ((_, successful, _, _, result), ()) = tokio::join!(run(&env), observe);
@@ -4272,7 +4393,7 @@ mod resume_loop_tests {
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
let mut env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
env.healer.source = HealRequestSource::AutoHeal;
let execution = crate::heal::storage::ReplacementExecution::for_test(Vec::new(), Vec::new());
env.healer = env
@@ -51,6 +51,25 @@ async fn replacement_env() -> Env {
)
.await
.expect("replacement intent should persist before scanning");
env.resume
.prepare_replacement_bucket_pass(env.storage.as_ref())
.await
.expect("the new replacement generation must admit its bucket identities before direct execution");
// Admission durably resets the companion checkpoint. Reload its current
// revision instead of retaining the previous generation's CAS instance.
env.checkpoint = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
.await
.expect("reload the admitted replacement pass checkpoint");
assert_eq!(
env.resume
.replacement_bucket_incarnation("b")
.await
.expect("the fixture bucket must belong to the replacement pass"),
env.storage
.admit_bucket_incarnation("b")
.await
.expect("fixture bucket identity")
);
env.healer = ErasureSetHealer::new(
env.storage.clone(),
Arc::new(RwLock::new(HealProgress::new())),
@@ -881,6 +881,13 @@ async fn auto_format_fixture(
// Preserve the constructor's explicit open/closed state before any fixture await.
let node_limit = node.as_ref().map(|node| node.snapshot().limit);
let fixture = resume_admission_fixture(node.clone()).await;
// The endpoint monitor independently probes format.json after five seconds.
// Drain it before aging the cache so the path-only pause observes this scanner.
for pool in &fixture._store.ecstore.pools {
tokio::time::timeout(Duration::from_secs(30), pool.stop_endpoint_monitor_for_test())
.await
.expect("endpoint reconnection must stop before the scanner owns format probes");
}
let disks = crate::heal::local_disk_map_read()
.await
.values()
+24 -1
View File
@@ -27,6 +27,7 @@ use super::{
BUCKET_META_PREFIX, DiskError, DiskStore, HealDiskExt as _, RUSTFS_META_BUCKET, storage_api::owner::EcstoreDiskBytes,
};
mod bucket_pass;
mod checkpoint;
mod gc;
mod handoff;
@@ -345,6 +346,16 @@ pub struct ResumeState {
/// input after restart.
#[serde(default)]
pub replacement_buckets: Vec<String>,
/// Current namespace pass, separate from the immutable generation plan.
/// Legacy snapshots have no binding and must discard every old cursor.
#[serde(default)]
pub replacement_bucket_incarnations: Option<std::collections::BTreeMap<String, Uuid>>,
/// Last proven retired identity per name, retained for returning stale volumes.
#[serde(default)]
pub replacement_retired_bucket_incarnations: std::collections::BTreeMap<String, Uuid>,
/// Reapply the checkpoint rewind after any interruption between the two writes.
#[serde(default)]
pub replacement_bucket_pass_reset_pending: bool,
/// A task-scoped generation assigned before formatting an automatic
/// replacement. A new unformatted replacement receives a new generation
/// and therefore cannot reuse an older disk's cursor or checkpoint.
@@ -454,6 +465,9 @@ impl ResumeState {
replacement_targets: Vec::new(),
replacement_target_identities: Vec::new(),
replacement_buckets: Vec::new(),
replacement_bucket_incarnations: None,
replacement_retired_bucket_incarnations: std::collections::BTreeMap::new(),
replacement_bucket_pass_reset_pending: false,
replacement_generation: None,
replacement_phase: ReplacementPhase::None,
replacement_execution_protocol: 0,
@@ -985,7 +999,16 @@ impl ResumeManager {
return Err(replacement_recovery_conflict("invalid retired legacy generation"));
}
}
if state.replacement_execution_protocol != 1
let valid_execution_protocol = match state.replacement_execution_protocol {
1 => state.replacement_bucket_incarnations.is_none(),
2 => state
.replacement_bucket_incarnations
.as_ref()
.is_some_and(|identities| identities.values().all(|incarnation| !incarnation.is_nil())),
_ => false,
};
if !valid_execution_protocol
|| state.replacement_retired_bucket_incarnations.values().any(Uuid::is_nil)
|| state.replacement_predecessor.as_deref()
!= state.replacement_lineage.last().map(|link| link.predecessor.as_str())
{
+411
View File
@@ -0,0 +1,411 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::heal::storage::HealStorageAPI;
use crate::heal::storage_api::storage::DiskSetSelector;
use std::collections::BTreeMap;
async fn namespace_snapshot(storage: &dyn HealStorageAPI, state: &ResumeState) -> Result<BTreeMap<String, Uuid>> {
let (pool, set) = crate::heal::utils::parse_set_disk_id(&state.set_disk_id)?;
// This API rejects partial topology and propagates read-quorum failures.
let buckets = storage
.list_buckets_for_admin_heal(Some(DiskSetSelector::new(pool, set)))
.await?;
let mut identities = BTreeMap::new();
for bucket in buckets {
let Some(incarnation) = storage.replacement_bucket_incarnation(&bucket.name).await? else {
// A returning disk may still list a retired volume. Only the admitted
// old UUID plus its durable receipt can discharge that obligation.
if let Some(expected) = state
.replacement_bucket_incarnations
.as_ref()
.and_then(|map| map.get(&bucket.name))
.or_else(|| state.replacement_retired_bucket_incarnations.get(&bucket.name))
&& storage.mrf_bucket_incarnation_retired(&bucket.name, *expected).await?
{
continue;
}
return Err(Error::ReplacementNamespaceChanged);
};
if incarnation.is_nil() {
return Err(Error::InvalidBucketIncarnation);
}
identities.insert(bucket.name, incarnation);
}
Ok(identities)
}
impl ResumeManager {
pub(crate) async fn replacement_bucket_incarnation(&self, bucket: &str) -> Result<Uuid> {
self.state
.read()
.await
.replacement_bucket_incarnations
.as_ref()
.and_then(|identities| identities.get(bucket).copied())
.ok_or(Error::ReplacementNamespaceChanged)
}
pub(crate) async fn prepare_replacement_bucket_pass(&self, storage: &dyn HealStorageAPI) -> Result<Vec<String>> {
let previous = self.get_state().await;
let current = namespace_snapshot(storage, &previous).await?;
if let Some(identities) = previous.replacement_bucket_incarnations.as_ref() {
for (bucket, expected) in identities {
if current.get(bucket) != Some(expected) && !storage.mrf_bucket_incarnation_retired(bucket, *expected).await? {
// Absence or a different current identity alone is not a deletion receipt.
return Err(Error::ReplacementNamespaceChanged);
}
}
}
if previous.replacement_bucket_incarnations.as_ref() != Some(&current) {
let mut state = self.state.write().await;
if let Some(identities) = previous.replacement_bucket_incarnations.as_ref() {
for (bucket, expected) in identities {
if current.get(bucket) != Some(expected) {
state
.replacement_retired_bucket_incarnations
.insert(bucket.clone(), *expected);
}
}
}
state.replacement_bucket_incarnations = Some(current.clone());
if state.replacement_generation.is_some() {
// Protocol-1 readers reject this before interpreting a new pass cursor.
state.replacement_execution_protocol = 2;
}
state.replacement_bucket_pass_reset_pending = true;
state.reset_for_retry();
state.pending_buckets = current.keys().cloned().collect();
state.total_objects = 0;
state.total_bytes = 0;
state.baseline_generation = None;
state.baseline_known = false;
drop(state);
self.save_state_strict().await?;
}
if self.get_state().await.replacement_bucket_pass_reset_pending {
let checkpoint = if CheckpointManager::has_checkpoint(&self.disk, &previous.task_id).await {
CheckpointManager::load_from_disk(self.disk.clone(), &previous.task_id).await?
} else {
CheckpointManager::new(self.disk.clone(), previous.task_id.clone()).await?
};
// Unlike the throttled retry writer this is durable before clearing the intent.
checkpoint.reset_after_delete_recovery().await?;
self.state.write().await.replacement_bucket_pass_reset_pending = false;
self.save_state_strict().await?;
}
Ok(current.into_keys().collect())
}
pub(crate) async fn verify_replacement_bucket_pass(&self, storage: &dyn HealStorageAPI) -> Result<()> {
let state = self.get_state().await;
let current = namespace_snapshot(storage, &state).await?;
if state.replacement_bucket_incarnations.as_ref() != Some(&current) {
return Err(Error::ReplacementNamespaceChanged);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::heal::storage::{ECStoreHealStorage, HealStorageAPI};
use crate::heal::storage_api::{
storage::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _},
test_support::{
ObjectOptions, PutObjReader, delete_config, init_bucket_metadata_sys, isolated_store_over_temp_disks, read_config,
save_config,
},
};
use rustfs_heal_contracts::heal_channel::HealOpts;
use tokio_util::sync::CancellationToken;
#[tokio::test]
#[serial_test::serial]
async fn replacement_bucket_pass_retires_deleted_identity_and_restarts_recreated_bucket() {
// Keep the concrete storage futures off the test executor stack.
let (directories, store) = Box::pin(isolated_store_over_temp_disks()).await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "replacement-pass-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create original generation");
let old = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let storage = ECStoreHealStorage::new(store.clone());
let (_anchor_dir, disk) = super::super::tests::schema_test_disk().await;
let task_id = ResumeUtils::generate_task_id();
let manager = ResumeManager::new_replacement_intent(
disk.clone(),
task_id.clone(),
"pool_0_set_0".to_string(),
vec![bucket.to_owned()],
vec!["replacement-a".to_string()],
vec![ReplacementTargetIdentity {
endpoint: "replacement-a".to_string(),
canonical_path: "/mnt/replacement-a".to_string(),
physical_device_ids: vec!["device-a".to_string()],
filesystem_identity: "1:2:3".to_string(),
}],
)
.await
.expect("persist generation");
// A legacy name-only cursor cannot be attributed to the currently live bucket.
manager
.set_resume_cursor(Some("unbound-legacy-cursor".to_string()))
.await
.expect("legacy cursor");
let legacy_checkpoint = CheckpointManager::new(disk.clone(), task_id.clone())
.await
.expect("legacy checkpoint");
legacy_checkpoint
.add_processed_object("legacy-key".to_string())
.await
.expect("legacy dedup");
legacy_checkpoint.complete_page(0, 99).await.expect("persist legacy cursor");
assert_eq!(
manager
.prepare_replacement_bucket_pass(&storage)
.await
.expect("admit original pass"),
[bucket]
);
assert_eq!(
manager
.get_state()
.await
.replacement_bucket_incarnations
.as_ref()
.and_then(|m| m.get(bucket))
.copied(),
Some(old)
);
assert!(
manager.get_state().await.resume_cursor.is_none(),
"legacy cursor must not enter the fresh pass"
);
let fresh_checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task_id)
.await
.expect("fresh checkpoint")
.get_checkpoint()
.await;
assert_eq!(fresh_checkpoint.current_object_index, 0);
assert!(fresh_checkpoint.processed_objects.is_empty());
manager
.set_resume_cursor(Some("old-generation-cursor".to_string()))
.await
.expect("persist old cursor");
let checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task_id)
.await
.expect("checkpoint");
checkpoint
.add_processed_object("same-key".to_string())
.await
.expect("old dedup entry");
checkpoint.complete_page(0, 7).await.expect("persist old position and dedup");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete original generation");
assert!(matches!(
manager.verify_replacement_bucket_pass(&storage).await,
Err(Error::ReplacementNamespaceChanged)
));
manager
.record_replacement_failure(&Error::ReplacementNamespaceChanged, 99)
.await
.expect("defer without exhausting replacement");
assert_eq!(manager.get_state().await.retry_count, 0);
let resumed = ResumeManager::load_replacement_intent(disk.clone(), &task_id)
.await
.expect("restart replacement manager");
let retirement_path = format!("bucket-retirements/{bucket}/{old}.json");
let retirement = Box::pin(read_config(store.clone(), &retirement_path))
.await
.expect("authoritative retirement");
Box::pin(delete_config(store.clone(), &retirement_path))
.await
.expect("simulate missing retirement evidence");
assert!(
matches!(
resumed.prepare_replacement_bucket_pass(&storage).await,
Err(Error::ReplacementNamespaceChanged)
),
"physical absence alone cannot retire an admitted identity"
);
Box::pin(save_config(store.clone(), &retirement_path, b"corrupt retirement".to_vec()))
.await
.expect("inject unreadable proof");
assert!(
matches!(resumed.prepare_replacement_bucket_pass(&storage).await, Err(Error::Storage(_))),
"a corrupt proof cannot be treated as an absent bucket"
);
assert_eq!(resumed.get_state().await.resume_cursor.as_deref(), Some("old-generation-cursor"));
Box::pin(save_config(store.clone(), &retirement_path, retirement))
.await
.expect("restore proof");
assert!(
resumed
.prepare_replacement_bucket_pass(&storage)
.await
.expect("retirement proves deleted plan item")
.is_empty()
);
// A disk returning after DELETE may expose the retired physical volume
// again; the persisted receipt identity still discharges that old name.
tokio::fs::create_dir(directories[0].path().join(bucket))
.await
.expect("return stale retired volume");
let returned = ResumeManager::load_replacement_intent(disk.clone(), &task_id)
.await
.expect("restart after retirement");
assert!(
returned
.prepare_replacement_bucket_pass(&storage)
.await
.expect("retired stale volume is not a new bucket")
.is_empty()
);
tokio::fs::remove_dir(directories[0].path().join(bucket))
.await
.expect("remove stale test volume");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate same name");
let new = store.bucket_incarnation_id_from_disk(bucket).await.expect("new identity");
assert_ne!(new, old);
Box::pin(store.put_object(
bucket,
"same-key",
&mut PutObjReader::from_vec(b"new payload".to_vec()),
&ObjectOptions::default(),
))
.await
.expect("new generation payload");
assert_eq!(
resumed
.prepare_replacement_bucket_pass(&storage)
.await
.expect("new generation gets independent pass"),
[bucket]
);
let state = resumed.get_state().await;
assert_eq!(
state.replacement_execution_protocol, 2,
"older binaries must reject identity-bound cursors"
);
assert_eq!(
state
.replacement_bucket_incarnations
.as_ref()
.and_then(|m| m.get(bucket))
.copied(),
Some(new)
);
assert!(state.resume_cursor.is_none());
let checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task_id)
.await
.expect("restarted checkpoint")
.get_checkpoint()
.await;
assert_eq!((checkpoint.current_bucket_index, checkpoint.current_object_index), (0, 0));
assert!(checkpoint.processed_objects.is_empty());
let opts = HealOpts {
pool: Some(0),
set: Some(0),
..Default::default()
};
assert!(
storage
.heal_replacement_object_at_incarnation(bucket, "same-key", None, old, &opts, &CancellationToken::new())
.await
.is_err(),
"a stale worker cannot mutate a recreated bucket"
);
let targets = directories
.iter()
.map(|dir| dir.path().to_string_lossy().into_owned())
.collect::<Vec<_>>();
assert!(
storage
.heal_replacement_bucket_metadata_at_incarnation(bucket, old, &opts, &targets)
.await
.is_err(),
"a stale metadata pass cannot certify the new incarnation"
);
storage
.heal_replacement_bucket_metadata_at_incarnation(bucket, new, &opts, &targets)
.await
.expect("current incarnation can repair metadata");
let mut reader =
Box::pin(store.get_object_reader(bucket, "same-key", None, Default::default(), &ObjectOptions::default()))
.await
.expect("open new generation payload");
assert_eq!(reader.read_all().await.expect("read new payload"), b"new payload");
drop(reader);
assert_eq!(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("new identity remains"),
new
);
// Simulate interruption after publishing the reset obligation, before
// replacing the old checkpoint.
let stale_checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task_id)
.await
.expect("checkpoint before interruption");
stale_checkpoint
.add_processed_object("stale-before-reset".to_string())
.await
.expect("stale dedup");
stale_checkpoint
.complete_page(0, 42)
.await
.expect("persist interrupted checkpoint");
resumed.state.write().await.replacement_bucket_pass_reset_pending = true;
resumed.save_state_strict().await.expect("persist reset obligation");
let interrupted = ResumeManager::load_replacement_intent(disk.clone(), &task_id)
.await
.expect("restart interrupted pass");
interrupted
.prepare_replacement_bucket_pass(&storage)
.await
.expect("finish interrupted checkpoint rewind");
let repaired_checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task_id)
.await
.expect("repaired checkpoint")
.get_checkpoint()
.await;
assert_eq!(repaired_checkpoint.current_object_index, 0);
assert!(repaired_checkpoint.processed_objects.is_empty());
assert!(!interrupted.get_state().await.replacement_bucket_pass_reset_pending);
let loaded = ResumeManager::load_replacement_intent(disk, &task_id)
.await
.expect("new pass survives restart");
loaded
.verify_replacement_bucket_pass(&storage)
.await
.expect("new pass matches current namespace");
}
}
+8 -2
View File
@@ -277,7 +277,7 @@ impl ResumeManager {
}
return self.ensure_handoff_successor().await;
}
if state.replacement_execution_protocol != 1
if !matches!(state.replacement_execution_protocol, 1 | 2)
|| !matches!(
state.replacement_phase,
ReplacementPhase::Intent
@@ -458,6 +458,12 @@ impl ResumeManager {
predecessor.replacement_targets.clone(),
link.targets.clone(),
);
if predecessor.replacement_execution_protocol == 2 {
state.replacement_execution_protocol = 2;
state.replacement_bucket_incarnations = predecessor.replacement_bucket_incarnations.clone();
state.replacement_retired_bucket_incarnations = predecessor.replacement_retired_bucket_incarnations.clone();
state.replacement_bucket_pass_reset_pending = true;
}
state.replacement_phase = ReplacementPhase::OwnershipPending;
// The constructor marks this compatibility grant consumed for every
// successor, including descendants of a recovered older generation.
@@ -580,7 +586,7 @@ fn preformat_handoff_retry_candidate(state: &ResumeState) -> bool {
&& state.replacement_generation.as_deref() == Some(state.task_id.as_str())
&& !state.replacement_preformat_handoff_retry_done
&& state.replacement_phase == ReplacementPhase::Intent
&& state.replacement_execution_protocol == 1
&& matches!(state.replacement_execution_protocol, 1 | 2)
&& state.max_retries >= 2
&& state.replacement_predecessor.is_none()
&& state.replacement_handoff.is_none()
@@ -918,3 +918,33 @@ async fn committed_predecessor_is_collectible_once_its_successor_is_verified() {
"the verified successor keeps its own record until cleanup"
);
}
#[tokio::test]
async fn handoff_preserves_bucket_pass_protocol_and_rewinds_successor_progress() {
let (_dirs, parent, execution) = fixture(1).await;
let identities = [("bucket-a".to_string(), Uuid::new_v4())].into_iter().collect();
{
let mut state = parent.state.write().await;
state.replacement_execution_protocol = 2;
state.replacement_bucket_incarnations = Some(identities);
state.resume_cursor = Some("old-pass-cursor".to_string());
}
parent.save_state_strict().await.expect("persist identity-bound predecessor");
let child = parent
.prepare_replacement_handoff(&execution, Vec::new())
.await
.expect("handoff protocol two");
let state = child.get_state().await;
assert_eq!(state.replacement_execution_protocol, 2);
assert_eq!(
state.replacement_bucket_incarnations,
parent.get_state().await.replacement_bucket_incarnations
);
assert!(state.replacement_bucket_pass_reset_pending);
assert!(state.resume_cursor.is_none());
assert!(state.completed_buckets.is_empty());
let recovered = ResumeManager::load_replacement_intent(child.disk.clone(), &state.task_id)
.await
.expect("reload protocol-two successor");
assert_eq!(recovered.get_state().await, state);
}
+20 -1
View File
@@ -345,6 +345,16 @@ pub(crate) fn merge_replacement_recovery_candidate(
let candidate_revision = candidate.state.replacement_revision;
let current_revision = current.state.replacement_revision;
let candidate_protocol = candidate.state.replacement_execution_protocol;
let current_protocol = current.state.replacement_execution_protocol;
if (candidate_protocol > current_protocol && candidate_revision <= current_revision)
|| (candidate_protocol < current_protocol && candidate_revision >= current_revision)
{
return Err(Error::ReplacementGenerationConflict {
task_id: candidate.state.task_id,
reason: "replacement execution protocol regresses with durable revision".to_string(),
});
}
if candidate_revision == current_revision && candidate.state != current.state {
return Err(Error::ReplacementGenerationConflict {
task_id: candidate.state.task_id,
@@ -374,7 +384,10 @@ fn same_replacement_generation_binding(left: &ResumeState, right: &ResumeState)
&& left.replacement_target_identities == right.replacement_target_identities
&& left.replacement_buckets == right.replacement_buckets
&& left.replacement_generation == right.replacement_generation
&& left.replacement_execution_protocol == right.replacement_execution_protocol
// A revision-ordered protocol-1 -> protocol-2 pass admission preserves
// the generation's target ownership and immutable lineage.
&& matches!(left.replacement_execution_protocol, 1 | 2)
&& matches!(right.replacement_execution_protocol, 1 | 2)
&& left.replacement_predecessor == right.replacement_predecessor
&& left.replacement_lineage == right.replacement_lineage
&& left.replacement_legacy_import == right.replacement_legacy_import
@@ -592,6 +605,12 @@ impl ResumeManager {
if state.replacement_generation.as_deref() != Some(state.task_id.as_str()) {
return Err(replacement_recovery_conflict("replacement failure has no matching generation"));
}
if matches!(error, Error::ReplacementNamespaceChanged | Error::BucketDiscoveryIncomplete) {
// Namespace churn is not evidence that the replacement is irreparable.
state.error_message = Some(error.to_string());
drop(state);
return self.save_state_strict().await;
}
if let Error::DanglingDeleteDeferred { retry_not_before } = error {
if !super::dangling_delete_deadline_is_representable(*retry_not_before) {
return Err(Error::TaskExecutionFailed {
+126
View File
@@ -3629,3 +3629,129 @@ async fn blocked_checkpoint_is_not_revalidated_before_its_schedule() {
assert_eq!(serde_json::from_slice::<serde_json::Value>(&retained).expect("marker JSON"), marker);
temp_dir.close().expect("remove scheduled revalidation fixture");
}
#[test]
fn replacement_candidate_merge_rejects_protocol_downgrade_in_either_read_order() {
for downgrade in [false, true] {
for reverse in [false, true] {
let task_id = ResumeUtils::generate_task_id();
let mut protocol_one = replacement_candidate_state(&task_id);
protocol_one.replacement_revision = if downgrade { 11 } else { 9 };
let mut protocol_two = replacement_candidate_state(&task_id);
protocol_two.replacement_execution_protocol = 2;
protocol_two.replacement_bucket_incarnations = Some(Default::default());
protocol_two.replacement_revision = 10;
let (first, second) = if reverse {
(protocol_two, protocol_one)
} else {
(protocol_one, protocol_two)
};
let mut selected = None;
merge_replacement_recovery_candidate(
&mut selected,
ReplacementRecoveryCandidate::new(first, "anchor-a", false).expect("first durable copy"),
)
.expect("initial observation");
let result = merge_replacement_recovery_candidate(
&mut selected,
ReplacementRecoveryCandidate::new(second, "anchor-b", true).expect("second durable copy"),
);
if downgrade {
assert!(
matches!(result, Err(Error::ReplacementGenerationConflict { .. })),
"protocol 1 at a newer revision must not erase protocol 2, reverse={reverse}"
);
} else {
result.expect("protocol 1 at an older revision may advance to protocol 2");
let state = selected.expect("selected upgraded generation").state;
assert_eq!(state.replacement_execution_protocol, 2);
assert_eq!(state.replacement_revision, 10);
}
}
}
}
#[tokio::test]
#[serial_test::serial]
async fn replacement_recovery_survives_same_named_bucket_deletion_and_recreation() {
use crate::heal::storage_api::{
storage::{BucketOperations as _, DeleteBucketOptions, DiskSetSelector, MakeBucketOptions, StorageAdminApi as _},
test_support::{init_bucket_metadata_sys, isolated_store_over_temp_disks},
};
let (_directories, store) = Box::pin(isolated_store_over_temp_disks()).await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "ahm-replacement";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create legal same-name bucket");
let disk = store
.disk_set_inventory(DiskSetSelector::new(0, 0))
.await
.expect("survivor inventory")
.into_iter()
.flatten()
.next()
.expect("survivor disk");
let task_id = ResumeUtils::generate_task_id();
let manager = ResumeManager::new_replacement_intent(
disk.clone(),
task_id.clone(),
"pool_0_set_0".to_string(),
vec!["other-bucket".to_string()],
vec!["replacement-a".to_string()],
vec![ReplacementTargetIdentity {
endpoint: "replacement-a".to_string(),
canonical_path: "/mnt/replacement-a".to_string(),
physical_device_ids: vec!["device-a".to_string()],
filesystem_identity: "1:2:3".to_string(),
}],
)
.await
.expect("persist sealed replacement intent");
let expected = manager.get_state().await;
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete same-name user bucket");
let resumed = ResumeManager::load_replacement_intent(disk.clone(), &task_id)
.await
.expect("sealed intent remains recoverable");
assert_eq!(resumed.get_state().await, expected);
assert_eq!(
ResumeUtils::get_replacement_intent_tasks(&disk)
.await
.expect("scanner recovery enumeration"),
std::slice::from_ref(&task_id)
);
resumed
.mark_replacement_completed_and_verified()
.await
.expect("replacement can finish after deletion");
resumed
.ensure_replacement_completion_proof()
.await
.expect("persist completion proof");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate same-name user bucket");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete recreated bucket");
let proof = ResumeManager::load_replacement_completion_proof(disk.clone(), &task_id)
.await
.expect("completion proof survives recreation");
assert_eq!(proof.task_id, task_id);
assert_eq!(
ResumeManager::load_replacement_intent(disk, &task_id)
.await
.expect("verified intent survives")
.get_state()
.await
.replacement_phase,
ReplacementPhase::Verified
);
}
+86
View File
@@ -476,6 +476,15 @@ pub trait HealStorageAPI: Send + Sync {
Ok(None)
}
/// Published identity for a replacement pass; None proves missing metadata only.
async fn replacement_bucket_incarnation(&self, bucket: &str) -> Result<Option<Uuid>> {
match self.admit_bucket_incarnation(bucket).await {
Ok(id) => Ok(Some(id)),
Err(Error::StaleBucketIncarnation { .. }) => Ok(None),
Err(error) => Err(error),
}
}
/// Whether the exact old bucket generation has a durable deletion record.
/// Alternate backends must fail closed unless they can validate the same
/// deployment, bucket and incarnation identity.
@@ -598,6 +607,23 @@ pub trait HealStorageAPI: Send + Sync {
self.heal_object(bucket, object, version_id, opts).await
}
/// Replacement writes retain both cancellation ownership and the admitted bucket fence.
async fn heal_replacement_object_at_incarnation(
&self,
bucket: &str,
object: &str,
version_id: Option<&str>,
expected: Uuid,
opts: &HealOpts,
task_cancel: &CancellationToken,
) -> Result<HealStorageObjectResult> {
if task_cancel.is_cancelled() {
return Err(Error::TaskCancelled);
}
self.heal_object_at_incarnation(bucket, object, version_id, expected, opts)
.await
}
async fn heal_object_with_receipt(
&self,
bucket: &str,
@@ -663,6 +689,17 @@ pub trait HealStorageAPI: Send + Sync {
Err(Error::Storage(StorageError::PreconditionFailed))
}
async fn heal_replacement_bucket_metadata_at_incarnation(
&self,
_bucket: &str,
_expected: Uuid,
_opts: &HealOpts,
_targets: &[String],
) -> Result<()> {
// An unfenced fallback could certify a different same-name bucket.
Err(Error::Storage(StorageError::PreconditionFailed))
}
/// Read target-specific physical evidence for one replacement version.
///
/// This is only used by automatic replacement healing after the normal
@@ -1316,6 +1353,35 @@ impl HealStorageAPI for ECStoreHealStorage {
.await
}
async fn heal_replacement_object_at_incarnation(
&self,
bucket: &str,
object: &str,
version_id: Option<&str>,
expected: Uuid,
opts: &HealOpts,
task_cancel: &CancellationToken,
) -> Result<HealStorageObjectResult> {
run_background_object_inner(
self.admission.as_ref(),
|| self.ecstore.physical_heal_plan(object, opts).map_err(Error::Storage),
opts,
Some(task_cancel),
|result: &HealStorageObjectResult| result.error.is_none(),
boxed_object_operation(|| async {
let result = self
.ecstore
.heal_object_at_incarnation(bucket, object, version_id.unwrap_or_default(), expected, opts)
.await
.map_err(|error| incarnation_storage_error(bucket, expected, error))?;
Ok(self
.object_result_with_receipt(bucket, object, version_id, opts, result, Some(expected))
.await)
}),
)
.await
}
async fn heal_mrf_object_at_incarnation(
&self,
bucket: &str,
@@ -1520,6 +1586,13 @@ impl HealStorageAPI for ECStoreHealStorage {
}
}
async fn replacement_bucket_incarnation(&self, bucket: &str) -> Result<Option<Uuid>> {
self.ecstore
.replacement_bucket_incarnation_from_disk(bucket)
.await
.map_err(Error::Storage)
}
async fn mrf_bucket_incarnation_retired(&self, bucket: &str, expected: Uuid) -> Result<bool> {
self.ecstore
.is_bucket_incarnation_retired(bucket, expected)
@@ -1959,6 +2032,19 @@ impl HealStorageAPI for ECStoreHealStorage {
.map_err(Error::Storage)
}
async fn heal_replacement_bucket_metadata_at_incarnation(
&self,
bucket: &str,
expected: Uuid,
opts: &HealOpts,
targets: &[String],
) -> Result<()> {
self.ecstore
.heal_replacement_bucket_metadata_at_incarnation(bucket, expected, opts, targets)
.await
.map_err(|error| incarnation_storage_error(bucket, expected, error))
}
async fn replacement_targets_have_version(
&self,
bucket: &str,
+10 -1
View File
@@ -98,10 +98,19 @@ pub(crate) mod storage {
};
#[cfg(test)]
pub(crate) use super::storage_contracts::DeleteBucketOptions;
pub(crate) use super::storage_contracts::{DeleteBucketOptions, MakeBucketOptions};
}
#[cfg(test)]
pub(crate) mod status {
pub(crate) use super::storage_contracts::BucketInfo;
}
#[cfg(test)]
pub(crate) mod test_support {
pub(crate) use rustfs_ecstore::api::bucket::metadata_sys::{
init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
pub(crate) use rustfs_ecstore::api::config::com::{delete_config, read_config, save_config};
pub(crate) use rustfs_ecstore::api::object::{ObjectOptions, PutObjReader};
}
+16 -1
View File
@@ -1310,7 +1310,22 @@ impl HealTask {
.load(Ordering::Acquire)
.max(self.retry_attempts)
.saturating_add(1);
manager.record_replacement_failure(&failure, attempt).await
// A deletion can race any stage, including hidden-transaction recovery.
// Only a successful complete inventory may reclassify that failure;
// lookup/quorum failures remain the original storage failure.
if manager.get_state().await.replacement_bucket_incarnations.is_some()
&& !matches!(failure, Error::TaskCancelled | Error::TaskTimeout)
&& matches!(
manager.verify_replacement_bucket_pass(self.storage.as_ref()).await,
Err(Error::ReplacementNamespaceChanged)
)
{
manager
.record_replacement_failure(&Error::ReplacementNamespaceChanged, attempt)
.await
} else {
manager.record_replacement_failure(&failure, attempt).await
}
}
.await;
match persisted {
+21 -3
View File
@@ -247,7 +247,12 @@ impl HealTask {
*self.replacement_resume_disk.write().await = Some(disk.clone());
let state = manager.get_state().await;
self.replacement_start_retry_count.store(state.retry_count, Ordering::Release);
buckets = state.replacement_buckets;
buckets = if state.completed {
state.replacement_buckets
} else {
self.await_with_control(manager.prepare_replacement_bucket_pass(self.storage.as_ref()))
.await?
};
Some((disk, manager, identities))
} else {
None
@@ -478,14 +483,27 @@ impl HealTask {
self.verify_replacement_identity_fence(expected_identities, &set_disk_id, "bucket prepass")
.await?;
}
let expected = if let Some((_, manager, _)) = replacement_resume.as_ref() {
Some(manager.replacement_bucket_incarnation(bucket).await?)
} else {
None
};
let heal_result = self
.await_with_control(self.storage.heal_bucket(bucket, &bucket_heal_opts))
.await_with_control(async {
match expected {
Some(id) => self.storage.heal_bucket_at_incarnation(bucket, id, &bucket_heal_opts).await,
None => self.storage.heal_bucket(bucket, &bucket_heal_opts).await,
}
})
.await;
match heal_result {
Ok(result) => {
self.record_result_item(result).await;
}
Err(err) => {
if let Some((_, manager, _)) = replacement_resume.as_ref() {
manager.verify_replacement_bucket_pass(self.storage.as_ref()).await?;
}
warn!(
target: "rustfs::heal::task",
event = EVENT_HEAL_ERASURE_SET_RESULT,
@@ -505,7 +523,7 @@ impl HealTask {
if let Some(result) = self
.await_with_control(
self.storage
.recover_delete_transactions(bucket, None, None, &bucket_heal_opts),
.recover_delete_transactions(bucket, None, expected, &bucket_heal_opts),
)
.await?
{
+12 -2
View File
@@ -1056,6 +1056,7 @@ async fn automatic_replacement_uses_target_scoped_format() {
let temp = TempDir::new().expect("temporary resume disk directory should be created");
let disk = make_resume_disk(&temp).await;
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
resume_disk: Mutex::new(Some(disk.clone())),
..Default::default()
@@ -1264,6 +1265,7 @@ async fn automatic_replacement_rejects_a_new_identity_after_format() {
let first_identity = replacement_identity("replacement-a", "device-a", "filesystem-a");
let second_identity = replacement_identity("replacement-a", "device-b", "filesystem-b");
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
replacement_target_identity_sequences: Mutex::new(VecDeque::from([
vec![first_identity.clone()],
@@ -1330,6 +1332,7 @@ async fn automatic_replacement_reuses_an_existing_non_target_resume_anchor() {
.await
.expect("existing intent should be stored on the non-target anchor");
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
replacement_resume_disk: Mutex::new(Some(anchor.clone())),
..Default::default()
@@ -2998,6 +3001,7 @@ async fn replacement_dropped_waiter_keeps_executor_until_format_and_failure_pers
let entered = Arc::new(tokio::sync::Notify::new());
let release = Arc::new(tokio::sync::Notify::new());
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
replacement_target_identity_sequences: Mutex::new(VecDeque::from([vec![identity.clone()], vec![identity]])),
replacement_execution_fixture: Mutex::new(Some(execution)),
@@ -3012,7 +3016,9 @@ async fn replacement_dropped_waiter_keeps_executor_until_format_and_failure_pers
let task = task.clone();
async move { task.execute().await }
});
entered.notified().await;
tokio::time::timeout(Duration::from_secs(5), entered.notified())
.await
.expect("owned format I/O must start");
waiter.abort();
assert!(waiter.await.expect_err("waiter aborted").is_cancelled());
task.cancel_token.cancel();
@@ -3122,6 +3128,7 @@ async fn replacement_format_failures_keep_the_durable_budget_across_new_executor
request.heal_endpoints = vec!["replacement-a".to_string()];
for _attempt in 1..=3 {
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
resume_disk: Mutex::new(Some(disk.clone())),
format_error: Mutex::new(Some(Error::other("injected format failure"))),
@@ -5683,6 +5690,7 @@ async fn replacement_owned_executor_keeps_manager_drain_pending_through_failure_
let entered = Arc::new(tokio::sync::Notify::new());
let release = Arc::new(tokio::sync::Notify::new());
let storage = Arc::new(MockStorage {
bucket_incarnation_id: Mutex::new(Some(Uuid::from_u128(1))),
replacement_target_identities_ready: Mutex::new(true),
replacement_target_identity_sequences: Mutex::new(VecDeque::from([vec![identity.clone()], vec![identity]])),
replacement_execution_fixture: Mutex::new(Some(execution)),
@@ -5699,7 +5707,9 @@ async fn replacement_owned_executor_keeps_manager_drain_pending_through_failure_
let task = task.clone();
async move { task.execute_owned(Some(parent)).await }
});
entered.notified().await;
tokio::time::timeout(Duration::from_secs(5), entered.notified())
.await
.expect("owned format I/O must start");
waiter.abort();
assert!(waiter.await.expect_err("waiter aborted").is_cancelled());
task.cancel_token.cancel();
+387 -81
View File
@@ -15,7 +15,6 @@
use std::{
collections::{HashMap, HashSet},
ops::{Deref, DerefMut},
ptr,
sync::{Arc, Mutex, Weak},
};
@@ -64,21 +63,46 @@ impl Default for CacheState {
pub struct Cache {
state: ArcSwap<CacheState>,
write_lock: Mutex<()>,
reload_lock: AsyncMutex<()>,
reload_changes: Mutex<Weak<Mutex<ReloadChanges>>>,
service_account_mutation_lock: AsyncMutex<()>,
sts_account_mutation_locks: Arc<StsMutationLockRegistry>,
sts_account_mutation_locks: Arc<MutationLockRegistry>,
parent_mutation_locks: Arc<MutationLockRegistry>,
}
struct StsMutationLockRegistry {
locks: Mutex<HashMap<String, Weak<AsyncMutex<StsMutationLockState>>>>,
#[derive(Default)]
struct MutationLockRegistry {
locks: Mutex<HashMap<String, Weak<AsyncMutex<MutationLockState>>>>,
}
pub(crate) struct StsMutationLockState {
pub(crate) struct MutationLockState {
access_key: String,
registry: Weak<StsMutationLockRegistry>,
lock: Weak<AsyncMutex<StsMutationLockState>>,
registry: Weak<MutationLockRegistry>,
lock: Weak<AsyncMutex<MutationLockState>>,
}
impl Drop for StsMutationLockState {
impl MutationLockRegistry {
fn lock_for(self: &Arc<Self>, access_key: &str) -> Arc<AsyncMutex<MutationLockState>> {
let mut locks = self.locks.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
if let Some(lock) = locks.get(access_key).and_then(Weak::upgrade) {
return lock;
}
let registry = Arc::downgrade(self);
let access_key_owned = access_key.to_string();
let lock = Arc::new_cyclic(|lock| {
AsyncMutex::new(MutationLockState {
access_key: access_key_owned,
registry,
lock: lock.clone(),
})
});
locks.insert(access_key.to_string(), Arc::downgrade(&lock));
lock
}
}
impl Drop for MutationLockState {
fn drop(&mut self) {
let Some(registry) = self.registry.upgrade() else {
return;
@@ -95,10 +119,52 @@ impl Default for Cache {
Self {
state: ArcSwap::new(Arc::new(CacheState::default())),
write_lock: Mutex::new(()),
reload_lock: AsyncMutex::new(()),
reload_changes: Mutex::new(Weak::new()),
service_account_mutation_lock: AsyncMutex::new(()),
sts_account_mutation_locks: Arc::new(StsMutationLockRegistry {
locks: Mutex::new(HashMap::new()),
}),
sts_account_mutation_locks: Arc::new(MutationLockRegistry::default()),
parent_mutation_locks: Arc::new(MutationLockRegistry::default()),
}
}
}
#[derive(Clone, Copy, Hash, PartialEq, Eq)]
pub(crate) enum CachePartition {
PolicyDocs,
Users,
UserPolicies,
StsAccounts,
StsPolicies,
Groups,
UserGroupMemberships,
GroupPolicies,
}
type ReloadChanges = HashMap<CachePartition, HashSet<String>>;
/// A reload journals only keys changed during its own lifetime, including
/// deletions. Unrelated STS traffic cannot suppress a persisted revocation.
pub(crate) struct CacheReload<'a> {
_guard: tokio::sync::MutexGuard<'a, ()>,
changes: Arc<Mutex<ReloadChanges>>,
}
impl CacheReload<'_> {
pub(crate) fn merge_partition<T: Clone>(
&self,
partition: CachePartition,
loaded: &mut CacheEntity<T>,
current: &CacheEntity<T>,
) {
let changes = self.changes.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
if let Some(keys) = changes.get(&partition) {
for key in keys {
if let Some(value) = current.get(key) {
loaded.insert(key.clone(), value.clone());
} else {
loaded.remove(key);
}
}
}
}
}
@@ -106,31 +172,27 @@ impl Default for Cache {
pub(crate) type CacheSnapshot = Guard<Arc<CacheState>>;
impl Cache {
pub(crate) async fn begin_reload(&self) -> CacheReload<'_> {
// Registration and publication both take write_lock before the journal
// lock, so no mutation can fall between the reload and its journal.
let guard = self.reload_lock.lock().await;
let changes = Arc::new(Mutex::new(HashMap::new()));
let _write = self.write_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
*self.reload_changes.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) = Arc::downgrade(&changes);
CacheReload { _guard: guard, changes }
}
pub(crate) fn service_account_mutation_lock(&self) -> &AsyncMutex<()> {
&self.service_account_mutation_lock
}
pub(crate) fn sts_account_mutation_lock(&self, access_key: &str) -> Arc<AsyncMutex<StsMutationLockState>> {
let mut locks = self
.sts_account_mutation_locks
.locks
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
if let Some(lock) = locks.get(access_key).and_then(Weak::upgrade) {
return lock;
}
pub(crate) fn sts_account_mutation_lock(&self, access_key: &str) -> Arc<AsyncMutex<MutationLockState>> {
self.sts_account_mutation_locks.lock_for(access_key)
}
let registry = Arc::downgrade(&self.sts_account_mutation_locks);
let access_key_owned = access_key.to_string();
let lock = Arc::new_cyclic(|lock| {
AsyncMutex::new(StsMutationLockState {
access_key: access_key_owned,
registry,
lock: lock.clone(),
})
});
locks.insert(access_key.to_string(), Arc::downgrade(&lock));
lock
/// Orders derived-credential publication with changes to its regular parent.
pub(crate) fn parent_mutation_lock(&self, access_key: &str) -> Arc<AsyncMutex<MutationLockState>> {
self.parent_mutation_locks.lock_for(access_key)
}
pub(crate) fn snapshot(&self) -> CacheSnapshot {
@@ -142,7 +204,13 @@ impl Cache {
let current = self.state.load_full();
let mut locked = LockedCache {
state: CacheState::clone(&current),
#[cfg(test)]
current_ptr: Arc::as_ptr(&current),
reload_changes: self
.reload_changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.upgrade(),
dirty: false,
};
let ret = f(&mut locked);
@@ -215,7 +283,9 @@ impl Cache {
pub(crate) struct LockedCache {
state: CacheState,
#[cfg(test)]
current_ptr: *const CacheState,
reload_changes: Option<Arc<Mutex<ReloadChanges>>>,
dirty: bool,
}
@@ -224,11 +294,19 @@ impl LockedCache {
&self.state
}
#[cfg(test)]
pub(crate) fn matches_snapshot(&self, snapshot: &CacheSnapshot) -> bool {
ptr::eq(self.current_ptr, Arc::as_ptr(snapshot))
std::ptr::eq(self.current_ptr, Arc::as_ptr(snapshot))
}
fn exec<T: Clone>(target: &mut Arc<CacheEntity<T>>, t: OffsetDateTime, mut op: impl FnMut(&mut CacheEntity<T>)) -> bool {
fn exec<T: Clone>(
target: &mut Arc<CacheEntity<T>>,
t: OffsetDateTime,
changes: Option<&Arc<Mutex<ReloadChanges>>>,
partition: CachePartition,
key: &str,
mut op: impl FnMut(&mut CacheEntity<T>),
) -> bool {
if target.load_time >= t {
return false;
}
@@ -236,6 +314,14 @@ impl LockedCache {
let mut new = CacheEntity::clone(target);
op(&mut new);
*target = Arc::new(new);
if let Some(changes) = changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(partition)
.or_default()
.insert(key.to_string());
}
true
}
@@ -244,139 +330,315 @@ impl LockedCache {
}
pub(crate) fn replace_policy_docs(&mut self, value: CacheEntity<PolicyDoc>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::PolicyDocs)
.or_default()
.extend(self.state.policy_docs.keys().chain(value.keys()).cloned());
}
self.state.policy_docs = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_users(&mut self, value: CacheEntity<UserIdentity>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::Users)
.or_default()
.extend(self.state.users.keys().chain(value.keys()).cloned());
}
self.state.users = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_user_policies(&mut self, value: CacheEntity<MappedPolicy>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::UserPolicies)
.or_default()
.extend(self.state.user_policies.keys().chain(value.keys()).cloned());
}
self.state.user_policies = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_sts_accounts(&mut self, value: CacheEntity<UserIdentity>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::StsAccounts)
.or_default()
.extend(self.state.sts_accounts.keys().chain(value.keys()).cloned());
}
self.state.sts_accounts = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_sts_policies(&mut self, value: CacheEntity<MappedPolicy>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::StsPolicies)
.or_default()
.extend(self.state.sts_policies.keys().chain(value.keys()).cloned());
}
self.state.sts_policies = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_groups(&mut self, value: CacheEntity<GroupInfo>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::Groups)
.or_default()
.extend(self.state.groups.keys().chain(value.keys()).cloned());
}
self.state.groups = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_group_policies(&mut self, value: CacheEntity<MappedPolicy>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::GroupPolicies)
.or_default()
.extend(self.state.group_policies.keys().chain(value.keys()).cloned());
}
self.state.group_policies = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn replace_user_group_memberships(&mut self, value: CacheEntity<HashSet<String>>) {
if let Some(changes) = &self.reload_changes {
changes
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.entry(CachePartition::UserGroupMemberships)
.or_default()
.extend(self.state.user_group_memberships.keys().chain(value.keys()).cloned());
}
self.state.user_group_memberships = Self::replaced(value);
self.dirty = true;
}
pub(crate) fn add_or_update_policy_doc(&mut self, key: &str, value: &PolicyDoc, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.policy_docs, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.policy_docs,
t,
self.reload_changes.as_ref(),
CachePartition::PolicyDocs,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_user(&mut self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.users, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.users,
t,
self.reload_changes.as_ref(),
CachePartition::Users,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_user_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.user_policies, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.user_policies,
t,
self.reload_changes.as_ref(),
CachePartition::UserPolicies,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_sts_account(&mut self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.sts_accounts, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.sts_accounts,
t,
self.reload_changes.as_ref(),
CachePartition::StsAccounts,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_sts_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.sts_policies, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.sts_policies,
t,
self.reload_changes.as_ref(),
CachePartition::StsPolicies,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_group(&mut self, key: &str, value: &GroupInfo, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.groups, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.groups,
t,
self.reload_changes.as_ref(),
CachePartition::Groups,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_user_group_membership(&mut self, key: &str, value: &HashSet<String>, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.user_group_memberships, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.user_group_memberships,
t,
self.reload_changes.as_ref(),
CachePartition::UserGroupMemberships,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn add_or_update_group_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.group_policies, t, |map| {
map.insert(key.to_string(), value.clone());
});
self.dirty |= Self::exec(
&mut self.state.group_policies,
t,
self.reload_changes.as_ref(),
CachePartition::GroupPolicies,
key,
|map| {
map.insert(key.to_string(), value.clone());
},
);
}
pub(crate) fn delete_policy_doc(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.policy_docs, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.policy_docs,
t,
self.reload_changes.as_ref(),
CachePartition::PolicyDocs,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_user(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.users, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.users,
t,
self.reload_changes.as_ref(),
CachePartition::Users,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_user_policy(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.user_policies, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.user_policies,
t,
self.reload_changes.as_ref(),
CachePartition::UserPolicies,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_user_group_membership(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.user_group_memberships, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.user_group_memberships,
t,
self.reload_changes.as_ref(),
CachePartition::UserGroupMemberships,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_sts_account(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.sts_accounts, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.sts_accounts,
t,
self.reload_changes.as_ref(),
CachePartition::StsAccounts,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_sts_policy(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.sts_policies, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.sts_policies,
t,
self.reload_changes.as_ref(),
CachePartition::StsPolicies,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_group(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.groups, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.groups,
t,
self.reload_changes.as_ref(),
CachePartition::Groups,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn delete_group_policy(&mut self, key: &str, t: OffsetDateTime) {
self.dirty |= Self::exec(&mut self.state.group_policies, t, |map| {
map.remove(key);
});
self.dirty |= Self::exec(
&mut self.state.group_policies,
t,
self.reload_changes.as_ref(),
CachePartition::GroupPolicies,
key,
|map| {
map.remove(key);
},
);
}
pub(crate) fn build_user_group_memberships(&mut self) {
@@ -494,6 +756,50 @@ mod tests {
use crate::cache::Cache;
use crate::store::MappedPolicy;
#[tokio::test]
async fn reload_journal_preserves_same_key_mutations_and_expires_on_cancellation() {
use super::{CacheEntity, CachePartition};
let cache = Cache::default();
let stale = UserIdentity {
version: 1,
..Default::default()
};
let current = UserIdentity {
version: 2,
..Default::default()
};
let reload = cache.begin_reload().await;
cache.add_or_update_user("updated", &current, OffsetDateTime::now_utc());
cache.add_or_update_user("removed", &current, OffsetDateTime::now_utc());
cache.delete_user("removed", OffsetDateTime::now_utc());
// Even an absent key's deletion is recorded: storage may have returned
// an entry created and deleted entirely during the reload's reads.
cache.delete_user("absent", OffsetDateTime::now_utc());
let mut loaded = CacheEntity::new(std::collections::HashMap::from([
("updated".to_string(), stale.clone()),
("removed".to_string(), stale.clone()),
("absent".to_string(), stale.clone()),
("disk-only".to_string(), stale),
]));
cache.with_write_lock(|locked| {
reload.merge_partition(CachePartition::Users, &mut loaded, &locked.state().users);
locked.replace_users(loaded);
});
let snapshot = cache.snapshot();
assert_eq!(snapshot.users["updated"].version, 2);
assert!(!snapshot.users.contains_key("removed"));
assert!(!snapshot.users.contains_key("absent"));
assert!(snapshot.users.contains_key("disk-only"));
drop(reload);
assert!(cache.reload_changes.lock().expect("journal lock").upgrade().is_none());
let cancelled = cache.begin_reload().await;
cache.delete_user("disk-only", OffsetDateTime::now_utc());
drop(cancelled);
assert!(cache.reload_changes.lock().expect("journal lock").upgrade().is_none());
let next = cache.begin_reload().await;
assert!(next.changes.lock().expect("new journal").is_empty());
}
#[test]
fn sts_mutation_locks_are_keyed_and_prune_unused_entries() {
let cache = Cache::default();
+10
View File
@@ -16,6 +16,7 @@ use super::{FederatedClaims, VerifiedFederatedIdentity};
use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC};
use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive};
use rustfs_utils::HashAlgorithm;
use serde_json::Value;
use std::{collections::HashMap, sync::Arc};
use tracing::debug;
@@ -66,6 +67,15 @@ impl FederatedAuthorization {
&self.provider_id
}
/// Issuer from the verified upstream identity.
pub fn issuer(&self) -> Option<&str> {
self.claims
.raw
.get("iss")
.and_then(Value::as_str)
.filter(|issuer| !issuer.is_empty())
}
pub fn subject(&self) -> &str {
&self.claims.sub
}
+149 -7
View File
@@ -415,6 +415,16 @@ where
self.api.save_iam_config(IAMFormat::new_version_1(), path).await
}
/// Current built-in IAM group membership, independent of credential snapshots.
pub fn user_group_memberships(&self, name: &str) -> Vec<String> {
self.cache
.snapshot()
.user_group_memberships
.get(name)
.map(|groups| groups.iter().cloned().collect())
.unwrap_or_default()
}
pub async fn get_user(&self, access_key: &str) -> Option<UserIdentity> {
let cache = self.cache.snapshot();
cache
@@ -1005,6 +1015,17 @@ where
}
pub async fn sts_policy_db_get(&self, name: &str, groups: &Option<Vec<String>>) -> Result<Vec<String>> {
self.sts_policy_db_get_with_identity_mapping(name, groups)
.await
.map(|(policies, _)| policies)
}
/// Resolve STS policies and whether that same lookup found a nonempty identity mapping.
pub async fn sts_policy_db_get_with_identity_mapping(
&self,
name: &str,
groups: &Option<Vec<String>>,
) -> Result<(Vec<String>, bool)> {
if name.is_empty() {
return Err(Error::InvalidArgument);
}
@@ -1031,7 +1052,10 @@ where
}
};
self.policy_db_get_with_groups(mapped_policy.to_slice(), groups).await
let direct_policies = mapped_policy.to_slice();
let has_identity_mapping = !direct_policies.is_empty();
let policies = self.policy_db_get_with_groups(direct_policies, groups).await?;
Ok((policies, has_identity_mapping))
}
async fn policy_db_get_with_groups(&self, mut policies: Vec<String>, groups: &Option<Vec<String>>) -> Result<Vec<String>> {
@@ -1158,7 +1182,7 @@ where
if let Some(groups) = u.credentials.groups.as_ref() {
for group in groups.iter() {
if groups_cache.get(group).filter(|v| v.status == STATUS_DISABLED).is_some() {
return Ok((Vec::new(), OffsetDateTime::now_utc()));
continue;
}
let mp = match group_policies.get(group) {
@@ -1189,7 +1213,7 @@ where
for group in user_group_memberships.get(name).cloned().unwrap_or_default().iter() {
if groups_cache.get(group).filter(|v| v.status == STATUS_DISABLED).is_some() {
return Ok((Vec::new(), OffsetDateTime::now_utc()));
continue;
}
let mp = match group_policies.get(group) {
@@ -1486,7 +1510,27 @@ where
args: &AddOrUpdateUserReq,
updated_at: OffsetDateTime,
) -> Result<OffsetDateTime> {
self.add_user_at_with_secret_change(access_key, args, updated_at)
.await
.map(|(updated_at, _, _guard)| updated_at)
}
/// Return whether this write replaced a different regular-user secret.
pub(crate) async fn add_user_at_with_secret_change(
&self,
access_key: &str,
args: &AddOrUpdateUserReq,
updated_at: OffsetDateTime,
) -> Result<(OffsetDateTime, bool, tokio::sync::OwnedMutexGuard<crate::cache::MutationLockState>)> {
let parent_lock = self.cache.parent_mutation_lock(access_key);
let parent_guard = parent_lock.lock_owned().await;
let _service_guard = self.cache.service_account_mutation_lock().lock().await;
let sts_lock = self.cache.sts_account_mutation_lock(access_key);
let _sts_guard = sts_lock.lock().await;
let cache = self.cache.snapshot();
if cache.sts_accounts.contains_key(access_key) {
return Err(Error::AccessKeyAlreadyExists);
}
let users = Arc::clone(&cache.users);
if let Some(x) = users.get(access_key) {
debug!(
@@ -1496,10 +1540,13 @@ where
state = "existing",
"Updating existing IAM user"
);
if x.credentials.is_temp() {
if x.credentials.is_temp() || x.credentials.is_service_account() {
return Err(Error::AccessKeyAlreadyExists);
}
}
let secret_changed = users
.get(access_key)
.is_some_and(|user| user.credentials.secret_key != args.secret_key);
drop(cache);
drop(users);
@@ -1523,7 +1570,7 @@ where
self.update_user_with_claims(access_key, user_entry)?;
Ok(updated_at)
Ok((updated_at, secret_changed, parent_guard))
}
pub async fn delete_user(&self, access_key: &str, utype: UserType) -> Result<()> {
@@ -1540,12 +1587,17 @@ where
return Err(Error::InvalidArgument);
}
let parent_lock = (utype == UserType::Reg).then(|| self.cache.parent_mutation_lock(access_key));
let _parent_guard = match &parent_lock {
Some(lock) => Some(lock.lock().await),
None => None,
};
let sts_mutation_lock = (utype == UserType::Sts).then(|| self.cache.sts_account_mutation_lock(access_key));
let _sts_mutation_guard = match &sts_mutation_lock {
Some(lock) => Some(lock.lock().await),
None => None,
};
let _service_account_guard = if utype == UserType::Svc {
let _service_account_guard = if matches!(utype, UserType::Svc | UserType::Reg) {
Some(self.cache.service_account_mutation_lock().lock().await)
} else {
None
@@ -1640,17 +1692,46 @@ where
}
pub async fn update_user_secret_key(&self, access_key: &str, secret_key: &str) -> Result<(OffsetDateTime, AccountStatus)> {
self.update_user_secret_key_with_guard(access_key, secret_key, None)
.await
.map(|(updated_at, status, _guard)| (updated_at, status))
}
pub(crate) async fn update_user_secret_key_with_guard(
&self,
access_key: &str,
secret_key: &str,
expected_secret: Option<&str>,
) -> Result<(
OffsetDateTime,
AccountStatus,
tokio::sync::OwnedMutexGuard<crate::cache::MutationLockState>,
)> {
if access_key.is_empty() || secret_key.is_empty() {
return Err(Error::InvalidArgument);
}
let parent_lock = self.cache.parent_mutation_lock(access_key);
let parent_guard = parent_lock.lock_owned().await;
let cache = self.cache.snapshot();
let users = Arc::clone(&cache.users);
let u = match users.get(access_key) {
Some(u) => u,
None if expected_secret.is_some() => return Err(Error::IAMActionNotAllowed),
None => return Err(Error::NoSuchUser(access_key.to_string())),
};
if let Some(expected) = expected_secret {
use subtle::ConstantTimeEq as _;
if !u.credentials.is_valid()
|| u.credentials.is_temp()
|| u.credentials.is_service_account()
|| !bool::from(u.credentials.secret_key.as_bytes().ct_eq(expected.as_bytes()))
{
return Err(Error::IAMActionNotAllowed);
}
}
let mut cred = u.credentials.clone();
cred.secret_key = secret_key.to_string();
@@ -1672,7 +1753,7 @@ where
.await?;
self.update_user_with_claims(access_key, u)?;
Ok((updated_at, status))
Ok((updated_at, status, parent_guard))
}
/// Add SSH public key for a user (for SFTP authentication)
@@ -1681,6 +1762,8 @@ where
return Err(Error::InvalidArgument);
}
let parent_lock = self.cache.parent_mutation_lock(access_key);
let _parent_guard = parent_lock.lock().await;
let cache = self.cache.snapshot();
let users = Arc::clone(&cache.users);
let u = match users.get(access_key) {
@@ -1720,6 +1803,8 @@ where
return Err(Error::InvalidArgument);
}
let parent_lock = self.cache.parent_mutation_lock(access_key);
let _parent_guard = parent_lock.lock().await;
let cache = self.cache.snapshot();
let users = Arc::clone(&cache.users);
let u = match users.get(access_key) {
@@ -2238,6 +2323,11 @@ where
Ok(())
}
pub async fn user_notification_handler(&self, name: &str, user_type: UserType) -> Result<()> {
let parent_lock = (user_type == UserType::Reg).then(|| self.cache.parent_mutation_lock(name));
let _parent_guard = match &parent_lock {
Some(lock) => Some(lock.lock().await),
None => None,
};
let sts_mutation_lock = (user_type == UserType::Sts).then(|| self.cache.sts_account_mutation_lock(name));
let _sts_mutation_guard = match &sts_mutation_lock {
Some(lock) => Some(lock.lock().await),
@@ -2985,6 +3075,58 @@ mod tests {
}
}
#[tokio::test]
async fn disabled_group_preserves_direct_and_other_group_policies() {
for credential_groups in [false, true] {
let iam = build_test_iam_cache(FailingInitialLoadStore);
let groups = vec!["disabled".to_string(), "enabled".to_string()];
let user = UserIdentity::new(Credentials {
access_key: "alice".to_string(),
secret_key: "alice-secret".to_string(),
status: STATUS_ENABLED.to_string(),
groups: credential_groups.then(|| groups.clone()),
..Default::default()
});
let now = OffsetDateTime::now_utc();
iam.cache.add_or_update_user("alice", &user, now);
iam.cache
.add_or_update_user_policy("alice", &MappedPolicy::new("direct-deny"), now);
if !credential_groups {
iam.cache
.add_or_update_user_group_membership("alice", &groups.iter().cloned().collect(), now);
}
for (group, status, policy) in [
("disabled", STATUS_DISABLED, "disabled-grant"),
("enabled", STATUS_ENABLED, "other-group-deny"),
] {
iam.cache.add_or_update_group(
group,
&GroupInfo {
status: status.to_string(),
members: vec!["alice".to_string()],
..Default::default()
},
now,
);
iam.cache.add_or_update_group_policy(group, &MappedPolicy::new(policy), now);
}
let policies: HashSet<_> = iam
.policy_db_get("alice", &None)
.await
.expect("user policy lookup")
.into_iter()
.collect();
assert_eq!(policies, HashSet::from(["direct-deny".to_string(), "other-group-deny".to_string()]));
assert!(
iam.policy_db_get_internal("disabled", true, false)
.await
.expect("disabled group lookup")
.0
.is_empty()
);
}
}
#[tokio::test]
async fn sts_policy_lookup_ignores_colliding_regular_user_mapping() {
let iam = build_test_iam_cache(FailingInitialLoadStore);
+59 -14
View File
@@ -18,18 +18,19 @@ use super::{
use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC};
use openidconnect::core::{CoreIdToken, CoreIdTokenClaims, CoreIdTokenVerifier, CoreJsonWebKeySet, CoreJwsSigningAlgorithm};
use openidconnect::{
AsyncHttpClient, ClaimsVerificationError, ClientId, ClientSecret, DiscoveryError, IssuerUrl, JsonWebKeySetUrl, NonceVerifier,
AsyncHttpClient, ClaimsVerificationError, ClientId, DiscoveryError, IssuerUrl, JsonWebKeySetUrl, NonceVerifier,
ProviderMetadataWithLogout, SignatureVerificationError, TokenUrl,
};
use serde::Deserialize;
use std::collections::HashMap;
use std::sync::RwLock;
use std::sync::{Arc, Mutex, RwLock};
use std::time::{Duration as StdDuration, Instant};
use tokio::time::sleep;
use tracing::{debug, warn};
use url::Url;
pub(super) const OIDC_JWKS_REFRESH_INTERVAL: StdDuration = StdDuration::from_secs(24 * 60 * 60);
const OIDC_JWKS_REFRESH_COOLDOWN: StdDuration = StdDuration::from_secs(60);
pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRIES: usize = 3;
pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY: StdDuration = StdDuration::from_millis(50);
pub(super) const OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY: &str = "JWKS request blocked by outbound policy";
@@ -97,9 +98,14 @@ pub(super) struct ProviderState {
pub(super) discovered_at: Instant,
}
type RefreshAttempt = Option<(Instant, Result<ProviderState, String>)>;
type ProviderRefreshLock = Arc<tokio::sync::Mutex<RefreshAttempt>>;
#[derive(Clone)]
pub(super) struct ProviderRuntime {
states: RwLock<HashMap<String, ProviderState>>,
http_client: ReqwestHttpClient,
states: Arc<RwLock<HashMap<String, ProviderState>>>,
http_client: Arc<ReqwestHttpClient>,
refreshes: Arc<Mutex<HashMap<String, ProviderRefreshLock>>>,
}
// Workload issuers do not implement the browser authorization flow. Keep their
@@ -200,16 +206,25 @@ impl DiscoveredProviderMetadata {
pub(super) fn verifier(&self, config: &OidcProviderConfig) -> CoreIdTokenVerifier<'static> {
let client_id = ClientId::new(config.client_id.clone());
let secret = config.client_secret.as_ref().map(|secret| ClientSecret::new(secret.clone()));
let (issuer, jwks, algorithms) = match self {
Self::Console(metadata) => (metadata.issuer(), metadata.jwks(), metadata.id_token_signing_alg_values_supported()),
Self::Workload(metadata) => (&metadata.issuer, &metadata.jwks, &metadata.id_token_signing_alg_values_supported),
};
let verifier = match secret {
Some(secret) => CoreIdTokenVerifier::new_confidential_client(client_id, secret, issuer.clone(), jwks.clone()),
None => CoreIdTokenVerifier::new_public_client(client_id, issuer.clone(), jwks.clone()),
};
verifier.set_allowed_algs(algorithms.clone())
// Caller-supplied tokens must prove possession of the issuer's signing
// key. A client secret is shared with clients and cannot provide that proof.
let algorithms = algorithms
.iter()
.filter(|algorithm| {
!matches!(
algorithm,
CoreJwsSigningAlgorithm::HmacSha256
| CoreJwsSigningAlgorithm::HmacSha384
| CoreJwsSigningAlgorithm::HmacSha512
)
})
.cloned()
.collect::<Vec<_>>();
CoreIdTokenVerifier::new_public_client(client_id, issuer.clone(), jwks.clone()).set_allowed_algs(algorithms)
}
}
@@ -273,8 +288,9 @@ impl ProviderState {
impl ProviderRuntime {
pub(super) fn new(http_client: ReqwestHttpClient, states: HashMap<String, ProviderState>) -> Self {
Self {
states: RwLock::new(states),
http_client,
states: Arc::new(RwLock::new(states)),
http_client: Arc::new(http_client),
refreshes: Arc::new(Mutex::new(HashMap::new())),
}
}
@@ -326,9 +342,38 @@ impl ProviderRuntime {
provider_id: &str,
config: &OidcProviderConfig,
) -> Result<ProviderState, String> {
self.recover_provider_state(provider_id, config)
let refresh = self
.refreshes
.lock()
.map_err(|_| "provider refresh registry is unavailable".to_string())?
.entry(provider_id.to_string())
.or_default()
.clone();
let mut attempt = refresh.lock_owned().await;
if let Some((completed, result)) = attempt.as_ref()
&& completed.elapsed() < OIDC_JWKS_REFRESH_COOLDOWN
{
return result.clone();
}
// The admitted fetch owns the per-provider lock. A disconnected request
// cannot abandon discovery while consuming the provider's cooldown.
// Waiting callers share its completed result, including genuine failures.
let runtime = tokio::runtime::Handle::try_current().map_err(|err| err.to_string())?;
let provider_runtime = self.clone();
let provider_id = provider_id.to_string();
let config = config.clone();
runtime
.spawn(async move {
let result = provider_runtime
.recover_provider_state(&provider_id, &config)
.await
.map_err(|err| err.to_string());
*attempt = Some((Instant::now(), result.clone()));
result
})
.await
.map_err(|err| err.to_string())
.map_err(|err| err.to_string())?
}
pub(super) async fn recover_provider_state(
+162 -17
View File
@@ -180,7 +180,7 @@ fn workload_metadata_rejects_duplicate_fields() {
}
}
#[test]
fn workload_verifier_preserves_algorithm_secret_and_audience_policy() {
fn caller_supplied_tokens_cannot_be_signed_with_the_client_secret() {
let secret = Nonce::new_random().secret().to_string();
let mut config = build_mocked_oidc_provider_config("workload", "https://issuer.example.com");
config.client_secret = Some(secret.clone());
@@ -195,7 +195,8 @@ fn workload_verifier_preserves_algorithm_secret_and_audience_policy() {
let token: CoreIdToken = signed.parse().unwrap();
for workload in [false, true] {
for (algorithms, accepted) in [
(serde_json::json!(["HS256", "unsupported-future-algorithm"]), true),
(serde_json::json!(["RS256", "HS256"]), false),
(serde_json::json!(["HS256", "unsupported-future-algorithm"]), false),
(serde_json::json!(["ES256"]), false),
(serde_json::json!(["unsupported-future-algorithm"]), false),
] {
@@ -216,21 +217,6 @@ fn workload_verifier_preserves_algorithm_secret_and_audience_policy() {
accepted,
"workload={workload}, algorithms={algorithms}"
);
if accepted {
assert!(
token.claims(&metadata.verifier(&config), |_: Option<&Nonce>| Ok(())).is_err(),
"additional audiences require explicit trust"
);
let mut wrong_secret = config.clone();
wrong_secret.client_secret = Some(Nonce::new_random().secret().to_string());
let verifier = metadata
.verifier(&wrong_secret)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
assert!(
token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_err(),
"incorrect client secret must fail"
);
}
}
}
}
@@ -274,6 +260,19 @@ fn configured_verifier_rejects_malformed_hmac_signature_lengths() {
let metadata = DiscoveredProviderMetadata::parse(&serde_json::to_vec(&document).expect("encode metadata"), true)
.expect("parse configured provider metadata");
let verifier = metadata.verifier(&config);
assert!(
verify_id_token_claims(&valid, &verifier, |_: Option<&Nonce>| Ok(())).is_err(),
"caller-supplied HMAC tokens must be rejected"
);
// The authorization-code flow obtains tokens directly from the
// provider and still supports confidential-client verification.
let verifier = openidconnect::core::CoreIdTokenVerifier::new_confidential_client(
openidconnect::ClientId::new(config.client_id.clone()),
openidconnect::ClientSecret::new(secret.to_owned()),
metadata.issuer().clone(),
openidconnect::core::CoreJsonWebKeySet::new(vec![]),
)
.set_allowed_algs(vec![serde_json::from_value(serde_json::json!(name)).expect("parse signing algorithm")]);
assert!(
verify_id_token_claims(&valid, &verifier, |_: Option<&Nonce>| Ok(())).is_ok(),
"valid {name} token must pass for workload={workload}"
@@ -780,3 +779,149 @@ async fn test_validate_oidc_provider_config_returns_detailed_errors() {
assert!(err.contains(&format!("issuer '{base}/application/o/rustfs/'")));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn jwks_refresh_is_single_flight_and_survives_request_cancellation() {
let (initial_key, initial_jwk) = oidc_es256_key_and_jwk("initial");
let (rotated_key, rotated_jwk) = oidc_es256_key_and_jwk("rotated");
let (wrong_key, _) = oidc_es256_key_and_jwk("wrong");
let requests = Arc::new(AtomicUsize::new(0));
let started = Arc::new(tokio::sync::Notify::new());
let (release_tx, release_rx) = std::sync::mpsc::channel();
let server_requests = Arc::clone(&requests);
let server_started = Arc::clone(&started);
let Some((base, server)) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), format!("{base}/jwks"), "/jwks".to_string()),
6,
"ES256",
false,
move |fetch| {
let request = server_requests.fetch_add(1, Ordering::SeqCst);
if request == 2 {
server_started.notify_one();
release_rx
.recv_timeout(StdDuration::from_secs(5))
.expect("release refresh response");
}
serde_json::json!({"keys": [if fetch == 0 { &initial_jwk } else { &rotated_jwk }]}).to_string()
},
) else {
return;
};
let mut config = build_mocked_oidc_provider_config("bounded-refresh", &base);
config.issuer = Some(base.clone());
let http = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).expect("allow test origin"));
let state = discover_provider(&config, &http).await.expect("initial discovery");
let sys = Arc::new(OidcSys {
configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]),
provider_runtime: ProviderRuntime::new(http, HashMap::from([(config.id.clone(), state)])),
state_store: OidcStateStore::new(),
pending_discovery: Default::default(),
});
let now = time::OffsetDateTime::now_utc().unix_timestamp();
let payload = serde_json::json!({"iss": base, "sub": "user", "aud": config.client_id, "iat": now, "exp": now + 300});
let sign = |kid: &str, claims: &serde_json::Value, key: &EncodingKey| {
let mut header = Header::new(Algorithm::ES256);
header.kid = Some(kid.to_string());
jsonwebtoken::encode(&header, claims, key).expect("sign test token")
};
for (claim, value) in [
("aud", serde_json::json!("wrong-audience")),
("exp", serde_json::json!(now - 300)),
] {
let mut invalid = payload.clone();
invalid[claim] = value;
assert!(
sys.verify_web_identity_token(&sign("initial", &invalid, &initial_key))
.await
.is_err()
);
}
assert_eq!(requests.load(Ordering::SeqCst), 2, "claim failures cannot refresh signing keys");
let invalid = sign("rotated", &payload, &wrong_key);
let attack = {
let sys = Arc::clone(&sys);
let invalid = invalid.clone();
tokio::spawn(async move { sys.verify_web_identity_token(&invalid).await })
};
tokio::time::timeout(StdDuration::from_secs(5), started.notified())
.await
.expect("refresh started");
attack.abort();
assert!(attack.await.expect_err("request canceled").is_cancelled());
let mut retries = Vec::new();
for _ in 0..16 {
let sys = Arc::clone(&sys);
let invalid = invalid.clone();
retries.push(tokio::spawn(async move { sys.verify_web_identity_token(&invalid).await }));
}
release_tx.send(()).expect("release provider");
for retry in retries {
assert!(
tokio::time::timeout(StdDuration::from_secs(5), retry)
.await
.expect("retry timeout")
.expect("retry join")
.is_err()
);
}
let rotated = sign("rotated", &payload, &rotated_key);
sys.verify_web_identity_token(&rotated)
.await
.expect("completed refresh must accept rotated keys after the original caller left");
assert_eq!(requests.load(Ordering::SeqCst), 4, "one discovery/JWKS pair for all rejected tokens");
sys.provider_runtime
.refresh_provider_state("independent-provider", &config)
.await
.expect("provider cooldowns must be independent");
assert_eq!(requests.load(Ordering::SeqCst), 6);
server.join().expect("mock server");
}
#[tokio::test]
async fn jwks_refresh_cools_down_failures_and_recovers_after_the_interval() {
let (_, initial_jwk) = oidc_es256_key_and_jwk("initial");
let (_, recovered_jwk) = oidc_es256_key_and_jwk("recovered");
let requests = Arc::new(AtomicUsize::new(0));
let server_requests = Arc::clone(&requests);
let Some((base, server)) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), format!("{base}/jwks"), "/jwks".to_string()),
6,
"ES256",
false,
move |fetch| {
server_requests.fetch_add(1, Ordering::SeqCst);
match fetch {
0 => serde_json::json!({"keys": [initial_jwk]}).to_string(),
1 => "invalid JWKS".to_string(),
_ => serde_json::json!({"keys": [recovered_jwk]}).to_string(),
}
},
) else {
return;
};
let mut config = build_mocked_oidc_provider_config("recovering-refresh", &base);
config.issuer = Some(base.clone());
let http = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).expect("allow test origin"));
let initial = discover_provider(&config, &http).await.expect("initial discovery");
let runtime = ProviderRuntime::new(http, HashMap::from([(config.id.clone(), initial.clone())]));
assert!(runtime.refresh_provider_state(&config.id, &config).await.is_err());
assert_eq!(requests.load(Ordering::SeqCst), 4);
for _ in 0..8 {
assert!(runtime.refresh_provider_state(&config.id, &config).await.is_err());
}
assert_eq!(requests.load(Ordering::SeqCst), 4, "completed failures also consume a bounded cooldown");
assert_eq!(
runtime.get_provider_state(&config.id).expect("cached keys").discovered_at,
initial.discovered_at
);
let refresh = runtime.refreshes.lock().expect("registry")[&config.id].clone();
refresh.lock().await.as_mut().expect("completed attempt").0 = Instant::now() - OIDC_JWKS_REFRESH_COOLDOWN;
runtime
.refresh_provider_state(&config.id, &config)
.await
.expect("retry after cooldown");
assert_eq!(requests.load(Ordering::SeqCst), 6);
assert!(runtime.get_provider_state(&config.id).expect("recovered keys").discovered_at > initial.discovered_at);
server.join().expect("mock server");
}
+8 -2
View File
@@ -33,8 +33,8 @@ use crate::{
};
use openidconnect::core::{CoreAuthenticationFlow, CoreClient, CoreIdToken};
use openidconnect::{
Audience, AuthType, AuthorizationCode, ClientId, ClientSecret, CsrfToken, LogoutRequest, Nonce, PkceCodeChallenge,
PkceCodeVerifier, PostLogoutRedirectUrl, RedirectUrl, RequestTokenError, Scope,
Audience, AuthType, AuthorizationCode, ClaimsVerificationError, ClientId, ClientSecret, CsrfToken, LogoutRequest, Nonce,
PkceCodeChallenge, PkceCodeVerifier, PostLogoutRedirectUrl, RedirectUrl, RequestTokenError, Scope,
};
use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive};
use rustfs_utils::logging::LogThrottle;
@@ -537,6 +537,9 @@ impl OidcSys {
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
let verified = verify_id_token_claims(id_token, &verifier, &Nonce::new(session.nonce.clone()));
if let Err(e) = verified {
if !matches!(e, ClaimsVerificationError::SignatureVerification(_)) {
return Err(format!("ID token verification failed: {e}"));
}
let verification_error = e.to_string();
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
@@ -767,6 +770,9 @@ impl OidcSys {
.verifier(&config)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
if let Err(e) = verify_id_token_claims(&id_token, &verifier, |_: Option<&Nonce>| Ok(())) {
if !matches!(e, ClaimsVerificationError::SignatureVerification(_)) {
return Err(format!("ID token verification failed: {e}"));
}
state = self
.provider_runtime
.refresh_provider_state(&provider_id, &config)
+14
View File
@@ -14,6 +14,20 @@
use std::sync::Arc;
pub(crate) fn is_unreadable_bucket_tags(error: &crate::error::Error) -> bool {
let crate::error::Error::Io(error) = error else {
return false;
};
let Some(storage_error) = error.get_ref().and_then(|inner| inner.downcast_ref::<IamStorageError>()) else {
return false;
};
rustfs_ecstore::api::bucket::metadata::unreadable_config_refusal(storage_error)
.is_some_and(|failure| failure.config_file == rustfs_ecstore::api::bucket::metadata::BUCKET_TAGGING_CONFIG)
}
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::bucket::metadata::UnreadableBucketConfig;
pub(crate) use rustfs_ecstore::api::config::RUSTFS_CONFIG_PREFIX as IAM_CONFIG_ROOT_PREFIX;
use rustfs_ecstore::api::config::com::{
delete_config as ecstore_delete_config, read_config_no_lock as ecstore_read_config_no_lock,
+146 -37
View File
@@ -1154,7 +1154,7 @@ impl Store for ObjectStore {
// node-counted lock quorum (rustfs#4304; startup contract rustfs#4056).
const LOAD_ALL_MODE: LoadMode = LoadMode::BootstrapNoLock;
let cache_snapshot = cache.snapshot();
let reload = cache.begin_reload().await;
let listed_config_items = self.list_all_iamconfig_items().await?;
let mut policy_docs_cache = CacheEntity::new(get_default_policies());
@@ -1201,26 +1201,20 @@ impl Store for ObjectStore {
}
// groups
let mut groups_cache = None;
let mut groups_cache = CacheEntity::default();
if let Some(item_name_list) = listed_config_items.get(GROUPS_LIST_KEY) {
let mut items_cache = CacheEntity::default();
for item in item_name_list.iter() {
let name = rustfs_utils::path::dir(item);
debug!(group = %name, "IAM group loaded");
if let Err(err) = self.load_group_with(&name, &mut items_cache, LOAD_ALL_MODE).await {
if let Err(err) = self.load_group_with(&name, &mut groups_cache, LOAD_ALL_MODE).await {
return Err(Error::other(format!("load group failed: {err}")));
};
}
groups_cache = Some(items_cache);
}
// user policies
let mut user_policies_cache = None;
let mut user_policies_cache = CacheEntity::default();
if let Some(item_name_list) = listed_config_items.get(POLICY_DB_USERS_LIST_KEY) {
let mut items_cache = CacheEntity::default();
// Load in fixed-size chunks so each mapping is fetched exactly once
// (backlog#806 — the split_off loop was O(n^2)).
for chunk in item_name_list.chunks(32) {
@@ -1235,32 +1229,26 @@ impl Store for ObjectStore {
let name = chunk[idx].trim_end_matches(".json").to_owned();
debug!(user = %name, "IAM user policy loaded");
items_cache.insert(name, p);
user_policies_cache.insert(name, p);
}
}
user_policies_cache = Some(items_cache);
}
// group policy
let mut group_policies_cache = None;
let mut group_policies_cache = CacheEntity::default();
if let Some(item_name_list) = listed_config_items.get(POLICY_DB_GROUPS_LIST_KEY) {
let mut items_cache = CacheEntity::default();
for item in item_name_list.iter() {
let name = item.trim_end_matches(".json");
debug!(group = %name, "IAM group policy loaded");
if let Err(err) = self
.load_mapped_policy_with(name, UserType::Reg, true, &mut items_cache, LOAD_ALL_MODE)
.load_mapped_policy_with(name, UserType::Reg, true, &mut group_policies_cache, LOAD_ALL_MODE)
.await
&& !is_err_no_such_policy(&err)
{
return Err(Error::other(format!("load group policy failed: {err}")));
};
}
group_policies_cache = Some(items_cache);
}
let mut sts_policies_cache = CacheEntity::default();
@@ -1330,25 +1318,48 @@ impl Store for ObjectStore {
}
}
#[cfg(test)]
if let Ok(barrier) = LOAD_ALL_COMMIT_BARRIER.try_with(std::sync::Arc::clone) {
barrier.0.notify_one();
barrier.1.notified().await;
}
cache.with_write_lock(|cache| {
if cache.matches_snapshot(&cache_snapshot) {
cache.replace_policy_docs(policy_docs_cache);
if let Some(groups_cache) = groups_cache {
cache.replace_groups(groups_cache);
}
if let Some(user_policies_cache) = user_policies_cache {
cache.replace_user_policies(user_policies_cache);
}
if let Some(group_policies_cache) = group_policies_cache {
cache.replace_group_policies(group_policies_cache);
}
cache.replace_users(user_items_cache);
cache.replace_sts_accounts(sts_items_cache);
cache.replace_sts_policies(sts_policies_cache);
cache.build_user_group_memberships();
} else {
warn!("IAM full reload cache commit skipped due to concurrent cache changes");
}
reload.merge_partition(
crate::cache::CachePartition::PolicyDocs,
&mut policy_docs_cache,
&cache.state().policy_docs,
);
reload.merge_partition(crate::cache::CachePartition::Groups, &mut groups_cache, &cache.state().groups);
reload.merge_partition(
crate::cache::CachePartition::UserPolicies,
&mut user_policies_cache,
&cache.state().user_policies,
);
reload.merge_partition(
crate::cache::CachePartition::GroupPolicies,
&mut group_policies_cache,
&cache.state().group_policies,
);
reload.merge_partition(crate::cache::CachePartition::Users, &mut user_items_cache, &cache.state().users);
reload.merge_partition(
crate::cache::CachePartition::StsAccounts,
&mut sts_items_cache,
&cache.state().sts_accounts,
);
reload.merge_partition(
crate::cache::CachePartition::StsPolicies,
&mut sts_policies_cache,
&cache.state().sts_policies,
);
cache.replace_policy_docs(policy_docs_cache);
cache.replace_groups(groups_cache);
cache.replace_user_policies(user_policies_cache);
cache.replace_group_policies(group_policies_cache);
cache.replace_users(user_items_cache);
cache.replace_sts_accounts(sts_items_cache);
cache.replace_sts_policies(sts_policies_cache);
cache.build_user_group_memberships();
});
Ok(())
@@ -1363,6 +1374,11 @@ fn map_delete_user_identity_error(name: &str, err: Error) -> Error {
}
}
#[cfg(test)]
tokio::task_local! {
static LOAD_ALL_COMMIT_BARRIER: std::sync::Arc<(tokio::sync::Notify, tokio::sync::Notify)>;
}
#[cfg(test)]
mod tests {
use super::{DecryptSource, LoadMode, ObjectStore, map_delete_user_identity_error};
@@ -1372,6 +1388,99 @@ mod tests {
use serial_test::serial;
use temp_env::with_vars;
#[test]
#[serial]
fn test_load_all_clears_partitions_with_empty_prefix() {
use crate::cache::Cache;
use crate::store::{GroupInfo, MappedPolicy, Store, UserType};
std::thread::Builder::new()
.stack_size(8 * 1024 * 1024)
.spawn(|| {
let runtime = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.expect("build IAM reload test runtime");
runtime.block_on(async {
let _credentials = test_cred();
let temp = tempfile::tempdir().expect("create IAM reload store directory");
let env = Box::pin(
rustfs_test_utils::TestECStoreEnv::builder()
.base_dir(temp.path())
.init_bucket_metadata(false)
.build(),
)
.await;
let store = ObjectStore::new(env.ecstore);
store
.save_group_info("reload-group", GroupInfo::new(vec!["reload-user".to_string()]))
.await
.expect("seed group membership");
store
.save_mapped_policy("reload-user", UserType::Reg, false, MappedPolicy::new("readwrite"), None)
.await
.expect("seed user policy");
store
.save_mapped_policy("reload-group", UserType::Reg, true, MappedPolicy::new("readonly"), None)
.await
.expect("seed group policy");
let cache = Cache::default();
store.load_all(&cache).await.expect("load nonempty IAM partitions");
{
let snapshot = cache.snapshot();
assert!(snapshot.groups.contains_key("reload-group"));
assert_eq!(snapshot.user_policies["reload-user"].policies, "readwrite");
assert_eq!(snapshot.group_policies["reload-group"].policies, "readonly");
assert!(snapshot.user_group_memberships["reload-user"].contains("reload-group"));
}
// Mutate durable state without notifying this cache, as on
// a peer that missed the last mapping/group deletion.
store
.delete_mapped_policy("reload-user", UserType::Reg, false)
.await
.expect("remove last user mapping");
store
.delete_mapped_policy("reload-group", UserType::Reg, true)
.await
.expect("remove last group mapping");
store.delete_group_info("reload-group").await.expect("remove last group");
assert!(cache.snapshot().groups.contains_key("reload-group"));
// Simulate a missed peer revocation while STS publication
// and a separate user update overlap this full disk reload.
let stale = rustfs_policy::auth::UserIdentity {
version: 1,
..Default::default()
};
cache.add_or_update_user("revoked-user", &stale, time::OffsetDateTime::now_utc());
let barrier = std::sync::Arc::new((tokio::sync::Notify::new(), tokio::sync::Notify::new()));
let loading = super::LOAD_ALL_COMMIT_BARRIER.scope(std::sync::Arc::clone(&barrier), store.load_all(&cache));
let concurrent = async {
barrier.0.notified().await;
cache.add_or_update_sts_account("new-session", &stale, time::OffsetDateTime::now_utc());
cache.add_or_update_user("new-service", &stale, time::OffsetDateTime::now_utc());
barrier.1.notify_one();
};
let (loaded, ()) = tokio::join!(loading, concurrent);
loaded.expect("reconcile revoked identities despite STS traffic");
let snapshot = cache.snapshot();
assert!(!snapshot.users.contains_key("revoked-user"));
assert!(snapshot.users.contains_key("new-service"));
assert!(snapshot.sts_accounts.contains_key("new-session"));
assert!(snapshot.groups.is_empty());
assert!(snapshot.user_policies.is_empty());
assert!(snapshot.group_policies.is_empty());
assert!(snapshot.user_group_memberships.is_empty());
assert!(!snapshot.policy_docs.is_empty(), "built-in policies must remain available");
});
})
.expect("spawn IAM reload test thread")
.join()
.expect("IAM reload test should complete");
}
/// rustfs#4304 / startup contract rustfs#4056: the bulk snapshot load
/// (`load_all`) must never depend on the node-counted namespace lock
/// quorum, so its reads have to carry `no_lock = true` down to the
+1013 -18
View File
File diff suppressed because it is too large Load Diff
+11
View File
@@ -85,6 +85,17 @@ impl KeystoneAuthProvider {
Ok(self.keystone_token_to_credentials(&keystone_token))
}
/// Revalidate a long-lived subscription directly with Keystone, bypassing
/// the request cache so token revocation terminates an existing stream.
/// Returns the same authentication and expiry errors as a new token login.
pub async fn revalidate_subscription_token(&self, token: &str) -> Result<Credentials> {
let validated = self.client.validate_token(token).await?;
if validated.is_expired() {
return Err(KeystoneError::TokenExpired);
}
Ok(self.keystone_token_to_credentials(&validated))
}
/// Authenticate using EC2 credentials (S3 API with AWS SigV4)
pub async fn authenticate_with_ec2(&self, access_key: &str, signature: &str, string_to_sign: &str) -> Result<Credentials> {
// Check cache
+142 -6
View File
@@ -121,7 +121,7 @@ impl KeystoneClient {
.await
.map_err(|e| {
error!("Failed to send token validation request: {}", e);
KeystoneError::HttpError(e.to_string())
KeystoneError::HttpError(e.without_url().to_string())
})?;
let status = response.status();
@@ -138,7 +138,10 @@ impl KeystoneClient {
)));
}
let body: serde_json::Value = response.json().await.map_err(|e| KeystoneError::ParseError(e.to_string()))?;
let body: serde_json::Value = response
.json()
.await
.map_err(|e| KeystoneError::ParseError(e.without_url().to_string()))?;
self.parse_token_v3(&body)
}
@@ -260,13 +263,16 @@ impl KeystoneClient {
.header("X-Auth-Token", admin_token)
.send()
.await
.map_err(|e| KeystoneError::HttpError(e.to_string()))?;
.map_err(|e| KeystoneError::HttpError(e.without_url().to_string()))?;
if !response.status().is_success() {
return Ok(vec![]);
}
let body: serde_json::Value = response.json().await.map_err(|e| KeystoneError::ParseError(e.to_string()))?;
let body: serde_json::Value = response
.json()
.await
.map_err(|e| KeystoneError::ParseError(e.without_url().to_string()))?;
let credentials = body
.get("credentials")
@@ -365,7 +371,7 @@ impl KeystoneClient {
.json(&auth_payload)
.send()
.await
.map_err(|e| KeystoneError::HttpError(e.to_string()))?;
.map_err(|e| KeystoneError::HttpError(e.without_url().to_string()))?;
if !response.status().is_success() {
return Err(KeystoneError::AuthenticationFailed("Admin authentication failed".to_string()));
@@ -379,7 +385,10 @@ impl KeystoneClient {
.to_string();
// Parse expiration from response body
let body: serde_json::Value = response.json().await.map_err(|e| KeystoneError::ParseError(e.to_string()))?;
let body: serde_json::Value = response
.json()
.await
.map_err(|e| KeystoneError::ParseError(e.without_url().to_string()))?;
let expires_at = body
.get("token")
@@ -410,6 +419,133 @@ impl KeystoneClient {
mod tests {
use super::*;
#[tokio::test]
async fn subscription_token_refresh_bypasses_a_still_valid_login_cache() {
use std::io::{Read, Write};
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("fake Keystone");
let endpoint = format!("http://{}", listener.local_addr().unwrap());
let expires = (OffsetDateTime::now_utc() + time::Duration::hours(1))
.format(&time::format_description::well_known::Rfc3339)
.unwrap();
let body = json!({"token":{"user":{"id":"listener","name":"listener"},
"roles":[{"name":"Member"}],"expires_at":expires}})
.to_string();
let server = std::thread::spawn(move || {
for response in [
format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
),
"HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".to_string(),
] {
let (mut socket, _) = listener.accept().expect("token validation request");
socket.set_read_timeout(Some(std::time::Duration::from_secs(5))).unwrap();
let mut request = [0; 4096];
let length = socket.read(&mut request).expect("request headers");
assert!(String::from_utf8_lossy(&request[..length]).contains("subscription-token"));
socket.write_all(response.as_bytes()).expect("validation response");
}
});
let mut client = KeystoneClient::new(
endpoint,
KeystoneVersion::V3,
None,
None,
None,
"Default".into(),
true,
std::time::Duration::from_secs(5),
);
client.client = reqwest::Client::builder()
.no_proxy()
.timeout(std::time::Duration::from_secs(5))
.build()
.unwrap();
let provider = crate::KeystoneAuthProvider::new(client, 10, std::time::Duration::from_secs(60), true);
assert!(provider.authenticate_with_token("subscription-token").await.is_ok());
assert!(
provider.authenticate_with_token("subscription-token").await.is_ok(),
"ordinary login still uses the valid cached token"
);
assert!(matches!(
provider.revalidate_subscription_token("subscription-token").await,
Err(KeystoneError::InvalidToken)
));
server.join().expect("fake Keystone finished");
}
#[tokio::test]
async fn middleware_keystone_failures_do_not_disclose_upstream_locations() {
use http_body_util::{BodyExt, Full};
use std::io::{Read, Write};
use tower::{Layer, ServiceExt};
for (response, details) in [
(None, "Keystone token validation failed"),
(
Some("HTTP/1.1 200 OK\r\nContent-Length: 8\r\nConnection: close\r\n\r\nnot-json"),
"Keystone token validation failed",
),
(
Some("HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"),
"Invalid token",
),
] {
for header in ["X-Auth-Token", "X-Storage-Token"] {
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind fake Keystone");
let endpoint = format!("http://{}/private-deployment", listener.local_addr().unwrap());
let server = response.map(|response| {
std::thread::spawn(move || {
let (mut stream, _) = listener.accept().expect("Keystone request");
stream.set_read_timeout(Some(std::time::Duration::from_secs(5))).unwrap();
let mut request = [0; 4096];
assert!(stream.read(&mut request).expect("read token validation request") > 0);
stream.write_all(response.as_bytes()).expect("send fake response");
})
});
let mut client = KeystoneClient::new(
endpoint.clone(),
KeystoneVersion::V3,
None,
None,
None,
"Default".to_owned(),
true,
std::time::Duration::from_secs(5),
);
client.client = reqwest::Client::builder()
.no_proxy()
.timeout(std::time::Duration::from_secs(5))
.build()
.unwrap();
let provider = crate::KeystoneAuthProvider::new(client, 1, std::time::Duration::from_secs(1), false);
let inner = tower::service_fn(|_| async {
Ok::<_, std::convert::Infallible>(http::Response::new(Full::new(bytes::Bytes::new())))
});
let response = crate::middleware::KeystoneAuthLayer::new(Some(Arc::new(provider)))
.layer(inner)
.oneshot(
http::Request::builder()
.header(header, "private-token")
.body(Full::new(bytes::Bytes::new()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), http::StatusCode::UNAUTHORIZED);
let body = response.into_body().collect().await.unwrap().to_bytes();
let body = std::str::from_utf8(&body).unwrap();
assert!(body.contains(&format!("<Details>{details}</Details>")));
for secret in [&endpoint, "private-deployment", "private-token", "/v3/auth/tokens"] {
assert!(!body.contains(secret), "upstream diagnostics must not appear in the response");
}
if let Some(server) = server {
server.join().unwrap();
}
}
}
}
#[test]
fn test_client_creation() {
let client = KeystoneClient::new(
+5 -1
View File
@@ -173,7 +173,11 @@ where
<Message>Invalid Keystone token</Message>
<Details>{}</Details>
</Error>"#,
xml_escape(&e.to_string())
xml_escape(match e {
crate::KeystoneError::InvalidToken => "Invalid token",
crate::KeystoneError::TokenExpired => "Token expired",
_ => "Keystone token validation failed",
})
);
let body: BoxBody = Full::new(Bytes::from(error_xml))
+11 -1
View File
@@ -214,6 +214,9 @@ impl KmsManager {
}
async fn create_key_inner(&self, request: CreateKeyRequest) -> Result<CreateKeyResponse> {
if request.key_name.as_deref() == Some(crate::probe::PROBE_KEY_ID) {
return Err(KmsError::invalid_operation("the KMS probe key is reserved for internal health checks"));
}
// A blank name is neither "generate one" (that is `None`) nor a usable
// id: the Local backend would write a key file with an empty stem and
// the Vault backends would address their mount root, each failing with
@@ -438,6 +441,9 @@ impl KmsManager {
/// any backend sees the request. The backends repeat the window bound as a
/// defensive assertion for callers that hold a backend handle directly.
async fn delete_key_inner(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse> {
if request.key_id == crate::probe::PROBE_KEY_ID {
return Err(KmsError::invalid_operation("the KMS probe key cannot be deleted through key management"));
}
self.check_deletion_request(&request)?;
if request.force_immediate.unwrap_or(false) {
self.refuse_referenced_immediate_deletion(&request.key_id).await?;
@@ -591,7 +597,11 @@ impl KmsManager {
/// Disable a key on behalf of `context`'s principal
pub async fn disable_key_with_context(&self, key_id: &str, context: &OperationContext) -> Result<()> {
let started = Instant::now();
let result = self.backend.disable_key(key_id).await;
let result = if key_id == crate::probe::PROBE_KEY_ID {
Err(KmsError::invalid_operation("the KMS probe key cannot be disabled through key management"))
} else {
self.backend.disable_key(key_id).await
};
if result.is_ok() {
self.invalidate_cached_metadata(key_id).await;
}
+51
View File
@@ -700,6 +700,57 @@ mod tests {
);
}
#[tokio::test]
async fn key_management_cannot_preempt_or_disable_the_readiness_probe() {
let temp_dir = tempfile::tempdir().expect("key directory");
let backend = local_backend(&temp_dir).await;
let mut config = KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults();
config.allow_immediate_deletion = true;
let manager = crate::manager::KmsManager::new(backend.clone(), config);
let request = CreateKeyRequest {
key_name: Some(PROBE_KEY_ID.to_owned()),
..Default::default()
};
assert!(matches!(manager.create_key(request).await, Err(KmsError::InvalidOperation { .. })));
let worker = worker(backend.clone());
assert_eq!(worker.round().await, ProbeResult::Success, "the probe still provisions its reserved key");
assert!(matches!(manager.disable_key(PROBE_KEY_ID).await, Err(KmsError::InvalidOperation { .. })));
for immediate in [false, true] {
assert!(matches!(
manager
.delete_key(DeleteKeyRequest {
key_id: PROBE_KEY_ID.to_owned(),
pending_window_in_days: Some(7),
force_immediate: Some(immediate),
confirm_key_id: Some(PROBE_KEY_ID.to_owned())
})
.await,
Err(KmsError::InvalidOperation { .. })
));
assert_eq!(worker.round().await, ProbeResult::Success);
}
let customer = manager
.create_key(CreateKeyRequest {
key_name: Some("customer-probe-control".to_owned()),
..Default::default()
})
.await
.expect("customer key");
manager
.disable_key(&customer.key_id)
.await
.expect("customer key can be disabled");
manager
.delete_key(DeleteKeyRequest {
key_id: customer.key_id,
pending_window_in_days: Some(7),
..Default::default()
})
.await
.expect("customer deletion remains available");
assert_eq!(worker.round().await, ProbeResult::Success);
}
#[test]
fn read_only_backend_is_reported_unsupported_without_failures() {
let (snapshot, result) = record_metrics(|| {
+1
View File
@@ -45,6 +45,7 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-metrics/hotpath-cpu"
hotpath.workspace = true
rustfs-io-metrics = { workspace = true }
rustfs-utils = { workspace = true }
rustfs-common = { workspace = true }
async-trait.workspace = true
futures.workspace = true
serde = { workspace = true, features = ["derive"] }
+255
View File
@@ -694,6 +694,132 @@ impl DistributedLock {
self.acquire_guard(&req).await
}
/// Acquire exclusive locks for many independent resources in one batched
/// round: one [`LockClient::acquire_locks_batch`] RPC per client instead of
/// one quorum round-trip per key. Returns one guard or error per resource.
///
/// Bulk operations (S3 `DeleteObjects`) are the only multi-key write-lock
/// holders, and their keys are independent. Per-key outcomes are
/// deadlock-free without any serial fallback: the wire call is a single
/// round per client and every lock server orders concurrent batches by the
/// same global (shard, key) order, so no acquirer can hold one key while
/// waiting for another. A key that fails to reach quorum releases its
/// partial grants (awaited) before its error is returned.
///
/// Guards keep the same TTL heartbeat and lock-lost semantics as
/// [`Self::acquire_guard`]. Contention is waited out server-side up to
/// `timeout` (each request carries it), matching the single-key path.
pub async fn acquire_guard_batch(
&self,
resources: &[ObjectKey],
owner: &str,
timeout: Duration,
ttl: Duration,
) -> Vec<Result<DistributedLockGuard>> {
if resources.is_empty() {
return Vec::new();
}
if self.clients.is_empty() {
return resources
.iter()
.map(|_| Err(LockError::internal("No lock clients available")))
.collect();
}
let required_quorum = self.required_quorum(LockType::Exclusive);
let attempt_started = Instant::now();
let requests: Vec<LockRequest> = resources
.iter()
.map(|resource| {
LockRequest::new(resource.clone(), LockType::Exclusive, owner)
.with_acquire_timeout(timeout)
.with_ttl(ttl)
})
.collect();
// One batched round per client; a client-level failure is a per-key
// failure for every key that still needed that client's vote.
let rounds = join_all(self.clients.iter().map(|client| {
let requests = requests.clone();
async move { client.acquire_locks_batch(&requests).await }
}))
.await;
let mut results: Vec<Result<DistributedLockGuard>> = Vec::with_capacity(resources.len());
let mut orphaned_entries: Vec<(LockId, Arc<dyn LockClient>)> = Vec::new();
for (key_idx, resource) in resources.iter().enumerate() {
let mut entries: Vec<HeldLockEntry> = Vec::new();
let mut failure: Option<String> = None;
for (client_idx, client) in self.clients.iter().enumerate() {
match &rounds[client_idx] {
Ok(responses) => match responses.get(key_idx) {
Some(response) if response.success => {
let lock_id = response
.lock_info
.as_ref()
.map(|info| info.id.clone())
.unwrap_or_else(|| LockId::new_unique(resource));
entries.push(HeldLockEntry {
lock_id,
client: client.clone(),
valid_until: attempt_started.checked_add(ttl).unwrap_or(attempt_started),
});
}
Some(response) => {
failure.get_or_insert_with(|| {
response
.error
.clone()
.unwrap_or_else(|| "distributed lock acquisition failed".to_string())
});
}
None => {
failure.get_or_insert_with(|| "client returned incomplete batch lock response".to_string());
}
},
Err(err) => {
failure.get_or_insert_with(|| err.to_string());
}
}
}
if entries.len() >= required_quorum {
let heartbeat = LockHeartbeatConfig {
ttl,
interval: derive_refresh_interval(entries.len(), ttl, None),
refresh_quorum: required_quorum,
owner: owner.to_string(),
resource: resource.clone(),
};
results.push(Ok(DistributedLockGuard::new(
LockId::new_unique(resource),
entries,
LockType::Exclusive,
heartbeat,
)));
} else {
let achieved = entries.len();
orphaned_entries.extend(entries.iter().map(HeldLockEntry::release_entry));
if let Some(error) = failure {
tracing::debug!(resource = %resource, owner, required_quorum, achieved, "batch lock acquire failed: {error}");
}
results.push(Err(LockError::QuorumNotReached {
required: required_quorum,
achieved,
}));
}
}
// One bounded synchronous pass so the caller cannot race these
// unlocks on an immediate retry; anything a slow peer still holds
// keeps retrying in the background instead of parking the request
// on the deferred unlock schedule.
if !orphaned_entries.is_empty() {
Self::release_entries_bounded(orphaned_entries, "distributed_lock_batch_quorum_rollback").await;
}
results
}
fn spawn_lock_requests_until(
&self,
pending: &mut JoinSet<LockAcquireTaskResult>,
@@ -755,6 +881,19 @@ impl DistributedLock {
Self::release_entries_deferred(pending, context).await;
}
/// Release entries with one synchronous pass, then continue the retry
/// schedule in the background.
///
/// The batch-acquire rollback path runs on the request: it must not park
/// the caller on the deferred unlock schedule when a peer is unreachable,
/// while still confirming the healthy quorum is free before returning.
async fn release_entries_bounded(entries: Vec<(LockId, Arc<dyn LockClient>)>, context: &'static str) {
let pending = Self::release_pending_once(entries, 1, context).await;
if !pending.is_empty() {
Self::spawn_release_cleanup(pending, context);
}
}
async fn release_pending_once(
pending: Vec<(LockId, Arc<dyn LockClient>)>,
attempt: usize,
@@ -2856,4 +2995,120 @@ mod tests {
drop(guard);
}
/// Per-object scripted client: grants only the named objects and records
/// every released lock's resource.
#[derive(Debug)]
struct ScriptedBatchClient {
grants: Vec<String>,
releases: Arc<Mutex<Vec<String>>>,
}
impl ScriptedBatchClient {
fn build(grants: &[&str]) -> (Arc<dyn LockClient>, Arc<Mutex<Vec<String>>>) {
let releases = Arc::new(Mutex::new(Vec::new()));
let client = Arc::new(Self {
grants: grants.iter().map(|object| (*object).to_string()).collect(),
releases: Arc::clone(&releases),
}) as Arc<dyn LockClient>;
(client, releases)
}
}
#[async_trait::async_trait]
impl LockClient for ScriptedBatchClient {
async fn acquire_lock(&self, request: &LockRequest) -> crate::Result<LockResponse> {
if self.grants.iter().any(|object| object == request.resource.object.as_ref()) {
let info = LockInfo {
id: request.lock_id.clone(),
resource: request.resource.clone(),
lock_type: request.lock_type,
status: crate::types::LockStatus::Acquired,
owner: request.owner.clone(),
acquired_at: std::time::SystemTime::now(),
expires_at: std::time::SystemTime::now() + request.ttl,
last_refreshed: std::time::SystemTime::now(),
metadata: request.metadata.clone(),
priority: request.priority,
wait_start_time: None,
};
Ok(LockResponse::success(info, Duration::ZERO))
} else {
Ok(LockResponse::failure("denied by test script", Duration::ZERO))
}
}
async fn release(&self, lock_id: &LockId) -> crate::Result<bool> {
self.releases
.lock()
.expect("release log")
.push(lock_id.resource.object.to_string());
Ok(true)
}
async fn refresh(&self, _lock_id: &LockId) -> crate::Result<bool> {
Ok(true)
}
async fn force_release(&self, lock_id: &LockId) -> crate::Result<bool> {
self.release(lock_id).await
}
async fn check_status(&self, _lock_id: &LockId) -> crate::Result<Option<LockInfo>> {
Ok(None)
}
async fn get_stats(&self) -> crate::Result<LockStats> {
Ok(LockStats::default())
}
async fn close(&self) -> crate::Result<()> {
Ok(())
}
async fn is_online(&self) -> bool {
true
}
async fn is_local(&self) -> bool {
false
}
}
/// A batch key granted by fewer than quorum clients fails alone, and its
/// partial grants are released synchronously before the call returns
/// (PR9155 review coverage for the distributed rollback path).
#[tokio::test]
async fn acquire_guard_batch_releases_partial_grants_below_quorum() {
let (c0, c0_releases) = ScriptedBatchClient::build(&["kept", "lost"]);
let (c1, _c1_releases) = ScriptedBatchClient::build(&["kept"]);
let (c2, _c2_releases) = ScriptedBatchClient::build(&[]);
let lock = DistributedLock::new("batch-test".to_string(), vec![c0, c1, c2], 2);
let resources = vec![ObjectKey::new("bucket", "kept"), ObjectKey::new("bucket", "lost")];
let results = lock
.acquire_guard_batch(&resources, "owner", Duration::from_millis(200), Duration::from_secs(30))
.await;
assert!(results[0].is_ok(), "kept reached quorum: {:?}", results[0].as_ref().err());
let error = results[1].as_ref().expect_err("lost must fail below quorum");
assert!(
matches!(
error,
LockError::QuorumNotReached {
required: 2,
achieved: 1
}
),
"lost must report its partial vote: {error:?}"
);
// Exactly the orphaned grant is released before the call returns,
// while "kept" stays held by its live guard.
assert_eq!(
c0_releases.lock().expect("release log").clone(),
vec!["lost".to_string()],
"only the below-quorum key's partial grant is released synchronously"
);
drop(results);
}
}
+3
View File
@@ -31,10 +31,13 @@ pub mod manager_trait;
pub mod metrics;
pub mod object_pool;
pub mod optimized_notify;
mod publication;
pub mod shard;
pub mod state;
pub mod types;
#[cfg(test)]
mod publication_tests;
#[cfg(test)]
mod tests;
+2
View File
@@ -95,6 +95,8 @@ impl ObjectLockState {
// Reset atomic state
self.atomic_state = crate::fast_lock::state::AtomicLockState::new();
self.publication = std::sync::OnceLock::new();
// Clear owners
*self.current_owner.write() = None;
self.shared_owners.write().clear();
+97
View File
@@ -0,0 +1,97 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! FIFO admission only for the bucket publication fence. Consecutive readers
//! form a group; a later writer cannot overtake them, nor can later readers
//! overtake a queued writer. This does not limit a current holder's I/O time.
use parking_lot::Mutex;
use std::{collections::VecDeque, sync::Arc};
use tokio::sync::Notify;
use super::{
state::ObjectLockState,
types::{LockMode, ObjectKey, ObjectLockRequest},
};
pub(super) fn applies(key: &ObjectKey) -> bool {
key.version.is_none() && key.object.as_ref() == rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH
}
#[derive(Debug, Default)]
pub(super) struct PublicationQueue {
entries: Mutex<VecDeque<(Arc<()>, LockMode)>>,
pub(super) changed: Notify,
}
pub(super) struct PublicationTicket {
pub(super) state: Arc<ObjectLockState>,
pub(super) queue: Arc<PublicationQueue>,
token: Arc<()>,
mode: LockMode,
}
impl PublicationTicket {
// Called while the shard map is locked. The waiter counter pins the map
// entry until ticket destruction, including cancellation between polls.
pub(super) fn new(state: Arc<ObjectLockState>, mode: LockMode) -> Option<Self> {
let counted = match mode {
LockMode::Shared => state.atomic_state.inc_readers_waiting(),
LockMode::Exclusive => state.atomic_state.inc_writers_waiting(),
};
if !counted {
return None;
}
let queue = state
.publication
.get_or_init(|| Arc::new(PublicationQueue::default()))
.clone();
let token = Arc::new(());
queue.entries.lock().push_back((token.clone(), mode));
Some(Self {
state,
queue,
token,
mode,
})
}
pub(super) fn try_acquire(&self, request: &ObjectLockRequest) -> bool {
// Lock order: shard map -> queue -> owner metadata. Release never
// locks the queue; ticket destruction only locks the queue.
let entries = self.queue.entries.lock();
let mut found = false;
for (token, mode) in entries.iter() {
if Arc::ptr_eq(token, &self.token) {
found = true;
break;
}
if self.mode == LockMode::Exclusive || *mode == LockMode::Exclusive {
return false;
}
}
if !found {
return false;
}
match self.mode {
LockMode::Shared => self.state.try_acquire_admitted_shared(&request.owner, request.lock_timeout),
LockMode::Exclusive => self.state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout),
}
}
}
impl Drop for PublicationTicket {
fn drop(&mut self) {
{
let mut entries = self.queue.entries.lock();
if let Some(index) = entries.iter().position(|(token, _)| Arc::ptr_eq(token, &self.token)) {
entries.remove(index);
}
}
match self.mode {
LockMode::Shared => self.state.atomic_state.dec_readers_waiting(),
LockMode::Exclusive => self.state.atomic_state.dec_writers_waiting(),
}
self.queue.changed.notify_waiters();
}
}
@@ -0,0 +1,327 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
use super::*;
use std::sync::Arc;
use tokio::{task::JoinHandle, time::timeout};
fn manager() -> Arc<FastObjectLockManager> {
Arc::new(FastObjectLockManager::with_config(LockConfig {
shard_count: 1,
max_idle_time: Duration::ZERO,
..Default::default()
}))
}
fn key() -> ObjectKey {
ObjectKey::new("publication-bucket", rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
}
fn queued(manager: &Arc<FastObjectLockManager>, owner: &str, mode: LockMode) -> JoinHandle<FastLockGuard> {
let manager = manager.clone();
let owner = owner.to_owned();
tokio::spawn(async move {
let mut request = ObjectLockRequest::new_read(key(), owner);
request.mode = mode;
manager
.acquire_lock(request.with_acquire_timeout(Duration::from_secs(5)))
.await
.expect("queued publication lock")
})
}
async fn waiters(manager: &FastObjectLockManager, count: u64) {
timeout(Duration::from_secs(3), async {
while manager.current_lock_counts().1 != count {
tokio::task::yield_now().await;
}
})
.await
.expect("waiter admission must be observable");
}
async fn acquired(task: JoinHandle<FastLockGuard>) -> FastLockGuard {
timeout(Duration::from_secs(3), task)
.await
.expect("queue must progress")
.expect("acquire task")
}
#[tokio::test]
async fn publication_queue_serves_read_groups_and_writers_in_arrival_order() {
let manager = manager();
let holder = manager.acquire_write_lock(key(), "holder").await.expect("initial holder");
let w1 = queued(&manager, "w1", LockMode::Exclusive);
waiters(&manager, 1).await;
let r1 = queued(&manager, "r1", LockMode::Shared);
waiters(&manager, 2).await;
let r2 = queued(&manager, "r2", LockMode::Shared);
waiters(&manager, 3).await;
let w2 = queued(&manager, "w2", LockMode::Exclusive);
waiters(&manager, 4).await;
let r3 = queued(&manager, "r3", LockMode::Shared);
waiters(&manager, 5).await;
assert_eq!(manager.cleanup_expired_traditional().await, 0, "queued state must stay mapped");
drop(holder);
assert_eq!(
manager.cleanup_expired_traditional().await,
0,
"an unlocked state with queued tickets must stay mapped"
);
let w1 = acquired(w1).await;
assert!(!r1.is_finished() && !r2.is_finished() && !w2.is_finished() && !r3.is_finished());
drop(w1);
let r1 = acquired(r1).await;
let r2 = acquired(r2).await;
assert!(!w2.is_finished() && !r3.is_finished(), "later requests cannot pass the reader group");
drop(r1);
assert!(!w2.is_finished(), "the second reader still holds the fence");
drop(r2);
let w2 = acquired(w2).await;
assert!(!r3.is_finished(), "a later reader cannot starve the queued writer");
drop(w2);
drop(acquired(r3).await);
assert_eq!(manager.current_lock_counts(), (0, 0));
}
#[tokio::test]
async fn publication_queue_cancellation_and_timeout_remove_their_turns() {
let manager = manager();
let holder = manager.acquire_write_lock(key(), "holder").await.expect("initial holder");
let cancelled = queued(&manager, "cancelled", LockMode::Exclusive);
waiters(&manager, 1).await;
let expiring = tokio::spawn({
let manager = manager.clone();
async move {
manager
.acquire_lock(ObjectLockRequest::new_write(key(), "expires").with_acquire_timeout(Duration::from_millis(500)))
.await
}
});
waiters(&manager, 2).await;
let reader = queued(&manager, "survivor", LockMode::Shared);
waiters(&manager, 3).await;
cancelled.abort();
assert!(cancelled.await.expect_err("cancel waiter").is_cancelled());
assert!(matches!(expiring.await.expect("timeout task"), Err(LockResult::Timeout)));
waiters(&manager, 1).await;
assert_eq!(manager.cleanup_expired_traditional().await, 0);
drop(holder);
drop(acquired(reader).await);
assert_eq!(manager.current_lock_counts(), (0, 0));
timeout(Duration::from_secs(3), async {
while manager.cleanup_expired_traditional().await == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("an empty queue must allow map cleanup");
// An empty queue neither keeps a stale turn nor prevents subsequent writers.
drop(manager.acquire_write_lock(key(), "next").await.expect("queue recovered"));
}
#[tokio::test]
async fn publication_queue_batch_fast_path_cannot_bypass_a_reader() {
for all_or_nothing in [false, true] {
let manager = manager();
let holder = manager.acquire_write_lock(key(), "holder").await.expect("initial holder");
let reader = queued(&manager, "queued-reader", LockMode::Shared);
waiters(&manager, 1).await;
drop(holder);
// This future can try the batch fast path before the reader runs again.
let batch = manager.acquire_locks_batch(
BatchLockRequest::new("batch")
.add_write_lock(key())
.with_all_or_nothing(all_or_nothing),
);
tokio::pin!(batch);
let reader = tokio::select! {
biased;
_ = &mut batch => panic!("batch overtook an already queued reader"),
reader = acquired(reader) => reader,
};
drop(reader);
let batch = timeout(Duration::from_secs(3), batch)
.await
.expect("batch progresses after reader");
assert!(batch.all_acquired);
drop(batch);
assert_eq!(manager.current_lock_counts(), (0, 0));
}
}
#[test]
fn publication_queue_is_scoped_to_the_unversioned_bucket_fence() {
assert!(publication::applies(&key()));
assert!(!publication::applies(&ObjectKey::new("publication-bucket", "plain-object")));
assert!(!publication::applies(&ObjectKey::with_version(
"publication-bucket",
rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH,
"version"
)));
}
async fn cluster_waiters(managers: &[Arc<FastObjectLockManager>], minimum: u64) {
timeout(Duration::from_secs(3), async {
while !managers.iter().all(|manager| manager.current_lock_counts().1 >= minimum) {
tokio::task::yield_now().await;
}
})
.await
.expect("all real local peers registered the waiter");
}
fn cluster_queued(lock: &Arc<crate::NamespaceLock>, owner: &str, kind: crate::LockType) -> JoinHandle<crate::NamespaceLockGuard> {
let lock = lock.clone();
let owner = owner.to_owned();
tokio::spawn(async move {
lock.acquire_guard(
&crate::LockRequest::new(key(), kind, owner)
.with_acquire_timeout(Duration::from_secs(5))
.with_ttl(Duration::from_secs(30)),
)
.await
.expect("distributed acquire")
.expect("distributed quorum")
})
}
async fn cluster_acquired(task: JoinHandle<crate::NamespaceLockGuard>) -> crate::NamespaceLockGuard {
timeout(Duration::from_secs(3), task)
.await
.expect("quorum must progress")
.expect("cluster task")
}
#[tokio::test]
async fn publication_queue_distributed_quorum_preserves_bidirectional_progress() {
let managers = (0..3).map(|_| manager()).collect::<Vec<_>>();
let clients = managers
.iter()
.map(|manager| {
Arc::new(crate::LocalClient::with_manager(Arc::new(crate::GlobalLockManager::Enabled(
manager.clone(),
)))) as Arc<dyn crate::LockClient>
})
.collect::<Vec<_>>();
let lock = Arc::new(crate::NamespaceLock::with_clients("publication-fairness".to_owned(), clients));
// Hold every voter while establishing a deterministic arrival order.
// A distributed holder may return at quorum and release its spare vote.
let mut holders = Vec::new();
for manager in &managers {
holders.push(manager.acquire_write_lock(key(), "holder").await.expect("hold voter"));
}
let w1 = cluster_queued(&lock, "w1", crate::LockType::Exclusive);
cluster_waiters(&managers, 1).await;
let r1 = cluster_queued(&lock, "r1", crate::LockType::Shared);
cluster_waiters(&managers, 2).await;
let w2 = cluster_queued(&lock, "w2", crate::LockType::Exclusive);
cluster_waiters(&managers, 3).await;
let r2 = cluster_queued(&lock, "r2", crate::LockType::Shared);
cluster_waiters(&managers, 4).await;
drop(holders);
let w1 = cluster_acquired(w1).await;
assert!(!r1.is_finished() && !w2.is_finished() && !r2.is_finished());
drop(w1);
let r1 = cluster_acquired(r1).await;
assert!(!w2.is_finished() && !r2.is_finished());
drop(r1);
let w2 = cluster_acquired(w2).await;
assert!(!r2.is_finished());
drop(w2);
drop(cluster_acquired(r2).await);
timeout(Duration::from_secs(3), async {
while managers.iter().any(|manager| manager.current_lock_counts() != (0, 0)) {
tokio::task::yield_now().await;
}
})
.await
.expect("all quorum guards and pending cleanup must release");
}
#[tokio::test]
async fn publication_queue_distributed_cancel_and_timeout_leave_no_orphan_turns() {
let managers = (0..3).map(|_| manager()).collect::<Vec<_>>();
let clients = managers
.iter()
.map(|manager| {
Arc::new(crate::LocalClient::with_manager(Arc::new(crate::GlobalLockManager::Enabled(
manager.clone(),
)))) as Arc<dyn crate::LockClient>
})
.collect::<Vec<_>>();
let lock = Arc::new(crate::NamespaceLock::with_clients("publication-cancellation".to_owned(), clients));
let mut holders = Vec::new();
for manager in &managers {
holders.push(manager.acquire_write_lock(key(), "holder").await.expect("hold voter"));
}
let cancelled = cluster_queued(&lock, "cancelled", crate::LockType::Exclusive);
cluster_waiters(&managers, 1).await;
cancelled.abort();
assert!(cancelled.await.expect_err("cancel distributed waiter").is_cancelled());
timeout(Duration::from_secs(3), async {
while managers.iter().any(|manager| manager.current_lock_counts().1 != 0) {
tokio::task::yield_now().await;
}
})
.await
.expect("aborted peer attempts must remove their tickets");
let timed_out = lock
.acquire_guard(
&crate::LockRequest::new(key(), crate::LockType::Exclusive, "timeout")
.with_acquire_timeout(Duration::from_millis(100)),
)
.await
.expect("contention is a failed acquire");
assert!(timed_out.is_none());
timeout(Duration::from_secs(3), async {
while managers.iter().any(|manager| manager.current_lock_counts().1 != 0) {
tokio::task::yield_now().await;
}
})
.await
.expect("timed out peer attempts must remove their tickets");
let reader = cluster_queued(&lock, "next-reader", crate::LockType::Shared);
cluster_waiters(&managers, 1).await;
let writer = cluster_queued(&lock, "next-writer", crate::LockType::Exclusive);
cluster_waiters(&managers, 2).await;
drop(holders);
let reader = cluster_acquired(reader).await;
assert!(!writer.is_finished());
drop(reader);
drop(cluster_acquired(writer).await);
timeout(Duration::from_secs(3), async {
while managers.iter().any(|manager| manager.current_lock_counts() != (0, 0)) {
tokio::task::yield_now().await;
}
})
.await
.expect("distributed holders and queues must drain");
}
#[tokio::test]
async fn publication_queue_does_not_change_ordinary_key_writer_preference() {
let manager = manager();
let ordinary = ObjectKey::new("publication-bucket", "ordinary-key");
let holder = manager
.acquire_write_lock(ordinary.clone(), "holder")
.await
.expect("ordinary holder");
let reader = tokio::spawn({
let manager = manager.clone();
let ordinary = ordinary.clone();
async move { manager.acquire_read_lock(ordinary, "reader").await.expect("ordinary reader") }
});
waiters(&manager, 1).await;
let writer = tokio::spawn({
let manager = manager.clone();
async move { manager.acquire_write_lock(ordinary, "writer").await.expect("ordinary writer") }
});
waiters(&manager, 2).await;
drop(holder);
let writer = acquired(writer).await;
assert!(!reader.is_finished(), "ordinary keys retain writer preference");
drop(writer);
drop(acquired(reader).await);
assert_eq!(manager.current_lock_counts(), (0, 0));
}
+55 -1
View File
@@ -21,6 +21,7 @@ use tokio::time::timeout;
use crate::fast_lock::{
metrics::ShardMetrics,
object_pool::ObjectStatePool,
publication::{self, PublicationTicket},
state::ObjectLockState,
types::{LockMode, LockResult, ObjectKey, ObjectLockRequest},
};
@@ -134,6 +135,9 @@ impl LockShard {
/// Acquire lock with fast path optimization
pub async fn acquire_lock(&self, request: &ObjectLockRequest) -> Result<(), LockResult> {
let start_time = Instant::now();
if publication::applies(&request.key) {
return self.acquire_publication(request, start_time).await;
}
// Try fast path first
if let Some(_state) = self.try_fast_path(request) {
@@ -145,15 +149,65 @@ impl LockShard {
self.acquire_lock_slow_path(request, start_time).await
}
fn publication_ticket(&self, request: &ObjectLockRequest) -> Option<PublicationTicket> {
let mut objects = self.objects.write();
let state = objects
.entry(request.key.clone())
.or_insert_with(|| Arc::new(*self.object_pool.acquire()))
.clone();
PublicationTicket::new(state, request.mode)
}
async fn acquire_publication(&self, request: &ObjectLockRequest, started: Instant) -> Result<(), LockResult> {
let Some(ticket) = self.publication_ticket(request) else {
return Err(LockResult::Timeout);
};
let mut first_attempt = true;
loop {
// Register before checking so a release/cancellation cannot be lost
// between the eligibility check and the notification wait.
let notified = ticket.queue.changed.notified();
let remaining = request.acquire_timeout.saturating_sub(started.elapsed());
if !first_attempt && remaining.is_zero() {
self.metrics.record_timeout();
return Err(LockResult::Timeout);
}
if ticket.try_acquire(request) {
self.metrics.record_slow_path_success();
return Ok(());
}
first_attempt = false;
if remaining.is_zero() || timeout(remaining, notified).await.is_err() {
self.metrics.record_timeout();
return Err(LockResult::Timeout);
}
}
}
/// Try fast path only (without fallback to slow path)
pub fn try_fast_path_only(&self, request: &ObjectLockRequest) -> bool {
if publication::applies(&request.key) {
// Batch acquisition must join the same queue, even for a no-wait
// attempt. Dropping a failed ticket does not consume a queue turn.
return self
.publication_ticket(request)
.is_some_and(|ticket| ticket.try_acquire(request));
}
// Early check to avoid unnecessary lock contention
if let Some(state) = self.objects.read().get(&request.key)
&& !state.atomic_state.is_fast_path_available(request.mode)
{
return false;
}
self.try_fast_path(request).is_some()
// Count the acquisition like `acquire_lock` does: batch callers use
// this path for every uncontended key, and the metric promises the
// total of successful acquisitions.
if self.try_fast_path(request).is_some() {
self.metrics.record_fast_path_success();
true
} else {
false
}
}
/// Try fast path lock acquisition (lock-free when possible)
+32 -4
View File
@@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, SystemTime};
use tokio::sync::Notify;
@@ -78,13 +78,21 @@ impl AtomicLockState {
/// Try to acquire shared lock (fast path)
pub fn try_acquire_shared(&self) -> bool {
self.try_acquire_shared_with_mask(NO_WRITER_AND_NO_WAITING_WRITERS)
}
// Publication admission already orders readers against waiting writers.
fn try_acquire_admitted_shared(&self) -> bool {
self.try_acquire_shared_with_mask(WRITER_FLAG_MASK)
}
fn try_acquire_shared_with_mask(&self, blocked_mask: u64) -> bool {
self.update_access_time();
loop {
let current = self.state.load(Ordering::Acquire);
// Fast path check - cannot acquire if there's a writer or writers waiting
if (current & NO_WRITER_AND_NO_WAITING_WRITERS) != 0 {
if (current & blocked_mask) != 0 {
return false;
}
@@ -334,6 +342,7 @@ pub struct ObjectLockState {
// First cache line: Most frequently accessed data
/// Atomic state for fast operations
pub atomic_state: AtomicLockState,
pub(super) publication: OnceLock<Arc<super::publication::PublicationQueue>>,
// Second cache line: Notification mechanisms
/// Notification for readers (traditional)
@@ -377,6 +386,7 @@ impl ObjectLockState {
pub fn new() -> Self {
Self {
atomic_state: AtomicLockState::new(),
publication: OnceLock::new(),
read_notify: Notify::new(),
write_notify: Notify::new(),
optimized_notify: OptimizedNotify::new(),
@@ -392,6 +402,19 @@ impl ObjectLockState {
return false;
}
self.record_shared_owner(owner, lock_timeout);
true
}
pub(super) fn try_acquire_admitted_shared(&self, owner: &Arc<str>, lock_timeout: Duration) -> bool {
if !self.atomic_state.try_acquire_admitted_shared() {
return false;
}
self.record_shared_owner(owner, lock_timeout);
true
}
fn record_shared_owner(&self, owner: &Arc<str>, lock_timeout: Duration) {
self.atomic_state.update_access_time();
let mut shared = self.shared_owners.write();
if let Some(entry) = shared.iter_mut().find(|entry| entry.owner.as_ref() == owner.as_ref()) {
@@ -406,7 +429,6 @@ impl ObjectLockState {
lock_timeout,
});
}
true
}
/// Try fast path exclusive lock acquisition
@@ -437,6 +459,9 @@ impl ObjectLockState {
Some(shared.remove(pos))
};
if self.atomic_state.release_shared() {
if let Some(queue) = self.publication.get() {
queue.changed.notify_waiters();
}
if shared.is_empty() {
drop(shared);
self.optimized_notify.notify_writer();
@@ -480,6 +505,9 @@ impl ObjectLockState {
if self.atomic_state.release_exclusive() {
*current = None;
drop(current);
if let Some(queue) = self.publication.get() {
queue.changed.notify_waiters();
}
// Notify waiters using optimized system - prefer writers over readers
if self
.atomic_state
+58 -1
View File
@@ -15,7 +15,7 @@
use crate::{
GlobalLockManager, ObjectKey,
error::Result,
fast_lock::{FastLockGuard, LockManager, LockMode, ObjectLockRequest},
fast_lock::{FastLockGuard, LockManager, LockMode, LockResult, ObjectLockRequest},
types::{LockPriority, LockRequest, LockType},
};
use std::sync::Arc;
@@ -112,4 +112,61 @@ impl LocalLock {
.with_ttl(ttl);
self.acquire_guard(&req).await
}
/// Acquire exclusive locks for many independent resources in one batched
/// pass; see [`crate::distributed_lock::DistributedLock::acquire_guard_batch`].
/// The manager's batch acquire sorts every request into one global
/// (shard, key) order, so concurrent batches cannot deadlock. Returns one
/// guard or error per resource, in `resources` order.
pub async fn acquire_guard_batch(
&self,
resources: &[ObjectKey],
owner: &str,
timeout: Duration,
ttl: Duration,
) -> Vec<Result<FastLockGuard>> {
if resources.is_empty() {
return Vec::new();
}
let batch = crate::fast_lock::BatchLockRequest {
requests: resources
.iter()
.map(|resource| {
ObjectLockRequest::new_write(resource.clone(), owner)
.with_acquire_timeout(timeout)
.with_lock_timeout(ttl)
})
.collect(),
owner: owner.into(),
all_or_nothing: false,
};
let outcome = self.manager.acquire_locks_batch(batch).await;
let mut guards = outcome
.successful_locks
.into_iter()
.zip(outcome.guards)
.collect::<std::collections::HashMap<ObjectKey, FastLockGuard>>();
let mut failures = outcome
.failed_locks
.into_iter()
.collect::<std::collections::HashMap<ObjectKey, LockResult>>();
resources
.iter()
.map(|resource| {
if let Some(guard) = guards.remove(resource) {
return Ok(guard);
}
match failures.remove(resource) {
Some(LockResult::Timeout) => Err(crate::error::LockError::timeout(resource.to_string(), timeout)),
Some(LockResult::Conflict { current_owner, .. }) => Err(crate::error::LockError::AlreadyLocked {
resource: resource.to_string(),
owner: current_owner.to_string(),
}),
Some(LockResult::Acquired) | None => {
Err(crate::error::LockError::internal("batch lock acquire returned no outcome for key"))
}
}
})
.collect()
}
}
+43
View File
@@ -88,6 +88,17 @@ impl NamespaceLockWrapper {
pub async fn get_read_lock(&self, timeout: Duration) -> std::result::Result<NamespaceLockGuard, crate::error::LockError> {
self.lock.get_read_lock(self.resource.clone(), &self.owner, timeout).await
}
/// Acquire write locks for `resources` in one batched round using this
/// wrapper's owner; see [`NamespaceLock::acquire_write_locks_batch`].
/// One guard or error per resource, in `resources` order.
pub async fn get_write_locks_batch(
&self,
resources: &[ObjectKey],
timeout: Duration,
) -> Vec<std::result::Result<NamespaceLockGuard, crate::error::LockError>> {
self.lock.acquire_write_locks_batch(resources, &self.owner, timeout).await
}
}
impl NamespaceLockGuard {
@@ -349,6 +360,38 @@ impl NamespaceLock {
}
}
/// Acquire exclusive locks for many independent resources in one batched
/// round; see [`crate::distributed_lock::DistributedLock::acquire_guard_batch`].
/// Returns one guard or error per resource, in `resources` order.
///
/// This is the lock primitive bulk mutations (S3 `DeleteObjects`) should
/// use: per-key quorum round-trips serialize a 1000-key batch, while the
/// batched form is one wire round per lock client and stays deadlock-free
/// without any canonical serial fallback (every lock server orders
/// concurrent batches by the same global key order).
pub async fn acquire_write_locks_batch(
&self,
resources: &[ObjectKey],
owner: &str,
timeout: Duration,
) -> Vec<std::result::Result<NamespaceLockGuard, crate::error::LockError>> {
let ttl = crate::fast_lock::DEFAULT_LOCK_TIMEOUT;
match self {
Self::Distributed(lock) => lock
.acquire_guard_batch(resources, owner, timeout, ttl)
.await
.into_iter()
.map(|result| result.map(NamespaceLockGuard::Standard))
.collect(),
Self::Local(lock) => lock
.acquire_guard_batch(resources, owner, timeout, ttl)
.await
.into_iter()
.map(|result| result.map(NamespaceLockGuard::Fast))
.collect(),
}
}
/// Get health information
pub async fn get_health(&self) -> crate::types::HealthInfo {
let lock_stats = self.get_stats().await;
+51
View File
@@ -41,6 +41,16 @@ pub(super) fn pyroscope_log_filter() -> FilterFn {
filter_fn(allows_non_pyroscope_dependency_event)
}
/// s3s verbose events include response bodies and expected signatures. Keep
/// diagnostics spans, but never let these events reach logs or OTLP exporters.
pub(super) fn s3s_log_filter() -> FilterFn {
filter_fn(|metadata| {
!(metadata.is_event()
&& (metadata.target() == "s3s" || metadata.target().starts_with("s3s::"))
&& *metadata.level() > tracing::Level::INFO)
})
}
/// Build an `EnvFilter` from the given log level string.
///
/// If `default_level` is provided, it is used directly. Otherwise, the
@@ -339,6 +349,47 @@ mod tests {
format!("{HTTP_SERVER_LOG_TARGET}={level}")
}
#[test]
fn s3s_filter_blocks_credentials_despite_explicit_verbose_directives() {
use tracing_subscriber::prelude::*;
for directive in ["trace", concat!("trace,", "s3s::service=trace,", "s3s::ops::signature=debug")] {
let writer = SharedWriter::default();
let captured = Arc::clone(&writer.0);
let subscriber = tracing_subscriber::registry()
.with(EnvFilter::new(directive))
.with(s3s_log_filter())
.with(
tracing_subscriber::fmt::layer()
.without_time()
.with_ansi(false)
.with_writer(writer)
.with_span_events(tracing_subscriber::fmt::format::FmtSpan::NEW),
);
with_default(subscriber, || {
let span = tracing::debug_span!(target: "s3s::service", "safe-operation-span");
let _entered = span.enter();
tracing::debug!(target: "s3s::service", response = "secret-response", "response dump");
tracing::trace!(target: "s3s", response = "secret-trace", "response dump");
tracing::debug!(target: "s3s::ops::signature", expected = "secret-signature", "signature mismatch");
tracing::info!(target: "s3s::service", "visible-info");
tracing::warn!(target: "s3s::service", "visible-warning");
tracing::debug!(target: "rustfs", "visible-rustfs-debug");
tracing::debug!(target: "s3s_other", "visible-other-target");
});
let output = String::from_utf8(captured.lock().expect("captured logs").clone()).expect("UTF-8 logs");
assert!(!output.contains("secret"), "{output}");
for expected in [
"safe-operation-span",
"visible-info",
"visible-warning",
"visible-rustfs-debug",
"visible-other-target",
] {
assert!(output.contains(expected), "missing {expected}: {output}");
}
}
}
#[test]
fn test_is_verbose_level() {
assert!(is_verbose_level("debug"));
+3 -1
View File
@@ -36,7 +36,7 @@ use crate::cleaner::LogCleaner;
use crate::cleaner::types::{CompressionAlgorithm, FileMatchMode};
use crate::config::OtelConfig;
use crate::global::{METRIC_LOG_CLEANER_RUN_FAILURES_TOTAL, METRIC_LOG_CLEANER_RUNS_TOTAL, set_observability_metric_enabled};
use crate::telemetry::filter::{build_env_filter, effective_verbose_logging, pyroscope_log_filter};
use crate::telemetry::filter::{build_env_filter, effective_verbose_logging, pyroscope_log_filter, s3s_log_filter};
use crate::telemetry::rolling::{RollingAppender, Rotation};
use metrics::counter;
use rustfs_config::observability::{
@@ -369,6 +369,7 @@ fn init_stdout_only(_config: &OtelConfig, logger_level: &str, is_production: boo
tracing_subscriber::registry()
.with(env_filter)
.with(pyroscope_log_filter())
.with(s3s_log_filter())
.with(ErrorLayer::default())
.with(fmt_layer)
.try_init()
@@ -486,6 +487,7 @@ fn init_file_logging_internal(
tracing_subscriber::registry()
.with(env_filter)
.with(pyroscope_log_filter())
.with(s3s_log_filter())
.with(ErrorLayer::default())
.with(file_layer)
.with(stdout_layer)
+4 -1
View File
@@ -39,7 +39,7 @@
use crate::cleaner::types::FileMatchMode;
use crate::config::OtelConfig;
use crate::global::set_observability_metric_enabled;
use crate::telemetry::filter::{build_env_filter, pyroscope_log_filter};
use crate::telemetry::filter::{build_env_filter, pyroscope_log_filter, s3s_log_filter};
use crate::telemetry::guard::{OtelGuard, ProfilingAgent};
use crate::telemetry::local::{build_json_log_layer, spawn_cleanup_task};
use crate::telemetry::recorder::{Recorder, install_process_global_recorder};
@@ -296,6 +296,7 @@ pub(super) fn init_observability_http(
tracing_subscriber::registry()
.with(filter)
.with(pyroscope_log_filter())
.with(s3s_log_filter())
.with(ErrorLayer::default())
.with(file_layer_opt)
.with(stdout_layer_opt)
@@ -1196,6 +1197,7 @@ mod tests {
let subscriber = tracing_subscriber::registry()
.with(env_filter)
.with(pyroscope_log_filter())
.with(s3s_log_filter())
.with(
tracing_subscriber::fmt::layer()
.with_ansi(false)
@@ -1219,6 +1221,7 @@ mod tests {
tracing_subscriber::registry()
.with(build_env_filter("info", None))
.with(pyroscope_log_filter())
.with(s3s_log_filter())
.with(tracing_subscriber::fmt::layer().with_ansi(false).without_time())
.try_init()
.expect("install isolated LogTracer subscriber");
+1 -1
View File
@@ -20,7 +20,7 @@ const ARN_PARTITION_RUSTFS: &str = "rustfs";
const ARN_SERVICE_IAM: &str = "iam";
const ARN_RESOURCE_TYPE_ROLE: &str = "role";
#[derive(Debug, PartialEq, Eq, Hash)]
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ARN {
pub partition: String,
pub service: String,

Some files were not shown because too many files have changed in this diff Show More