Record end-to-end durations for ReadAll, WriteAll, ReadMultiple, and BatchReadVersion alongside existing request, error, and payload metrics. This makes direct unary payload ABBA results observable through the internode operation duration family.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Release the per-peer prewarm deduplication marker when the best-effort dial fails, allowing a later RemoteDisk construction to retry the connection warmup without changing lazy connect fallback behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
SFTP subdirectory mkdir and rmdir sent the on-disk __XLDIR__ marker
through the public S3 API. After the suffix was reserved, those calls
were rejected as invalid client keys, so ordinary directory operations
failed while bucket mkdir still succeeded.
Send the logical trailing-slash key and let storage encode the marker.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
* fix(ecstore,heal,select): bound storage-layer work driven by client input
Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:
- Multipart: upload ids are canonicalized before they name a staging path
or a lock key (backlog#3027); trailing-slash keys are encoded like every
other handler (backlog#3055); unlisted parts are removed at completion
(backlog#3106); per-bucket uploads are found through a signed per-disk
index instead of a scan of the shared staging area (backlog#3121);
encrypted part ETags carry a per-upload secret so a write-only user
cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
(backlog#3038); a delete marker on a key whose xl.meta path is a
directory, and sidecar names used as child keys, no longer wedge the
journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
counts delete markers against max-keys (backlog#3108), and rejects a
marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
bucket is skipped instead of failing the rebuild (backlog#3040), and
directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
modification time (backlog#3073, backlog#3074); Snowball transport
headers are not forwarded to the tier (backlog#3090); an epoch
modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
(backlog#3131); SelectObjectContent releases its namespace locks once
the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
(backlog#3129); a bucket-metadata reload timeout no longer marks the
peer offline (backlog#3124); replication target requests carry an
operation deadline (backlog#3102); the scanner resumes its interrupted
wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
batches, Parquet dictionary bounds and pre-planner projection checks
(backlog#3075, backlog#3076, backlog#3077, backlog#3078).
* test(e2e): seed legacy duplicate bucket tags offline for the OPA contract
* fix(s3): authorize what handlers store and bound request-driven work
Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.
Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
the tag set it copies (backlog#3034); HeadObject with the replication
check header still authorizes s3:GetObject (backlog#3047); the
x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
recursive force-delete with a REPLICA header authorizes every
descendant as a delete (backlog#3070); include-deleted listings require
s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
table isolation and uses the same prefix it authorizes (backlog#3053,
backlog#3054); If-Match probing needs read permission (backlog#3097);
Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
returns lock and tag headers only with permission (backlog#3113);
IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
(backlog#3049); bucket policies are parsed once and cached
(backlog#3032); keystone: access keys no longer verify with an empty
secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
(backlog#3023); PutObject takes buffer and foreground admission only
after body bytes arrive and bounds the wait (backlog#3031,
backlog#3084); xl.meta size is bounded on the write path
(backlog#3100); XML compat layers stream instead of buffering whole
bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
completion (backlog#3130); MinIO admin decryption runs Argon2id off the
async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
(backlog#3092, backlog#3093); control characters in keys and lifecycle
rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
Content-MD5 and admin/STS reject SigV2 replays with altered query or
body (backlog#3037, backlog#3026); request logs redact presigned
queries and handler spans skip credentials (backlog#3068,
backlog#3107); SSE-C listings and completions withhold plaintext
metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
keys (backlog#3126); notifications fire after a client disconnect
(backlog#3127); ListenNotification streams re-check access
(backlog#3119).
* fix(api): drop imports left unused by the base merge
* test(s3): move the KMS multipart denial test to the app layer
The test drives app-layer multipart use cases, so it belongs with them;
living in the SSE module it imported app code from the infra layer and
failed the layer dependency guard. The SSE test helpers it borrows are
exposed through the storage and app facades.
* fix(s3): keep the SSE-C key fingerprint on keyless multipart completion
The backlog#3050 fix withheld both the plaintext checksums and the
customer-key MD5 header from a CompleteMultipartUpload that did not
present the key. The checksums are a digest of the plaintext and stay
hidden; the key MD5 identifies the key only, and the S3 conformance suite
(test_copy_part_enc) expects it on a keyless completion, so it is echoed
again.
* fix(tier): stream GET bodies and document replication budgets
* fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes
Two lifecycle regressions surfaced once the storage and request batches
ran together against main:
- The multipart index owner cloned the disk's pinned publication root and
bootstrapped on a task of its own. That kept the mount alive past the
last LocalDisk owner (the disk lifecycle allows exactly three holders of
the root FD) and left a stray task behind, which the local-disk and
read-repair tests detect. The owner now keeps only the root path and
reopens it for identity checks, and the bootstrap scan rides the disk's
existing cleanup task, where stopping the disk abandons it and a
reconnect resumes it.
- The disk-read admission wrapper defers releasing its permit until an
in-flight read completes, so a cancelled client cannot widen disk
concurrency. The cold-fill producer, however, drops a read it has
already classified as stalled when its deadline fires, and that permit
stayed pinned for the lifetime of the stuck read. The producer now
abandons the admission explicitly on timeout.
* test(ecstore): pass the index owner to the Linux cleanup-loop test
* fix(s3): release cold-fill admission after watchdog handoff
* fix(s3,ecstore): close the round-1 adversarial review findings
Adversarial review of the combined request- and storage-layer batch found
fixes that fell short of their issues, two behaviour regressions for
legitimate clients and several unrequested costs:
- backlog#3038: a bulk DeleteObjects reserved the whole per-disk
inspection allowance up front for every key in flight and exhausted the
16 Mi request budget on any set wider than eight drives, failing healthy
keys with SlowDown. Calls are now admitted while budget remains and
charged for the entries they actually visited. Object-scoped discovery
charges plain child keys at one entry per 64 names instead of one each,
so deleting a missing key that names a folder prefix completes instead
of answering 503 forever while a hostile prefix stays bounded. A peer
still on the previous binary is reported
as an unsupported participant, the repository's rolling-upgrade
contract, instead of as a budget verdict.
- backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was
still replayable with another body; it is refused with MissingContentMD5
while SigV4 keeps the optional-MD5 contract.
- backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before
authorization like PUT and COPY, so a replica delete cannot remove a
version the policy never saw.
- backlog#3121: the multipart index no longer fsyncs a marker and four
directories per multipart metadata write (it is rebuilt on every start),
and ListMultipartUploads discovers roots concurrently.
- backlog#3077: Parquet SelectObjectContent decodes budget-sized batches
instead of one row per batch.
- backlog#3031: a full upload buffer pool hands out an unpooled buffer
instead of refusing the upload; the progress floor already bounds stalls.
- backlog#3023: the GET disk-read idle cut-off is configurable
(RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused
client is not truncated after 30 s.
- backlog#3075: JSON Lines schema inference keeps the sample collected at
the 4 MiB bound instead of failing the query.
- backlog#3024: compat admin decryption queues briefly for a slot instead
of refusing the third concurrent request.
- backlog#3113: HEAD skips its permission probes for the bucket owner.
- backlog#3117: control characters in a lifecycle rule ID are rejected on
PutBucketLifecycle as well.
The lock-crate publication FIFO serves backlog#3111 and moves to the
table-catalog PR. Also fixed: a lossy admin error message, an unused
s3-client function, two unused test-util features, a duplicated key
validation, the s3s footprint baseline, and a stale log event name in the
reverse-proxy runbook. The ODM multipart write-back test runs on the large
test stack like its sibling.
* fix(table-catalog): bound maintenance work and isolate table namespaces (#9174)
* fix(table-catalog): bound maintenance work and isolate table namespaces
Scan findings in the Iceberg table catalog:
- Metadata maintenance deleted every unreferenced *.metadata.json in a
table's directory without checking whether a renamed table in another
namespace still owned those files (backlog#3060); it now resolves the
owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
writer schema and no recursion depth limit; a self-referencing record
schema overflowed the stack (backlog#3080). Schemas are validated for
depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
memory; a decompression bomb OOM-killed the server (backlog#3085). Row
groups are streamed through a byte budget and oversized inputs are
refused.
- An S3 request for a key outside an indexed table prefix listed the
bucket's whole namespaces/ catalog prefix and failed once it held more
than 4096 objects (backlog#3086); a per-bucket directory index answers
index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
left the bucket's catalog entry locked at 503 (backlog#3088); the rename
runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
starving every object write (backlog#3111); commits now hold it under a
deadline and object writes are admitted between commits.
The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.
The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.
* fix(table-catalog): close the review gaps in the scan-fix batch
Adversarial review of the table-catalog batch found the fixes narrower
than the issues they close and two cost regressions:
- backlog#3085: the compaction footer check bounded nothing a page header
could not override; parquet sizes its page buffers from the header.
Walk every column chunk's page headers (Thrift compact, no decompression)
and hold each page and the chunk total to the budget before decoding.
The stored-input budget is per rewrite group, so a multi-group job no
longer fails after writing earlier outputs.
- backlog#3086: the default index-miss scan still counted maintenance
reports toward its 4096-object limit; only table entries count now, and
the listing itself is bounded to 64 pages. The directory index remains
an optional accelerator.
- backlog#3111: a create wrote its metadata file under the fence as the
first durable mutation, which disarmed the deadline before the unbounded
registration inventory. The write precedes publication now. The deadline
gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS,
default 30 s, never below the lock-acquire timeout) instead of reusing
the 5 s lock timeout, and the ownership inventory reads entries
concurrently.
- Directory candidates are written only while a directory exists; the
backfill covers earlier entries. A directory on the migration-fence
path is reported as a conflict instead of a parse error.
- Dead test hooks, a stale comment and the support-matrix text that did
not match the code are removed or corrected.
* fix(select): preserve complete samples and Parquet batch limits
* fix(s3,ecstore): close the round-2 adversarial review findings
DeleteObject resolved x-rustfs-source-version-id for authorization on
every request while storage consumes it only for a REPLICA request, so a
non-replica caller with s3:DeleteObjectVersion could trigger a plain
delete of the current version. The hook now resolves the header under
the same predicate as storage, and the regression test covers the
non-replica case.
HEAD no longer skips its permission probes for the bucket owner; an
explicit bucket-policy Deny on the metadata actions binds the owner too.
Multipart index pages verify each root against the staging area and
prune markers whose staging root is gone, under the marker's own lease
with a re-check, so the per-drive index stays proportional to live
uploads instead of every key ever multipart-initiated on the drive.
SigV2 configuration PUTs that declare an empty body need no Content-MD5:
there is no body to replay. The minimum-progress floor stays independent
of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now
says so instead of claiming 0 disables it. The quota bootstrap window
(503 until the background scan publishes the ledger) is kept, because
ledger admission must not come from scanner reporting; the compatibility
matrix now documents it.
The S3 compatibility matrix gains a section for the client-visible
validation bounds, and the drive-timeout tuning guide tabulates
RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS.
* fix(lock): queue table-bucket publication writers fairly
Table commits take the bucket publication fence as a write lock; under a
stream of commits the lock's try-and-retry acquisition let a new commit
jump ahead of object writers that had been waiting longer, which is part
of what starved them (backlog#3111). Requests for the publication lock
path now take a FIFO ticket per shard: a waiter is admitted only when it
is at the head of the queue, cancelled waiters release their turn, and
batch acquisitions join the same queue. Other lock paths are unchanged.
This moved here from the request-layer batch, where no cited issue
covered it; it serves the table-catalog issue this PR closes.
* fix(table-catalog): close the round-2 adversarial review findings
Parquet page-header walker: read the size fields by field id as parquet
does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so
a header encoding uncompressed_page_size as I64 can no longer bypass the
budget while parquet allocates it. Bound every list, set and map count by
the bytes remaining in the chunk, and reject bool collection elements
(parquet skips them with zero width, so either width would desynchronize
the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>.
Start each column-chunk walk at the dictionary page offset whenever the
footer declares one, exactly as parquet's byte_range does, instead of
min(dictionary, data).
Footer pre-walk: before parquet decodes a footer, walk FileMetaData with
the hardened cursor and charge the declared row_groups count times the
RowGroupMetaData size to the decode budget, because parquet 59 reserves
that Vec before reading one element.
Preparation deadline: clamp the
RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400
seconds so Instant + Duration cannot overflow.
Index-miss scan: after each listing page, jump the cursor past the
maintenance subtree the page ended in, and allow one extra page per table
entry already listed, so job-report history scales with the catalog while
a report flood under one table stays bounded.
Migration fence: read the fence-path object once and decide from its
stored kind whether it is directory state or a fence.
Expired or failed create: remove the initial metadata file when the
registration fails before any catalog write may have begun; nothing else
references it. Correct the comment and support matrix that claimed
maintenance reclaims it.
Tests cover each walker divergence, the footer row-group reservation, the
deadline ceiling, the paging budget for honest catalogs and for a report
flood, and the removal of the unpublished metadata file.
* test(admin): register the catalog directory routes in the route matrix
The table-catalog batch added GET, POST and DELETE
{warehouse}/catalog/directory on both the Iceberg and the compat admin
prefixes without listing them in the admin route matrix, so
test_admin_route_matrix_matches_registered_routes failed on CI.
* fix(s3,ecstore): close the round-3 adversarial review findings
Multipart index pages now treat the cursor as the last examined marker
rather than the last root, so a page that pruned a stale marker still
validates and ListMultipartUploads keeps using the index. A marker is
written before its staging root is published, so pruning spares markers
younger than a five-minute grace period and an existing marker is
touched on every new upload; the prune re-checks both under the marker
lease. Bounded delete-recovery inspection rides the durable-mutation
lane like the unbounded path instead of the one-worker foreground lane.
The on-demand-migration inline body budget scales with the object size
(at least 64 KiB/s) instead of a flat 120 s. The control-body digest
service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no
inactivity timeout" like every other reader of that knob.
Quota reconciliation throttle objects are deleted together with a
retired generation's ledger, so bucket turnover leaves no residue. The PUT guard
test shim is gone; tests call the shared guard. Bounded DiskAPI defaults
answer MethodNotAllowed like their siblings. Stale
budget docs, an unreachable CSV arm, an always-Some limit parameter,
commented-out constants and a misplaced doc comment are cleaned up;
weak test assertions name the expected error; lifecycle rule-ID control
characters gain a test. The compatibility matrix records the CopyObject
tag-copy permission, the exact SigV2 empty-body predicate, replica and
migration writes of control-character keys, and the SigV2 DeleteObjects
digest rule.
* fix(table-catalog,heal): close the round-3 review findings
A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job,
TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before
any output is written, and name the ceiling in the commit-path errors;
document the effective cap in the support matrix and scheduler runbook.
B. Keep TaskCancelled/TaskTimeout classification on replacement passes:
the namespace verification runs only for completed or storage-failed
passes, in the erasure healer and the task bucket prepass.
C. Compare the replacement bucket pass per bucket: a recorded identity
that is gone or replaced still errors or restarts the pass, buckets
created mid-pass are appended to the identities, pending_buckets and a
persisted positional order without rewinding progress or the
checkpoint; verification ignores additions.
D. Read the migration-fence object once in the object-backed write permit
and dispatch on its kind.
E. Drop the strong store's copy of maintenance_cleanup_roots.
F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and
pause_after_metadata; make compact_parquet_data_files and
PublicationTicket.state private.
G. Document the body-revision identity, the legacy revalidation budget
and its HTTP 503 in the support matrix.
H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it
from heal through the storage_api facade.
* fix(ecstore): cap the index pages one disk may answer
Pruned multipart index pages may carry few roots, so the page cursor no
longer bounds how many pages a peer can make a listing fetch. Cut a disk
off after 4096 pages (about one million examined markers) so it
contributes no coverage and the listing falls back to the staging scan.
* fix(ecstore,table-catalog): close the round-4 adversarial review findings
Multipart index markers dated in the future (a backwards clock step
between recording a marker and publishing its staging root) were judged
old and could be pruned while their upload was in flight; a future-dated
marker is now kept. The marker touch on EEXIST is best effort, so a
filesystem that refuses to set times no longer fails the upload commit.
Pruning runs on the durable-mutation lane instead of the one-worker
foreground lane.
Quota reconciliation keeps one throttle object per bucket incarnation (a
quota revision change replaces it instead of orphaning the previous
revision's object), the retirement cleanup deletes it ahead of the ledger
and also when the ledger is already gone, and a scan the retirement
already refused writes nothing back. Bounded DiskAPI defaults answer
UnsupportedDisk, the verdict the aggregate gives a peer without the
protocol. The compaction ceiling pre-check runs over every rewrite group
before any output is written, and the worker's ceiling error no longer
names a placeholder group.
Docs: the inline on-demand-migration client budget, the protocol-2
downgrade conflict of replacement passes, the exact footprint numbers.
* fix(ecstore): retry metadata writes rejected before staging
* fix(ecstore): reuse registered disks during startup
* fix(ecstore): retain remote health monitors during reconnect
* test(heal): release the commit barrier during graceful shutdown
* test(e2e): register graceful shutdown barrier regressions
* fix(ecstore): roll back interrupted pool metadata commits
* test(upgrade): honor the 1.0.4 delete recovery protocol
* fix(heal): bound admission during object rate pressure
* test(e2e): handle rebalance completion before stop
* test(e2e): observe rebalance activity before the start reply
* test(rebalance): hold the first run until stop cancels admission
Use explicit buckets that preserve millisecond lock acquisitions while retaining visibility into long-held delete and recovery locks. Add an exporter-backed regression test for both histogram instruments.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): treat duplicate CreateBucket as owned by the requester
The duplicate-bucket response was decided from the requester's root-owner
flag, so every IAM user recreating a bucket received 409
BucketAlreadyExists, including a user recreating a bucket it created.
BucketAlreadyExists means the bucket belongs to another account; a
deployment owns a single namespace, so the answer is purely region-based:
the default region returns the legacy 200 OK, any other region returns
BucketAlreadyOwnedByYou.
Both the existing-bucket fast path and the BucketExists path now share one
helper. Authorization still runs first, so a requester without
s3:CreateBucket gets AccessDenied and cannot probe bucket existence.
Refs #8493
* test(ci): refresh e2e selections for duplicate CreateBucket
The two new create_bucket_region cases join the e2e-smoke and e2e-full
selections, which main also extended with multipart_upload_id tests.
smoke is measured from a local nextest listing: removing the two new cases
reproduces main's committed digest exactly. e2e-full is derived the same way
from main's listing artifact (run 41933823399, babfeaed0) plus the two
multipart_upload_id cases, which reproduces main's committed linux digest
before the new cases are added; CI re-verifies both platforms.
test_bucket_create_exists_nonowner asserts BucketAlreadyExists for a second
identity, which a single-owner deployment cannot produce. Move it to
excluded_tests.txt with the deployment-ownership rationale; the IAM-user
duplicate paths stay covered by crates/e2e_test/src/create_bucket_region_test.rs.
UploadPart, UploadPartCopy, ListParts and CompleteMultipartUpload returned 400 InvalidArgument for a malformed upload ID, while AbortMultipartUpload returned 404 NoSuchUpload. AWS S3 documents NoSuchUpload for an upload ID that might be invalid.
Report MalformedUploadID as NoSuchUpload through one helper used by all operations that address an existing upload, replacing the Abort-only special case. ListMultipartUploads keeps InvalidArgument for a malformed upload-id-marker.
Add the multipart_upload_id_test e2e module to the e2e-smoke profile. It asserts 404 NoSuchUpload from every operation that addresses an existing upload, for a malformed upload ID and for an aborted one, and updates the e2e-smoke and Linux e2e-full selection digests.
Co-authored-by: Karl Bastarache <bastarachek@google.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
* perf(ecstore): probe bulk-delete write locks in parallel waves
Bulk delete acquired one distributed write lock per key in serialized
order, paying a full quorum round-trip per key. Probe the unique keys in
bounded parallel waves with a short timeout instead; any contention drops
the whole wave and the canonical serial order resolves it as before, so
two overlapping batches can never deadlock.
* perf(ecstore): probe bulk-delete transaction locks in parallel
The set-layer delete-transaction locks were also acquired one quorum
round-trip per key after the raw write locks. Apply the same optimistic
all-or-nothing probe wave with a canonical serial fallback, and keep the
per-key error reporting the fallback already provided.
* perf(ecstore): pre-fetch bulk-delete source metadata in parallel
The admission loop ran one metadata quorum fan-out per key in sequence
on top of the per-key lock round-trips. Pre-fetch the source reads the
loop needs with bounded parallelism under the locks already held, and
derive the per-key read inputs from one shared helper so the pre-fetch
and the loop cannot disagree.
* test(ecstore): drop unused import in batch-delete probe test
* refactor(ecstore,lock): await wave releases and unify probe waves
Adversarial review of the bulk-delete lock waves found three real
issues. Distributed guard release is fire-and-forget on drop, so the
serial fallback could re-acquire keys while unlock RPCs were still in
flight and block on locks it already owned: guards are now released
with an awaited unlock before the fallback runs. The wave also probed
every key after the first failure, adding up to one probe timeout per
chunk of pure overhead under contention: probing now stops at the
first failing chunk. Finally the two copy-pasted wave implementations
are replaced by one shared probe_lock_wave, and the per-key delete
read inputs are resolved once instead of being recomputed.
Tests now assert the wave/fallback counters so they prove which path
ran, and an overlapping-batch test pins the no-deadlock property.
* refactor(ecstore,lock): batch bulk-delete lock rounds per key
Replace the optimistic probe waves with the bulk lock primitive the lock
layer already supports on the wire: one lock_batch RPC per lock client
for a whole DeleteObjects key set, per-key quorum guards with the usual
TTL heartbeat, and per-key outcomes. The batched round is deadlock-free
by construction (every lock server orders concurrent batches by the
same global (shard, key) order), so the probe timeout, the wave
release-and-wait dance, and the canonical serial fallback all go away.
Per-key lock outcomes are now reported per key end to end: a locked-out
key fails alone in the response while the rest of the batch proceeds,
matching the S3 DeleteObjects contract and the set layer's existing
behavior. The set layer's raw-write-lock and delete-transaction-lock
waves fold into the same primitive via one shared resource-folding
helper, and the fast-lock batch fast path now counts acquisitions so
shard metrics keep their meaning.
* perf(ecstore): move journal publish off the one-worker namespace lane
A bulk delete publishes one recovery transaction per key, and every
journal inspection and settle-plan closure ran on the per-disk
foreground namespace lane (one worker), serializing the whole publish
phase per drive behind every other mutation. Route those closures onto
the durable-mutation lane the fused delete prepare already uses (eight
workers per disk, fsyncs coalesce in the filesystem journal), and keep
32 journal publications in flight so a 16-drive set can feed those
lanes.
* test(ecstore): flatten lock-error iteration in overlap test
* fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock
Review findings on the batched lock rounds. A batch entry whose recovery
rolls back an interrupted prepared delete can bring a retained version
back; a later duplicate of the same key still held its pre-fetched
source-missing read, skipped the object-lock check, and deleted the
restored version. Recovery now evicts the pre-fetched reads of every
other entry naming the same object, so those entries take the loop's
live re-read and see the restored version.
The batch rollback release awaited the full unlock retry schedule,
including the deferred background pass, parking the request behind an
unreachable peer. The synchronous phase is now a single release pass
with the deferred retries spawned in the background.
* fix(ecstore): release local batch locks before deferred recovery
A set-layer batch delete that runs deferred key recovery re-acquires a
write lock on the same keys through recover_delete_object, but the raw
local batch guards were only ever released at function exit — the
distributed path releases its batch ids before the deferred loop and the
local path missed the mirror release, so the recovery deadlocked against
the batch's own locks and surfaced as a spurious Lock(Timeout) for every
deferred key.
A regression test plants the interrupted prepared delete this needs and
pins the end-to-end contract: a batch naming a GOVERNANCE-retained
version twice must deny both entries after the first entry's recovery
rolls the prepared delete back, and the restored version must survive.
* test(lock): cover batch acquire partial-grant rollback
The distributed acquire_guard_batch rollback path — a key granted by
fewer than quorum clients must fail alone and release its partial grants
before the call returns — was only reachable in distributed mode and had
no coverage. A scripted per-object LockClient pins both halves: the
below-quorum key reports QuorumNotReached with its partial vote, and the
orphaned grant is released synchronously while the quorate key's guard
stays held.
Track active descriptor volumes, exact paths, and component prefixes in FdCacheTracker so unmatched invalidate_under, invalidate_volume, and invalidate_exact calls return without scanning Moka or waking the background maintenance task.
Scan findings in the alternate-protocol front ends and the KMS and
Keystone integrations:
- WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE,
checked that the file existed, and then built their TLS listeners
with with_no_client_auth(), so mTLS was never enforced. A shared
protocols::common::tls::server_config builds the rustls config with a
client-certificate verifier from the configured CA, startup fails on
an invalid CA, and a CA cannot be combined with cleartext
(backlog#3069).
- The KMS readiness probe used a fixed key id that the key-lifecycle
admin routes accepted like any customer key, so kms:DisableKey on it
failed readiness on every node. The probe key is reserved from key
management and a disabled probe key no longer counts as a backend
failure (backlog#3115).
- The Keystone middleware answered anonymous requests carrying a junk
X-Auth-Token with the configured Keystone URL in the body while
Keystone was unreachable; failures no longer disclose upstream
locations, and the login cache is bypassed when a subscription token
refresh is required (backlog#3116).
Scan findings in crates whose behavior is driven by untrusted input and
that stand alone from the storage core:
- trusted-proxies: the simple implementation takes the client address
from the right-most untrusted X-Forwarded-For hop instead of the
left-most (backlog#3082); the legacy validator never falls back to the
proxy identity when the chain is rejected and reads X-Forwarded-For
unless RFC 7239 is explicitly selected (backlog#3051).
- targets: webhook and similar outbound clients drain responses up to a
byte limit instead of buffering arbitrary bodies (backlog#3062); the
on-disk notification queue has a shared byte budget in addition to the
entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES
(backlog#3122); AMQP and Redis manifests declare the broker url as a
secret and bind credentials to their connection fields, and the
plugin-instance PUT refuses to restore a redacted secret when the
connection fields changed (backlog#3065, backlog#3066).
- obs: the debug-level s3s response dump is filtered so STS and
service-account credentials never reach the log (backlog#3029).
Scan findings in the IAM crate and the admin identity handlers, grouped
because they share the IamCache mutation lock, the STS session sweep and
the admin client-address plumbing:
- Admin and STS authorization used the socket peer as aws:SourceIp
behind a trusted proxy; they now read the verified client address the
S3 path already uses (backlog#3025), and the admin usage route passes
the bucket named in the path to the s3:ListBucket check (backlog#3033).
- AssumeRoleWithWebIdentity accepted HS256 ID tokens MACed with the OIDC
client secret; only provider-signed tokens verify (backlog#3028), and
JWKS refresh is single-flight with a failure cool-down (backlog#3103).
- A disabled group emptied the member's whole policy list, dropping
every Deny; direct and other-group policies are preserved (backlog#3042).
- Identity rewrites (secret change, add-user, status) ran without a lock
and without re-reading the record, so a user's password change could
undo an admin disable or reset (backlog#3048). add-user on an existing
user and in-flight AssumeRole both left sessions minted under the old
secret valid (backlog#3071, backlog#3079); rotation now revokes under
the per-key mutation lock and derived-credential publication is ordered
with it.
- aws:groups never carried a built-in user's IAM group membership and
jwt:iss was always the local marker, so group- and issuer-scoped
Denies never matched; both now come from live IAM state
(backlog#3095, backlog#3096).
- OPA mode skipped the account-status gates for service accounts and STS
sessions of a disabled user (backlog#3057).
- The periodic IAM reload published nothing when any cache write raced
it and kept stale partitions when a prefix listed empty, so a missed
revocation never applied (backlog#3064, backlog#3114).
- update-group-members never evaluated admin:RemoveUserFromGroup
(backlog#3098); ExportIAM compared the site-replicator account against
a name it never has (backlog#3045); add-user accepted the access key of
a live STS session or service account (backlog#3134); a service account
could create or extend a lifetime beyond its own (backlog#3112);
sts:DurationSeconds and svc:DurationSeconds were never populated
(backlog#3136); temporary-account-info returned service accounts
(backlog#3137); an access key the object layer refuses no longer holds
the service-account lock through the write retries (backlog#3133).
A put-like write without a version id to a key that is not
versioning-Enabled replaces the key's null version in place. The
commit-time Object Lock gate in put_object_with_old_current_size_inner
ran only when the write carried an explicit version id, and the
request-layer pre-check reads the key's latest version. When the
retained null version was not the latest entry (a UUID version or a
delete marker sat above it), a user holding only s3:PutObject could
overwrite it together with its COMPLIANCE retention or legal hold, and
CompleteMultipartUpload had no commit-time gate at all.
Name the version a commit replaces (the explicit id, or the nil id when
the write is not versioned and the bucket's Object Lock snapshot is not
ConfirmedAbsent) and run the existing WORM gate against it, tolerating
a null delete marker. complete_multipart_upload gets the same gate at
its commit point, and ECStore::handle_complete_multipart_upload attaches
the bucket's Object Lock snapshot the way a single PUT already does.
Refs rustfs/backlog#3019 (ANT-2026-EH17ZP59).
Eight scan findings in the shared policy engine, each a case where a
Deny did not apply or an Allow was broader than written:
- A bare `*` Action was parsed as `s3:*`, so `Deny *` never matched
admin:, kms: or sts: actions. `Action::All` now denies every family
while the historical S3-only Allow is preserved (backlog#3041).
- A bare IPv6 address in aws:SourceIp got `/32` appended, matching a
2^96-address network; a bare address now means one host, and IPv4
clients on a dual-stack listener are matched through their mapped
form (backlog#3030).
- Date conditions on aws:EpochTime parsed the epoch seconds as RFC 3339
and never matched, so a time-bounded Deny failed open (backlog#3036).
- NumericNotEquals and DateNotEquals evaluated to false when the key was
absent; negated operators now follow the AWS rule and match
(backlog#3123).
- Every trailing IfExists on a condition operator added a level of
recursion without bound; the chain is rejected before parsing
(backlog#3091).
- Policy variable substitution re-expanded its own output for up to ten
passes, so an access key holding k placeholders grew to k^10; values
are now substituted once as literal data (backlog#3067).
- A bucket-policy Service principal was matched against the signing
access key, so a service account named after the service received its
grants; Service principals round-trip but never match (backlog#3081).
- Resource matching cleaned the trailing slash off a directory object,
so a Deny on `prefix/*` missed `prefix/` itself (backlog#3125).
Also deduplicate large Resource sets through a HashSet instead of a
quadratic scan.
DeleteObjects ran its Object Lock check on one version of a key and then
removed a different one. For an entry without a version id on a key that
is not versioning-Enabled, delete_objects_with_accounting judged the
key's latest entry and then had the disks remove the null version. When
a UUID version or a delete marker sat above a null version under
COMPLIANCE retention or legal hold, that retention was never read and a
user holding only s3:DeleteObject could remove the retained version. The
single-object DELETE on a Suspended key had the same gap.
Add SetDisks::check_null_version_object_lock_delete, which looks up the
null version under the lock the caller already holds and runs the
existing check_object_lock_delete against it when the latest entry
carries a UUID version id. The batch path is not gated on
source_missing because a delete marker on top makes the latest-version
lookup report the key as missing. Plain unversioned keys and
versioning-Enabled keys keep a single lookup and unchanged behavior.
Refs rustfs/backlog#3017 (ANT-2026-6ED2F7II).
Storage encodes a directory object `photos/` as `photos__XLDIR__/xl.meta`
and LocalDisk::scan_dir decodes every directory entry ending in the
suffix back into a marker. The S3 write path never refused the suffix in
client keys, so a 1-byte PutObject to the key `__XLDIR__` made the walk
schedule the directory being scanned as its own child and recurse until
the stack overflowed, aborting the server on every restart until the
directory was removed by hand. Keys below a `*__XLDIR__/` segment were
stored but hidden from every listing and from quota accounting.
validate_object_key now rejects any client key containing the suffix,
which covers PutObject, CopyObject, Snowball extraction and the admin
internal put; CreateMultipartUpload and CompleteMultipartUpload now run
the same validation so uploads created before the key was reserved
cannot publish it. scan_dir additionally ignores a marker whose decoded
name is empty, so malformed markers left by older writers or corrupt
disks can no longer recurse into their parent.
Refs rustfs/backlog#3020 (ANT-2026-SBJ1JLC2).
* fix(replication): replay the MRF journal periodically
The persisted MRF journal was replayed only at startup. Delete-marker
purges whose watch window drained with targets unreachable, and
replications that overflowed the worker queues, reach the journal while
the server runs, so they waited for a restart to converge.
Move the startup pass into replay_mrf_journal and rerun it every five
minutes from a background loop. The existing recovery leader lock keeps
one pass at a time across the cluster, including the startup pass.
* fix(replication): space periodic MRF replay across nodes
Every node runs the MRF replay loop and the recovery leader lock only
serializes passes, so an N-node cluster replayed the same retained
backlog N times per interval while a target stayed down.
Record the start time of a pass that finds entries in a shared replay
stamp and skip a periodic pass when any node started one within the
interval. The startup pass still always runs.
* fix(heal): treat SlowDown as a rate signal and bound page churn
* refactor(heal): unify page rate control and drop checkpoint clones
* refactor(heal): one pass-wide rate controller, single SlowDown observe
* fix(heal): address SlowDown backoff and checkpoint baseline review
Replace the single-mutex PREPARED_DELETE_RECEIPTS and ACTIVE_MULTIPART_SOURCE_HEADS maps in rustfs-ecstore with a 64-shard ShardedPathCache backed by SWAR canonical byte keys and xxh3_64 hashing. Replace PUT_FILE_TARGET_LOCKS in rustfs internode RPC with a 64-shard ShardedPutFileTargetLockRegistry using stack-buffered binary keys, in-place dead-Weak slot reuse, and amortized garbage collection.
Large metacache elements (an xl.meta grown past 16 MiB with its version
history) were accumulated in fixed 64 KiB reads, paying hundreds of
read rounds per entry on the listing path. Double the per-round chunk up
to a 4 MiB bound so each reservation stays incremental and bounded
(rustfs/rustfs#2715 corrupt-length-prefix safety is unchanged) while
wide listings and oversized entries decode in far fewer rounds.
Add regression coverage for rustfs/rustfs#9120: a listing whose
aggregate stream exceeds the per-value 16 MiB bound, and an oversized
entry listed among ordinary neighbors with peek/skip interleaving.
The rustfs_cluster_servers_offline_total gauge was event-driven only:
record_peer_reachable fired solely after a fresh connector.connect()
succeeded. When post-recovery traffic was served by a cached channel
(tonic auto-reconnect, addr-form cache-key split, or delete-recovery
channel), no fresh dial occurred and the gauge stayed stale until the
process restarted.
Add record_peer_reachable_if_offline (fast-path read / slow-path write)
and wire it into every successful peer interaction:
- remote-disk recovery probe success (gRPC DiskInfo)
- periodic TCP connectivity check success (bounded-interval reconcile)
- data-path RPC success (execute_with_timeout_for_op_and_health_action)
- lock-path RPC success (record_rpc_success and detached completion)
Fixesrustfs/rustfs#9119
Tracking: rustfs/backlog#3138
Remove pull_request from the distributed and upgrade workflows. Distributed
stays on the nightly cron, and upgrade stays on the Monday cron plus
version tags. Both remain manual-dispatch lanes and stay outside the
required merge checks.
Co-authored-by: majinghe <majinghe@users.noreply.github.com>
Give each mixed-workload request a bounded policy for the server's
explicit namespace lock timeout response and Retry-After advice. Keep
unrelated SDK failures and data mismatches immediately visible.
Preserve 24 workers, four rounds, cross-node checks and all retained
objects. Cover exhausted SDK retries, bodyless HEAD, fatal responses,
persistent contention and incorrect acknowledged bytes, and include the
new cases in the protected distributed profile inventory.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh crates and adopt openidconnect 4.1
Use published OIDC/OAuth2 releases and reject malformed HMAC signature
lengths before upstream verification. Refresh workspace and fuzz locks,
and advance the local zstd compatibility patch to 0.13.4 while retaining
the fixed 0.14.3 codec backend.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh compatible lockfile resolutions
Resolve Prost build and derive dependencies with itertools 0.14 and
prefer windows-sys 0.61.2 within the existing supported version ranges.
Retain all package versions, sources and checksums.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): separate cleanup checks from short probe windows
Use the functional measurement budget for the cleanup HTTP fixture so
setup and measured I/O can finish before checking delete status handling.
Exercise both normal scheduling and an in-flight PUT clock advance, and
include target classification in unexpected-failure diagnostics.
Retain the cleanup deadline, outer timeout and delete-stage assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate stalled multipart disk timeouts
Apply the short injected timeout only to the deliberately stalled disk.
Healthy disks retain their normal operation budgets, while logical lock,
physical owner, immediate CopyObject and final shard assertions remain.
Keep overrides scoped across cloned and reconnected handles, preserve
shorter caller deadlines and explicit zero timeouts, and report drive
health when the real CopyObject assertion fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): retry metadata write lock quorum admission
Recognize typed write-lock quorum failures for rebalance.bin within the
existing bounded metadata admission retry policy. Each attempt must
acquire quorum before loading or saving metadata; data writes are not
replayed by this retry.
Cover recovered and persistent quorum failures, cancellation after a
failed attempt, and rejection of unrelated resource or read-lock errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add an opt-in `fips` Cargo feature to `rustfs` and `rustfs-tls-runtime`
that enables `rustls/fips` and `tokio-rustls/fips` (activating
`aws-lc-rs/fips` / `aws-lc-fips-sys` and `rustls-webpki/aws-lc-rs-fips`).
When compiled with `--features fips`, `startup_runtime_hooks` installs
`rustls::crypto::default_fips_provider()` as the process-wide default
crypto provider and asserts `provider.fips()` at startup.
Also lock `aws-lc-fips-sys v0.14.4` in `Cargo.lock`, allow the `OpenSSL`
license on `aws-lc-fips-sys` in `deny.toml`, and update
`docs/operations/kms-cryptographic-compliance.md` to document the scope
of `-p rustfs --features fips` vs. `rustfs-crypto`.
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Characterize how scanner metadata decoding and object listing handle a leading free version and a later delete marker.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): fuse multipart commit, cache active upload heads, and scale fsync workers (#9077)
- Cache active multipart upload heads, committed part records, and per-part
transaction state in ACTIVE_MULTIPART_SOURCE_HEADS with invalidation on
mutation, retirement, abort, and delete paths.
- Fuse destination metadata preparation into MultipartFusedCommitOutcome,
coalesce prepared metadata writes before sync_data(), and gate direct
local part transaction settlement on disk.is_local().
- Retain quorum commit receipt publication before physical delete cleanup
using PREPARED_DELETE_RECEIPTS, batch TARGET and PREPARED syncs, and skip
redundant pre-reservation directory syncs for inline deletes.
- Scale DiskBlockingLane::Fsync to 8 workers per disk with idle_workers
accounting and bounded per-disk batch fairness so concurrent fsyncs
coalesce in the filesystem journal.
- Cache verified small stripe proof records in VERIFIED_STRIPE_RECORDS while
recomputing payload digests on every read.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): align test/prod commit and cache paths and hold source lock across cache publication
* fix(ecstore): preserve fused commit cancellation/panic semantics and retire excess idle fsync workers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
Replace the immediate post-replacement leader identity comparison with a
polling stabilization loop (up to 20 x 500ms). A leader-lock refresh gap
during old-binary replacement can make leader() briefly observe a different
node; the loop waits for the measured new coordinator to be confirmed. A
persistent change to an old-binary node still fails with the observed
identity in the error message.
Removes the redundant immediate assertion that the stabilization loop
supersedes. Also consolidates the non-mixed leader/pid resolution to avoid
a double leader query.
backlog#2918 retires the usage floor as a quota proof. After dual-corruption
recovery has been attempted, an unreadable floor is reported and rebuilt
through a CAS overwrite with a bootstrap marker rather than stopping the
leader iteration for a supervisor retry. Transient I/O errors that prevent
reading the floor revision fall through to the existing stop-and-retry path.
The identity-less floor regression from rustfs/rustfs#8670 now verifies
automatic rebuild and recovery instead of bounded backoff.
When sequential test fixtures share the bootstrap InstanceContext, the
BucketMetadataSys stores a Weak<ECStore> to avoid a reference cycle. The
ExpiryState holds an Arc<ECStore> that keeps the store alive past the
test's own Arc drop; the next fixture's resize_workers releases it and
the weak goes dangling before init_bucket_metadata_sys installs a fresh
BucketMetadataSys. Any object_store() call in that window panicked.
Add Drop for ECStore that clears the context's BucketMetadataSys slot
when its weak handle is stale, so the next fixture always finds a clean
slot. Fixes the heal absence_receipt_regressions and dangling_grace
test failures under sequential execution.
cxymds