* fix(ecstore,heal,select): bound storage-layer work driven by client input
Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:
- Multipart: upload ids are canonicalized before they name a staging path
or a lock key (backlog#3027); trailing-slash keys are encoded like every
other handler (backlog#3055); unlisted parts are removed at completion
(backlog#3106); per-bucket uploads are found through a signed per-disk
index instead of a scan of the shared staging area (backlog#3121);
encrypted part ETags carry a per-upload secret so a write-only user
cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
(backlog#3038); a delete marker on a key whose xl.meta path is a
directory, and sidecar names used as child keys, no longer wedge the
journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
counts delete markers against max-keys (backlog#3108), and rejects a
marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
bucket is skipped instead of failing the rebuild (backlog#3040), and
directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
modification time (backlog#3073, backlog#3074); Snowball transport
headers are not forwarded to the tier (backlog#3090); an epoch
modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
(backlog#3131); SelectObjectContent releases its namespace locks once
the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
(backlog#3129); a bucket-metadata reload timeout no longer marks the
peer offline (backlog#3124); replication target requests carry an
operation deadline (backlog#3102); the scanner resumes its interrupted
wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
batches, Parquet dictionary bounds and pre-planner projection checks
(backlog#3075, backlog#3076, backlog#3077, backlog#3078).
* test(e2e): seed legacy duplicate bucket tags offline for the OPA contract
* fix(s3): authorize what handlers store and bound request-driven work
Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.
Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
the tag set it copies (backlog#3034); HeadObject with the replication
check header still authorizes s3:GetObject (backlog#3047); the
x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
recursive force-delete with a REPLICA header authorizes every
descendant as a delete (backlog#3070); include-deleted listings require
s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
table isolation and uses the same prefix it authorizes (backlog#3053,
backlog#3054); If-Match probing needs read permission (backlog#3097);
Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
returns lock and tag headers only with permission (backlog#3113);
IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
(backlog#3049); bucket policies are parsed once and cached
(backlog#3032); keystone: access keys no longer verify with an empty
secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
(backlog#3023); PutObject takes buffer and foreground admission only
after body bytes arrive and bounds the wait (backlog#3031,
backlog#3084); xl.meta size is bounded on the write path
(backlog#3100); XML compat layers stream instead of buffering whole
bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
completion (backlog#3130); MinIO admin decryption runs Argon2id off the
async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
(backlog#3092, backlog#3093); control characters in keys and lifecycle
rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
Content-MD5 and admin/STS reject SigV2 replays with altered query or
body (backlog#3037, backlog#3026); request logs redact presigned
queries and handler spans skip credentials (backlog#3068,
backlog#3107); SSE-C listings and completions withhold plaintext
metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
keys (backlog#3126); notifications fire after a client disconnect
(backlog#3127); ListenNotification streams re-check access
(backlog#3119).
* fix(api): drop imports left unused by the base merge
* test(s3): move the KMS multipart denial test to the app layer
The test drives app-layer multipart use cases, so it belongs with them;
living in the SSE module it imported app code from the infra layer and
failed the layer dependency guard. The SSE test helpers it borrows are
exposed through the storage and app facades.
* fix(s3): keep the SSE-C key fingerprint on keyless multipart completion
The backlog#3050 fix withheld both the plaintext checksums and the
customer-key MD5 header from a CompleteMultipartUpload that did not
present the key. The checksums are a digest of the plaintext and stay
hidden; the key MD5 identifies the key only, and the S3 conformance suite
(test_copy_part_enc) expects it on a keyless completion, so it is echoed
again.
* fix(tier): stream GET bodies and document replication budgets
* fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes
Two lifecycle regressions surfaced once the storage and request batches
ran together against main:
- The multipart index owner cloned the disk's pinned publication root and
bootstrapped on a task of its own. That kept the mount alive past the
last LocalDisk owner (the disk lifecycle allows exactly three holders of
the root FD) and left a stray task behind, which the local-disk and
read-repair tests detect. The owner now keeps only the root path and
reopens it for identity checks, and the bootstrap scan rides the disk's
existing cleanup task, where stopping the disk abandons it and a
reconnect resumes it.
- The disk-read admission wrapper defers releasing its permit until an
in-flight read completes, so a cancelled client cannot widen disk
concurrency. The cold-fill producer, however, drops a read it has
already classified as stalled when its deadline fires, and that permit
stayed pinned for the lifetime of the stuck read. The producer now
abandons the admission explicitly on timeout.
* test(ecstore): pass the index owner to the Linux cleanup-loop test
* fix(s3): release cold-fill admission after watchdog handoff
* fix(s3,ecstore): close the round-1 adversarial review findings
Adversarial review of the combined request- and storage-layer batch found
fixes that fell short of their issues, two behaviour regressions for
legitimate clients and several unrequested costs:
- backlog#3038: a bulk DeleteObjects reserved the whole per-disk
inspection allowance up front for every key in flight and exhausted the
16 Mi request budget on any set wider than eight drives, failing healthy
keys with SlowDown. Calls are now admitted while budget remains and
charged for the entries they actually visited. Object-scoped discovery
charges plain child keys at one entry per 64 names instead of one each,
so deleting a missing key that names a folder prefix completes instead
of answering 503 forever while a hostile prefix stays bounded. A peer
still on the previous binary is reported
as an unsupported participant, the repository's rolling-upgrade
contract, instead of as a budget verdict.
- backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was
still replayable with another body; it is refused with MissingContentMD5
while SigV4 keeps the optional-MD5 contract.
- backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before
authorization like PUT and COPY, so a replica delete cannot remove a
version the policy never saw.
- backlog#3121: the multipart index no longer fsyncs a marker and four
directories per multipart metadata write (it is rebuilt on every start),
and ListMultipartUploads discovers roots concurrently.
- backlog#3077: Parquet SelectObjectContent decodes budget-sized batches
instead of one row per batch.
- backlog#3031: a full upload buffer pool hands out an unpooled buffer
instead of refusing the upload; the progress floor already bounds stalls.
- backlog#3023: the GET disk-read idle cut-off is configurable
(RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused
client is not truncated after 30 s.
- backlog#3075: JSON Lines schema inference keeps the sample collected at
the 4 MiB bound instead of failing the query.
- backlog#3024: compat admin decryption queues briefly for a slot instead
of refusing the third concurrent request.
- backlog#3113: HEAD skips its permission probes for the bucket owner.
- backlog#3117: control characters in a lifecycle rule ID are rejected on
PutBucketLifecycle as well.
The lock-crate publication FIFO serves backlog#3111 and moves to the
table-catalog PR. Also fixed: a lossy admin error message, an unused
s3-client function, two unused test-util features, a duplicated key
validation, the s3s footprint baseline, and a stale log event name in the
reverse-proxy runbook. The ODM multipart write-back test runs on the large
test stack like its sibling.
* fix(table-catalog): bound maintenance work and isolate table namespaces (#9174)
* fix(table-catalog): bound maintenance work and isolate table namespaces
Scan findings in the Iceberg table catalog:
- Metadata maintenance deleted every unreferenced *.metadata.json in a
table's directory without checking whether a renamed table in another
namespace still owned those files (backlog#3060); it now resolves the
owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
writer schema and no recursion depth limit; a self-referencing record
schema overflowed the stack (backlog#3080). Schemas are validated for
depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
memory; a decompression bomb OOM-killed the server (backlog#3085). Row
groups are streamed through a byte budget and oversized inputs are
refused.
- An S3 request for a key outside an indexed table prefix listed the
bucket's whole namespaces/ catalog prefix and failed once it held more
than 4096 objects (backlog#3086); a per-bucket directory index answers
index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
left the bucket's catalog entry locked at 503 (backlog#3088); the rename
runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
starving every object write (backlog#3111); commits now hold it under a
deadline and object writes are admitted between commits.
The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.
The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.
* fix(table-catalog): close the review gaps in the scan-fix batch
Adversarial review of the table-catalog batch found the fixes narrower
than the issues they close and two cost regressions:
- backlog#3085: the compaction footer check bounded nothing a page header
could not override; parquet sizes its page buffers from the header.
Walk every column chunk's page headers (Thrift compact, no decompression)
and hold each page and the chunk total to the budget before decoding.
The stored-input budget is per rewrite group, so a multi-group job no
longer fails after writing earlier outputs.
- backlog#3086: the default index-miss scan still counted maintenance
reports toward its 4096-object limit; only table entries count now, and
the listing itself is bounded to 64 pages. The directory index remains
an optional accelerator.
- backlog#3111: a create wrote its metadata file under the fence as the
first durable mutation, which disarmed the deadline before the unbounded
registration inventory. The write precedes publication now. The deadline
gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS,
default 30 s, never below the lock-acquire timeout) instead of reusing
the 5 s lock timeout, and the ownership inventory reads entries
concurrently.
- Directory candidates are written only while a directory exists; the
backfill covers earlier entries. A directory on the migration-fence
path is reported as a conflict instead of a parse error.
- Dead test hooks, a stale comment and the support-matrix text that did
not match the code are removed or corrected.
* fix(select): preserve complete samples and Parquet batch limits
* fix(s3,ecstore): close the round-2 adversarial review findings
DeleteObject resolved x-rustfs-source-version-id for authorization on
every request while storage consumes it only for a REPLICA request, so a
non-replica caller with s3:DeleteObjectVersion could trigger a plain
delete of the current version. The hook now resolves the header under
the same predicate as storage, and the regression test covers the
non-replica case.
HEAD no longer skips its permission probes for the bucket owner; an
explicit bucket-policy Deny on the metadata actions binds the owner too.
Multipart index pages verify each root against the staging area and
prune markers whose staging root is gone, under the marker's own lease
with a re-check, so the per-drive index stays proportional to live
uploads instead of every key ever multipart-initiated on the drive.
SigV2 configuration PUTs that declare an empty body need no Content-MD5:
there is no body to replay. The minimum-progress floor stays independent
of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now
says so instead of claiming 0 disables it. The quota bootstrap window
(503 until the background scan publishes the ledger) is kept, because
ledger admission must not come from scanner reporting; the compatibility
matrix now documents it.
The S3 compatibility matrix gains a section for the client-visible
validation bounds, and the drive-timeout tuning guide tabulates
RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS.
* fix(lock): queue table-bucket publication writers fairly
Table commits take the bucket publication fence as a write lock; under a
stream of commits the lock's try-and-retry acquisition let a new commit
jump ahead of object writers that had been waiting longer, which is part
of what starved them (backlog#3111). Requests for the publication lock
path now take a FIFO ticket per shard: a waiter is admitted only when it
is at the head of the queue, cancelled waiters release their turn, and
batch acquisitions join the same queue. Other lock paths are unchanged.
This moved here from the request-layer batch, where no cited issue
covered it; it serves the table-catalog issue this PR closes.
* fix(table-catalog): close the round-2 adversarial review findings
Parquet page-header walker: read the size fields by field id as parquet
does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so
a header encoding uncompressed_page_size as I64 can no longer bypass the
budget while parquet allocates it. Bound every list, set and map count by
the bytes remaining in the chunk, and reject bool collection elements
(parquet skips them with zero width, so either width would desynchronize
the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>.
Start each column-chunk walk at the dictionary page offset whenever the
footer declares one, exactly as parquet's byte_range does, instead of
min(dictionary, data).
Footer pre-walk: before parquet decodes a footer, walk FileMetaData with
the hardened cursor and charge the declared row_groups count times the
RowGroupMetaData size to the decode budget, because parquet 59 reserves
that Vec before reading one element.
Preparation deadline: clamp the
RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400
seconds so Instant + Duration cannot overflow.
Index-miss scan: after each listing page, jump the cursor past the
maintenance subtree the page ended in, and allow one extra page per table
entry already listed, so job-report history scales with the catalog while
a report flood under one table stays bounded.
Migration fence: read the fence-path object once and decide from its
stored kind whether it is directory state or a fence.
Expired or failed create: remove the initial metadata file when the
registration fails before any catalog write may have begun; nothing else
references it. Correct the comment and support matrix that claimed
maintenance reclaims it.
Tests cover each walker divergence, the footer row-group reservation, the
deadline ceiling, the paging budget for honest catalogs and for a report
flood, and the removal of the unpublished metadata file.
* test(admin): register the catalog directory routes in the route matrix
The table-catalog batch added GET, POST and DELETE
{warehouse}/catalog/directory on both the Iceberg and the compat admin
prefixes without listing them in the admin route matrix, so
test_admin_route_matrix_matches_registered_routes failed on CI.
* fix(s3,ecstore): close the round-3 adversarial review findings
Multipart index pages now treat the cursor as the last examined marker
rather than the last root, so a page that pruned a stale marker still
validates and ListMultipartUploads keeps using the index. A marker is
written before its staging root is published, so pruning spares markers
younger than a five-minute grace period and an existing marker is
touched on every new upload; the prune re-checks both under the marker
lease. Bounded delete-recovery inspection rides the durable-mutation
lane like the unbounded path instead of the one-worker foreground lane.
The on-demand-migration inline body budget scales with the object size
(at least 64 KiB/s) instead of a flat 120 s. The control-body digest
service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no
inactivity timeout" like every other reader of that knob.
Quota reconciliation throttle objects are deleted together with a
retired generation's ledger, so bucket turnover leaves no residue. The PUT guard
test shim is gone; tests call the shared guard. Bounded DiskAPI defaults
answer MethodNotAllowed like their siblings. Stale
budget docs, an unreachable CSV arm, an always-Some limit parameter,
commented-out constants and a misplaced doc comment are cleaned up;
weak test assertions name the expected error; lifecycle rule-ID control
characters gain a test. The compatibility matrix records the CopyObject
tag-copy permission, the exact SigV2 empty-body predicate, replica and
migration writes of control-character keys, and the SigV2 DeleteObjects
digest rule.
* fix(table-catalog,heal): close the round-3 review findings
A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job,
TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before
any output is written, and name the ceiling in the commit-path errors;
document the effective cap in the support matrix and scheduler runbook.
B. Keep TaskCancelled/TaskTimeout classification on replacement passes:
the namespace verification runs only for completed or storage-failed
passes, in the erasure healer and the task bucket prepass.
C. Compare the replacement bucket pass per bucket: a recorded identity
that is gone or replaced still errors or restarts the pass, buckets
created mid-pass are appended to the identities, pending_buckets and a
persisted positional order without rewinding progress or the
checkpoint; verification ignores additions.
D. Read the migration-fence object once in the object-backed write permit
and dispatch on its kind.
E. Drop the strong store's copy of maintenance_cleanup_roots.
F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and
pause_after_metadata; make compact_parquet_data_files and
PublicationTicket.state private.
G. Document the body-revision identity, the legacy revalidation budget
and its HTTP 503 in the support matrix.
H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it
from heal through the storage_api facade.
* fix(ecstore): cap the index pages one disk may answer
Pruned multipart index pages may carry few roots, so the page cursor no
longer bounds how many pages a peer can make a listing fetch. Cut a disk
off after 4096 pages (about one million examined markers) so it
contributes no coverage and the listing falls back to the staging scan.
* fix(ecstore,table-catalog): close the round-4 adversarial review findings
Multipart index markers dated in the future (a backwards clock step
between recording a marker and publishing its staging root) were judged
old and could be pruned while their upload was in flight; a future-dated
marker is now kept. The marker touch on EEXIST is best effort, so a
filesystem that refuses to set times no longer fails the upload commit.
Pruning runs on the durable-mutation lane instead of the one-worker
foreground lane.
Quota reconciliation keeps one throttle object per bucket incarnation (a
quota revision change replaces it instead of orphaning the previous
revision's object), the retirement cleanup deletes it ahead of the ledger
and also when the ledger is already gone, and a scan the retirement
already refused writes nothing back. Bounded DiskAPI defaults answer
UnsupportedDisk, the verdict the aggregate gives a peer without the
protocol. The compaction ceiling pre-check runs over every rewrite group
before any output is written, and the worker's ceiling error no longer
names a placeholder group.
Docs: the inline on-demand-migration client budget, the protocol-2
downgrade conflict of replacement passes, the exact footprint numbers.
FileMetaShallowVersion::into_fileinfo and to_fileinfo_with_part_checksums previously decoded an owned FileMetaVersion and then converted it via shared reference (&self), cloning meta_user, meta_sys, part_etags, and part_indices before dropping the temporary FileMetaVersion and scanning the metadata map up to 10 times for optional internal suffixes. Additionally, metadata_compat helpers (get_str, contains_key_str, remove_str, get_consistent_value) allocated two temporary heap Strings via both_keys(suffix) on every absent-key fallback.
- Replace both_keys(suffix) in case-insensitive fallback scans with allocation-free prefix+suffix slice matching.
- Add consuming into_fileinfo_with_part_checksums on FileMetaVersion and MetaObject to move decoded part vectors and metadata maps in place.
- Fold transition-version metadata formatting and optional suffix presence detection into the single meta_sys pass, and skip target_delete_marker_versions in get_internal_replication_state when no delete-marker version keys are present.
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs
`get_volume_name` returned the volume root truncated at its NUL and the
result was then passed to `GetVolumeInformationW` as a PCWSTR, an
out-of-bounds read. The disk-space and volume APIs also received the
caller's raw path, so they failed beyond MAX_PATH, and a UNC share root
lacked the trailing backslash these APIs require.
- Wrap every Win32 path in a NUL-terminated `WidePath` that rejects
interior NULs; volume roots returned by `GetVolumePathNameW` keep
their terminator.
- Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after
`GetFullPathNameW` normalization, and give directory queries a
trailing backslash.
- Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume
root as documented; size buffers to the documented MAX_PATH + 1 or
the input length.
- Report Win32 error codes instead of raw HRESULTs so `ErrorKind`
matches `std::fs`.
- Keep volume ids in their conventional `C:\` / `\\server\share\` form.
Add Windows-only `os::` tests for long directories, UNC share roots and
non-empty fstype, and run them in the Windows filesystem workflow.
* fix(utils): report Windows network volumes to the fs guard
Windows names the server's filesystem (usually NTFS) for SMB shares, so
the startup unsupported-filesystem guard, which matches fstype names
such as cifs or smb2, never fired for UNC paths or mapped drives.
- Add `DiskInfo::remote`. On Windows it is set when the directory handle
answers `FileRemoteProtocolInfo` (only network redirectors do, and the
handle follows symlinks and junctions to their target) or the volume
root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because
the client fstype already names network mounts.
- Treat remote volumes as unsupported in the startup guard and label
them "<fstype> on network share".
- Cover UNC shares, paths beyond MAX_PATH below a share, a directory
symlink to a share and a `net use` mapped drive with Windows-only
`os::` tests that skip when `\\localhost\<drive>$` is unreachable;
the Windows filesystem workflow maps the drive before running them.
* chore(deps): replace yanked inout 0.2.3 with 0.2.4
cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump.
* fix(utils): detect network targets from the final NT path
On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive.
Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): share metadata cache identity across Windows key aliases
Windows disks treat `\` as a path separator and `path_join_buf` drops the
empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all
address the xl.meta of `a/b`. The GetObject metadata cache keyed and
hashed the raw spelling, so DELETE `a\b` retired nothing cached for
`a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the
2s cache TTL.
Fold the object name into the same segment form the disk resolves to
before computing the cache hash, key and lookup key. Off Windows the
name stays verbatim and borrowed, so `\` remains an ordinary key byte.
* fix(ecstore): serialize Windows backslash key aliases on one namespace lock
Windows disks treat `\` as a path separator and drop the empty segments
it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta
of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and
heals through different spellings ran concurrently on one xl.meta and a
read-modify-write in rename_data could drop a version.
Fold the object name into its disk identity (rustfs_utils::path::
object_name_disk_identity, a no-op off Windows) wherever a namespace
lock key is formed:
- SetDisks::new_ns_lock, which every object lock path goes through;
- the batch delete lock request and its dedup/lookups, which build
ObjectKeys directly.
Two consumers compared lock identities by spelling and follow along:
- WriteCommitGuard records the folded key from the lock resource, so
matching a borrowed guard now folds the target name too;
- CopyObject holds the source read guard until EOF on streaming reads.
`a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares
one lock, so it now takes the self-copy lock instead of waiting on
itself until the lock timeout.
Distributed lock keys are folded by the requesting node only; lock
servers compare them verbatim.
* refactor(ecstore): key the metadata cache by the shared disk identity
The metadata cache fix carried its own copy of the Windows backslash
folding rule. Use rustfs_utils::path::object_name_disk_identity, which
namespace locks already use, so cache and lock identities cannot drift.
* ci(windows): run filesystem tests when key identity rules change
The Windows backslash folding for object names lives in utils path.rs
and is consumed by the namespace lock and write commit guard; run the
windows_ tests when any of them changes.
* fix(ecstore): keep Windows disk roots out of verbatim form
The disk root comes from GetFinalPathNameByHandleW, which returns a
verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a
data directory past 260 characters (or one reached through a UNC share)
kept the prefix. Disk IO extends the root by joining `/`-separated
relative paths as strings (`read_raw` builds `<object>/xl.meta` with
`path_join`), and inside a verbatim path `/` is not a separator: the
first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME
until startup gave up.
Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim
prefixes without a length limit while still keeping paths whose
components Win32 would rewrite, and use it for the publication root.
std re-adds the extended-length prefix for long paths.
Cover a root past MAX_PATH with a disk-level read/write test and run all
`dunce::` tests in the Windows filesystem workflow.
* fix(ecstore): query Win32 volume APIs with extended long disk roots
A disk root past the legacy limit used to reach `rustfs_utils::os` in
verbatim form, which the Win32 volume APIs accept. Now that the root
stays non-verbatim, GetDiskFreeSpaceExW rejected it with
ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem
workflow. Pass long roots to `get_info` and `get_physical_device_ids` in
extended-length form; shorter roots stay unchanged so reported volume
identities keep their `C:\` form.
* fix(ecstore): admit CopyObject destinations like PutObject
Object-key platform checks were split across call sites with gaps:
CopyObject only ran check_bucket_and_object_names on its destination and
then reached the pools' put_object directly, so it admitted keys PutObject
rejects (over 1024 bytes, leading slash, and on Windows reserved characters
or unaddressable segments, the "writes but cannot be read" case of #3449).
- rustfs-utils: expose check_windows_file_name as the single per-segment
Win32 rule set; dunce's simplified() reuses it. Reserved device names now
include COM/LPT with superscript digits and CONIN$/CONOUT$.
- ecstore: replace the duplicated reserved-name table with that function;
Windows also rejects control characters 0x01-0x1F (NUL keeps its
platform-independent ObjectNameInvalid answer).
- ecstore: add check_new_object_args as the shared admission entry for
PutObject, cross-key CopyObject destinations and CreateMultipartUpload.
In-place copies (metadata replace, restore) keep the copy-only check so
existing keys stay rewritable. CreateMultipartUpload now also rejects
over-NAME_MAX segments up front instead of failing at completion.
- CI: run bucket::utils and dunce tests in the Windows filesystem job.
* fix(ecstore): reject new backslash keys on Windows hosts
A Windows disk treats '\' as a path separator, so a new key `a\b` lands on
the same xl.meta as `a/b` and the two keys overwrite each other.
Reject '\' only where a key enters the namespace (check_new_object_args:
PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and
only on Windows. Reads, deletes, in-flight multipart operations, in-place
copies, data movement and internal meta buckets keep accepting it, so keys
already stored with '\' stay readable, deletable and movable.
Pending owner ruling: whether new '\' keys should be rejected at all, and
whether reads/deletes should keep accepting them.
* ci(windows): quote filtered test runs and drop duplicate path
tempfile's Windows `persist` passes the raw paths to MoveFileExW and
SetFileAttributesW, bypassing std's extended-length handling. Once
`<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete
of that key failed with 500, while PUT and GET kept working.
Add `rustfs_utils::extended_length_path`, which normalizes a path with
GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it
for the staging directory and destination handed to tempfile. Cover the
single and batch delete paths with a deep-key regression test and run the
helper's Windows assertions in the Windows filesystem workflow.
Fixes#8545
* fix(iam): require an explicit permission for force-delete
A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): keep force-delete header names static
The bucket force-delete helper must pass a static header name into the
SDK request mutator.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): move the force-delete header into the request mutator
The SDK request customizer requires a static header name owned by the
closure.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* fix(iam): keep force-delete out of NotAction grants
NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.
Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.
---------
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Record the upload id on the completed object so a lost-response retry
returns that object's ETag instead of NoSuchUpload, while a different
part list or a replaced object keeps the existing errors.
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
This is particularly useful for integration testing of services that use
an S3 API, as it lets many different instances bind a port on different
IPs.
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(replication): resolve drifted replicas via a target version ledger
A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).
Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.
The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.
* fix(scanner): drop the unused Digest import
Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.
* fix(admin): probe replication-check mutations by the assigned version id (#7373)
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names (#7377)
* fix(admin): probe replication-check mutations by the assigned version id
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names
A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).
Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.
* fix(admin): send replication-check marker creation without a version id
Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:
- the DeleteMarker phase put the assigned version id on its DELETE. A
RustFS peer reads the source-deletemarker header and creates a marker,
but a generic S3 target executes it as a permanent delete of the probe
version, so VersionDelete then answered NoSuchVersion. Use the same wire
shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
already removed as a failure (RustFS/MinIO answer 204 there).
Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): drop the unused Digest import
* fix(replication): carry the compression layout through SSE-C passthrough (#7372)
* fix(replication): queue an in-flight version only once (#7376)
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1823 step 10, batch 1 of the repo-wide item-allow sweep. 227 bare #[allow(dead_code)] remain across 83 files; this takes the 19 in utils, notify, checksums, policy, keystone and trusted-proxies, which are small enough to verify end to end.
Removing all 19 first, before writing any reason, matters: 8 of them suppress nothing. Every allow in utils, one in policy and three in notify sit on items that are publicly reachable, so dead_code never applied to them — the same shape as the swift module and kms's dek.rs. Writing a reason onto a no-op allow would dress noise up as considered judgement, so those are simply deleted.
Three items are genuinely dead and go with their allows: notify's new_target_id_set, the AWS metadata fetcher's get_metadata_token, and policy's empty `pub struct Value;`, none of which is referenced anywhere in the tree.
The remaining eight keep an allow, now saying why the item survives rather than who calls it. Two are exercised only by their own crate's tests (checksums' MD5_HEADER_NAME, policy's is_match_as_pattern_prefix). Four are fields written but never read back: keystone's verify_ssl, parsed from config after the reqwest client is already built; keystone's client handle, which keeps the Keystone client alive for the mapper's lifetime; the AWS IMDS endpoint, kept beside the client while requests build their own URLs; and notify's rules_map, whose own comment retains it for snapshot-time judgements no code performs.
checksums' Md5 needed the most care. Crc32, Sha256 and seven others each have an arm in ChecksumAlgorithm::into_impl, and Md5 has none, which reads like a missing algorithm. It is not: ChecksumAlgorithm has no Md5 variant at all. S3 carries Content-MD5 as its own header, separate from the x-amz-checksum-* family, and this impl exists so both paths share the Checksum trait. The reason records that, so the next reader does not re-derive it.
One measurement note for anyone continuing this sweep: cargo does not re-emit warnings for cached compilations, so a per-crate loop of `cargo check -p <crate>` under-reports. checksums showed zero that way while actually carrying three. Touch the sources and check the crates in one invocation, then attribute by path.
Verification: the six crates are warning-free under cargo check --tests; clippy --lib --tests -D warnings clean; cargo nextest run 1096 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 10).
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
The policy crate's Go path.Clean port and rustfs-utils' Windows-aware clean look like duplicates but are not interchangeable: S3 ARN/resource matching must treat backslashes as object-name data, never as separators, so adopting the utils version would change policy evaluation semantics on Windows — a security-adjacent behavior change. Record that judgment as bidirectional do-not-merge notes on both implementations, per the issue's adversarial ruling.
Comment-only change.
Ref rustfs/backlog#1833 (PR7).
The shared module rustfs_utils::http::object_encryption_keys is the single source of truth for encryption metadata key names, but three call sites still carried their own copies or bare literals: crates/kms/src/service.rs (two private constants plus four bare x-rustfs-encryption-* literals on both the write and read path), rustfs/src/app/select_object.rs (six SELECT_* copies), and rustfs/src/storage/options.rs (two private prefix copies now imported from header_compat). All values are unchanged, so the change is a compiler-verified rename.
The reader-only x-rustfs-internal-server-side-encryption- family gets a named constant with the verified judgment recorded on it: no writer emits these keys anywhere in the repo (the SSE writer persists the MinIO-branded keys verbatim for interop), the two comments claiming the dual-key invariant writes this twin were wrong and are corrected, and the defensive redaction/strip readers are kept because removing them is risk-asymmetric.
rustfs-kms's rustfs-utils dependency now declares the http feature it uses instead of relying on feature unification from sibling crates.
Refs rustfs/backlog#1775, rustfs/backlog#1562.
backlog#1833 PR1 prescribed deduplicating crates/replication/src/http.rs onto the canonical rustfs-utils http modules via a re-export facade. That plan conflicts with a standing architecture guard the issue's review missed: check_architecture_migration_rules.sh rejects any rustfs-utils import or dependency from the replication crate ("replication crate HTTP/helper contracts must not import or depend on rustfs-utils"), the same way it bans rustfs-filemeta and rustfs-storage-api — the wire-contract crate deliberately has zero internal dependencies.
So this lands the issue's fallback shape instead (the same bidirectional do-not-merge pattern the issue itself prescribes for the policy path.rs cluster): a module doc on replication/http.rs naming the canonical owners and the guard that forces the local copy, mirror notes on utils' metadata_compat.rs and header_compat.rs, and a new test pinning every duplicated constant to its literal wire value so the two copies cannot drift silently.
No production code changed.
Ref rustfs/backlog#1833 (PR1).
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
Groundwork for encrypted-object replication (backlog#1783, PR-A of 3):
- classify_replication_source_encryption: accept the AES256 marker that
every stored SSE-C object carries; the SseC arm was unreachable.
- Fail closed on sealed material without an SSE marker (MinIO-written
objects) instead of replicating ciphertext as plaintext.
- Replace the dead VALID_SSE_REPLICATION_HEADERS table with a transport
map keyed by the metadata keys the SSE writer actually persists, shared
via the new rustfs_utils::http::object_encryption_keys module.
- Structurally strip all encryption metadata from outbound replication
(x-rustfs-encryption-* envelopes previously passed the filters).
- Skip decrypt_checksums for encrypted objects at the boundary so its
is_multipart=false (a response-path contract) cannot misroute
encrypted multipart objects once managed replication opens.
- Redact X-Rustfs-Replication-* SSE transport values in FileInfo Debug.
A reconciliation test pins that every key encryption_material_to_metadata
produces is either transport-mapped or stripped. All four SSE replication
e2e contracts still assert FAILED unchanged.
Chris An