Build the inventory only for coordinator-owned cross-drive healing and expose the regression marker directly through the scan storage domain.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Avoid building and cloning the cross-drive inventory for single-drive scans and scans with healing suppressed. Exercise durable usage publication and cursor advancement through the runtime cycle.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(http): send deferred HTTP/2 100 Continue
Backport upstream Expect handling onto Hyper 1.12.0 and cover the transport handshake and rejection boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(http): use uppercase Expect token in fixture
Preserve case-insensitive Expect coverage without triggering the spelling guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(http): compact deferred Continue body state
Preserve the existing body size bound and H2 frame path while removing redundant state from the temporary Expect allocation. Cover spawned readers, size hints, trailers, and EOF.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(http): align Continue backport with upstream
Remove the local body-layout micro-optimization so the runtime patch exactly matches upstream Hyper PR #4217 while retaining RustFS regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(storage): keep authorization tag lookup in infra
Move the store-backed ExistingObjectTag lookup behind the authorization core and keep legacy error conversion at the FS interface boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tests): align CI fixtures with current contracts
Distinguish AddUser primary and secondary admin gates, parse SigV4 civil time in UTC, and provide the required Object Lock snapshot for restore metadata copies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix(ecstore): do not republish pool.bin for an unreachable drive
The pool.bin heal treated every drive whose state was not "ok" as a drive
to repair, offline and faulty drives included. A write cannot reach those
drives: while one was unreachable, each read repair republished pool.bin
to the reachable drives only, failed with SystemMetadataRepairRequired and
was retried, writing a new version every time. The drive came back one
version behind, while every node reported it ready.
In a set of three drives at EC:1, losing another drive in that window
left one current and one stale replica: pool.bin had no read quorum, the
node with the replaced drive could not start, and neither could any other
node that restarted.
Repair pool.bin only for a reachable drive that lacks the current copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ecstore): require replica damage before pool metadata heal
A timed-out or otherwise inconclusive metadata probe can leave a valid
pool.bin replica behind. Require confirmed missing, stale, or corrupt
metadata before publishing a new generation through the existing fence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* style(admin): remove trailing whitespace in identity test
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(rustfs): restore gateway builds and clear CI lints
Backport 142cff57a and 2e286d322 from main. Reuse the admitted
request headers for POST archive controls and return the existing
S3 error type directly at both call sites.
Remove the unused form metadata projection left by the same merge
and keep the gateway policy test on the production field path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Regular parents now fence STS sessions by durable generation, so a session
stored without one is rejected as an unknown key. Publish the test sessions
with set_temp_user_from_parent, as the STS handler does.
The three UploadPart timeout tests drove the early-response drain around
a hand-written s3s S3 service. They now drive it around the gateway
stack as production configures it (GatewayStack::assemble_serving, a
test-only constructor over the same configured builder), serving an
UploadPart handler that reads its whole body as RustFS's handler does.
Every assertion is kept: 400 RequestTimeout, the HTTP/1 transport
retained for the drain and drained of the late bytes, the HTTP/2 body
dropped alone, and other HTTP/2 and HTTP/3 streams still served.
The gateway times a stalled body out with its own request-body idle
bound, so that bound is what the fixture sets; it does not follow a
paused tokio clock, so the HTTP/1-and-HTTP/2 case runs in real time
(200 ms bound) instead of advancing a paused clock.
Kept as written, the HTTP/2 assertion failed: GatewayStack::call added
`Connection: close` to any answer whose connection intent is to close,
HTTP/2 included, where the header is forbidden and only the stream
ends. It is now added to HTTP/1 answers only.
The retired legacy metadata route carried the website-host and metadata=true listing case tables, and a test that its authorization attaches the table listing snapshot. Re-create both against the gateway stack, observing the listing a caller receives, and probe the native admin and control-plane hosts for the unsigned x-amz header refusal.
The gateway stack was called through rustfs-gateway's tower adapter. Under the RustFS profile
that adapter wraps the request body, takes it back when the handler drops it unread, and reads
it in a task of its own spawned on the ambient runtime (unread_body.rs, Retention::settle). That
task took the body away from EarlyResponseBodyService before its abandon slot could see it, so
RustFS's own drain never ran: its task was no longer spawned under the HTTP/1-2 task owner,
graceful shutdown could report its children retired while a detached drain still held the
transport body, and the early-response drain metric and log went silent. On origin/main the
legacy stack was the default assembly, so the listener test never crossed the adapter.
GatewayStack::call now calls the service's own entry and keeps the adapter's connection
verdict (connection_intent_of). The inert drain_unread_request_bodies setting is removed.
EarlyResponseBodyService is again the only HTTP/1 unread-body drain, with Connection: close
and the configured idle bound.
The gateway-stack test that asserted the adapter's drain timeout now asserts that the stack
releases an unread body when it answers. The configured-timeout half moves to
gateway_rejected_http1_body_uses_the_configured_drain_timeout, which composes
EarlyResponseBodyService over the real gateway stack for a 1 s and a zero timeout.
Mutation: restoring the adapter call turns all three red:
http12_actual_early_body_drain_is_owned_until_release (the drain never starts),
rejected_http1_body_is_released_to_the_host_drain and
gateway_rejected_http1_body_uses_the_configured_drain_timeout.
Neither side of dual_stack_differential is started with the retired
stack selector any more, which the binary under test now refuses: both
sides run the binary under test, or side A runs the released reference
binary named by RUSTFS_E2E_REFERENCE_BINARY. RUSTFS_E2E_STACK_B is
gone; the injected-region control is unchanged.
server/layer.rs and server/hybrid.rs no longer reference s3s and leave
the edge allowlist; the baseline ceilings drop to the measured values
(files 50, s3_error! lines 103).
SigV4HeaderGuard, S3ErrorMessageCompat, DoubleSlashListBucketsCompat,
VirtualHostStyleHint and ObjectAttributesEtagFix patched the s3s edge
and have no work left: the gateway_stack legacy_layers tests pin what
each made the legacy stack answer against the gateway stack alone. The
layer unit tests go with the layers, and the server-domain flag that
only gated VirtualHostStyleHint goes with it.
EmptyBodyContentLengthCompat keeps its admin, console and internode
halves and loses its S3 half, which the gateway stack replaces by
leaving the bodies of bodyless operations unread; its S3 tests now
assert that such requests pass through unchanged.
The S3 service under the middleware now answers in EdgeBody, a boxed
Bytes body, with a boxed error, instead of s3s::Body and
s3s::HttpError. The gateway stack, the native console and the refusals
written in server/http.rs meet in that one type; Swift and the
internode RPC lane convert into it from rustfs_s3_types::Body.
ControlBodyDigestService is generic over its inner response body, and
its control-plane XML body limit is a named constant equal to the limit
the s3s edge applied.
Remove the S3Stack selector, the legacy s3s service assembly and the
gateway-stack cargo feature: the gateway stack, its native admin and
console hosts and the control-plane host are always assembled, and the
gateway crates are ordinary dependencies. The debug-only compile_error
guard goes with the feature.
RUSTFS_S3_STACK is retired: a server or an embedded server started with
it set, to any value including gateway or an empty string, refuses to
start with an error naming the variable and this release. The message
and its tests share one constant.
Tests that ran only on the legacy stack, or selected between stacks,
are retired; tests that ran on both now run on the gateway stack alone.
The s3s metadata/website route is no longer mounted and stays only
until the metadata-route branch replaces it.
Each entry names a legacy-only test, or the unit test of an s3s patch
layer, that the flip deletes, with the gateway-stack test that keeps
its contract (rustfs/backlog#2783). The commits that delete them delete
their entries.
The gateway selection gains a website service, chosen by its host claim
before the Keystone refusal, the native console, the native admin and the
control plane, as the legacy metadata route claims those requests first.
A parity test sends both request classes to both stacks over one store.
Install the RustFS dialect rows of ListObjectsV2WithMetadata and
ListObjectVersionsWithMetadata on the S3 table, let the early gate allow
them, and register one handler that runs the shared access phase, the
listing authorization and the shared listing answer. The rows keep their
overlaps with the standard listings and each other; the admin extension
rows they also shadow live on the control-plane host.
The legacy MetadataRoute now hands website requests to an owned WebsiteHost,
which reads through the owned GetObject/HeadObject authorization and use
cases with an anonymous request envelope, and hands metadata=true listings
to an owned module that authorizes and answers them on the request envelope.
The WebsiteRead marker moves with the website rules.
The gateway PutObject path carried no hotpath measurement. Measure execute_put_object_owned under the label the use case had before the legacy edge split, and extend the contract test to require it.
The legacy-only tests for keys outside the floor and for CompleteMultipartUpload request shapes now run on both stacks. The gateway answers every NUL and over-long key with one uniform refusal before the bucket is consulted, and differs from the legacy table on six completion shapes; both are pinned explicitly.
Rename storage/ecfs_extend.rs to storage/s3_support.rs, point its request-marker
imports at storage::request_markers, and move parse_object_version_id and
ListObjectUnorderedQuery out of storage/ecfs.rs into it. The module no longer
depends on FS, S3Access or storage::access. Behaviour is unchanged.
Authenticate the actual request signing secret against the authoritative regular-owner record before reading the registration body. Carry its epoch and identity revision through publication without rebinding to a newer owner.
Reject stale peer signing secrets and concurrent owner changes, including same-secret recreation, rotation back to an old secret, revision-only rewrites, and legacy-initialization readback races. Preserve the independent STS generation and unchanged-secret upsert policy.
Add deterministic real-store and handler regressions and document the secret-reuse boundary. This is a separate correction layer for #3266 above the generation-aware integration.
Validation: cargo fmt --all --check, git diff --check 05d9825, scripts/check_doc_paths.sh, and scripts/check_no_planning_docs.sh passed. Compilation, runtime tests, and Clippy are deferred as requested.
A custom error code nobody gave a status (the tier handlers' TierNotFound) was refused as a bare InternalError, so peers matching on the code never saw it. The legacy writer answers such an error 500 with its own code and message; handler_error now does the same for every status-less custom code. Errors with an unwritable header are still refused.
The authorization core moved under storage/authorization still called the
legacy FS type for this lookup, a reverse dependency the layer guard refuses.
The lookup now lives beside its only policy caller and FS delegates to it.
* perf(ecstore): reduce allocations for bounded bitrot frames
* test(s3): cover dynamic block layouts over HTTP
* perf: measure quantized range read shape
* chore: checkpoint quantized read metrics draft
Save work in progress for collaboration. Counts omit retired readers and transient or cancelled hedges. Frame semantics, async recorder coverage and disabled-metrics overhead remain unresolved; these counters are not ready for performance conclusions.
Validation: rustfmt and ecstore check passed; io-metrics 120, quantized 4 and hashed-read 3 tests passed. No new live or performance validation.
* perf(ecstore): reduce dynamic layout read overhead
* fix(ecstore): avoid prefetching conservative short range tails
* perf(ecstore): compact deferred shard reader state
* perf(ecstore): include shard checksums in file buffer cap
* fix(ecstore): preserve bounded quantum frames during heal
* perf(ecstore): reuse owned quantized frame batches
* perf(ecstore): batch q reads and retain shared copy framing
* fix(ecstore): preserve typed quantized read errors
* perf(ecstore): avoid zero-fill and serial copies in local batch reads
The local batch reader zero-initialized every window at a nonzero offset
before reading it, and copied every shard serially inside one blocking
task. Two-segment q-window range reads (cross-shard, cross-block) pay
both costs twice per request.
Read each window with a positioned read into uninitialized Vec capacity,
and dispatch each shard separately when every window is at least 512 KiB,
where copies dominate blocking-pool round trips. Smaller windows keep one
coalesced dispatch.
* perf(ecstore): keep readahead pipelined for local batch windows
A single window-sized pread on cold pages waits on each kernel readahead
chunk in turn. Seek to the window start and fill the buffer with
read_to_end over a length-limited reader instead: it still reads into
uninitialized spare capacity, but its growing sequential reads keep
readahead pipelined. Warm windows cost the same.
* Revert "perf(ecstore): keep readahead pipelined for local batch windows"
This reverts commit c252fca525.
On the ESSD-backed cloud footprint, the growing read_to_end reads were
slower than one window-sized positioned read for cold windows at nonzero
offsets (cross-block -27% on 4 MiB layouts, cross-shard -21% on 64 KiB
layouts versus the previous batch reader). The local benchmark that
motivated the change ran on a WSL virtual disk and did not represent the
target storage. Keep the single positioned read into spare capacity and
the per-shard dispatch for large windows.
* perf(ecstore): choose the batch read method by window size
Measured on ESSD-backed cloud storage, a cold window of about 1 MiB is
fastest as one positioned read, which completes as a single device
request, while windows of 2 MiB and more are faster with growing
sequential reads that keep readahead pipelined (one large read waits on
each readahead chunk in turn). Use one positioned read up to 1.5 MiB and
a seek plus length-limited read_to_end above it. Both paths read into
uninitialized spare capacity; per-shard dispatch for large windows is
unchanged.
* perf(ecstore): read local batch windows with an exact readahead hint
Kernel readahead sizes its first request from the read size. A cold
16 KiB window at offset 0 pulled 64-128 KiB, and a single 2 MiB read was
split into serial readahead chunks that cached almost twice the window,
which is why neither the single positioned read nor growing sequential
reads won for every window size on block storage.
Announce each window with POSIX_FADV_WILLNEED, which submits exactly the
window at once, and then fill the exact spare capacity with positioned
reads. This replaces the size-based choice between the two read methods.
The hint is advisory and its failure does not fail the read. A residency
test checks that cold reads cache only their window; it failed with the
hint disabled (a 5-page window cached 32 pages).
* perf(ecstore): use half-shard read quantum at 1 MiB EC layout
* revert(release): defer certification rollout for 1.0.5 (#9254)
Revert e5563e6004 only on release/1.0.5. Its trusted verifier workflow and immutable-candidate integration are not available yet. Restore the previous publication path under the approved release policy; retain the completed product regression and preview acceptance requirements.
* fix(ecstore): preserve legacy full-shard framing for default 1 MiB layout
* perf(ecstore): enable direct-memory GET and coalesce cross-shard range reads for bounded-quantum blocks
* chore(ecstore): remove unused schedule_bitrot_reader_task wrapper
* fix(ecstore): run delete-recovery discovery off the metadata lane
Bucket-wide delete-recovery discovery starts with the server and walks the
whole bucket tree in pages of up to 4096 entries, one lstat per entry. Each
page ran on the per-disk namespace queue's foreground lane, which has a
single worker and also serves GET metadata reads. On a cold tree a page
holds that worker for hundreds of milliseconds, so every GET on the disk
waits behind it until the walk finishes.
Run the page on the queue's sync lane instead, as the delete-transaction
inspection already does for the same reason. The namespace lease and the
page bounds are unchanged.
The regression test occupies the foreground lane and requires a discovery
page to complete within ten seconds; it times out without this change.
* test(ecstore): align read_appending rollback assertions on error
* fix: align multipart and EC read paths with current workspace APIs
---------
Co-authored-by: Xiaoyang Han <lux1an@qq.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
The gateway handlers, the owned envelope entry points, the protocol client and
the admin account-permissions probe now receive the owned S3 error directly.
authorize_request_core, its entry points, the denial log and the ListBuckets and
action-potential discovery decisions return the owned S3 error type; the legacy
hooks and tests convert at the call. The two source-reading guards that looked
for the table data-plane functions in access.rs (moved by the previous slice)
now read storage/authorization/table_data_plane.rs.
A JSON comparison of two bodies that are not JSON no longer agrees, which
turns two vacuous cases into byte comparisons. The table gains unknown,
deleted and service-account keys and a repeated Authorization line, the
caller probe gains header and query session tokens, and the console
listener's isolation is pinned in front of the control-plane host.
The control-plane host ran S3Router::admit inside its handler, after the
gateway had buffered the whole body, so an anonymous chunked admin PUT
made the host hold up to 20 MiB before answering 403. The host now
attaches each request's head before the gateway reads it and decides it
by S3Router::admit at the route-authorization stage, holds at most the
legacy 1 MiB of a claimed body, and answers a repeated Authorization line
as legacy RustFS answers an anonymous caller, never serving it as one.
The gateway selection no longer holds the legacy service: claimed routes go to the control-plane host, and a selection without one answers 501. Tests drive every request class through both selections and assert neither reaches the other stack, and compare the host with the legacy stack over real IAM and signatures.
The host dispatches every admin, table-catalog, profiling, STS form, replication, listen and Object Lambda request through S3Router::admit and the existing route table, with the legacy stack's identity, body ceiling and answers. Probe tests pin which requests reach a handler and as whom.
Design (T2.11): serve every route the admin router claims on the gateway
stack without the s3s service.
- Extension point: a RustFS-owned dialect on a second gateway service, the
control-plane host. One claimed operation, rustfs:ControlPlane, takes the
six RustFS claims (admin, MinIO alias, both table-catalog prefixes, both
profiling triggers) exactly, with a trailing slash and every descendant,
for every method; a RustFS form operation takes the STS form at /; the
eight query-discriminated extension and Object Lambda rows are the
gateway dialect's own S3-table operations. No gateway change is needed.
- Identity: the gateway verifies whatever is presented (floor admits
header, presigned and anonymous; the caller secret is handed to every
operation, as s3s hands it to every custom route) and the envelope is
built with RequestEnvelope::from_gateway, the mechanism the native admin
bridge already uses.
- Extracted here: S3Router::admit, check_access's rule over owned method,
URI, headers and "signed", and router::dispatch, the route-table answer.
The legacy S3Route impl now calls both; the console and Object Lambda
branches stay with the legacy call, and an envelope-based Object Lambda
entry is added for the gateway host.
The squash of #9489 carried two hunks of an unlanded POST change that call
post_effective_headers, which main does not define, so every build with the
gateway-stack feature failed. Read the request headers as before; the POST
change reapplies cleanly on top. Also drops trailing whitespace that fails
cargo fmt --check.
* fix(ecstore): admit degraded local startup at read quorum
Keep absent local volumes in configured offline slots and admit existing native formats with nonvoting blank replacements. Preserve legacy validation and elected migration authority before degraded reads. Add startup, migration, readiness and real S3 cold-start regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): separate startup reads from metadata writers
Install committed startup metadata under a shared namespace fence when no writer work is required. Revalidate under the existing exclusive pipeline before bootstrap, recovery, repair, or topology side effects, and check fence validity at publication.
Cache existing-format read admission and avoid legacy scans below native read quorum for ineligible nonwriters. Cover shared-lock contention, authority bytes, released fences, and completed-process and cold-store retries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): separate legacy read compatibility from migration
Validate observed legacy layouts against an established native read quorum without requiring legacy migration quorum. Reuse known unavailable native slots to avoid repeated peer RPCs. Keep migration behind its original elected-writer, availability and strict-majority requirements, and grant no bootstrap authority to read-compatible outcomes.
Add full-loader regressions for blank plus unreachable or missing slots, both creator roles, conflicting evidence and unchanged authority files.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): route startup test through storage facade
Use the existing internal NamespaceLocking re-export in the startup regression, preserving the architecture boundary checked by CI. Production code and the trait identity are unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(iam): preserve virtual parent collision invariant
Keep the federated binding regression focused on colliding regular and STS policy mappings. Virtual OIDC parent keys are now intentionally reserved and cannot identify regular users; that invariant already has direct IAM coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(gateway): project POST form metadata safely
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(gateway): preserve form authority across feature boundaries
Keep archive extraction decisions on projected POST form headers in gateway builds while preserving ordinary header handling without that feature. Update the checksum regression to model the signed form source instead of a forbidden wire object property.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The request facts trait, its accessors and the access-phase admission no longer
live in the legacy access hook file; they return the owned S3 error type.
Moves the rustfs/gateway pin to ab5242b0: under the legacy RustFS policy
readings a starts-with condition over a field the form does not carry is
read against the empty string, as the shipped server reads it.
The console listener now answers 404 with an empty body for every route
that is not a console or control-plane route (9cefecb04), so an
unauthenticated S3 root there is no longer a 403 from the S3 service.
The old expectation described the behaviour before that hardening.
Moves the rustfs/gateway pin to 6096b921 (dotted custom error codes, the
x-rustfs-error-cause header, completion checksum types) and maps the
cause a 503 carries onto the gateway's closed header member, so the
gateway stack answers 503 with its cause instead of a generic 500.
Moves the rustfs/gateway pin to 98cb70d1: on the RustFS profile an
unsigned browser-form policy is enforced as the shipped server enforces
it and the request stays anonymous.
Moves the rustfs/gateway pin to fb4d21fc: an empty DeleteObjects entry
ETag is carried per entry, and the RustFS profile reads aws-chunked
bodies as the shipped server does (one empty line before signed
trailers, legacy trailer bounds, XAmzContentSHA256Mismatch, legacy
framing refusal codes).
A use-case error raised with a code and no message reached the gateway
stack as an empty <Message>, where the legacy edge has always sent the
per-code default. The owned code type now carries that table, checked
row by row against the pinned s3s, and the one error edge applies it.
* fix(ecstore): reserve emergency space for delete journals
Keep bounded physical metadata headroom for recoverable deletion on Unix without bypassing quorum or rollback.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): recover delete metadata space on demand
Share a pinned reserve owner per physical disk and keep healthy deletion free of reserve filesystem probes. Retry only disposable staging and idempotent metadata primitives after actual ENOSPC, preserving transaction publication and quorum boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): retry concurrent full-disk deletes
Serialize ENOSPC waiters per physical disk and give each safe metadata primitive one bounded retry against shared released capacity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): account for delete reserves in receipt fixture
Reject undersized tmpfs mounts after including the eight per-disk delete reserves and keep a self-test for the capacity boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): refresh resume admission fixture lease
Refresh the existing node admission sample immediately before starting resume GC so the cancellation test reaches the native-open ownership boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): inject receipt ENOSPC on hardlink path
Keep the receipt fault at the atomic allocation boundary after the stage-two hardlink fast path landed, so reserve recovery remains covered.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Resolve multipart ownership from storage metadata and share authorization across both request edges. Preserve the existing legacy adapter while binding gateway claims to the recorded namespace.
* perf(ecstore): defer flat inline preflight and lazy-pin commit dirs
Eliminate redundant directory opens, ancestor prefix probes, and metadata stats on the Linux local-disk object commit path:
- Short-circuit validate_existing_local_disk_prefix_at on ENOENT for normalized relative paths under RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS.
- Defer source directory pinning and new-destination fstat in RenameCommitGuard on flat inline commits.
- Defer destination volume preflight to the commit guard preparation step on flat inline commits while preserving VolumeNotFound/VolumeAccessDenied preflight rejection proofs.
- Stage rollback backups by linking the held predecessor file descriptor via /proc/self/fd/<fd> and extend hardlinked rollback staging to non-inline overwrites.
* fix(ecstore): sync linked predecessor metadata when sync_backup is enabled
---------
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Wire both admin aliases through AdminBridge and the existing owned ServerInfo authorization. Keep the business handler and gateway pin unchanged; add no API or instance endpoint.
Add three test entry points covering eight allowed and 80 refusal/fallback rows, both hosts and aliases, complete JSON/XML, response headers, and unread unsigned bodies with signed controls. Production diff: +12/-3 lines.
Baseline and mutation base: 94a138b994. The unmodified native-selection test failed as expected. One compiled cohort passed eight controls, then eight faults failed their intended assertions: authorization bypass, native selection fallback, JSON, content type, Content-Length, refusal XML, premature body polling, and premature handler entry.
Restored sources were hash-checked and actually recompiled on 0278e1cd0e: 31 scoped tests, fmt, both RustFS Clippy modes, script-tests and test wiring passed. Existing snapshot checks now accumulate wire failures before asserting.
Not verified: CI, socket end-to-end or the full workspace suite.
Validation: cargo fmt --all --check; both rustfs all-target Clippy modes with -D warnings; 31 scoped restored tests; make script-tests; test wiring. No CI, socket-level end-to-end run, or full-workspace suite was run.
One compiled fault cohort first passed eight controls. Authorization bypass, native-route fallback, corrupted JSON, content type, content length and refusal XML each failed native_admin_extensions_catalog_real_signature_iam_and_wire; route fallback also failed native_admin_selection_is_exact_and_host_independent. Premature body polling and handler entry each failed native_admin_extensions_catalog_unsigned_before_body. Temporary instrumentation was removed, exact hashes restored and touched, and the product actually recompiled.
Restored test and lint base: 0278e1cd0e. Existing snapshot wire assertions remain enforced through the shared failure collector.
Integration verification base: 913edbc536 with the Iceberg config increment retained. The final dialect registers ten operations and four claims. Twenty-six selected tests passed: fourteen native admin HTTP tests, four mounted admin fixture tests, seven original extension handler tests and the exact edge selector. Both all-target RustFS Clippy modes, formatting, script-tests, current test wiring and prepush guards passed. Source hashes matched after verification. No new assertions or business behavior were introduced during integration.
Route both config GET aliases through GetCatalogConfigHandler. Keep its owned IAM, session-claim and warehouse handling, with the shared Iceberg JSON error outlet. Only these exact GET paths use the native bridge.
The matrix covers 12 allowed and 52 refused rows across both aliases and hosts, including full JSON, temporary credentials, query ordering and observed handler entry. Pin early duplicate-query rejection and the native anonymous message explicitly. The legacy baseline ran on b84985286.
Seven faults made the IAM, table-token, warehouse-prefix, alias-selection, handler-counter, JSON-outlet and complete anonymous-message assertions fail. Each had a passing control with the same compiled source. Restore hashes and a real restored build were checked.
Validation on 0278e1cd0: 16 scoped tests, cargo fmt and RustFS Clippy with and without gateway-stack passed. make script-tests passed with unchanged scripts. Production diff: +18/-9, excluding test-only probes.
No catalog storage, credential vending, catalog commits or other Iceberg operations were migrated. Full workspace, TCP/external SDK e2e and CI were not run.
Route the protocol client and legacy adapter through one owned multipart
copy use case, preserving storage ownership, fallback context, completion
accounting, conditional order, XML ETags, and error messages and causes.
Share copy-source validation and retain the original 1024-byte key limit.
Validation: original-main fixtures, 23 focused runtime tests, 38 effective
behavior mutations, restored-source runtime, default and gateway-stack
all-target Clippy, script tests, and final integration check and prepush.
Reduce syscall and sync overhead on local-disk DELETE and PUT paths:
- Hardlink the Stage 1 delete-prepared regular file to delete-quorum-committed on the cached Stage 2 local publish path when Stage 1 synced the journal, avoiding a duplicate temp-file create, write, fdatasync, and rename per drive while retaining the staged wrapper fallback on conflict or durability escalation.
- Accept bare DeleteRecoveryJournal payloads as unfinished (finished = false) commit receipts in parse_commit_receipt while keeping delete-prepared validation strictly DeleteRecoveryJournal.
- Reuse validated io_get_bucket_and_object_path components when constructing the root-relative metadata lease key in delete_version_inner on Linux.
- Track live /proc/self/fd/<fd> PublicationRoot handles via Weak<File> in NATIVE_PUBLICATION_HISTORY so record_native_publication skips redundant statx calls on pinned uncollided procfs roots.
* feat(ecstore): align inline budgets and binary batch decoding
Use a fixed 128 KiB per-shard inline budget with versioned admission and
preserve historical configuration markers and persisted object layouts.
Decode binary delete batches independently of the legacy JSON list while
retaining JSON-only requests and dual-write senders. Reject malformed
binary entries before disk mutation and preallocate the selected batch.
Cover budget boundaries, layout coexistence, MinIO metadata fixtures,
binary-only batches and JSON compatibility. Document geometry and
rollback boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): add isolated Docker inline geometry validation
Build clean-source Linux validation images from a frozen Git bundle.
Pin running image identities and binary hashes, and inspect persisted
object geometry through the existing filemeta decoder.
Provide an owned four-node Docker adapter for the existing ABBA runner.
Guard resource ownership, isolate loopback access, serialize state
mutations, and verify HEAD, full GET and Range contents.
Keep PUT content probes available through an isolated Linux mc adapter
when a host client cannot initialize. Cover ownership and failure paths
with script tests wired into the existing runner.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reuse bounded verified inline bodies
Share the materialization contract between metadata pre-verification
and buffered inline GETs. Bound responses to 2 MiB and a limited frame
count, and reuse an already verified body within the selected snapshot.
Route proof-protected parts through the readers that bind PartProofReader.
Cover larger single-block bodies, allocation bounds and proof validation
after a shard's bitrot checksum has been recomputed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(ecstore): support bounded multi-block inline objects
Keep the default EC block and accumulated shard budget while bounding inline frame storage and verified-body reuse. Preserve transformed and protected write admission and existing metadata formats.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate inline transport comparison modes
Pin and verify both MessagePack-only gates for each Docker cell, preserve an explicit JSON compatibility cell and probe exact accumulated shard boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): cover inline rollback and mixed-reader fleets
Add reusable compatibility fixtures for null and versioned objects, cross-block ranges, unfinished multipart uploads and batch version deletion. Record coordinator identity and per-version geometry, preserve retained-data manifests and archive manifests on dataset resets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify bounded inline read and write paths
Share checked frame decoding across metadata preverification and GET, resolve write admission once, and use one checked physical shard length for inline and disk writers. Preserve proof-bound readers and erasure recovery for damaged or missing shards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): persist inline compatibility mutation receipts
Save successful multipart and delete receipts before fallible content and geometry checks. Resume completed mutations through verification without repeating writes, and cover read failures, geometry failures, and command retries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(gateway): isolate capability probes from multipart listings
Give successful capability probes their own buckets so CreateMultipartUpload does not change the exact two-upload pagination fixtures. Preserve all listing-boundary assertions on both HTTP stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): honor fleet-confirmed codecs for batch deletion
Build DeleteVersions requests with one codec policy for the complete batch. Emit JSON compatibility fields by default and binary-only batches after both existing fleet gates pass. Cover request fields, canonical binding and actual remote sender, handler and disk deletion in both modes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): verify inline shards through borrowed frames
Share borrowed frame verification across buffered GET, bitrot readers and deep inline healing. Validate exact physical framing before scanning memory slices. Cover short prefixes, legacy padded frames, damage and trailing data, and provide a reproducible manual ABBA kernel comparison.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): account for manual inline verification benchmark
Raise the ECStore ignored-test ceiling from 2 to 3 for the opt-in
ABBA comparison of borrowed and streaming bitrot verification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reject inexact inline shard framing
Validate the complete encoded length of every selected inline data shard
before decoding or caching a verified object body. Return to the erasure
reader on missing, truncated, or trailing payload bytes.
Extend decoder fixtures across single and multi-block geometry, legacy
padding, unhashed payloads, and explicit bitrot verification bypass. Cover
metadata early-stop rejection and GET recovery for trailing shard bytes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): share inline shard reader construction
Build owned-metadata and borrowed-metadata readers through one Send iterator-based factory. Preserve shard ordering, empty slots, bitrot settings and caller-bound integrity verification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): move endpoint buffers into batch delete requests
Transfer the prepared endpoint String into the protobuf request and assert allocation identity across every batch codec mode. This removes the intermediate endpoint copy during request construction.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(gateway): remove superseded pagination isolation
Use the upstream capability probes that clean up their own multipart sessions. Run probes against the seeded pagination bucket while keeping all exact page and cursor assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(table-catalog): recognize borrowed multipart guard calls
Count owned and borrowed bucket/key guard forms while requiring all four production calls. Cover mixed split-file calls and reject wrong resources, missing error propagation and test-only mentions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): size heal cleanup fixtures above inline budgets
Use an environment-independent per-pool fixture policy and payload sizes above the aggregate inline budget. Assert external placement before pausing old-data cleanup, rebuilding the committed generation and checking physical shards and the final payload.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(table-catalog): count borrowed object metadata guards
Recognize the borrowed bucket/key guards in migrated legal-hold and retention entrypoints. Validate two object and two key guards and cover incorrect resources, missing propagation and test-only mentions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Register GET /{rustfs,minio}/admin/v4/cluster/snapshot with the existing owned handler and ServerInfo authorization. The read-only use case is unchanged.
Reuse the admin fixture for full JSON and refusal XML comparisons, response headers, exact route selection, and unsigned rejection before reading a body. The native matrix covers 8 allowed and 80 refusal or fallback rows; the body observer adds four signed controls and four unsigned refusals. Update the existing registration census.
Validation on b84985286: 35 scoped tests, RustFS Clippy with and without gateway-stack, cargo fmt, and make script-tests passed. Full workspace, distributed e2e, and CI were not run.
Mutations: IAM bypass, route removal, JSON replacement, and event-time corruption turned the signature/IAM/wire contract red. Reading the unsigned body turned the body observer red. Content-Type, Content-Length, and an extra XML Resource node each turned the corresponding wire assertion red. A temporary entry counter observed signed calls and no unsigned handler entry.
Production diff: +12/-2. No new public API or business logic.
* fix(targets): decode Kafka notification record keys
Build Kafka records from the decoded object identity so record keys match
the notification envelope for immediate delivery and persisted replay.
Cover form encoding boundaries, historical queue round trips, and early
rejection of invalid UTF-8 metadata.
Fixes#9437
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(targets): share allocation-efficient event key construction
Use a borrowed form decoder and exact-capacity key buffer for notification
envelopes and Kafka records. Reuse the established percent decoder and
preserve the owned public decoder API and persisted replay metadata.
Add exhaustive malformed-escape and UTF-8 compatibility checks, plus a
regression for borrowing plain object names.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Route the protocol client through its installed multipart context and owned
values. Preserve request headers, response and error projection, counting,
and audit completion. UploadId keeps one scalar compatibility projection.
UploadPartCopy, PostObject, gateway handlers and ownership rules are unchanged.
On f68ef7d5f: fmt, both all-targets Clippy modes, 37 scoped tests, and script-tests
passed. The 14-fault cohort had a green 37-test control; restored sources were
hash-checked, touched and recompiled before the final 37-test pass. On 489316e7a:
gateway-stack all-targets check and prepush-quick passed after rebase.
New runtime cases: 13 negative, 2 positive. Projection matrices cover 60 input,
64 output and 6 error rows. Non-test lines: +122/-20. No existing test was
deleted, weakened or ignored.
Effective mutations and failing tests (under protocols::client::multipart::
tests::upload_part_tests):
- part-number, length, md5, sse, eager-body, metric, audit:
upload_part_refusals_match_original_without_persisting_parts
- checksum: upload_part_input_projection_matches_original_fields
- output-field, etag-order:
upload_part_output_projection_matches_original_fields_and_refusals
- error-default, error-explicit:
upload_part_error_projection_matches_original_defaults
- ambient, observer: upload_part_client_preserves_stored_bytes_and_bound_context
Not run: full workspace, protocol wire lanes, released-binary differential,
live KMS/SSE integration, or CI.
Replace the protocol client compatibility dispatch with owned request and
response projections while preserving source validation, conditions, dates,
ETags, error answers, bound storage, fallback behavior, and operation counts.
Validated against the existing client and library behavior on f68ef7d5.
Final gateway-feature tests passed 11/11, including existing PutObject
regressions; default-mode tests passed 6/6. Both all-target Clippy modes,
script tests, wiring checks, and static preflight passed.
Twenty-two effective mutations covered observation controls, operation
counts, context selection, storage effects, source and condition parsing,
optional fields, dates, output fields, error facts, and request credentials.
Restored sources were rebuilt before the final passing controls.
This does not register UploadPartCopy or change the default S3 stack.
External migration lanes and release-reference testing remain outstanding.
Route CreateMultipartUpload, CompleteMultipartUpload and AbortMultipartUpload through the installed server's multipart use case. Keep the measured protocol projections and operation counts.
The upload-id scalar compatibility helper remains in Complete and Abort. UploadPart, UploadPartCopy, PostObject and persisted-owner work are outside this change.
Validation: 11 new tests; the runtime contract has 8 refusal cases and 2 successful flows. The restored selection passed 59/59 on d3abf8023. Both Clippy modes, fmt, script-tests, test wiring and prepush-quick passed. Non-test diff: +231/-42. The original client baseline ran on a98bd9028.
Effective mutations (38), grouped by the test that failed:
- multipart_create_projection_matches_original_fields_and_times: create-input-name, create-input-metadata, create-input-tagging, create-input-storage, create-input-sse, create-input-kms, create-input-md5, create-input-mode, create-input-hold, create-input-date, timestamp.
- multipart_refusals_match_original_client_errors: complete-error-message.
- complete_input_preserves_part_fields: legacy-part-checksum.
- multipart_create_projection_preserves_refusals: create-input-refusal.
- multipart_create_output_matches_original_for_stored_names_and_empty_values: create-output-name, create-output-strings.
- multipart_complete_projection_matches_original_parts_and_conditions: complete-condition, complete-parts, complete-part-number, complete-part-etag, complete-part-checksum, complete-condition-refusal.
- multipart_completion_projection_preserves_original_escaped_tags: complete-output-escape, complete-output-weak.
- multipart_completion_projection_preserves_header_only_field_behavior: complete-output-headerdrop, complete-output-header-validation.
- protocol_multipart_lifecycle_stores_parts_and_aborts: count-create, count-complete, count-abort, complete-location, complete-storage.
- multipart_uses_bound_store_with_a_foreign_ambient_context: create-storage, abort-storage, bound-store.
- multipart_envelopes_preserve_original_request_facts: request-facts.
- protocol_policy_denials_and_request_conditions: permission-create, permission-complete, permission-abort.
The final restored run had no leaky summary; an earlier control marked one pure projection test leaky. The linker reported the non-failing __eh_frame warning. The ambient fault's test exited 100 with target=false/ambient=true; its wrapper exited 1 because it expected generic assertion wording.
Not verified: CI, network protocol suites and the released-reference comparison.
Call the existing bucket use case directly from the protocol client. Preserve
bucket-name materialization, response facts, edge counts and bound or fallback
store selection. Keep legacy DTO imports confined to tests.
Tests: one new runtime matrix, with seven rejection and four success rows,
measured first against main at 70022e51c. Restored gateway bucket tests passed 4/4.
The old dispatch failed the source wiring guard; owned dispatch passed it.
Mutation checks: normal controls passed; the runtime matrix rejected storage
bypass, missing fallback, metric loss, ambient context selection, strict names,
and changed error code, message, status or headers. Isolated storage, fallback
and ambient-context faults retained normal operation counts. Restored sources
were touched and actually recompiled before the runtime pass.
Validation: fmt, package Clippy in both feature modes, script-tests and test
wiring. Production diff: +19/-14 lines. No new API or storage use case.
Not verified: CI, full workspace tests or live protocol end-to-end tests.
Use the bucket use case for ListObjectsV2 and both bucket listing entry points. Keep entry encoding, timestamps, ETags and refreshed session facts while removing three legacy FS calls. Other protocol operations are outside this increment. Non-test diff: +102/-82.
Added six tests: projection rows have 34 refusals and six accepted values; runtime cases have five refusals and eight successes, plus bucket exclusions and four credential rows. Existing assertions were retained.
Validation: baseline 2fea17d57 (2/2); restored build on a98bd9028 (15/15); all-target Clippy default and gateway-stack; fmt, script-tests, test wiring and prepush-quick. Rebased onto 88515bb7d with no task-file or pin overlap; all task blobs match the tested source.
Eight independent controls and 31 intended mutation failures:
- protocol_listing_output_matches_legacy_projection: strict Name, oversized key, invalid/missing ETag, XML escaping, weak tag, prefix, marker, encoding, timestamp.
- protocol_listing_bucket_projection_preserves_strict_names_and_milliseconds: bucket filtering, timestamp.
- protocol_listing_pages_and_encoding: cursor, count, encoding.
- protocol_listing_refusals_and_limits: empty/unknown credentials, counts.
- protocol_listing_bucket_context_and_defaults: count, bound context.
- protocol_listing_session_credential_projection_matches_legacy: token, secret.
- protocol_session_bucket_listing_revalidates_the_session_key: request context, IP, secure transport, headers, refresh, accepted count, refusal count, store context.
- request_extensions_preserve_authenticated_identity_and_source_ip: owner.
Experiments were removed, hashes checked and the restored product actually recompiled. Full protocol listing E2E and CI were not run.
Register data usage, plugin catalog and module-switch GET reads through
the existing AdminBridge and handlers under both admin aliases. Keep
neighboring methods and object-lambda routing on their existing paths.
Validation on 65057691: unchanged-production legacy baseline passed;
restored-source scoped tests passed 78/78. The new native HTTP matrix
covers 28 success and 236 refusal or routing-boundary rows. Exact replies
are pinned for both aliases and ordinary and bucket Hosts.
Three independent no-fault controls and 21 effective mutations passed
in one compiled cohort: selector removal/widening/lambda, each handler
binding, each IAM gate and action, both narrowed data-usage any-of gates,
signature verification, lambda dispatch, response bytes/status/request
id/content type and Keystone precedence. Exact refusal observations also
caught the disabled-key signature mutation. No existing assertion was
removed or weakened.
Gateway and default all-targets Clippy, script tests, test wiring and the
local prepush guards passed. No gateway pin or API changed.
Not verified: production listeners and full middleware, session
credentials, non-default persisted usage/module-switch data, full
workspace tests or CI.
Call the existing bucket use cases directly from the protocol client. Preserve root lookups, bucket validation, request context, response facts and operation counts. Other protocol operations stay as they are.
Non-test lines: +33/-40. Added cases: 17 refusals, 16 successes; reused three HeadBucket policy refusals. Protocol tests passed 39/39 in default and gateway-stack builds. Both rustfs all-target Clippy modes and make script-tests passed.
Four no-fault controls passed; 22 faulted test runs failed as intended. Mutations covered input routing, name refusals, output and error facts, counters, deletion, context resolution and authorization allow/deny. The force-delete fault also changed the observed bucket/object state. The raw HeadBucket NUL-name projection went red before its response text was restored.
Tests and Clippy ran on 2fea17d57. Script-tests ran on 2ca689f48; their sources are unchanged. Rebased onto 65057691 with unchanged client and gateway-pin blobs. Network transport e2e, workspace tests and CI were not run.
Serve the storage info, runtime capabilities and object cache statistics
GET aliases through the native admin service. Call the existing full IAM
gates and owned handlers, preserve raw request facts and keep the Keystone
guard first.
Add two contract matrices (42 allowed and 168 refused rows), with expanded
selector, registry, envelope and readiness coverage. On 2fea17d57d, the
restored source passed 72 scoped tests, fmt and both rustfs all-targets
Clippy modes. Script tests and test wiring also passed.
29 effective fault runs cover selection, lambda precedence, all three
dispatch/action/authorization bindings, signature verification, source IP,
raw URI/query, request context, readiness, response bytes/status/headers/
length and the registry. Nine controls ran in separate processes against
the same binary before those faults.
Not verified: live listeners and full production middleware, session
credentials, active cache statistics, CI or the full workspace tests.
Pair a single-use Keystone proof with sanitized owned authorization facts.
Preserve the original session-presence fact for policy principal projection,
and scope legacy credentials during both service call construction and polling.
Bind the gateway host-token principal to that owned identity without an IAM
credential or task-local fallback.
Initialize Keystone before gateway HTTP assembly and reuse the provider for
listeners and later auth integration setup. Refuse token-bearing claimed
non-S3 requests before console, admin or legacy dispatch.
Add 14 test functions. The primary ListBuckets/HeadBucket matrix has 13 rejected
and 7 allowed requests. Change enabled-Keystone startup to succeed after provider
initialization; retain both configuration tests, including disabled and invalid
settings. Production changes: +304/-77 lines, excluding test bodies.
Mutation checks (20 effective boundaries; each has a green control):
- owner, groups, username: keystone_http_tokens_authorize_real_s3_by_roles_groups_and_username
- pair-name: keystone_envelope_refuses_missing_and_mismatched_owned_identity
- provider, expiry: keystone_bridge_refuses_provider_mismatch_and_post_validation_expiry
- stale-established, stale-owned: keystone_stack_clears_both_payloads_before_a_signed_request
- legacy-before-deny, native-before-deny, admin-before-deny: gateway_keystone_tokens_are_refused_before_non_s3_legacy_dispatch
- scope: disabled_middleware_clears_identity_before_inner_call_and_poll
- principal-session: resource_free_conditions_preserve_legacy_projection_and_erased_keystone_session_facts
- secret, session, session-fact, replay, poison: request_identity_erases_secrets_and_consumes_its_proof_once
- repeat-init, missing-child: keystone_initialization_reuses_the_provider_for_concurrent_calls
Verification on 2ca689f: 214 focused tests passed after restoring and rebuilding
the production sources; all 33 Keystone package tests passed. Both package-scoped
all-target Clippy configurations, formatting, source guards, script tests and
prepush checks passed. Temporary fault code was removed.
No gateway API, dependency pin or PostObject changes. A deployed Keystone server,
the complete listener token lane and CI were not exercised locally.
Project protocol PutObject directly into the owned write input and project the answer back without the legacy DTO path. Keep the measured normalization, refusals, error facts, server context and operation counts at the client edge.
Non-test lines: +196/-18. Eight added tests include 25 runtime cases (18 negative, 7 positive).
Verified on f4266a4576 with gateway pin 44f9a2422d745ffba952347ec91b0332ef2667a4: protocol tests 36/36 in default and gateway-stack builds; both all-target Clippy builds; script-tests; prepush-quick.
Sixteen runtime mutations failed their intended tests from one fixed binary after all eight controls passed:
- empty: protocol_put_optional_values_match_legacy
- condition, specific-none: protocol_put_condition_grammar_matches_legacy
- input-checksum: protocol_put_checksum_input_matches_legacy
- retain-nanos, retain-range: protocol_put_retain_until_matches_legacy
- output-etag, output-fields, output-checksum: protocol_put_output_matches_legacy
- error-default, error-facts: protocol_put_error_projection_matches_legacy
- event-hold, event-days, event-years, body, metrics: protocol_put_runtime_matches_legacy
The experimental code was removed and the restored implementation was recompiled and tested in both builds. The earlier incorrect metric lookup is not counted as evidence.
Scope is protocol PutObject. Network protocol E2E, the full workspace and CI were not run.
Serve GET /rustfs/admin/v3/info and /minio/admin/v3/info through a
scoped native service using the existing signature verifier and the real
ServerInfoHandler IAM gate. Keep object-lambda routing ahead of these
aliases and retain the raw URI, optional ReqInfo, headers and body.
The pinned RustFS profile already handles claims on bucket-style hosts.
Outside tests and the generated lockfile: +180/-14 lines. Add seven tests,
including 32 authorization refusals and 16 authorized info responses, both
SourceIp directions and signed legacy-service comparisons. No existing
assertion was weakened. Duplicate-query signing is tested as a fixture
refusal, with routing coverage retained separately.
Validation on f4266a4576: fmt, both rustfs
Clippy modes, 42 scoped tests, script-tests and prepush-quick.
Twenty-four mutation runs failed as intended across 22 faults:
- native_admin_selection_is_exact_and_host_independent: remove/widen
selection and lose object-lambda priority.
- native_admin_real_iam_and_legacy_contract: bypass signature verification
or IAM, change the action or source IP, omit pools, rewrite the URI,
or lose object-lambda priority.
- native_admin_envelope_preserves_raw_alias_host_facts_and_optional_req_info:
rewrite the URI, drop/default ReqInfo, or drop client, peer or request ID.
- native_admin_complete_response_keeps_bytes_status_and_content_type:
corrupt bytes, status, content type, extra headers or content length.
- native_admin_authenticated_request_refuses_an_uninstalled_server_context:
remove the readiness check.
- native_admin_registers_one_operation_and_both_claims: register an extra op.
- native_admin_signing_fixture_refuses_duplicate_query: accept duplicate keys.
The same mutation binary passed all seven controls before fault runs;
the restored production tree was then rebuilt and tested.
Not exercised: network listeners, session credentials, the combined
Keystone change, CI or other admin endpoints.
* fix(rpc): scope canary errors to grpc failures
Exclude expected read-version remote responses without hiding transport, gRPC status, response-decode, or other gRPC operation failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(rpc): verify canary revisions from image history
Use immutable image history only when recognized config labels are absent, keep mismatches fail-closed, and record the provenance source in verifier output.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(audit): restore bucket config audits and metadata guards
Audit accelerate, request-payment, and website configuration mutations through the shared owned request scope, including failures and the website deletion status.
Validate the replication mutation delegation before checking peer reload notifications. Count borrowed reserved-prefix guard calls and cover missing production calls, test-only mentions, audit completion, and stale bucket incarnations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Route both GetObject calls, HeadObject and DeleteObject to the owned use cases. Keep the request facts and millisecond timestamp behavior. PutObject stays unchanged; 12 operations and 13 trait calls remain.
The client edge is an interface in the layer guard. Its public re-export is covered too, and app-to-client imports are rejected by fixtures.
Runtime wiring: +177/-101 non-test lines. The new matrix has 18 negative and 13 positive operation cases, plus the extended identity-copy control.
Validation: 28 protocol tests passed in both default and gateway-stack modes; both all-target Clippy modes passed. Script tests, layer controls and focused architecture/footprint/test guards passed. Five source faults failed at the intended claims, range, Head ETag, deletion-witness and version assertions. The precision oracles failed before the millisecond projection was added.
Runtime and Clippy ran on e9da1d309; initial controls and faults ran on 2308dee4b2. Rebase to 00ecde125 changed none of the seven task files or gateway pins. CI and HTTP transport were not checked.
Use the existing console router when the gateway console is enabled. Keep lambda and extension precedence, the legacy selection, and outer middleware assembly unchanged. The common body conversion stays at the edge.
Add test-only assets and one contract with 10 native and 52 excluded routing probes. Non-test source: +89/-16. The restored run passed 22 console/stack tests on e9da1d309; both Clippy modes, script-tests and prepush-quick passed.
native_console_route_contract went red for three independent mutations: remove the native branch (service-call counts), widen the shared prefix (neighboring S3 path), and route every claim to console (anonymous admin refusal). Restoring the sources triggered a real product recompile.
Full outer HTTP wire checks and a production UI bundle were not tested. Admin, STS and website migration remain separate work.
Share the batch deletion engine between legacy and native edges with owned inputs and results. Keep exact per-entry authorization, version and retention handling, ordered errors, quiet responses, metrics, and audit aggregation. Preserve frozen resource resolution and rejected-key echoes.
On 2308dee4 with gateway 44f9a242, the focused suite passed 208 tests before and after deliberate mutations. Both rustfs Clippy configurations, script tests, test wiring, formatting, and prepush guards passed. Four added runtime tests cover mixed successes and per-entry refusals, frozen fields, and rejected-key state.
Five targeted tests failed when batch target clearing, denial error mapping, and raw-key echo were changed. Sources were restored byte for byte and product artifacts recompiled.
The legacy edge matched 62 observed baseline responses including all headers, XML and object states. Native decoder and expected-owner differences are tracked separately. Actual webhook records matched for mixed and quiet requests across the baseline and both candidate stacks.
Full workspace tests and CI were not run. Verification is scoped to this migration increment.
Use the validated request offset and full timestamp precision when projecting PutObject and PutObjectRetention dates into authorization and storage.
Signed requests verify exact stored dates and version isolation for positive and negative offsets and UTC. Policy tests cover denied dates and the typed date supplied by PutObject authorization.
On b2b4f0c4, 175 focused tests and six raw xl.meta checks passed. The UTC-only control and five targeted mutations failed at their intended assertions. Both Clippy feature configurations passed.
Share GetObject authorization and the existing Select query producer across both stacks. Carry live events through the typed response with the projected SSE headers. The producer retains its single deadline, snapshot checks, bounded channel and receiver cancellation.
On c5b4a8577, the restored build passed 207 focused tests, both RustFS Clippy feature modes, formatting, script tests and test wiring. Signed socket probes matched all 17 legacy cases and 12 gateway cases. Five gateway refusals omit the legacy default message, tracked in rustfs/backlog#2684.
The fault build passed all 98 controls. Bypassing authorization failed on both stacks. Removing receiver-close handling, dropping the terminal Records permit, skipping the Stats fence, replacing an error with End, and dropping progress or scan range each failed the corresponding test.
Verification used local stores and SSE-C. The full workspace suite and external KMS were outside the local run.
Share the authorization snapshot and restore admission between both stacks.
Preserve HTTP 200 for both success paths and emit ObjectRestorePost on new
admission only.
Verified on c0763a43 with 114 focused tests, four serialized ILM tests, and
16 signed wire responses per stack. Faults covered shared authorization,
generation capture and admission, SELECT rejection, worker parameters and
liveness, and both audit and event paths. Cluster lock loss and mixed-version
rollout remain unverified.
Share each operation's authorization and metadata write with the legacy edge.
Keep their request ordering, version and bucket-incarnation fences, replication
completion, and notification suppression. Retention checks the locked metadata
again before committing a change.
The two stacks were exercised through signed HTTP requests and independent
disk metadata reads. Faults in authorization, body projection, version
selection, counting, auditing, notification suppression, scanner invalidation,
the bucket fence, and the commit-time retention check reached their assertions.
Share the bucket Object Lock write between the legacy and gateway edges. Keep lock configuration and versioning as sequential writes against the authorized bucket incarnation, followed by peer reload, site replication and scanner invalidation.
Cover eight refusal or failure cases and three successful cases with durable metadata and signed requests. The two stacks run the same request cases and refusal matrix.
The mutation run detected both missing write fences, swallowed peer failure, propagated site failure, early or absent scanner invalidation, missing audit, wrong authorization action or resource, dropped request body and incorrect operation counting.
Validation used debug builds on d461908bf. The final lint pass elides an equivalent lifetime in a test helper. Peer and site failures were injected locally, with real multi-site delivery left unverified.
Route PutBucketReplication and DeleteBucketReplication through the owned bucket use case. Share site checks, metadata fencing, target cleanup recovery, and audit completion with the legacy edge.
Signed requests exercise 28 refusals and six accepted edits across both stacks. The direct storage tests cover six failure paths and two successful edits. Twenty-four faults turned their expected disk, authorization, locking, callback, audit, or metric assertions red.
The restored gateway-stack build passed 132 selected tests on 75b293f09, including both HTTP stacks. Both Clippy variants and the mutation cohort passed on fb3e2818c. The fixed legacy CLI matched all six response records. The standalone default runtime suite, multi-site delivery, and CI were unverified.
Share the owned write paths across both S3 stacks. Preserve the policy
document bytes, default KMS key resolution, bucket tag validation, and
incarnation fences before reload and replication callbacks.
Add three successful writes and three rejected stale writes on disk.
Extend the two-stack refusal matrix with six successful and 24 denied
requests. Cover the encryption and tag validation order before lookup.
The fault cohort exercised the action and resource checks, value
validation and order, raw policy bytes, default KMS readback, legacy
policy count, durable writes, incarnation fences, reload, replication,
audit, and scanner invalidation. Each selected assertion failed.
Local checks used base 75b293f09. Fixed CLI comparisons covered legacy
responses, audit entries, and operation counts. Live peer replication,
external KMS providers, and CI were outside the local checks.
Share the three generation-fenced deletes across both S3 stacks. Keep
their existing IAM actions and use the shared IAM scope for tag deletion.
Add six focused tests covering 21 refusals and nine successes, and add the
three operations to the refusal matrix. Substituting IAM actions and
buckets, removing the tagging scope, and suppressing the gateway counter
fail the authorization and metrics checks. The storage tests detect a
lost fence, a skipped write, missing reload or replication calls, omitted
scanner invalidation, and a changed audit status.
Verified on 4294c41cc with scoped nextest, both Clippy modes, script tests,
quick checks, and local CLI comparisons. Peer network delivery was
outside the local checks.
Route PutBucketAcl and PutBucketLogging through shared bucket use cases.
Preserve the ACL response for header-only requests and the bucket lookup
before the XML policy refusal. Write logging configuration using the
authorized bucket incarnation and the existing persistence format.
Verify 111 focused tests on 4294c41cc, including the storage access suite,
and run both Clippy modes, script tests, and local wire comparisons. The
wire matrix exercises 15 refusal cases and seven successful writes on
each stack.
Change both IAM actions and resources, audit statuses, and counters in
the fault cohort. Reverse ACL lookup and XML policy checks, drop policy
presence, corrupt logging bytes, reverse logging conversion order, and
remove the incarnation guard. The regressions detect each fault, and
the stale-authorization test reads the replacement bucket from disk.
Route Get, Put, and DeletePublicAccessBlock through shared bucket use
cases. Preserve the persisted switch values, mutation incarnation fence,
reload behavior, audit status, and missing-configuration response.
Add six focused tests covering 25 refusal cases and 17 success cases
across both stacks and storage. Extend the refusal and counter matrix
for all three operations.
Substitute each IAM action and bucket, corrupt each operation's value,
change the missing-config status, and suppress each edge counter. The
round-trip and matrix tests reject those faults. Put and Delete tests
reject a missing incarnation fence, the wrong reload mode, and wrong
audit status. Read the durable replacement-bucket bytes after disabling
both fences.
Verify on 92050cdca with scoped nextest, both Clippy modes, script tests,
and local wire tests. Peer reload delivery was outside these local tests.
Serve accelerate, request-payment and website writes, including website
deletion, through shared use cases. Both S3 edges enforce the bucket
policy and the metadata incarnation fence before changing stored data.
Verified on 92050cdca with formatting, Clippy on both RustFS feature
sets, script tests, 21 RustFS tests and 27 operation and metrics tests.
The fixed legacy CLI supplied 42 HTTP comparison cases per stack.
The wire assertions caught altered IAM actions and resources, missing
denial audits and operation counters, and skipped website validation.
The stale-incarnation test caught changes to the replacement bucket's
disk bytes. The lookup-order test caught early configuration conversion.
Refs rustfs/backlog#2734
* fix(metrics): label grouped fdatasync batches accurately
Record the serialized user-space fdatasync coordinator as grouped and reserve parallel for independent per-file fanout. This keeps performance dashboards from conflating two different dispatch models.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): separate grouping from timeout race
Keep the same-disk batching test focused on queue aggregation. Claimed-follower timeout ownership remains covered by its dedicated deterministic regression, avoiding a valid pre-claim timeout race in the batching fixture.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(table-catalog): accept borrowed multipart guard calls
Count both owned String and borrowed &str spellings of the same reserved-prefix guard. UploadPart now obtains borrowed bucket and key values from its typed target, while the production guard remains in place.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(grpc): attribute rebuilds and bound bulk responses
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(grpc): attribute rebuild success after publication
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(obs): add exact HTTP/2 ping timeout diagnostics
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(multipart): run refusal matrix on large stack
Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.
Keep the original test name and all refusal-order assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): pin nixl-sys workspace source
Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.
Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): authenticate msgpack peer capability
Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(grpc): bind peer capability to connection generations (#9433)
* feat(grpc): track physical connection generations
Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): bind msgpack capability to connections
Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): guard stale generations before dispatch
Reject generation-bound requests before inner transport call or first payload frame when tonic readiness observes a different physical connection, while leaving ordinary mutations unguarded until the send-policy follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): add per-peer msgpack rollout gate
Add the default-off peer-capability environment gate, cached evaluation, fixed-cardinality send-decision metrics, and rollout constraints for stable peer-affine connections.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): apply per-peer msgpack send policy
Capture one capability decision per eligible RPC, bind peer-mode requests to boot and connection generations, and rebuild at most one dual-write request only after a locally proven pre-dispatch rejection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(rpc): verify msgpack peer canary evidence
Add a strict three-phase four-node evidence verifier with raw Prometheus and workload receipts, adversarial tests, script-test wiring, and the operational manifest contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(rpc): align batch read manifest encoder
Match the BatchReadVersion compatibility manifest to the request-owned encoder expression used by the production send path.
Restore the owning ECStore source guard while preserving wire and runtime behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(rpc): orchestrate msgpack peer canary
Run the fixed mixed/all-new/rollback four-node matrix with prebuilt images, real workload and readback, raw Prometheus capture, strict manifest generation, and adversarial fake integration tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(rpc): verify canary image and telemetry identities
* test(rpc): bind canary evidence provenance
Scope canary metrics to one injected cluster identity and bind candidate, baseline, and per-phase running containers to immutable image IDs and exact build revisions.
Reject foreign-cluster metrics, stale revision labels, same-image tags, and phase image mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(table-catalog): accept borrowed multipart guard calls
Count the exact borrowed bucket/key guard call introduced by UploadPart target ownership while preserving the four-production-call contract.
Cover the borrowed spelling in both the positive fixture and the test-only false-positive regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Share XML tag authorization and storage execution across both stacks.
Preserve the requested version, replication-target proxy, and metadata
completion order.
The dual-stack tests exercise 37 refusal and 13 success scenarios.
The restored integration run passed 148 tests on 26db9b19a, and the
proxy cleanup verification passed 134 run, 134 passed.
Both feature configurations passed Clippy, and script tests passed.
Fault injection caught native and shared authorization bypasses,
missing XML facts, and incorrect action, resource and tag key facts.
It also caught duplicate counting, incorrect persisted tags, missing
or duplicate success events, and event or dirty publication on a miss.
Live replication targets and ECStore write faults remain unverified.
Serve bucket acceleration, request payment, website, and logging
configuration through shared persisted read paths. Keep each operation's
existing authorization and legacy response projection. Append distinct
metric labels without shifting existing indexes.
The scoped RustFS, S3 operation, and metrics suite passed 283 tests on
993464fbb. The integrated configuration and refusal suites passed 21
tests on 292974baf. Both feature modes passed all-targets Clippy.
Standalone checks compared 23 successful and 17 refused legacy requests
against cfdd4b268 and exercised the gateway answers on the same data.
Action, resource, anonymous, projection, default, missing-bucket, counter,
and audit faults failed their original assertions. Metric registry tests
also detected duplicate labels, changed index order, and an added event.
Verification is scoped to these local suites and wire probes.
Share bucket-scoped authorization and persisted configuration reads across
both stacks. Carry the stored configurations in gateway responses and
preserve the operation-specific errors for missing configs.
On RustFS 993464fbb, 120 scoped tests passed, including both refusal
matrices and six new tests with 24 refusal and 12 success cases.
Live HTTP checks covered response documents, operation counts and
authorization-denial audit records. Both feature configurations passed
Clippy, and formatting, script tests and source guards passed.
Changing each IAM action or resource admitted a forbidden read.
Duplicating each edge count and changing each stored output failed its
test. Returning 404 for missing encryption failed the 400 assertion.
Suppressing gateway denial audit removed all nine expected records.
Runtime verification used single-node debug servers.
Share the read use case and both IAM decisions across the legacy and
gateway handlers. Keep attribute selection, stored metadata and
pagination in the use case, and encode the result at each edge.
The 6f93ac5f control matched 55 legacy wire cases and the restored
sources passed 139 scoped tests. Both stacks exercised 59 signed
requests, including 31 legacy refusals and 32 gateway refusals.
Seventeen injected faults each turned a targeted assertion red. They
cover the four IAM actions, resource and version binding, empty-version
selection, handler authorization, SSE-C, selection, both checksum paths,
paging, ETag, Last-Modified, counting and notification suppression.
Share one owned write path across the gateway and legacy edges. Project
runtime destinations at each edge and defer consumption of the persisted
conversion result until after filter validation. Keep IAM scope, the
incarnation fence, and the existing validate, write, reload, clear, and
add order. Count the request at its edge and audit it in the use case.
Add three refusal tests and four tests that exercise success and failure. The legacy HTTP baseline passed on 97a7639f. On 6f93ac5f the
38-test control passed. Six fault injections exposed wrong IAM action,
wrong IAM resource, a missing incarnation fence, duplicate counting,
the wrong audit status, and missing reload callbacks. The first three
changed the stored bytes guarded by their refusal tests.
The restored tree rebuilt and passed all 38 scoped tests. Both package
Clippy configurations, script tests, test wiring, and prepush passed.
Cross-node reload delivery and remote audit sink delivery were outside
the local test scope.
Use the same authorization and audited use case on both S3 stacks. Bind
the gateway claim to the owner selected under the upload write lock
before staged parts can be removed. Preserve the existing generation,
lifecycle and quorum checks.
Verified the fixed legacy baseline, the integrated native wire flow, 212
focused tests and both Clippy modes. Fault checks exercised owner
selection, deletion and orphan handling, scoped IAM, generation
transport, audit, operation counts and dirty usage.
Audit delivery and separate fault injection of the existing final
lock-loss and metadata-incarnation fences were outside this verification.
Resolve the source from the gateway read proof. Share source and
destination authorization, tag checks, and the existing storage copy
between both S3 stacks.
Checked the legacy wire baseline, both CopyObject wire contracts, and the
access and refusal suites. Nine temporary faults triggered the intended
assertions. The restored build passed seven focused tests with gateway
75757292, and Clippy passed with and without gateway-stack.
Seed a lifecycle configuration so the admitted request removes real state.
Verified the matrix census and both refusal matrices in the CopyObject development checkout on 93947f22c.
Share authorization and pagination across both S3 edges. Resolve the
native upload claim using the owner from the same quorum-selected
metadata as the parts page. Preserve typed storage errors and write
the page through each edge's response projection.
The scoped suite includes all 98 storage access tests, real upload
sessions, stored-owner combinations, both wire paths, and the storage
API. Twenty-three fault modes exercised the owner, authorization,
lookup, error, counter, and projection assertions.
Validated 168 focused tests after restoring and recompiling the candidate
on c2b5fbb plus the isolated fixture cleanup. The owner matrix covers
three positive and six negative cases; each wire IAM matrix covers one
positive and seven negative cases. Both feature configurations passed
Clippy. Full migration acceptance lanes remain pending.
Abort the session returned by the successful matrix call through the
fixture store so later pagination assertions see their own sessions.
Both listing boundary tests failed on c2b5fbb and passed with the
fixture cleanup. Both refusal matrices passed with the same cleanup.
Share the incarnation-fenced lifecycle deletion across both edges, preserving reload and site-hook ordering and one audit record.
Validated 37 focused tests on 77da63397, both all-target Clippy configurations, script tests and wiring guards. Thirteen effective faults covered deletion, authorization wiring, incarnation fences, counting, reload and hook ordering, and audit behavior. Live peer delivery and final migration acceptance lanes remain unverified.
* perf(grpc): attribute rebuilds and bound bulk responses
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(grpc): attribute rebuild success after publication
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(obs): add exact HTTP/2 ping timeout diagnostics
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(multipart): run refusal matrix on large stack
Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.
Keep the original test name and all refusal-order assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): pin nixl-sys workspace source
Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.
Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): authenticate msgpack peer capability
Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(grpc): bind peer capability to connection generations (#9433)
* feat(grpc): track physical connection generations
Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(rpc): bind msgpack capability to connections
Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Stage inline overwrite rollback backups (xl.meta.bkp) directly relative
to the pinned destination directory descriptor in RenameCommitGuard
instead of staging in .rustfs.sys/tmp and walking both paths a second
time. Also fuse post-commit xl.meta.bkp removal and empty-ancestor
directory pruning into a single blocking dispatch while stopping at live
object roots.
gVisor presents a host bind mount as 9p, so statfs reports V9FS.
When the mountinfo options include both trans=fd and directfs, the
startup filesystem guard treats that endpoint as local POSIX storage
and logs that the host type is not visible. Real 9p and virtio shares
still warn.
Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
* fix(iam): authorize delegated service account creation
* docs(iam): clarify explicit grants for custom administrator policies
* fix(iam): include service account delegation in admin wildcards
* style(policy): format delegated authorization test cases
* fix(iam): keep delegated authorization and replication consistent
* fix(ci): include delegated account test in E2E selections
* feat(diagnostics): restore allocator and runtime metrics
Port the missing live-byte accounting and serving-runtime observations from 13a30abca. Reuse the current MiMalloc GlobalAlloc wrapper, expose explicit metric availability, and cover allocation operations and feature selection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): detach queued fdatasync followers safely
Adapt the follower timeout experiment from 138bc69f6 with exclusive queue ownership transfer. Move queued paths and resource units into independent sync, retain claimed batch ownership through completion, and record observed queue time. Cover publication before the original batch resumes, sync failures, claimed batches, and queue accounting.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): index sync followers and isolate missing sources
Use reusable slab slots and monotonic identities for constant-time queued follower extraction. Avoid retaining follower group references when the timeout is disabled. Preserve claimed-batch ownership and reject stale tickets after slot reuse.
Keep missing-source results local to their requests so healthy requests still synchronize their own files. Preserve fail-fast behavior for other disk I/O failures. Repair failure-probe synchronization and exercise source disappearance, reused slots, disabled tracking, cancellation and bounded admission on Linux.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ci): apply storage stack to multipart refusal fixture
Route the real-store absent-binding refusal-order fixture through the existing bounded 4 MiB storage setup in both default and CI profiles. The Linux CI failure reproduced with the default 2 MiB stack. Quota, legal-hold, error, cause, audit and notification assertions execute through the established storage fixture budget.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): make sync worker idle handoff atomic
Move batch claim, in-flight accounting and completion onto the existing worker guard. Publish the idle state under the queue lock so an enqueue during cleanup establishes successor-worker ownership. Keep stale cleanup conditional on current ownership.
Add a controlled interleaving regression that detaches the final queued follower and enqueues a successor after the empty claim. The old worker leaves that request without an owner; the refactored worker hands it off and completes one physical sync.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): await grouped sync resource release
Wait for native producer admission units to reach zero after the blocking sync finishes. This removes a scheduler-dependent assertion window exposed by the latest main integration while preserving the cancellation and ownership checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Share authorization between both CreateMultipartUpload edges. Keep the
decoded legacy fields intact, preserve the replication destination
version and apply the captured bucket incarnation before creating the
upload. Leave checksum, encryption and retention metadata on the same
storage path.
Verify the frozen legacy behavior on 2c2af51a before the migration and
exercise both stacks with 66 refusal cases and 22 success controls.
Read back sessions, version, retention and encryption metadata.
Fault probes made authorization tests fail on both edges when an action
or the resource binding was lost. Storage tests detected a different
destination version and unexpected sessions when generation, quota, KMS
or SSE checks were removed. The wire and
helper tests failed for early name rejection, lost timestamp precision,
metadata, checksum, SSE response, the operation count and unexpected
notification.
On 77da6339, 185 scoped tests passed with a 32 MiB test stack. The default
stack run passed 184 tests and reproduced the existing overflow in
completion_absent_binding_refusal_order_baseline. Both all-target
Clippy variants, test wiring, script tests and prepush passed. After
integration with 97a7639f, gateway all-target check and prepush passed.
Whole-workspace CI and rio-v2 were outside the local verification.
Share the streaming multipart write path and staged authorization across both edges. Resolve the native upload ID against the active storage record at the existing lookup. Carry checksums, trailers, SSE values and the stored ETag through each response projection.
Add seven tests for application lookup, route authorization, framing and durable part state. Exercise 62 refusals and 10 successful requests across those fixtures. The existing SSE-C completion test also covers the migrated UploadPart core.
Verification on 1467d32695: fixed legacy contract, 147 gateway-feature tests and 119 default tests, both all-target Clippy legs, script tests, test wiring and prepush checks. The test scopes include the entire access module. The gateway scope includes PutObject and both refusal matrices.
Twenty-two local fault controls exercised edge authorization, policy and generation, resource and upload ID projection, owner binding, lookup count and error ordering, logical length, trailer publication, checksum validation and typed producer errors. Each fault failed its intended test after the same artifact passed with the selector unset.
Linux RDMA and a native KMS denial through UploadPart remain unverified.
Validate lifecycle status across both request adapters and preserve mutation error priority, ordered rule IDs, and stored dates. Forward the original bucket and request envelope at the legacy edge. Pin the gateway lifecycle date carrier.
The priority fixture covers seven refusal cases and one success case. All 39 scoped tests passed after source restoration. Fifteen fault switches triggered 16 expected failures covering priority, early conversion, metrics and audit, action binding, transition writes, date precision, offsets, and rule indices.
The tests and both Clippy modes passed on 0567078 with gateway 2a6b8d1. The gateway all-target check and prepush passed after rebasing onto 1467d32. Full-workspace and remote CI results remain unverified.
Register PutObject and share write authorization with legacy PUT and
browser POST. Carry authenticated trailer declarations and captured
frames into the owned write, preserve typed eager-read errors, and keep
the table-catalog guard tied to the actual write call.
Verified on RustFS 0567078f5 with gateway d75b6ce on macOS. All 1022
affected tests passed with a 4 MiB test stack, along with both
all-target Clippy configurations, script tests, test wiring, and
prepush checks. A separate gateway wire run stored and read an empty
object using the rebuilt server binary.
Fourteen single-fault checks covered trailer selection and capture,
typed producer failures, tag policies, the shared write grants, and
POST policy projection. The reserved-prefix source guard also rejected
a fixture whose owned write call was replaced.
Route bucket versioning writes through the shared authorization and audited use case. Preserve legacy configuration preparation, the incarnation fence, refusal order, persisted XML, and the reload, replication, and scanner callbacks. Count each admitted write once at its edge.
Verified against RustFS 0567078f5 and gateway d75b6ceb. The signed wire cases cover six accepted writes and nine refusals per stack. The affected suite passed 29 tests, the typed-error suite passed 26, and both Clippy configurations, script tests, wiring, and prepush checks passed. The wire cases and both refusal matrices passed again after boxing the test fixture's SDK error.
Eight fault controls went red for both authorization hooks, the legacy counter, an omitted write, a dropped empty excluded prefix, Object Lock precedence, replication refusal, and the custom-status error. CI, the Linux reference comparison, and live multi-node callback delivery remain unverified.
Register GetObject and share its authorization and native response facts
with the legacy edge. Preserve conditional headers, range completion,
stream errors, body ownership, SSE reads and request accounting. Pin the
gateway 304 response-override fix without unrelated dependency changes.
Keep the incoming native Put path and update its profiling guard source.
Validation on b7e7a6156:
- Normal and mutation-disabled controls: 9/9 each.
- Affected suite: 754/755; the profiling guard still read the old Put file.
After correcting that source path, its targeted rerun passed 1/1.
- All-target Clippy passed with gateway-stack and default features.
- Formatting, script-tests, test wiring and prepush-quick passed.
- Authorization matrix: 40 refusals and 8 allowances across both edges.
Mutations: swapped version action, redirected resource, ignored denial,
and bypassed Served authorization failed the permission or HTTP checks.
Earlier effective mutations covered body copying, premature owner drop,
range status, missing 304 headers, a 304 body, wrapped stream error causes
and duplicate accounting. No assertions were removed or weakened.
Linux RDMA paths were not compiled or exercised. GetObjectAttributes,
PutObject gateway registration and the default-stack switch are not part
of this increment.
Use native gateway DTOs for stored notification reads, register the handler,
and keep both edges on the shared bucket authorizer. The legacy edge now
uses the compatibility seam. Notification writes and the default stack are
unchanged. Non-test changes: +53/-36 lines.
Add three refusal-oriented tests and two configuration-read tests, including
successful IAM controls. Cover absent and stored configurations, reserved
bucket shapes, unknown stored XML, event/filter values, bucket-bound IAM
allow/implicit deny/explicit deny, and one handled-operation count.
Mutation controls on da5414c: bypassing each edge's authorization made its
notification_authorization_binds_the_bucket_on_the_*_stack test fail because
the wrong bucket was served. Dropping the stored queue made both
*_stack_answers_the_stored_notification_configuration tests fail. Restore
hashes were checked and sources were touched before a real rebuild.
Validation: 279 scoped tests passed on da5414c; 70 scoped tests passed after
rebasing to fd8dc41. Both rustfs all-target Clippy builds passed on fd8dc41,
as did fmt, test wiring and prepush-quick. Script tests passed on da5414c.
The earlier broad run reported one existing CORS test as leaky with exit 0.
CI and the full workspace suite were not run.
Use RequestEnvelope, a native upload body, raw write fields and owned results in the PutObject core. Legacy and RDMA ingress project once; internal writes share the core. Preserve archive batching, shared SSE and object-lock validation, completion ownership, audit events and response facts. Keep the merged RDMA length, cache lifetime and response-header behavior.
No gateway registration or UploadPart migration is included. Source outside test files/modules: +820/-383 (including conditional test imports).
Validation on fd8dc41f0c: fmt; default and gateway-stack all-target Clippy; 903 scoped tests with RUST_MIN_STACK=4194304; script-tests; test wiring; prepush quick checks. The archive authorization matrix has six refusal cases and three allowed cases, with persisted object/metadata checks. A fixed a279 legacy TCP baseline has 48 matching response records; the two retained socket tests cover PUT and POST behavior. The recorder observes drained counter deltas alongside the official cumulative 0/1/2/3 operation totals.
Mutation evidence: 44 distinct faults, grouped below by the test that rejected them. The counter and archive authorization additions each passed an unflagged control before their five mutations. The other fault controls and failures were retained from the same unchanged implementation.
- app::object::native_body::tests::legacy_body_preserves_demand_pending_and_length_validation_location: body-eager, body-length
- app::object::native_body::tests::legacy_producer_error_keeps_its_concrete_type_and_is_not_eof: producer-source
- app::object::native_body::tests::internal_producer_error_preserves_io_kind_and_nested_source_after_data: producer-source
- app::object::native_body::tests::native_trailers_are_published_only_at_eof: trailer-pending, trailer-value
- app::object::native_body::tests::native_framing_and_classified_errors_are_not_unwrapped: native-byte-count, native-class, native-kind
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_defers_invalid_raw_values_to_the_existing_validators: input-bucket, input-key, input-storage_class, input-server_side_encryption, input-object_lock_mode, input-object_lock_legal_hold_status
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_keeps_each_checksum_and_unknown_length: input-checksum_crc32, input-checksum_crc32c, input-checksum_sha1, input-checksum_sha256, input-checksum_crc64nvme, input-length, input-trailer
- storage::ecfs::legacy_edge::object_write::native_put_tests::owned_response_projection_preserves_status_headers_and_all_checksums: output-e_tag, output-checksum_crc32, output-checksum_crc32c, output-checksum_sha1, output-checksum_sha256, output-checksum_crc64nvme, output-status, output-headers
- app::object::put::tests::put_read_quorum_failures_are_normalized_at_each_stage: error-status, error-message, error-source
- app::object::put::tests::put_commit_lock_failure_keeps_request_operation_in_spawned_owner: error-headers
- app::object::put::tests::cancelled_put_request_completes_post_commit_publication: completion
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_mode_without_retain_until_date: lock-pairing
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_retain_until_date_without_mode: lock-pairing
- storage::ecfs_test::tests::test_parse_object_lock_retention: lock-presence
- app::object::extract::tests::snowball_member_mtime_requires_member_replication_authorization: archive-replication
- table_catalog::tests::put_object_reserved_prefix_guard_rejects_broken_native_delegation: guard-reason
- object_write::legacy_owned_put_preserves_refusal_order_checksums_and_short_body: refusal-priority, output-headers, output-checksum_crc32
- app::object::put::tests::put_edge_counts_polled_requests_once_before_usecase_refusal: count-missing, count-double
- app::object::extract::tests::snowball_owned_member_privileges_gate_storage: archive-tagging, archive-retention, archive-legal_hold
Existing expected values are unchanged.
Not verified: CI, Linux RDMA execution, performance, and gateway native-frame trailer consumption through this business core. The native trailer adapter itself is covered; gateway integration is a separate increment. The local suite uses a 4 MiB stack and makes no claim about default-stack behavior.
* feat: S3-over-RDMA with NIXL and cuObject transports
A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL.
Applied onto main from #9305 (head 0f3c19a0f); the stacked #9302-#9304
commits are already on main as eca95f297. Conflicts were resolved where
this change met newer main: crates/ecstore/src/api/mod.rs (kept the
internode-transport exports, added the nixl module),
crates/rio/src/http_reader.rs (internode_http_request uses the pooled
get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs
(write_authenticated_put_file stays a plain fn returning a future, now
pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart
branch moved into multipart_usecase/upload_part.rs after the
per-operation split), Cargo.lock (main's lock plus only this change's
new packages).
* fix(rdma): build the RDMA crates without vendor SDKs and pass guards
Maintainer changes on top of the contributed S3-over-RDMA work:
- rustfs-cuobject: cuobject is an optional, default-features = false
dependency behind a new `server` feature (enabled by the rustfs
`cuobject` feature); the crate root is #![cfg(feature = "server")], so
without the feature it exports nothing and a workspace build needs no
NVIDIA cuObject SDK.
- The NIXL agent handle (OnceLock, install_nixl_server, nixl_server)
moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app
init installs it. The layer-dependency guard rejects infra -> app.
- New code reaches ECStore through a cfg-gated `ecstore_rdma` facade
module and s3s types through the existing re-export hubs
(architecture-migration and s3s-footprint guards).
- SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window
the unsafe-allowance guard checks.
- transport_config_known_backends_are_current_oss_values loops over
KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name
itself.
- New non-required workflow rdma-feature-check.yml runs
`cargo check --locked -p rustfs --features nixl` on Linux (paths
filter, weekly, manual), registered in scheduled-validations.json.
* fix(ecstore): keep NIXL internode error messages constant
The three io::Error::other(format!(..)) sites in nixl_internode.rs
(read refused, write refused, write byte-count mismatch) now return
constant messages; the status, response body length and byte counts go
to a debug! line in the same place, the level this file already uses for
a failed NIXL peer call. No response body is logged, as on the HTTP
internode path.
The error-format ratchet exists because per-disk formatted messages make
otherwise identical errors compare unequal and split the quorum
reduction; constant messages keep them in one bucket. A typed transport
error would carry the detail without that problem; this takes the
mechanical route.
* fix(ecstore): flush staged NIXL writes before waiting for buffers
* fix(rdma): build rustfs-nixl only on Linux
nixl-sys's stub build links -lstdc++, which macOS does not ship, so
`cargo test -p rustfs-nixl` (and a workspace test run) failed to link on
macOS. rustfs-nixl's dependencies are now Linux target dependencies and
its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl`
feature code that was not already Linux-gated (the storage/rpc nixl_read
module, open_shard_contained, the internode route arms and the facade
module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux
builds are unchanged: the workspace tests still build and run the
crate's tests through the stub.
* ci(rdma): alert on scheduled failures of the RDMA feature check
The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes.
* fix(rdma): put rustfs-nixl behind a sys feature
nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device.
---------
Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Rain Jiang <rain-jiang@outlook.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
* feat(obs): split PUT commit and remote rename timing
Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* bench(filemeta): measure inline rename RPC encoding
Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(bench): use exact inline rename shard fixture
Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(obs): measure remote rename field bytes
Record detail-gated JSON and msgpack field lengths after successful remote
rename request encoding so msgpack-only rehearsals can prove sender behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preserve partial carry drain window
Use the production drain window for real partial-bucket checkpoints so CI storage latency cannot truncate source-cache persistence.
Align the certified partial-cycle assertions with carried-value semantics while retaining mixed-coverage and final convergence checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Resolve empty-bucket requests through the existing storage lookup at the
legacy edge. Keep authorization, operation counting and nonempty bucket
conversion unchanged. No gateway bucket-name contract changes.
Add two tests covering two positive and two negative cases: protocol root,
ordinary bucket, missing bucket and uninitialized storage. Check response
status/headers and one count per request. Disabling the compatibility
branch makes both tests fail; restoring it passes both after recompilation.
Verified on 4182d55d1: focused tests, gateway Clippy, and a rebuilt binary
with libunftp 0.25 using FTP_TLS: login 230, root and ordinary CWD 250,
missing bucket 550. Default Clippy and script-tests passed on 925e5d0c0.
After integrating a279b4bb6, gateway Clippy and prepush-quick passed.
The wire probe and focused tests were not rerun after that integration.
Use shared request authorization and native completion responses across both S3 stacks. Keep storage responsible for final commit and replay decisions.
Update jsonwebtoken to 12.0.0 and libunftp to 0.25.0, including the JWT key construction API changes required by the new release. Refresh the lockfile through two requested cargo update and cargo upgrade rounds.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
perf(grpc): prewarm isolated bulk channel pools
Extend the existing opt-in peer prewarm to the configured bulk channel pool. Control prewarm remains one channel, isolated bulk prewarm fills the per-peer pool, and legacy shared mode reuses the control cache. Add a cache-reuse regression for the legacy mode.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): coalesce shard, integrity index, and commit syncs
- Enable sync-on-close (WriteMode::TruncateSync with sync_parent_dir: false)
in LocalDisk::create_file for multipart staging writes so part.N and
part.N.integrity.<gen> run fdatasync before POSIX_FADV_DONTNEED page-cache
reclaim and register in PRESYNCED_TMP_FILES without polluting
PRESYNCED_TMP_DATA_DIRS.
- Keep LazyLocalFileWriter open across write_bitrot_shard when shard integrity
is enabled and coalesce the .idx sidecar creation, write, and fdatasync into
poll_shutdown via set_pending_integrity_index + MultiWriter::finish_and_shutdown,
skipping duplicate PreparedIntegrity::write file creations for prewritten disks.
- In fused_attempt (commit.rs), sync prepared_metadata_source (xl.meta) before
os::fsync_dir_std(src_data_path) when src_data_path is presynced so the
active journal transaction commits both xl.meta data extents and the staged
data directory entry in one flush instead of two.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
* perf(ecstore): skip redundant .idx parent creation and header re-read when presynced
- Mark parent_exists = true in StdBackend::open_write for temporary .integrity.
and .proof. sidecars so open_lazy_local_file_std opens the file directly in
its existing staging directory before falling back to create_dir_all_std.
- Skip reopening and re-reading the 32-byte .idx header in
LocalDisk::prepare_part_transaction when take_presynced_tmp_file confirms the
sidecar was just written and synced by LazyLocalFileWriter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
* fix(ecstore): validate presynced multipart integrity indexes
Validate the proof file type, length, and header before preparing a multipart transaction. Use a presync receipt to skip the redundant file sync after validation.
Cover valid, truncated, and mismatched-header indexes carrying presync receipts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): wait for tail rename drain in inline_put_direct_commit_handles_post_encode_rename_failures
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
Share HEAD authorization and its use case across both request edges. Keep
stored and proxy ETag parsing distinct, preserve conditional headers and
version bytes, and finish auditing before terminal header encoding. Pin
gateway to 1afff6a for the measured 304 response profile.
Verified 785 affected and embedded tests, then 12 tests after test-only
module/import cleanup. Both all-target Clippy configurations, script tests,
wiring and prepush checks passed. Tests cover malformed stored tags,
separate replication denial, conditional status conflicts and ordinary
success controls; the proxy oracle checks 779 spellings.
Observed 39 effective isolated faults: authorization/delegation, execution,
metadata exposure, stored/legacy/proxy tags, raw header escaping/repetition,
framing conflicts, length/counting, range boundaries, checksum and storage
class, ordinary completion, terminal code/status/default resolution, and
each of the seven 304 fields. An ineffective fallback mutation was retained
as a failed experiment and replaced with an explicit-status-only fault.
The 785-test base was c13232996; final module tests and Clippy ran on
fc22c11d1. No GetObject migration added. Cross-binary pinned acceptance,
remote proxy/ODM integration, SSE-C real-storage wire and performance were
not verified. Existing assertions were not weakened.
* fix(health): bound quorum probes and record timeout observations
Collect fresh local and remote observations concurrently and finish once
each configured set has sufficient storage and lock quorum evidence.
Validate configured peer identities and disk slots before counting votes.
Apply the cluster deadline to cache and singleflight waiting, retain flight
ownership through timeout publication, and record timeout observations once.
Cover slow peers, unavailable quorum, identity validation, cancellation,
cache ownership and compatibility with focused regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(health): coalesce probes and count quorum incrementally
Share the completion generation with overlapping uncached requests while
retaining fresh observations for later requests and bounded flight ownership.
Build validated topology thresholds once and count unique disk and lock
votes incrementally instead of rescanning collected inventories.
Honor the existing inventory metrics flag through RPC and storage owners.
Health probes omit operation metric snapshots while retaining fresh disk
identity, capacity and runtime checks. Move owned inventories across local
aggregation layers and preserve the full-metrics administrative APIs.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(health): cover failed uncached flight recovery
Cover a later waiter sharing its overlapping failed round before an independent request starts a fresh successful observation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(health): group lock membership in one endpoint pass
Group configured lock members by set in one pass and reject impossible set counts before allocation. Retain per-set thresholds and add a bounded malformed-topology regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(health): publish startup topology in gating fixture
Publish the shared four-disk fixture's endpoint topology before initializing disks, matching storage startup. Cluster health probes can then validate fresh quorum observations while the readiness test exercises S3 admission publication.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(obs): split PUT commit and remote rename timing
Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* bench(filemeta): measure inline rename RPC encoding
Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(bench): use exact inline rename shard fixture
Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Select gateway-stack for replication and protocol gateway legs and use matching build and run features. Legacy feature selections remain unchanged.
Verified on 0460ad63a: 20 e2e binary tests, full make script-tests, actionlint with shellcheck, CI workflow check, and prepush quick checks. The workflow contract exercises four passing build/run selections, four propagated failures and four mismatched-feature rejections. Removing either gateway feature, mismatching either run selection, and emptying the matrix each made the contract test fail.
No Rust compilation or live nightly execution was performed; actual gateway startup and suite outcomes remain to be observed in CI.
* feat(admin): add caller-bound effective permission queries
Expose bounded advisory checks through the existing IAM, bucket-policy, table, and KMS authorization paths. Preserve three-state decisions for incomplete request facts and bypass publication guards only for internal probes while real operations continue to authorize independently.
* fix(auth): preserve effective permission probe boundaries
Use canonical action scopes and verified signing facts, keep unavailable conditions unknown across table metadata gates, and resolve table resources without catalog repair. Route admin test dependencies through existing facades and keep legacy S3 error usage within its migration baseline.
Observe active, completed and retained multipart ownership without replay, recovery, acknowledgement or reclamation. Only active uploads supply preflight metadata. Completed and retained records require validated physical evidence; completion still owns publication.
Preserve native inspection errors before reporting absence. Completed quorum retains precedence, and UnsupportedDisk keeps its compatibility behavior.
Validation: 17 binding tests and all 7,453 ecstore tests passed with four workers and no skips. Both default and internode-transport,test-util all-target Clippy passed with warnings denied. Formatting, script tests, test wiring and prepush checks passed.
Combined mutations of inspection error propagation, read quorum selection and UnsupportedDisk handling produced four intended runtime failures while two controls passed. Sources were restored by hash and actually recompiled before the full final run.
Application and CompleteMultipartUpload integration are not included.
Reuse the full CI job's pinned mc and Vault provisioning, version checks,
and iptables capability warning before both e2e-full nightly stack runs.
Keep existing test filters, profile selection and report-only policy.
Validation on 4053e694c: actionlint with shellcheck, YAML step equivalence,
test wiring, CI workflow contract, prepush-quick and full script-tests.
The first script-tests attempt hit an unchanged scanner fixture JSON read;
the fixture control and final full rerun passed without source changes.
This does not establish the cause or repair of that fixture failure.
Executed the fixed Linux mc and Vault downloads, SHA-256 checks and version
commands in a private prefix. No test assertions were added or changed
(negative/positive 0/0); no mutations or Rust builds were needed.
The complete nightly workflow and E2E cases were not run for this change.
Keep SSE configuration, requests, materials and read response fields on
native strings. Project legacy enums at the DTO boundaries and retain the
existing validation order, bucket defaults and KMS authorization behavior.
Carrier diagnostics omit customer keys and encryption key material.
Production lines: +186/-166. Add 5 negative diagnostic tests, 0 positives.
Do not implement or register an owned PutObject operation.
Validation: fmt, default and gateway-stack all-target Clippy, 777 affected
unit tests, script-tests with aligned Python, test wiring and prepush-quick.
Tests ran on 9ea8e8d37f9670f8da3dc5cb6ed5fe6e6e9bb592; rebased onto
a9dd5f0fd1 with no overlapping files or pin change.
Mutation witnesses (8 sites, 9 failing cases): restore derived Debug for
SseTypeV2, EncryptionRequest, DecryptionRequest, EncryptionMaterial and
DecryptionMaterial -> all five native_*debug_withholds* tests fail; change
unknown default mapping -> shared and copy unknown-default tests fail;
admit an unsupported algorithm -> rejects_invalid_sse_algorithm fails;
project AES reads as KMS -> classification_withholds_the_wrapping_key_for_sse_s3_reads fails.
Restore exact source hashes, touch the SSE source and rebuild: 777/777 pass.
Not verified: full workspace, optional rio-v2/FIPS builds, or CI for this commit.
No intended legacy protocol change; diagnostics intentionally omit key fields.
Moka maintenance can logically evict an entry while epoch reclamation still
owns its allocation. Check a cache miss, tear down the cache, and observe
entry destruction through Weak before asserting response ownership and the
final reservation release. Keep the original five-byte and zero-byte checks;
only their physical-owner precondition changes. No runtime behavior changes.
Validated on 4bb272ed6 with default cache/runtime-memory features on macOS:
140 package tests passed, all-target Clippy, formatting, test wiring, and
prepush quick checks. Linux has one extra cgroup-only test, not run locally.
Mutation checks: copying response bytes breaks the live claim assertion;
skipping reservation release breaks the final zero assertion; retaining an
entry breaks the Weak assertion; skipping cache removal breaks the miss
assertion; detaching the response after eviction breaks the post-teardown
five-byte assertion. All failed through their intended runtime assertions.
No tests added or removed. No cache policy or reclamation timing changes.
Reuse the outer transport layer's deployment setting in the gateway builder. The existing false default allows customer keys over plaintext with the gateway's explicit acknowledgement; true keeps both key sides strict. Transport facts remain per request.
The shared signer fixture now accepts headers before signing. The new test exercises 45 scenarios (38 refusals, 7 test-handler successes), checks signed SSE headers and raw-key handoff, and runs the inner plaintext gate separately from the outer layer.
Validation on ebe31dfdb5: fmt, all-target Clippy with and without gateway-stack, test wiring, and 63 gateway-stack/handler/transport tests passed. Production diff: +29/-14 lines.
Fault controls fail ssec_deployment_policy_and_key_handoff: forced strict/allow, false secure transport, re-signed key/input rewrites, skipped handler, response-key echo and wrong refusal code. Separate fixture checks reject anonymous allowance and unsigned SSE headers. The original strict selector fails a valid signed plaintext request with InvalidRequest after a TLS handler control succeeds.
Not verified: copy-source key wire handoff, storage use cases, real Served IAM authorization, network TLS termination, cutover acceptance or CI. No production operation was added.
Restore only bindgen 0.73.2's syn edge to the already locked syn 3.0.10.
Its >=2,<4 range permits this version and prettyplease 0.3 requires syn 3.
No package version, checksum, manifest, feature or formatter changes.
Regression evidence: the frozen lock failed the FIPS tls-runtime test
build with E0277 in bindgen's prettyplease::unparse call. The corrected
locked graph builds and runs provider_policy_isolated in both modes.
No new tests or assertions; no behavior mutation was needed for this
compile failure. Product source lines changed: 0; lock lines: +1/-1.
Verified: tls-runtime all-targets Clippy and provider_policy_isolated
(1 passed in each mode), default on ebd4bb2ba and FIPS on f2a43376d.
The final rebase only adds two test-guard fixes. cargo tree --locked,
fmt, diff whitespace and prepush-quick passed on the final base.
Not verified: full RustFS/workspace suites, the remaining nightly FIPS
consumer matrix, or a new Linux CI run. No runtime TLS policy changed.
Follow DeleteObject's typed and legacy entrypoints through the shared fields adapter into its owned implementation. Preserve the existing mutation guards and key/prefix/store ordering checks.
Let the TRACE mutation probe locate either ListObjectsV2 parameter shape exactly once, then require the real logging guard to reject DEBUG.
Validation on ebd4bb2ba8: default and gateway-stack all-target Clippy, three focused nextest tests, full make script-tests, split probe (61 checks), test wiring, shell syntax, and prepush-quick passed. Product lines: +0/-0.
Source-guard evidence: one valid source control and 11 negative fixtures; missing methods, bypassed delegation, missing guards, and reversed ordering fail with the intended diagnostic. The old checker fails on current source. Disabling each typed, legacy, or fields delegation check makes the fixture test fail. TRACE marker evidence: two accepted shapes and five rejected missing/debug/duplicate shapes; TRACE-to-DEBUG mutation is rejected by the real logging guard.
No runtime code or logging policy changes. Full workspace and Linux CI were not run locally; source-guard mutations do not prove runtime authorization.
Move ListObjects, ListObjectsV2, ListObjectVersions and ListMultipartUploads
to owned use cases and gateway handlers. Keep legacy encoding, stored ETags,
optional facts and cursor boundaries at their existing edges. ListParts and
multipart create/complete/abort are outside this increment.
Non-test changed lines: +973/-659, excluding test modules and test-only
imports/reexports; comments and whitespace are included.
Validation: 282 focused tests passed on ed3a9e670d,
including all storage::access::tests and both stack refusal matrices. The four
listing matrix rows exercise 32 refusals and 8 allowed requests across both
stacks; these are migrated coverage, not 40 new tests. Stored-key, entity-tag,
encoding and pagination tables provide additional compatibility coverage.
The 1024-byte stored-key positive runs on Linux; macOS attempts the write and
asserts FileNameTooLong and absence because of its full-path limit.
Final RustFS all-target Clippy passed with and without gateway-stack on ed3a9e670.
The affected e2e_test package Clippy passed on e3858df4fa.
After rebasing over the adjacent response-helper change, one gateway-stack
all-target cargo check passed on ebe31dfdb5.
Formatting, test wiring, seam validation and prepush-quick passed.
Ten effective local mutations were reviewed on d5f640243952b571111bb3c97a98850f04a92172:
- Count: key_count_includes_objects_and_prefixes_and_a_keyless_entry_is_not_an_object.
- Marker: list_objects_next_marker_follows_the_cursor_then_the_last_raw_entry.
- Key floor: every_key_the_store_can_hold_is_represented_unchanged.
- Nil version: versions_delete_markers_and_markers_are_answered_raw.
- Upload fields: list_multipart_uploads_output_maps_uploads_and_prefixes_raw.
- XML escaping: listing_entries_are_encoded_per_segment_only_for_url_and_tags_are_escaped
and the_legacy_stack_lists_every_stored_entity_tag.
- Three direct authorization mutations (ListObjectsV2, ListObjects,
ListObjectVersions): the_refusal_matrix_holds_on_the_gateway_stack.
- Shared multipart authorization: both stack refusal matrices; this also covers
ListMultipartUploads delegation. Its separate direct early-return experiment
failed for a root request and is excluded from authorization evidence.
All temporary experiment code was removed before the final product build.
Not verified: full dual-stack differential run, full workspace CI, or Linux
runtime execution locally. Bootstrapping operations for the differential suite
are not all migrated yet. Historical wire baselines cover a narrower surface.
Add Answered<Output> for use cases that return a status override or extra
response headers. The legacy seam preserves those fields on S3Response;
the gateway helper passes them to Resp through its supported API. Bare
outputs share the seam implementation and retain their default response.
No operation consumes the carrier yet. Non-test changes: +87/-4 lines.
Validated on ed3a9e670d:
- cargo clippy --locked -p rustfs --all-targets -- -D warnings, both
default and --features gateway-stack.
- cargo nextest run --locked -p rustfs --features gateway-stack --lib
--no-tests fail -E 'test(storage::ecfs::legacy_edge::tests) |
test(server::gateway_handlers::tests)': 23 passed.
- cargo fmt --all --check, prepush-quick, test wiring, CI workflow
contract and validation registry passed.
Three added tests cover carrier values and defaults (0 refusal / 3
positive tests). Representative mutations discarded status and headers
on both edges: an_answered_status_and_headers_reach_the_legacy_response_unchanged
failed its status assertion; the corresponding gateway answer test
failed its status/header tuple. Both tests ran and failed. Restore hashes
matched and sources were touched before the passing ordinary rebuild.
Not verified: wire header delivery or encoder-owned/header-collision
refusals; these belong with the first HeadObject/GetObject consumers.
The legacy header assertion was not independently mutated. No new
operation, authorization path, wire claim, or legacy operation behavior
change is included. Full workspace tests and CI were not run locally.
Refs: rustfs/backlog#2767, rustfs/backlog#2734
Move DeleteObject to an owned use case and share authorization between the legacy and gateway edges. Keep the legacy request fields at the entry so validation and audit run in order. Give the refusal matrix its own bucket.
Tests: 168 passed on e97dc74d9; the restored mutation control passed 9 tests before an equivalent test-only Clippy fix. RustFS Clippy passed in both configurations; e2e_test Clippy and test wiring passed. The recursive authorization table covers 24 refusals and 16 allows. The header and caller tables add three refusals and five allows across both request forms; the wire cases add key, version and caller refusals.
Mutations: bypassing the primary, force, authenticated-caller, or governance decisions turned their refusal cases into allows. Skipping descendant delete, replication, or bypass decisions turned the respective refusal rows into allows. Removing context restoration failed both allowed and refused rows. Disabling the helper removed the invalid-key audit. The gateway root matrix stayed green for each mutation except the primary decision, where its root row passed and the unauthorized rows were served.
Not verified: the full dual-stack differential scenario requires PutObject and bucket versioning on the gateway stack. CI has not run for this candidate.
Serve object ACL reads and writes, legal hold and retention reads, tagging reads and deletes, and torrent requests through shared use cases and authorization.
On base af93bf466, 131 object metadata and access tests, four refusal matrix tests, and five targeted behavior controls passed. Mutations covered each authorization, version actions, retention UTC conversion and range, ACL responses, tags, lock checks, missing objects, version parsing, and store initialization. Both package clippy configurations passed.
After rebasing onto e97dc74d9, the gateway-stack all-target check and all four refusal matrix tests passed. Full workspace tests and differential scenarios were not run.
Use the owner ID RustFS reports for gateway expected-owner checks. A missing bucket is still resolved by its handler; no per-bucket owner lookup is added.
The gateway wire table covers six refusals and four successes across ordinary, missing, and reserved-shaped buckets and ListBuckets. The same table runs against the legacy stack. The source unit test covers a reported owner and an unavailable owner.
Mutations: removing the source makes the gateway table fail; returning the wrong owner makes the gateway table and owner unit test fail; returning the wrong owner only for the missing bucket changes its matching-owner response from 404 to 403; returning an owner when none exists makes the owner unit test fail.
Verified against base e97dc74d9: both RustFS clippy configurations, three focused tests, test wiring, and prepush quick checks. Successful SDK responses are checked as success, without asserting their HTTP status. Full workspace tests and CI were not run locally.
Verified both bucket guards, default and gateway-stack Clippy, formatting, test wiring, and prepush checks. Removing the CORS notification and flipping its maintenance flag each failed the guard; the restored control passed.
Tests ran on e97dc74d9 before rebasing over unrelated KMS changes at 3c59e4f30. Full workspace tests were not run locally.
* fix(kms): tolerate Vault reads that lag the client's own writes
A Vault or OpenBao node that answers reads from its own copy (an OpenBao standby, a Vault Enterprise performance standby) can serve the state from before a write for a short while after acknowledging it, and can refuse a write it is able to reject from that copy without forwarding it.
- Create a KV2 key in one create-only write that carries usage, description and tags, instead of writing the record and reading it back to attach them.
- Pause with jittered backoff before re-reading after a lost check-and-set write, and allow five attempts instead of three, in both backends.
- Remember for five seconds what this client last wrote or deleted at each record and re-read an answer that is older. The last answer is returned once the attempts are spent, so this only delays a read.
- Retry a Transit key delete that is refused because the deletion switch is not visible yet.
* docs(kms): describe running the Vault backends against OpenBao
* fix(kms): keep the remembered write version from moving backwards
Two writes of one record by the same client can have their acknowledgements processed out of order. Recording the lower version last made it the floor again, so a read served from that older version was accepted. The marker is now updated atomically and only raised within a record's lifetime; a write after this client's own delete starts the count again.
Route the operation through the owned use case and shared authorization.
The tests cover two bucket names and a missing bucket on both stacks.
Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), and test wiring. Bypassing location authorization made the
gateway refusal matrix fail for anonymous and no-policy callers. The main
legacy comparison passed. Non-default regions were not verified.
Share the use case and authorization across both edges. Preserve the
force-delete header rules and operation accounting.
Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), test wiring and main legacy comparison. The force-delete table
covers eight refusals and six successful cases; additional tests cover header
precedence and anonymous invalid input. Bypassing DeleteBucket authorization
made the gateway refusal matrix fail for anonymous and no-policy callers;
bypassing ForceDeleteBucket made the false-header authorization test fail.
Site replication and multi-node deletion were not verified.
Route PutBucketCors and DeleteBucketCors through owned use cases and shared authorization, preserving audit, storage, and legacy edge behavior.
Coverage includes eight refusal and two root matrix scenarios per stack, a wrong-resource refusal, missing-context refusal, and CORS persistence cases. The scoped suite passed 125 tests on 05554bee0. Bypassing Put or Delete authorization failed the gateway refusal matrix; skipping each storage mutation failed the CORS persistence test; allowing a missing context failed the access unit test.
Not verified: full workspace CI and the missing-bucket plus invalid-configuration error ordering. The delete audit status remains 200 while the response is 204.
* fix(replication): stop re-sending existing objects every scan cycle
Existing objects on a site-replication source were re-queued and
re-sent on every scanner cycle, so the replica kept rewriting the same
versions and its reported bucket size grew while the object count stayed
flat.
Two defects combined:
- The per-target resync stamp was computed by the worker but never
written back to the source object, so the resync decision kept
returning "replicate" for every object older than the reset date. The
status writeback now persists the stamp under both internal metadata
prefixes, and a successful full-object send of an existing object is
stamped as well.
- The target HEAD only reported the tag count, so the tag comparison
always saw a difference for tagged objects and fell back to a full
re-send. The replication HEAD now sends
x-amz-tagging-directive: ACCESS and the server answers with the tag
set under the same authorization as the tag count. Targets that do not
return the tag set still compare as different, which costs one extra
send instead of a loop.
Also drop replicate_all_stat_options, which built the directive but was
never passed to the HEAD call.
Refs #9354
* test(replication): use relative paths in the resync stamp test
Read stored entity tags with the existing legacy spelling rules before
constructing gateway values. Keep the original control-character coverage.
This prepares the shared helper without adding an operation caller.
Tests: four focused tests passed on 05554bee0, covering eleven spellings,
weak empty tags, control characters, and invalid header values. The table
covers seven malformed or boundary spellings and four ordinary spellings.
Mutations: weak, quotes, bare weak prefix, nested quotes and plain values
failed the spelling table; control characters failed the preservation and
header tests; empty weak tags failed the weak-tag test; removing the version
header failed the header test.
Not verified: operation-level wire responses and full workspace tests.
Share replication header, write ETag, and staged multipart authorization across request forms, preserving the existing decisions and marker cleanup.
The parity table covers eight denials and six allowed cases. Mutations of each authorization call, replication marker and header source, ETag header source, and staged marker cleanup on success and failure all failed the parity test.
Tests and both Clippy configurations passed on 05554bee0. The gateway handlers are not included in this increment. The script test suite did not pass locally: four unchanged scanner fixture tests failed twice with expired fixture deadlines.
* docs: add Docker Swarm with the SwarmCLI chart to the README quickstart
Option 7 links to the Docker Swarm guide on docs.rustfs.com and to the
community rustfs chart, noting that Eldara Tech maintains both.
* docs: point README Option 7 at the SwarmCLI chart guide's new path
The docs guide moved from Container to Cloud Native as SwarmCLI Chart.
* docs: name the SwarmCLI chart's key features in README Option 7
Secrets, versioned upgrades and rollbacks, Traefik HTTPS; and that its
Console is off and rustfsadmin refused, unlike Accessing RustFS below.
* docs: call the SwarmCLI chart open-source rather than community in README
It is maintained by Eldara Tech, which 'community' contradicted.
* docs: add the SwarmCLI chart's distributed mode to README Option 7
The chart guide now covers distributed, erasure-coded clusters as well
as a single-node service, so Option 7 no longer says single-node only.
After every successful write, enqueue_immediate_expiry took the bucket
metadata-transaction read lock and read the bucket incarnation before
checking the cached bucket metadata that decides the outcome. A cached
entry without any lifecycle document can never enqueue an expiry.
Return early in that case. Buckets with lifecycle rules, unreadable
lifecycle documents, a pending creation commit or no cached entry keep
the authoritative path.
Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Move test helpers to dev-dependencies and clean unused build
requirements and workspace metadata.
Use profiling compatibility entrypoints for probe-free crates and
verify the application's hotpath, allocation, and CPU feature graphs.
Validate the affected libraries, FTPS/SFTP test targets, and 1729
existing library tests.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): persist restartable format bootstrap authority
Persist all-drive creator plans before format publication and resume exact deployment and drive slots after interrupted fresh formation. Consume authority durably before startup publication and validate regular control files on full and streamed reads.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): finalize format bootstrap once
Order bootstrap retirement before initialized identity publication, preserve established expansion preflight repair, and require followers to observe committed topology and complete identities.
Publish bootstrap controls through one authenticated atomic RPC on the dedicated fsync lane. Keep cancellation-safe cache ownership in the native operation and retain interoperable replacement healing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): fence anchored format publication retries
Select the durable first-slot plan before fanout and complete immutable anchored plans across retries. Retain physical-root namespace ownership through native publication, retirement and cache invalidation, rejecting conflicting or retired control records.
Reuse main's unary RPC lifecycle recorder for validated outcomes and exact request, byte and cancellation accounting.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(test): align Windows and test-util compilation gates
Match physical publication-hook tests to their non-Windows provider. Compile pool-capacity overrides with their test-util consumers, and keep alias-list mutation inside the Unix branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): target native format publication paths
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* style(rpc): format merged request imports
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* test(ilm): account for completed immediate transitions
PUT and lifecycle configuration updates can enqueue transitions before a manual scan. Assert complete per-object accounting and verify a repeated scan skips the committed transition without duplicating the remote object or changing GET content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ilm): stabilize continuation eligibility across restart
Wait for configuration-triggered transitions to commit before testing opaque continuation tokens. Assert both pages skip exactly one completed object, never enqueue work and preserve the two remote objects across restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Strict recoverable-delete activation renamed the prepared transaction
directory on the disk's single-worker foreground lane, which also serves
metadata reads, and then queued its directory fsyncs as a separate
fsync-lane job. A stalled activation rename blocked unrelated metadata
reads on the same disk.
Run the rename, the fsync of the pinned object directory and, for a
missing original, the ancestor fsyncs in one fsync-lane closure. The set
and order of fsyncs are unchanged; relaxed and none modes keep the
previous rename path.
Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(deps): refresh Prost and WebPKI dependencies
Run cargo update --verbose and cargo upgrade --verbose twice from main.
Upgrade the Prost family to 0.14.6 and rustls-webpki to 0.103.17.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): document shard buffer unsafe allowance
Document the arena block validity and exclusivity contract beside the
unsafe_code allowance to satisfy the existing guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test: avoid typo false positive in PUT body chunks
Use hell and o as fixture chunks that concatenate to hello.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): upgrade rustfs-erasure-codec to 9.1.0
Run cargo update --verbose and cargo upgrade --verbose again.
Resolve rustfs-erasure-codec 9.1.0 and refresh the workspace lockfile.
Validate the integrated legacy GET producer stack fix with all 28
reported scenarios at a 2 MiB stack in both nextest profiles, plus
codec compatibility cases and the producer future-size regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): follow moved request denial audit calls
Inspect both legacy access and owned object authorization production
sources when checking request-level denial audit wiring.
Add regression coverage for a moved call, removal of that call, and
audit calls confined to test code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(heal): add bounded MRF management and streaming audits
Publish lightweight responsibility summaries and expose authenticated, identity-bound cluster administration without reading object payloads. Stream audit-only reads while retaining staged verified bytes and full target readback for migration.
* test(admin): keep MRF coverage within gateway boundaries
Exercise owned MRF handlers through the shared admin test seam and keep wire-level authentication assertions in the router tests, without introducing another direct s3s consumer.
* test(e2e): complete MRF lifecycle lock fixture RPC
* docs: clarify integrity job resume refusals
Explain that a failed lock acquisition does not prove a live worker, and require job-status reconciliation before retrying an unknown request outcome.
* test(admin): include MRF cluster routes in route matrix
Keep the exhaustive admin route matrix aligned with the four registered cluster management endpoints, including the concrete node and responsibility ID probe.
* fix(ecstore): make retired part settlement idempotent
Resolve a failed source admission against the canonical transaction while
retaining SourceRW ownership. Treat proven absence as already settled when
cached directory descriptors still refer to an upload retired into trash.
Cover a real pinned part transaction through certified Abort, existing closed
transactions and dangling aliases. Preserve active settlement barriers and
rooted no-follow path validation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound offline OPA fixture future stack usage
Construct offline storage futures in a separate noninlined frame before
boxing them. Keep the bucket-tag fixture within the normal test thread
stack while retaining real metadata writes and authorization assertions.
The affected debug test and the full STS/OPA module pass with a 2 MiB
thread stack after the original CI smoke stack overflow.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): isolate OPA contract on a dedicated stack
Run the bucket-tag OPA contract on a dedicated 32 MiB thread with a current-thread Tokio runtime. Nested ECStore initialization poll frames can overflow the default test stack even when the outer future is boxed.
Remove the superseded future-boxing helper. Validate all eight STS/OPA tests and targeted runs with 1 MiB and 512 KiB entry stacks on Linux ARM64, plus all-target Clippy and formatting checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(lock): extend contended writer retries within caller budget
Keep the initial probe, shared locks and batch waits at their existing peer limit. Allow later exclusive contention rounds to use a bounded longer wait so read holders outlasting one short round do not repeatedly erase writer progress. Preserve the caller deadline, quorum and grant cleanup, and cover the original failure with real LocalClient grants.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(admin): expose physical holder counts in top locks
Serialize the node-local physical holder count from the existing fast-lock snapshot for each resource/version/mode/owner cohort. Keep row totals and lease-metadata validation, and cover repeated owners, stale leases and the actual JSON field.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(admin): clarify node-local holder sampling
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
GetBucketPolicy, GetBucketPolicyStatus and GetBucketAcl are served by the
gateway stack and by the legacy stack through one owned use case each. The
legacy access hooks call authorize_bucket_request, the function the gateway
edge calls; the owner's bypass of a bucket-policy Deny for GetBucketPolicy stays
in the shared authorization core. BucketAcl gets its use case in
bucket_usecase/acl.rs: the answer (one owner, full control, a grantee carrying
no identity) is unchanged.
Not built: GetBucketNotificationConfiguration, the four mutations and
PutBucketAcl (they wait for shared helpers).
Removed code: build_get_bucket_acl_output in storage/s3_api/acl.rs, replaced by
the use case.
Removed tests: test_build_get_bucket_acl_output_contains_full_control_grant_and_owner,
covered by the_gateway_stack_answers_the_bucket_acl and
the_legacy_stack_answers_the_bucket_acl (one owner, full control, grantee with
no id and no display name, NoSuchBucket for a missing bucket) in
tests/embedded_gateway_stack_test/bucket_access.rs.
Tests: refusal-matrix rows for the three reads on both stacks; NoSuchBucketPolicy,
the verbatim policy document, the owner reading a policy that denies everyone,
IsPublic following the stored policy, and the ACL answer, on both stacks;
use-case tests ported to the owned signatures.
Mutations: the gateway authorize answering Ok(()) in each of the three reads
turns that operation's anonymous and no-policy rows of the refusal matrix red.
Legacy answers compared with main: see the previous commit (40 rows, identical,
including the missing content-type of the stored-policy answer).
GetBucketVersioning is served by the gateway stack, and by the legacy stack
through the same owned use case (gateway input and output types). The legacy
access hook now calls authorize_bucket_request, the function the gateway edge
calls. The stored status is carried verbatim as the gateway status enumeration,
so a stored value is never dropped. As before, MFADelete is not answered.
Not built: PutBucketVersioning (it waits for the shared bucket-configuration
mutation wrapper and the owned audit call).
Tests: refusal-matrix row on both stacks (anonymous and a user without a policy
are refused, root is served and counted once); GetBucketVersioning answers every
stored bucket name the SDK reaches (a-s3alias, sthree-x, x--ol-s3), a stored
Enabled status, NoSuchBucket for a missing bucket, and the 400 InvalidBucketName
both stacks give xn--abc, on both stacks; the ported use-case test.
Differential cases: an unconfigured and a suspended bucket.
Mutations: the gateway authorize answering Ok(()) turns the anonymous and
no-policy rows of the refusal matrix red; dropping the status from the use case
turns the stored-status assertion red (left None, right Enabled).
Legacy answers compared with main (status, headers and body; the SDK output for
successes): 40 rows, for plain-bucket, sthree-x, a-s3alias, x--ol-s3 and a
missing bucket, each unconfigured and configured, for GetBucketVersioning and
the three reads of the next commit. All identical.
Batch A3 of rustfs/backlog#2765, second increment: the gateway stack serves
GetBucketLifecycleConfiguration, GetBucketReplication and
GetObjectLockConfiguration through the owned use cases from the previous
commit.
- `impl Served` for the three operations and their rows in the A3 family. Each
`authorize` calls `authorize_bucket_request` with the action its legacy hook
asks (GetBucketLifecycle, GetReplicationConfiguration,
GetBucketObjectLockConfiguration) on the decoded bucket.
- The three rows of the refusal matrix. Each row stores its configuration
through the bucket metadata first, so root is served without depending on the
Put operations.
- The behaviour test from the previous commit also runs on the gateway stack,
for an ordinary and a reserved-shaped bucket name.
Tests (negative / positive): refusal matrix 3 rows x 4 refused callers (12
negative, 3 positive; each is counted once under its legacy label);
gateway-stack behaviour test: NoSuchBucket 404 (3), each read's own 404 (6),
wrong-resource AccessDenied 403 (6), stored contents (6 positive).
Mutations: the three `authorize` bodies allow everything. Red:
the_refusal_matrix_holds_on_the_gateway_stack (each of the three reads serves
the anonymous and the no-policy caller), and
the_gateway_stack_answers_the_configuration_reads_as_stored (the wrong-resource
caller is served). The legacy-stack variants stay green.
Not built: the five mutations.
Not verified: the differential's gateway side (CreateBucket, PutObject and the
Put configuration operations are not all served yet; Linux reference only).
storage::access::tests::denial_exits_without_a_context_call_the_audit_function
fails on this base: the source guard looks for the force-delete exit in
access.rs, and that exit now lives in access_owned/object_misc.rs. It is not
this change.
CreateBucket is served end to end by the gateway stack; the legacy stack
serves it through the same owned use case.
- Use case: execute_create_bucket(envelope, gateway input) -> gateway
output, on the fresh admitted task as before (the text guard names the
new inner call). The audit entry and event come from the owned helper.
- Authorization: authorize_create_bucket in storage/access_owned/
bucket_core.rs is the one composite hook both edges call: s3:CreateBucket
with the body's LocationConstraint as a condition value (first decision
without the presigned-query hoist), then PutBucketObjectLockConfiguration
and PutBucketVersioning when object lock is requested.
- Legacy edge: record_s3_op, then through_gateway_seam.
- Gateway edge: impl Served and one ROWS entry.
Not built: no change to what CreateBucket answers; DeleteBucket and
GetBucketLocation follow in their own commits.
Tests (negatives 9, positives 8):
- embedded, both stacks: the_{gateway,legacy}_stack_creates_buckets_as_before
(fresh, idempotent re-create, foreign and EU constraint refused, the four
reserved name families created, xn-- refused, x--x-s3 refused by the SDK
on the client); refusal-matrix row (anonymous, wrong secret, unknown key,
no policy refused; root served, counted once).
- parity: owned_and_legacy_create_bucket_authorization_use_location_constraint
gains six composite cases (location, object lock) comparing decision and
handed-on bucket, legacy hook against owned function.
- use case: tests ported to the owned signature; empty LocationConstraint
reads as none.
- differential: CreateBucket cases for a foreign constraint, EU and the
reserved name families.
Mutation: Served::authorize returns Ok(()) ->
the_refusal_matrix_holds_on_the_gateway_stack red ("CreateBucket serves the
anonymous caller", "... the no policy caller").
Changed assertion: guards.rs names the inner call
execute_create_bucket_inner(envelope, input, mutation_permit) instead of
(req, mutation_permit); same check, new signature.
Metric (ledger candidate, no wire change): the legacy edge now counts every
CreateBucket call, including one its own validation refuses.
Not verified: the differential run against 1.0.5 (Linux CI).
A typed cause now survives the s3s -> owned -> s3s round trip when the
code is unchanged. Before, an s3s error that crossed into
`rustfs_s3_types::S3Error` carried its whole self as a `LegacyCause`
wrapper, and crossing back set that wrapper as the cause, so a caller
downcasting the cause after an owned use case (a legacy edge's
diagnostics, a test asserting `StorageError::ErasureReadQuorum`) found a
wrapper. Crossing back now hands the original s3s error on and applies
the message, request id, status and headers on top; an error our side
re-coded keeps the wrapper, which shows the cause's text, as before.
Test (1 added, both directions):
`a_round_trip_hands_the_typed_cause_back_unless_the_code_changed` - an
untouched round trip is exactly the original (code, message, explicit
status, headers, typed cause); a changed message is applied with the
typed cause kept; a re-coded error keeps only the cause's text.
Mutation: always keeping the wrapper (the hand-back arm dropped) makes
the new test red.
The source-level guard that every denial exit without a context calls
audit_request_denial read only storage/access.rs. 8eb89dd54 moved
DeleteObject's decision, with its recursive-force-delete exit, into
access_owned/object_misc.rs and left the guard behind, so the guard went
red although the exit never stopped auditing. The guard now reads each
exit in the file that holds it.
Tests: the storage::access lib module, run whole: 113 passed.
Mutation: removing the audit_request_denial call from the exit in
access_owned/object_misc.rs turns the guard red with
"\"recursive_force_delete_requires_authenticated_caller\" exit does not
call audit_request_denial".
The policy inputs some hooks take from a request body are now owned
types that both request forms carry and install through
`AuthorizationRequest`, so the gateway edge no longer needs s3s DTOs:
- `RequestedObjectLockRetention { mode, retain_until }`.
- `RequestedObjectLegalHold(Option<String>)`, the status.
- `RequestedObjectTags(Vec<(String, String)>)`. It keeps only complete
key/value pairs; those are the only tags a condition ever named.
Each type converts from its s3s DTO once, at the legacy insert site
(`From<&ObjectLockRetention>`, `From<&ObjectLockLegalHold>`,
`From<&Tagging>`). The conditions built from them are unchanged.
New trait installers: `install_requested_retention`,
`install_requested_retain_until`, `install_requested_legal_hold` and
`install_requested_object_tags`. The legacy hooks use them.
`set_requested_object_lock_retain_until` is now generic and takes an
owned `OffsetDateTime`.
`RequestEnvelope::fork_for_authorization` copies a request for one
more authorization, such as an archive member. Every field is copied,
the access phase's output included. The table publication guards stay
one shared set, so a lock retained through the copy is held for the
original's write. Anything else installed on the copy does not show on
the original.
Tests:
- A new parity test installs body inputs on both forms. The installed
bodies, not conflicting headers, must describe the request, and an
incomplete tag must name nothing.
- A fork test checks that the copy reads like the original and that
what is installed on the copy stays off the original.
- The object-shape column of the decision table now runs on a forked
envelope.
Batch A3 of rustfs/backlog#2765, first increment: GetBucketLifecycleConfiguration,
GetBucketReplication and GetObjectLockConfiguration are owned use cases
(gateway DTO in, gateway DTO out). The legacy stack serves them through
`through_gateway_seam`. Nothing is registered on the gateway stack yet; that
comes in the next commit, once the gateway error mapping answers a missing
bucket 404.
What changes:
- `app::dto_values::stored_config(&T: BucketConfigXml, parse)` is the one way a
stored bucket configuration becomes a gateway DTO: its persisted bytes, read
by the gateway's DTO bridge (`parse_*_dto`). A refusal is InternalError.
- `execute_get_bucket_lifecycle_configuration` and `execute_get_bucket_replication`
take and answer gateway DTOs. `execute_get_object_lock_configuration` is new
in `bucket_usecase/object_lock.rs`, moved from the inline legacy edge body.
Codes and messages are unchanged.
- The legacy edges of the three reads count the operation and go through the
seam. The input members the seam converts are `bucket` and
`expected_bucket_owner` for each.
- The three legacy access hooks call `authorize_bucket_request` with their
legacy action (GetBucketLifecycle, GetReplicationConfiguration,
GetBucketObjectLockConfiguration).
- `storage/s3_compat/{lifecycle,replication}.rs`: the reverse (stored -> s3s)
direction is no longer on the request path and is compiled only under
`cfg(test)`, as the oracle of the parity tests below and of replication's
existing round-trip tests. It goes, with retirement entries, when the Put
edges cross the seam.
Tests (negative / positive):
- Legacy answer parity against main's own conversion: lifecycle (XML bytes,
with DelMarkerExpiration, ExpiredObjectAllVersions, Transition,
NoncurrentVersionTransition, an And/Tag filter and dates, one with a +08:00
offset), replication (XML bytes, with DeleteReplication,
ExistingObjectReplication, ReplicaModifications) and object lock (value; it
holds no timestamp). 3 positive.
- dto_values: a stored configuration round-trips byte for byte through the
bridge (1 positive); a bridge refusal is InternalError (1 negative).
- Use cases: store not initialized is InternalError for each read (3 negative;
the replication one is ported to the owned signature).
- Embedded, legacy stack, for an ordinary bucket and a reserved-shaped one
(`sthree-a3-stored`, `a3-empty-s3alias`): missing bucket is NoSuchBucket 404
(3), no configuration gives each read's own 404 code (6), a caller allowed on
other buckets only is AccessDenied 403 (6), the stored contents are answered
(6 positive). 15 negative, 6 positive.
- Differential (`scenarios/extra/bucket_replication_lifecycle.rs`): lifecycle
with every member and a dated rule on a reserved-shaped bucket, lock reads
before and after a Years rule, absent-configuration reads.
Mutations (run on this tree before the rebase onto ea56aa923):
- M1, `stored_config` rewrites a stored member (WARM -> COLD) before the
bridge: red in s3_compat::lifecycle::tests::the_legacy_answer_through_the_seam_is_the_answer_of_the_reverse_direction,
dto_values::tests::a_stored_configuration_is_answered_with_every_stored_member,
and embedded the_legacy_stack_answers_the_configuration_reads_as_stored.
- M2, the GetBucketLifecycleConfiguration legacy hook allows everything: red in
embedded the_legacy_stack_answers_the_configuration_reads_as_stored (the
wrong-resource caller was served).
Not built: the gateway edge (`impl Served`, rows, refusal-matrix rows, the
gateway-stack variant of the behaviour test), the five mutations.
Not verified: the differential (needs the Linux reference binary, so it is
CI's); the e2e_test crate built only by clippy.
The audit coverage guard required the legacy wrapper for every mutating
operation without a notification event. A migrated operation writes its
audit entry in the use case instead, so both stacks record the same entry,
and its legacy edge drops the wrapper. The guard now accepts exactly one
audit site per operation: the legacy wrapper in ecfs.rs / legacy_edge,
or the owned call under src/app. Both sites at once or neither is a failure.
Calls are matched with whitespace ignored, so rustfmt wrapping and trailing
commas still match.
Accepted owned shapes under src/app:
audit_only_operation(envelope, S3Operation::X, ...)
OperationHelper::audit_only_from_view(&envelope.audit_view(), S3Operation::X)
SourceSet gains count_ignoring_whitespace for this.
Tests: the production guard (all 22 operations, still on the legacy
wrapper); fixtures accepted (legacy only, owned helper wrapped by rustfmt
with a trailing comma, owned envelope wrapper wrapped); fixtures refused
(neither, legacy plus each owned form, an owned call only under #[cfg(test)],
a wrapper and an owned call for UploadPartCopy, which must not count for
UploadPart). Negatives 6, positives 4.
Mutation: owned helper needle renamed so it never matches; both fixture
tests went red (the_guard_accepts_exactly_one_audit_place,
the_guard_refuses_an_operation_audited_nowhere_or_twice), the production
guard stayed green. An earlier run of the fixtures caught a needle that
missed rustfmt's trailing comma; fixed before this commit.
Not verified: nothing beyond the audit_coverage_tests module was run.
The gateway stack's one error mapping now answers every code the
gateway refuses without a context (its is_contextual set: NoSuchKey,
NoSuchVersion, NoSuchBucket, PermanentRedirect, TemporaryRedirect,
NotModified, AuthorizationHeaderMalformed, MethodNotAllowed,
BucketAlreadyOwnedByYou, AccessForbidden) and InvalidRange through
the gateway's LegacyRustfsRefusal, stating the facts the error's own
response headers carry:
- NotModified: ETag, Last-Modified, Cache-Control, Content-Location,
Expires, Vary and the evaluated x-amz-version-id;
- InvalidRange: Content-Range: bytes */<length>;
- NoSuchKey / MethodNotAllowed on a delete marker:
x-amz-delete-marker: true, x-amz-version-id, Last-Modified.
s3s takes no part. A fact a code may not state, an unreadable or
repeated fact, or a refusal the gateway cannot state is answered 500
InternalError rather than sent with a member missing. Before this, each
of these codes failed the gateway's context check, so for example
HeadBucket on a missing bucket could not answer NoSuchBucket on the
gateway stack. The list is pinned against the gateway's own behaviour
in both directions.
PermanentRedirect and TemporaryRedirect are answered 500: the legacy
refusal states only 4xx, 5xx and the 304 of NotModified. RustFS raises
neither.
Adds crate::app::dto_values::stored_etag(&str) -> ETag, the one
conversion of a stored entity tag for every answer that carries one:
a tag the gateway reader accepts is read as it reads it, any other
(empty, a control character) is carried byte for byte
(ETag::from_stored). Nothing calls it yet; batch C's reads and B2's
listings are its first callers.
Not built: no operation of batch C is registered here; the
embedded-server tests of these answers come with HeadObject.
Tests: 12 new in gateway_handlers::error with 18 refused rows (facts a
code may not state, unreadable or incomplete facts, redirects, a hidden
missing key) and the list checked against the gateway both ways, versus
15 answered rows; 2 in dto_values (1 carried-verbatim, 1 spellings).
Changed assertions: every_header_of_the_closed_set_is_carried uses
SlowDown instead of InvalidRange, because InvalidRange is now answered
in the legacy form; it still covers every header of the ordinary
closed set.
Mutations: run right after this commit lands, reported on
rustfs/backlog#2767; a mutation that stays green is fixed by the next
commit.
Refs: rustfs/backlog#2767
Move the listing authorization of batch B2 into
`storage/access_owned/listing_multipart.rs`, generic over
`AuthorizationRequest`, so the legacy access hooks and the gateway edge
call one copy:
- `authorize_bucket_listing` (ListObjects, ListObjectsV2,
ListObjectVersions, the metadata listings): the bucket on `ReqInfo`, the
listing action, `ListBucketVersions` as well for `include-deleted: true`
on a `ListBucket` listing, then the table snapshot. The repeated-query
`InvalidRequest` and the anonymous `ListBucket` fallback stay inside the
policy decision.
- `authorize_list_objects_v2`: the on-demand migration generation captured
before the decision and installed after an allow.
- `prepare_table_data_plane_list_access` and its fingerprint/cursor-key
helpers move here unchanged, now generic; the snapshot reaches the use
case through the new `AuthorizationRequest::install_table_list_access`.
The ListObjects, ListObjectsV2 and ListObjectVersions hooks reduce to one
call each; ListMultipartUploads keeps its generation guard and calls the
moved snapshot. Same actions, same order, same refusals.
Tests (1 added, 1 extended; negatives in the extended table outnumber the
allows): `table_list_access_reaches_the_use_case_from_either_edge`; the
include-deleted table (`include_deleted_listing_enforces_version_permissions`)
now decides every case on an envelope too and requires the same answer.
Mutations: dropping the `include-deleted` follow-up makes the extended
table red; an envelope installer that installs nothing makes the new test
red.
Not built here: the gateway rows and the owned use cases (they wait for
the gateway re-pin).
Verified: the touched tests (19, listing/table access/metadata route)
ran on f33271a74 (before re-pin 1); feature and default clippy
--all-targets ran there too; feature clippy --all-targets ran again on
941fb4fc1, the final base.
CopyObject and UploadPartCopy authorized their copy source with two
identical blocks in their S3Access hooks. Both hooks now call
`access_owned::object_commit::authorize_copy_source`, generic over
`AuthorizationRequest`, which the gateway stack's `Served::authorize`
for both operations will call too (batch E1, rustfs/backlog#2769). The
order is the hooks' order, unchanged: an access-point or Outposts ARN
source is NotImplemented before any policy is read; the source is
recorded for the `x-amz-copy-source` condition; its bucket incarnation
is captured; reading it is authorized (s3:GetObjectVersion when a
version is named); a failed capture is surfaced only after the allow,
and the incarnation is handed on. The destination is decided afterwards
by each hook, as before. `AuthorizationRequest` gains the two installers
the shape needs, implemented for `S3Request` and `RequestEnvelope`.
Measurement for the batch (recorded here as the coordinator asked):
none of the three E1 operations commits a 200 before its work ends, so
the gateway edges will answer with `Resp::new`, not `Resp::commit`.
The pinned 1.0.5 reference binary could not be run on this host (the
asset is Linux x86-64 only and no container runtime was usable), so the
current legacy stack was measured instead: the debug binary at e8847f6f1,
default stack, on 127.0.0.1, driven by `curl --aws-sigv4`:
- CompleteMultipartUpload with a wrong part ETag: 400 InvalidPart with
Content-Length; success: 200 with Content-Length, not chunked.
- CopyObject, missing source: 404 NoSuchKey; success: 200 with
Content-Length.
- UploadPartCopy, missing source: 404 NoSuchKey; range beyond the
source: 416 InvalidRange; success: 200 with Content-Length.
The 1.0.5 source agrees (tag 1.0.5, s3s 5761ddfe): s3s writes a
keep-alive body only for CompleteMultipartUpload and only when the
output's `future` member is set (crates/s3s/src/ops/multipart.rs:27);
RustFS never sets it, and CopyObject and UploadPartCopy have no such
path.
Not built: the destination half of either hook, the gateway edges
(they arrive with each operation's own increment), and the copy-source
read guard (task H's generic form is used as is).
Lines outside tests: +105 / -59 (access.rs, envelope.rs, the new
access_owned/object_commit.rs).
Tests (6 negative cases, 2 positive in the parity table; the hook test
has 2 negative cases and 1 control per operation): copy-source parity
on both edges, comparing decision,
recorded source, handed-on incarnation and ReqInfo for an ARN source,
a source without a read policy, a named version under s3:GetObject,
a deny with a failed capture, a failed capture after an allow, a
missing ReqInfo, and the two allows; and both legacy hooks refuse a
denied source and an ARN source without ReqInfo ever naming the
destination.
Mutations (two grouped runs of the two new tests, each reverted):
- ARN source answered Ok: parity "an ARN source is NotImplemented..."
and hook "CopyObject: a denied source"/ARN case red.
- source decision skipped: parity "a source without a read policy..."
and "a deny wins over a failed incarnation capture" red.
- version swap dropped (always s3:GetObject): parity "a named version
needs s3:GetObjectVersion" red.
- capture failure surfaced before the decision: parity "a deny wins
over a failed incarnation capture" and "a failed incarnation capture
is surfaced after the allow" red.
- destination check before the source (CopyObject hook): hook
"CopyObject: a denied source" red.
Not verified: UploadPartCopy's source-before-destination order is not
distinguishable by an answer on the legacy hook (both halves refuse
AccessDenied); it is pinned on the gateway edge with that operation.
The gateway stack is unchanged by this commit.
`run_owned_mutation` owned a mutation on its own task only when the
mutation answered an s3s `S3Response`. The owned use cases of batch E1
(CompleteMultipartUpload first) answer a gateway DTO with the owned
error type, so the helper is now generic over the result:
`run_owned_mutation<R, E, F>(operation: F) -> Result<R, E>` where
`F: Future<Output = Result<R, E>> + Send + 'static`, `R: Send + 'static`
and `E: From<rustfs_s3_types::S3Error> + Send + 'static`. A lost owner
task is still InternalError "object mutation owner task failed"; the
s3s error is reached through the existing `From` conversion, so the
legacy callers (DeleteObject, DeleteObjects, CompleteMultipartUpload)
compile and answer unchanged. The task-locals it carries into the
spawned task (Keystone credentials, the S3 HTTP operation context) are
unchanged.
Not built: no new caller in this commit; the owned
CompleteMultipartUpload use case is the first.
Tests: none added; the existing callers' tests ran unchanged (71 run
in the touched modules, all passed). Not mutation-checked: a pure
signature generalisation with no new assertion.
This reverts commit f6d64bc1f.
The change was merged outside the merge queue, so the full end-to-end
merge gate never ran on it. On main that gate fails 25 multi-node
cluster tests since f6d64bc1f (distributed_startup_regression_test,
inline_fast_path_cluster_test, heal_erasure_disk_rebuild_test,
tier_stats_cluster_test, upgrade_compatibility_test): every node answers
503 and never becomes ready, logging "Local endpoint has no format on
the system filesystem; restore its data mount" for its fresh data
directories. The run before it (c43e415b0) passed all of them.
Reverting restores the previous startup behaviour so the change can be
re-landed through the queue together with a decision on how fresh
URL-style clusters on a system filesystem (and the cluster test
harness) are meant to start.
The stack-overflow fix c83f47871 in set_disk/ops/object.rs is kept.
Brings rustfs/gateway#1613 (listing members a stored entry may lack are
written as no element), #1614 (ListMultipartUploads url encoding under
the RustFS profile) and #1616 (PutObject and UploadPart carry the legacy
content length). No RustFS caller uses the changed converters yet.
`authorize_bucket_config_mutation(server_ctx, req, bucket, action)` is
now generic over `AuthorizationRequest`, so the legacy hooks and the
gateway edge run the same shape. The shape records the bucket on
`ReqInfo` and snapshots the bucket incarnation under the lifecycle
namespace read lock before the decision. It surfaces a snapshot failure
only after an Allow, and on Allow it installs the snapshot (new trait
installer `install_bucket_config_mutation`) for
`bucket_config_mutation_incarnation`.
The 22 legacy hooks of the shape now call it, with no other change:
- Delete: BucketCors, BucketEncryption, BucketLifecycle, BucketPolicy,
BucketReplication, BucketTagging, BucketWebsite and
PublicAccessBlock.
- Put: BucketAccelerateConfiguration, BucketCors, BucketEncryption,
BucketLifecycleConfiguration, BucketLogging,
BucketNotificationConfiguration, BucketPolicy, BucketReplication,
BucketRequestPayment, BucketTagging, BucketVersioning, BucketWebsite,
ObjectLockConfiguration and PublicAccessBlock.
DeleteBucketTagging keeps its `with_bucket_tag_deletion` scope at the
call site.
Tests: the decision fixture of the owned-versus-legacy parity tests is
now one shared function. A new parity test runs eight cases through
both request forms and compares the decision and the snapshot handed
on. Three are allowed: IAM user, anonymous via bucket policy, owner.
Five are refused: wrong resource, no policy for the action, anonymous
on a private bucket, anonymous on a missing bucket, owner on a missing
bucket.
Not built here: the gateway edges of these operations (batches A1-A4).
DeleteObject's composite access decision moves out of the legacy S3Access
hook into access_owned::object_misc::authorize_delete_object, generic over
AuthorizationRequest, so the legacy hook and the gateway edge call the same
code. The legacy hook loads the bucket generation, calls it, and installs
the guard it returns; its order and answers do not change: NoSuchBucket
before policy, DeleteObject[Version] for the version storage deletes (the
source-version header for a replica delete), ForceDeleteObject, the
authenticated-caller rule for a recursive force delete, and
BypassGovernanceRetention.
Not built here: the gateway edge of DeleteObject and its use case on the
owned DTOs (they route the legacy body through the seam and wait for the
re-pin).
Tests (access_owned/object_misc_test.rs): one table test runs both
request forms (s3s request and owned envelope) through the decision and
compares the answer and the state handed on (ReqInfo bucket, key, version;
guard ready or failed): 5 refused (missing bucket before policy,
anonymous, caller without a policy, no caller installed, invalid
force-delete value) and 3 allowed (replica delete naming no version gets
its source version, any other delete ignores that header, an unavailable
generation is handed on as a failed guard).
Mutations (each run collects every failing case):
- NoSuchBucket check removed, invalid force-delete read as false, replica
version swap removed (one build): the missing-bucket, invalid
force-delete and replica cases went red, in both request forms
- policy decision skipped: the anonymous and no-policy cases went red,
in both request forms
Verified locally: clippy -D warnings with and without gateway-stack (on
the base before the last rebase, which touched none of these files); 64
delete-related lib tests (app::object::delete, the lifecycle delete tests,
storage::access delete tests) passed, and the parity test passed in its
own run.
Not mutation-checked here: ForceDeleteObject, the authenticated-caller rule
and BypassGovernanceRetention need a caller a policy partly allows; the
existing use-case tests cover the predicate.
The pin carries floor-only bucket names across the s3s seam and a typed
ConversionError::name_kind (rustfs/gateway#1610), so HeadBucket's legacy
edge goes through the generated conversion again. seam_error now lives in
crate::app::dto_values and ecfs.rs delegates to it: a bucket name below
the floor stays a server fault, because s3s admits only names the floor
accepts; any other refusal is a server fault as before. handler_error
carries the four stored headers the pin adds to ErrorHeader.
One legacy difference is observable: a key the seam refuses (empty,
longer than 1024 bytes, NUL) is answered 400 InvalidArgument with the
fixed message "Invalid object key", where the message text previously
came from the store or the key validator.
An operation is counted once per request by its edge, after authorization
and before the use case: the gateway's blanket handler already does so, and
a migrated operation's legacy edge calls record_s3_op at the top of its
impl S3 body. OperationHelper::from_view therefore no longer counts; the
legacy constructor OperationHelper::new keeps counting exactly as before for
operations that are not migrated yet. Without this, every owned mutation
that builds its helper with from_view would count twice on the gateway stack.
Not built: no caller of from_view changes here; the first one is the owned
CreateBucket use case.
Tests (1 negative, 1 positive in one test):
only_the_legacy_helper_counts_the_operation - the owned helper counts
nothing; the legacy one counts once.
Mutation: record_s3_op(op) put back into from_view ->
only_the_legacy_helper_counts_the_operation red ("the owned helper counts
nothing").
Local gate: fmt; clippy --all-targets -D warnings with gateway-stack and in
the default build; nextest storage::helper + app::owned_audit_test, 28 passed.
`resource_free_conditions_read_the_same_client_through_both_forms`
compared two condition maps read at different instants, so
`CurrentTime` (and, across a second boundary, `EpochTime`) differed
between the legacy and the owned read, and the test failed. Both clock
keys are now removed before the comparison. Every request-derived key
must still match, and `SourceIp` is still pinned to its expected value.
This corrects a test introduced in 9c394af48.
Since eca95f297 (#9304) every GET through the legacy duplex pipeline
built `run_legacy_producer_segment` futures that embedded the whole
`get_object_with_fileinfo` read future inline, wrapped again in an
`async move` block and the shard-emitter scope. `get_object_reader`
builds one such producer per downstream arm and per segment, so its
debug poll frame grew to 641,888 bytes. Any test that read an object
back through a real store on a 2 MiB thread overflowed the stack and
aborted (28 workspace tests and one e2e test on main, e.g.
rustfs-heal mrf_partial_write_test, heal_integration_test, and
rustfs app::object::internal_put::tests).
The read future is now boxed inside `run_legacy_producer_segment`, so
the producer future is pointer-sized. No behaviour change.
Measured (debug, aarch64-apple-darwin, line-tables-only):
- run_legacy_producer_segment future: 110,200 -> 320 bytes
- <SetDisks as ObjectIO>::get_object_reader poll frame:
641,888 -> 202,352 bytes (prologue stack allocation)
Regression test (negative): set_disk::ops::object::
legacy_producer_stack_tests::legacy_producer_future_keeps_the_read_pipeline_off_the_stack
asserts the producer future is at most 1024 bytes. Mutation: removing
the Box::pin makes it fail with "got 110200 bytes".
Verified locally: the 7 previously aborting rustfs-heal tests pass;
the rustfs lib tests that aborted on main (app::object::internal_put,
on_demand_migration_put, put sse_c, owned_authorization_test,
storage::access, object_zip_download, directory_index, data_usage
snapshot gating, snowball extract) pass, 94 of 94 selected.
clippy -p rustfs-ecstore --all-targets reports only the existing
unused PoolSpaceForTest items in store/rebalance.rs, not this change.
Not run locally: the full workspace, the e2e test
sts_query_compat_test::test_bucket_tags_opa_contract, and release-build
frame sizes.
* fix(ecstore): isolate unavailable local drives during startup
Keep missing local endpoints in their configured slots and initialize healthy
local drives before applying the existing format and service quorums.
Protect unformatted Linux URL endpoints from writes to the system filesystem,
validate legacy admission, and retry missing local slots during format waits.
Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): prepare local drives before registry publication
Prepare local disks with bounded concurrency outside inventory locks and publish validated batches under per-instance writer coordination. Preserve existing registrations on failed or cancelled full initialization, reuse current-root handles, and withdraw stale reopened slots.
Unify initial-format and reconnect publication, retain raw and explicit-context constructor contracts, and move filesystem queries into owned native jobs. Centralize format validation and reject special format files before startup writes.
Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(ecstore): tighten formatted-error ratchet after refactor
Drop the three removed runtime registration call sites from the shrink-only baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test: align drive fixtures with startup admission
Expect the disk-scoped missing-root error and enable the existing directory test mode for the synthetic URL-based receiver fixture.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify local drive admission and publication
Share bounded preparation and pre-I/O membership snapshots across local
startup and retry paths, with coherent registry and topology publication.
Coalesce native startup preflight and preserve root and buffer ownership.
Keep reconnect ID caches instance-scoped and retain admitted health monitors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): pin multipart index queue aliases to owner lifetime
Derive the Linux index alias from the existing identity descriptor so a
retired caller descriptor cannot redirect rebuild or reconnect queue lookup.
Retain the original descriptor assertion and cover cross-disk alias reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(protos): balance peer channels and synchronize TLS eviction
Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(metrics): observe internode unary request lifetimes
Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(perf): make internode gRPC experiments reproducible
Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(perf): isolate transport knobs and redact dry runs
Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(observability): qualify gRPC windows and trace shutdown
Reject stale, incomplete or maintenance-active measurement evidence using
producer timestamps and independent process identities. Preserve raw inputs,
bind evidence hashes, and exercise failure paths through the script entry point.
Record HTTP/2 graceful, child-join and TLS reloader retirement phases without
changing shared deadlines, join ownership or unclean-shutdown marker policy.
Document RPC coverage, fixed-window behavior and separate maintenance workloads.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(perf): add authenticated ReadAll and fixed-rate Ping probe
Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers.
Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(perf): resolve RPC probe disks by explicit UUID
Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata.
Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(perf): reseed temporary RPC fixtures after restart
Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(server): cancel unfinished TLS handshakes during shutdown
Cancel pre-request TLS acceptance when the listener retires so silent
peers cannot retain graceful watchers beyond the shared drain budget.
Keep established TLS/H2 response draining and original child joins intact.
Add real TCP/TLS shutdown regressions and document fresh native OTLP
observation fences and exec-confirmed diagnostic test units.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(grpc): reuse bulk keys and fix IPv6 TLS verification
Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression.
Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(grpc): keep eviction race behind cache helpers
Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(grpc): load native trust and preserve mTLS client identity
Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline.
Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(grpc): account for TLS setup failures and cancelled dials
Own explicit dial metrics from validated endpoint setup through connection completion. Record TLS preparation failures and acquisition cancellation once while keeping peer-health decisions in the connect-result branches.
Cover malformed client identity, queued configuration cancellation and successful coalesced dials, and document the metric scope.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The state an access hook hands to a use case is now read and installed
through the `AuthorizationRequest` trait. Its two implementors are the
s3s request and `RequestEnvelope`, so the legacy hooks and the gateway
edge call the same functions:
- These readers are now generic over `AuthorizationRequest`:
`apply_bucket_generation_guard`,
`apply_copy_source_bucket_generation_guard` (both on one shared
fence), `odm_read_generation`, `prepare_odm_read_generation`,
`bucket_config_mutation_incarnation`,
`replication_request_authorized`,
`retained_table_data_plane_publication_guards`,
`request_context_from_req`, `resource_free_condition_values`,
`req_info_ref`, `req_info_mut`, `load_bucket_generation_from_store`
and `SseKmsPrincipal::from_request`.
- `load_bucket_generation` and `capture_odm_read_generation` now take
the server context explicitly. The legacy hooks pass
`FS::server_ctx`.
- New trait accessors: request context; the bucket, pending-delete,
copy-source and ODM generation guards; the bucket-config mutation
snapshot.
- New trait installers: the bucket, pending-delete and ODM generation
guards. The legacy hooks now install those guards through them.
- New: `authorize_object_request(req, bucket, key, version_id,
action)`, the object twin of `authorize_bucket_request`, with no
version swap. `versioned_read_action` and `versioned_tagging_action`
become crate-visible so callers can pick the action first. The
GetObjectAcl, PutObjectAcl, GetObjectLegalHold and GetObjectRetention
hooks now call it.
- Removed: `request_context_from_extensions`, which had no caller left.
Each reader keeps the same answers and the same error codes and
messages.
Not built here: the bucket-configuration mutation shape, the
body-input newtypes, and the replication-only, write-ETag and
staged-part checks.
Tests: `storage/access_readers_test.rs` covers each reader through
both forms with the state present and absent (12 refusal
cases, 11 accepted or absent ones), plus the installers on the envelope. The
existing owned-versus-legacy decision table now also runs every case
through `authorize_object_request` on the envelope.
Add `storage/access_owned/` with one empty module for each migration
batch stem: bucket_core, bucket_configs, bucket_access,
bucket_replication_lifecycle, object_meta, listing_multipart,
object_read, object_write, object_commit and object_misc. A batch puts
an authorization shape that only its own operations use into its own
file, so parallel batches never edit the same `mod` list. Shapes that
several batches share stay in `storage/access.rs`.
The directory is mounted as a child module of `access.rs`
(`#[path = "access_owned/mod.rs"]`), so the shapes can use the guard
state that `access.rs` keeps private without widening its visibility.
The gateway edge reaches the modules through
`storage_api::server::http::access::access_owned`, feature-gated in the
same way as `authorize_bucket_request`. The re-export carries
`#[expect(unused_imports)]` until the first batch uses it.
No behaviour change. No tests, since the modules are empty.
xn--abc is refused by the edge (400) and the SDK sends nothing for a
--x-s3 name, as on main before the use case moved onto the gateway's
types (measured with an embedded-server probe on main). The corrected
test was not run locally: the build host had no disk for it.
The gateway stack's request-body bridge (batch D: PutObject, UploadPart,
PostObject) reads the gateway's streamed body type, so the rustfs crate
needs rustfs-gateway-stream as a dependency. Added as an optional
dependency enabled only by the gateway-stack feature, without the
tokio-io feature: the bridge does not need it. The workspace entry
already existed (rustfs-protocols depends on it), so the lock gains one
edge and no package.
Not built: the bridge itself; batch D writes it with its first caller.
No tests: a dependency edge has no behaviour of its own.
Verified: check_connect_release_boundary.py --features-only and
--self-test, ci_gate.py --self-test and --check-workflow.
Not compiled locally: the host is short of disk; the crate already
builds in the workspace through rustfs-protocols.
The walking skeleton of the gateway edge, behind the gateway-stack cargo
feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary
serves ListBuckets and HeadBucket end to end and answers every other S3
operation 501 NotImplemented. A default build carries neither the stack
selection nor the wrapper: its request path is unchanged.
- One per-operation unit: impl Served for dto::Op with a metric label, an
authorize without a default body, and an execute. The only impl Handler is
the blanket one over Served (access phase, authorize, count, execute, one
error mapping). authorize calls the function the legacy access hook calls.
- One table laid out per family (ten family files named after the migration
batches); it registers the handlers and is the whole early gate.
- A shared access phase used by both edges, fed on the gateway edge from the
gateway's verdict.
- Stored bucket names are materialised under the gateway floor only, so names
the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds
its input the same way until the gateway re-pin (rustfs/gateway#1610).
- With the gateway selected the legacy service receives exactly what the
legacy router claims (extension routes, object lambda, control plane).
- A start-up line names the stack the mounted service answers with.
- The release-boundary guard reads feature closures from cargo metadata.
Squashed from gw/T2.5-gateway-stack at aca434fdf onto ce9d0b153, with two
changes made while landing: the new CI job reads the ci-dev cache under its
current digest key, and the cache-reader count in scripts/test_ci_plan.py is
7 (this job is the seventh reader).
Checked: cargo clippy --locked -p rustfs --all-targets -- -D warnings in both
builds on aca434fdf (main up to eca95f297); on the squashed tree the Quick
Checks guards, the spell check, ci_gate.py --self-test and --check-workflow,
and the release-boundary tests. Not run on the squashed tree: a compile of
the rustfs crate (six small commits landed on main since the clippy run) and
any test. The legacy HeadBucket name table carries four provisional statuses
for names no store holds; they are corrected by a follow-up when the probe
of main reports.
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776):
.github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance
corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh
--external) against the candidate binary on RUSTFS_S3_STACK=legacy and
=gateway, one server per stack. The suites are checked out at the gateway rev
Cargo.toml pins. Nightly schedule and workflow_dispatch only.
- Each stack must be observed: the start step writes `ready` only when
/health/ready answers while its own process is alive and the server's
server_starting event names the requested stack (scripts/dual_stack.py
stack); otherwise `did not start: ...`. Summaries print the observed stack.
- Verdicts come from files the run leaves on disk (startup record, stack
evidence, liveness after the corpus, exit codes, report.json, matrix.json),
never from an earlier step's exit status.
- Conformance: the rustfs profile describes the gateway edge, so only the
gateway stack is judged (scripts/check_conformance_rulings.py against
.config/conformance-rulings.toml). Red: an unruled failure, a ruled case
failing differently from its ruling's `failure`, a stale ruling, and a skip
on the server's own answer (endpoint not answering, refused fixture). The
legacy run is an unjudged reference: it must serve, stay up and complete.
- Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail`
or `sut-unregistered` cell, a client that could not be installed, or an
unfinished matrix; legacy is the baseline and every legacy pass must pass on
gateway.
- The gateway leg and the stack comparison are continue-on-error, report-only
until T2.10.
Not built: no re-pin (write cases need a gateway rev with
--allow-external-writes plus that flag on the lane's run line); no legacy
rulings; no use of the runner's own --rulings ledger (stale rulings do not
block there); no call to the gateway's check_compat_matrix.sh (it does not
fail `fail` cells or clients that never ran); no JUnit (the pinned runner
writes a raw U+0001); no install cache; toolchain versions copied from the
gateway's client-matrix.yml at 7915d069.
Tests: --self-test of both checkers in make script-tests, 40 tests, 31
negative. Mutations, each red: unruled failure ignored
(test_an_unruled_failure_is_red); stale only when passed
(test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green
(test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit
(test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted
(test_an_environment_exit_is_red); legacy judged or tallied
(test_the_legacy_reference_is_counted_never_judged); dead server green
(test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture
skips not red (test_a_skip_on_the_endpoint_not_answering_is_red,
test_a_skip_on_a_refused_fixture_is_red); ruling on id only
(test_a_ruled_case_failing_differently_is_red); unknown verdict
(test_an_unknown_verdict_is_refused); stack not compared
(test_a_server_of_the_other_stack_is_red); no stack evidence
(test_ready_without_stack_evidence_is_red); failed cell, skipped client,
swallowed matrix exit (test_a_failed_cell_is_red,
test_a_client_that_could_not_be_installed_is_skipped_never_passed,
test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset
gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red,
test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed,
test_diff_without_both_manifests_is_red); diffs
(test_diff_lists_changed_and_absent_cases_only,
test_diff_without_both_reports_compares_nothing).
Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did
not start (gateway stack is not assembled yet, T2.5); legacy reference
completed, 140 of 890 cases executable against an external endpoint; legacy
matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0.
Not verified: the stack observation and liveness record in CI (exercised
locally under bash -eo pipefail with a stand-in server and main's
dual_stack.py, both directions); a scheduled run; a diff with gateway content.
ListTargetSubscriptions rendered each bucket's stored notification
configuration as the s3s DTO (an XML round trip) and walked it as JSON.
It now reads the routes of the stored PersistedNotificationConfiguration
directly through its behavior() projection: same order (queue, topic,
lambda), same exact destination match, same filter-rule reading
("Prefix" before "prefix", a rule without a value supplies none). The
conversion step and its two internal-error exits are gone; the function no
longer fails. admin/handlers/event.rs no longer references s3s.
Tests: the existing subscription test now starts from stored XML, the way
the handler loads it; a new test pins rule-name precedence, valueless and
nameless rules, and exact (case- and whitespace-sensitive) destination
matching. Both were run against the old code first and passed unchanged,
then against the new code. Mutation: not run yet (the build was interrupted); it follows separately.
Not verified: the workspace suite (only the touched modules' tests ran).
Part of the s3s removal tail (rustfs/backlog TAIL, feeding T4.2):
- storage/options.rs: the empty Object Lock value check compares against
the lowercase rustfs_utils names the file already imports instead of the
s3s header constants; the comparison stays case-insensitive.
- admin/handlers/pools.rs tests: compare ApiError codes through the owned
code (the idiom the file already uses) and read headers and status from
the owned S3Error conversion, from which the s3s one is derived.
- init.rs: the error logged when a bucket's notification rules cannot be
registered is built with rustfs_s3_types::s3_error!; code and message are
unchanged, the Debug spelling in that log line is the owned type's.
Not changed: init.rs still builds s3s notification DTOs, because the notify
rule conversion it calls takes them.
Tests: empty_object_lock_values_are_not_user_metadata (4 skipped keys, 1
kept), run against the old code first and passed. Mutation: not run yet (the build was interrupted); it follows separately.
Not verified: the workspace suite (only the touched modules' tests ran).
Two quick checks failed on main after eca95f297:
- scripts/check_unsafe_code_allowances.sh requires a SAFETY: comment next
to an unsafe_code allowance. The comment above the allowance in
crates/ecstore/src/erasure/codec/shard_buf.rs already stated the
contract; it now carries the marker the guard looks for.
- The spell checker reads the byte literal b"hel" in a put.rs test as a
misspelling. The two frames are now b"he" and b"llo"; the test still
joins them into "hello".
No behaviour change. Checked locally: the unsafe-allowance guard, typos
over the whole tree, cargo fmt --check. Not run: clippy or tests (a
comment and a test literal).
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY:pre-commit
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checktest-count-ratchet-checktest-seams-checktest-retirements-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast pre-commit checks without clippy/full tests
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checkrelease-feature-boundary-checktest-count-ratchet-checktest-seams-checktest-retirements-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY:pre-pr
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checktest-count-ratchet-checktest-seams-checktest-retirements-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkoffline-enrollment-e2e-checkclippy-checktest## Run full pre-PR checks with clippy and tests
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checkrelease-feature-boundary-checktest-count-ratchet-checktest-seams-checktest-retirements-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkoffline-enrollment-e2e-checkclippy-checktest## Run full pre-PR checks with clippy and tests
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
**Use this when** validating inline placement, block geometry, Range reads and reader rollback in a task-owned local Docker cluster.
The adapter reuses the repository's four-node compose topology with 16 logical Docker volumes and EC12+4. It binds one S3 port to loopback, keeps strict durability and checks running binary hashes and persisted object geometry. Docker volumes share the host storage substrate. Performance observations describe this local virtualized lab.
## Build a frozen server image
Use the current stable toolchain pinned in the lab Dockerfile. Supply a Git bundle and the metadata inspector source as the build context. The server is compiled from a clean checkout before the diagnostic example is added.
```bash
set -euo pipefail
LAB_BUILD_DIR="$(mktemp -d)"
LAB_REVISION="$(git rev-parse HEAD)"
git bundle create "$LAB_BUILD_DIR/repository.bundle" HEAD
Select a new compose project and a private state directory. The adapter generates test credentials there. Existing resources require matching ownership labels. Configure checks the image's source revision and pins immutable image IDs. Each operation takes an exclusive state lock. Deployment waits for `/health/ready` on every node, covering storage, IAM, lock quorum and peer dependencies.
The probe verifies complete bytes, HEAD sizes, Range contents, EC12+4 and the actual persisted block. It records inline markers for 4 KiB, 1 MiB, 1 MiB + 1 byte, 1.25 MiB, 1,572,856 bytes, 1,572,857 bytes and 1.5 MiB objects in `probe.json`. The server-only image includes the existing filemeta decoder through a small geometry inspector. Console assets and the administration CLI are outside this lab image.
## ABBA deployment hook
Use `inline_docker_lab.py ... deploy` in the existing `run_hotpath_warp_abba.sh` external deploy hook. Export `RUSTFS_ERASURE_SHARD_BYTES` according to `HOTPATH_ABBA_PHASE`. The adapter uses the supplied leg, phase, bucket, binary and evidence-file context, resets only owned lab volumes, checks running configuration and binary hashes, creates the fresh bucket and writes the runner's deployment evidence.
Use the same frozen binary for both sides of a geometry experiment. Fix the inline budget and encoding mode, then compare zero with one candidate shard value. Include the CPU, memory and image identities from every node in each result. The state `config.json` supplies the generated test credentials for the load driver. Set `DOCKER_CONTEXT` to the selected context when the driver captures Docker resource snapshots.
For content compatibility across an image or geometry change, configure the next selected image and artifact using the same state, then deploy with `--keep-data` and an existing bucket. This retains owned volumes. Compare existing objects and version IDs with the manifest prepared before the change.
## Cleanup
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" down --purge
```
Cleanup checks ownership for containers, volumes and networks. A configuration cannot adopt another project's resources. Raw measurements, state, Git bundles and build outputs stay in task artifact directories.
Unit checks run with `scripts/python_bin.sh scripts/test_inline_docker_lab.py` and are included in `make script-tests`.
## Linux content-probe client
The PUT after-probe requires a working `mc`. For a macOS client affected by cgo initialization failures, compile the same frozen public mc module for Linux with CGO disabled, build it with `Dockerfile.mc`, and use `scripts/inline_docker_mc.sh` as the process-local `mc` command. Record the module version, checksum and resulting image ID with the run.
The adapter rewrites only the lab loopback alias endpoint to the private `node1` endpoint. Alias configuration stays in the probe's own `MC_CONFIG_DIR`, and `cat` streams raw bytes. The existing HEAD, GET and hash checks continue to execute.
`deploy --rpc-mode json-compat` explicitly clears both MessagePack-only flags on every node. `deploy --rpc-mode msgpack-only` enables both flags for the selected homogeneous lab fleet. Deployment evidence records the mode and checks both running flags. The default is `json-compat`. The ABBA hook may select the same mode through `RUSTFS_INLINE_LAB_RPC_MODE`.
First compare the base and bounded-multi-block binaries at `--shard-bytes 0` in JSON compatibility mode. Then repeat with MessagePack-only on both sides. A same-binary transport ABBA isolates the encoding effect. Keep the 128 KiB budget, CPU sets, strict durability, load and object set fixed. Check actual markers and block sizes at the boundary sizes before timing. Retain baseline-drift and candidate gates, error logs and raw request distributions.
For a rolling downgrade, redeploy the current binary with `--keep-data --rpc-mode json-compat` before introducing an older node. Check old and new reads, version IDs and batch deletes against the persisted manifest. A running MessagePack-only flag pair establishes the lab configuration. Compatibility with a particular old reader requires the explicit reader test.
## Preserve a compatibility manifest
Run the seed once in a fresh lab bucket. It records full byte oracles and the observed version inventory, including any additional version matching a known body, and leaves a 5 MiB multipart part ready for the next reader. Each bucket has its own private `compat-<bucket>.json` manifest. Keep the same state and volumes across image changes. A fresh deployment or cleanup that resets volumes moves existing compatibility manifests into `compat-history`. Retained-data deployments preserve the active manifests.
The check compares every persisted version ID, HEAD length, complete SHA256, cross-block Range and per-version geometry from `xl.meta`. The seed includes zero-length objects, null versions, the 192 KiB versioned admission boundary and the accumulated 128 KiB shard boundary. Seed a second bucket on the new writer before the downgrade, then check both manifests using the selected old reader. Complete an old upload on the new binary and a new upload on the old binary. After returning to the current fleet, `compat-delete --bucket <bucket>` deletes the recorded versions in one S3 batch and checks the exact remaining inventory and bytes.
Successful multipart-completion and batch-delete receipts are saved before subsequent content and geometry checks. If either check fails, rerun the same command with the same state directory to resume verification from the saved completion state. After a seed check fails with a saved manifest, use `compat-check` to resume verification.
For mixed-version testing, retain JSON compatibility mode throughout the rolling replacement. Verify each node's immutable image and binary, then perform manifest checks and new writes through both coordinator versions using `--coordinator node1` and `--coordinator node4`. The manifest records the actual writer coordinator revision from the running image. Stop one owned node to check read availability at EC12+4, restore it and check the manifests again. Record the exact oldest tested source revision, toolchain and build profile with the evidence. The version boundary is established by these executed cells.
An owned lab can replace just its fourth node through a compose image override. Select the frozen old image first and verify its revision label. Keep `compose.json` in JSON compatibility mode for this cell.
Inspect all four running image IDs and binary hashes before recording the mixed-fleet result. Redeploy the configured current image with `--keep-data --rpc-mode json-compat` to restore a homogeneous fleet. The same manifest checks apply after stopping and restarting one owned node through the base compose file.
@@ -34,6 +34,10 @@ If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rust
- **High Performance**: Optimized configurations for batching, compression, and memory management.
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
## HTTP Request Latency Percentiles
The main RustFS dashboard shows request latency percentiles in milliseconds. The HTTP server request, response-body chunk, and response-body stream histograms export bucket boundaries in seconds. During a rolling upgrade or rollback that changes the bucket layout, compute percentiles from nodes that share one layout and from a query window entirely on that layout. The bundled dashboard and recording rules use a five-minute window. The mean from `_sum` divided by `_count` remains usable across the change.
## GET Performance Optimization Dashboards
Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Deployment
- **The S3 protocol stack is now rustfs/gateway**, and it is the only S3 stack. The legacy s3s-based stack is removed, and so are the `RUSTFS_S3_STACK` environment variable and the `gateway-stack` cargo feature. There is no on-disk format change. `RUSTFS_S3_STACK` no longer selects anything: a server started with it set, to any value, refuses to start and prints `RUSTFS_S3_STACK is no longer supported; the gateway stack is the only S3 stack since <version>`, where `<version>` is the release's version. Remove the variable from service units, container environments, Helm values and orchestration files before upgrading. To roll back, run the previous release binary on the same data.
- Before upgrading, verify that the configured root access key is at least 3 bytes and the root secret key is at least 8 bytes. RustFS now rejects shorter values at startup and identifies the setting to update. Version 1.0.5 accepted these settings at startup and returned HTTP 500 when root requested STS credentials. Coordinate root credential changes across all nodes and clients. See [root credential rotation considerations](docs/operations/two-factor-auth.md#root-credentials-cannot-be-changed-at-runtime).
- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
@@ -44,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Vault KMS backends no longer fail when the server answers reads from a replica**: a Vault or OpenBao node that serves reads locally (an OpenBao standby, a Vault Enterprise performance standby) can return the state from before a write for a short while after acknowledging it. Against such a node the KV2 backend failed every key creation with `404 Key not found` while leaving the key stored without its description and tags, failed operations issued right after a create or update with `Key not found` or `Concurrent modification`, and the Transit backend failed a disable followed quickly by an enable. Key creation on the KV2 backend is now a single create-only write that carries the description and tags. Both backends now re-read a key record when the answer is older than a write the same RustFS node made in the previous five seconds, pause before re-reading after a lost check-and-set write and allow five attempts instead of three, and retry a Transit key delete that is refused because the deletion switch is not visible yet. A different RustFS node reading the same replica can still observe the earlier state until the replica catches up; see [OpenBao as the Vault KMS backend](docs/operations/kms-openbao-compatibility.md).
- **KMS keys could not be described by ARN through the path route**: `GET /rustfs/admin/v3/kms/keys/{key_id}` read the path segment verbatim, so a key id that has to be percent-encoded to fit in one segment — an AWS key ARN, with its `:` and `/` — was looked up, authorized and audited under its encoded spelling and answered `404`. The segment is now percent-decoded first; a segment that is not valid UTF-8 after decoding is refused with `400 InvalidArgument`. The `describe-key?keyId=` form was unaffected.
- **SSE-KMS objects written through the AWS KMS backend could not be read**: `GetObject`, `CopyObject` and every other read of an `aws:kms` object failed with `400 InvalidRequest` ("SSE-S3 requires RUSTFS_SSE_S3_MASTER_KEY to be set ...") on a deployment using `RUSTFS_KMS_BACKEND=aws`, although the write and `HeadObject` succeeded. The AWS backend stored the `GenerateDataKey` blob as it came from AWS, the read path took those unrecognized bytes for a locally wrapped key, and AWS was never asked to decrypt. The backend now stores the blob inside the same data-key envelope the other backends use, and a blob stored without one is routed to AWS when the AWS backend is active. Objects written by 1.0.4 and 1.0.5 are intact and become readable after the upgrade, with no rewrite. Setting `RUSTFS_SSE_S3_MASTER_KEY` was never a workaround. An object written by this release is not readable by an older one; the same was already true of every object the older release wrote itself.
### 7\. Docker Swarm with the SwarmCLI chart (Option 7)
Follow the [SwarmCLI chart guide](https://docs.rustfs.com/en/installation/cloud-native/swarmcli-chart) to run RustFS on Docker Swarm, as a single-node service or a distributed, erasure-coded cluster with one peer per node, with the open-source [SwarmCLI `rustfs` chart](https://github.com/Eldara-Tech/swarmcli-charts/tree/main/charts/rustfs). The chart keeps the root credentials in Docker secrets and refuses the default `rustfsadmin` keys, upgrades and rolls back as versioned releases, and can serve the S3 API over HTTPS through Traefik. Its Console stays off until you enable it. The chart and [SwarmCLI](https://github.com/Eldara-Tech/swarmcli) are maintained by Eldara Tech, not by the RustFS project.
- When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance.
- An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`.
Scanner bucket quantum controls:
-`RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES`, `RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES`, and `RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES` cap one bucket admission. Their persistent keys are `scanner.bucket_quantum_duration`, `scanner.bucket_quantum_objects`, `scanner.bucket_quantum_directories`, `scanner.bucket_quantum_entries`, and `scanner.bucket_quantum_name_bytes`; duration is in whole seconds and the remaining keys are work counts or bytes.
- All five defaults are `0` for controlled rollout. A positive value enables that dimension; `0` disables that dimension, and all five zero values disable the bucket quantum. Parent cycle limits remain in force. Invalid unsigned integers and durations outside the platform timer range are rejected; environment overrides take precedence but cannot hide invalid persisted values. Admin scanner status reports each effective value and source under `runtime_config.bucket_quantum_*`, using `bucket_quantum_duration_seconds` for duration.
- Disabled defaults are an initial rollout state, not completion of usage-convergence acceptance. Production defaults require matched-topology A/B measurements and the remote progress/fairness checks in [Scanner runtime controls](../../docs/operations/scanner-runtime-controls.md#bucket-quantum-rollout-and-acceptance). These controls do not certify partial usage or acknowledge pending dirty work.
Scanner recovery retry controls:
-`RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS` caps the wait between automatic startup recovery retries. The persistent admin key is `scanner.recovery_backoff_cap`; the environment takes precedence over the persistent value, and the unset default remains 600 seconds (10 minutes). Values must be positive integer seconds within the platform timer range; `0`, malformed values, and overflow are rejected. A valid environment override does not hide an invalid persistent value from `admin config set` validation.
@@ -291,6 +302,8 @@ Drive timeout profile preset:
RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them.
A local POSIX filesystem bind-mounted into a gVisor sandbox with directfs is a supported storage path. gVisor presents that bind mount as `9p`/`V9FS`, so the host filesystem type is not visible; the startup guard recognizes mountinfo options `trans=fd` and `directfs` and does not treat the mount as unsupported. Genuine 9p, virtio, and NFS shares remain unsupported.
## 📄 License
This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set`RUSTFS_E2E_REFERENCE_BINARY`to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Both sides run the binary under test on its only (gateway) stack unless`RUSTFS_E2E_REFERENCE_BINARY`names a released `rustfs` binary, which then runs side A. A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode.`RUSTFS_E2E_STACK_B_REGION` starts side B in another region to inject a difference. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.