Compare commits

...

350 Commits

Author SHA1 Message Date
Chris An 4a1ce44a37 Merge remote-tracking branch 'origin/main' into fix/s3-post-object-location 2026-11-19 17:51:17 +08:00
overtrue 55be4e3f79 fix(notify): route rule engine storage imports through the storage_api boundary 2026-11-19 17:34:58 +08:00
overtrue 5be4ef65e8 test(table-catalog): issue ordinary temporary credentials through the parent-bound path
Regular parents now fence STS sessions by durable generation, so a session
stored without one is rejected as an unknown key. Publish the test sessions
with set_temp_user_from_parent, as the STS handler does.
2026-11-19 16:59:19 +08:00
Chris An ce20ed4a4b Merge branch 'main' into fix/s3-post-object-location
Keep PostObjectLocationFixLayer on the gateway-only external HTTP stack,
ahead of ConditionalCorsLayer, and drop the legacy-stack Location
regression test now that the retired S3 stack selector is gone.
2026-11-19 16:43:56 +08:00
overtrue 24f3e783c2 fix(server): keep Connection off HTTP/2 answers and test the upload drain on the gateway stack
The three UploadPart timeout tests drove the early-response drain around
a hand-written s3s S3 service. They now drive it around the gateway
stack as production configures it (GatewayStack::assemble_serving, a
test-only constructor over the same configured builder), serving an
UploadPart handler that reads its whole body as RustFS's handler does.
Every assertion is kept: 400 RequestTimeout, the HTTP/1 transport
retained for the drain and drained of the late bytes, the HTTP/2 body
dropped alone, and other HTTP/2 and HTTP/3 streams still served.

The gateway times a stalled body out with its own request-body idle
bound, so that bound is what the fixture sets; it does not follow a
paused tokio clock, so the HTTP/1-and-HTTP/2 case runs in real time
(200 ms bound) instead of advancing a paused clock.

Kept as written, the HTTP/2 assertion failed: GatewayStack::call added
`Connection: close` to any answer whose connection intent is to close,
HTTP/2 included, where the header is forbidden and only the stream
ends. It is now added to HTTP/1 answers only.
2026-11-19 16:15:49 +08:00
overtrue 583df55d90 test(server): name the metadata route table case type for clippy 2026-11-19 16:15:49 +08:00
overtrue 670ae01f7a test(server): restore website, metadata listing and table isolation coverage on the gateway stack
The retired legacy metadata route carried the website-host and metadata=true listing case tables, and a test that its authorization attaches the table listing snapshot. Re-create both against the gateway stack, observing the listing a caller receives, and probe the native admin and control-plane hosts for the unsigned x-amz header refusal.
2026-11-19 16:15:49 +08:00
overtrue 3a1f0c09b3 fix(server): let the owned early-response drain take bodies the gateway leaves unread
The gateway stack was called through rustfs-gateway's tower adapter. Under the RustFS profile
that adapter wraps the request body, takes it back when the handler drops it unread, and reads
it in a task of its own spawned on the ambient runtime (unread_body.rs, Retention::settle). That
task took the body away from EarlyResponseBodyService before its abandon slot could see it, so
RustFS's own drain never ran: its task was no longer spawned under the HTTP/1-2 task owner,
graceful shutdown could report its children retired while a detached drain still held the
transport body, and the early-response drain metric and log went silent. On origin/main the
legacy stack was the default assembly, so the listener test never crossed the adapter.

GatewayStack::call now calls the service's own entry and keeps the adapter's connection
verdict (connection_intent_of). The inert drain_unread_request_bodies setting is removed.
EarlyResponseBodyService is again the only HTTP/1 unread-body drain, with Connection: close
and the configured idle bound.

The gateway-stack test that asserted the adapter's drain timeout now asserts that the stack
releases an unread body when it answers. The configured-timeout half moves to
gateway_rejected_http1_body_uses_the_configured_drain_timeout, which composes
EarlyResponseBodyService over the real gateway stack for a 1 s and a zero timeout.

Mutation: restoring the adapter call turns all three red:
http12_actual_early_body_drain_is_owned_until_release (the drain never starts),
rejected_http1_body_is_released_to_the_host_drain and
gateway_rejected_http1_body_uses_the_configured_drain_timeout.
2026-11-19 16:15:49 +08:00
overtrue eed50fc5ff test(e2e): run the differential against the only stack
Neither side of dual_stack_differential is started with the retired
stack selector any more, which the binary under test now refuses: both
sides run the binary under test, or side A runs the released reference
binary named by RUSTFS_E2E_REFERENCE_BINARY. RUSTFS_E2E_STACK_B is
gone; the injected-region control is unchanged.
2026-11-19 16:15:49 +08:00
overtrue e462fafb65 docs: describe the gateway stack as the only S3 stack in the CI gates and the changelog 2026-11-19 16:15:49 +08:00
overtrue 9c0bc265c3 ci: drop RUSTFS_S3_STACK from reference parity, nextest notes and the connect boundary guard 2026-11-19 16:15:49 +08:00
overtrue 74bf3ca5fa ci(nightly): run the dual-stack nightly on the gateway stack and archive the both-stack scripts 2026-11-19 16:15:49 +08:00
overtrue 31c4f4a7ff ci(acceptance): judge one server and drop the legacy reference run and the stack diff 2026-11-19 16:15:48 +08:00
overtrue 3fb8d861f0 ci(e2e): run the distributed, replication and migration-interop lanes on one stack 2026-11-19 16:15:48 +08:00
overtrue 4b386794e1 ci(mint): run one server on the gateway stack and drop the stack comparison 2026-11-19 16:15:48 +08:00
overtrue 23a2593aac ci: fold the gateway-stack lanes into the main lint, test and e2e jobs 2026-11-19 16:15:48 +08:00
overtrue 7b1befa6f9 chore(ci): shrink the s3s footprint to what the gateway-only stack keeps
server/layer.rs and server/hybrid.rs no longer reference s3s and leave
the edge allowlist; the baseline ceilings drop to the measured values
(files 50, s3_error! lines 103).
2026-11-19 16:15:48 +08:00
overtrue 2f3359df69 refactor(server): delete the five s3s patch layers the gateway stack replaces
SigV4HeaderGuard, S3ErrorMessageCompat, DoubleSlashListBucketsCompat,
VirtualHostStyleHint and ObjectAttributesEtagFix patched the s3s edge
and have no work left: the gateway_stack legacy_layers tests pin what
each made the legacy stack answer against the gateway stack alone. The
layer unit tests go with the layers, and the server-domain flag that
only gated VirtualHostStyleHint goes with it.

EmptyBodyContentLengthCompat keeps its admin, console and internode
halves and loses its S3 half, which the gateway stack replaces by
leaving the bodies of bodyless operations unread; its S3 tests now
assert that such requests pass through unchanged.
2026-11-19 16:15:48 +08:00
overtrue ea522c18b6 refactor(server): serve the outer tower stack with an owned body and error type
The S3 service under the middleware now answers in EdgeBody, a boxed
Bytes body, with a boxed error, instead of s3s::Body and
s3s::HttpError. The gateway stack, the native console and the refusals
written in server/http.rs meet in that one type; Swift and the
internode RPC lane convert into it from rustfs_s3_types::Body.
ControlBodyDigestService is generic over its inner response body, and
its control-plane XML body limit is a named constant equal to the limit
the s3s edge applied.
2026-11-19 16:15:48 +08:00
overtrue 9f13eb22a5 refactor(server)!: make the gateway stack the only S3 stack
Remove the S3Stack selector, the legacy s3s service assembly and the
gateway-stack cargo feature: the gateway stack, its native admin and
console hosts and the control-plane host are always assembled, and the
gateway crates are ordinary dependencies. The debug-only compile_error
guard goes with the feature.

RUSTFS_S3_STACK is retired: a server or an embedded server started with
it set, to any value including gateway or an empty string, refuses to
start with an error naming the variable and this release. The message
and its tests share one constant.

Tests that ran only on the legacy stack, or selected between stacks,
are retired; tests that ran on both now run on the gateway stack alone.
The s3s metadata/website route is no longer mounted and stays only
until the metadata-route branch replaces it.
2026-11-19 16:15:48 +08:00
overtrue fec2d4bb4b test: register the legacy-stack tests the gateway flip retires
Each entry names a legacy-only test, or the unit test of an s3s patch
layer, that the flip deletes, with the gateway-stack test that keeps
its contract (rustfs/backlog#2783). The commits that delete them delete
their entries.
2026-11-19 16:15:47 +08:00
overtrue 42a1417ed0 fix(notify): bind dispatch rules to bucket metadata snapshots 2026-11-19 10:15:02 +08:00
Chris An 7b7f96924b test: use typo-safe UTF-8 fixtures for POST Object Location
crate-ci typo tokenizes the caf prefix in caf%C3%A9 as a misspelling of
calf. Keep a UTF-8 percent-encoding case, but avoid that false positive.
2026-11-19 08:14:42 +08:00
overtrue a3d6b59d03 feat(server): answer website hosts on the gateway stack before S3 routing
The gateway selection gains a website service, chosen by its host claim
before the Keystone refusal, the native console, the native admin and the
control plane, as the legacy metadata route claims those requests first.
A parity test sends both request classes to both stacks over one store.
2026-11-19 07:44:46 +08:00
overtrue 924eb06968 feat(gateway): serve metadata=true listings on the gateway stack
Install the RustFS dialect rows of ListObjectsV2WithMetadata and
ListObjectVersionsWithMetadata on the S3 table, let the early gate allow
them, and register one handler that runs the shared access phase, the
listing authorization and the shared listing answer. The rows keep their
overlaps with the standard listings and each other; the admin extension
rows they also shadow live on the control-plane host.
2026-11-19 07:44:46 +08:00
overtrue 5a44944095 refactor(app): serve website hosts and metadata listings through owned modules
The legacy MetadataRoute now hands website requests to an owned WebsiteHost,
which reads through the owned GetObject/HeadObject authorization and use
cases with an anonymous request envelope, and hands metadata=true listings
to an owned module that authorizes and answers them on the request envelope.
The WebsiteRead marker moves with the website rules.
2026-11-19 07:44:45 +08:00
overtrue 8541ab829a perf(gateway): attribute the owned PutObject path in hotpath profiles
The gateway PutObject path carried no hotpath measurement. Measure execute_put_object_owned under the label the use case had before the legacy edge split, and extend the contract test to require it.
2026-11-19 07:27:26 +08:00
overtrue b71c6fabc3 test(gateway): run the raw XML and error-code checks of GetObjectAttributes on both stacks
The legacy-only assertions now apply to the gateway stack too, with the five rows on which the gateway answers differently pinned explicitly.
2026-11-19 07:27:26 +08:00
overtrue ed5927cf5d test(gateway): hold the gateway stack to the key-floor and completion tables
The legacy-only tests for keys outside the floor and for CompleteMultipartUpload request shapes now run on both stacks. The gateway answers every NUL and over-long key with one uniform refusal before the bucket is consulted, and differs from the legacy table on six completion shapes; both are pinned explicitly.
2026-11-19 07:27:26 +08:00
Chris An 4c869cd08c fix(server): emit absolute encoded POST Object Location
Browser POST uploads with success_action_status=201 currently return a
relative, unencoded Location such as /bucket/a+b c%d.txt. Clients that
decode the field the way AWS formats it treat + as a space.

Rebuild PostResponse Location as an absolute URI from the request host,
scheme, and addressing style, percent-encoding the object key with RFC
3986 path rules (space as %20, slash preserved as a separator).

Fixes #9259
2026-11-19 06:42:14 +08:00
overtrue c794df64b8 refactor(storage): rehome live helpers out of the legacy edge
Rename storage/ecfs_extend.rs to storage/s3_support.rs, point its request-marker
imports at storage::request_markers, and move parse_object_version_id and
ListObjectUnorderedQuery out of storage/ecfs.rs into it. The module no longer
depends on FS, S3Access or storage::access. Behaviour is unchanged.
2026-11-19 06:12:30 +08:00
overtrue 5e69593836 test(iam): align log boundary with conditional writer refusal 2026-11-19 04:21:09 +08:00
overtrue e811755b8d fix(iam): bind plugin registration to request owner snapshots
Authenticate the actual request signing secret against the authoritative regular-owner record before reading the registration body. Carry its epoch and identity revision through publication without rebinding to a newer owner.

Reject stale peer signing secrets and concurrent owner changes, including same-secret recreation, rotation back to an old secret, revision-only rewrites, and legacy-initialization readback races. Preserve the independent STS generation and unchanged-secret upsert policy.

Add deterministic real-store and handler regressions and document the secret-reuse boundary. This is a separate correction layer for #3266 above the generation-aware integration.

Validation: cargo fmt --all --check, git diff --check 05d9825, scripts/check_doc_paths.sh, and scripts/check_no_planning_docs.sh passed. Compilation, runtime tests, and Clippy are deferred as requested.
2026-11-19 04:21:09 +08:00
overtrue efed8a35d2 fix(iam): integrate plugin epochs with generation-aware identity writes 2026-11-19 04:21:09 +08:00
overtrue b656745305 fix(iam): bind plugin delegations to durable owner epochs 2026-11-19 04:21:09 +08:00
overtrue ef974ad719 fix(iam): fence regular-user STS sessions by durable parent generation 2026-11-19 04:21:09 +08:00
overtrue 3af84de8f1 test(storage): iterate quorum fixture disks directly 2026-11-19 04:21:09 +08:00
overtrue d19a2ac726 test(replication): verify per-bucket refresh bookkeeping 2026-11-19 04:21:09 +08:00
overtrue 0524cce44b fix(replication): scope target clients and health to source buckets 2026-11-19 04:21:09 +08:00
overtrue 0fac11d57b refactor(storage): mount access_owned and its probe test outside access.rs 2026-11-19 02:29:45 +08:00
overtrue 46c58d8758 refactor(storage): switch access_owned to owned error types and explicit imports 2026-11-19 02:29:45 +08:00
overtrue aa24d21807 refactor(storage): rewrite authorize_copy_object_tags over owned tags and move it to access_owned 2026-11-19 01:51:44 +08:00
overtrue 3afea8ecf9 refactor(storage): move the plain authorize-action selectors to authorization_actions 2026-11-19 01:51:44 +08:00
overtrue f1fdb8c2c3 fix(gateway): answer a custom code without a status as 500 like legacy
A custom error code nobody gave a status (the tier handlers' TierNotFound) was refused as a bare InternalError, so peers matching on the code never saw it. The legacy writer answers such an error 500 with its own code and message; handler_error now does the same for every status-less custom code. Errors with an unwritable header are still refused.
2026-11-19 01:21:28 +08:00
overtrue e0afeba48f refactor(storage): read existing object tags for policy inside the authorization module
The authorization core moved under storage/authorization still called the
legacy FS type for this lookup, a reverse dependency the layer guard refuses.
The lookup now lives beside its only policy caller and FS delegates to it.
2026-11-19 01:02:44 +08:00
Henry Guo 9e57f83885 feat(storage): add opt-in EC layouts and frame-aware local reads (#9255)
* perf(ecstore): reduce allocations for bounded bitrot frames

* test(s3): cover dynamic block layouts over HTTP

* perf: measure quantized range read shape

* chore: checkpoint quantized read metrics draft

Save work in progress for collaboration. Counts omit retired readers and transient or cancelled hedges. Frame semantics, async recorder coverage and disabled-metrics overhead remain unresolved; these counters are not ready for performance conclusions.

Validation: rustfmt and ecstore check passed; io-metrics 120, quantized 4 and hashed-read 3 tests passed. No new live or performance validation.

* perf(ecstore): reduce dynamic layout read overhead

* fix(ecstore): avoid prefetching conservative short range tails

* perf(ecstore): compact deferred shard reader state

* perf(ecstore): include shard checksums in file buffer cap

* fix(ecstore): preserve bounded quantum frames during heal

* perf(ecstore): reuse owned quantized frame batches

* perf(ecstore): batch q reads and retain shared copy framing

* fix(ecstore): preserve typed quantized read errors

* perf(ecstore): avoid zero-fill and serial copies in local batch reads

The local batch reader zero-initialized every window at a nonzero offset
before reading it, and copied every shard serially inside one blocking
task. Two-segment q-window range reads (cross-shard, cross-block) pay
both costs twice per request.

Read each window with a positioned read into uninitialized Vec capacity,
and dispatch each shard separately when every window is at least 512 KiB,
where copies dominate blocking-pool round trips. Smaller windows keep one
coalesced dispatch.

* perf(ecstore): keep readahead pipelined for local batch windows

A single window-sized pread on cold pages waits on each kernel readahead
chunk in turn. Seek to the window start and fill the buffer with
read_to_end over a length-limited reader instead: it still reads into
uninitialized spare capacity, but its growing sequential reads keep
readahead pipelined. Warm windows cost the same.

* Revert "perf(ecstore): keep readahead pipelined for local batch windows"

This reverts commit c252fca525.

On the ESSD-backed cloud footprint, the growing read_to_end reads were
slower than one window-sized positioned read for cold windows at nonzero
offsets (cross-block -27% on 4 MiB layouts, cross-shard -21% on 64 KiB
layouts versus the previous batch reader). The local benchmark that
motivated the change ran on a WSL virtual disk and did not represent the
target storage. Keep the single positioned read into spare capacity and
the per-shard dispatch for large windows.

* perf(ecstore): choose the batch read method by window size

Measured on ESSD-backed cloud storage, a cold window of about 1 MiB is
fastest as one positioned read, which completes as a single device
request, while windows of 2 MiB and more are faster with growing
sequential reads that keep readahead pipelined (one large read waits on
each readahead chunk in turn). Use one positioned read up to 1.5 MiB and
a seek plus length-limited read_to_end above it. Both paths read into
uninitialized spare capacity; per-shard dispatch for large windows is
unchanged.

* perf(ecstore): read local batch windows with an exact readahead hint

Kernel readahead sizes its first request from the read size. A cold
16 KiB window at offset 0 pulled 64-128 KiB, and a single 2 MiB read was
split into serial readahead chunks that cached almost twice the window,
which is why neither the single positioned read nor growing sequential
reads won for every window size on block storage.

Announce each window with POSIX_FADV_WILLNEED, which submits exactly the
window at once, and then fill the exact spare capacity with positioned
reads. This replaces the size-based choice between the two read methods.
The hint is advisory and its failure does not fail the read. A residency
test checks that cold reads cache only their window; it failed with the
hint disabled (a 5-page window cached 32 pages).

* perf(ecstore): use half-shard read quantum at 1 MiB EC layout

* revert(release): defer certification rollout for 1.0.5 (#9254)

Revert e5563e6004 only on release/1.0.5. Its trusted verifier workflow and immutable-candidate integration are not available yet. Restore the previous publication path under the approved release policy; retain the completed product regression and preview acceptance requirements.

* fix(ecstore): preserve legacy full-shard framing for default 1 MiB layout

* perf(ecstore): enable direct-memory GET and coalesce cross-shard range reads for bounded-quantum blocks

* chore(ecstore): remove unused schedule_bitrot_reader_task wrapper

* fix(ecstore): run delete-recovery discovery off the metadata lane

Bucket-wide delete-recovery discovery starts with the server and walks the
whole bucket tree in pages of up to 4096 entries, one lstat per entry. Each
page ran on the per-disk namespace queue's foreground lane, which has a
single worker and also serves GET metadata reads. On a cold tree a page
holds that worker for hundreds of milliseconds, so every GET on the disk
waits behind it until the walk finishes.

Run the page on the queue's sync lane instead, as the delete-transaction
inspection already does for the same reason. The namespace lease and the
page bounds are unchanged.

The regression test occupies the foreground lane and requires a discovery
page to complete within ten seconds; it times out without this change.

* test(ecstore): align read_appending rollback assertions on error

* fix: align multipart and EC read paths with current workspace APIs

---------

Co-authored-by: Xiaoyang Han <lux1an@qq.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-19 00:48:49 +08:00
overtrue 283332f6f8 refactor(gateway): drop the error conversions the owned authorization core made redundant
The gateway handlers, the owned envelope entry points, the protocol client and
the admin account-permissions probe now receive the owned S3 error directly.
2026-11-18 22:16:18 +08:00
overtrue 23fff7976b refactor(storage): import request markers and ReqInfo from their own modules
The admin account-permissions handler, the storage facade and the compat envelope
tests stop reaching them through storage::access.
2026-11-18 22:16:18 +08:00
overtrue 49274736f2 refactor(storage): move the test envelope payloads into storage/authorization 2026-11-18 22:16:18 +08:00
overtrue 9bc60944be refactor(storage): move the policy decision and discovery checks into storage/authorization
authorize_request_core, its entry points, the denial log and the ListBuckets and
action-potential discovery decisions return the owned S3 error type; the legacy
hooks and tests convert at the call. The two source-reading guards that looked
for the table data-plane functions in access.rs (moved by the previous slice)
now read storage/authorization/table_data_plane.rs.
2026-11-18 22:16:18 +08:00
overtrue 011d8bee6b refactor(storage): move authorization condition building into storage/authorization
The request facts, condition map and existing-object-tag conditions return the
owned S3 error type; the legacy tests convert at the call.
2026-11-18 22:16:18 +08:00
overtrue 66eb65da06 fix(iam): revoke group-only grants for existing OIDC credentials 2026-11-18 21:47:39 +08:00
overtrue 85898d94f9 test(server): name the legacy service through the listener's own import 2026-11-18 21:14:46 +08:00
overtrue caefc17dd1 test(gateway): build the layer fixture with the control-plane host's builder shape 2026-11-18 21:14:46 +08:00
overtrue c677030610 test(server): compare control-plane answers strictly and widen the caller table
A JSON comparison of two bodies that are not JSON no longer agrees, which
turns two vacuous cases into byte comparisons. The table gains unknown,
deleted and service-account keys and a repeated Authorization line, the
caller probe gains header and query session tokens, and the console
listener's isolation is pinned in front of the control-plane host.
2026-11-18 21:14:46 +08:00
overtrue 08deaeea38 fix(server): decide claimed control-plane requests before reading their body
The control-plane host ran S3Router::admit inside its handler, after the
gateway had buffered the whole body, so an anonymous chunked admin PUT
made the host hold up to 20 MiB before answering 403. The host now
attaches each request's head before the gateway reads it and decides it
by S3Router::admit at the route-authorization stage, holds at most the
legacy 1 MiB of a claimed body, and answers a repeated Authorization line
as legacy RustFS answers an anonymous caller, never serving it as one.
2026-11-18 21:14:46 +08:00
overtrue 17b01efc89 test(server): compare the content type of control-plane JSON answers 2026-11-18 21:14:46 +08:00
overtrue 88d8cc86b1 feat(server): remove the legacy fallback from the gateway stack
The gateway selection no longer holds the legacy service: claimed routes go to the control-plane host, and a selection without one answers 501. Tests drive every request class through both selections and assert neither reaches the other stack, and compare the host with the legacy stack over real IAM and signatures.
2026-11-18 21:14:46 +08:00
overtrue 640f872054 feat(gateway): serve claimed control-plane routes on a gateway host
The host dispatches every admin, table-catalog, profiling, STS form, replication, listen and Object Lambda request through S3Router::admit and the existing route table, with the legacy stack's identity, body ceiling and answers. Probe tests pin which requests reach a handler and as whom.
2026-11-18 21:14:46 +08:00
overtrue ee333a2774 refactor(admin): decide control-plane access by one owned rule
Design (T2.11): serve every route the admin router claims on the gateway
stack without the s3s service.

- Extension point: a RustFS-owned dialect on a second gateway service, the
  control-plane host. One claimed operation, rustfs:ControlPlane, takes the
  six RustFS claims (admin, MinIO alias, both table-catalog prefixes, both
  profiling triggers) exactly, with a trailing slash and every descendant,
  for every method; a RustFS form operation takes the STS form at /; the
  eight query-discriminated extension and Object Lambda rows are the
  gateway dialect's own S3-table operations. No gateway change is needed.
- Identity: the gateway verifies whatever is presented (floor admits
  header, presigned and anonymous; the caller secret is handed to every
  operation, as s3s hands it to every custom route) and the envelope is
  built with RequestEnvelope::from_gateway, the mechanism the native admin
  bridge already uses.
- Extracted here: S3Router::admit, check_access's rule over owned method,
  URI, headers and "signed", and router::dispatch, the route-table answer.
  The legacy S3Route impl now calls both; the console and Object Lambda
  branches stay with the legacy call, and an envelope-based Object Lambda
  entry is added for the gateway host.
2026-11-18 21:14:46 +08:00
overtrue 177572b0d9 fix(auth): enforce explicit denies on own-password and MFA changes 2026-11-18 21:13:39 +08:00
overtrue 2e286d3220 chore(rustfs): remove redundant S3 error conversions flagged by Clippy 2026-11-18 21:13:39 +08:00
overtrue cfa93b9b40 fix(iam): enforce account status permission on user upserts 2026-11-18 21:13:39 +08:00
overtrue 142cff57a3 fix(gateway): restore the gateway-stack build after the rename-batch merge
The squash of #9489 carried two hunks of an unlanded POST change that call
post_effective_headers, which main does not define, so every build with the
gateway-stack feature failed. Read the request headers as before; the POST
change reapplies cleanly on top. Also drops trailing whitespace that fails
cargo fmt --check.
2026-11-18 20:41:23 +08:00
Hauser Zhi 447f18db43 fix(ecstore): preserve degraded reads during cold startup (#9490)
* fix(ecstore): admit degraded local startup at read quorum

Keep absent local volumes in configured offline slots and admit existing native formats with nonvoting blank replacements. Preserve legacy validation and elected migration authority before degraded reads. Add startup, migration, readiness and real S3 cold-start regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): separate startup reads from metadata writers

Install committed startup metadata under a shared namespace fence when no writer work is required. Revalidate under the existing exclusive pipeline before bootstrap, recovery, repair, or topology side effects, and check fence validity at publication.

Cache existing-format read admission and avoid legacy scans below native read quorum for ineligible nonwriters. Cover shared-lock contention, authority bytes, released fences, and completed-process and cold-store retries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): separate legacy read compatibility from migration

Validate observed legacy layouts against an established native read quorum without requiring legacy migration quorum. Reuse known unavailable native slots to avoid repeated peer RPCs. Keep migration behind its original elected-writer, availability and strict-majority requirements, and grant no bootstrap authority to read-compatible outcomes.

Add full-loader regressions for blank plus unreachable or missing slots, both creator roles, conflicting evidence and unchanged authority files.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route startup test through storage facade

Use the existing internal NamespaceLocking re-export in the startup regression, preserving the architecture boundary checked by CI. Production code and the trait identity are unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(iam): preserve virtual parent collision invariant

Keep the federated binding regression focused on colliding regular and STS policy mappings. Virtual OIDC parent keys are now intentionally reserved and cannot identify regular users; that invariant already has direct IAM coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(gateway): project POST form metadata safely

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(gateway): preserve form authority across feature boundaries

Keep archive extraction decisions on projected POST form headers in gateway builds while preserving ordinary header handling without that feature. Update the checksum regression to model the signed form source instead of a forbidden wire object property.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-18 20:25:44 +08:00
Hauser Zhi 975e232a63 chore(deps): refresh dependabot dependency set (#9492) 2026-11-18 20:17:43 +08:00
Hauser Zhi d921f3cb24 perf(ecstore): batch remote rename fanout (#9489) 2026-11-18 20:17:11 +08:00
overtrue 62117054d9 fix(iam): serialize group edits against durable membership 2026-11-18 19:21:55 +08:00
overtrue 2f51a39c64 test(gateway): prove the gateway stack serves an unsigned x-amz-cf-id without SigV4HeaderGuard 2026-11-18 19:19:33 +08:00
overtrue 292c288896 build(gateway): pin the unsigned x-amz-cf-id reading of the RustFS profile 2026-11-18 19:19:33 +08:00
cxymds d8519549d1 test(ssec): honor deferred refusals and legacy error responses (#9503)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-18 19:16:02 +08:00
cxymds c3efe4be2a fix(gateway): align POST controls and console isolation tests (#9501)
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-18 19:15:05 +08:00
Chris An f56b2185b5 fix(storage): use the runtime consumer boundary for request access (#9508) 2026-11-18 19:10:50 +08:00
overtrue d0218bb870 refactor(storage): move table data-plane authorization into storage/authorization
The table publication guards, list cursor and table-resource checks return the
owned S3 error type; the legacy tests convert at the call.
2026-11-18 18:38:36 +08:00
overtrue 10cb203c76 refactor(storage): move the bucket-generation fences into storage/authorization
The generation guards, snapshots and their loaders return the owned S3 error
type; legacy-typed callers convert at the call site.
2026-11-18 18:38:36 +08:00
overtrue 1d3137e52a refactor(storage): move the authorization request trait and access phase into storage/authorization
The request facts trait, its accessors and the access-phase admission no longer
live in the legacy access hook file; they return the owned S3 error type.
2026-11-18 18:38:36 +08:00
overtrue a94c00fc83 test(gateway): prove the gateway stack writes a bare attributes ETag without ObjectAttributesEtagFix 2026-11-18 18:07:09 +08:00
overtrue bef37ecd6a test(gateway): prove the gateway stack serves lengthless bodyless requests without EmptyBodyContentLengthCompat 2026-11-18 18:07:09 +08:00
overtrue 9b24e8cef1 test(gateway): prove the gateway stack hints at virtual hosts without VirtualHostStyleHint 2026-11-18 18:07:09 +08:00
overtrue 71fb13ed31 test(gateway): prove the gateway stack lists buckets on GET // without DoubleSlashListBucketsCompat 2026-11-18 18:07:09 +08:00
overtrue 0f60640e8d test(gateway): prove the gateway stack words SignatureDoesNotMatch without S3ErrorMessageCompat 2026-11-18 18:07:09 +08:00
overtrue 1cd2b12df1 test(gateway): prove the gateway stack refuses unsigned amz headers without SigV4HeaderGuard 2026-11-18 18:07:09 +08:00
overtrue b108663fcc refactor(storage): return the owned result type from the bulk denial audit
The moved type brought an s3s import into a file that had none.
2026-11-18 16:36:08 +08:00
overtrue 396c1d8972 refactor(storage): rehome PermissionProbeRequestFact and WebsiteRead under storage/request_markers 2026-11-18 16:11:35 +08:00
overtrue 52402c762b refactor(storage): move request markers and object action selectors out of access.rs 2026-11-18 16:11:35 +08:00
overtrue 7cbe41ad6f refactor(storage): move ReqInfo out of access.rs and type its region as the owned Region 2026-11-18 16:11:35 +08:00
overtrue 5f4188d1da refactor(app): move the WebsiteRead marker out of metadata_route 2026-11-18 16:11:35 +08:00
overtrue 18a3416a7e refactor(storage): move BulkDenialAudit into denial_audit 2026-11-18 16:11:35 +08:00
overtrue a0bf1a5ef6 refactor(app): move PermissionProbeRequestFact out of storage/access.rs 2026-11-18 16:11:34 +08:00
overtrue ac6bc21852 build(gateway): pin the legacy reading of an absent starts-with field
Moves the rustfs/gateway pin to ab5242b0: under the legacy RustFS policy
readings a starts-with condition over a field the form does not carry is
read against the empty string, as the shipped server reads it.
2026-11-18 14:59:39 +08:00
overtrue b417d25165 fix(auth): require separate permission to change OIDC providers 2026-11-18 14:37:04 +08:00
overtrue d8e8ced097 fix(replication): bound SDK control responses before aggregation 2026-11-18 13:42:19 +08:00
overtrue 86d9da837a test(e2e): expect the console listener to serve no S3 root
The console listener now answers 404 with an empty body for every route
that is not a console or control-plane route (9cefecb04), so an
unauthenticated S3 root there is no longer a 403 from the S3 service.
The old expectation described the behaviour before that hardening.
2026-11-18 13:31:27 +08:00
overtrue 7efd4e59f4 fix(iam): reject invalid user paths before mutation locks (#3269) 2026-11-18 12:57:00 +08:00
overtrue 3a23b87b31 fix(admin): check active IAM parent for STS MFA mutations (#3268) 2026-11-18 12:57:00 +08:00
overtrue 200eb34834 fix(storage): require complete retention rules (#3267) 2026-11-18 12:57:00 +08:00
overtrue cc5c0f6945 fix(multipart): refuse queued copies holding source locks (#3265) 2026-11-18 12:38:41 +08:00
overtrue 07769f052d fix(iam): detach cache partitions once for batch removals (#3264) 2026-11-18 12:38:40 +08:00
overtrue 061e5464a5 fix(storage): borrow destination options for materialized copy 2026-11-18 12:38:19 +08:00
overtrue ce232d7a79 fix: preserve suspended versioning on copy writes (#3261) 2026-11-18 12:38:19 +08:00
overtrue 8067ad1076 fix: verify synced multipart heads against their receipt (#3260) 2026-11-18 12:38:19 +08:00
overtrue ad7c0748c5 fix: preserve empty fields in delete generation identity (#3258) 2026-11-18 12:38:19 +08:00
overtrue 9192c36712 fix(gateway): answer a deferred SSE header refusal after authorization 2026-11-18 12:36:31 +08:00
overtrue 7ace7b6d77 build(gateway): pin 2aced663 for the customer-key refusals and deferral switch 2026-11-18 12:36:31 +08:00
overtrue 4ee8651837 fix: align tier admission and scanner name handling (#3257) 2026-11-18 12:10:40 +08:00
overtrue 6b0e445f0e fix(quota): hash the storage spelling for every shard lookup (#3249) 2026-11-18 12:03:13 +08:00
overtrue 6babd93693 fix(iam): reserve storage directory suffix in IAM names (#3248) 2026-11-18 12:03:13 +08:00
overtrue 748e422651 fix(storage): check retention on null-version self copies (#3247) 2026-11-18 11:55:45 +08:00
overtrue c840cd34a0 fix(lifecycle): require positive days for all copy restores (#3245) 2026-11-18 11:55:45 +08:00
overtrue 889acaae1c fix(tier): restrict endpoint authority to HTTP transports (#3242) 2026-11-18 11:55:21 +08:00
overtrue 257f14fdbf fix(tier): reject URL delimiters in bucket names (#3241) 2026-11-18 11:55:21 +08:00
overtrue 667f82bf1a fix(s3): require an explicit legal hold status (#3239) 2026-11-18 11:55:21 +08:00
overtrue 8f12b81f09 fix(gateway): state the unavailable cause through the gateway error header
Moves the rustfs/gateway pin to 6096b921 (dotted custom error codes, the
x-rustfs-error-cause header, completion checksum types) and maps the
cause a 503 carries onto the gateway's closed header member, so the
gateway stack answers 503 with its cause instead of a generic 500.
2026-11-18 11:32:24 +08:00
overtrue 92ad98b1e5 fix(gateway): leave the error document's Content-Type to the gateway when mapping a raised error 2026-11-18 11:32:24 +08:00
overtrue 69f8b27728 build(gateway): pin unsigned POST policy parity
Moves the rustfs/gateway pin to 98cb70d1: on the RustFS profile an
unsigned browser-form policy is enforced as the shipped server enforces
it and the request stays anonymous.
2026-11-18 11:32:24 +08:00
overtrue 50952e10cf style(storage): format listing budget conditions 2026-11-18 11:26:48 +08:00
overtrue 3ddbb80b20 fix(iam): limit bootstrap listing to cache partitions (#3237) 2026-11-18 11:18:46 +08:00
overtrue 941fa254fc fix(storage): exclude pre-marker objects from walk budgets (#3236) 2026-11-18 11:18:46 +08:00
overtrue d4af3cee3e style(auth): format identity checks and simplify test input 2026-11-18 11:18:27 +08:00
overtrue 850640f452 fix(admin): retain STS principal facts in ZIP tokens (#3234) 2026-11-18 10:49:29 +08:00
overtrue 1baaf0b84d fix(iam): reserve federated parent keys during user writes (#3233) 2026-11-18 10:49:29 +08:00
overtrue 83032a3671 style(storage): format local part metadata dispatch 2026-11-18 10:42:45 +08:00
overtrue e110baa3e0 fix(storage): bound legacy prefetch by stored part size (#3231) 2026-11-18 10:41:41 +08:00
overtrue f90137b73f fix(storage): dispatch part metadata path and stamp IO (#3230) 2026-11-18 10:41:41 +08:00
overtrue a5b69359d2 fix(auth): require an entity tag for the if-match policy key (#3229) 2026-11-18 10:15:54 +08:00
overtrue c6a52d87cf fix(iam): serialize table credential issuance with rotation (#3228) 2026-11-18 10:15:54 +08:00
overtrue 3e24761f0a fix(admin): redact secrets in full config exports (#3227) 2026-11-18 10:15:35 +08:00
overtrue a5b7c413a3 refactor(protocols): authorize protocol operations through the owned request envelope 2026-11-18 09:52:10 +08:00
overtrue 00f98e81cd refactor(protocols): bind the protocol client to its server context slot 2026-11-18 09:52:10 +08:00
overtrue 66a7a17c97 refactor(protocols): return owned S3 errors from the protocol client 2026-11-18 09:52:10 +08:00
overtrue 4e8245491e test(tagging): authorize through the owned config-mutation check 2026-11-18 09:52:10 +08:00
overtrue e14274a6d1 test(lifecycle): build persisted lifecycle fixtures without the s3s converter 2026-11-18 09:52:10 +08:00
overtrue 04d51631fb refactor(admin): serve object-lambda GET through owned authorization and use case 2026-11-18 09:52:10 +08:00
overtrue 25e4cf3599 refactor(rustfs): build the admin decode test body from rustfs_s3_types::Body 2026-11-18 09:52:10 +08:00
overtrue 167ca5aa28 refactor(rustfs): use rustfs_s3_types::Region outside the legacy edge 2026-11-18 09:52:10 +08:00
overtrue b097226a6f fix(delete): consume source metadata in bounded windows (#3223) 2026-11-18 09:34:55 +08:00
overtrue b2e7e79e32 fix(recovery): exclude independently owned UUID child keys (#3222) 2026-11-18 09:34:35 +08:00
overtrue 77dad01e14 test(storage): verify external overwrite by data generation 2026-11-18 09:30:12 +08:00
overtrue cc58e9f52e fix(storage): derive inline layout from destination encoding (#3263) 2026-11-18 09:30:12 +08:00
overtrue 6225de66bc style(storage): format retention metadata validation 2026-11-18 09:30:12 +08:00
overtrue 4e1bd4ebbd fix(lifecycle): bound immediate configuration scan admission 2026-11-18 09:30:12 +08:00
overtrue 8b37479a26 fix(storage): keep default retention for empty lock metadata 2026-11-18 09:30:12 +08:00
overtrue b4f28e9418 test(multipart): use a valid historical source timestamp 2026-11-18 09:30:12 +08:00
overtrue 6b86c33679 fix(multipart): use encoded ID consistently for staging (#3221) 2026-11-18 09:30:12 +08:00
overtrue f7c7e8e3af fix(disk): cap positive bucket cache and omit misses (#3220) 2026-11-18 09:30:12 +08:00
overtrue b3348ce739 test(metadata): check bucket policy size boundary and rejected updates 2026-11-18 09:30:12 +08:00
overtrue c27b6410e0 fix(metadata): bound configs and narrow admission parsing (#3219) 2026-11-18 09:30:12 +08:00
overtrue 14bb4b21c5 fix(auth): omit anonymous identity conditions (#3218) 2026-11-18 09:30:12 +08:00
overtrue 6f7207b536 fix(heal): exclude completed keys before charging page budgets (#3217) 2026-11-18 09:30:12 +08:00
overtrue 699a001c61 test(ecstore): supply lock snapshot in restore finalization fixture 2026-11-18 09:30:12 +08:00
overtrue a4bfcc0828 test(auth): compare the correct S3 error type 2026-11-18 09:30:12 +08:00
overtrue 2aa0e71f16 test(auth): resolve versioning trait in regression 2026-11-18 09:30:12 +08:00
overtrue ea43f96c73 test(auth): cover null versions below excluded parents (#3240) 2026-11-18 09:30:12 +08:00
overtrue 0486e880d7 fix: enforce security invariant for 5TIGS9Y7 2026-11-18 09:30:12 +08:00
overtrue 4e4cbcb065 fix: enforce security invariant for LG8HTUD5 2026-11-18 09:30:11 +08:00
overtrue 9cefecb045 fix: enforce security invariant for CMISVGAS 2026-11-18 09:30:11 +08:00
overtrue 217e8224f8 build(gateway): pin aws-chunked and DeleteObjects parity fixes
Moves the rustfs/gateway pin to fb4d21fc: an empty DeleteObjects entry
ETag is carried per entry, and the RustFS profile reads aws-chunked
bodies as the shipped server does (one empty line before signed
trailers, legacy trailer bounds, XAmzContentSHA256Mismatch, legacy
framing refusal codes).
2026-11-18 07:33:04 +08:00
overtrue 2a661f4382 refactor(e2e_test): use owned Body and header constants instead of s3s 2026-11-18 07:32:38 +08:00
overtrue 2af5a12c52 fix(gateway): accept a region list on the gateway stack
RUSTFS_REGION may list several regions; the gateway assembly passed the
whole list as one region name and refused to start.
2026-11-18 06:37:23 +08:00
overtrue 2ce53cdec8 fix(gateway): state accept-ranges once on SOSAPI and ODM GET answers 2026-11-18 06:37:23 +08:00
overtrue 5d95a06b29 fix(s3): refuse a version marker without a key marker as InvalidArgument
ListObjectVersions answered 501 NotImplemented for a malformed cursor;
a client-caused refusal is a 400 (rustfs/backlog#2684 K-7).
2026-11-18 06:37:23 +08:00
overtrue 180a4b0c6a fix(gateway): write the legacy default message for a message-less error
A use-case error raised with a code and no message reached the gateway
stack as an empty <Message>, where the legacy edge has always sent the
per-code default. The owned code type now carries that table, checked
row by row against the pinned s3s, and the one error edge applies it.
2026-11-18 06:37:23 +08:00
overtrue fa30644cbb fix(policy): fail closed on unresolved resource scopes 2026-11-18 05:53:57 +08:00
overtrue ee27c0d7db fix(policy): enforce nested resource recursion limit 2026-11-18 05:53:57 +08:00
overtrue 4220570e3d fix(policy): preserve nested resource variable semantics 2026-11-18 05:53:57 +08:00
overtrue 5afe8585e5 fix(policy): reject empty resource variable values 2026-11-18 05:53:57 +08:00
overtrue ce313fcc61 fix(policy): preserve exact wildcard character matches 2026-11-18 05:53:57 +08:00
overtrue 1b001fdb2b perf(policy): keep ordinary wildcard matching allocation-free 2026-11-18 05:53:57 +08:00
overtrue 15555f9441 fix(policy): preserve deny across variable alternatives 2026-11-18 05:53:57 +08:00
overtrue 943b2ae531 fix(policy): keep substituted resource variables literal 2026-11-18 05:53:57 +08:00
Chris An 5fd15193f9 fix(startup): skip update checks when client initialization fails (#9495) 2026-11-18 05:46:17 +08:00
overtrue 4b13deb567 fix(ecstore): preserve target version and lock checks during multipart restore 2026-11-18 04:57:28 +08:00
Chris An 4aded279ae docs(storage): clarify drive isolation for separate clusters (#9493) 2026-11-18 04:45:19 +08:00
overtrue 75b3dce2e0 fix(test): preserve parity parameter identifiers 2026-11-18 03:12:19 +08:00
overtrue 9df834a955 feat(gateway): bind preflighted POST archive replay 2026-11-18 03:10:37 +08:00
overtrue 969191ebe6 fix(s3): refresh archive member tag conditions 2026-11-18 01:12:50 +08:00
overtrue 17f1164581 test(gateway): reject incomplete declared POST forms 2026-11-18 01:06:03 +08:00
overtrue 6b60b64b81 fix(s3): enforce archive member policy overrides 2026-11-18 01:04:20 +08:00
overtrue 3ab62090c2 fix(gateway): fail closed on unbound POST archives 2026-11-18 00:32:50 +08:00
overtrue 1c4a7c2f8b feat(gateway): serve browser object uploads 2026-11-18 00:32:50 +08:00
overtrue dc83d478f8 fix(test): preserve colons in pytest parameter IDs 2026-11-18 00:29:35 +08:00
overtrue a18231a2b3 build(gateway): pin verified POST policy support 2026-11-18 00:25:06 +08:00
overtrue 0717dc1b0d fix(gateway): honor configured unread body timeout 2026-11-18 00:18:06 +08:00
overtrue a68e299baa fix(s3): enforce copy source bucket owner 2026-11-17 23:58:32 +08:00
Hauser Zhi f2003af73d fix(ecstore): recover deletes on full disks (#9465)
* fix(ecstore): reserve emergency space for delete journals

Keep bounded physical metadata headroom for recoverable deletion on Unix without bypassing quorum or rollback.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): recover delete metadata space on demand

Share a pinned reserve owner per physical disk and keep healthy deletion free of reserve filesystem probes. Retry only disposable staging and idempotent metadata primitives after actual ENOSPC, preserving transaction publication and quorum boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): retry concurrent full-disk deletes

Serialize ENOSPC waiters per physical disk and give each safe metadata primitive one bounded retry against shared released capacity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): account for delete reserves in receipt fixture

Reject undersized tmpfs mounts after including the eight per-disk delete reserves and keep a self-test for the capacity boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): refresh resume admission fixture lease

Refresh the existing node admission sample immediately before starting resume GC so the cancellation test reaches the native-open ownership boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): inject receipt ENOSPC on hardlink path

Keep the receipt fault at the atomic allocation boundary after the stage-two hardlink fast path landed, so reserve recovery remains covered.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-17 07:53:38 +00:00
overtrue f47840b3c2 feat(gateway): serve native UploadPartCopy
Resolve multipart ownership from storage metadata and share authorization across both request edges. Preserve the existing legacy adapter while binding gateway claims to the recorded namespace.
2026-11-17 11:00:34 +08:00
Hauser Zhi 7fb9b3501f fix(ecstore): allow missing deletes at degraded quorum (#9478) 2026-11-17 09:28:48 +08:00
Hauser Zhi cbe5b16758 test(rpc): verify timeout boundary evidence (#9482) 2026-11-17 09:28:21 +08:00
GatewayJ a33fa8a226 fix(rpc): retain cancelled streams and repair CI validation (#9439)
* fix(build): disable cuobject defaults at the workspace level

* fix(rpc): retain cancelled streams during metadata read bursts

* fix(deps): use published NIXL bindings with scoped advisory policy

* fix(test): size multipart refusal fixture threads for real storage

* fix(release): wait for uploaded artifact metadata visibility

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 16:22:06 +00:00
Ramakrishna Chilaka bfff795635 perf(ecstore): defer flat inline preflight and lazy-pin commit dirs (#9459)
* perf(ecstore): defer flat inline preflight and lazy-pin commit dirs

Eliminate redundant directory opens, ancestor prefix probes, and metadata stats on the Linux local-disk object commit path:
- Short-circuit validate_existing_local_disk_prefix_at on ENOENT for normalized relative paths under RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS.
- Defer source directory pinning and new-destination fstat in RenameCommitGuard on flat inline commits.
- Defer destination volume preflight to the commit guard preparation step on flat inline commits while preserving VolumeNotFound/VolumeAccessDenied preflight rejection proofs.
- Stage rollback backups by linking the held predecessor file descriptor via /proc/self/fd/<fd> and extend hardlinked rollback staging to non-inline overwrites.

* fix(ecstore): sync linked predecessor metadata when sync_backup is enabled

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 15:53:23 +00:00
overtrue fb104ab2c1 feat(gateway): serve extension catalog on the native admin stack
Wire both admin aliases through AdminBridge and the existing owned ServerInfo authorization. Keep the business handler and gateway pin unchanged; add no API or instance endpoint.

Add three test entry points covering eight allowed and 80 refusal/fallback rows, both hosts and aliases, complete JSON/XML, response headers, and unread unsigned bodies with signed controls. Production diff: +12/-3 lines.

Baseline and mutation base: 94a138b994. The unmodified native-selection test failed as expected. One compiled cohort passed eight controls, then eight faults failed their intended assertions: authorization bypass, native selection fallback, JSON, content type, Content-Length, refusal XML, premature body polling, and premature handler entry.

Restored sources were hash-checked and actually recompiled on 0278e1cd0e: 31 scoped tests, fmt, both RustFS Clippy modes, script-tests and test wiring passed. Existing snapshot checks now accumulate wire failures before asserting.

Not verified: CI, socket end-to-end or the full workspace suite.

Validation: cargo fmt --all --check; both rustfs all-target Clippy modes with -D warnings; 31 scoped restored tests; make script-tests; test wiring. No CI, socket-level end-to-end run, or full-workspace suite was run.

One compiled fault cohort first passed eight controls. Authorization bypass, native-route fallback, corrupted JSON, content type, content length and refusal XML each failed native_admin_extensions_catalog_real_signature_iam_and_wire; route fallback also failed native_admin_selection_is_exact_and_host_independent. Premature body polling and handler entry each failed native_admin_extensions_catalog_unsigned_before_body. Temporary instrumentation was removed, exact hashes restored and touched, and the product actually recompiled.

Restored test and lint base: 0278e1cd0e. Existing snapshot wire assertions remain enforced through the shared failure collector.

Integration verification base: 913edbc536 with the Iceberg config increment retained. The final dialect registers ten operations and four claims. Twenty-six selected tests passed: fourteen native admin HTTP tests, four mounted admin fixture tests, seven original extension handler tests and the exact edge selector. Both all-target RustFS Clippy modes, formatting, script-tests, current test wiring and prepush guards passed. Source hashes matched after verification. No new assertions or business behavior were introduced during integration.
2026-11-17 05:43:38 +08:00
overtrue 251eee0cba feat(gateway): serve Iceberg catalog config natively
Route both config GET aliases through GetCatalogConfigHandler. Keep its owned IAM, session-claim and warehouse handling, with the shared Iceberg JSON error outlet. Only these exact GET paths use the native bridge.

The matrix covers 12 allowed and 52 refused rows across both aliases and hosts, including full JSON, temporary credentials, query ordering and observed handler entry. Pin early duplicate-query rejection and the native anonymous message explicitly. The legacy baseline ran on b84985286.

Seven faults made the IAM, table-token, warehouse-prefix, alias-selection, handler-counter, JSON-outlet and complete anonymous-message assertions fail. Each had a passing control with the same compiled source. Restore hashes and a real restored build were checked.

Validation on 0278e1cd0: 16 scoped tests, cargo fmt and RustFS Clippy with and without gateway-stack passed. make script-tests passed with unchanged scripts. Production diff: +18/-9, excluding test-only probes.

No catalog storage, credential vending, catalog commits or other Iceberg operations were migrated. Full workspace, TCP/external SDK e2e and CI were not run.
2026-11-17 05:43:38 +08:00
overtrue 676875576f refactor(protocols): own UploadPartCopy client boundary
Route the protocol client and legacy adapter through one owned multipart
copy use case, preserving storage ownership, fallback context, completion
accounting, conditional order, XML ETags, and error messages and causes.
Share copy-source validation and retain the original 1024-byte key limit.

Validation: original-main fixtures, 23 focused runtime tests, 38 effective
behavior mutations, restored-source runtime, default and gateway-stack
all-target Clippy, script tests, and final integration check and prepush.
2026-11-17 05:40:29 +08:00
hector a54e2ea386 feat(runtime): raise default thread stack size to 2 MiB (#9452)
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 15:35:37 +00:00
Pernav Jain 913edbc536 fix(s3): return NoSuchVersion for missing Object Lock versions (#9468)
fix(s3): return NoSuchVersion for lock reads

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 11:09:55 +00:00
Ramakrishna Chilaka 909b7c3c6f perf(ecstore): hardlink stage-2 delete receipts and pin live proc roots (#9470)
Reduce syscall and sync overhead on local-disk DELETE and PUT paths:
- Hardlink the Stage 1 delete-prepared regular file to delete-quorum-committed on the cached Stage 2 local publish path when Stage 1 synced the journal, avoiding a duplicate temp-file create, write, fdatasync, and rename per drive while retaining the staged wrapper fallback on conflict or durability escalation.
- Accept bare DeleteRecoveryJournal payloads as unfinished (finished = false) commit receipts in parse_commit_receipt while keeping delete-prepared validation strictly DeleteRecoveryJournal.
- Reuse validated io_get_bucket_and_object_path components when constructing the root-relative metadata lease key in delete_version_inner on Linux.
- Track live /proc/self/fd/<fd> PublicationRoot handles via Weak<File> in NATIVE_PUBLICATION_HISTORY so record_native_publication skips redundant statx calls on pinned uncollided procfs roots.
2026-11-16 11:00:11 +00:00
Hauser Zhi 571d100a27 perf(ecstore): align inline placement and batch deletion codecs (#9449)
* feat(ecstore): align inline budgets and binary batch decoding

Use a fixed 128 KiB per-shard inline budget with versioned admission and
preserve historical configuration markers and persisted object layouts.

Decode binary delete batches independently of the legacy JSON list while
retaining JSON-only requests and dual-write senders. Reject malformed
binary entries before disk mutation and preallocate the selected batch.

Cover budget boundaries, layout coexistence, MinIO metadata fixtures,
binary-only batches and JSON compatibility. Document geometry and
rollback boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): add isolated Docker inline geometry validation

Build clean-source Linux validation images from a frozen Git bundle.
Pin running image identities and binary hashes, and inspect persisted
object geometry through the existing filemeta decoder.

Provide an owned four-node Docker adapter for the existing ABBA runner.
Guard resource ownership, isolate loopback access, serialize state
mutations, and verify HEAD, full GET and Range contents.

Keep PUT content probes available through an isolated Linux mc adapter
when a host client cannot initialize. Cover ownership and failure paths
with script tests wired into the existing runner.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse bounded verified inline bodies

Share the materialization contract between metadata pre-verification
and buffered inline GETs. Bound responses to 2 MiB and a limited frame
count, and reuse an already verified body within the selected snapshot.

Route proof-protected parts through the readers that bind PartProofReader.
Cover larger single-block bodies, allocation bounds and proof validation
after a shard's bitrot checksum has been recomputed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(ecstore): support bounded multi-block inline objects

Keep the default EC block and accumulated shard budget while bounding inline frame storage and verified-body reuse. Preserve transformed and protected write admission and existing metadata formats.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate inline transport comparison modes

Pin and verify both MessagePack-only gates for each Docker cell, preserve an explicit JSON compatibility cell and probe exact accumulated shard boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover inline rollback and mixed-reader fleets

Add reusable compatibility fixtures for null and versioned objects, cross-block ranges, unfinished multipart uploads and batch version deletion. Record coordinator identity and per-version geometry, preserve retained-data manifests and archive manifests on dataset resets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify bounded inline read and write paths

Share checked frame decoding across metadata preverification and GET, resolve write admission once, and use one checked physical shard length for inline and disk writers. Preserve proof-bound readers and erasure recovery for damaged or missing shards.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): persist inline compatibility mutation receipts

Save successful multipart and delete receipts before fallible content and geometry checks. Resume completed mutations through verification without repeating writes, and cover read failures, geometry failures, and command retries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(gateway): isolate capability probes from multipart listings

Give successful capability probes their own buckets so CreateMultipartUpload does not change the exact two-upload pagination fixtures. Preserve all listing-boundary assertions on both HTTP stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(rpc): honor fleet-confirmed codecs for batch deletion

Build DeleteVersions requests with one codec policy for the complete batch. Emit JSON compatibility fields by default and binary-only batches after both existing fleet gates pass. Cover request fields, canonical binding and actual remote sender, handler and disk deletion in both modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): verify inline shards through borrowed frames

Share borrowed frame verification across buffered GET, bitrot readers and deep inline healing. Validate exact physical framing before scanning memory slices. Cover short prefixes, legacy padded frames, damage and trailing data, and provide a reproducible manual ABBA kernel comparison.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): account for manual inline verification benchmark

Raise the ECStore ignored-test ceiling from 2 to 3 for the opt-in
ABBA comparison of borrowed and streaming bitrot verification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject inexact inline shard framing

Validate the complete encoded length of every selected inline data shard
before decoding or caching a verified object body. Return to the erasure
reader on missing, truncated, or trailing payload bytes.

Extend decoder fixtures across single and multi-block geometry, legacy
padding, unhashed payloads, and explicit bitrot verification bypass. Cover
metadata early-stop rejection and GET recovery for trailing shard bytes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): share inline shard reader construction

Build owned-metadata and borrowed-metadata readers through one Send iterator-based factory. Preserve shard ordering, empty slots, bitrot settings and caller-bound integrity verification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(rpc): move endpoint buffers into batch delete requests

Transfer the prepared endpoint String into the protobuf request and assert allocation identity across every batch codec mode. This removes the intermediate endpoint copy during request construction.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(gateway): remove superseded pagination isolation

Use the upstream capability probes that clean up their own multipart sessions. Run probes against the seeded pagination bucket while keeping all exact page and cursor assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): recognize borrowed multipart guard calls

Count owned and borrowed bucket/key guard forms while requiring all four production calls. Cover mixed split-file calls and reject wrong resources, missing error propagation and test-only mentions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): size heal cleanup fixtures above inline budgets

Use an environment-independent per-pool fixture policy and payload sizes above the aggregate inline budget. Assert external placement before pausing old-data cleanup, rebuilding the committed generation and checking physical shards and the final payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): count borrowed object metadata guards

Recognize the borrowed bucket/key guards in migrated legal-hold and retention entrypoints. Validate two object and two key guards and cover incorrect resources, missing propagation and test-only mentions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-16 10:11:19 +00:00
unixglo e4ac38a011 fix(s3-client): decode URL-encoded listing names (#9448)
Signed-off-by: unixglo <unixunit@outlook.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 09:31:08 +00:00
Hauser Zhi 849cdd6319 feat(ecstore): stage quorum-proof metadata cache (#9454)
* feat(lock): carry cache fence quorum proofs

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(ecstore): stage proof-bound distributed cache

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(ecstore): refine cache activation blockers

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-16 08:38:54 +00:00
Hauser Zhi dc7f4e943c fix(ecstore): unify commit durability and reduce inline dispatches (#9451)
* fix(ecstore): honor file-sync policy in inline fallback commits

Apply metadata, directory, and recovery synchronization policies separately while retaining rollback publication barriers. Add sync-budget regressions and small-object I/O profiling guidance.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify recovery-aware commit synchronization

Share recovery-directory policy across fused and fallback commits. Complete fresh file-sync publication within preparation and verify queue budgets, rollback recovery, and inline-to-external overwrites.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate sync counts across reused disk handles

Consume the metadata synchronization count for every external write attempt before releasing a Linux procfs-backed fixture, including failed publication and retry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-16 08:06:58 +00:00
GatewayJ 0278e1cd0e fix(iceberg): recover renames and unblock namespace deletion (#9440)
* fix(iceberg): recover renames and unblock namespace deletion

* fix(iceberg): fence rename recovery with restore lease

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-16 23:54:03 +08:00
overtrue 94a138b994 feat(gateway): serve cluster snapshots on the native admin stack
Register GET /{rustfs,minio}/admin/v4/cluster/snapshot with the existing owned handler and ServerInfo authorization. The read-only use case is unchanged.

Reuse the admin fixture for full JSON and refusal XML comparisons, response headers, exact route selection, and unsigned rejection before reading a body. The native matrix covers 8 allowed and 80 refusal or fallback rows; the body observer adds four signed controls and four unsigned refusals. Update the existing registration census.

Validation on b84985286: 35 scoped tests, RustFS Clippy with and without gateway-stack, cargo fmt, and make script-tests passed. Full workspace, distributed e2e, and CI were not run.

Mutations: IAM bypass, route removal, JSON replacement, and event-time corruption turned the signature/IAM/wire contract red. Reading the unsigned body turned the body observer red. Content-Type, Content-Length, and an extra XML Resource node each turned the corresponding wire assertion red. A temporary entry counter observed signed calls and no unsigned handler entry.

Production diff: +12/-2. No new public API or business logic.
2026-11-16 23:18:03 +08:00
Hauser Zhi b3cdea0c15 fix(targets): decode Kafka notification record keys (#9450)
* fix(targets): decode Kafka notification record keys

Build Kafka records from the decoded object identity so record keys match
the notification envelope for immediate delivery and persisted replay.
Cover form encoding boundaries, historical queue round trips, and early
rejection of invalid UTF-8 metadata.

Fixes #9437

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(targets): share allocation-efficient event key construction

Use a borrowed form decoder and exact-capacity key buffer for notification
envelopes and Kafka records. Reuse the established percent decoder and
preserve the owned public decoder API and persisted replay metadata.

Add exhaustive malformed-escape and UTF-8 compatibility checks, plus a
regression for borrowing plain object names.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-16 07:49:17 +00:00
overtrue b849852863 refactor(protocols): call owned UploadPart use case
Route the protocol client through its installed multipart context and owned
values. Preserve request headers, response and error projection, counting,
and audit completion. UploadId keeps one scalar compatibility projection.
UploadPartCopy, PostObject, gateway handlers and ownership rules are unchanged.

On f68ef7d5f: fmt, both all-targets Clippy modes, 37 scoped tests, and script-tests
passed. The 14-fault cohort had a green 37-test control; restored sources were
hash-checked, touched and recompiled before the final 37-test pass. On 489316e7a:
gateway-stack all-targets check and prepush-quick passed after rebase.

New runtime cases: 13 negative, 2 positive. Projection matrices cover 60 input,
64 output and 6 error rows. Non-test lines: +122/-20. No existing test was
deleted, weakened or ignored.

Effective mutations and failing tests (under protocols::client::multipart::
tests::upload_part_tests):
- part-number, length, md5, sse, eager-body, metric, audit:
  upload_part_refusals_match_original_without_persisting_parts
- checksum: upload_part_input_projection_matches_original_fields
- output-field, etag-order:
  upload_part_output_projection_matches_original_fields_and_refusals
- error-default, error-explicit:
  upload_part_error_projection_matches_original_defaults
- ambient, observer: upload_part_client_preserves_stored_bytes_and_bound_context

Not run: full workspace, protocol wire lanes, released-binary differential,
live KMS/SSE integration, or CI.
2026-11-16 21:40:00 +08:00
overtrue 489316e7a2 refactor(protocol): dispatch CopyObject through the owned usecase
Replace the protocol client compatibility dispatch with owned request and
response projections while preserving source validation, conditions, dates,
ETags, error answers, bound storage, fallback behavior, and operation counts.

Validated against the existing client and library behavior on f68ef7d5.
Final gateway-feature tests passed 11/11, including existing PutObject
regressions; default-mode tests passed 6/6. Both all-target Clippy modes,
script tests, wiring checks, and static preflight passed.

Twenty-two effective mutations covered observation controls, operation
counts, context selection, storage effects, source and condition parsing,
optional fields, dates, output fields, error facts, and request credentials.
Restored sources were rebuilt before the final passing controls.

This does not register UploadPartCopy or change the default S3 stack.
External migration lanes and release-reference testing remain outstanding.
2026-11-16 21:06:46 +08:00
overtrue f68ef7d5f6 refactor(protocols): use owned multipart lifecycle paths
Route CreateMultipartUpload, CompleteMultipartUpload and AbortMultipartUpload through the installed server's multipart use case. Keep the measured protocol projections and operation counts.

The upload-id scalar compatibility helper remains in Complete and Abort. UploadPart, UploadPartCopy, PostObject and persisted-owner work are outside this change.

Validation: 11 new tests; the runtime contract has 8 refusal cases and 2 successful flows. The restored selection passed 59/59 on d3abf8023. Both Clippy modes, fmt, script-tests, test wiring and prepush-quick passed. Non-test diff: +231/-42. The original client baseline ran on a98bd9028.

Effective mutations (38), grouped by the test that failed:
- multipart_create_projection_matches_original_fields_and_times: create-input-name, create-input-metadata, create-input-tagging, create-input-storage, create-input-sse, create-input-kms, create-input-md5, create-input-mode, create-input-hold, create-input-date, timestamp.
- multipart_refusals_match_original_client_errors: complete-error-message.
- complete_input_preserves_part_fields: legacy-part-checksum.
- multipart_create_projection_preserves_refusals: create-input-refusal.
- multipart_create_output_matches_original_for_stored_names_and_empty_values: create-output-name, create-output-strings.
- multipart_complete_projection_matches_original_parts_and_conditions: complete-condition, complete-parts, complete-part-number, complete-part-etag, complete-part-checksum, complete-condition-refusal.
- multipart_completion_projection_preserves_original_escaped_tags: complete-output-escape, complete-output-weak.
- multipart_completion_projection_preserves_header_only_field_behavior: complete-output-headerdrop, complete-output-header-validation.
- protocol_multipart_lifecycle_stores_parts_and_aborts: count-create, count-complete, count-abort, complete-location, complete-storage.
- multipart_uses_bound_store_with_a_foreign_ambient_context: create-storage, abort-storage, bound-store.
- multipart_envelopes_preserve_original_request_facts: request-facts.
- protocol_policy_denials_and_request_conditions: permission-create, permission-complete, permission-abort.

The final restored run had no leaky summary; an earlier control marked one pure projection test leaky. The linker reported the non-failing __eh_frame warning. The ambient fault's test exited 100 with target=false/ambient=true; its wrapper exited 1 because it expected generic assertion wording.

Not verified: CI, network protocol suites and the released-reference comparison.
2026-11-16 19:09:59 +08:00
overtrue d3abf80237 refactor(protocols): call owned CreateBucket use case
Call the existing bucket use case directly from the protocol client. Preserve
bucket-name materialization, response facts, edge counts and bound or fallback
store selection. Keep legacy DTO imports confined to tests.

Tests: one new runtime matrix, with seven rejection and four success rows,
measured first against main at 70022e51c. Restored gateway bucket tests passed 4/4.
The old dispatch failed the source wiring guard; owned dispatch passed it.

Mutation checks: normal controls passed; the runtime matrix rejected storage
bypass, missing fallback, metric loss, ambient context selection, strict names,
and changed error code, message, status or headers. Isolated storage, fallback
and ambient-context faults retained normal operation counts. Restored sources
were touched and actually recompiled before the runtime pass.

Validation: fmt, package Clippy in both feature modes, script-tests and test
wiring. Production diff: +19/-14 lines. No new API or storage use case.
Not verified: CI, full workspace tests or live protocol end-to-end tests.
2026-11-16 18:17:06 +08:00
overtrue 70022e51c4 refactor(protocols): call owned bucket use cases for listings
Use the bucket use case for ListObjectsV2 and both bucket listing entry points. Keep entry encoding, timestamps, ETags and refreshed session facts while removing three legacy FS calls. Other protocol operations are outside this increment. Non-test diff: +102/-82.

Added six tests: projection rows have 34 refusals and six accepted values; runtime cases have five refusals and eight successes, plus bucket exclusions and four credential rows. Existing assertions were retained.

Validation: baseline 2fea17d57 (2/2); restored build on a98bd9028 (15/15); all-target Clippy default and gateway-stack; fmt, script-tests, test wiring and prepush-quick. Rebased onto 88515bb7d with no task-file or pin overlap; all task blobs match the tested source.

Eight independent controls and 31 intended mutation failures:
- protocol_listing_output_matches_legacy_projection: strict Name, oversized key, invalid/missing ETag, XML escaping, weak tag, prefix, marker, encoding, timestamp.
- protocol_listing_bucket_projection_preserves_strict_names_and_milliseconds: bucket filtering, timestamp.
- protocol_listing_pages_and_encoding: cursor, count, encoding.
- protocol_listing_refusals_and_limits: empty/unknown credentials, counts.
- protocol_listing_bucket_context_and_defaults: count, bound context.
- protocol_listing_session_credential_projection_matches_legacy: token, secret.
- protocol_session_bucket_listing_revalidates_the_session_key: request context, IP, secure transport, headers, refresh, accepted count, refusal count, store context.
- request_extensions_preserve_authenticated_identity_and_source_ip: owner.

Experiments were removed, hashes checked and the restored product actually recompiled. Full protocol listing E2E and CI were not run.
2026-11-16 16:58:48 +08:00
overtrue 88515bb7d0 feat(gateway): serve admin usage and catalog reads
Register data usage, plugin catalog and module-switch GET reads through
the existing AdminBridge and handlers under both admin aliases. Keep
neighboring methods and object-lambda routing on their existing paths.

Validation on 65057691: unchanged-production legacy baseline passed;
restored-source scoped tests passed 78/78. The new native HTTP matrix
covers 28 success and 236 refusal or routing-boundary rows. Exact replies
are pinned for both aliases and ordinary and bucket Hosts.

Three independent no-fault controls and 21 effective mutations passed
in one compiled cohort: selector removal/widening/lambda, each handler
binding, each IAM gate and action, both narrowed data-usage any-of gates,
signature verification, lambda dispatch, response bytes/status/request
id/content type and Keystone precedence. Exact refusal observations also
caught the disabled-key signature mutation. No existing assertion was
removed or weakened.

Gateway and default all-targets Clippy, script tests, test wiring and the
local prepush guards passed. No gateway pin or API changed.

Not verified: production listeners and full middleware, session
credentials, non-default persisted usage/module-switch data, full
workspace tests or CI.
2026-11-16 16:43:17 +08:00
overtrue a98bd9028d refactor(protocols): use owned bucket head and delete paths
Call the existing bucket use cases directly from the protocol client. Preserve root lookups, bucket validation, request context, response facts and operation counts. Other protocol operations stay as they are.

Non-test lines: +33/-40. Added cases: 17 refusals, 16 successes; reused three HeadBucket policy refusals. Protocol tests passed 39/39 in default and gateway-stack builds. Both rustfs all-target Clippy modes and make script-tests passed.

Four no-fault controls passed; 22 faulted test runs failed as intended. Mutations covered input routing, name refusals, output and error facts, counters, deletion, context resolution and authorization allow/deny. The force-delete fault also changed the observed bucket/object state. The raw HeadBucket NUL-name projection went red before its response text was restored.

Tests and Clippy ran on 2fea17d57. Script-tests ran on 2ca689f48; their sources are unchanged. Rebased onto 65057691 with unchanged client and gateway-pin blobs. Network transport e2e, workspace tests and CI were not run.
2026-11-16 16:12:18 +08:00
overtrue 65057691b6 feat(gateway): serve signed admin system reads
Serve the storage info, runtime capabilities and object cache statistics
GET aliases through the native admin service. Call the existing full IAM
gates and owned handlers, preserve raw request facts and keep the Keystone
guard first.

Add two contract matrices (42 allowed and 168 refused rows), with expanded
selector, registry, envelope and readiness coverage. On 2fea17d57d, the
restored source passed 72 scoped tests, fmt and both rustfs all-targets
Clippy modes. Script tests and test wiring also passed.

29 effective fault runs cover selection, lambda precedence, all three
dispatch/action/authorization bindings, signature verification, source IP,
raw URI/query, request context, readiness, response bytes/status/headers/
length and the registry. Nine controls ran in separate processes against
the same binary before those faults.

Not verified: live listeners and full production middleware, session
credentials, active cache statistics, CI or the full workspace tests.
2026-11-16 15:35:13 +08:00
overtrue 2fea17d57d feat(auth): bridge Keystone identities into the gateway stack
Pair a single-use Keystone proof with sanitized owned authorization facts.
Preserve the original session-presence fact for policy principal projection,
and scope legacy credentials during both service call construction and polling.
Bind the gateway host-token principal to that owned identity without an IAM
credential or task-local fallback.

Initialize Keystone before gateway HTTP assembly and reuse the provider for
listeners and later auth integration setup. Refuse token-bearing claimed
non-S3 requests before console, admin or legacy dispatch.

Add 14 test functions. The primary ListBuckets/HeadBucket matrix has 13 rejected
and 7 allowed requests. Change enabled-Keystone startup to succeed after provider
initialization; retain both configuration tests, including disabled and invalid
settings. Production changes: +304/-77 lines, excluding test bodies.

Mutation checks (20 effective boundaries; each has a green control):
- owner, groups, username: keystone_http_tokens_authorize_real_s3_by_roles_groups_and_username
- pair-name: keystone_envelope_refuses_missing_and_mismatched_owned_identity
- provider, expiry: keystone_bridge_refuses_provider_mismatch_and_post_validation_expiry
- stale-established, stale-owned: keystone_stack_clears_both_payloads_before_a_signed_request
- legacy-before-deny, native-before-deny, admin-before-deny: gateway_keystone_tokens_are_refused_before_non_s3_legacy_dispatch
- scope: disabled_middleware_clears_identity_before_inner_call_and_poll
- principal-session: resource_free_conditions_preserve_legacy_projection_and_erased_keystone_session_facts
- secret, session, session-fact, replay, poison: request_identity_erases_secrets_and_consumes_its_proof_once
- repeat-init, missing-child: keystone_initialization_reuses_the_provider_for_concurrent_calls

Verification on 2ca689f: 214 focused tests passed after restoring and rebuilding
the production sources; all 33 Keystone package tests passed. Both package-scoped
all-target Clippy configurations, formatting, source guards, script tests and
prepush checks passed. Temporary fault code was removed.

No gateway API, dependency pin or PostObject changes. A deployed Keystone server,
the complete listener token lane and CI were not exercised locally.
2026-11-16 15:02:13 +08:00
overtrue 2ca689f486 refactor(protocols): call the owned PutObject use case
Project protocol PutObject directly into the owned write input and project the answer back without the legacy DTO path. Keep the measured normalization, refusals, error facts, server context and operation counts at the client edge.

Non-test lines: +196/-18. Eight added tests include 25 runtime cases (18 negative, 7 positive).

Verified on f4266a4576 with gateway pin 44f9a2422d745ffba952347ec91b0332ef2667a4: protocol tests 36/36 in default and gateway-stack builds; both all-target Clippy builds; script-tests; prepush-quick.

Sixteen runtime mutations failed their intended tests from one fixed binary after all eight controls passed:
- empty: protocol_put_optional_values_match_legacy
- condition, specific-none: protocol_put_condition_grammar_matches_legacy
- input-checksum: protocol_put_checksum_input_matches_legacy
- retain-nanos, retain-range: protocol_put_retain_until_matches_legacy
- output-etag, output-fields, output-checksum: protocol_put_output_matches_legacy
- error-default, error-facts: protocol_put_error_projection_matches_legacy
- event-hold, event-days, event-years, body, metrics: protocol_put_runtime_matches_legacy

The experimental code was removed and the restored implementation was recompiled and tested in both builds. The earlier incorrect metric lookup is not counted as evidence.

Scope is protocol PutObject. Network protocol E2E, the full workspace and CI were not run.
2026-11-16 14:15:53 +08:00
overtrue b2c30d69c2 feat(gateway): bridge the signed admin info aliases
Serve GET /rustfs/admin/v3/info and /minio/admin/v3/info through a
scoped native service using the existing signature verifier and the real
ServerInfoHandler IAM gate. Keep object-lambda routing ahead of these
aliases and retain the raw URI, optional ReqInfo, headers and body.
The pinned RustFS profile already handles claims on bucket-style hosts.

Outside tests and the generated lockfile: +180/-14 lines. Add seven tests,
including 32 authorization refusals and 16 authorized info responses, both
SourceIp directions and signed legacy-service comparisons. No existing
assertion was weakened. Duplicate-query signing is tested as a fixture
refusal, with routing coverage retained separately.

Validation on f4266a4576: fmt, both rustfs
Clippy modes, 42 scoped tests, script-tests and prepush-quick.

Twenty-four mutation runs failed as intended across 22 faults:
- native_admin_selection_is_exact_and_host_independent: remove/widen
  selection and lose object-lambda priority.
- native_admin_real_iam_and_legacy_contract: bypass signature verification
  or IAM, change the action or source IP, omit pools, rewrite the URI,
  or lose object-lambda priority.
- native_admin_envelope_preserves_raw_alias_host_facts_and_optional_req_info:
  rewrite the URI, drop/default ReqInfo, or drop client, peer or request ID.
- native_admin_complete_response_keeps_bytes_status_and_content_type:
  corrupt bytes, status, content type, extra headers or content length.
- native_admin_authenticated_request_refuses_an_uninstalled_server_context:
  remove the readiness check.
- native_admin_registers_one_operation_and_both_claims: register an extra op.
- native_admin_signing_fixture_refuses_duplicate_query: accept duplicate keys.
The same mutation binary passed all seven controls before fault runs;
the restored production tree was then rebuilt and tested.

Not exercised: network listeners, session credentials, the combined
Keystone change, CI or other admin endpoints.
2026-11-16 14:01:00 +08:00
Hauser Zhi fb9f19c6d2 fix(rpc): scope canary errors to grpc failures (#9453)
* fix(rpc): scope canary errors to grpc failures

Exclude expected read-version remote responses without hiding transport, gRPC status, response-decode, or other gRPC operation failures.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rpc): verify canary revisions from image history

Use immutable image history only when recognized config labels are absent, keep mismatches fail-closed, and record the provenance source in verifier output.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(audit): restore bucket config audits and metadata guards

Audit accelerate, request-payment, and website configuration mutations through the shared owned request scope, including failures and the website deletion status.

Validate the replication mutation delegation before checking peer reload notifications. Count borrowed reserved-prefix guard calls and cover missing production calls, test-only mentions, audit completion, and stale bucket incarnations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-16 13:57:56 +08:00
overtrue f4266a4576 refactor(protocols): use owned object reads and deletion
Route both GetObject calls, HeadObject and DeleteObject to the owned use cases. Keep the request facts and millisecond timestamp behavior. PutObject stays unchanged; 12 operations and 13 trait calls remain.

The client edge is an interface in the layer guard. Its public re-export is covered too, and app-to-client imports are rejected by fixtures.

Runtime wiring: +177/-101 non-test lines. The new matrix has 18 negative and 13 positive operation cases, plus the extended identity-copy control.

Validation: 28 protocol tests passed in both default and gateway-stack modes; both all-target Clippy modes passed. Script tests, layer controls and focused architecture/footprint/test guards passed. Five source faults failed at the intended claims, range, Head ETag, deletion-witness and version assertions. The precision oracles failed before the millisecond projection was added.

Runtime and Clippy ran on e9da1d309; initial controls and faults ran on 2308dee4b2. Rebase to 00ecde125 changed none of the seven task files or gateway pins. CI and HTTP transport were not checked.
2026-11-16 12:11:38 +08:00
overtrue 00ecde1250 feat(gateway): dispatch console routes through native Axum
Use the existing console router when the gateway console is enabled. Keep lambda and extension precedence, the legacy selection, and outer middleware assembly unchanged. The common body conversion stays at the edge.

Add test-only assets and one contract with 10 native and 52 excluded routing probes. Non-test source: +89/-16. The restored run passed 22 console/stack tests on e9da1d309; both Clippy modes, script-tests and prepush-quick passed.

native_console_route_contract went red for three independent mutations: remove the native branch (service-call counts), widen the shared prefix (neighboring S3 path), and route every claim to console (anonymous admin refusal). Restoring the sources triggered a real product recompile.

Full outer HTTP wire checks and a production UI bundle were not tested. Admin, STS and website migration remain separate work.
2026-11-16 11:47:40 +08:00
overtrue e9da1d3096 feat(gateway): serve DeleteObjects through the owned engine
Share the batch deletion engine between legacy and native edges with owned inputs and results. Keep exact per-entry authorization, version and retention handling, ordered errors, quiet responses, metrics, and audit aggregation. Preserve frozen resource resolution and rejected-key echoes.

On 2308dee4 with gateway 44f9a242, the focused suite passed 208 tests before and after deliberate mutations. Both rustfs Clippy configurations, script tests, test wiring, formatting, and prepush guards passed. Four added runtime tests cover mixed successes and per-entry refusals, frozen fields, and rejected-key state.

Five targeted tests failed when batch target clearing, denial error mapping, and raw-key echo were changed. Sources were restored byte for byte and product artifacts recompiled.

The legacy edge matched 62 observed baseline responses including all headers, XML and object states. Native decoder and expected-owner differences are tracked separately. Actual webhook records matched for mixed and quiet requests across the baseline and both candidate stacks.

Full workspace tests and CI were not run. Verification is scoped to this migration increment.
2026-11-16 10:24:21 +08:00
overtrue 2308dee4b2 fix(gateway): preserve object retention date offsets
Use the validated request offset and full timestamp precision when projecting PutObject and PutObjectRetention dates into authorization and storage.

Signed requests verify exact stored dates and version isolation for positive and negative offsets and UTC. Policy tests cover denied dates and the typed date supplied by PutObject authorization.

On b2b4f0c4, 175 focused tests and six raw xl.meta checks passed. The UTC-only control and five targeted mutations failed at their intended assertions. Both Clippy feature configurations passed.
2026-11-16 08:39:36 +08:00
overtrue b2b4f0c491 feat(gateway): serve SelectObjectContent natively
Share GetObject authorization and the existing Select query producer across both stacks. Carry live events through the typed response with the projected SSE headers. The producer retains its single deadline, snapshot checks, bounded channel and receiver cancellation.

On c5b4a8577, the restored build passed 207 focused tests, both RustFS Clippy feature modes, formatting, script tests and test wiring. Signed socket probes matched all 17 legacy cases and 12 gateway cases. Five gateway refusals omit the legacy default message, tracked in rustfs/backlog#2684.

The fault build passed all 98 controls. Bypassing authorization failed on both stacks. Removing receiver-close handling, dropping the terminal Records permit, skipping the Stats fence, replacing an error with End, and dropping progress or scan range each failed the corresponding test.

Verification used local stores and SSE-C. The full workspace suite and external KMS were outside the local run.
2026-11-16 07:29:41 +08:00
overtrue c5b4a85778 feat(gateway): serve RestoreObject through the owned use case
Share the authorization snapshot and restore admission between both stacks.
Preserve HTTP 200 for both success paths and emit ObjectRestorePost on new
admission only.

Verified on c0763a43 with 114 focused tests, four serialized ILM tests, and
16 signed wire responses per stack. Faults covered shared authorization,
generation capture and admission, SELECT rejection, worker parameters and
liveness, and both audit and event paths. Cluster lock loss and mixed-version
rollout remain unverified.
2026-11-16 06:30:48 +08:00
overtrue 4ae180ea40 feat(gateway): serve object legal hold and retention writes
Share each operation's authorization and metadata write with the legacy edge.
Keep their request ordering, version and bucket-incarnation fences, replication
completion, and notification suppression. Retention checks the locked metadata
again before committing a change.

The two stacks were exercised through signed HTTP requests and independent
disk metadata reads. Faults in authorization, body projection, version
selection, counting, auditing, notification suppression, scanner invalidation,
the bucket fence, and the commit-time retention check reached their assertions.
2026-11-16 06:19:43 +08:00
overtrue c0763a43f5 feat(gateway): serve bucket object lock configuration
Share the bucket Object Lock write between the legacy and gateway edges. Keep lock configuration and versioning as sequential writes against the authorized bucket incarnation, followed by peer reload, site replication and scanner invalidation.

Cover eight refusal or failure cases and three successful cases with durable metadata and signed requests. The two stacks run the same request cases and refusal matrix.

The mutation run detected both missing write fences, swallowed peer failure, propagated site failure, early or absent scanner invalidation, missing audit, wrong authorization action or resource, dropped request body and incorrect operation counting.

Validation used debug builds on d461908bf. The final lint pass elides an equivalent lifetime in a test helper. Peer and site failures were injected locally, with real multi-site delivery left unverified.
2026-11-16 05:35:18 +08:00
overtrue d461908bf0 feat(gateway): serve bucket replication configuration writes
Route PutBucketReplication and DeleteBucketReplication through the owned bucket use case. Share site checks, metadata fencing, target cleanup recovery, and audit completion with the legacy edge.

Signed requests exercise 28 refusals and six accepted edits across both stacks. The direct storage tests cover six failure paths and two successful edits. Twenty-four faults turned their expected disk, authorization, locking, callback, audit, or metric assertions red.

The restored gateway-stack build passed 132 selected tests on 75b293f09, including both HTTP stacks. Both Clippy variants and the mutation cohort passed on fb3e2818c. The fixed legacy CLI matched all six response records. The standalone default runtime suite, multi-site delivery, and CI were unverified.
2026-11-16 04:32:24 +08:00
overtrue 5b8876ad0e feat(gateway): serve bucket policy, encryption, and tagging writes
Share the owned write paths across both S3 stacks. Preserve the policy
document bytes, default KMS key resolution, bucket tag validation, and
incarnation fences before reload and replication callbacks.

Add three successful writes and three rejected stale writes on disk.
Extend the two-stack refusal matrix with six successful and 24 denied
requests. Cover the encryption and tag validation order before lookup.

The fault cohort exercised the action and resource checks, value
validation and order, raw policy bytes, default KMS readback, legacy
policy count, durable writes, incarnation fences, reload, replication,
audit, and scanner invalidation. Each selected assertion failed.

Local checks used base 75b293f09. Fixed CLI comparisons covered legacy
responses, audit entries, and operation counts. Live peer replication,
external KMS providers, and CI were outside the local checks.
2026-11-16 04:24:32 +08:00
overtrue 75b293f094 feat(gateway): serve encryption, policy, and tagging deletes
Share the three generation-fenced deletes across both S3 stacks. Keep
their existing IAM actions and use the shared IAM scope for tag deletion.

Add six focused tests covering 21 refusals and nine successes, and add the
three operations to the refusal matrix. Substituting IAM actions and
buckets, removing the tagging scope, and suppressing the gateway counter
fail the authorization and metrics checks. The storage tests detect a
lost fence, a skipped write, missing reload or replication calls, omitted
scanner invalidation, and a changed audit status.

Verified on 4294c41cc with scoped nextest, both Clippy modes, script tests,
quick checks, and local CLI comparisons. Peer network delivery was
outside the local checks.
2026-11-16 03:21:06 +08:00
overtrue fb3e2818c7 feat(gateway): serve bucket ACL and logging writes
Route PutBucketAcl and PutBucketLogging through shared bucket use cases.
Preserve the ACL response for header-only requests and the bucket lookup
before the XML policy refusal. Write logging configuration using the
authorized bucket incarnation and the existing persistence format.

Verify 111 focused tests on 4294c41cc, including the storage access suite,
and run both Clippy modes, script tests, and local wire comparisons. The
wire matrix exercises 15 refusal cases and seven successful writes on
each stack.

Change both IAM actions and resources, audit statuses, and counters in
the fault cohort. Reverse ACL lookup and XML policy checks, drop policy
presence, corrupt logging bytes, reverse logging conversion order, and
remove the incarnation guard. The regressions detect each fault, and
the stale-authorization test reads the replacement bucket from disk.
2026-11-16 03:08:10 +08:00
overtrue 4294c41ccf feat(gateway): serve bucket public access block operations
Route Get, Put, and DeletePublicAccessBlock through shared bucket use
cases. Preserve the persisted switch values, mutation incarnation fence,
reload behavior, audit status, and missing-configuration response.

Add six focused tests covering 25 refusal cases and 17 success cases
across both stacks and storage. Extend the refusal and counter matrix
for all three operations.

Substitute each IAM action and bucket, corrupt each operation's value,
change the missing-config status, and suppress each edge counter. The
round-trip and matrix tests reject those faults. Put and Delete tests
reject a missing incarnation fence, the wrong reload mode, and wrong
audit status. Read the durable replacement-bucket bytes after disabling
both fences.

Verify on 92050cdca with scoped nextest, both Clippy modes, script tests,
and local wire tests. Peer reload delivery was outside these local tests.
2026-11-16 02:10:24 +08:00
overtrue 3cb0aa137e feat(gateway): serve core bucket configuration mutations
Serve accelerate, request-payment and website writes, including website
deletion, through shared use cases. Both S3 edges enforce the bucket
policy and the metadata incarnation fence before changing stored data.

Verified on 92050cdca with formatting, Clippy on both RustFS feature
sets, script tests, 21 RustFS tests and 27 operation and metrics tests.
The fixed legacy CLI supplied 42 HTTP comparison cases per stack.

The wire assertions caught altered IAM actions and resources, missing
denial audits and operation counters, and skipped website validation.
The stale-incarnation test caught changes to the replacement bucket's
disk bytes. The lookup-order test caught early configuration conversion.

Refs rustfs/backlog#2734
2026-11-16 02:07:32 +08:00
Hauser Zhi a8cf62f8f3 fix(metrics): label grouped fdatasync batches accurately (#9446)
* fix(metrics): label grouped fdatasync batches accurately

Record the serialized user-space fdatasync coordinator as grouped and reserve parallel for independent per-file fanout. This keeps performance dashboards from conflating two different dispatch models.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): separate grouping from timeout race

Keep the same-disk batching test focused on queue aggregation. Claimed-follower timeout ownership remains covered by its dedicated deterministic regression, avoiding a valid pre-claim timeout race in the batching fixture.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): accept borrowed multipart guard calls

Count both owned String and borrowed &str spellings of the same reserved-prefix guard. UploadPart now obtains borrowed bucket and key values from its typed target, while the production guard remains in place.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 12:51:57 +00:00
Hauser Zhi d0fd6425e4 feat(rpc): guard stale generations before dispatch (#9435)
* perf(grpc): attribute rebuilds and bound bulk responses

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): attribute rebuild success after publication

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): add exact HTTP/2 ping timeout diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): run refusal matrix on large stack

Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.

Keep the original test name and all refusal-order assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): pin nixl-sys workspace source

Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.

Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): authenticate msgpack peer capability

Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(grpc): bind peer capability to connection generations (#9433)

* feat(grpc): track physical connection generations

Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): bind msgpack capability to connections

Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): guard stale generations before dispatch

Reject generation-bound requests before inner transport call or first payload frame when tonic readiness observes a different physical connection, while leaving ordinary mutations unguarded until the send-policy follow-up.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): add per-peer msgpack rollout gate

Add the default-off peer-capability environment gate, cached evaluation, fixed-cardinality send-decision metrics, and rollout constraints for stable peer-affine connections.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): apply per-peer msgpack send policy

Capture one capability decision per eligible RPC, bind peer-mode requests to boot and connection generations, and rebuild at most one dual-write request only after a locally proven pre-dispatch rejection.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): verify msgpack peer canary evidence

Add a strict three-phase four-node evidence verifier with raw Prometheus and workload receipts, adversarial tests, script-test wiring, and the operational manifest contract.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): align batch read manifest encoder

Match the BatchReadVersion compatibility manifest to the request-owned encoder expression used by the production send path.

Restore the owning ECStore source guard while preserving wire and runtime behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): orchestrate msgpack peer canary

Run the fixed mixed/all-new/rollback four-node matrix with prebuilt images, real workload and readback, raw Prometheus capture, strict manifest generation, and adversarial fake integration tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rpc): verify canary image and telemetry identities

* test(rpc): bind canary evidence provenance

Scope canary metrics to one injected cluster identity and bind candidate, baseline, and per-phase running containers to immutable image IDs and exact build revisions.

Reject foreign-cluster metrics, stale revision labels, same-image tags, and phase image mismatches.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): accept borrowed multipart guard calls

Count the exact borrowed bucket/key guard call introduced by UploadPart target ownership while preserving the four-production-call contract.

Cover the borrowed spelling in both the positive fixture and the test-only false-positive regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-15 12:37:25 +00:00
overtrue 92050cdca6 feat(gateway): serve PutObjectTagging through the shared use case
Share XML tag authorization and storage execution across both stacks.
Preserve the requested version, replication-target proxy, and metadata
completion order.

The dual-stack tests exercise 37 refusal and 13 success scenarios.
The restored integration run passed 148 tests on 26db9b19a, and the
proxy cleanup verification passed 134 run, 134 passed.
Both feature configurations passed Clippy, and script tests passed.

Fault injection caught native and shared authorization bypasses,
missing XML facts, and incorrect action, resource and tag key facts.
It also caught duplicate counting, incorrect persisted tags, missing
or duplicate success events, and event or dirty publication on a miss.

Live replication targets and ECStore write faults remain unverified.
2026-11-16 00:29:49 +08:00
overtrue 26db9b19af feat(gateway): serve core bucket configuration reads
Serve bucket acceleration, request payment, website, and logging
configuration through shared persisted read paths. Keep each operation's
existing authorization and legacy response projection. Append distinct
metric labels without shifting existing indexes.

The scoped RustFS, S3 operation, and metrics suite passed 283 tests on
993464fbb. The integrated configuration and refusal suites passed 21
tests on 292974baf. Both feature modes passed all-targets Clippy.

Standalone checks compared 23 successful and 17 refused legacy requests
against cfdd4b268 and exercised the gateway answers on the same data.
Action, resource, anonymous, projection, default, missing-bucket, counter,
and audit faults failed their original assertions. Metric registry tests
also detected duplicate labels, changed index order, and an added event.
Verification is scoped to these local suites and wire probes.
2026-11-15 23:29:07 +08:00
overtrue 292974baff feat(gateway): serve bucket CORS, encryption and tagging reads
Share bucket-scoped authorization and persisted configuration reads across
both stacks. Carry the stored configurations in gateway responses and
preserve the operation-specific errors for missing configs.

On RustFS 993464fbb, 120 scoped tests passed, including both refusal
matrices and six new tests with 24 refusal and 12 success cases.
Live HTTP checks covered response documents, operation counts and
authorization-denial audit records. Both feature configurations passed
Clippy, and formatting, script tests and source guards passed.

Changing each IAM action or resource admitted a forbidden read.
Duplicating each edge count and changing each stored output failed its
test. Returning 404 for missing encryption failed the 400 assertion.
Suppressing gateway denial audit removed all nine expected records.
Runtime verification used single-node debug servers.
2026-11-15 22:48:03 +08:00
overtrue 993464fbbb feat(gateway): serve GetObjectAttributes through the owned path
Share the read use case and both IAM decisions across the legacy and
gateway handlers. Keep attribute selection, stored metadata and
pagination in the use case, and encode the result at each edge.

The 6f93ac5f control matched 55 legacy wire cases and the restored
sources passed 139 scoped tests. Both stacks exercised 59 signed
requests, including 31 legacy refusals and 32 gateway refusals.

Seventeen injected faults each turned a targeted assertion red. They
cover the four IAM actions, resource and version binding, empty-version
selection, handler authorization, SSE-C, selection, both checksum paths,
paging, ETag, Last-Modified, counting and notification suppression.
2026-11-15 21:41:16 +08:00
overtrue cfdd4b2686 feat(gateway): serve PutBucketNotificationConfiguration
Share one owned write path across the gateway and legacy edges. Project
runtime destinations at each edge and defer consumption of the persisted
conversion result until after filter validation. Keep IAM scope, the
incarnation fence, and the existing validate, write, reload, clear, and
add order. Count the request at its edge and audit it in the use case.

Add three refusal tests and four tests that exercise success and failure. The legacy HTTP baseline passed on 97a7639f. On 6f93ac5f the
38-test control passed. Six fault injections exposed wrong IAM action,
wrong IAM resource, a missing incarnation fence, duplicate counting,
the wrong audit status, and missing reload callbacks. The first three
changed the stored bytes guarded by their refusal tests.

The restored tree rebuilt and passed all 38 scoped tests. Both package
Clippy configurations, script tests, test wiring, and prepush passed.
Cross-node reload delivery and remote audit sink delivery were outside
the local test scope.
2026-11-15 21:18:37 +08:00
overtrue 6f93ac5f47 feat(gateway): serve AbortMultipartUpload
Use the same authorization and audited use case on both S3 stacks. Bind
the gateway claim to the owner selected under the upload write lock
before staged parts can be removed. Preserve the existing generation,
lifecycle and quorum checks.

Verified the fixed legacy baseline, the integrated native wire flow, 212
focused tests and both Clippy modes. Fault checks exercised owner
selection, deletion and orphan handling, scoped IAM, generation
transport, audit, operation counts and dirty usage.

Audit delivery and separate fault injection of the existing final
lock-loss and metadata-incarnation fences were outside this verification.
2026-11-15 18:05:23 +08:00
overtrue 620fdc286a feat(gateway): serve CopyObject through the owned path
Resolve the source from the gateway read proof. Share source and
destination authorization, tag checks, and the existing storage copy
between both S3 stacks.

Checked the legacy wire baseline, both CopyObject wire contracts, and the
access and refusal suites. Nine temporary faults triggered the intended
assertions. The restored build passed seven focused tests with gateway
75757292, and Clippy passed with and without gateway-stack.
2026-11-15 18:02:37 +08:00
overtrue 58310c569b chore: pin RustFS to gateway mainflow update
Refresh the four RustFS gateway dependency pins to gateway commit 75757292b9617080cbb1bb7f880db8da9b34912c. Update the gateway facade and types versions; keep stream and HTTP versions unchanged.

Verification: cargo check --locked -p rustfs --all-targets --features gateway-stack, layer dependency guard, architecture migration guard, and prepush-quick passed. No Rust sources or tests changed.
2026-11-15 17:42:10 +08:00
overtrue e36c7452ac test: cover DeleteBucketLifecycle in the refusal matrix
Seed a lifecycle configuration so the admitted request removes real state.

Verified the matrix census and both refusal matrices in the CopyObject development checkout on 93947f22c.
2026-11-15 16:41:34 +08:00
overtrue 48e97253b5 feat(gateway): serve ListParts through the owned path
Share authorization and pagination across both S3 edges. Resolve the
native upload claim using the owner from the same quorum-selected
metadata as the parts page. Preserve typed storage errors and write
the page through each edge's response projection.

The scoped suite includes all 98 storage access tests, real upload
sessions, stored-owner combinations, both wire paths, and the storage
API. Twenty-three fault modes exercised the owner, authorization,
lookup, error, counter, and projection assertions.

Validated 168 focused tests after restoring and recompiling the candidate
on c2b5fbb plus the isolated fixture cleanup. The owner matrix covers
three positive and six negative cases; each wire IAM matrix covers one
positive and seven negative cases. Both feature configurations passed
Clippy. Full migration acceptance lanes remain pending.
2026-11-15 16:24:31 +08:00
overtrue ce5bd8b117 test: isolate CreateMultipartUpload in the refusal matrix
Abort the session returned by the successful matrix call through the
fixture store so later pagination assertions see their own sessions.

Both listing boundary tests failed on c2b5fbb and passed with the
fixture cleanup. Both refusal matrices passed with the same cleanup.
2026-11-15 15:32:04 +08:00
overtrue 93947f22ce feat(gateway): serve DeleteBucketLifecycle through the owned use case
Share the incarnation-fenced lifecycle deletion across both edges, preserving reload and site-hook ordering and one audit record.

Validated 37 focused tests on 77da63397, both all-target Clippy configurations, script tests and wiring guards. Thirteen effective faults covered deletion, authorization wiring, incarnation fences, counting, reload and hook ordering, and audit behavior. Live peer delivery and final migration acceptance lanes remain unverified.
2026-11-15 15:28:43 +08:00
Hauser Zhi 8f361e3a6f perf(ecstore): share rename fanout metadata (#9432)
* perf(ecstore): share rename fanout metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): stabilize async regression fixtures

Run the multipart refusal matrix on the shared large-stack harness, wait for scanner metadata commit tails before restart discovery, and observe io_uring owner release alongside in-flight retirement.

Keep the production paths unchanged while preserving each original failure oracle.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:05:20 +08:00
Hauser Zhi 3cee9c154b perf(grpc): attribute rebuilds and bound bulk responses (#9420)
* perf(grpc): attribute rebuilds and bound bulk responses

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): attribute rebuild success after publication

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): add exact HTTP/2 ping timeout diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): run refusal matrix on large stack

Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.

Keep the original test name and all refusal-order assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): pin nixl-sys workspace source

Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.

Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): authenticate msgpack peer capability

Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(grpc): bind peer capability to connection generations (#9433)

* feat(grpc): track physical connection generations

Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): bind msgpack capability to connections

Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:05:09 +08:00
Ramakrishna Chilaka 504cc06d0c perf(ecstore): stage inline rollback backups and fuse backup reclaim (#9445)
Stage inline overwrite rollback backups (xl.meta.bkp) directly relative
to the pinned destination directory descriptor in RenameCommitGuard
instead of staging in .rustfs.sys/tmp and walking both paths a second
time. Also fuse post-commit xl.meta.bkp removal and empty-ancestor
directory pruning into a single blocking dispatch while stopping at live
object roots.
2026-11-15 15:04:18 +08:00
Hauser Zhi 44145ef327 chore(deps): update flake.lock (#9441)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4fe25e1' (2026-11-06)
  → 'github:NixOS/nixpkgs/2074cc6' (2026-11-12)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/d3fe742' (2026-11-07)
  → 'github:oxalica/rust-overlay/5b87ee0' (2026-11-14)
2026-11-15 15:04:11 +08:00
Hauser Zhi ba9e8342f8 test(storage): add fail-closed quiescence gate (#9436)
* test(storage): add fail-closed quiescence gate

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): gate benchmark rounds on quiescence

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): bound quiescence Docker calls

Start the storage-quiescence deadline before Docker discovery, pass the remaining budget to every subprocess, and reject samples completed after the deadline.

Cover delayed initial lookup and delayed exec through the supported Docker binary override.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:04:04 +08:00
Hauser Zhi a41019802d test(storage): expose direct-write validation controls (#9430)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:53 +08:00
Hauser Zhi 4313812e24 perf(ecstore): keep inline commits fused with file grouping (#9427)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:45 +08:00
hector 5c58a56c4c fix: recognize gVisor directfs bind mounts in the FS guard (#9426)
gVisor presents a host bind mount as 9p, so statfs reports V9FS.
When the mountinfo options include both trans=fd and directfs, the
startup filesystem guard treats that endpoint as local POSIX storage
and logs that the host type is not visible. Real 9p and virtio shares
still warn.

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 15:03:37 +08:00
GatewayJ 8196603a76 fix(iam): authorize delegated service account creation (#9408)
* fix(iam): authorize delegated service account creation

* docs(iam): clarify explicit grants for custom administrator policies

* fix(iam): include service account delegation in admin wildcards

* style(policy): format delegated authorization test cases

* fix(iam): keep delegated authorization and replication consistent

* fix(ci): include delegated account test in E2E selections
2026-11-15 15:03:24 +08:00
Hauser Zhi 1e62cbaeaf perf(ecstore): integrate safe sync and allocator diagnostics (#9428)
* feat(diagnostics): restore allocator and runtime metrics

Port the missing live-byte accounting and serving-runtime observations from 13a30abca. Reuse the current MiMalloc GlobalAlloc wrapper, expose explicit metric availability, and cover allocation operations and feature selection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): detach queued fdatasync followers safely

Adapt the follower timeout experiment from 138bc69f6 with exclusive queue ownership transfer. Move queued paths and resource units into independent sync, retain claimed batch ownership through completion, and record observed queue time. Cover publication before the original batch resumes, sync failures, claimed batches, and queue accounting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): index sync followers and isolate missing sources

Use reusable slab slots and monotonic identities for constant-time queued follower extraction. Avoid retaining follower group references when the timeout is disabled. Preserve claimed-batch ownership and reject stale tickets after slot reuse.

Keep missing-source results local to their requests so healthy requests still synchronize their own files. Preserve fail-fast behavior for other disk I/O failures. Repair failure-probe synchronization and exercise source disappearance, reused slots, disabled tracking, cancellation and bounded admission on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): apply storage stack to multipart refusal fixture

Route the real-store absent-binding refusal-order fixture through the existing bounded 4 MiB storage setup in both default and CI profiles. The Linux CI failure reproduced with the default 2 MiB stack. Quota, legal-hold, error, cause, audit and notification assertions execute through the established storage fixture budget.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): make sync worker idle handoff atomic

Move batch claim, in-flight accounting and completion onto the existing worker guard. Publish the idle state under the queue lock so an enqueue during cleanup establishes successor-worker ownership. Keep stale cleanup conditional on current ownership.

Add a controlled interleaving regression that detaches the final queued follower and enqueues a successor after the empty claim. The old worker leaves that request without an owner; the refactored worker hands it off and completes one physical sync.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): await grouped sync resource release

Wait for native producer admission units to reach zero after the blocking sync finishes. This removes a scheduler-dependent assertion window exposed by the latest main integration while preserving the cancellation and ownership checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 14:58:54 +08:00
Hauser Zhi da1817ae01 test(rpc): expose msgpack fleet confirmation (#9429) 2026-11-15 14:54:23 +08:00
overtrue c2b5fbb138 feat(gateway): serve CreateMultipartUpload through the owned path
Share authorization between both CreateMultipartUpload edges. Keep the
decoded legacy fields intact, preserve the replication destination
version and apply the captured bucket incarnation before creating the
upload. Leave checksum, encryption and retention metadata on the same
storage path.

Verify the frozen legacy behavior on 2c2af51a before the migration and
exercise both stacks with 66 refusal cases and 22 success controls.
Read back sessions, version, retention and encryption metadata.

Fault probes made authorization tests fail on both edges when an action
or the resource binding was lost. Storage tests detected a different
destination version and unexpected sessions when generation, quota, KMS
or SSE checks were removed. The wire and
helper tests failed for early name rejection, lost timestamp precision,
metadata, checksum, SSE response, the operation count and unexpected
notification.

On 77da6339, 185 scoped tests passed with a 32 MiB test stack. The default
stack run passed 184 tests and reproduced the existing overflow in
completion_absent_binding_refusal_order_baseline. Both all-target
Clippy variants, test wiring, script tests and prepush passed. After
integration with 97a7639f, gateway all-target check and prepush passed.
Whole-workspace CI and rio-v2 were outside the local verification.
2026-11-15 14:08:38 +08:00
overtrue 97a7639f59 feat(s3): serve UploadPart through the gateway
Share the streaming multipart write path and staged authorization across both edges. Resolve the native upload ID against the active storage record at the existing lookup. Carry checksums, trailers, SSE values and the stored ETag through each response projection.

Add seven tests for application lookup, route authorization, framing and durable part state. Exercise 62 refusals and 10 successful requests across those fixtures. The existing SSE-C completion test also covers the migrated UploadPart core.

Verification on 1467d32695: fixed legacy contract, 147 gateway-feature tests and 119 default tests, both all-target Clippy legs, script tests, test wiring and prepush checks. The test scopes include the entire access module. The gateway scope includes PutObject and both refusal matrices.

Twenty-two local fault controls exercised edge authorization, policy and generation, resource and upload ID projection, owner binding, lookup count and error ordering, logical length, trailer publication, checksum validation and typed producer errors. Each fault failed its intended test after the same artifact passed with the selector unset.

Linux RDMA and a native KMS denial through UploadPart remain unverified.
2026-11-15 13:41:36 +08:00
overtrue 77da63397c feat(s3): serve PutBucketLifecycleConfiguration through the gateway
Validate lifecycle status across both request adapters and preserve mutation error priority, ordered rule IDs, and stored dates. Forward the original bucket and request envelope at the legacy edge. Pin the gateway lifecycle date carrier.

The priority fixture covers seven refusal cases and one success case. All 39 scoped tests passed after source restoration. Fifteen fault switches triggered 16 expected failures covering priority, early conversion, metrics and audit, action binding, transition writes, date precision, offsets, and rule indices.

The tests and both Clippy modes passed on 0567078 with gateway 2a6b8d1. The gateway all-target check and prepush passed after rebasing onto 1467d32. Full-workspace and remote CI results remain unverified.
2026-11-15 13:08:53 +08:00
overtrue 1467d32695 feat(gateway): serve PutObject through the owned write path
Register PutObject and share write authorization with legacy PUT and
browser POST. Carry authenticated trailer declarations and captured
frames into the owned write, preserve typed eager-read errors, and keep
the table-catalog guard tied to the actual write call.

Verified on RustFS 0567078f5 with gateway d75b6ce on macOS. All 1022
affected tests passed with a 4 MiB test stack, along with both
all-target Clippy configurations, script tests, test wiring, and
prepush checks. A separate gateway wire run stored and read an empty
object using the rebuilt server binary.

Fourteen single-fault checks covered trailer selection and capture,
typed producer failures, tag policies, the shared write grants, and
POST policy projection. The reserved-prefix source guard also rejected
a fixture whose owned write call was replaced.
2026-11-15 11:51:26 +08:00
overtrue 2c2af51a46 feat(gateway): serve PutBucketVersioning
Route bucket versioning writes through the shared authorization and audited use case. Preserve legacy configuration preparation, the incarnation fence, refusal order, persisted XML, and the reload, replication, and scanner callbacks. Count each admitted write once at its edge.

Verified against RustFS 0567078f5 and gateway d75b6ceb. The signed wire cases cover six accepted writes and nine refusals per stack. The affected suite passed 29 tests, the typed-error suite passed 26, and both Clippy configurations, script tests, wiring, and prepush checks passed. The wire cases and both refusal matrices passed again after boxing the test fixture's SDK error.

Eight fault controls went red for both authorization hooks, the legacy counter, an omitted write, a dropped empty excluded prefix, Object Lock precedence, replication refusal, and the custom-status error. CI, the Linux reference comparison, and live multi-node callback delivery remain unverified.
2026-11-15 11:33:25 +08:00
overtrue 0567078f5b feat(gateway): serve GetObject through the owned read path
Register GetObject and share its authorization and native response facts
with the legacy edge. Preserve conditional headers, range completion,
stream errors, body ownership, SSE reads and request accounting. Pin the
gateway 304 response-override fix without unrelated dependency changes.
Keep the incoming native Put path and update its profiling guard source.

Validation on b7e7a6156:
- Normal and mutation-disabled controls: 9/9 each.
- Affected suite: 754/755; the profiling guard still read the old Put file.
  After correcting that source path, its targeted rerun passed 1/1.
- All-target Clippy passed with gateway-stack and default features.
- Formatting, script-tests, test wiring and prepush-quick passed.
- Authorization matrix: 40 refusals and 8 allowances across both edges.

Mutations: swapped version action, redirected resource, ignored denial,
and bypassed Served authorization failed the permission or HTTP checks.
Earlier effective mutations covered body copying, premature owner drop,
range status, missing 304 headers, a 304 body, wrapped stream error causes
and duplicate accounting. No assertions were removed or weakened.

Linux RDMA paths were not compiled or exercised. GetObjectAttributes,
PutObject gateway registration and the default-stack switch are not part
of this increment.
2026-11-15 07:40:48 +08:00
overtrue b7e7a6156c feat(gateway): serve bucket notification configuration reads
Use native gateway DTOs for stored notification reads, register the handler,
and keep both edges on the shared bucket authorizer. The legacy edge now
uses the compatibility seam. Notification writes and the default stack are
unchanged. Non-test changes: +53/-36 lines.

Add three refusal-oriented tests and two configuration-read tests, including
successful IAM controls. Cover absent and stored configurations, reserved
bucket shapes, unknown stored XML, event/filter values, bucket-bound IAM
allow/implicit deny/explicit deny, and one handled-operation count.

Mutation controls on da5414c: bypassing each edge's authorization made its
notification_authorization_binds_the_bucket_on_the_*_stack test fail because
the wrong bucket was served. Dropping the stored queue made both
*_stack_answers_the_stored_notification_configuration tests fail. Restore
hashes were checked and sources were touched before a real rebuild.

Validation: 279 scoped tests passed on da5414c; 70 scoped tests passed after
rebasing to fd8dc41. Both rustfs all-target Clippy builds passed on fd8dc41,
as did fmt, test wiring and prepush-quick. Script tests passed on da5414c.
The earlier broad run reported one existing CORS test as leaky with exit 0.
CI and the full workspace suite were not run.
2026-11-15 06:08:43 +08:00
overtrue df4c4916c2 refactor(storage): own the PutObject write core
Use RequestEnvelope, a native upload body, raw write fields and owned results in the PutObject core. Legacy and RDMA ingress project once; internal writes share the core. Preserve archive batching, shared SSE and object-lock validation, completion ownership, audit events and response facts. Keep the merged RDMA length, cache lifetime and response-header behavior.

No gateway registration or UploadPart migration is included. Source outside test files/modules: +820/-383 (including conditional test imports).

Validation on fd8dc41f0c: fmt; default and gateway-stack all-target Clippy; 903 scoped tests with RUST_MIN_STACK=4194304; script-tests; test wiring; prepush quick checks. The archive authorization matrix has six refusal cases and three allowed cases, with persisted object/metadata checks. A fixed a279 legacy TCP baseline has 48 matching response records; the two retained socket tests cover PUT and POST behavior. The recorder observes drained counter deltas alongside the official cumulative 0/1/2/3 operation totals.

Mutation evidence: 44 distinct faults, grouped below by the test that rejected them. The counter and archive authorization additions each passed an unflagged control before their five mutations. The other fault controls and failures were retained from the same unchanged implementation.

- app::object::native_body::tests::legacy_body_preserves_demand_pending_and_length_validation_location: body-eager, body-length
- app::object::native_body::tests::legacy_producer_error_keeps_its_concrete_type_and_is_not_eof: producer-source
- app::object::native_body::tests::internal_producer_error_preserves_io_kind_and_nested_source_after_data: producer-source
- app::object::native_body::tests::native_trailers_are_published_only_at_eof: trailer-pending, trailer-value
- app::object::native_body::tests::native_framing_and_classified_errors_are_not_unwrapped: native-byte-count, native-class, native-kind
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_defers_invalid_raw_values_to_the_existing_validators: input-bucket, input-key, input-storage_class, input-server_side_encryption, input-object_lock_mode, input-object_lock_legal_hold_status
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_keeps_each_checksum_and_unknown_length: input-checksum_crc32, input-checksum_crc32c, input-checksum_sha1, input-checksum_sha256, input-checksum_crc64nvme, input-length, input-trailer
- storage::ecfs::legacy_edge::object_write::native_put_tests::owned_response_projection_preserves_status_headers_and_all_checksums: output-e_tag, output-checksum_crc32, output-checksum_crc32c, output-checksum_sha1, output-checksum_sha256, output-checksum_crc64nvme, output-status, output-headers
- app::object::put::tests::put_read_quorum_failures_are_normalized_at_each_stage: error-status, error-message, error-source
- app::object::put::tests::put_commit_lock_failure_keeps_request_operation_in_spawned_owner: error-headers
- app::object::put::tests::cancelled_put_request_completes_post_commit_publication: completion
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_mode_without_retain_until_date: lock-pairing
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_retain_until_date_without_mode: lock-pairing
- storage::ecfs_test::tests::test_parse_object_lock_retention: lock-presence
- app::object::extract::tests::snowball_member_mtime_requires_member_replication_authorization: archive-replication
- table_catalog::tests::put_object_reserved_prefix_guard_rejects_broken_native_delegation: guard-reason
- object_write::legacy_owned_put_preserves_refusal_order_checksums_and_short_body: refusal-priority, output-headers, output-checksum_crc32
- app::object::put::tests::put_edge_counts_polled_requests_once_before_usecase_refusal: count-missing, count-double
- app::object::extract::tests::snowball_owned_member_privileges_gate_storage: archive-tagging, archive-retention, archive-legal_hold

Existing expected values are unchanged.

Not verified: CI, Linux RDMA execution, performance, and gateway native-frame trailer consumption through this business core. The native trailer adapter itself is covered; gateway integration is a separate increment. The local suite uses a 4 MiB stack and makes no claim about default-stack behavior.
2026-11-15 01:20:53 +08:00
Hauser Zhi fd8dc41f0c feat(ecstore): add file-sync durability tier (#9422)
* refactor(ecstore): split durability sync capabilities

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): classify durability barriers

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(ecstore): add file-sync durability tier

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): harden durability docker validation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): isolate durability PUT benchmarks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): expose file sync grouping controls

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 00:17:34 +08:00
Chris An 645bb63809 feat: S3-over-RDMA with NIXL and cuObject transports (#9394)
* feat: S3-over-RDMA with NIXL and cuObject transports

A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL.

Applied onto main from #9305 (head 0f3c19a0f); the stacked #9302-#9304
commits are already on main as eca95f297. Conflicts were resolved where
this change met newer main: crates/ecstore/src/api/mod.rs (kept the
internode-transport exports, added the nixl module),
crates/rio/src/http_reader.rs (internode_http_request uses the pooled
get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs
(write_authenticated_put_file stays a plain fn returning a future, now
pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart
branch moved into multipart_usecase/upload_part.rs after the
per-operation split), Cargo.lock (main's lock plus only this change's
new packages).

* fix(rdma): build the RDMA crates without vendor SDKs and pass guards

Maintainer changes on top of the contributed S3-over-RDMA work:

- rustfs-cuobject: cuobject is an optional, default-features = false
  dependency behind a new `server` feature (enabled by the rustfs
  `cuobject` feature); the crate root is #![cfg(feature = "server")], so
  without the feature it exports nothing and a workspace build needs no
  NVIDIA cuObject SDK.
- The NIXL agent handle (OnceLock, install_nixl_server, nixl_server)
  moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app
  init installs it. The layer-dependency guard rejects infra -> app.
- New code reaches ECStore through a cfg-gated `ecstore_rdma` facade
  module and s3s types through the existing re-export hubs
  (architecture-migration and s3s-footprint guards).
- SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window
  the unsafe-allowance guard checks.
- transport_config_known_backends_are_current_oss_values loops over
  KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name
  itself.
- New non-required workflow rdma-feature-check.yml runs
  `cargo check --locked -p rustfs --features nixl` on Linux (paths
  filter, weekly, manual), registered in scheduled-validations.json.

* fix(ecstore): keep NIXL internode error messages constant

The three io::Error::other(format!(..)) sites in nixl_internode.rs
(read refused, write refused, write byte-count mismatch) now return
constant messages; the status, response body length and byte counts go
to a debug! line in the same place, the level this file already uses for
a failed NIXL peer call. No response body is logged, as on the HTTP
internode path.

The error-format ratchet exists because per-disk formatted messages make
otherwise identical errors compare unequal and split the quorum
reduction; constant messages keep them in one bucket. A typed transport
error would carry the detail without that problem; this takes the
mechanical route.

* fix(ecstore): flush staged NIXL writes before waiting for buffers

* fix(rdma): build rustfs-nixl only on Linux

nixl-sys's stub build links -lstdc++, which macOS does not ship, so
`cargo test -p rustfs-nixl` (and a workspace test run) failed to link on
macOS. rustfs-nixl's dependencies are now Linux target dependencies and
its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl`
feature code that was not already Linux-gated (the storage/rpc nixl_read
module, open_shard_contained, the internode route arms and the facade
module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux
builds are unchanged: the workspace tests still build and run the
crate's tests through the stub.

* ci(rdma): alert on scheduled failures of the RDMA feature check

The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes.

* fix(rdma): put rustfs-nixl behind a sys feature

nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device.

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Rain Jiang <rain-jiang@outlook.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 00:10:10 +08:00
Hauser Zhi 5f9e5d62df feat(obs): measure remote rename field bytes (#9417)
* feat(obs): split PUT commit and remote rename timing

Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* bench(filemeta): measure inline rename RPC encoding

Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): use exact inline rename shard fixture

Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): measure remote rename field bytes

Record detail-gated JSON and msgpack field lengths after successful remote
rename request encoding so msgpack-only rehearsals can prove sender behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): preserve partial carry drain window

Use the production drain window for real partial-bucket checkpoints so CI storage latency cannot truncate source-cache persistence.

Align the certified partial-cycle assertions with carried-value semantics while retaining mixed-coverage and final convergence checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 00:00:09 +08:00
overtrue da5414c345 fix(protocols): preserve HeadBucket storage root compatibility
Resolve empty-bucket requests through the existing storage lookup at the
legacy edge. Keep authorization, operation counting and nonempty bucket
conversion unchanged. No gateway bucket-name contract changes.

Add two tests covering two positive and two negative cases: protocol root,
ordinary bucket, missing bucket and uninitialized storage. Check response
status/headers and one count per request. Disabling the compatibility
branch makes both tests fail; restoring it passes both after recompilation.

Verified on 4182d55d1: focused tests, gateway Clippy, and a rebuilt binary
with libunftp 0.25 using FTP_TLS: login 230, root and ordinary CWD 250,
missing bucket 550. Default Clippy and script-tests passed on 925e5d0c0.
After integrating a279b4bb6, gateway Clippy and prepush-quick passed.
The wire probe and focused tests were not rerun after that integration.
2026-11-14 20:57:53 +08:00
overtrue a279b4bb63 feat(gateway): serve owned multipart completion
Use shared request authorization and native completion responses across both S3 stacks. Keep storage responsible for final commit and replay decisions.
2026-11-14 20:42:23 +08:00
Hauser Zhi 4182d55d11 chore(deps): upgrade JWT and FTP dependencies (#9419)
Update jsonwebtoken to 12.0.0 and libunftp to 0.25.0, including the JWT key construction API changes required by the new release. Refresh the lockfile through two requested cargo update and cargo upgrade rounds.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 09:11:54 +00:00
Hauser Zhi d4af2377fe perf(grpc): prewarm isolated bulk channels (#9418)
perf(grpc): prewarm isolated bulk channel pools

Extend the existing opt-in peer prewarm to the configured bulk channel pool. Control prewarm remains one channel, isolated bulk prewarm fills the per-peer pool, and legacy shared mode reuses the control cache. Add a cache-reuse regression for the legacy mode.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 09:02:49 +00:00
Ramakrishna Chilaka f5e8f17461 perf(ecstore): coalesce shard, integrity index, and commit syncs (#9407)
* perf(ecstore): coalesce shard, integrity index, and commit syncs

- Enable sync-on-close (WriteMode::TruncateSync with sync_parent_dir: false)
  in LocalDisk::create_file for multipart staging writes so part.N and
  part.N.integrity.<gen> run fdatasync before POSIX_FADV_DONTNEED page-cache
  reclaim and register in PRESYNCED_TMP_FILES without polluting
  PRESYNCED_TMP_DATA_DIRS.
- Keep LazyLocalFileWriter open across write_bitrot_shard when shard integrity
  is enabled and coalesce the .idx sidecar creation, write, and fdatasync into
  poll_shutdown via set_pending_integrity_index + MultiWriter::finish_and_shutdown,
  skipping duplicate PreparedIntegrity::write file creations for prewritten disks.
- In fused_attempt (commit.rs), sync prepared_metadata_source (xl.meta) before
  os::fsync_dir_std(src_data_path) when src_data_path is presynced so the
  active journal transaction commits both xl.meta data extents and the staged
  data directory entry in one flush instead of two.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

* perf(ecstore): skip redundant .idx parent creation and header re-read when presynced

- Mark parent_exists = true in StdBackend::open_write for temporary .integrity.
  and .proof. sidecars so open_lazy_local_file_std opens the file directly in
  its existing staging directory before falling back to create_dir_all_std.
- Skip reopening and re-reading the 32-byte .idx header in
  LocalDisk::prepare_part_transaction when take_presynced_tmp_file confirms the
  sidecar was just written and synced by LazyLocalFileWriter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

* fix(ecstore): validate presynced multipart integrity indexes

Validate the proof file type, length, and header before preparing a multipart transaction. Use a presync receipt to skip the redundant file sync after validation.

Cover valid, truncated, and mismatched-header indexes carrying presync receipts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): wait for tail rename drain in inline_put_direct_commit_handles_post_encode_rename_failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-14 18:03:56 +08:00
overtrue 925e5d0c0c feat(gateway): serve HeadObject through native response facts
Share HEAD authorization and its use case across both request edges. Keep
stored and proxy ETag parsing distinct, preserve conditional headers and
version bytes, and finish auditing before terminal header encoding. Pin
gateway to 1afff6a for the measured 304 response profile.

Verified 785 affected and embedded tests, then 12 tests after test-only
module/import cleanup. Both all-target Clippy configurations, script tests,
wiring and prepush checks passed. Tests cover malformed stored tags,
separate replication denial, conditional status conflicts and ordinary
success controls; the proxy oracle checks 779 spellings.

Observed 39 effective isolated faults: authorization/delegation, execution,
metadata exposure, stored/legacy/proxy tags, raw header escaping/repetition,
framing conflicts, length/counting, range boundaries, checksum and storage
class, ordinary completion, terminal code/status/default resolution, and
each of the seven 304 fields. An ineffective fallback mutation was retained
as a failed experiment and replaced with an explicit-status-only fault.

The 785-test base was c13232996; final module tests and Clippy ran on
fc22c11d1. No GetObject migration added. Cross-binary pinned acceptance,
remote proxy/ODM integration, SSE-C real-storage wire and performance were
not verified. Existing assertions were not weakened.
2026-11-14 17:54:44 +08:00
Hauser Zhi 247aecf7b8 fix(health): bound quorum probes and readiness observations (#9410)
* fix(health): bound quorum probes and record timeout observations

Collect fresh local and remote observations concurrently and finish once
each configured set has sufficient storage and lock quorum evidence.
Validate configured peer identities and disk slots before counting votes.

Apply the cluster deadline to cache and singleflight waiting, retain flight
ownership through timeout publication, and record timeout observations once.
Cover slow peers, unavailable quorum, identity validation, cancellation,
cache ownership and compatibility with focused regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(health): coalesce probes and count quorum incrementally

Share the completion generation with overlapping uncached requests while
retaining fresh observations for later requests and bounded flight ownership.
Build validated topology thresholds once and count unique disk and lock
votes incrementally instead of rescanning collected inventories.

Honor the existing inventory metrics flag through RPC and storage owners.
Health probes omit operation metric snapshots while retaining fresh disk
identity, capacity and runtime checks. Move owned inventories across local
aggregation layers and preserve the full-metrics administrative APIs.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(health): cover failed uncached flight recovery

Cover a later waiter sharing its overlapping failed round before an independent request starts a fresh successful observation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(health): group lock membership in one endpoint pass

Group configured lock members by set in one pass and reject impossible set counts before allocation. Retain per-set thresholds and add a bounded malformed-topology regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(health): publish startup topology in gating fixture

Publish the shared four-disk fixture's endpoint topology before initializing disks, matching storage startup. Cluster health probes can then validate fresh quorum observations while the readiness test exercises S3 admission publication.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 06:24:21 +00:00
Hauser Zhi fc22c11d1a feat(obs): split PUT commit and remote rename timing (#9416)
* feat(obs): split PUT commit and remote rename timing

Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* bench(filemeta): measure inline rename RPC encoding

Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): use exact inline rename shard fixture

Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 16:19:00 +08:00
overtrue 0eec25e4c0 fix(ci): compile gateway features in nightly matrix legs
Select gateway-stack for replication and protocol gateway legs and use matching build and run features. Legacy feature selections remain unchanged.

Verified on 0460ad63a: 20 e2e binary tests, full make script-tests, actionlint with shellcheck, CI workflow check, and prepush quick checks. The workflow contract exercises four passing build/run selections, four propagated failures and four mismatched-feature rejections. Removing either gateway feature, mismatching either run selection, and emptying the matrix each made the contract test fail.

No Rust compilation or live nightly execution was performed; actual gateway startup and suite outcomes remain to be observed in CI.
2026-11-14 15:28:03 +08:00
cxymds c132329961 fix(scanner): preserve usage convergence across bounded cycles (#9397)
* feat(usage): record bucket provenance and complete observations

* fix(scanner): preserve usage progress across bounded scan cycles

* docs(scanner): explain usage provenance and bounded convergence

* chore(scanner): correct requeuing spelling in runtime guidance

* fix(scanner): route usage contracts through domain facades

* fix(usage): keep admin projection futures sendable

* fix(scanner): expose endpoint layout in production facade

* fix(scanner): preserve checkpoint ownership during budget drain

* test(scanner): remove redundant fixture clones

* test(ecstore): update Windows rename publication calls

* fix(ci): restore scanner test exports and lint assertions

* test(scanner): preserve production drain for real checkpoints

* fix(scanner): retire obsolete work without clearing responsibility

* test(table-catalog): follow DeleteObject guard delegation

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-14 15:07:51 +08:00
cxymds 0f503fc613 feat(admin): expose caller-bound effective permissions (#9414)
* feat(admin): add caller-bound effective permission queries

Expose bounded advisory checks through the existing IAM, bucket-policy, table, and KMS authorization paths. Preserve three-state decisions for incomplete request facts and bypass publication guards only for internal probes while real operations continue to authorize independently.

* fix(auth): preserve effective permission probe boundaries

Use canonical action scopes and verified signing facts, keep unavailable conditions unknown across table metadata gates, and resolve table resources without catalog repair. Route admin test dependencies through existing facades and keep legacy S3 error usage within its migration baseline.
2026-11-14 15:07:24 +08:00
overtrue 0460ad63a0 feat(ecstore): add read-only multipart completion bindings
Observe active, completed and retained multipart ownership without replay, recovery, acknowledgement or reclamation. Only active uploads supply preflight metadata. Completed and retained records require validated physical evidence; completion still owns publication.

Preserve native inspection errors before reporting absence. Completed quorum retains precedence, and UnsupportedDisk keeps its compatibility behavior.

Validation: 17 binding tests and all 7,453 ecstore tests passed with four workers and no skips. Both default and internode-transport,test-util all-target Clippy passed with warnings denied. Formatting, script tests, test wiring and prepush checks passed.

Combined mutations of inspection error propagation, read quorum selection and UnsupportedDisk handling produced four intended runtime failures while two controls passed. Sources were restored by hash and actually recompiled before the full final run.

Application and CompleteMultipartUpload integration are not included.
2026-11-14 14:01:10 +08:00
Nonot 28da2e12a8 feat(initContainer): add resource configuration for init containers (#9400)
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-14 13:01:38 +08:00
overtrue 41225da343 fix(ci): provision runtime tools for both full nightly stacks
Reuse the full CI job's pinned mc and Vault provisioning, version checks,
and iptables capability warning before both e2e-full nightly stack runs.
Keep existing test filters, profile selection and report-only policy.

Validation on 4053e694c: actionlint with shellcheck, YAML step equivalence,
test wiring, CI workflow contract, prepush-quick and full script-tests.
The first script-tests attempt hit an unchanged scanner fixture JSON read;
the fixture control and final full rerun passed without source changes.
This does not establish the cause or repair of that fixture failure.

Executed the fixed Linux mc and Vault downloads, SHA-256 checks and version
commands in a private prefix. No test assertions were added or changed
(negative/positive 0/0); no mutations or Rust builds were needed.
The complete nightly workflow and E2E cases were not run for this change.
2026-11-14 12:50:34 +08:00
overtrue 4053e694cb feat(storage): carry native values through shared SSE
Keep SSE configuration, requests, materials and read response fields on
native strings. Project legacy enums at the DTO boundaries and retain the
existing validation order, bucket defaults and KMS authorization behavior.
Carrier diagnostics omit customer keys and encryption key material.

Production lines: +186/-166. Add 5 negative diagnostic tests, 0 positives.
Do not implement or register an owned PutObject operation.

Validation: fmt, default and gateway-stack all-target Clippy, 777 affected
unit tests, script-tests with aligned Python, test wiring and prepush-quick.
Tests ran on 9ea8e8d37f9670f8da3dc5cb6ed5fe6e6e9bb592; rebased onto
a9dd5f0fd1 with no overlapping files or pin change.

Mutation witnesses (8 sites, 9 failing cases): restore derived Debug for
SseTypeV2, EncryptionRequest, DecryptionRequest, EncryptionMaterial and
DecryptionMaterial -> all five native_*debug_withholds* tests fail; change
unknown default mapping -> shared and copy unknown-default tests fail;
admit an unsupported algorithm -> rejects_invalid_sse_algorithm fails;
project AES reads as KMS -> classification_withholds_the_wrapping_key_for_sse_s3_reads fails.
Restore exact source hashes, touch the SSE source and rebuild: 777/777 pass.

Not verified: full workspace, optional rio-v2/FIPS builds, or CI for this commit.
No intended legacy protocol change; diagnostics intentionally omit key fields.
2026-11-14 12:25:38 +08:00
overtrue a9dd5f0fd1 test(cache): observe entry destruction before claim release
Moka maintenance can logically evict an entry while epoch reclamation still
owns its allocation. Check a cache miss, tear down the cache, and observe
entry destruction through Weak before asserting response ownership and the
final reservation release. Keep the original five-byte and zero-byte checks;
only their physical-owner precondition changes. No runtime behavior changes.

Validated on 4bb272ed6 with default cache/runtime-memory features on macOS:
140 package tests passed, all-target Clippy, formatting, test wiring, and
prepush quick checks. Linux has one extra cgroup-only test, not run locally.

Mutation checks: copying response bytes breaks the live claim assertion;
skipping reservation release breaks the final zero assertion; retaining an
entry breaks the Weak assertion; skipping cache removal breaks the miss
assertion; detaching the response after eviction breaks the post-teardown
five-byte assertion. All failed through their intended runtime assertions.
No tests added or removed. No cache policy or reclamation timing changes.
2026-11-14 11:58:09 +08:00
overtrue 4bb272ed64 fix(gateway): honor the SSE-C transport setting
Reuse the outer transport layer's deployment setting in the gateway builder. The existing false default allows customer keys over plaintext with the gateway's explicit acknowledgement; true keeps both key sides strict. Transport facts remain per request.

The shared signer fixture now accepts headers before signing. The new test exercises 45 scenarios (38 refusals, 7 test-handler successes), checks signed SSE headers and raw-key handoff, and runs the inner plaintext gate separately from the outer layer.

Validation on ebe31dfdb5: fmt, all-target Clippy with and without gateway-stack, test wiring, and 63 gateway-stack/handler/transport tests passed. Production diff: +29/-14 lines.

Fault controls fail ssec_deployment_policy_and_key_handoff: forced strict/allow, false secure transport, re-signed key/input rewrites, skipped handler, response-key echo and wrong refusal code. Separate fixture checks reject anonymous allowance and unsigned SSE headers. The original strict selector fails a valid signed plaintext request with InvalidRequest after a TLS handler control succeeds.

Not verified: copy-source key wire handoff, storage use cases, real Served IAM authorization, network TLS termination, cutover acceptance or CI. No production operation was added.
2026-11-14 10:46:32 +08:00
overtrue 9ea8e8d37f fix(deps): align bindgen and prettyplease on syn 3
Restore only bindgen 0.73.2's syn edge to the already locked syn 3.0.10.
Its >=2,<4 range permits this version and prettyplease 0.3 requires syn 3.
No package version, checksum, manifest, feature or formatter changes.

Regression evidence: the frozen lock failed the FIPS tls-runtime test
build with E0277 in bindgen's prettyplease::unparse call. The corrected
locked graph builds and runs provider_policy_isolated in both modes.
No new tests or assertions; no behavior mutation was needed for this
compile failure. Product source lines changed: 0; lock lines: +1/-1.

Verified: tls-runtime all-targets Clippy and provider_policy_isolated
(1 passed in each mode), default on ebd4bb2ba and FIPS on f2a43376d.
The final rebase only adds two test-guard fixes. cargo tree --locked,
fmt, diff whitespace and prepush-quick passed on the final base.

Not verified: full RustFS/workspace suites, the remaining nightly FIPS
consumer matrix, or a new Linux CI run. No runtime TLS policy changed.
2026-11-14 10:35:45 +08:00
overtrue f2a43376de test: adapt owned entrypoint guard fixtures
Follow DeleteObject's typed and legacy entrypoints through the shared fields adapter into its owned implementation. Preserve the existing mutation guards and key/prefix/store ordering checks.

Let the TRACE mutation probe locate either ListObjectsV2 parameter shape exactly once, then require the real logging guard to reject DEBUG.

Validation on ebd4bb2ba8: default and gateway-stack all-target Clippy, three focused nextest tests, full make script-tests, split probe (61 checks), test wiring, shell syntax, and prepush-quick passed. Product lines: +0/-0.

Source-guard evidence: one valid source control and 11 negative fixtures; missing methods, bypassed delegation, missing guards, and reversed ordering fail with the intended diagnostic. The old checker fails on current source. Disabling each typed, legacy, or fields delegation check makes the fixture test fail. TRACE marker evidence: two accepted shapes and five rejected missing/debug/duplicate shapes; TRACE-to-DEBUG mutation is rejected by the real logging guard.

No runtime code or logging policy changes. Full workspace and Linux CI were not run locally; source-guard mutations do not prove runtime authorization.
2026-11-14 10:12:26 +08:00
overtrue ebd4bb2ba8 feat(gateway): serve object and multipart-upload listings
Move ListObjects, ListObjectsV2, ListObjectVersions and ListMultipartUploads
to owned use cases and gateway handlers. Keep legacy encoding, stored ETags,
optional facts and cursor boundaries at their existing edges. ListParts and
multipart create/complete/abort are outside this increment.

Non-test changed lines: +973/-659, excluding test modules and test-only
imports/reexports; comments and whitespace are included.

Validation: 282 focused tests passed on ed3a9e670d,
including all storage::access::tests and both stack refusal matrices. The four
listing matrix rows exercise 32 refusals and 8 allowed requests across both
stacks; these are migrated coverage, not 40 new tests. Stored-key, entity-tag,
encoding and pagination tables provide additional compatibility coverage.
The 1024-byte stored-key positive runs on Linux; macOS attempts the write and
asserts FileNameTooLong and absence because of its full-path limit.

Final RustFS all-target Clippy passed with and without gateway-stack on ed3a9e670.
The affected e2e_test package Clippy passed on e3858df4fa.
After rebasing over the adjacent response-helper change, one gateway-stack
all-target cargo check passed on ebe31dfdb5.
Formatting, test wiring, seam validation and prepush-quick passed.

Ten effective local mutations were reviewed on d5f640243952b571111bb3c97a98850f04a92172:
- Count: key_count_includes_objects_and_prefixes_and_a_keyless_entry_is_not_an_object.
- Marker: list_objects_next_marker_follows_the_cursor_then_the_last_raw_entry.
- Key floor: every_key_the_store_can_hold_is_represented_unchanged.
- Nil version: versions_delete_markers_and_markers_are_answered_raw.
- Upload fields: list_multipart_uploads_output_maps_uploads_and_prefixes_raw.
- XML escaping: listing_entries_are_encoded_per_segment_only_for_url_and_tags_are_escaped
  and the_legacy_stack_lists_every_stored_entity_tag.
- Three direct authorization mutations (ListObjectsV2, ListObjects,
  ListObjectVersions): the_refusal_matrix_holds_on_the_gateway_stack.
- Shared multipart authorization: both stack refusal matrices; this also covers
  ListMultipartUploads delegation. Its separate direct early-return experiment
  failed for a root request and is excluded from authorization evidence.
All temporary experiment code was removed before the final product build.

Not verified: full dual-stack differential run, full workspace CI, or Linux
runtime execution locally. Bootstrapping operations for the differential suite
are not all migrated yet. Historical wire baselines cover a narrower surface.
2026-11-14 09:23:04 +08:00
overtrue ebe31dfdb5 feat(app): carry answered status and headers across both edges
Add Answered<Output> for use cases that return a status override or extra
response headers. The legacy seam preserves those fields on S3Response;
the gateway helper passes them to Resp through its supported API. Bare
outputs share the seam implementation and retain their default response.
No operation consumes the carrier yet. Non-test changes: +87/-4 lines.

Validated on ed3a9e670d:
- cargo clippy --locked -p rustfs --all-targets -- -D warnings, both
  default and --features gateway-stack.
- cargo nextest run --locked -p rustfs --features gateway-stack --lib
  --no-tests fail -E 'test(storage::ecfs::legacy_edge::tests) |
  test(server::gateway_handlers::tests)': 23 passed.
- cargo fmt --all --check, prepush-quick, test wiring, CI workflow
  contract and validation registry passed.

Three added tests cover carrier values and defaults (0 refusal / 3
positive tests). Representative mutations discarded status and headers
on both edges: an_answered_status_and_headers_reach_the_legacy_response_unchanged
failed its status assertion; the corresponding gateway answer test
failed its status/header tuple. Both tests ran and failed. Restore hashes
matched and sources were touched before the passing ordinary rebuild.

Not verified: wire header delivery or encoder-owned/header-collision
refusals; these belong with the first HeadObject/GetObject consumers.
The legacy header assertion was not independently mutated. No new
operation, authorization path, wire claim, or legacy operation behavior
change is included. Full workspace tests and CI were not run locally.

Refs: rustfs/backlog#2767, rustfs/backlog#2734
2026-11-14 09:07:52 +08:00
overtrue ed3a9e670d feat(gateway): serve DeleteObject on both stacks
Move DeleteObject to an owned use case and share authorization between the legacy and gateway edges. Keep the legacy request fields at the entry so validation and audit run in order. Give the refusal matrix its own bucket.

Tests: 168 passed on e97dc74d9; the restored mutation control passed 9 tests before an equivalent test-only Clippy fix. RustFS Clippy passed in both configurations; e2e_test Clippy and test wiring passed. The recursive authorization table covers 24 refusals and 16 allows. The header and caller tables add three refusals and five allows across both request forms; the wire cases add key, version and caller refusals.

Mutations: bypassing the primary, force, authenticated-caller, or governance decisions turned their refusal cases into allows. Skipping descendant delete, replication, or bypass decisions turned the respective refusal rows into allows. Removing context restoration failed both allowed and refused rows. Disabling the helper removed the invalid-key audit. The gateway root matrix stayed green for each mutation except the primary decision, where its root row passed and the unauthorized rows were served.

Not verified: the full dual-stack differential scenario requires PutObject and bucket versioning on the gateway stack. CI has not run for this candidate.
2026-11-14 07:55:37 +08:00
overtrue e3858df4fa feat(gateway): serve object metadata operations
Serve object ACL reads and writes, legal hold and retention reads, tagging reads and deletes, and torrent requests through shared use cases and authorization.

On base af93bf466, 131 object metadata and access tests, four refusal matrix tests, and five targeted behavior controls passed. Mutations covered each authorization, version actions, retention UTC conversion and range, ACL responses, tags, lock checks, missing objects, version parsing, and store initialization. Both package clippy configurations passed.

After rebasing onto e97dc74d9, the gateway-stack all-target check and all four refusal matrix tests passed. Full workspace tests and differential scenarios were not run.
2026-11-14 07:36:21 +08:00
overtrue 8e64a3aa2f feat(server): configure gateway bucket owner source
Use the owner ID RustFS reports for gateway expected-owner checks. A missing bucket is still resolved by its handler; no per-bucket owner lookup is added.

The gateway wire table covers six refusals and four successes across ordinary, missing, and reserved-shaped buckets and ListBuckets. The same table runs against the legacy stack. The source unit test covers a reported owner and an unavailable owner.

Mutations: removing the source makes the gateway table fail; returning the wrong owner makes the gateway table and owner unit test fail; returning the wrong owner only for the missing bucket changes its matching-owner response from 404 to 403; returning an owner when none exists makes the owner unit test fail.

Verified against base e97dc74d9: both RustFS clippy configurations, three focused tests, test wiring, and prepush quick checks. Successful SDK responses are checked as success, without asserting their HTTP status. Full workspace tests and CI were not run locally.
2026-11-14 07:33:47 +08:00
overtrue 092c8c272d test: accept owned bucket in metadata reload guard
Verified both bucket guards, default and gateway-stack Clippy, formatting, test wiring, and prepush checks. Removing the CORS notification and flipping its maintenance flag each failed the guard; the restored control passed.

Tests ran on e97dc74d9 before rebasing over unrelated KMS changes at 3c59e4f30. Full workspace tests were not run locally.
2026-11-14 07:32:45 +08:00
唐小鸭 3c59e4f306 fix(kms): tolerate Vault reads that lag the client's own writes (#9406)
* fix(kms): tolerate Vault reads that lag the client's own writes

A Vault or OpenBao node that answers reads from its own copy (an OpenBao standby, a Vault Enterprise performance standby) can serve the state from before a write for a short while after acknowledging it, and can refuse a write it is able to reject from that copy without forwarding it.

- Create a KV2 key in one create-only write that carries usage, description and tags, instead of writing the record and reading it back to attach them.
- Pause with jittered backoff before re-reading after a lost check-and-set write, and allow five attempts instead of three, in both backends.
- Remember for five seconds what this client last wrote or deleted at each record and re-read an answer that is older. The last answer is returned once the attempts are spent, so this only delays a read.
- Retry a Transit key delete that is refused because the deletion switch is not visible yet.

* docs(kms): describe running the Vault backends against OpenBao

* fix(kms): keep the remembered write version from moving backwards

Two writes of one record by the same client can have their acknowledgements processed out of order. Recording the lower version last made it the floor again, so a read served from that older version was accepted. The marker is now updated atomically and only raised within a record's lifetime; a write after this client's own delete starts the count again.
2026-11-14 06:59:29 +08:00
overtrue e97dc74d93 feat(gateway): serve GetBucketLocation on both stacks
Route the operation through the owned use case and shared authorization.
The tests cover two bucket names and a missing bucket on both stacks.

Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), and test wiring. Bypassing location authorization made the
gateway refusal matrix fail for anonymous and no-policy callers. The main
legacy comparison passed. Non-default regions were not verified.
2026-11-14 05:21:23 +08:00
overtrue e83b6815ed feat(gateway): serve DeleteBucket on both stacks
Share the use case and authorization across both edges. Preserve the
force-delete header rules and operation accounting.

Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), test wiring and main legacy comparison. The force-delete table
covers eight refusals and six successful cases; additional tests cover header
precedence and anonymous invalid input. Bypassing DeleteBucket authorization
made the gateway refusal matrix fail for anonymous and no-policy callers;
bypassing ForceDeleteBucket made the false-header authorization test fail.
Site replication and multi-node deletion were not verified.
2026-11-14 05:21:23 +08:00
overtrue 20ea63e27e feat(gateway): serve bucket CORS mutations
Route PutBucketCors and DeleteBucketCors through owned use cases and shared authorization, preserving audit, storage, and legacy edge behavior.

Coverage includes eight refusal and two root matrix scenarios per stack, a wrong-resource refusal, missing-context refusal, and CORS persistence cases. The scoped suite passed 125 tests on 05554bee0. Bypassing Put or Delete authorization failed the gateway refusal matrix; skipping each storage mutation failed the CORS persistence test; allowing a missing context failed the access unit test.

Not verified: full workspace CI and the missing-bucket plus invalid-configuration error ordering. The delete audit status remains 200 while the response is 204.
2026-11-14 04:00:05 +08:00
唐小鸭 a984f42b0b fix(replication): stop re-sending existing objects every scan cycle (#9412)
* fix(replication): stop re-sending existing objects every scan cycle

Existing objects on a site-replication source were re-queued and
re-sent on every scanner cycle, so the replica kept rewriting the same
versions and its reported bucket size grew while the object count stayed
flat.

Two defects combined:

- The per-target resync stamp was computed by the worker but never
  written back to the source object, so the resync decision kept
  returning "replicate" for every object older than the reset date. The
  status writeback now persists the stamp under both internal metadata
  prefixes, and a successful full-object send of an existing object is
  stamped as well.
- The target HEAD only reported the tag count, so the tag comparison
  always saw a difference for tagged objects and fell back to a full
  re-send. The replication HEAD now sends
  x-amz-tagging-directive: ACCESS and the server answers with the tag
  set under the same authorization as the tag count. Targets that do not
  return the tag set still compare as different, which costs one extra
  send instead of a loop.

Also drop replicate_all_stat_options, which built the directive but was
never passed to the HEAD call.

Refs #9354

* test(replication): use relative paths in the resync stamp test
2026-11-14 03:59:46 +08:00
overtrue f9f52f9fad fix(app): preserve stored entity tag spellings
Read stored entity tags with the existing legacy spelling rules before
constructing gateway values. Keep the original control-character coverage.
This prepares the shared helper without adding an operation caller.

Tests: four focused tests passed on 05554bee0, covering eleven spellings,
weak empty tags, control characters, and invalid header values. The table
covers seven malformed or boundary spellings and four ordinary spellings.

Mutations: weak, quotes, bare weak prefix, nested quotes and plain values
failed the spelling table; control characters failed the preservation and
header tests; empty weak tags failed the weak-tag test; removing the version
header failed the header test.

Not verified: operation-level wire responses and full workspace tests.
2026-11-14 03:50:33 +08:00
overtrue af93bf4663 feat(auth): share write-path authorization checks
Share replication header, write ETag, and staged multipart authorization across request forms, preserving the existing decisions and marker cleanup.

The parity table covers eight denials and six allowed cases. Mutations of each authorization call, replication marker and header source, ETag header source, and staged marker cleanup on success and failure all failed the parity test.

Tests and both Clippy configurations passed on 05554bee0. The gateway handlers are not included in this increment. The script test suite did not pass locally: four unchanged scanner fixture tests failed twice with expired fixture deadlines.
2026-11-14 03:43:04 +08:00
clangenb 4db694f1c4 docs(readme): add Docker Swarm install via the SwarmCLI rustfs chart (#9405)
* docs: add Docker Swarm with the SwarmCLI chart to the README quickstart

Option 7 links to the Docker Swarm guide on docs.rustfs.com and to the
community rustfs chart, noting that Eldara Tech maintains both.

* docs: point README Option 7 at the SwarmCLI chart guide's new path

The docs guide moved from Container to Cloud Native as SwarmCLI Chart.

* docs: name the SwarmCLI chart's key features in README Option 7

Secrets, versioned upgrades and rollbacks, Traefik HTTPS; and that its
Console is off and rustfsadmin refused, unlike Accessing RustFS below.

* docs: call the SwarmCLI chart open-source rather than community in README

It is maintained by Eldara Tech, which 'community' contradicted.

* docs: add the SwarmCLI chart's distributed mode to README Option 7

The chart guide now covers distributed, erasure-coded clusters as well
as a single-node service, so Option 7 no longer says single-node only.
2026-11-13 14:53:55 +00:00
Adrián Pérez 0e8fb73e57 perf(lifecycle): skip immediate-expiry lookup without lifecycle rules (#9404)
After every successful write, enqueue_immediate_expiry took the bucket
metadata-transaction read lock and read the bucket incarnation before
checking the cached bucket metadata that decides the outcome. A cached
entry without any lifecycle document can never enqueue an expiry.

Return early in that case. Buckets with lifecycle rules, unreadable
lifecycle documents, a pending creation commit or no cached entry keep
the authoritative path.

Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:52:54 +00:00
Hauser Zhi 05554bee00 chore(deps): clean up unused dependency declarations (#9411)
Move test helpers to dev-dependencies and clean unused build
requirements and workspace metadata.

Use profiling compatibility entrypoints for probe-free crates and
verify the application's hotpath, allocation, and CPU feature graphs.

Validate the affected libraries, FTPS/SFTP test targets, and 1729
existing library tests.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 02:04:31 +08:00
Hauser Zhi 8d5faa5750 fix(ecstore): persist restartable format bootstrap authority (#9359)
* fix(ecstore): persist restartable format bootstrap authority

Persist all-drive creator plans before format publication and resume exact deployment and drive slots after interrupted fresh formation. Consume authority durably before startup publication and validate regular control files on full and streamed reads.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): finalize format bootstrap once

Order bootstrap retirement before initialized identity publication, preserve established expansion preflight repair, and require followers to observe committed topology and complete identities.

Publish bootstrap controls through one authenticated atomic RPC on the dedicated fsync lane. Keep cancellation-safe cache ownership in the native operation and retain interoperable replacement healing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): fence anchored format publication retries

Select the durable first-slot plan before fanout and complete immutable anchored plans across retries. Retain physical-root namespace ownership through native publication, retirement and cache invalidation, rejecting conflicting or retired control records.

Reuse main's unary RPC lifecycle recorder for validated outcomes and exact request, byte and cancellation accounting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(test): align Windows and test-util compilation gates

Match physical publication-hook tests to their non-Windows provider. Compile pool-capacity overrides with their test-util consumers, and keep alias-list mutation inside the Unix branch.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): target native format publication paths

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* style(rpc): format merged request imports

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-11-14 01:04:03 +08:00
Hauser Zhi a9f308b075 test(ilm): stabilize manual transition accounting and pagination (#9409)
* test(ilm): account for completed immediate transitions

PUT and lifecycle configuration updates can enqueue transitions before a manual scan. Assert complete per-object accounting and verify a repeated scan skips the committed transition without duplicating the remote object or changing GET content.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): stabilize continuation eligibility across restart

Wait for configuration-triggered transitions to commit before testing opaque continuation tokens. Assert both pages skip exactly one completed object, never enqueue work and preserve the two remote objects across restart.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-14 01:02:11 +08:00
Adrián Pérez 2b0d74027a fix(ecstore): keep delete activation off the metadata read lane (#9402)
Strict recoverable-delete activation renamed the prepared transaction
directory on the disk's single-worker foreground lane, which also serves
metadata reads, and then queued its directory fsyncs as a separate
fsync-lane job. A stalled activation rename blocked unrelated metadata
reads on the same disk.

Run the rename, the fsync of the pinned object directory and, for a
missing original, the ancestor fsyncs in one fsync-lane closure. The set
and order of fsyncs are unchanged; relaxed and none modes keep the
previous rename path.

Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:51:38 +00:00
Hauser Zhi 964cb463a2 chore(deps): refresh Prost and WebPKI dependencies (#9392)
* chore(deps): refresh Prost and WebPKI dependencies

Run cargo update --verbose and cargo upgrade --verbose twice from main.
Upgrade the Prost family to 0.14.6 and rustls-webpki to 0.103.17.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): document shard buffer unsafe allowance

Document the arena block validity and exclusivity contract beside the
unsafe_code allowance to satisfy the existing guard.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: avoid typo false positive in PUT body chunks

Use hell and o as fixture chunks that concatenate to hello.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): upgrade rustfs-erasure-codec to 9.1.0

Run cargo update --verbose and cargo upgrade --verbose again.
Resolve rustfs-erasure-codec 9.1.0 and refresh the workspace lockfile.

Validate the integrated legacy GET producer stack fix with all 28
reported scenarios at a 2 MiB stack in both nextest profiles, plus
codec compatibility cases and the producer future-size regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): follow moved request denial audit calls

Inspect both legacy access and owned object authorization production
sources when checking request-level denial audit wiring.

Add regression coverage for a moved call, removal of that call, and
audit calls confined to test code.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 14:15:42 +00:00
cxymds ef6af77026 feat(heal): add bounded MRF management and streaming audits (#9399)
* feat(heal): add bounded MRF management and streaming audits

Publish lightweight responsibility summaries and expose authenticated, identity-bound cluster administration without reading object payloads. Stream audit-only reads while retaining staged verified bytes and full target readback for migration.

* test(admin): keep MRF coverage within gateway boundaries

Exercise owned MRF handlers through the shared admin test seam and keep wire-level authentication assertions in the router tests, without introducing another direct s3s consumer.

* test(e2e): complete MRF lifecycle lock fixture RPC

* docs: clarify integrity job resume refusals

Explain that a failed lock acquisition does not prove a live worker, and require job-status reconciliation before retrying an unknown request outcome.

* test(admin): include MRF cluster routes in route matrix

Keep the exhaustive admin route matrix aligned with the four registered cluster management endpoints, including the concrete node and responsibility ID probe.
2026-11-13 23:03:02 +08:00
Hauser Zhi 707694c097 fix(ecstore): make retired part settlement idempotent (#9393)
* fix(ecstore): make retired part settlement idempotent

Resolve a failed source admission against the canonical transaction while
retaining SourceRW ownership. Treat proven absence as already settled when
cached directory descriptors still refer to an upload retired into trash.

Cover a real pinned part transaction through certified Abort, existing closed
transactions and dangling aliases. Preserve active settlement barriers and
rooted no-follow path validation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound offline OPA fixture future stack usage

Construct offline storage futures in a separate noninlined frame before
boxing them. Keep the bucket-tag fixture within the normal test thread
stack while retaining real metadata writes and authorization assertions.

The affected debug test and the full STS/OPA module pass with a 2 MiB
thread stack after the original CI smoke stack overflow.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): isolate OPA contract on a dedicated stack

Run the bucket-tag OPA contract on a dedicated 32 MiB thread with a current-thread Tokio runtime. Nested ECStore initialization poll frames can overflow the default test stack even when the outer future is boxed.

Remove the superseded future-boxing helper. Validate all eight STS/OPA tests and targeted runs with 1 MiB and 512 KiB entry stacks on Linux ARM64, plus all-target Clippy and formatting checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 08:25:50 +00:00
Hauser Zhi f621f28d01 fix(lock): lengthen contended writer retries and report holders (#9390)
* fix(lock): extend contended writer retries within caller budget

Keep the initial probe, shared locks and batch waits at their existing peer limit. Allow later exclusive contention rounds to use a bounded longer wait so read holders outlasting one short round do not repeatedly erase writer progress. Preserve the caller deadline, quorum and grant cleanup, and cover the original failure with real LocalClient grants.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(admin): expose physical holder counts in top locks

Serialize the node-local physical holder count from the existing fast-lock snapshot for each resource/version/mode/owner cohort. Keep row totals and lease-metadata validation, and cover repeated owners, stale leases and the actual JSON field.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(admin): clarify node-local holder sampling

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 07:23:24 +00:00
overtrue d57e781375 feat(gateway): serve the policy, status and ACL reads on the gateway stack
GetBucketPolicy, GetBucketPolicyStatus and GetBucketAcl are served by the
gateway stack and by the legacy stack through one owned use case each. The
legacy access hooks call authorize_bucket_request, the function the gateway
edge calls; the owner's bypass of a bucket-policy Deny for GetBucketPolicy stays
in the shared authorization core. BucketAcl gets its use case in
bucket_usecase/acl.rs: the answer (one owner, full control, a grantee carrying
no identity) is unchanged.

Not built: GetBucketNotificationConfiguration, the four mutations and
PutBucketAcl (they wait for shared helpers).

Removed code: build_get_bucket_acl_output in storage/s3_api/acl.rs, replaced by
the use case.

Removed tests: test_build_get_bucket_acl_output_contains_full_control_grant_and_owner,
covered by the_gateway_stack_answers_the_bucket_acl and
the_legacy_stack_answers_the_bucket_acl (one owner, full control, grantee with
no id and no display name, NoSuchBucket for a missing bucket) in
tests/embedded_gateway_stack_test/bucket_access.rs.

Tests: refusal-matrix rows for the three reads on both stacks; NoSuchBucketPolicy,
the verbatim policy document, the owner reading a policy that denies everyone,
IsPublic following the stored policy, and the ACL answer, on both stacks;
use-case tests ported to the owned signatures.

Mutations: the gateway authorize answering Ok(()) in each of the three reads
turns that operation's anonymous and no-policy rows of the refusal matrix red.

Legacy answers compared with main: see the previous commit (40 rows, identical,
including the missing content-type of the stored-policy answer).
2026-11-13 20:32:56 +08:00
overtrue da07ad6a08 feat(gateway): serve GetBucketVersioning on the gateway stack
GetBucketVersioning is served by the gateway stack, and by the legacy stack
through the same owned use case (gateway input and output types). The legacy
access hook now calls authorize_bucket_request, the function the gateway edge
calls. The stored status is carried verbatim as the gateway status enumeration,
so a stored value is never dropped. As before, MFADelete is not answered.

Not built: PutBucketVersioning (it waits for the shared bucket-configuration
mutation wrapper and the owned audit call).

Tests: refusal-matrix row on both stacks (anonymous and a user without a policy
are refused, root is served and counted once); GetBucketVersioning answers every
stored bucket name the SDK reaches (a-s3alias, sthree-x, x--ol-s3), a stored
Enabled status, NoSuchBucket for a missing bucket, and the 400 InvalidBucketName
both stacks give xn--abc, on both stacks; the ported use-case test.
Differential cases: an unconfigured and a suspended bucket.

Mutations: the gateway authorize answering Ok(()) turns the anonymous and
no-policy rows of the refusal matrix red; dropping the status from the use case
turns the stored-status assertion red (left None, right Enabled).

Legacy answers compared with main (status, headers and body; the SDK output for
successes): 40 rows, for plain-bucket, sthree-x, a-s3alias, x--ol-s3 and a
missing bucket, each unconfigured and configured, for GetBucketVersioning and
the three reads of the next commit. All identical.
2026-11-13 20:32:56 +08:00
overtrue 17081a6d23 feat(server): serve the lifecycle, replication and lock reads
Batch A3 of rustfs/backlog#2765, second increment: the gateway stack serves
GetBucketLifecycleConfiguration, GetBucketReplication and
GetObjectLockConfiguration through the owned use cases from the previous
commit.

- `impl Served` for the three operations and their rows in the A3 family. Each
  `authorize` calls `authorize_bucket_request` with the action its legacy hook
  asks (GetBucketLifecycle, GetReplicationConfiguration,
  GetBucketObjectLockConfiguration) on the decoded bucket.
- The three rows of the refusal matrix. Each row stores its configuration
  through the bucket metadata first, so root is served without depending on the
  Put operations.
- The behaviour test from the previous commit also runs on the gateway stack,
  for an ordinary and a reserved-shaped bucket name.

Tests (negative / positive): refusal matrix 3 rows x 4 refused callers (12
negative, 3 positive; each is counted once under its legacy label);
gateway-stack behaviour test: NoSuchBucket 404 (3), each read's own 404 (6),
wrong-resource AccessDenied 403 (6), stored contents (6 positive).

Mutations: the three `authorize` bodies allow everything. Red:
the_refusal_matrix_holds_on_the_gateway_stack (each of the three reads serves
the anonymous and the no-policy caller), and
the_gateway_stack_answers_the_configuration_reads_as_stored (the wrong-resource
caller is served). The legacy-stack variants stay green.

Not built: the five mutations.

Not verified: the differential's gateway side (CreateBucket, PutObject and the
Put configuration operations are not all served yet; Linux reference only).
storage::access::tests::denial_exits_without_a_context_call_the_audit_function
fails on this base: the source guard looks for the force-delete exit in
access.rs, and that exit now lives in access_owned/object_misc.rs. It is not
this change.
2026-11-13 20:17:47 +08:00
overtrue d36c3595ad feat(gateway): serve CreateBucket on the gateway stack
CreateBucket is served end to end by the gateway stack; the legacy stack
serves it through the same owned use case.

- Use case: execute_create_bucket(envelope, gateway input) -> gateway
  output, on the fresh admitted task as before (the text guard names the
  new inner call). The audit entry and event come from the owned helper.
- Authorization: authorize_create_bucket in storage/access_owned/
  bucket_core.rs is the one composite hook both edges call: s3:CreateBucket
  with the body's LocationConstraint as a condition value (first decision
  without the presigned-query hoist), then PutBucketObjectLockConfiguration
  and PutBucketVersioning when object lock is requested.
- Legacy edge: record_s3_op, then through_gateway_seam.
- Gateway edge: impl Served and one ROWS entry.

Not built: no change to what CreateBucket answers; DeleteBucket and
GetBucketLocation follow in their own commits.

Tests (negatives 9, positives 8):
- embedded, both stacks: the_{gateway,legacy}_stack_creates_buckets_as_before
  (fresh, idempotent re-create, foreign and EU constraint refused, the four
  reserved name families created, xn-- refused, x--x-s3 refused by the SDK
  on the client); refusal-matrix row (anonymous, wrong secret, unknown key,
  no policy refused; root served, counted once).
- parity: owned_and_legacy_create_bucket_authorization_use_location_constraint
  gains six composite cases (location, object lock) comparing decision and
  handed-on bucket, legacy hook against owned function.
- use case: tests ported to the owned signature; empty LocationConstraint
  reads as none.
- differential: CreateBucket cases for a foreign constraint, EU and the
  reserved name families.

Mutation: Served::authorize returns Ok(()) ->
the_refusal_matrix_holds_on_the_gateway_stack red ("CreateBucket serves the
anonymous caller", "... the no policy caller").

Changed assertion: guards.rs names the inner call
execute_create_bucket_inner(envelope, input, mutation_permit) instead of
(req, mutation_permit); same check, new signature.

Metric (ledger candidate, no wire change): the legacy edge now counts every
CreateBucket call, including one its own validation refuses.

Not verified: the differential run against 1.0.5 (Linux CI).
2026-11-13 19:15:24 +08:00
overtrue db5a910b69 fix(s3-types): keep a typed cause across the s3s round trip
A typed cause now survives the s3s -> owned -> s3s round trip when the
code is unchanged. Before, an s3s error that crossed into
`rustfs_s3_types::S3Error` carried its whole self as a `LegacyCause`
wrapper, and crossing back set that wrapper as the cause, so a caller
downcasting the cause after an owned use case (a legacy edge's
diagnostics, a test asserting `StorageError::ErasureReadQuorum`) found a
wrapper. Crossing back now hands the original s3s error on and applies
the message, request id, status and headers on top; an error our side
re-coded keeps the wrapper, which shows the cause's text, as before.

Test (1 added, both directions):
`a_round_trip_hands_the_typed_cause_back_unless_the_code_changed` - an
untouched round trip is exactly the original (code, message, explicit
status, headers, typed cause); a changed message is applied with the
typed cause kept; a re-coded error keeps only the cause's text.

Mutation: always keeping the wrapper (the hand-back arm dropped) makes
the new test red.
2026-11-13 19:08:01 +08:00
overtrue 77ce76fde9 test(storage): read the force-delete denial exit where it now lives
The source-level guard that every denial exit without a context calls
audit_request_denial read only storage/access.rs. 8eb89dd54 moved
DeleteObject's decision, with its recursive-force-delete exit, into
access_owned/object_misc.rs and left the guard behind, so the guard went
red although the exit never stopped auditing. The guard now reads each
exit in the file that holds it.

Tests: the storage::access lib module, run whole: 113 passed.
Mutation: removing the audit_request_denial call from the exit in
access_owned/object_misc.rs turns the guard red with
"\"recursive_force_delete_requires_authenticated_caller\" exit does not
call audit_request_denial".
2026-11-13 18:59:43 +08:00
overtrue d645456a90 refactor(storage): carry body-sourced policy inputs in owned form
The policy inputs some hooks take from a request body are now owned
types that both request forms carry and install through
`AuthorizationRequest`, so the gateway edge no longer needs s3s DTOs:

- `RequestedObjectLockRetention { mode, retain_until }`.
- `RequestedObjectLegalHold(Option<String>)`, the status.
- `RequestedObjectTags(Vec<(String, String)>)`. It keeps only complete
  key/value pairs; those are the only tags a condition ever named.

Each type converts from its s3s DTO once, at the legacy insert site
(`From<&ObjectLockRetention>`, `From<&ObjectLockLegalHold>`,
`From<&Tagging>`). The conditions built from them are unchanged.

New trait installers: `install_requested_retention`,
`install_requested_retain_until`, `install_requested_legal_hold` and
`install_requested_object_tags`. The legacy hooks use them.

`set_requested_object_lock_retain_until` is now generic and takes an
owned `OffsetDateTime`.

`RequestEnvelope::fork_for_authorization` copies a request for one
more authorization, such as an archive member. Every field is copied,
the access phase's output included. The table publication guards stay
one shared set, so a lock retained through the copy is held for the
original's write. Anything else installed on the copy does not show on
the original.

Tests:
- A new parity test installs body inputs on both forms. The installed
  bodies, not conflicting headers, must describe the request, and an
  incomplete tag must name nothing.
- A fork test checks that the copy reads like the original and that
  what is installed on the copy stays off the original.
- The object-shape column of the decision table now runs on a forked
  envelope.
2026-11-13 18:09:11 +08:00
overtrue fa13764bb7 feat(bucket): answer lifecycle, replication and lock reads as owned
Batch A3 of rustfs/backlog#2765, first increment: GetBucketLifecycleConfiguration,
GetBucketReplication and GetObjectLockConfiguration are owned use cases
(gateway DTO in, gateway DTO out). The legacy stack serves them through
`through_gateway_seam`. Nothing is registered on the gateway stack yet; that
comes in the next commit, once the gateway error mapping answers a missing
bucket 404.

What changes:
- `app::dto_values::stored_config(&T: BucketConfigXml, parse)` is the one way a
  stored bucket configuration becomes a gateway DTO: its persisted bytes, read
  by the gateway's DTO bridge (`parse_*_dto`). A refusal is InternalError.
- `execute_get_bucket_lifecycle_configuration` and `execute_get_bucket_replication`
  take and answer gateway DTOs. `execute_get_object_lock_configuration` is new
  in `bucket_usecase/object_lock.rs`, moved from the inline legacy edge body.
  Codes and messages are unchanged.
- The legacy edges of the three reads count the operation and go through the
  seam. The input members the seam converts are `bucket` and
  `expected_bucket_owner` for each.
- The three legacy access hooks call `authorize_bucket_request` with their
  legacy action (GetBucketLifecycle, GetReplicationConfiguration,
  GetBucketObjectLockConfiguration).
- `storage/s3_compat/{lifecycle,replication}.rs`: the reverse (stored -> s3s)
  direction is no longer on the request path and is compiled only under
  `cfg(test)`, as the oracle of the parity tests below and of replication's
  existing round-trip tests. It goes, with retirement entries, when the Put
  edges cross the seam.

Tests (negative / positive):
- Legacy answer parity against main's own conversion: lifecycle (XML bytes,
  with DelMarkerExpiration, ExpiredObjectAllVersions, Transition,
  NoncurrentVersionTransition, an And/Tag filter and dates, one with a +08:00
  offset), replication (XML bytes, with DeleteReplication,
  ExistingObjectReplication, ReplicaModifications) and object lock (value; it
  holds no timestamp). 3 positive.
- dto_values: a stored configuration round-trips byte for byte through the
  bridge (1 positive); a bridge refusal is InternalError (1 negative).
- Use cases: store not initialized is InternalError for each read (3 negative;
  the replication one is ported to the owned signature).
- Embedded, legacy stack, for an ordinary bucket and a reserved-shaped one
  (`sthree-a3-stored`, `a3-empty-s3alias`): missing bucket is NoSuchBucket 404
  (3), no configuration gives each read's own 404 code (6), a caller allowed on
  other buckets only is AccessDenied 403 (6), the stored contents are answered
  (6 positive). 15 negative, 6 positive.
- Differential (`scenarios/extra/bucket_replication_lifecycle.rs`): lifecycle
  with every member and a dated rule on a reserved-shaped bucket, lock reads
  before and after a Years rule, absent-configuration reads.

Mutations (run on this tree before the rebase onto ea56aa923):
- M1, `stored_config` rewrites a stored member (WARM -> COLD) before the
  bridge: red in s3_compat::lifecycle::tests::the_legacy_answer_through_the_seam_is_the_answer_of_the_reverse_direction,
  dto_values::tests::a_stored_configuration_is_answered_with_every_stored_member,
  and embedded the_legacy_stack_answers_the_configuration_reads_as_stored.
- M2, the GetBucketLifecycleConfiguration legacy hook allows everything: red in
  embedded the_legacy_stack_answers_the_configuration_reads_as_stored (the
  wrong-resource caller was served).

Not built: the gateway edge (`impl Served`, rows, refusal-matrix rows, the
gateway-stack variant of the behaviour test), the five mutations.

Not verified: the differential (needs the Linux reference binary, so it is
CI's); the e2e_test crate built only by clippy.
2026-11-13 17:57:35 +08:00
overtrue c221fb62a3 test(storage): audit each mutation without an event at exactly one place
The audit coverage guard required the legacy wrapper for every mutating
operation without a notification event. A migrated operation writes its
audit entry in the use case instead, so both stacks record the same entry,
and its legacy edge drops the wrapper. The guard now accepts exactly one
audit site per operation: the legacy wrapper in ecfs.rs / legacy_edge,
or the owned call under src/app. Both sites at once or neither is a failure.
Calls are matched with whitespace ignored, so rustfmt wrapping and trailing
commas still match.

Accepted owned shapes under src/app:
  audit_only_operation(envelope, S3Operation::X, ...)
  OperationHelper::audit_only_from_view(&envelope.audit_view(), S3Operation::X)

SourceSet gains count_ignoring_whitespace for this.

Tests: the production guard (all 22 operations, still on the legacy
wrapper); fixtures accepted (legacy only, owned helper wrapped by rustfmt
with a trailing comma, owned envelope wrapper wrapped); fixtures refused
(neither, legacy plus each owned form, an owned call only under #[cfg(test)],
a wrapper and an owned call for UploadPartCopy, which must not count for
UploadPart). Negatives 6, positives 4.

Mutation: owned helper needle renamed so it never matches; both fixture
tests went red (the_guard_accepts_exactly_one_audit_place,
the_guard_refuses_an_operation_audited_nowhere_or_twice), the production
guard stayed green. An earlier run of the fixtures caught a needle that
missed rustfmt's trailing comma; fixed before this commit.

Not verified: nothing beyond the audit_coverage_tests module was run.
2026-11-13 17:36:03 +08:00
overtrue ad94f6914a feat(gateway): answer RustFS refusal facts in their legacy form
The gateway stack's one error mapping now answers every code the
gateway refuses without a context (its is_contextual set: NoSuchKey,
NoSuchVersion, NoSuchBucket, PermanentRedirect, TemporaryRedirect,
NotModified, AuthorizationHeaderMalformed, MethodNotAllowed,
BucketAlreadyOwnedByYou, AccessForbidden) and InvalidRange through
the gateway's LegacyRustfsRefusal, stating the facts the error's own
response headers carry:
- NotModified: ETag, Last-Modified, Cache-Control, Content-Location,
  Expires, Vary and the evaluated x-amz-version-id;
- InvalidRange: Content-Range: bytes */<length>;
- NoSuchKey / MethodNotAllowed on a delete marker:
  x-amz-delete-marker: true, x-amz-version-id, Last-Modified.
s3s takes no part. A fact a code may not state, an unreadable or
repeated fact, or a refusal the gateway cannot state is answered 500
InternalError rather than sent with a member missing. Before this, each
of these codes failed the gateway's context check, so for example
HeadBucket on a missing bucket could not answer NoSuchBucket on the
gateway stack. The list is pinned against the gateway's own behaviour
in both directions.

PermanentRedirect and TemporaryRedirect are answered 500: the legacy
refusal states only 4xx, 5xx and the 304 of NotModified. RustFS raises
neither.

Adds crate::app::dto_values::stored_etag(&str) -> ETag, the one
conversion of a stored entity tag for every answer that carries one:
a tag the gateway reader accepts is read as it reads it, any other
(empty, a control character) is carried byte for byte
(ETag::from_stored). Nothing calls it yet; batch C's reads and B2's
listings are its first callers.

Not built: no operation of batch C is registered here; the
embedded-server tests of these answers come with HeadObject.

Tests: 12 new in gateway_handlers::error with 18 refused rows (facts a
code may not state, unreadable or incomplete facts, redirects, a hidden
missing key) and the list checked against the gateway both ways, versus
15 answered rows; 2 in dto_values (1 carried-verbatim, 1 spellings).

Changed assertions: every_header_of_the_closed_set_is_carried uses
SlowDown instead of InvalidRange, because InvalidRange is now answered
in the legacy form; it still covers every header of the ordinary
closed set.

Mutations: run right after this commit lands, reported on
rustfs/backlog#2767; a mutation that stays green is fixed by the next
commit.

Refs: rustfs/backlog#2767
2026-11-13 17:24:44 +08:00
overtrue 190f3686f4 refactor(storage): own the object-listing authorization for both edges
Move the listing authorization of batch B2 into
`storage/access_owned/listing_multipart.rs`, generic over
`AuthorizationRequest`, so the legacy access hooks and the gateway edge
call one copy:

- `authorize_bucket_listing` (ListObjects, ListObjectsV2,
  ListObjectVersions, the metadata listings): the bucket on `ReqInfo`, the
  listing action, `ListBucketVersions` as well for `include-deleted: true`
  on a `ListBucket` listing, then the table snapshot. The repeated-query
  `InvalidRequest` and the anonymous `ListBucket` fallback stay inside the
  policy decision.
- `authorize_list_objects_v2`: the on-demand migration generation captured
  before the decision and installed after an allow.
- `prepare_table_data_plane_list_access` and its fingerprint/cursor-key
  helpers move here unchanged, now generic; the snapshot reaches the use
  case through the new `AuthorizationRequest::install_table_list_access`.

The ListObjects, ListObjectsV2 and ListObjectVersions hooks reduce to one
call each; ListMultipartUploads keeps its generation guard and calls the
moved snapshot. Same actions, same order, same refusals.

Tests (1 added, 1 extended; negatives in the extended table outnumber the
allows): `table_list_access_reaches_the_use_case_from_either_edge`; the
include-deleted table (`include_deleted_listing_enforces_version_permissions`)
now decides every case on an envelope too and requires the same answer.

Mutations: dropping the `include-deleted` follow-up makes the extended
table red; an envelope installer that installs nothing makes the new test
red.

Not built here: the gateway rows and the owned use cases (they wait for
the gateway re-pin).

Verified: the touched tests (19, listing/table access/metadata route)
ran on f33271a74 (before re-pin 1); feature and default clippy
--all-targets ran there too; feature clippy --all-targets ran again on
941fb4fc1, the final base.
2026-11-13 17:13:13 +08:00
overtrue 941fb4fc11 refactor(access): one owned copy-source authorization for both copies
CopyObject and UploadPartCopy authorized their copy source with two
identical blocks in their S3Access hooks. Both hooks now call
`access_owned::object_commit::authorize_copy_source`, generic over
`AuthorizationRequest`, which the gateway stack's `Served::authorize`
for both operations will call too (batch E1, rustfs/backlog#2769). The
order is the hooks' order, unchanged: an access-point or Outposts ARN
source is NotImplemented before any policy is read; the source is
recorded for the `x-amz-copy-source` condition; its bucket incarnation
is captured; reading it is authorized (s3:GetObjectVersion when a
version is named); a failed capture is surfaced only after the allow,
and the incarnation is handed on. The destination is decided afterwards
by each hook, as before. `AuthorizationRequest` gains the two installers
the shape needs, implemented for `S3Request` and `RequestEnvelope`.

Measurement for the batch (recorded here as the coordinator asked):
none of the three E1 operations commits a 200 before its work ends, so
the gateway edges will answer with `Resp::new`, not `Resp::commit`.
The pinned 1.0.5 reference binary could not be run on this host (the
asset is Linux x86-64 only and no container runtime was usable), so the
current legacy stack was measured instead: the debug binary at e8847f6f1,
default stack, on 127.0.0.1, driven by `curl --aws-sigv4`:
- CompleteMultipartUpload with a wrong part ETag: 400 InvalidPart with
  Content-Length; success: 200 with Content-Length, not chunked.
- CopyObject, missing source: 404 NoSuchKey; success: 200 with
  Content-Length.
- UploadPartCopy, missing source: 404 NoSuchKey; range beyond the
  source: 416 InvalidRange; success: 200 with Content-Length.
The 1.0.5 source agrees (tag 1.0.5, s3s 5761ddfe): s3s writes a
keep-alive body only for CompleteMultipartUpload and only when the
output's `future` member is set (crates/s3s/src/ops/multipart.rs:27);
RustFS never sets it, and CopyObject and UploadPartCopy have no such
path.

Not built: the destination half of either hook, the gateway edges
(they arrive with each operation's own increment), and the copy-source
read guard (task H's generic form is used as is).

Lines outside tests: +105 / -59 (access.rs, envelope.rs, the new
access_owned/object_commit.rs).

Tests (6 negative cases, 2 positive in the parity table; the hook test
has 2 negative cases and 1 control per operation): copy-source parity
on both edges, comparing decision,
recorded source, handed-on incarnation and ReqInfo for an ARN source,
a source without a read policy, a named version under s3:GetObject,
a deny with a failed capture, a failed capture after an allow, a
missing ReqInfo, and the two allows; and both legacy hooks refuse a
denied source and an ARN source without ReqInfo ever naming the
destination.

Mutations (two grouped runs of the two new tests, each reverted):
- ARN source answered Ok: parity "an ARN source is NotImplemented..."
  and hook "CopyObject: a denied source"/ARN case red.
- source decision skipped: parity "a source without a read policy..."
  and "a deny wins over a failed incarnation capture" red.
- version swap dropped (always s3:GetObject): parity "a named version
  needs s3:GetObjectVersion" red.
- capture failure surfaced before the decision: parity "a deny wins
  over a failed incarnation capture" and "a failed incarnation capture
  is surfaced after the allow" red.
- destination check before the source (CopyObject hook): hook
  "CopyObject: a denied source" red.

Not verified: UploadPartCopy's source-before-destination order is not
distinguishable by an answer on the legacy hook (both halves refuse
AccessDenied); it is pinned on the gateway edge with that operation.
The gateway stack is unchanged by this commit.
2026-11-13 16:58:20 +08:00
overtrue 0e44453639 refactor(helper): make run_owned_mutation generic over its result
`run_owned_mutation` owned a mutation on its own task only when the
mutation answered an s3s `S3Response`. The owned use cases of batch E1
(CompleteMultipartUpload first) answer a gateway DTO with the owned
error type, so the helper is now generic over the result:
`run_owned_mutation<R, E, F>(operation: F) -> Result<R, E>` where
`F: Future<Output = Result<R, E>> + Send + 'static`, `R: Send + 'static`
and `E: From<rustfs_s3_types::S3Error> + Send + 'static`. A lost owner
task is still InternalError "object mutation owner task failed"; the
s3s error is reached through the existing `From` conversion, so the
legacy callers (DeleteObject, DeleteObjects, CompleteMultipartUpload)
compile and answer unchanged. The task-locals it carries into the
spawned task (Keystone credentials, the S3 HTTP operation context) are
unchanged.

Not built: no new caller in this commit; the owned
CompleteMultipartUpload use case is the first.

Tests: none added; the existing callers' tests ran unchanged (71 run
in the touched modules, all passed). Not mutation-checked: a pure
signature generalisation with no new assertion.
2026-11-13 16:58:19 +08:00
overtrue 0fde0f14a0 Revert "fix(ecstore): isolate unavailable local drives during startup (#9353)"
This reverts commit f6d64bc1f.

The change was merged outside the merge queue, so the full end-to-end
merge gate never ran on it. On main that gate fails 25 multi-node
cluster tests since f6d64bc1f (distributed_startup_regression_test,
inline_fast_path_cluster_test, heal_erasure_disk_rebuild_test,
tier_stats_cluster_test, upgrade_compatibility_test): every node answers
503 and never becomes ready, logging "Local endpoint has no format on
the system filesystem; restore its data mount" for its fresh data
directories. The run before it (c43e415b0) passed all of them.

Reverting restores the previous startup behaviour so the change can be
re-landed through the queue together with a decision on how fresh
URL-style clusters on a system filesystem (and the cluster test
harness) are meant to start.

The stack-overflow fix c83f47871 in set_disk/ops/object.rs is kept.
2026-11-13 16:27:06 +08:00
overtrue aa5f145983 build: re-pin the gateway to 1bbef575
Brings rustfs/gateway#1613 (listing members a stored entry may lack are
written as no element), #1614 (ListMultipartUploads url encoding under
the RustFS profile) and #1616 (PutObject and UploadPart carry the legacy
content length). No RustFS caller uses the changed converters yet.
2026-11-13 16:06:54 +08:00
overtrue ea56aa9238 refactor(storage): authorize bucket configuration changes on both edges
`authorize_bucket_config_mutation(server_ctx, req, bucket, action)` is
now generic over `AuthorizationRequest`, so the legacy hooks and the
gateway edge run the same shape. The shape records the bucket on
`ReqInfo` and snapshots the bucket incarnation under the lifecycle
namespace read lock before the decision. It surfaces a snapshot failure
only after an Allow, and on Allow it installs the snapshot (new trait
installer `install_bucket_config_mutation`) for
`bucket_config_mutation_incarnation`.

The 22 legacy hooks of the shape now call it, with no other change:

- Delete: BucketCors, BucketEncryption, BucketLifecycle, BucketPolicy,
  BucketReplication, BucketTagging, BucketWebsite and
  PublicAccessBlock.
- Put: BucketAccelerateConfiguration, BucketCors, BucketEncryption,
  BucketLifecycleConfiguration, BucketLogging,
  BucketNotificationConfiguration, BucketPolicy, BucketReplication,
  BucketRequestPayment, BucketTagging, BucketVersioning, BucketWebsite,
  ObjectLockConfiguration and PublicAccessBlock.

DeleteBucketTagging keeps its `with_bucket_tag_deletion` scope at the
call site.

Tests: the decision fixture of the owned-versus-legacy parity tests is
now one shared function. A new parity test runs eight cases through
both request forms and compares the decision and the snapshot handed
on. Three are allowed: IAM user, anonymous via bucket policy, owner.
Five are refused: wrong resource, no policy for the action, anonymous
on a private bucket, anonymous on a missing bucket, owner on a missing
bucket.

Not built here: the gateway edges of these operations (batches A1-A4).
2026-11-13 15:51:23 +08:00
overtrue 8eb89dd542 refactor(storage): decide DeleteObject in one owned function
DeleteObject's composite access decision moves out of the legacy S3Access
hook into access_owned::object_misc::authorize_delete_object, generic over
AuthorizationRequest, so the legacy hook and the gateway edge call the same
code. The legacy hook loads the bucket generation, calls it, and installs
the guard it returns; its order and answers do not change: NoSuchBucket
before policy, DeleteObject[Version] for the version storage deletes (the
source-version header for a replica delete), ForceDeleteObject, the
authenticated-caller rule for a recursive force delete, and
BypassGovernanceRetention.

Not built here: the gateway edge of DeleteObject and its use case on the
owned DTOs (they route the legacy body through the seam and wait for the
re-pin).

Tests (access_owned/object_misc_test.rs): one table test runs both
request forms (s3s request and owned envelope) through the decision and
compares the answer and the state handed on (ReqInfo bucket, key, version;
guard ready or failed): 5 refused (missing bucket before policy,
anonymous, caller without a policy, no caller installed, invalid
force-delete value) and 3 allowed (replica delete naming no version gets
its source version, any other delete ignores that header, an unavailable
generation is handed on as a failed guard).

Mutations (each run collects every failing case):
- NoSuchBucket check removed, invalid force-delete read as false, replica
  version swap removed (one build): the missing-bucket, invalid
  force-delete and replica cases went red, in both request forms
- policy decision skipped: the anonymous and no-policy cases went red,
  in both request forms

Verified locally: clippy -D warnings with and without gateway-stack (on
the base before the last rebase, which touched none of these files); 64
delete-related lib tests (app::object::delete, the lifecycle delete tests,
storage::access delete tests) passed, and the parity test passed in its
own run.

Not mutation-checked here: ForceDeleteObject, the authenticated-caller rule
and BypassGovernanceRetention need a caller a policy partly allows; the
existing use-case tests cover the predicate.
2026-11-13 15:26:49 +08:00
overtrue 4a1d542f2d build: re-pin the gateway to 66193a80 and map seam name refusals once
The pin carries floor-only bucket names across the s3s seam and a typed
ConversionError::name_kind (rustfs/gateway#1610), so HeadBucket's legacy
edge goes through the generated conversion again. seam_error now lives in
crate::app::dto_values and ecfs.rs delegates to it: a bucket name below
the floor stays a server fault, because s3s admits only names the floor
accepts; any other refusal is a server fault as before. handler_error
carries the four stored headers the pin adds to ErrorHeader.

One legacy difference is observable: a key the seam refuses (empty,
longer than 1024 bytes, NUL) is answered 400 InvalidArgument with the
fixed message "Invalid object key", where the message text previously
came from the store or the key validator.
2026-11-13 14:30:24 +08:00
overtrue 75f53303e1 fix(storage): count an operation once, at its edge, not in the owned helper
An operation is counted once per request by its edge, after authorization
and before the use case: the gateway's blanket handler already does so, and
a migrated operation's legacy edge calls record_s3_op at the top of its
impl S3 body. OperationHelper::from_view therefore no longer counts; the
legacy constructor OperationHelper::new keeps counting exactly as before for
operations that are not migrated yet. Without this, every owned mutation
that builds its helper with from_view would count twice on the gateway stack.

Not built: no caller of from_view changes here; the first one is the owned
CreateBucket use case.

Tests (1 negative, 1 positive in one test):
only_the_legacy_helper_counts_the_operation - the owned helper counts
nothing; the legacy one counts once.

Mutation: record_s3_op(op) put back into from_view ->
only_the_legacy_helper_counts_the_operation red ("the owned helper counts
nothing").

Local gate: fmt; clippy --all-targets -D warnings with gateway-stack and in
the default build; nextest storage::helper + app::owned_audit_test, 28 passed.
2026-11-13 14:25:35 +08:00
overtrue f33271a743 test(storage): compare resource-free conditions without the clock keys
`resource_free_conditions_read_the_same_client_through_both_forms`
compared two condition maps read at different instants, so
`CurrentTime` (and, across a second boundary, `EpochTime`) differed
between the legacy and the owned read, and the test failed. Both clock
keys are now removed before the comparison. Every request-derived key
must still match, and `SourceIp` is still pinned to its expected value.

This corrects a test introduced in 9c394af48.
2026-11-13 14:04:54 +08:00
overtrue c83f47871d fix(ecstore): box the legacy GET producer read pipeline
Since eca95f297 (#9304) every GET through the legacy duplex pipeline
built `run_legacy_producer_segment` futures that embedded the whole
`get_object_with_fileinfo` read future inline, wrapped again in an
`async move` block and the shard-emitter scope. `get_object_reader`
builds one such producer per downstream arm and per segment, so its
debug poll frame grew to 641,888 bytes. Any test that read an object
back through a real store on a 2 MiB thread overflowed the stack and
aborted (28 workspace tests and one e2e test on main, e.g.
rustfs-heal mrf_partial_write_test, heal_integration_test, and
rustfs app::object::internal_put::tests).

The read future is now boxed inside `run_legacy_producer_segment`, so
the producer future is pointer-sized. No behaviour change.

Measured (debug, aarch64-apple-darwin, line-tables-only):
- run_legacy_producer_segment future: 110,200 -> 320 bytes
- <SetDisks as ObjectIO>::get_object_reader poll frame:
  641,888 -> 202,352 bytes (prologue stack allocation)

Regression test (negative): set_disk::ops::object::
legacy_producer_stack_tests::legacy_producer_future_keeps_the_read_pipeline_off_the_stack
asserts the producer future is at most 1024 bytes. Mutation: removing
the Box::pin makes it fail with "got 110200 bytes".

Verified locally: the 7 previously aborting rustfs-heal tests pass;
the rustfs lib tests that aborted on main (app::object::internal_put,
on_demand_migration_put, put sse_c, owned_authorization_test,
storage::access, object_zip_download, directory_index, data_usage
snapshot gating, snowball extract) pass, 94 of 94 selected.
clippy -p rustfs-ecstore --all-targets reports only the existing
unused PoolSpaceForTest items in store/rebalance.rs, not this change.
Not run locally: the full workspace, the e2e test
sts_query_compat_test::test_bucket_tags_opa_contract, and release-build
frame sizes.
2026-11-13 13:52:41 +08:00
Hauser Zhi f6d64bc1fb fix(ecstore): isolate unavailable local drives during startup (#9353)
* fix(ecstore): isolate unavailable local drives during startup

Keep missing local endpoints in their configured slots and initialize healthy
local drives before applying the existing format and service quorums.
Protect unformatted Linux URL endpoints from writes to the system filesystem,
validate legacy admission, and retry missing local slots during format waits.

Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): prepare local drives before registry publication

Prepare local disks with bounded concurrency outside inventory locks and publish validated batches under per-instance writer coordination. Preserve existing registrations on failed or cancelled full initialization, reuse current-root handles, and withdraw stale reopened slots.

Unify initial-format and reconnect publication, retain raw and explicit-context constructor contracts, and move filesystem queries into owned native jobs. Centralize format validation and reject special format files before startup writes.

Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ecstore): tighten formatted-error ratchet after refactor

Drop the three removed runtime registration call sites from the shrink-only baseline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: align drive fixtures with startup admission

Expect the disk-scoped missing-root error and enable the existing directory test mode for the synthetic URL-based receiver fixture.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify local drive admission and publication

Share bounded preparation and pre-I/O membership snapshots across local
startup and retry paths, with coherent registry and topology publication.
Coalesce native startup preflight and preserve root and buffer ownership.
Keep reconnect ID caches instance-scoped and retain admitted health monitors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): pin multipart index queue aliases to owner lifetime

Derive the Linux index alias from the existing identity descriptor so a
retired caller descriptor cannot redirect rebuild or reconnect queue lookup.
Retain the original descriptor assertion and cover cross-disk alias reuse.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 00:04:54 -05:00
Hauser Zhi d37640fbb9 fix(grpc): record TLS setup failures and cancelled dial attempts (#9388)
* perf(protos): balance peer channels and synchronize TLS eviction

Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(metrics): observe internode unary request lifetimes

Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): make internode gRPC experiments reproducible

Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): isolate transport knobs and redact dry runs

Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(observability): qualify gRPC windows and trace shutdown

Reject stale, incomplete or maintenance-active measurement evidence using
producer timestamps and independent process identities. Preserve raw inputs,
bind evidence hashes, and exercise failure paths through the script entry point.

Record HTTP/2 graceful, child-join and TLS reloader retirement phases without
changing shared deadlines, join ownership or unclean-shutdown marker policy.
Document RPC coverage, fixed-window behavior and separate maintenance workloads.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): add authenticated ReadAll and fixed-rate Ping probe

Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers.

Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(perf): resolve RPC probe disks by explicit UUID

Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata.

Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(perf): reseed temporary RPC fixtures after restart

Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): cancel unfinished TLS handshakes during shutdown

Cancel pre-request TLS acceptance when the listener retires so silent
peers cannot retain graceful watchers beyond the shared drain budget.
Keep established TLS/H2 response draining and original child joins intact.

Add real TCP/TLS shutdown regressions and document fresh native OTLP
observation fences and exec-confirmed diagnostic test units.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(grpc): reuse bulk keys and fix IPv6 TLS verification

Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression.

Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(grpc): keep eviction race behind cache helpers

Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): load native trust and preserve mTLS client identity

Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline.

Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): account for TLS setup failures and cancelled dials

Own explicit dial metrics from validated endpoint setup through connection completion. Record TLS preparation failures and acquisition cancellation once while keeping peer-health decisions in the connect-result branches.

Cover malformed client identity, queued configuration cancellation and successful coalesced dials, and document the metric scope.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 00:04:41 -05:00
overtrue 9c394af489 refactor(storage): read and install authorization state on both edges
The state an access hook hands to a use case is now read and installed
through the `AuthorizationRequest` trait. Its two implementors are the
s3s request and `RequestEnvelope`, so the legacy hooks and the gateway
edge call the same functions:

- These readers are now generic over `AuthorizationRequest`:
  `apply_bucket_generation_guard`,
  `apply_copy_source_bucket_generation_guard` (both on one shared
  fence), `odm_read_generation`, `prepare_odm_read_generation`,
  `bucket_config_mutation_incarnation`,
  `replication_request_authorized`,
  `retained_table_data_plane_publication_guards`,
  `request_context_from_req`, `resource_free_condition_values`,
  `req_info_ref`, `req_info_mut`, `load_bucket_generation_from_store`
  and `SseKmsPrincipal::from_request`.
- `load_bucket_generation` and `capture_odm_read_generation` now take
  the server context explicitly. The legacy hooks pass
  `FS::server_ctx`.
- New trait accessors: request context; the bucket, pending-delete,
  copy-source and ODM generation guards; the bucket-config mutation
  snapshot.
- New trait installers: the bucket, pending-delete and ODM generation
  guards. The legacy hooks now install those guards through them.
- New: `authorize_object_request(req, bucket, key, version_id,
  action)`, the object twin of `authorize_bucket_request`, with no
  version swap. `versioned_read_action` and `versioned_tagging_action`
  become crate-visible so callers can pick the action first. The
  GetObjectAcl, PutObjectAcl, GetObjectLegalHold and GetObjectRetention
  hooks now call it.
- Removed: `request_context_from_extensions`, which had no caller left.

Each reader keeps the same answers and the same error codes and
messages.

Not built here: the bucket-configuration mutation shape, the
body-input newtypes, and the replication-only, write-ETag and
staged-part checks.

Tests: `storage/access_readers_test.rs` covers each reader through
both forms with the state present and absent (12 refusal
cases, 11 accepted or absent ones), plus the installers on the envelope. The
existing owned-versus-legacy decision table now also runs every case
through `authorize_object_request` on the envelope.
2026-11-13 12:38:19 +08:00
overtrue be2d3ca2cf refactor(storage): add one owned-authorization module per batch
Add `storage/access_owned/` with one empty module for each migration
batch stem: bucket_core, bucket_configs, bucket_access,
bucket_replication_lifecycle, object_meta, listing_multipart,
object_read, object_write, object_commit and object_misc. A batch puts
an authorization shape that only its own operations use into its own
file, so parallel batches never edit the same `mod` list. Shapes that
several batches share stay in `storage/access.rs`.

The directory is mounted as a child module of `access.rs`
(`#[path = "access_owned/mod.rs"]`), so the shapes can use the guard
state that `access.rs` keeps private without widening its visibility.
The gateway edge reaches the modules through
`storage_api::server::http::access::access_owned`, feature-gated in the
same way as `authorize_bucket_request`. The re-export carries
`#[expect(unused_imports)]` until the first batch uses it.

No behaviour change. No tests, since the modules are empty.
2026-11-13 12:38:19 +08:00
overtrue 1ce2e51358 test(server): pin legacy HeadBucket name answers as measured on main
xn--abc is refused by the edge (400) and the SDK sends nothing for a
--x-s3 name, as on main before the use case moved onto the gateway's
types (measured with an embedded-server probe on main). The corrected
test was not run locally: the build host had no disk for it.
2026-11-13 12:12:13 +08:00
overtrue 95c7e4218e build(rustfs): add rustfs-gateway-stream behind gateway-stack
The gateway stack's request-body bridge (batch D: PutObject, UploadPart,
PostObject) reads the gateway's streamed body type, so the rustfs crate
needs rustfs-gateway-stream as a dependency. Added as an optional
dependency enabled only by the gateway-stack feature, without the
tokio-io feature: the bridge does not need it. The workspace entry
already existed (rustfs-protocols depends on it), so the lock gains one
edge and no package.

Not built: the bridge itself; batch D writes it with its first caller.
No tests: a dependency edge has no behaviour of its own.
Verified: check_connect_release_boundary.py --features-only and
--self-test, ci_gate.py --self-test and --check-workflow.
Not compiled locally: the host is short of disk; the crate already
builds in the workspace through rustfs-protocols.
2026-11-13 12:04:29 +08:00
overtrue c43e415b09 feat(server): serve ListBuckets and HeadBucket on the gateway stack (#9296)
The walking skeleton of the gateway edge, behind the gateway-stack cargo
feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary
serves ListBuckets and HeadBucket end to end and answers every other S3
operation 501 NotImplemented. A default build carries neither the stack
selection nor the wrapper: its request path is unchanged.

- One per-operation unit: impl Served for dto::Op with a metric label, an
  authorize without a default body, and an execute. The only impl Handler is
  the blanket one over Served (access phase, authorize, count, execute, one
  error mapping). authorize calls the function the legacy access hook calls.
- One table laid out per family (ten family files named after the migration
  batches); it registers the handlers and is the whole early gate.
- A shared access phase used by both edges, fed on the gateway edge from the
  gateway's verdict.
- Stored bucket names are materialised under the gateway floor only, so names
  the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds
  its input the same way until the gateway re-pin (rustfs/gateway#1610).
- With the gateway selected the legacy service receives exactly what the
  legacy router claims (extension routes, object lambda, control plane).
- A start-up line names the stack the mounted service answers with.
- The release-boundary guard reads feature closures from cargo metadata.

Squashed from gw/T2.5-gateway-stack at aca434fdf onto ce9d0b153, with two
changes made while landing: the new CI job reads the ci-dev cache under its
current digest key, and the cache-reader count in scripts/test_ci_plan.py is
7 (this job is the seventh reader).

Checked: cargo clippy --locked -p rustfs --all-targets -- -D warnings in both
builds on aca434fdf (main up to eca95f297); on the squashed tree the Quick
Checks guards, the spell check, ci_gate.py --self-test and --check-workflow,
and the release-boundary tests. Not run on the squashed tree: a compile of
the rustfs crate (six small commits landed on main since the clippy run) and
any test. The legacy HeadBucket name table carries four provisional statuses
for names no store holds; they are corrected by a follow-up when the probe
of main reports.
2026-11-13 11:31:50 +08:00
cxymds ce9d0b153e fix(heal): retain legacy MRF responsibility after physical repair (#9391) 2026-11-13 11:03:53 +08:00
overtrue ef79dcbe19 ci: conformance and client-matrix lanes against both S3 stacks
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776):
.github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance
corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh
--external) against the candidate binary on RUSTFS_S3_STACK=legacy and
=gateway, one server per stack. The suites are checked out at the gateway rev
Cargo.toml pins. Nightly schedule and workflow_dispatch only.

- Each stack must be observed: the start step writes `ready` only when
  /health/ready answers while its own process is alive and the server's
  server_starting event names the requested stack (scripts/dual_stack.py
  stack); otherwise `did not start: ...`. Summaries print the observed stack.
- Verdicts come from files the run leaves on disk (startup record, stack
  evidence, liveness after the corpus, exit codes, report.json, matrix.json),
  never from an earlier step's exit status.
- Conformance: the rustfs profile describes the gateway edge, so only the
  gateway stack is judged (scripts/check_conformance_rulings.py against
  .config/conformance-rulings.toml). Red: an unruled failure, a ruled case
  failing differently from its ruling's `failure`, a stale ruling, and a skip
  on the server's own answer (endpoint not answering, refused fixture). The
  legacy run is an unjudged reference: it must serve, stay up and complete.
- Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail`
  or `sut-unregistered` cell, a client that could not be installed, or an
  unfinished matrix; legacy is the baseline and every legacy pass must pass on
  gateway.
- The gateway leg and the stack comparison are continue-on-error, report-only
  until T2.10.

Not built: no re-pin (write cases need a gateway rev with
--allow-external-writes plus that flag on the lane's run line); no legacy
rulings; no use of the runner's own --rulings ledger (stale rulings do not
block there); no call to the gateway's check_compat_matrix.sh (it does not
fail `fail` cells or clients that never ran); no JUnit (the pinned runner
writes a raw U+0001); no install cache; toolchain versions copied from the
gateway's client-matrix.yml at 7915d069.

Tests: --self-test of both checkers in make script-tests, 40 tests, 31
negative. Mutations, each red: unruled failure ignored
(test_an_unruled_failure_is_red); stale only when passed
(test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green
(test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit
(test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted
(test_an_environment_exit_is_red); legacy judged or tallied
(test_the_legacy_reference_is_counted_never_judged); dead server green
(test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture
skips not red (test_a_skip_on_the_endpoint_not_answering_is_red,
test_a_skip_on_a_refused_fixture_is_red); ruling on id only
(test_a_ruled_case_failing_differently_is_red); unknown verdict
(test_an_unknown_verdict_is_refused); stack not compared
(test_a_server_of_the_other_stack_is_red); no stack evidence
(test_ready_without_stack_evidence_is_red); failed cell, skipped client,
swallowed matrix exit (test_a_failed_cell_is_red,
test_a_client_that_could_not_be_installed_is_skipped_never_passed,
test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset
gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red,
test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed,
test_diff_without_both_manifests_is_red); diffs
(test_diff_lists_changed_and_absent_cases_only,
test_diff_without_both_reports_compares_nothing).

Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did
not start (gateway stack is not assembled yet, T2.5); legacy reference
completed, 140 of 890 cases executable against an external endpoint; legacy
matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0.

Not verified: the stack observation and liveness record in CI (exercised
locally under bash -eo pipefail with a stand-in server and main's
dual_stack.py, both directions); a scheduled run; a diff with gateway content.
2026-11-13 10:31:46 +08:00
overtrue dea9ac7fe2 refactor(admin): read target subscriptions from the stored configuration
ListTargetSubscriptions rendered each bucket's stored notification
configuration as the s3s DTO (an XML round trip) and walked it as JSON.
It now reads the routes of the stored PersistedNotificationConfiguration
directly through its behavior() projection: same order (queue, topic,
lambda), same exact destination match, same filter-rule reading
("Prefix" before "prefix", a rule without a value supplies none). The
conversion step and its two internal-error exits are gone; the function no
longer fails. admin/handlers/event.rs no longer references s3s.

Tests: the existing subscription test now starts from stored XML, the way
the handler loads it; a new test pins rule-name precedence, valueless and
nameless rules, and exact (case- and whitespace-sensitive) destination
matching. Both were run against the old code first and passed unchanged,
then against the new code. Mutation: not run yet (the build was interrupted); it follows separately.

Not verified: the workspace suite (only the touched modules' tests ran).
2026-11-13 10:27:53 +08:00
overtrue ca5ac98c6c refactor(rustfs): use owned error and header names in three files
Part of the s3s removal tail (rustfs/backlog TAIL, feeding T4.2):

- storage/options.rs: the empty Object Lock value check compares against
  the lowercase rustfs_utils names the file already imports instead of the
  s3s header constants; the comparison stays case-insensitive.
- admin/handlers/pools.rs tests: compare ApiError codes through the owned
  code (the idiom the file already uses) and read headers and status from
  the owned S3Error conversion, from which the s3s one is derived.
- init.rs: the error logged when a bucket's notification rules cannot be
  registered is built with rustfs_s3_types::s3_error!; code and message are
  unchanged, the Debug spelling in that log line is the owned type's.

Not changed: init.rs still builds s3s notification DTOs, because the notify
rule conversion it calls takes them.

Tests: empty_object_lock_values_are_not_user_metadata (4 skipped keys, 1
kept), run against the old code first and passed. Mutation: not run yet (the build was interrupted); it follows separately.

Not verified: the workspace suite (only the touched modules' tests ran).
2026-11-13 10:27:53 +08:00
overtrue 34be0a802f fix(ci): satisfy the unsafe-allowance and spelling checks on main
Two quick checks failed on main after eca95f297:

- scripts/check_unsafe_code_allowances.sh requires a SAFETY: comment next
  to an unsafe_code allowance. The comment above the allowance in
  crates/ecstore/src/erasure/codec/shard_buf.rs already stated the
  contract; it now carries the marker the guard looks for.
- The spell checker reads the byte literal b"hel" in a put.rs test as a
  misspelling. The two frames are now b"he" and b"llo"; the test still
  joins them into "hello".

No behaviour change. Checked locally: the unsafe-allowance guard, typos
over the whole tree, cargo fmt --check. Not run: clippy or tests (a
comment and a test literal).
2026-11-13 10:27:10 +08:00
Hauser Zhi 7e93d41c5d test(ecstore): reserve capacity for namespace ownership probes (#9382) 2026-11-12 21:14:42 -05:00
Rain Jiang eca95f2973 perf(ecstore): erasure read path for large GETs (#9304)
Lands the contributor's pull requests #9302 (opt-in lockless KV cache
buckets), #9303 (per-bucket XXH3 ETags; MD5 off the request task) and
#9304 (erasure read path for large GETs) as one change, because #9304
was stacked on the other two.

Maintainer changes on top of the contributed work:
- Merged main. crates/ecstore/src/erasure/coding/bitrot.rs: kept the
  PR's verify_chunked_frame/append_chunked_data restructuring and routed
  its hash-mismatch log through main's throttled log_hash_mismatch.
- rustfs/src/app/multipart_usecase.rs: took main's per-operation split
  and re-applied the PR's changes in place: multipart_etag_algorithm and
  its test in the parent module, the XXH3 pin in create.rs, the stage
  timer and from_stream_with_etag in upload_part.rs.
- Reworded one doc comment in crates/config/src/constants/object.rs that
  spelled an ecstore import path, which the architecture migration guard
  counts as a direct import.
- fix(config): made the MD5 offload opt-in. Each offloaded upload holds
  one blocking-pool thread until its body ends, and that pool also
  serves disk I/O, so many concurrent large uploads could starve disk
  writes. The default threshold is now 0 (disabled); operators enable
  it with RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES (for example 8388608).
2026-11-13 10:05:26 +08:00
hector 5a26862e2e ci: build macOS and Windows on a weekly schedule (#9286)
* ci: build macOS and Windows on a weekly schedule

Keep the 01:13 UTC cron on Linux and add a Sunday 16:00 UTC cron
for macOS and Windows. Select that subset from the firing cron so a
delayed runner cannot guess the weekday.

Co-authored-by: hector <majinghe@users.noreply.github.com>

* ci: move the weekly desktop cron off minute zero

---------

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-12 20:51:08 -05:00
hector b8efcb8dfa feat(release): package manual Linux builds from a git tag (#9332)
* feat(release): package manual Linux builds from a git tag

Let operators name an existing tag, branch, or commit. An empty ref
still packages the tip of origin/main, and a tag can supply the
package version by dropping one leading v.

Co-authored-by: hector <majinghe@users.noreply.github.com>

* fix(release): confirm historical tags with the tooling helper

The confirm step was calling the script from the tagged commit, which
exits 127 when that commit predates the script and cannot confirm when
it only has an older copy. Use a helper that implements confirm-source.

Co-authored-by: hector <majinghe@users.noreply.github.com>

---------

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-12 20:51:00 -05:00
overtrue 704b984c1f ci: both-stack e2e diff in the source workflows and a strict mint stack gate
T3.1/T3.3 (rustfs/backlog#2772, #2774). The existing nightly lanes get a stack
dimension and their legacy leg is the legacy side: e2e-replication-nightly (repl,
cluster, protocols), e2e-distributed and the ODM minio-source job. A gateway leg
runs scripts/dual_stack.py preflight after its build and stops there when no gateway
start is proven; the summary says 'skipped: gateway stack refuses startup: <why>'.
One summary job per workflow diffs the legs of the same run with junit_diff.py.
dual-stack-nightly.yml covers only e2e-smoke and e2e-full: one build, both stacks
from that binary, gateway legs only when the preflight proved a start.
junit_diff.py reports a missing, empty or case-less side as 'no result for <label>'.
mint.yml stays weekly, builds the image once and runs it per stack; mint_stack_diff.py
gates that every case legacy passes is present and passing on gateway with no new
failure (missing, NA, unknown status and truncated lines never pass). Gateway legs and
the compare job are report-only until T2.10.

Not built: a shared-binary preflight job in the three source workflows (one preflight
step per gateway leg); harness-only filtering for their five profiles; mc/Vault/iptables
in the e2e-full leg; an s3s-e2e both-stack run; rustfs_gateway=debug.
Tests: dual_stack 30 (preflight with fake binaries), mint_stack_diff 16, junit_diff 27;
negatives outnumber positives. Mutations (red test named) are in the PR 9367 description.
Not verified: the workflows have not run in CI in this shape (composites, stack matrices,
image artifact round trip, needs result of a red continue-on-error leg).
2026-11-13 09:21:25 +08:00
Chris An 598282bc63 refactor(admin): move replication and listen routes to owned requests (#9373)
The replication and listen extension routes take owned requests instead of s3s requests: handlers under admin/handlers/s3_extension, one authorization entry per route, the listen subscriber holds a body-less request envelope. Route-by-caller tests cover the seven routes (served and refused), including the empty events value.
2026-11-12 20:20:50 -05:00
Hauser Zhi 45156f313e perf(ecstore): reduce default fsync worker idle spin budget (#9386)
Use 256 idle iterations by default and preserve the configurable 1024
iteration ceiling. Update fallback regression cases and tuning guidance.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 20:17:38 -05:00
Joe C a6cd90b6fb feat(s3): accept a comma-separated region list in RUSTFS_REGION (#9338)
* feat(s3): accept a comma-separated region list in RUSTFS_REGION

Since #8739 CreateBucket rejects any LocationConstraint other than the
server's region. A single deployment that serves clients configured for
several regions can no longer create buckets for all of them.

RUSTFS_REGION / --region now accepts a comma-separated list. The first
entry stays the server's own region, so request signing and
GetBucketLocation are unchanged; every entry is accepted as a
CreateBucket LocationConstraint, and anything else is still rejected
with InvalidLocationConstraint.

The list is parsed once at startup and the extra regions are stored
write-once on the per-instance InstanceContext next to the region,
reaching the use-case through RegionInterface (default: none).

A comma was never a valid region (s3s accepts [a-z0-9-] only), so every
existing value keeps its meaning.

* test(e2e): pin e2e-smoke selection for the listed-regions test

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-12 20:17:30 -05:00
Ramakrishna Chilaka 248c56ec1f perf(ecstore): use pinned directory FDs for rename_part and delete_recovery (#9375)
- Reuse pinned txn_parent_fd and txn_dir_fd in LocalDisk::rename_part to perform FD-relative renameat and a single parent directory fsync for part data and publish.meta, avoiding two full-path prepare_rename_with_retry walks per drive (32 openat2 + 32 fstat + 32 close syscalls per UploadPart/UploadPartCopy across 8 drives).
- Probe open_part_dir_fd on transaction_parent before calling create_dir_all in prepare_part_transaction so Part 2..N uploads reuse the existing <upload_id>/<data_dir> directory with a single openat2.
- Remove the redundant file.metadata() (fstat) pre-check in read_child_regular_bytes and delegate directly to File::read_to_end, which already performs its own single fstat and buffer reservation.
- Use FD-relative openat, renameat, unlinkat, and unlinkat(AT_REMOVEDIR) via file_path_dir for <object>/<old_data_dir> staging and finalization in delete_recovery, and drop file_path_dir at the end of Stage 1 for inline unversioned deletes when no old_data_dir or surviving target exists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-12 20:17:22 -05:00
overtrue cb75405d8c ci: log the selected S3 stack at start-up and prove it from server logs
The both-stack lanes need the stack a server actually selected, not the variable
a workflow set. The start-up line gains the s3_stack field (below the stack match,
so a refused gateway request never logs it), and scripts/dual_stack.py stack reads
that line from server logs: it fails on a server of the other stack, on no server of
the requested stack, and on no server output; older-release servers count as unstamped.

Not built: the preflight and summary subcommands, the workflows and the mint gate.
Tests: 10 in scripts/test_dual_stack.py (7 negative, 3 positive), in make script-tests.
Mutation: ignoring foreign-stack starts turns test_a_server_of_the_other_stack_fails_the_proof
and test_the_other_direction_fails_too red.
Not verified: clippy with --features gateway-stack; rustfs nextest.
2026-11-13 09:08:23 +08:00
863 changed files with 151435 additions and 27633 deletions
+11
View File
@@ -31,3 +31,14 @@
# the flag, so the two can no longer drift apart unnoticed.
#
# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
# Exact internode HTTP/2 PING ACK timeout accounting is also deliberately
# default-off. Hyper exposes the exact timeout only through its unstable tracing
# event, so diagnostic builds must opt in to both the feature and cfg:
#
# RUSTFLAGS="--cfg hyper_unstable_tracing" \
# cargo build -p rustfs --features diagnostic-internode-http2-ping
#
# `crates/protos/build.rs` and `crates/obs/build.rs` fail fast if the feature and
# cfg drift apart. Do not move this cfg into global rustflags: doing so would
# recompile every dependency under an unstable, non-semver build contract.
+10
View File
@@ -0,0 +1,10 @@
# Rulings for the gateway conformance lane (.github/workflows/gateway-acceptance.yml).
#
# One row per case that fails on the GATEWAY stack for a reviewed reason: `case`; `ledger`, the
# rustfs/backlog#2684 entry that accepts the difference (R-0xx or K-n); a one-line `reason`; and
# `failure`, a prefix of the failure the ruling accepts, as the runner's report.json renders it
# (`[expect/status] at /expect/status: exchange #1, attempt 1: expected status 400, observed 403`).
# scripts/check_conformance_rulings.py fails the lane on a failed case without a row, on a ruled
# case whose failures do not start with its `failure`, and on a row whose case no longer fails or
# no longer exists. The legacy stack is a reference run and takes no rulings: the `rustfs` profile
# describes the gateway edge, not the legacy one.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=dde32c85d39a3c2eb1cad6912861883af2b3554fc6d04b86b3aeb2afbcc86c7e
sha256-linux=155b526f75743796c18c4c5202bbef3a1770b0ea4ba18b32910317a957ad36f2
sha256-darwin=24a8cfaa1b9125bd69d7d4a7ff752a8ab8ed350c761d90ae8472ef52393e2686
sha256-linux=014257197fbbfbb48cab2864da9e10d96ee877d478fa29d119f8186532ea293d
+1 -1
View File
@@ -1 +1 @@
sha256=caccbc5bcbfbe1b7548fbd184f1a6ae3aedec7cb84ed862d62dddbe27e75271f
sha256=fd5683273bba1eae3b4402a4e854a5d168ace845bd0e5fc801f8e336485a4394
+5
View File
@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: release-feature-boundary-check
release-feature-boundary-check: ## Check no release-reachable feature enables a debug-only one
@echo "🚧 Checking release feature boundaries..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --features-only
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+6
View File
@@ -39,7 +39,11 @@ script-tests: ## Run shell script tests
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_grpc_window.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_rpc_timeout_boundary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_msgpack_peer_canary.py
./scripts/test_internode_msgpack_peer_canary_runner.sh
./scripts/test_object_batch_bench_enhanced.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_inline_docker_lab.py
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
@@ -99,6 +103,8 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/check_conformance_rulings.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/compat_matrix_report.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_api_coverage.py
+15 -5
View File
@@ -99,6 +99,11 @@ threads-required = "num-test-threads"
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
@@ -136,8 +141,9 @@ setup = 'ecstore-large-stack'
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
# The multipart refusal fixture composes real-store setup, abort, quota and legal-hold futures.
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
@@ -405,6 +411,11 @@ threads-required = "num-test-threads"
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
@@ -455,7 +466,7 @@ filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
@@ -712,9 +723,8 @@ test-group = 'ecstore-serial-flaky'
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# starts two single-node servers on random ports with isolated temp
# dirs, and runs one SDK script covering every S3 operation against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
+2 -2
View File
@@ -9,9 +9,9 @@
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# name = "candidate-tuned"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
# env = { RUSTFS_EXAMPLE_SETTING = "on" }
schema = 1
-2
View File
@@ -28,8 +28,6 @@ rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
+2 -2
View File
@@ -13,6 +13,6 @@
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=63
s3_error_lines=217
files=50
s3_error_lines=103
ecstore_files=0
+1 -1
View File
@@ -22,7 +22,7 @@ rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7858 2
rustfs-ecstore 7858 3
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-gateway-adapter 74 0
@@ -34,6 +34,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -43,6 +47,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -56,7 +62,7 @@ services:
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "9000:9000"
- "${RUSTFS_NODE1_PORT:-9000}:9000"
networks:
- rustfs-cluster-net
@@ -81,6 +87,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -90,6 +100,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -103,7 +115,7 @@ services:
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "9001:9000"
- "${RUSTFS_NODE2_PORT:-9001}:9000"
networks:
- rustfs-cluster-net
@@ -128,6 +140,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -137,6 +153,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -150,7 +168,7 @@ services:
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "9002:9000"
- "${RUSTFS_NODE3_PORT:-9002}:9000"
networks:
- rustfs-cluster-net
@@ -175,6 +193,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -184,6 +206,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -197,7 +221,7 @@ services:
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "9003:9000"
- "${RUSTFS_NODE4_PORT:-9003}:9000"
networks:
- rustfs-cluster-net
+58
View File
@@ -0,0 +1,58 @@
# syntax=docker/dockerfile:1.6
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Server-only Linux validation image. The build context supplies a Git bundle.
FROM rust:1.100.0-trixie AS build-tools
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates clang lld libssl-dev pkg-config protobuf-compiler python3 python3-boto3 \
&& rm -rf /var/lib/apt/lists/*
ENV RUSTUP_TOOLCHAIN=1.100.0 \
CARGO_BUILD_JOBS=4 \
CARGO_NET_GIT_FETCH_WITH_CLI=true \
CARGO_TARGET_DIR=/work/rustfs/target \
RUSTFLAGS="-C link-arg=-fuse-ld=lld"
FROM build-tools AS build
ARG SOURCE_REVISION
COPY repository.bundle /tmp/repository.bundle
COPY inspect_geometry.rs /tmp/inspect_geometry.rs
RUN git clone /tmp/repository.bundle /work/rustfs \
&& git -C /work/rustfs checkout --detach "$SOURCE_REVISION" \
&& test "$(git -C /work/rustfs rev-parse HEAD)" = "$SOURCE_REVISION"
WORKDIR /work/rustfs
RUN --mount=type=cache,id=rustfs-inline-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=rustfs-inline-git,target=/usr/local/cargo/git \
--mount=type=cache,id=rustfs-inline-linux-target,target=/work/rustfs/target \
cargo build --locked --release --bin rustfs \
&& install -Dm755 target/release/rustfs /out/rustfs
# Build the diagnostic tool after preserving the clean-source server binary.
RUN cp /tmp/inspect_geometry.rs crates/filemeta/examples/inline_lab_inspect.rs
RUN --mount=type=cache,id=rustfs-inline-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=rustfs-inline-git,target=/usr/local/cargo/git \
--mount=type=cache,id=rustfs-inline-linux-target,target=/work/rustfs/target \
cargo build --locked --release -p rustfs-filemeta --example inline_lab_inspect \
&& install -Dm755 target/release/examples/inline_lab_inspect /out/inline_lab_inspect
FROM debian:trixie-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3t64 \
&& rm -rf /var/lib/apt/lists/*
ARG SOURCE_REVISION
LABEL org.opencontainers.image.revision="$SOURCE_REVISION" \
rustfs.validation="inline"
COPY --from=build /out/rustfs /usr/local/bin/rustfs
COPY --from=build /out/inline_lab_inspect /usr/local/bin/inline_lab_inspect
ENTRYPOINT ["/usr/local/bin/rustfs"]
+5
View File
@@ -0,0 +1,5 @@
FROM debian:trixie-slim
COPY mc /usr/bin/mc
RUN chmod 0755 /usr/bin/mc && /usr/bin/mc --version
LABEL rustfs.validation="inline"
ENTRYPOINT ["/usr/bin/mc"]
+133
View File
@@ -0,0 +1,133 @@
# Inline geometry Linux lab
**Use this when** validating inline placement, block geometry, Range reads and reader rollback in a task-owned local Docker cluster.
The adapter reuses the repository's four-node compose topology with 16 logical Docker volumes and EC12+4. It binds one S3 port to loopback, keeps strict durability and checks running binary hashes and persisted object geometry. Docker volumes share the host storage substrate. Performance observations describe this local virtualized lab.
## Build a frozen server image
Use the current stable toolchain pinned in the lab Dockerfile. Supply a Git bundle and the metadata inspector source as the build context. The server is compiled from a clean checkout before the diagnostic example is added.
```bash
set -euo pipefail
LAB_BUILD_DIR="$(mktemp -d)"
LAB_REVISION="$(git rev-parse HEAD)"
git bundle create "$LAB_BUILD_DIR/repository.bundle" HEAD
cp crates/filemeta/examples/inline_lab_inspect.rs "$LAB_BUILD_DIR/inspect_geometry.rs"
docker --context "$LAB_CONTEXT" build --target build-tools \
-t rustfs-inline-build-tools:1.100 \
-f .docker/inline-lab/Dockerfile .docker/inline-lab
docker --context "$LAB_CONTEXT" build \
--build-arg SOURCE_REVISION="$LAB_REVISION" \
-t rustfs-inline-validation:local \
-f .docker/inline-lab/Dockerfile "$LAB_BUILD_DIR"
LAB_EXPORT_ID="$(docker --context "$LAB_CONTEXT" create \
--label rustfs.validation=inline rustfs-inline-validation:local)"
docker --context "$LAB_CONTEXT" cp \
"$LAB_EXPORT_ID:/usr/local/bin/rustfs" "$LAB_BUILD_DIR/rustfs-linux"
docker --context "$LAB_CONTEXT" rm "$LAB_EXPORT_ID"
```
## Configure and probe
Select a new compose project and a private state directory. The adapter generates test credentials there. Existing resources require matching ownership labels. Configure checks the image's source revision and pins immutable image IDs. Each operation takes an exclusive state lock. Deployment waits for `/health/ready` on every node, covering storage, IAM, lock quorum and peer dependencies.
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" configure \
--context "$LAB_CONTEXT" --project "$LAB_PROJECT" \
--image rustfs-inline-validation:local --revision "$LAB_REVISION" \
--binary "$LAB_BUILD_DIR/rustfs-linux" --port 19080
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" deploy \
--bucket inline-geometry-probe --shard-bytes 0
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" probe \
--bucket inline-geometry-probe
```
The probe verifies complete bytes, HEAD sizes, Range contents, EC12+4 and the actual persisted block. It records inline markers for 4 KiB, 1 MiB, 1 MiB + 1 byte, 1.25 MiB, 1,572,856 bytes, 1,572,857 bytes and 1.5 MiB objects in `probe.json`. The server-only image includes the existing filemeta decoder through a small geometry inspector. Console assets and the administration CLI are outside this lab image.
## ABBA deployment hook
Use `inline_docker_lab.py ... deploy` in the existing `run_hotpath_warp_abba.sh` external deploy hook. Export `RUSTFS_ERASURE_SHARD_BYTES` according to `HOTPATH_ABBA_PHASE`. The adapter uses the supplied leg, phase, bucket, binary and evidence-file context, resets only owned lab volumes, checks running configuration and binary hashes, creates the fresh bucket and writes the runner's deployment evidence.
Use the same frozen binary for both sides of a geometry experiment. Fix the inline budget and encoding mode, then compare zero with one candidate shard value. Include the CPU, memory and image identities from every node in each result. The state `config.json` supplies the generated test credentials for the load driver. Set `DOCKER_CONTEXT` to the selected context when the driver captures Docker resource snapshots.
For content compatibility across an image or geometry change, configure the next selected image and artifact using the same state, then deploy with `--keep-data` and an existing bucket. This retains owned volumes. Compare existing objects and version IDs with the manifest prepared before the change.
## Cleanup
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" down --purge
```
Cleanup checks ownership for containers, volumes and networks. A configuration cannot adopt another project's resources. Raw measurements, state, Git bundles and build outputs stay in task artifact directories.
Unit checks run with `scripts/python_bin.sh scripts/test_inline_docker_lab.py` and are included in `make script-tests`.
## Linux content-probe client
The PUT after-probe requires a working `mc`. For a macOS client affected by cgo initialization failures, compile the same frozen public mc module for Linux with CGO disabled, build it with `Dockerfile.mc`, and use `scripts/inline_docker_mc.sh` as the process-local `mc` command. Record the module version, checksum and resulting image ID with the run.
The adapter rewrites only the lab loopback alias endpoint to the private `node1` endpoint. Alias configuration stays in the probe's own `MC_CONFIG_DIR`, and `cat` streams raw bytes. The existing HEAD, GET and hash checks continue to execute.
```bash
export RUSTFS_INLINE_LAB_STATE="$LAB_STATE"
export RUSTFS_INLINE_MC_IMAGE="$LAB_MC_IMAGE"
mkdir -p "$LAB_TOOLS"
ln -s "$PWD/scripts/inline_docker_mc.sh" "$LAB_TOOLS/mc"
export PATH="$LAB_TOOLS:$PATH"
```
## Compare transport and placement independently
`deploy --rpc-mode json-compat` explicitly clears both MessagePack-only flags on every node. `deploy --rpc-mode msgpack-only` enables both flags for the selected homogeneous lab fleet. Deployment evidence records the mode and checks both running flags. The default is `json-compat`. The ABBA hook may select the same mode through `RUSTFS_INLINE_LAB_RPC_MODE`.
First compare the base and bounded-multi-block binaries at `--shard-bytes 0` in JSON compatibility mode. Then repeat with MessagePack-only on both sides. A same-binary transport ABBA isolates the encoding effect. Keep the 128 KiB budget, CPU sets, strict durability, load and object set fixed. Check actual markers and block sizes at the boundary sizes before timing. Retain baseline-drift and candidate gates, error logs and raw request distributions.
For a rolling downgrade, redeploy the current binary with `--keep-data --rpc-mode json-compat` before introducing an older node. Check old and new reads, version IDs and batch deletes against the persisted manifest. A running MessagePack-only flag pair establishes the lab configuration. Compatibility with a particular old reader requires the explicit reader test.
## Preserve a compatibility manifest
Run the seed once in a fresh lab bucket. It records full byte oracles and the observed version inventory, including any additional version matching a known body, and leaves a 5 MiB multipart part ready for the next reader. Each bucket has its own private `compat-<bucket>.json` manifest. Keep the same state and volumes across image changes. A fresh deployment or cleanup that resets volumes moves existing compatibility manifests into `compat-history`. Retained-data deployments preserve the active manifests.
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-seed --bucket inline-compat
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check --bucket inline-compat
# Configure the next frozen image and binary, then retain the data.
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" deploy \
--bucket inline-compat --shard-bytes 0 --rpc-mode json-compat --keep-data
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check --bucket inline-compat
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-finish-mpu --bucket inline-compat
```
The check compares every persisted version ID, HEAD length, complete SHA256, cross-block Range and per-version geometry from `xl.meta`. The seed includes zero-length objects, null versions, the 192 KiB versioned admission boundary and the accumulated 128 KiB shard boundary. Seed a second bucket on the new writer before the downgrade, then check both manifests using the selected old reader. Complete an old upload on the new binary and a new upload on the old binary. After returning to the current fleet, `compat-delete --bucket <bucket>` deletes the recorded versions in one S3 batch and checks the exact remaining inventory and bytes.
Successful multipart-completion and batch-delete receipts are saved before subsequent content and geometry checks. If either check fails, rerun the same command with the same state directory to resume verification from the saved completion state. After a seed check fails with a saved manifest, use `compat-check` to resume verification.
For mixed-version testing, retain JSON compatibility mode throughout the rolling replacement. Verify each node's immutable image and binary, then perform manifest checks and new writes through both coordinator versions using `--coordinator node1` and `--coordinator node4`. The manifest records the actual writer coordinator revision from the running image. Stop one owned node to check read availability at EC12+4, restore it and check the manifests again. Record the exact oldest tested source revision, toolchain and build profile with the evidence. The version boundary is established by these executed cells.
An owned lab can replace just its fourth node through a compose image override. Select the frozen old image first and verify its revision label. Keep `compose.json` in JSON compatibility mode for this cell.
```bash
set -euo pipefail
OLD_IMAGE_ID="$(docker --context "$LAB_CONTEXT" image inspect "$OLD_IMAGE" --format '{{.Id}}')"
test "$(docker --context "$LAB_CONTEXT" image inspect "$OLD_IMAGE_ID" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}')" = "$OLD_REVISION"
INLINE_OLD_IMAGE_ID="$OLD_IMAGE_ID" scripts/python_bin.sh - <<'PYTHON' > "$LAB_STATE/mixed-image.json"
import json
import os
import sys
override = {"services": {"node4": {"image": os.environ["INLINE_OLD_IMAGE_ID"]}}}
json.dump(override, sys.stdout)
PYTHON
docker --context "$LAB_CONTEXT" compose --project-name "$LAB_PROJECT" \
--file "$LAB_STATE/compose.json" --file "$LAB_STATE/mixed-image.json" \
up --detach --no-deps --no-build --pull never node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check \
--bucket inline-compat --coordinator node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-seed \
--bucket inline-mixed-writer --coordinator node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check \
--bucket inline-mixed-writer --coordinator node1
```
Inspect all four running image IDs and binary hashes before recording the mixed-fleet result. Redeploy the configured current image with `--keep-data --rpc-mode json-compat` to restore a homogeneous fleet. The same manifest checks apply after stopping and restarting one owned node through the base compose file.
+4
View File
@@ -107,6 +107,10 @@ runs:
git cat-file -e "$FOOTPRINT_BASE^{commit}" 2>/dev/null || git fetch --no-tags --depth=1 origin "$FOOTPRINT_BASE"
./scripts/check_s3s_footprint.sh --base "$FOOTPRINT_BASE"
- name: Check release feature boundaries
shell: bash
run: python3 scripts/check_connect_release_boundary.py --features-only
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
+83 -4
View File
@@ -296,6 +296,34 @@
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/rdma-feature-check.yml",
"name": "S3-over-RDMA feature check",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"schedule",
"workflow_dispatch"
],
"cadence": "weekly",
"timezone": "UTC",
"schedule": [
"37 5 * * 1"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 192,
"result_owner": "rdma-feature-check.yml",
"disposition": "compiles the nixl feature and runs rustfs-nixl unit tests on Linux; the feature is off in every other build",
"failure_owner": "workflow",
"origin": "added",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/rustfs-validation-sweep.yml",
"name": "RustFS Validation Sweep",
@@ -360,14 +388,15 @@
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"13 1 * * *"
"13 1 * * *",
"17 16 * * 0"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"disposition": "Daily cron builds Linux; Sunday 16:17 UTC builds macOS and Windows. Public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -764,6 +793,31 @@
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/dual-stack-nightly.yml",
"name": "dual-stack-nightly",
"roles": [
"periodic",
"manual"
],
"events": [
"workflow_dispatch",
"schedule"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"13 3 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "dual-stack-nightly.yml",
"disposition": "nightly run of the two e2e profiles only ci.yml runs, on the gateway stack, with the JUnit reports retained for 30 days",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/e2e-distributed.yml",
"name": "e2e-distributed",
@@ -951,6 +1005,31 @@
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/gateway-acceptance.yml",
"name": "Gateway Acceptance",
"roles": [
"periodic",
"manual"
],
"events": [
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"29 3 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "gateway-acceptance.yml",
"disposition": "gateway conformance corpus (rustfs profile) and client matrix against the candidate on both S3 stacks",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/gateway-integration-sync.yml",
"name": "Gateway Integration Sync",
@@ -1023,7 +1102,7 @@
},
{
"workflow": ".github/workflows/manual-main-packages.yml",
"name": "Manual main packages",
"name": "Manual Linux packages",
"roles": [
"manual"
],
@@ -1037,7 +1116,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "manual-main-packages.yml",
"disposition": "Packages the origin/main tip as workflow artifacts only. No GitHub Release or download-channel writes.",
"disposition": "Packages a git tag, or the origin/main tip when the ref input is empty, as workflow artifacts only. No GitHub Release or download-channel writes.",
"monitor_reason": "Operator-requested packaging has no periodic freshness obligation.",
"origin": "manual-main-packages"
},
+1 -1
View File
@@ -81,6 +81,6 @@ Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicl
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. The daily `build.yml` schedule selects Linux; the Sunday 16:17 UTC schedule selects macOS and Windows. Manual dispatch can still select any platform list. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+27 -8
View File
@@ -12,7 +12,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly development artifacts and reusable immutable-candidate builds.
# Scheduled development artifacts and reusable immutable-candidate builds.
# The daily cron builds Linux. Sunday 16:17 UTC builds macOS and Windows.
# Public release writes live exclusively in release-publish.yml. Candidate callers
# select Linux or desktop and pin release inputs before any matrix starts.
@@ -30,7 +31,8 @@ on:
required: true
type: string
schedule:
- cron: "13 1 * * *" # Nightly 01:13 UTC
- cron: "13 1 * * *" # Daily Linux, 01:13 UTC
- cron: "17 16 * * 0" # Weekly macOS and Windows, Sunday 16:17 UTC
workflow_dispatch:
inputs:
publish:
@@ -219,16 +221,33 @@ jobs:
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
# Daily development builds cover Linux. Desktop remains available on
# manual dispatch and is required after candidate field acceptance.
# A direct schedule has no platforms input. Choose its subset from
# github.event.schedule, which is fixed when that cron fires, so a
# queued job cannot switch subsets because the runner clock moved.
# Callers that pass platforms keep that list, including a scheduled
# caller: inputs.platforms is set even though event_name stays schedule.
RAW_PLATFORMS: ${{ inputs.platforms || 'linux' }}
HAS_PLATFORM_INPUT: ${{ inputs.platforms && 'true' || 'false' }}
EVENT_NAME: ${{ github.event_name }}
EVENT_SCHEDULE: ${{ github.event.schedule }}
run: |
set -euo pipefail
selected="$RAW_PLATFORMS"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
if [[ "${EVENT_NAME}" == "schedule" && "${HAS_PLATFORM_INPUT}" != "true" ]]; then
if [[ "${EVENT_SCHEDULE}" == "13 1 * * *" ]]; then
selected="linux"
elif [[ "${EVENT_SCHEDULE}" == "17 16 * * 0" ]]; then
selected="desktop"
else
echo "Unrecognized schedule cron: ${EVENT_SCHEDULE}" >&2
exit 1
fi
else
selected="${RAW_PLATFORMS}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
fi
# linux-aarch64-* need a real aarch64 host (placement rule: hardware)
+3 -125
View File
@@ -92,7 +92,6 @@ jobs:
connect: ${{ steps.scope.outputs.connect || 'true' }}
packages: ${{ steps.scope.outputs.packages }}
full_validation: ${{ steps.scope.outputs.full_validation || 'true' }}
gateway_smoke: ${{ steps.scope.outputs.gateway_smoke || 'true' }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
@@ -118,8 +117,8 @@ jobs:
printf '%s\n' 'mode=full' >> "$GITHUB_OUTPUT"
echo "Base CI policy unavailable; running the full matrix."
fi
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=7$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'gateway_smoke=true' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=8$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
echo "Base policy does not cover this workflow contract; retaining the full matrix."
fi
- name: Retain verification plan
@@ -469,7 +468,7 @@ jobs:
# hold. `cargo check --release` is metadata-only and needs no
# link-concurrency limit; it is cold because ci-dev holds no release
# artifacts.
- name: Reject short credentials in release builds
- name: Reject debug-only features in release builds
run: python3 scripts/check_connect_release_boundary.py
- name: Check offline enrollment E2E root boundary
@@ -938,8 +937,6 @@ jobs:
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1060,125 +1057,6 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The test-binary
# download, identity, membership, run and completeness steps must stay
# identical to e2e-tests; scripts/ci_gate.py enforces that.
#
# When it runs: every push to main or gateway/integration, merge group,
# nightly schedule and manual dispatch, and every pull request that changes a
# gateway-stack input (GATEWAY_SMOKE_PATHS and the rustfs-gateway pins in
# scripts/ci_gate.py, published by Select CI scope as gateway_smoke).
# Why it is skipped on every other pull request: until T2.10 the gateway
# stack does not serve the whole smoke suite, so the lane fails on any
# revision. A pull request that touches none of its inputs can neither cause
# nor fix that failure, and a red check there says nothing about the change.
# Skipped is the honest state; a failure must never be rendered green, so no
# step here may carry continue-on-error. A selection that is missing or
# unreadable runs the lane. At T2.10, when the lane becomes required, it
# must run on every pull request again: delete the selection, do not extend
# it.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ classify-changes, build-rustfs-debug-binary ]
if: github.event_name != 'pull_request' || needs.classify-changes.outputs.gateway_smoke != 'false'
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install verification runtime
uses: ./.github/actions/test-runtime
- name: Download E2E test binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: e2e-test-binaries
path: ${{ runner.temp }}
- name: Verify E2E archive identity
run: python3 scripts/ci_test_artifact.py verify --configuration e2e --archive "$RUNNER_TEMP/e2e-tests.tar.zst"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the smoke
# selection guard, security exact-count check, and run below, avoiding a
# second compile of the same e2e_test target on cold runners (backlog#1645).
- name: Verify e2e smoke test membership
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json --workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all --failure-output final
- name: Verify e2e smoke execution completeness
run: python3 scripts/ci_test_artifact.py evidence "$RUNNER_TEMP/rustfs-e2e-smoke-list.json" target/nextest/e2e-smoke/junit.xml
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}-${{ github.run_attempt }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
+195
View File
@@ -0,0 +1,195 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly run of the two e2e profiles only ci.yml runs, e2e-smoke and e2e-full
# (rustfs/backlog#2772, T3.1), on the gateway stack, which is the only S3 stack.
# The workflow keeps its file and registry name from the period when it diffed two
# stacks. Each leg uploads its JUnit report and server logs for 30 days; the one
# comparison against the last legacy record (the `dual-stack-leg-<profile>-legacy`
# artifacts of the final both-stack run) is made by hand with scripts/junit_diff.py.
#
# Left out: `dual_stack_differential::` (servers with an explicit
# stack), `fake_s3_target::`, the upgrade write-readiness cases and one deadline
# case (no server), and `distributed_startup_regression_test::` (needs ci.yml's
# hooks-binary manifest). e2e-full uses ci.yml's pinned mc and Vault binaries
# and network fault-injection tools.
name: dual-stack-nightly
on:
workflow_dispatch:
schedule:
# 03:13 UTC nightly, clear of e2e-nightly (04:29), the ODM lane (05:23) and
# e2e-distributed (05:53).
- cron: "13 3 * * *"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
build:
name: Build once
if: github.repository == 'rustfs/rustfs'
# The runner and cache of ci.yml's build-rustfs-debug-binary.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build the rustfs binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload the binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-binary
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
leg:
name: ${{ matrix.profile }}
needs: [ build ]
runs-on: sm-standard-4
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
profile: [ e2e-smoke, e2e-full ]
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Download the binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: dual-stack-binary
path: target/debug
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
test -x "$HOME/.local/bin/awscurl"
- name: Install network fault-injection tools
if: matrix.profile == 'e2e-full'
run: |
sudo apt-get install -y iptables
sudo -n iptables --version
# The endpoint-blackhole heal scenario needs CAP_NET_ADMIN. Containerised
# runners can run iptables but not touch the rule set; the test then logs
# a skip instead of failing, so surface that here where it is visible.
if ! sudo -n iptables -w 5 -S OUTPUT >/dev/null 2>&1; then
echo "::warning::iptables cannot read the OUTPUT chain on this runner (no CAP_NET_ADMIN); the endpoint-blackhole heal scenario will be skipped"
fi
- name: Install mc
if: matrix.profile == 'e2e-full'
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
- name: Verify mc
if: matrix.profile == 'e2e-full'
run: mc --version
- name: Install Vault
if: matrix.profile == 'e2e-full'
run: |
VAULT_VERSION="1.17.6"
VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669 $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
- name: Verify Vault
if: matrix.profile == 'e2e-full'
run: |
"$RUSTFS_TEST_VAULT_BIN" version
- name: Run the profile
env:
PROFILE: ${{ matrix.profile }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/server-logs
run: |
chmod +x target/debug/rustfs
filter='not test(/^(dual_stack_differential|fake_s3_target|distributed_startup_regression_test)::/)'
filter="$filter & not test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)"
filter="$filter & not test(=list_objects_v2_pagination_test::tests::test_list_objects_v2_deadline_rejects_unresponsive_peer)"
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile "$PROFILE" -p e2e_test -E "$filter"
- name: Upload the leg
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-nightly-${{ matrix.profile }}
overwrite: true
path: |
target/nextest/${{ matrix.profile }}/junit.xml
${{ runner.temp }}/server-logs
if-no-files-found: warn
retention-days: 30
@@ -62,6 +62,7 @@ jobs:
timeout-minutes: 45
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ""
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -93,7 +94,7 @@ jobs:
# Build once and carry its source/binary identity into the test invocation.
- name: Build rustfs binary
run: |
python3 scripts/e2e_binary.py build --bins
python3 scripts/e2e_binary.py build --bins --features "$RUSTFS_BUILD_FEATURES"
- name: Verify replication e2e membership
env:
@@ -105,7 +106,7 @@ jobs:
- name: Run replication e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
run: python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
- name: Upload nextest junit report
if: always()
@@ -211,8 +212,10 @@ jobs:
- name: Run protocol e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
run: >-
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
run: |
set -o pipefail
mkdir -p "$RUSTFS_E2E_LOG_DIR"
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture 2>&1 | tee "$RUSTFS_E2E_LOG_DIR/nextest.log"
- name: Upload protocol diagnostics
if: always()
+336
View File
@@ -0,0 +1,336 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Gateway black-box acceptance lanes (rustfs/backlog#2775 T3.4, #2776 T3.5).
#
# Runs two suites of the rustfs/gateway repository against the candidate binary,
# one server:
#
# - the conformance corpus, `--profile rustfs`, judged by
# scripts/check_conformance_rulings.py against .config/conformance-rulings.toml
# (red: an unruled failure, a ruled case failing differently, a stale ruling,
# a skip on the server's own answer);
# - the client matrix, ci/compat/run_matrix.sh --external, judged by
# scripts/compat_matrix_report.py (no `fail`, no `sut-unregistered`, every
# client ran; a client that could not be installed is reported as skipped).
#
# The suites are checked out at the rustfs/gateway rev Cargo.toml pins, so the
# suite and the framework under test never drift apart. The verdict is computed
# from what the run left on disk (startup record, runner exit code, liveness
# after the corpus, report), never from the exit status of an earlier step.
#
# Not a pull-request gate. Scheduled failures are owned by the scheduled-validation
# watchdog (.github/scheduled-validations.json).
name: Gateway Acceptance
on:
schedule:
- cron: "29 3 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Read the gateway pin
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
gateway-rev: ${{ steps.pin.outputs.rev }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Read the rustfs/gateway rev from Cargo.toml
id: pin
run: |
set -euo pipefail
python3 - <<'PY' >> "$GITHUB_OUTPUT"
import re, sys, tomllib
dependencies = tomllib.load(open("Cargo.toml", "rb"))["workspace"]["dependencies"]
revs = {spec.get("rev") for spec in dependencies.values()
if isinstance(spec, dict) and spec.get("git") == "https://github.com/rustfs/gateway.git"}
if len(revs) != 1 or not re.fullmatch(r"[0-9a-f]{40}", str(next(iter(revs)))):
sys.exit(f"Cargo.toml must pin rustfs/gateway at exactly one full rev, found {sorted(map(str, revs))}")
print(f"rev={revs.pop()}")
PY
cat "$GITHUB_OUTPUT"
build-candidate:
name: Build candidate binary
# The same debug build as ci.yml's build-rustfs-debug-binary, reading the
# ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
build-gateway-tools:
name: Build gateway conformance runner and compat-sut
needs: [ plan ]
# The gateway workspace builds with its own rust-toolchain.toml, which rustup
# installs on first use; nothing here names a toolchain.
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Build
working-directory: gateway
run: |
cargo build --locked --release -p rustfs-gateway-conformance -p rustfs-gateway-compat-sut \
--bin rustfs-gateway-conformance --bin compat-sut
- name: Upload tools
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-tools
overwrite: true
path: |
gateway/target/release/rustfs-gateway-conformance
gateway/target/release/compat-sut
if-no-files-found: error
retention-days: 1
acceptance:
name: Acceptance
needs: [ plan, build-candidate, build-gateway-tools ]
# GitHub-hosted like the s3-tests targets of reference-parity.yml: the job
# only runs prebuilt binaries and the pinned clients, none of which needs Docker.
runs-on: ubuntu-latest
# About 3x the 15 minutes the job took in run 43325101234 (client install about
# 8, matrix about 3, corpus about 3), so a hung server costs 45 runner minutes, not more.
timeout-minutes: 45
env:
ENDPOINT: http://127.0.0.1:9000
# The conformance suite signs with these fixed fixture credentials (the AWS
# documentation example pair), so the candidate is started with them and the
# client matrix is handed the same pair.
ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Download candidate binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-candidate-binary
path: target/debug
- name: Download gateway tools
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-tools
path: tools
# Records `ready`, or what the process said when it exited, in lane/startup.txt.
- name: Start candidate
run: |
set -uo pipefail
chmod +x target/debug/rustfs tools/*
mkdir -p lane "$RUNNER_TEMP"/data/rustfs{0..3}
target/debug/rustfs --version | tee lane/candidate-version.txt
# The four directories share the runner's one disk; the bypass is the CI case the
# physical-disk guard sanctions (as in mint.yml).
RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true RUST_LOG=info target/debug/rustfs --address 127.0.0.1:9000 \
--access-key "$ACCESS_KEY" --secret-key "$SECRET_KEY" "$RUNNER_TEMP"/data/rustfs{0..3} \
> lane/rustfs.log 2>&1 &
pid=$!
echo "$pid" > lane/rustfs.pid
# Liveness first: a ready answer counts only while this process is still running.
for _ in $(seq 120); do
if ! kill -0 "$pid" 2>/dev/null; then
status=0
wait "$pid" || status=$?
echo "did not start: exit ${status}: $(grep -v '^[[:space:]]*$' lane/rustfs.log | tail -n 1)" > lane/startup.txt
cat lane/startup.txt
exit 1
fi
if curl -sf "$ENDPOINT/health/ready" >/dev/null; then
echo ready > lane/startup.txt
exit 0
fi
sleep 1
done
echo "did not start: /health/ready did not answer within 120 seconds" > lane/startup.txt
exit 1
# The exit code is data for the judge step, not this step's result.
- name: Run gateway conformance (rustfs profile)
run: |
set +e
tools/rustfs-gateway-conformance run --root gateway/conformance --profile rustfs \
--endpoint "$ENDPOINT" --allow-external-fixtures \
--json lane/report.json > lane/conformance.log 2>&1
echo "$?" > lane/runner-exit.txt
if kill -0 "$(cat lane/rustfs.pid)" 2>/dev/null; then echo alive; else echo exited; fi > lane/alive-after-conformance.txt
tail -n 20 lane/conformance.log
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
# Toolchains for the matrix's SDK driver programs, as the gateway's own
# client-matrix.yml sets them up; the client versions come from
# gateway/compat/versions.toml.
- uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
with:
go-version: "1.26"
cache: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22"
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1
with:
dotnet-version: "8.0.x"
- uses: actions/setup-java@cf277c60eb25467037889841efdb72551f06f6c3 # v4.9.1
with:
distribution: temurin
java-version: "21"
- name: Install Maven
env:
MAVEN_VERSION: "3.9.16"
MAVEN_SHA512: "831a8591fe20c8243b1dbe7d71e3244f31d1665b0804b2e825e38cbbe5ce0cafb8338851f90780735568773e0a6cd07bbec107cda0b896b008b861075358b6f6"
run: |
set -euo pipefail
archive="$RUNNER_TEMP/apache-maven.tar.gz"
curl -sSfL -o "$archive" "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz"
echo "${MAVEN_SHA512} ${archive}" | sha512sum -c -
tar -xzf "$archive" -C "$RUNNER_TEMP"
echo "$RUNNER_TEMP/apache-maven-${MAVEN_VERSION}/bin" >> "$GITHUB_PATH"
# One client at a time, so a client that cannot be installed is reported
# as skipped with its reason instead of stopping the others.
- name: Install pinned clients
working-directory: gateway
run: |
set -uo pipefail
python3 -m pip install --disable-pip-version-check "pyyaml==6.0.2"
installed=()
for client in $(python3 -c 'import tomllib; print(" ".join(sorted(tomllib.load(open("compat/versions.toml", "rb"))["clients"])))'); do
if bash ci/compat/install_clients.sh --clients "$client" > "../lane/install-${client}.log" 2>&1; then
installed+=("$client")
else
echo "${client}: skipped: $(tail -n 1 "../lane/install-${client}.log")" >> ../lane/skipped-clients.txt
fi
done
echo "COMPAT_CLIENTS=$(IFS=,; echo "${installed[*]}")" >> "$GITHUB_ENV"
cat ../lane/skipped-clients.txt 2>/dev/null || echo "every pinned client installed"
# The exit code is data for the judge step, not this step's result.
- name: Run client matrix (external)
working-directory: gateway
env:
GATEWAY_COMPAT_SUT_BIN: ${{ github.workspace }}/tools/compat-sut
GATEWAY_COMPAT_RUN_DIR: ${{ github.workspace }}/lane/compat
GATEWAY_COMPAT_MATRIX_OUT: ${{ github.workspace }}/lane/compat/matrix.json
run: |
set +e
export GATEWAY_COMPAT_ACCESS_KEY="$ACCESS_KEY" GATEWAY_COMPAT_SECRET_KEY="$SECRET_KEY"
PATH="$(go env GOPATH)/bin:$PATH"
if [[ -f ../lane/skipped-clients.txt ]]; then
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs --clients "$COMPAT_CLIENTS"
else
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs
fi
echo "$?" > ../lane/matrix-exit.txt
- name: Judge and summarize
if: always()
run: |
set -uo pipefail
status=0
python3 scripts/check_conformance_rulings.py --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/compat_matrix_report.py check --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
exit "$status"
- name: Upload lane results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-lane
overwrite: true
path: |
lane/*.txt
lane/*.log
lane/report.json
lane/compat/matrix.json
lane/compat/probe.jsonl
lane/compat/sut.log
lane/compat/results/**
if-no-files-found: error
retention-days: 14
+86 -38
View File
@@ -1,11 +1,13 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
# Manual packaging of the current origin/main tip. Workflow artifacts only.
# Dispatch uses this file from the selected branch. The build fetches
# refs/heads/main and checks out that commit, not the dispatch ref or a tag.
# Manual Linux zip/deb/rpm packaging. Workflow artifacts only.
# Dispatch uses this file from the selected branch. An empty ref packages
# the tip of refs/heads/main. A non-empty ref must already exist; the
# resolver fetches that tag, branch, or commit so checkout is not limited
# to the default branch. Nothing is published.
name: Manual main packages
name: Manual Linux packages
on:
workflow_dispatch:
@@ -19,10 +21,16 @@ on:
- arm
- all
default: all
version:
description: "Binary and package version: MAJOR.MINOR.PATCH, or MAJOR.MINOR.PATCH-(alpha|beta|rc).N. No leading v."
required: true
ref:
description: "Git tag to package. Leave empty to package the tip of origin/main. A branch name or full commit SHA is also accepted."
required: false
type: string
default: ""
version:
description: "Package version: MAJOR.MINOR.PATCH, or MAJOR.MINOR.PATCH-(alpha|beta|rc).N. No leading v. Required for origin/main. For a tag, leave empty to derive it from the tag; an explicit value is used as entered."
required: false
type: string
default: ""
permissions:
contents: read
@@ -34,48 +42,43 @@ env:
jobs:
prepare:
name: Resolve main tip and version
name: Resolve source and version
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
source_sha: ${{ steps.pin.outputs.source_sha }}
version: ${{ steps.version.outputs.version }}
source_sha: ${{ steps.source.outputs.source_sha }}
source_kind: ${{ steps.source.outputs.source_kind }}
source_name: ${{ steps.source.outputs.source_name }}
version: ${{ steps.source.outputs.version }}
packaging: ${{ steps.source.outputs.packaging }}
tool_sha: ${{ steps.source.outputs.tool_sha }}
confirm_from: ${{ steps.source.outputs.confirm_from }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
console: ${{ steps.inputs.outputs.console }}
matrix: ${{ steps.matrix.outputs.matrix }}
steps:
- name: Checkout main
# Tooling checkout. An empty ref then detaches this tree to the tip of
# origin/main. A tag or other ref is fetched and pinned without
# replacing this tree, because console resolution needs candidate.py.
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
fetch-depth: 0
persist-credentials: false
- name: Pin the tip of origin/main
id: pin
shell: bash
run: |
set -euo pipefail
GIT_TERMINAL_PROMPT=0 git fetch --no-tags origin refs/heads/main
sha=$(git rev-parse FETCH_HEAD)
git checkout --detach "$sha"
test "$(git rev-parse HEAD)" = "$sha"
printf 'source_sha=%s\n' "$sha" >> "$GITHUB_OUTPUT"
printf 'Pinned origin/main %s\n' "$sha"
- name: Validate package version
id: version
- name: Resolve source and version
id: source
shell: bash
env:
SOURCE_REF: ${{ inputs.ref }}
PACKAGE_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
bash scripts/release/package_manual_artifacts.sh validate \
--version "$PACKAGE_VERSION" \
--arch x86_64 \
--libc gnu | tee "$RUNNER_TEMP/package-version.txt"
printf 'version=%s\n' "$PACKAGE_VERSION" >> "$GITHUB_OUTPUT"
bash scripts/release/package_manual_artifacts.sh resolve-source \
--source "$SOURCE_REF" \
--version "$PACKAGE_VERSION" | tee -a "$GITHUB_OUTPUT"
- name: Install stable Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
@@ -134,22 +137,45 @@ jobs:
fail-fast: false
matrix: ${{ fromJson(needs.prepare.outputs.matrix) }}
steps:
- name: Checkout the pinned origin/main commit
- name: Checkout the pinned source commit
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.source_sha }}
fetch-depth: 0
persist-credentials: false
- name: Confirm the checkout is that main commit
# Tags cut before the manual packaging script still compile from their
# own tree. The official zip/deb/rpm recipe, and confirm-source when the
# tagged script does not implement it, come from the prepare checkout.
- name: Checkout packaging recipe
if: ${{ needs.prepare.outputs.packaging == 'tool' || needs.prepare.outputs.confirm_from == 'tool' }}
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.tool_sha }}
path: packaging-tools
fetch-depth: 1
persist-credentials: false
- name: Confirm the pinned source
shell: bash
env:
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_NAME: ${{ needs.prepare.outputs.source_name }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$SOURCE_SHA"
GIT_TERMINAL_PROMPT=0 git fetch --no-tags origin refs/heads/main
git merge-base --is-ancestor "$SOURCE_SHA" FETCH_HEAD
helper=scripts/release/package_manual_artifacts.sh
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
helper=packaging-tools/scripts/release/package_manual_artifacts.sh
fi
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
printf 'no packaging script implements confirm-source\n' >&2
exit 1
fi
bash "$helper" confirm-source \
--kind "$SOURCE_KIND" \
--name "$SOURCE_NAME" \
--sha "$SOURCE_SHA"
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -247,7 +273,15 @@ jobs:
exit 1
fi
if [[ -n "$BUILD_GLIBC" ]]; then
python3 scripts/release/check_glibc.py "$BUILD_GLIBC" "$bin"
glibc_check=scripts/release/check_glibc.py
if [[ ! -f "$glibc_check" ]]; then
glibc_check=packaging-tools/scripts/release/check_glibc.py
fi
if [[ ! -f "$glibc_check" ]]; then
printf 'glibc check script not found\n' >&2
exit 1
fi
python3 "$glibc_check" "$BUILD_GLIBC" "$bin"
fi
- name: Generate the CPU symbol catalog
@@ -285,9 +319,23 @@ jobs:
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_FEATURES: ${{ matrix.features }}
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
PACKAGING: ${{ needs.prepare.outputs.packaging }}
run: |
set -euo pipefail
case "$PACKAGING" in
source) script=scripts/release/package_manual_artifacts.sh ;;
tool) script=packaging-tools/scripts/release/package_manual_artifacts.sh ;;
*)
printf 'unexpected packaging recipe: %q\n' "$PACKAGING" >&2
exit 1
;;
esac
if [[ ! -f "$script" ]]; then
printf 'packaging script not found: %s\n' "$script" >&2
exit 1
fi
binary="target/${BUILD_TARGET}/release/rustfs"
extra=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
@@ -296,14 +344,14 @@ jobs:
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.sha256"
)
fi
bash scripts/release/package_manual_artifacts.sh package \
bash "$script" package \
--version "$PACKAGE_VERSION" \
--arch "$ARCH" \
--libc "$LIBC" \
--binary "$binary" \
--output dist \
"${extra[@]}"
printf 'Packaged %s from origin/main %s\n' "$PACKAGE_VERSION" "$SOURCE_SHA"
printf 'Packaged %s from %s %s\n' "$PACKAGE_VERSION" "$SOURCE_KIND" "$SOURCE_SHA"
- name: Verify the packaged binary
shell: bash
+38 -9
View File
@@ -108,14 +108,8 @@ defaults:
shell: bash
jobs:
mint:
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# sm-standard-2 has no docker.sock either (weekly runs failed at "Enable
# buildx" from 2026-09-27 to 2026-11-08), so keep this job on ubuntu-latest.
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
image:
name: Build the RustFS image once
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
@@ -134,6 +128,40 @@ jobs:
--cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
-t rustfs-ci \
-f Dockerfile.source .
docker save rustfs-ci --output "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Upload the image
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint-image
path: ${{ runner.temp }}/rustfs-ci.tar
retention-days: 1
mint:
name: mint
needs: [image]
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# sm-standard-2 has no docker.sock either (weekly runs failed at "Enable
# buildx" from 2026-09-27 to 2026-11-08), so keep this job on ubuntu-latest.
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download the image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: mint-image
path: ${{ runner.temp }}
- name: Load the image
run: docker load --input "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Start RustFS
run: |
@@ -151,6 +179,7 @@ jobs:
-e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
-e RUST_LOG=rustfs=info \
-v /tmp/rustfs-mint:/data \
rustfs-ci
@@ -255,7 +284,7 @@ jobs:
alert-on-failure:
name: Alert on scheduled failure
needs: [mint]
needs: [image, mint]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
# dispatch failures are already watched by a human.
@@ -146,6 +146,8 @@ jobs:
python3 ./scripts/check_test_wiring.py --check-profile e2e-odm-interop "${NEXTEST_LISTING}"
- name: Run the interop cases against MinIO
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test --no-tests=fail
- name: Build the MinIO interop report
+91
View File
@@ -0,0 +1,91 @@
name: S3-over-RDMA feature check
# The `nixl` and `cuobject` features are off in every other build, so the code
# behind them is compiled nowhere else. This job compiles the `nixl` half on
# Linux and runs rustfs-nixl's unit tests; nixl-sys builds through its stub
# fallback, so no NIXL SDK is needed.
# The cuObject half needs the NVIDIA cuObject SDK and is not compiled here.
# Not a required check and not part of the `Test and Lint` aggregate.
on:
pull_request:
paths:
- "crates/nixl/**"
- "crates/cuobject/**"
- "rustfs/src/storage/rpc/nixl*"
- "rustfs/src/app/object/cuobject.rs"
- "rustfs/src/app/object/get.rs"
- "rustfs/src/app/object/put.rs"
- "rustfs/src/app/multipart_usecase/upload_part.rs"
- "Cargo.toml"
- "rustfs/Cargo.toml"
- ".github/workflows/rdma-feature-check.yml"
schedule:
- cron: "37 5 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: rdma-feature-check-${{ github.ref }}
cancel-in-progress: true
jobs:
nixl-check:
name: cargo check and unit tests (nixl)
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-nixl
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Install only missing native build tools
run: |
packages=()
command -v g++ >/dev/null || packages+=(g++)
ldconfig -p | grep -q 'libclang' || packages+=(libclang-dev)
if (( ${#packages[@]} )); then
sudo apt-get update
sudo apt-get install --no-install-recommends -y "${packages[@]}"
fi
- name: Check the nixl feature build
run: cargo check --locked -p rustfs --features nixl
# These unit tests use a fake sysfs and plain host memory; none needs an
# RDMA device or the NIXL libraries.
- name: Run the rustfs-nixl unit tests
run: cargo test --locked -p rustfs-nixl --features sys
alert-on-failure:
name: Alert on scheduled failure
needs: [nixl-check]
# Only scheduled runs open or update the tracking issue; pull-request and
# manual runs stay quiet.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+4 -4
View File
@@ -32,9 +32,9 @@
#
# The `differential` job adds the 73-operation dual-stack differential
# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
# started without RUSTFS_S3_STACK), side B is the candidate built above on its
# legacy stack. It reuses the candidate build artifact, never builds the candidate
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY),
# side B is the candidate built above.
# It reuses the candidate build artifact, never builds the candidate
# twice, and prints reference_side, version_a and version_b from the report the test
# wrote; it fails when side A was not the reference or both sides report the same
# version. Every (op, field, a, b) difference goes to the job summary. Report-only
@@ -310,7 +310,7 @@ jobs:
chmod +x target/debug/rustfs
test -x "$REFERENCE_PATH"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
# Side A: the released binary. Side B: the candidate on its legacy stack.
# Side A: the released binary. Side B: the candidate.
export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
+5
View File
@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Deployment
- **The S3 protocol stack is now rustfs/gateway**, and it is the only S3 stack. The legacy s3s-based stack is removed, and so are the `RUSTFS_S3_STACK` environment variable and the `gateway-stack` cargo feature. There is no on-disk format change. `RUSTFS_S3_STACK` no longer selects anything: a server started with it set, to any value, refuses to start and prints `RUSTFS_S3_STACK is no longer supported; the gateway stack is the only S3 stack since <version>`, where `<version>` is the release's version. Remove the variable from service units, container environments, Helm values and orchestration files before upgrading. To roll back, run the previous release binary on the same data.
- Before upgrading, verify that the configured root access key is at least 3 bytes and the root secret key is at least 8 bytes. RustFS now rejects shorter values at startup and identifies the setting to update. Version 1.0.5 accepted these settings at startup and returned HTTP 500 when root requested STS credentials. Coordinate root credential changes across all nodes and clients. See [root credential rotation considerations](docs/operations/two-factor-auth.md#root-credentials-cannot-be-changed-at-runtime).
- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
@@ -44,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Vault KMS backends no longer fail when the server answers reads from a replica**: a Vault or OpenBao node that serves reads locally (an OpenBao standby, a Vault Enterprise performance standby) can return the state from before a write for a short while after acknowledging it. Against such a node the KV2 backend failed every key creation with `404 Key not found` while leaving the key stored without its description and tags, failed operations issued right after a create or update with `Key not found` or `Concurrent modification`, and the Transit backend failed a disable followed quickly by an enable. Key creation on the KV2 backend is now a single create-only write that carries the description and tags. Both backends now re-read a key record when the answer is older than a write the same RustFS node made in the previous five seconds, pause before re-reading after a lost check-and-set write and allow five attempts instead of three, and retry a Transit key delete that is refused because the deletion switch is not visible yet. A different RustFS node reading the same replica can still observe the earlier state until the replica catches up; see [OpenBao as the Vault KMS backend](docs/operations/kms-openbao-compatibility.md).
- **KMS keys could not be described by ARN through the path route**: `GET /rustfs/admin/v3/kms/keys/{key_id}` read the path segment verbatim, so a key id that has to be percent-encoded to fit in one segment — an AWS key ARN, with its `:` and `/` — was looked up, authorized and audited under its encoded spelling and answered `404`. The segment is now percent-decoded first; a segment that is not valid UTF-8 after decoding is refused with `400 InvalidArgument`. The `describe-key?keyId=` form was unaffected.
- **SSE-KMS objects written through the AWS KMS backend could not be read**: `GetObject`, `CopyObject` and every other read of an `aws:kms` object failed with `400 InvalidRequest` ("SSE-S3 requires RUSTFS_SSE_S3_MASTER_KEY to be set ...") on a deployment using `RUSTFS_KMS_BACKEND=aws`, although the write and `HeadObject` succeeded. The AWS backend stored the `GenerateDataKey` blob as it came from AWS, the read path took those unrecognized bytes for a locally wrapped key, and AWS was never asked to decrypt. The backend now stores the blob inside the same data-key envelope the other backends use, and a blob stored without one is routed to AWS when the AWS backend is active. Objects written by 1.0.4 and 1.0.5 are intact and become readable after the upgrade, with no rewrite. Setting `RUSTFS_SSE_S3_MASTER_KEY` was never a workaround. An object written by this release is not readable by an older one; the same was already true of every object the older release wrote itself.
@@ -75,6 +79,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
### Added
- **Multiple bucket regions**: `RUSTFS_REGION` / `--region` accepts a comma-separated list. The first entry stays the server's own region (request signing and `GetBucketLocation` are unchanged); every entry is accepted as a `CreateBucket` `LocationConstraint`, so one deployment can stand in for buckets that live in several AWS regions (local development and test rigs) now that location constraints are validated (#8739). Regions outside the list are still rejected with `InvalidLocationConstraint`. A single region behaves exactly as before; a comma was never a valid region, so no existing value changes meaning.
- **On-Demand Migration**: Lazy, pull-style migration of an existing S3-compatible bucket into RustFS. A local bucket is attached to an external source bucket; a GET for a key that does not exist locally fetches it from the source, streams it to the client, and stores it locally in the same pass, so every later read is served locally. The module is on by default; set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false` on every node to turn it off. A bucket with no source configured behaves exactly as before — the runtime never intervenes on its reads and makes no outbound call. Operator guide at `docs/operations/on-demand-migration.md`.
- Per-bucket configuration persisted as `on-demand-migration.json` in the bucket metadata: source provider (`s3`, `aws`, `minio`, `rustfs`, `r2`, `gcs`), endpoint, region, addressing style, credentials and TLS material, an optional key-prefix filter and source-prefix rewrite, and a policy block covering the inline size threshold, multipart part size, concurrency, queue capacity, timeouts, bandwidth limit and negative-cache TTL
- Admin routes under `/rustfs/admin/v3/on-demand-migration/{bucket}`: `PUT` (with `?dry-run=true` to validate and probe the source without saving), `GET`, `DELETE`, `GET .../status`, plus `POST .../backfill?op=start|cancel` and `GET .../backfill` for the background full-backfill job with its resumable checkpoint. Authorized by the new `admin:GetBucketOnDemandMigration` and `admin:SetBucketOnDemandMigration` actions; every response redacts `secret_key` and `session_token`
Generated
+779 -434
View File
File diff suppressed because it is too large Load Diff
+44 -37
View File
@@ -23,6 +23,8 @@ members = [
"crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/cuobject", # NVIDIA cuObject S3-over-RDMA staging and transfers (optional)
"crates/nixl", # NIXL/UCX S3-over-RDMA server side (optional)
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
@@ -78,7 +80,6 @@ repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
authors = ["RustFS Team"]
@@ -93,7 +94,6 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-gateway-adapter = { path = "crates/gateway-adapter", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
@@ -106,6 +106,8 @@ rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-cuobject = { path = "crates/cuobject", version = "1.0.5" }
rustfs-nixl = { path = "crates/nixl", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
@@ -149,14 +151,16 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "7915d06950f88611d883f6ab3d95efc90c65dec0", version = "0.40.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "7915d06950f88611d883f6ab3d95efc90c65dec0", version = "0.2.0", default-features = false }
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", version = "0.42.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", version = "0.2.0", default-features = false }
# The gateway framework itself, consumed for its extension traits by the ring-2
# adapter. `default-features = false` keeps the `server` feature, and with it the
# server crate, out of the build.
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "7915d06950f88611d883f6ab3d95efc90c65dec0", version = "0.62.0", default-features = false }
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", version = "0.66.0", default-features = false }
# Test-only: builds the `EffectiveHost` a host-resolution test queries with.
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "7915d06950f88611d883f6ab3d95efc90c65dec0", default-features = false }
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", default-features = false }
rustfs-gateway-core = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", version = "0.63.0", default-features = false }
rustfs-gateway-dialect-rustfs-admin = { git = "https://github.com/rustfs/gateway.git", rev = "d6840d54187382d31526317e61369734c759e2cd", version = "0.13.0", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -164,13 +168,13 @@ async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.52.0", default-features = false }
async-nats = { version = "0.53.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.1"
pollster = "1.0.2"
pulsar = { default-features = false, version = "=6.9.0" }
lapin = { default-features = false, version = "4.12.3" }
hyper = { version = "1.12.0" }
@@ -184,6 +188,7 @@ memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.1" }
slab = { version = "0.4.12" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
@@ -204,14 +209,14 @@ quinn = { version = "0.11.12", default-features = false, features = [
] }
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
apache-avro = { version = "0.23.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.2" }
csv-core = "0.1.13"
bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
prost = "0.14.5"
prost = "0.14.6"
quick-xml = "0.42.0"
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
@@ -233,11 +238,11 @@ blake2 = "0.11.0"
chacha20poly1305 = { version = "0.11.0" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "11.1.0" }
jsonwebtoken = { version = "12.0.0" }
openidconnect = { default-features = false, version = "4.1.0" }
pbkdf2 = "0.13.0"
p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] }
rsa = { version = "=0.10.0-rc.18" }
rsa = { version = "0.10.0", features = ["pkcs1v15"] }
rustls = { default-features = false, version = "0.23.46" }
rustls-native-certs = "0.8"
rustls-pki-types = "1.16.0"
@@ -245,13 +250,13 @@ x509-parser = "0.18.1"
sha1 = "0.11.0"
sha2 = "0.11.1"
subtle = "2.6"
zeroize = { version = "1.9.1" }
zeroize = { version = "1.9.2" }
proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.43" }
jiff = { version = "1.0.1" }
time = { version = "0.3.55" }
# Database
@@ -268,18 +273,18 @@ tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.2" }
aws-config = { version = "1.13.3" }
aws-credential-types = { version = "1.3.1" }
aws-sdk-kms = { default-features = false, version = "1.131.0" }
aws-sdk-s3 = { default-features = false, version = "1.159.0" }
aws-sdk-sts = { default-features = false, version = "1.126.0" }
aws-sdk-kms = { default-features = false, version = "1.133.0" }
aws-sdk-s3 = { default-features = false, version = "1.160.0" }
aws-sdk-sts = { default-features = false, version = "1.127.0" }
aws-smithy-async = { version = "1.3.1" }
aws-smithy-http-client = { default-features = false, version = "1.5.2" }
aws-smithy-runtime-api = { version = "1.21.0" }
aws-smithy-http-client = { default-features = false, version = "1.5.3" }
aws-smithy-runtime-api = { version = "1.22.0" }
aws-smithy-types = { version = "1.9.4" }
base64-simd = "0.8.0"
brotli = "9.1.1"
clap = { version = "4.6.7" }
brotli = "9.1.3"
clap = { version = "4.6.8" }
const-str = { version = "1.1.0" }
convert_case = "0.12.0"
criterion = { version = "0.8" }
@@ -287,7 +292,7 @@ crossbeam-queue = "0.3.14"
crossbeam-channel = "0.5.17"
crossbeam-deque = "0.8.8"
crossbeam-utils = "0.8.23"
datafusion = { default-features = false, version = "55.2.0" }
datafusion = { default-features = false, version = "55.3.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.15"
@@ -306,12 +311,14 @@ hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.1"
libc = "0.2.191"
cuobject = { version = "0.1.0", default-features = false }
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
nixl-sys = { git = "https://github.com/houseme/nixl", rev = "cc636c83f13d692e85b67e55aed092413d16fb41" }
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
lz4_flex = { version = "0.14.2", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
md-5 = "0.11.0"
mime_guess = "2.0.5"
@@ -330,14 +337,14 @@ percent-encoding = "2.3.2"
qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.3" }
rand = { version = "0.10.4" }
ratelimit = "2.0.0"
rayon = "1.12.0"
rustfs-erasure-codec = { version = "9.0.0" }
rustfs-erasure-codec = { version = "9.1.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
regex = { version = "1.13.2" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.8.1" }
redis = { version = "1.8.2" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.6" }
rust-embed = { version = "8.14.0" }
@@ -357,7 +364,7 @@ snap = "1.1.2"
starshard = { version = "2.4.1" }
strum = { version = "0.28.0" }
sync_wrapper = "1.0.2"
sysinfo = "0.40.3"
sysinfo = "0.40.4"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.4.1"
@@ -371,15 +378,15 @@ tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.29.2" }
uuid = { version = "1.29.3" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "9.5.1"
xxhash-rust = { version = "0.8.21" }
zip = "9.7.0"
zstd = "0.14.4"
# Observability and Metrics
@@ -396,11 +403,11 @@ opentelemetry-stdout = { version = "0.33.1" }
pyroscope = { version = "2.1.5" }
# FTP and SFTP
libunftp = { version = "0.24.0" }
libunftp = { version = "0.25.0" }
unftp-core = "0.1.1"
suppaftp = { version = "12.2.1" }
suppaftp = { version = "13.0.0" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.1" }
russh = { version = "0.65.2" }
russh-sftp = "3.0.1"
# WebDAV
@@ -409,7 +416,7 @@ dav-server = "0.12.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.4" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.31.2", default-features = false }
hotpath = { version = "0.32.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+7
View File
@@ -170,10 +170,17 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;; \
esac
# `RC_RELEASE=disabled` is reserved for local server-only validation when the
# GitHub release asset is unavailable. The placeholder fails if invoked, so a
# production image that needs the CLI must keep the default release download.
# TARGETARCH is only provided by BuildKit. Fall back to TARGETPLATFORM (for
# example set by docker-compose) and then to the build host, like the cargo
# step above, so every path that builds this file passes a real architecture.
RUN set -eu; \
if [ "${RC_RELEASE}" = "disabled" ]; then \
install -m 0755 /bin/false /usr/local/bin/rc; \
exit 0; \
fi; \
rc_arch="${TARGETARCH:-}"; \
if [ -z "${rc_arch}" ]; then \
case "${TARGETPLATFORM:-}" in \
+4
View File
@@ -311,6 +311,10 @@ x env use rustfs
rustfs --help
```
### 7\. Docker Swarm with the SwarmCLI chart (Option 7)
Follow the [SwarmCLI chart guide](https://docs.rustfs.com/en/installation/cloud-native/swarmcli-chart) to run RustFS on Docker Swarm, as a single-node service or a distributed, erasure-coded cluster with one peer per node, with the open-source [SwarmCLI `rustfs` chart](https://github.com/Eldara-Tech/swarmcli-charts/tree/main/charts/rustfs). The chart keeps the root credentials in Docker secrets and refuses the default `rustfsadmin` keys, upgrades and rolls back as versioned releases, and can serve the S3 API over HTTPS through Traefik. Its Console stays off until you enable it. The chart and [SwarmCLI](https://github.com/Eldara-Tech/swarmcli) are maintained by Eldara Tech, not by the RustFS project.
---
### Accessing RustFS
+4
View File
@@ -239,6 +239,10 @@ x env use rustfs
rustfs --help
```
### 7\. 使用 SwarmCLI Chart 部署到 Docker Swarm (Option 7)
请按照 [SwarmCLI Chart 指南](https://docs.rustfs.com/zh/installation/cloud-native/swarmcli-chart),使用开源的 [SwarmCLI `rustfs` Chart](https://github.com/Eldara-Tech/swarmcli-charts/tree/main/charts/rustfs) 在 Docker Swarm 上运行 RustFS,可部署为单节点服务,或每个节点一个对等节点、采用纠删码的分布式集群。该 Chart 将 root 凭证保存在 Docker Secret 中并拒绝默认的 `rustfsadmin` 密钥,以带版本的发布进行升级和回滚,并可通过 Traefik 以 HTTPS 提供 S3 API。控制台默认关闭,需要时再启用。该 Chart 和 [SwarmCLI](https://github.com/Eldara-Tech/swarmcli) 由 Eldara Tech 维护,并非 RustFS 项目的一部分。
---
### 访问 RustFS
+3 -4
View File
@@ -30,12 +30,11 @@ workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
+3 -4
View File
@@ -30,13 +30,12 @@ workspace = true
[features]
default = []
test-util = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
arc-swap.workspace = true
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
+157
View File
@@ -23,7 +23,80 @@ pub static GLOBAL_LOCAL_NODE_NAME: LazyLock<RwLock<String>> = LazyLock::new(|| R
pub static GLOBAL_RUSTFS_HOST: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("".to_string()));
pub static GLOBAL_RUSTFS_PORT: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("9000".to_string()));
pub static GLOBAL_RUSTFS_ADDR: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("".to_string()));
static NEXT_GRPC_CHANNEL_ID: AtomicU64 = AtomicU64::new(1);
static NEXT_GRPC_CONNECTION_GENERATION: AtomicU64 = AtomicU64::new(1);
#[derive(Debug)]
struct GrpcConnectionStateInner {
channel_id: u64,
generation: AtomicU64,
}
/// Lock-free identity and physical-connection generation shared by every clone of one gRPC channel.
#[derive(Clone, Debug)]
pub struct GrpcConnectionState(Arc<GrpcConnectionStateInner>);
/// A point-in-time identity for the physical connection currently used by a gRPC channel.
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct GrpcConnectionSnapshot {
pub channel_id: u64,
pub generation: u64,
}
impl GrpcConnectionState {
pub fn new() -> Self {
Self(Arc::new(GrpcConnectionStateInner {
channel_id: next_nonzero_id(&NEXT_GRPC_CHANNEL_ID),
generation: AtomicU64::new(0),
}))
}
/// Publish a successfully opened physical transport before tonic makes it request-ready.
pub fn record_connected(&self) -> GrpcConnectionSnapshot {
let generation = next_nonzero_id(&NEXT_GRPC_CONNECTION_GENERATION);
self.0.generation.store(generation, Ordering::Release);
GrpcConnectionSnapshot {
channel_id: self.0.channel_id,
generation,
}
}
/// Return `None` until the first physical transport has connected successfully.
pub fn snapshot(&self) -> Option<GrpcConnectionSnapshot> {
let generation = self.0.generation.load(Ordering::Acquire);
(generation != 0).then_some(GrpcConnectionSnapshot {
channel_id: self.0.channel_id,
generation,
})
}
}
impl Default for GrpcConnectionState {
fn default() -> Self {
Self::new()
}
}
fn next_nonzero_id(counter: &AtomicU64) -> u64 {
loop {
let id = counter.fetch_add(1, Ordering::Relaxed);
if id != 0 {
return id;
}
}
}
pub static GLOBAL_CONN_MAP: LazyLock<RwLock<HashMap<String, Channel>>> = LazyLock::new(|| RwLock::new(HashMap::new()));
#[derive(Clone)]
struct TrackedGrpcConnection {
channel: Channel,
connection_state: GrpcConnectionState,
}
static GLOBAL_TRACKED_CONN_MAP: LazyLock<RwLock<HashMap<String, TrackedGrpcConnection>>> =
LazyLock::new(|| RwLock::new(HashMap::new()));
static GLOBAL_CONN_CACHE_WRITER: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
@@ -231,6 +304,16 @@ pub async fn evict_connection(addr: &str) {
/// Evict a stale/dead connection from the global connection cache with an explicit log level.
pub async fn evict_connection_with_log_level(addr: &str, log_level: ConnectionEvictionLogLevel) {
let _ = remove_connection_with_log_level(addr, log_level).await;
}
/// Remove a cached connection and report whether it existed. Transport owners use this to
/// attach a rebuild reason only to real cache transitions.
pub async fn remove_connection_with_log_level(addr: &str, log_level: ConnectionEvictionLogLevel) -> bool {
// Writes are rare and serialized so the optimized tracked cache and public compatibility map
// cannot publish different channel clones for one key.
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.remove(addr);
let removed = GLOBAL_CONN_MAP.write().await.remove(addr);
if removed.is_some() {
match log_level {
@@ -254,6 +337,7 @@ pub async fn evict_connection_with_log_level(addr: &str, log_level: ConnectionEv
}
}
}
removed.is_some()
}
/// Get a cached gRPC connection for the given address.
@@ -261,8 +345,37 @@ pub async fn cached_connection(addr: &str) -> Option<Channel> {
GLOBAL_CONN_MAP.read().await.get(addr).cloned()
}
/// Get a cached channel together with its shared physical-connection state, when tracked.
pub async fn cached_connection_with_state(addr: &str) -> Option<(Channel, Option<GrpcConnectionState>)> {
if let Some(cached) = GLOBAL_TRACKED_CONN_MAP.read().await.get(addr).cloned() {
return Some((cached.channel, Some(cached.connection_state)));
}
// A miss may race a low-frequency publish/evict. Serialize only this fallback, recheck the
// tracked map, then consult legacy entries injected through the compatibility API.
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
if let Some(cached) = GLOBAL_TRACKED_CONN_MAP.read().await.get(addr).cloned() {
return Some((cached.channel, Some(cached.connection_state)));
}
cached_connection(addr).await.map(|channel| (channel, None))
}
/// Cache a gRPC connection for the given address.
pub async fn cache_connection(addr: String, channel: Channel) {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.remove(&addr);
GLOBAL_CONN_MAP.write().await.insert(addr, channel);
}
/// Cache a gRPC channel and its physical-connection state as one coherent entry.
pub async fn cache_connection_with_state(addr: String, channel: Channel, connection_state: GrpcConnectionState) {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.insert(
addr.clone(),
TrackedGrpcConnection {
channel: channel.clone(),
connection_state,
},
);
GLOBAL_CONN_MAP.write().await.insert(addr, channel);
}
@@ -279,6 +392,8 @@ pub async fn has_cached_connection(addr: &str) -> bool {
/// Clear all cached connections. Useful for full cluster reset/recovery.
pub async fn clear_all_connections() {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.clear();
let mut map = GLOBAL_CONN_MAP.write().await;
let count = map.len();
map.clear();
@@ -363,4 +478,46 @@ mod tls_snapshot_tests {
assert!(!debug.contains("PRIVATE-KEY-MARKER"));
assert!(debug.contains("[REDACTED]"));
}
#[tokio::test]
async fn conditional_connection_removal_preserves_legacy_evict_api() {
let key = "http://conditional-eviction-test.invalid";
cache_connection(
key.to_owned(),
tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy(),
)
.await;
assert!(remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
assert!(!remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
let _: () = evict_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await;
}
#[tokio::test]
async fn cached_connection_keeps_channel_and_transport_state_coherent() {
let key = "http://tracked-connection-test.invalid";
let channel = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
let state = GrpcConnectionState::new();
assert!(state.snapshot().is_none());
let first = state.record_connected();
cache_connection_with_state(key.to_owned(), channel, state.clone()).await;
let (_, cached_state) = cached_connection_with_state(key).await.expect("tracked cache entry");
let cached_state = cached_state.expect("tracked cache state");
assert_eq!(cached_state.snapshot(), Some(first));
let second = state.record_connected();
assert_eq!(cached_state.snapshot(), Some(second));
assert_eq!(first.channel_id, second.channel_id);
assert_ne!(first.generation, second.generation);
cache_connection(
key.to_owned(),
tonic::transport::Endpoint::from_static("http://127.0.0.1:2").connect_lazy(),
)
.await;
let (_, replacement_state) = cached_connection_with_state(key).await.expect("legacy replacement entry");
assert!(replacement_state.is_none(), "legacy replacement must remove stale tracked state");
assert!(remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
assert!(cached_connection_with_state(key).await.is_none());
}
}
+3 -4
View File
@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,9 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
+13
View File
@@ -121,6 +121,11 @@ when restoring `/rustfs/console` for a custom OEM build.
- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES` (canonical)
- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical)
- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical)
- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical)
@@ -130,6 +135,12 @@ Scanner cycle budget controls:
- When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance.
- An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`.
Scanner bucket quantum controls:
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES`, `RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES`, and `RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES` cap one bucket admission. Their persistent keys are `scanner.bucket_quantum_duration`, `scanner.bucket_quantum_objects`, `scanner.bucket_quantum_directories`, `scanner.bucket_quantum_entries`, and `scanner.bucket_quantum_name_bytes`; duration is in whole seconds and the remaining keys are work counts or bytes.
- All five defaults are `0` for controlled rollout. A positive value enables that dimension; `0` disables that dimension, and all five zero values disable the bucket quantum. Parent cycle limits remain in force. Invalid unsigned integers and durations outside the platform timer range are rejected; environment overrides take precedence but cannot hide invalid persisted values. Admin scanner status reports each effective value and source under `runtime_config.bucket_quantum_*`, using `bucket_quantum_duration_seconds` for duration.
- Disabled defaults are an initial rollout state, not completion of usage-convergence acceptance. Production defaults require matched-topology A/B measurements and the remote progress/fairness checks in [Scanner runtime controls](../../docs/operations/scanner-runtime-controls.md#bucket-quantum-rollout-and-acceptance). These controls do not certify partial usage or acknowledge pending dirty work.
Scanner recovery retry controls:
- `RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS` caps the wait between automatic startup recovery retries. The persistent admin key is `scanner.recovery_backoff_cap`; the environment takes precedence over the persistent value, and the unset default remains 600 seconds (10 minutes). Values must be positive integer seconds within the platform timer range; `0`, malformed values, and overflow are rejected. A valid environment override does not hide an invalid persistent value from `admin config set` validation.
@@ -291,6 +302,8 @@ Drive timeout profile preset:
RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them.
A local POSIX filesystem bind-mounted into a gVisor sandbox with directfs is a supported storage path. gVisor presents that bind mount as `9p`/`V9FS`, so the host filesystem type is not visible; the startup guard recognizes mountinfo options `trans=fd` and `directfs` and does not treat the mount as unsupported. Genuine 9p, virtio, and NFS shares remain unsupported.
## 📄 License
This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
+2 -2
View File
@@ -35,8 +35,8 @@ pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
/// Additional time beyond the configured object-lock acquisition deadline.
pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
/// Timeout for cluster health readiness collectors (milliseconds).
/// This bounds expensive storage and lock quorum checks used by cluster probes.
/// End-to-end budget for cluster health readiness probes (milliseconds).
/// Includes cache/singleflight waiting, storage observations, and lock checks.
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
+145 -2
View File
@@ -121,9 +121,17 @@ pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
/// Permit current per-peer capability proof to replace fleet-wide confirmation.
///
/// This remains ineffective unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY`] is also enabled. Unknown,
/// unsupported, revoked, or stale connection generations keep dual-writing JSON.
pub const ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: bool = false;
// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY);
/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
@@ -294,6 +302,14 @@ pub const DEFAULT_INTERNODE_BULK_CONCURRENCY_LIMIT: usize = 64;
pub const ENV_INTERNODE_BULK_BUFFER_SIZE: &str = "RUSTFS_INTERNODE_BULK_BUFFER_SIZE";
pub const DEFAULT_INTERNODE_BULK_BUFFER_SIZE: usize = 1024;
/// Per-peer limit held through complete bulk-unary response decoding and validation. Zero keeps
/// the compatibility behavior with no additional admission gate.
pub const ENV_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT: &str = "RUSTFS_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT";
pub const DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT: usize = 0;
pub const ENV_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS: &str = "RUSTFS_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS";
pub const DEFAULT_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS: u64 = 1_000;
const _: () = assert!(DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT == 0);
/// Profile selector for conservative internode HTTP data-plane client tuning.
pub const ENV_INTERNODE_HTTP_TUNING_PROFILE: &str = "RUSTFS_INTERNODE_HTTP_TUNING_PROFILE";
pub const DEFAULT_INTERNODE_HTTP_TUNING_PROFILE: &str = "legacy";
@@ -333,8 +349,130 @@ pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
/// Legacy alias for "tcp-http". Both values select the TCP/HTTP transport backend.
pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
/// Internode reads whose data moves over NIXL/UCX (InfiniBand rails or RoCE by
/// LID/GID, no IP needed); requests, writes and walks stay on TCP/HTTP. Requires
/// a binary built with the `nixl` feature and `RUSTFS_NIXL_ENABLE`; until the
/// NIXL agent is up, and for any read it cannot serve, it uses TCP/HTTP.
pub const INTERNODE_DATA_TRANSPORT_NIXL: &str = "nixl";
/// Known internode transport backend names accepted by the config parser.
pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP];
pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[
DEFAULT_INTERNODE_DATA_TRANSPORT,
INTERNODE_DATA_TRANSPORT_TCP,
INTERNODE_DATA_TRANSPORT_NIXL,
];
/// Check the configured RDMA devices at startup and leave out any that is down,
/// slower than its peers, or loses its link during a short watch.
pub const ENV_RUSTFS_NIXL_DEVICE_CHECK: &str = "RUSTFS_NIXL_DEVICE_CHECK";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_CHECK: bool = true;
/// Leave out a device whose link dropped this many times more since boot than
/// the quietest device of its link layer; 0 skips this test.
pub const ENV_RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS: &str = "RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS: u64 = 16;
/// How long the startup device check watches the link counters; 0 skips the watch.
pub const ENV_RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS: &str = "RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS: u64 = 3_000;
/// NIXL internode reads in flight per remote shard stream.
pub const ENV_RUSTFS_NIXL_INTERNODE_DEPTH: &str = "RUSTFS_NIXL_INTERNODE_DEPTH";
pub const DEFAULT_RUSTFS_NIXL_INTERNODE_DEPTH: usize = 4;
/// Whether S3 GETs and PUTs carrying the NIXL headers are served over NIXL/UCX
/// S3-over-RDMA (requires a build with the `nixl` feature and libnixl). The
/// pipeline settings below (`RUSTFS_CUOBJECT_GET_FANOUT*`, `_INFLIGHT_WRITES`,
/// `_SHARD_ARENA_BYTES`) apply to this transport as well.
pub const ENV_RUSTFS_NIXL_ENABLE: &str = "RUSTFS_NIXL_ENABLE";
pub const DEFAULT_RUSTFS_NIXL_ENABLE: bool = false;
/// This node's NIXL agent name; defaults to `rustfs-<first local endpoint host>`.
pub const ENV_RUSTFS_NIXL_AGENT_NAME: &str = "RUSTFS_NIXL_AGENT_NAME";
/// Comma-separated UCX device names (`mlx5_9,mlx5_10`; NIXL adds the port); empty lets UCX use every device.
pub const ENV_RUSTFS_NIXL_UCX_DEVICES: &str = "RUSTFS_NIXL_UCX_DEVICES";
/// UCX progress workers of the agent.
pub const ENV_RUSTFS_NIXL_NUM_WORKERS: &str = "RUSTFS_NIXL_NUM_WORKERS";
pub const DEFAULT_RUSTFS_NIXL_NUM_WORKERS: u32 = 1;
/// Staging chunk size per transfer descriptor to a client.
pub const ENV_RUSTFS_NIXL_CHUNK_BYTES: &str = "RUSTFS_NIXL_CHUNK_BYTES";
pub const DEFAULT_RUSTFS_NIXL_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Registered staging memory of the agent.
pub const ENV_RUSTFS_NIXL_POOL_BYTES: &str = "RUSTFS_NIXL_POOL_BYTES";
pub const DEFAULT_RUSTFS_NIXL_POOL_BYTES: usize = 1024 * 1024 * 1024;
/// Transfers in flight per node at most.
pub const ENV_RUSTFS_NIXL_MAX_INFLIGHT: &str = "RUSTFS_NIXL_MAX_INFLIGHT";
pub const DEFAULT_RUSTFS_NIXL_MAX_INFLIGHT: usize = 256;
/// A transfer that has not completed after this many seconds fails the request.
pub const ENV_RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS: &str = "RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS";
pub const DEFAULT_RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS: u64 = 60;
/// Whether S3 GETs carrying an `x-amz-rdma-token` are served over cuObject S3-over-RDMA
/// (requires a build with the `cuobject` feature and libcuobjserver).
pub const ENV_RUSTFS_CUOBJECT_ENABLE: &str = "RUSTFS_CUOBJECT_ENABLE";
pub const DEFAULT_RUSTFS_CUOBJECT_ENABLE: bool = false;
/// Comma-separated RDMA-capable IP addresses that get a cuObject server each.
/// Defaults to the hosts of this node's own endpoints.
pub const ENV_RUSTFS_CUOBJECT_ADDRS: &str = "RUSTFS_CUOBJECT_ADDRS";
/// UDP/RDMA control port of the cuObject servers.
pub const ENV_RUSTFS_CUOBJECT_PORT: &str = "RUSTFS_CUOBJECT_PORT";
pub const DEFAULT_RUSTFS_CUOBJECT_PORT: u16 = 18515;
/// Staging chunk size per RDMA write to a client.
pub const ENV_RUSTFS_CUOBJECT_CHUNK_BYTES: &str = "RUSTFS_CUOBJECT_CHUNK_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Registered staging memory per cuObject server.
pub const ENV_RUSTFS_CUOBJECT_POOL_BYTES: &str = "RUSTFS_CUOBJECT_POOL_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_POOL_BYTES: usize = 1024 * 1024 * 1024;
/// Dynamically connected initiators per cuObject server (concurrent client connections).
/// Staging chunks (RDMA writes) kept in flight per GET, so decoding never waits
/// for the write of the previous chunk; each costs one chunk of the pool.
pub const ENV_RUSTFS_CUOBJECT_INFLIGHT_WRITES: &str = "RUSTFS_CUOBJECT_INFLIGHT_WRITES";
pub const DEFAULT_RUSTFS_CUOBJECT_INFLIGHT_WRITES: usize = 3;
/// Staging chunks (RDMA reads from the client buffer) kept in flight per
/// RDMA PUT body, so ingest never waits on one chunk's round trip at a time;
/// each costs one chunk of the pool.
pub const ENV_RUSTFS_CUOBJECT_PUT_READ_AHEAD: &str = "RUSTFS_CUOBJECT_PUT_READ_AHEAD";
pub const DEFAULT_RUSTFS_CUOBJECT_PUT_READ_AHEAD: usize = 4;
/// Concurrent erasure-block-aligned segments one cuObject GET is split into
/// on the server (each segment decodes and RDMA-writes its own stripes), so a
/// single stream keeps several stripes in flight. 1 disables the fan-out.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT: &str = "RUSTFS_CUOBJECT_GET_FANOUT";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT: usize = 8;
/// Smallest segment worth its own producer (rounded up to whole erasure
/// blocks): each segment pays a reader setup per part it touches, so short
/// requests stay sequential.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES: &str = "RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES: u64 = 192 * 1024 * 1024;
/// Segment producers a node runs across all its cuObject GETs: each new GET
/// gets a fair share (budget / open GETs, at least 1, at most the fan-out)
/// and never waits for one, so many-stream load degrades to sequential GETs
/// instead of a reader-setup storm.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT_BUDGET: &str = "RUSTFS_CUOBJECT_GET_FANOUT_BUDGET";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT_BUDGET: usize = 16;
/// Registered host memory the erasure decoder reads shards into so a
/// cuObject GET can RDMA-write data shards straight from those buffers
/// (zero-copy emit); 0 keeps the staging copy. Blocks are
/// `RUSTFS_ERASURE_SHARD_BYTES` (4 MiB when unset); objects with larger
/// shards fall back to heap buffers and the staging copy.
pub const ENV_RUSTFS_CUOBJECT_SHARD_ARENA_BYTES: &str = "RUSTFS_CUOBJECT_SHARD_ARENA_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_SHARD_ARENA_BYTES: usize = 0;
pub const ENV_RUSTFS_CUOBJECT_NUM_DCIS: &str = "RUSTFS_CUOBJECT_NUM_DCIS";
pub const DEFAULT_RUSTFS_CUOBJECT_NUM_DCIS: u32 = 128;
/// Complete cuObject transfers through the library's asynchronous mode: the
/// write is submitted with a handle and the task waits on the channel's
/// completion descriptor instead of holding a blocking thread for the whole
/// transfer. `false` keeps one blocking library call per transfer.
pub const ENV_RUSTFS_CUOBJECT_ASYNC_TRANSFERS: &str = "RUSTFS_CUOBJECT_ASYNC_TRANSFERS";
pub const DEFAULT_RUSTFS_CUOBJECT_ASYNC_TRANSFERS: bool = true;
#[cfg(test)]
mod tests {
@@ -379,8 +517,12 @@ mod tests {
fn internode_channel_isolation_defaults_and_env_names() {
// The isolation-off default is asserted at compile time next to the constant definition.
assert_eq!(DEFAULT_INTERNODE_BULK_CHANNELS, 2);
assert_eq!(DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT, 0);
assert_eq!(DEFAULT_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS, 1_000);
assert_eq!(ENV_INTERNODE_CHANNEL_ISOLATION, "RUSTFS_INTERNODE_CHANNEL_ISOLATION");
assert_eq!(ENV_INTERNODE_BULK_CHANNELS, "RUSTFS_INTERNODE_BULK_CHANNELS");
assert_eq!(ENV_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT, "RUSTFS_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT");
assert_eq!(ENV_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS, "RUSTFS_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS");
}
#[test]
@@ -391,6 +533,7 @@ mod tests {
ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
"RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
);
assert_eq!(ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY, "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY");
}
#[test]
@@ -465,6 +608,6 @@ mod tests {
assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
assert_eq!(INTERNODE_DATA_TRANSPORT_TCP, "tcp");
assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp"]);
assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp", "nixl"]);
}
}
+82
View File
@@ -117,6 +117,30 @@ pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITR
/// Default: bitrot verification is enabled on GetObject reads (do not skip).
pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
/// Comma-separated buckets that hold a content-addressed cache (an LLM KV
/// cache) and serve PutObject, GetObject and HeadObject without distributed
/// locks or authoritative bucket-metadata reads. See the ecstore
/// `bucket::lockless_cache` module for what such a bucket gives up.
///
/// Default is empty: every bucket keeps the locked paths.
pub const ENV_LOCKLESS_CACHE_BUCKETS: &str = "RUSTFS_LOCKLESS_CACHE_BUCKETS";
/// Smallest declared body, in bytes, whose ETag MD5 is computed on a
/// blocking-pool thread while the body streams into the erasure encoder,
/// instead of inline on the request's task. The ETag and Content-MD5
/// checks are unchanged; `0` keeps every MD5 inline.
///
/// Each offloaded upload holds one blocking-pool thread until its body ends,
/// and that pool also serves disk I/O, so size the threshold against the
/// expected number of concurrent large uploads before enabling it.
pub const ENV_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES: &str = "RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES";
/// Default: disabled; every ETag MD5 is computed inline. Set
/// `RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES` (for example `8388608`) to opt in.
pub const DEFAULT_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES: u64 = 0;
const _: () = assert!(DEFAULT_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES == 0);
/// Create independent shard commitments for new writes after the fleet is upgraded.
/// Existing protected objects and multipart uploads retain their protection.
pub const ENV_SHARD_INTEGRITY_WRITE: &str = "RUSTFS_SHARD_INTEGRITY_WRITE";
@@ -128,6 +152,33 @@ pub const DEFAULT_SHARD_INTEGRITY_WRITE: bool = false;
pub const ENV_SHARD_INTEGRITY_FLEET_CONFIRMED: &str = "RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED";
pub const DEFAULT_SHARD_INTEGRITY_FLEET_CONFIRMED: bool = false;
/// Request parsing switch for the per-object erasure-code block-size hint.
/// The default keeps the feature inert during rolling upgrades.
pub const ENV_EC_BLOCK_SIZE_HINT_ENABLE: &str = "RUSTFS_EC_BLOCK_SIZE_HINT_ENABLE";
pub const DEFAULT_EC_BLOCK_SIZE_HINT_ENABLE: bool = false;
/// Operator confirmation that every writer and reader in the fleet understands
/// the persisted erasure block-size geometry.
pub const ENV_EC_BLOCK_SIZE_HINT_FLEET_CONFIRMED: &str = "RUSTFS_EC_BLOCK_SIZE_HINT_FLEET_CONFIRMED";
pub const DEFAULT_EC_BLOCK_SIZE_HINT_FLEET_CONFIRMED: bool = false;
/// Write new EC shards with a bounded bitrot frame size so Range GETs do not
/// inherit the full persisted layout block as their physical read quantum.
/// The marker is only emitted after the fleet confirmation gate is enabled;
/// readers without this feature continue to consume legacy objects unchanged.
pub const ENV_EC_READ_QUANTUM_ENABLE: &str = "RUSTFS_EC_READ_QUANTUM_ENABLE";
pub const DEFAULT_EC_READ_QUANTUM_ENABLE: bool = false;
/// Operator confirmation that every reader and writer understands the
/// persisted EC read-quantum marker and its per-frame bitrot layout.
pub const ENV_EC_READ_QUANTUM_FLEET_CONFIRMED: &str = "RUSTFS_EC_READ_QUANTUM_FLEET_CONFIRMED";
pub const DEFAULT_EC_READ_QUANTUM_FLEET_CONFIRMED: bool = false;
const _: () = assert!(!DEFAULT_EC_BLOCK_SIZE_HINT_ENABLE);
const _: () = assert!(!DEFAULT_EC_BLOCK_SIZE_HINT_FLEET_CONFIRMED);
const _: () = assert!(!DEFAULT_EC_READ_QUANTUM_ENABLE);
const _: () = assert!(!DEFAULT_EC_READ_QUANTUM_FLEET_CONFIRMED);
const _: () = assert!(!DEFAULT_SHARD_INTEGRITY_WRITE);
const _: () = assert!(!DEFAULT_SHARD_INTEGRITY_FLEET_CONFIRMED);
@@ -298,6 +349,37 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
/// Default disk read timeout in seconds.
pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
/// Environment variable for the smallest bitrot frame (hash + shard bytes)
/// that is verified on a blocking thread instead of inline on the decode task
/// (`0` disables off-thread verification).
///
/// The shard reads of one stripe complete on one task, so with large shards
/// their hashes were computed one after another on one core: at 4 MiB shards
/// a 12+4 stripe spent about 10 ms hashing serially. Frames at or above this
/// size are hashed on the blocking pool so the shards of a stripe verify in
/// parallel; small frames stay inline because a thread hop would cost more
/// than the hash.
pub const ENV_BITROT_VERIFY_OFFLOAD_MIN_BYTES: &str = "RUSTFS_BITROT_VERIFY_OFFLOAD_MIN_BYTES";
/// Default smallest off-thread bitrot frame: 2 MiB (frames of the legacy 1 MiB
/// block, and 1 MiB shards, stay inline).
pub const DEFAULT_BITROT_VERIFY_OFFLOAD_MIN_BYTES: usize = 2 * 1024 * 1024;
/// Environment variable for the erasure shard size, in bytes, of newly written
/// objects (`0`, the default, keeps the historical 1 MiB erasure block).
///
/// When set, the erasure block of a new object is `data shards x shard bytes`,
/// so every shard read and write is exactly this large and aligned: 4 MiB on a
/// 12+4 layout gives a 48 MiB block, 48x fewer stripes per object than today.
/// Existing objects keep the block recorded in their `xl.meta`. System
/// metadata under `.rustfs.sys` always uses the 1 MiB block. The value must be
/// a multiple of 4 KiB and at most `MAX_ERASURE_SHARD_BYTES`; invalid values
/// are ignored with a warning.
pub const ENV_ERASURE_SHARD_BYTES: &str = "RUSTFS_ERASURE_SHARD_BYTES";
/// Default erasure shard size override (`0` = derive from the 1 MiB block).
pub const DEFAULT_ERASURE_SHARD_BYTES: usize = 0;
/// Largest accepted erasure shard size override (64 MiB).
pub const MAX_ERASURE_SHARD_BYTES: usize = 64 * 1024 * 1024;
/// Environment variable for the per-shard erasure write stall timeout (seconds).
///
/// A single shard write (or shard-writer shutdown) that makes no forward
+16 -1
View File
@@ -48,7 +48,7 @@ pub const ENV_RUNTIME_DIAL9_S3_PREFIX: &str = "RUSTFS_RUNTIME_DIAL9_S3_PREFIX";
pub const DEFAULT_WORKER_THREADS: usize = 16;
pub const DEFAULT_MAX_BLOCKING_THREADS: usize = 1024;
pub const DEFAULT_THREAD_PRINT_ENABLED: bool = false;
pub const DEFAULT_THREAD_STACK_SIZE: usize = MI_B; // 1 MiB
pub const DEFAULT_THREAD_STACK_SIZE: usize = 2 * MI_B; // 2 MiB
pub const DEFAULT_THREAD_KEEP_ALIVE: u64 = 60; // seconds
pub const DEFAULT_GLOBAL_QUEUE_INTERVAL: u32 = 31;
pub const DEFAULT_THREAD_NAME: &str = "rustfs-worker";
@@ -114,6 +114,14 @@ pub const ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: &str = "RUSTFS_OBJECT_FILE_
pub const ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD";
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = true;
/// Bytes of page-cache read-ahead a ranged file stream keeps queued in front
/// of its read position (`posix_fadvise(WILLNEED)`, Linux only; `0` disables).
/// A shard stream served to a peer reads 4 MiB chunks one after another; the
/// kernel's own read-ahead window ramps up slowly for large reads, so without
/// this each chunk waits for its own device round trips.
pub const ENV_OBJECT_STREAM_READ_AHEAD_BYTES: &str = "RUSTFS_OBJECT_STREAM_READ_AHEAD_BYTES";
pub const DEFAULT_OBJECT_STREAM_READ_AHEAD_BYTES: usize = 16 * 1024 * 1024;
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
/// Threshold for small object seek support in bytes.
@@ -158,3 +166,10 @@ pub const ENV_OBJECT_DATA_CACHE_MIN_FREE_MEMORY_PERCENT: &str = "RUSTFS_OBJECT_D
pub const ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_PER_CPU: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_CONCURRENCY_PER_CPU";
pub const ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_MAX: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_CONCURRENCY_MAX";
pub const ENV_OBJECT_DATA_CACHE_IDENTITY_KEYS_MAX: &str = "RUSTFS_OBJECT_DATA_CACHE_IDENTITY_KEYS_MAX";
/// Fill the object data cache from the PUT body as it streams in, so the first
/// GET after a write is a hit (write-once KV-cache chunks). Only objects that
/// fit the cache entry limit and take the plain streaming PUT path (no
/// compression, no SSE) are filled; the body is kept in memory once more
/// during the PUT.
pub const ENV_OBJECT_DATA_CACHE_FILL_ON_PUT: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_ON_PUT";
pub const DEFAULT_OBJECT_DATA_CACHE_FILL_ON_PUT: bool = false;
+50
View File
@@ -44,6 +44,21 @@ pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
/// Scanner config key capping directories entered by one cycle.
pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
/// Per-bucket admission limit in seconds; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_DURATION: &str = "bucket_quantum_duration";
/// Per-bucket admission limit in objects; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_OBJECTS: &str = "bucket_quantum_objects";
/// Per-bucket admission limit in directories; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_DIRECTORIES: &str = "bucket_quantum_directories";
/// Per-bucket admission limit in raw entries; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_ENTRIES: &str = "bucket_quantum_entries";
/// Per-bucket admission limit in raw entry-name bytes; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_NAME_BYTES: &str = "bucket_quantum_name_bytes";
/// Scanner config key setting the periodic bitrot scan cycle in seconds.
pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
@@ -93,6 +108,11 @@ pub const SCANNER_KEYS: &[&str] = &[
SCANNER_CYCLE_MAX_DURATION,
SCANNER_CYCLE_MAX_OBJECTS,
SCANNER_CYCLE_MAX_DIRECTORIES,
SCANNER_BUCKET_QUANTUM_DURATION,
SCANNER_BUCKET_QUANTUM_OBJECTS,
SCANNER_BUCKET_QUANTUM_DIRECTORIES,
SCANNER_BUCKET_QUANTUM_ENTRIES,
SCANNER_BUCKET_QUANTUM_NAME_BYTES,
SCANNER_BITROT_CYCLE,
SCANNER_IDLE_MODE,
SCANNER_CACHE_SAVE_TIMEOUT,
@@ -141,6 +161,36 @@ pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECT
/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
/// Per-bucket admission limit in seconds; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_DURATION_SECS: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_DURATION_SECS: u64 = 0;
/// Per-bucket admission limit in objects; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_OBJECTS: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_OBJECTS: u64 = 0;
/// Per-bucket admission limit in directories; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_DIRECTORIES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_DIRECTORIES: u64 = 0;
/// Per-bucket admission limit in raw entries; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_ENTRIES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_ENTRIES: u64 = 0;
/// Per-bucket admission limit in raw entry-name bytes; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_NAME_BYTES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_NAME_BYTES: u64 = 0;
/// Environment variable that selects the scanner speed preset.
/// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
/// Controls the sleep factor, maximum sleep duration, and cycle interval.
+4
View File
@@ -55,6 +55,9 @@ pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRIC
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
/// Path of the sampled GetObject stage summary file (defaults to `/var/log/rustfs/get-stage-summary.log`).
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_FILE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_FILE";
pub const DEFAULT_OBS_GET_STAGE_LOCAL_SUMMARY_FILE: &str = "/var/log/rustfs/get-stage-summary.log";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
@@ -178,6 +181,7 @@ mod tests {
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
);
assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_FILE, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_FILE");
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
+3 -4
View File
@@ -26,12 +26,11 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
+4 -5
View File
@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -37,7 +36,7 @@ jsonwebtoken = { workspace = true, features = ["aws_lc_rs"] }
base64-simd = { workspace = true }
pbkdf2 = { workspace = true, optional = true }
rand = { workspace = true, features = ["serde"] }
rsa = { workspace = true, features = ["sha2"] }
rsa = { workspace = true }
serde = { workspace = true, features = ["derive"] }
sha2 = { workspace = true, optional = true }
thiserror.workspace = true
@@ -50,9 +49,9 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
fips = []
crypto = [
"dep:aes-gcm",
+2 -5
View File
@@ -18,9 +18,6 @@ use crate::Error;
use crate::jwt::Claims;
pub fn decode(token: &str, token_secret: &[u8]) -> Result<TokenData<Claims>, Error> {
Ok(jsonwebtoken::decode(
token,
&DecodingKey::from_secret(token_secret),
&Validation::new(Algorithm::HS512),
)?)
let key = DecodingKey::from_secret(token_secret)?;
Ok(jsonwebtoken::decode(token, &key, &Validation::new(Algorithm::HS512))?)
}
+2 -5
View File
@@ -18,9 +18,6 @@ use crate::Error;
use crate::jwt::Claims;
pub fn encode(token_secret: &[u8], claims: &Claims) -> Result<String, Error> {
Ok(jsonwebtoken::encode(
&Header::new(Algorithm::HS512),
claims,
&EncodingKey::from_secret(token_secret),
)?)
let key = EncodingKey::from_secret(token_secret)?;
Ok(jsonwebtoken::encode(&Header::new(Algorithm::HS512), claims, &key)?)
}
+16 -16
View File
@@ -13,9 +13,9 @@
// limitations under the License.
use rsa::{
Pkcs1v15Encrypt, RsaPrivateKey, RsaPublicKey,
Pkcs1v15Encrypt, PrivateKey, PublicKey,
pkcs8::{DecodePrivateKey, DecodePublicKey},
pss::{BlindedSigningKey, Signature, VerifyingKey},
pss::{Signature, SigningKey, VerifyingKey},
sha2::Sha256,
signature::{RandomizedSigner, Verifier},
traits::PublicKeyParts,
@@ -37,7 +37,7 @@ pub struct Token {
pub fn gencode(token: &Token, key: &str) -> Result<String> {
let data = serde_json::to_vec(token)?;
let mut rng = rand::rng();
let public_key = RsaPublicKey::from_public_key_pem(key).map_err(Error::other)?;
let public_key = PublicKey::from_public_key_pem(key).map_err(Error::other)?;
let encrypted_data = public_key.encrypt(&mut rng, Pkcs1v15Encrypt, &data).map_err(Error::other)?;
Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&encrypted_data))
}
@@ -51,7 +51,7 @@ pub fn parse(token: &str, key: &str) -> Result<Token> {
let encrypted_data = base64_simd::URL_SAFE_NO_PAD
.decode_to_vec(token.as_bytes())
.map_err(Error::other)?;
let private_key = RsaPrivateKey::from_pkcs8_pem(key).map_err(Error::other)?;
let private_key = PrivateKey::from_pkcs8_pem(key).map_err(Error::other)?;
let decrypted_data = private_key.decrypt(Pkcs1v15Encrypt, &encrypted_data).map_err(Error::other)?;
serde_json::from_slice(&decrypted_data).map_err(Error::other)
}
@@ -63,8 +63,8 @@ pub fn parse(token: &str, key: &str) -> Result<Token> {
pub fn sign_license_token(token: &Token, private_key_pem: &str) -> Result<String> {
let payload = serde_json::to_vec(token)?;
let mut rng = rand::rng();
let private_key = RsaPrivateKey::from_pkcs8_pem(private_key_pem).map_err(Error::other)?;
let signing_key = BlindedSigningKey::<Sha256>::new(private_key);
let private_key = PrivateKey::from_pkcs8_pem(private_key_pem).map_err(Error::other)?;
let signing_key = SigningKey::<Sha256>::new(private_key);
let signature: Signature = signing_key.try_sign_with_rng(&mut rng, &payload).map_err(Error::other)?;
let signature: Box<[u8]> = signature.into();
@@ -80,7 +80,7 @@ pub fn parse_signed_license_token(token: &str, public_key_pem: &str) -> Result<T
let signed_payload = base64_simd::URL_SAFE_NO_PAD
.decode_to_vec(token.as_bytes())
.map_err(Error::other)?;
let public_key = RsaPublicKey::from_public_key_pem(public_key_pem).map_err(Error::other)?;
let public_key = PublicKey::from_public_key_pem(public_key_pem).map_err(Error::other)?;
let signature_len = public_key.size();
if signed_payload.len() <= signature_len {
@@ -103,7 +103,7 @@ pub fn parse_license_with_public_key(license: &str, public_key: &str) -> Result<
mod tests {
use super::*;
use rsa::{
RsaPrivateKey,
PrivateKey,
pkcs8::{EncodePrivateKey, EncodePublicKey, LineEnding},
};
use std::time::{SystemTime, UNIX_EPOCH};
@@ -112,8 +112,8 @@ mod tests {
fn test_sign_license_token_and_parse_signed_license_token() {
let mut rng = rand::rng();
let bits = 2048;
let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
let public_key = RsaPublicKey::from(&private_key);
let private_key = PrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
let public_key = PublicKey::from(&private_key);
let private_key_pem = private_key
.to_pkcs8_pem(LineEnding::LF)
@@ -144,8 +144,8 @@ mod tests {
fn test_legacy_gencode_and_parse_roundtrip() {
let mut rng = rand::rng();
let bits = 2048;
let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
let public_key = RsaPublicKey::from(&private_key);
let private_key = PrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
let public_key = PublicKey::from(&private_key);
let private_key_pem = private_key
.to_pkcs8_pem(LineEnding::LF)
@@ -173,8 +173,8 @@ mod tests {
#[test]
fn test_parse_signed_license_token_rejects_tampered_payload() {
let mut rng = rand::rng();
let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
let public_key = RsaPublicKey::from(&private_key);
let private_key = PrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
let public_key = PublicKey::from(&private_key);
let private_key_pem = private_key
.to_pkcs8_pem(LineEnding::LF)
.expect("failed to encode private key pem");
@@ -206,8 +206,8 @@ mod tests {
#[test]
fn test_parse_signed_license_token_rejects_invalid_token() {
let mut rng = rand::rng();
let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
let public_key = RsaPublicKey::from(&private_key);
let private_key = PrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
let public_key = PublicKey::from(&private_key);
let public_key_pem = public_key
.to_public_key_pem(LineEnding::LF)
.expect("failed to encode public key pem");
+26
View File
@@ -0,0 +1,26 @@
[package]
name = "rustfs-cuobject"
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
homepage.workspace = true
description = "RustFS staging pool and transfers for NVIDIA cuObject S3-over-RDMA (GPUDirect object loading)"
keywords = ["rdma", "cuobject", "gpudirect", "s3"]
categories = ["network-programming"]
documentation = "https://docs.rs/rustfs-cuobject/latest/rustfs_cuobject/"
[lints]
workspace = true
[dependencies]
bytes = { workspace = true, optional = true }
cuobject = { workspace = true, default-features = false, optional = true }
parking_lot = { workspace = true, optional = true }
thiserror = { workspace = true, optional = true }
tokio = { workspace = true, features = ["net", "rt-multi-thread", "sync", "time"], optional = true }
[features]
# Compiles the cuObject server bindings (needs the NVIDIA cuObject SDK).
server = ["dep:bytes", "dep:cuobject", "cuobject/server", "dep:parking_lot", "dep:thiserror", "dep:tokio"]
+775
View File
@@ -0,0 +1,775 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Server side of NVIDIA cuObject S3-over-RDMA.
//!
//! A client that supports the protocol (NVIDIA cuObject, NIXL's object plugin,
//! …) sends an ordinary S3 GET with an `x-amz-rdma-token` header describing a
//! registered client buffer (host or GPU memory). Instead of streaming the
//! body over HTTP, the server RDMA-writes the object bytes into that buffer
//! through NVIDIA's `libcuobjserver` and answers with an empty body plus
//! `x-amz-rdma-reply` and `x-amz-rdma-bytes-transferred`.
//!
//! The library bindings come from the [`cuobject`] crate. This crate owns
//! RustFS's pool of registered, page-aligned host staging buffers: the regular
//! read path fills a staging chunk and
//! [`CuObjectServer::write_to_client_owned`] pushes it.
//!
//! Everything below needs the `server` feature, which compiles the
//! NVIDIA cuObject server bindings; without it the crate is empty, so a
//! workspace build does not need the cuObject SDK.
#![cfg(feature = "server")]
use bytes::Bytes;
use cuobject::{Channel, Completion, RegisteredBuffer, Server, ServerOptions, SgBuffer, SgEntry, Source};
use std::collections::VecDeque;
use std::net::IpAddr;
use std::num::NonZeroUsize;
use std::sync::Arc;
use std::time::Duration;
use thiserror::Error;
use tokio::io::Interest;
use tokio::io::unix::AsyncFd;
use tokio::sync::Semaphore;
/// Header carrying the client's RDMA descriptor on a request.
pub const RDMA_TOKEN_HEADER: &str = cuobject::TOKEN_HEADER;
/// Header carrying the transfer status on the response (HTTP-style code).
pub const RDMA_REPLY_HEADER: &str = cuobject::REPLY_HEADER;
/// Header carrying the payload byte count moved over RDMA.
pub const RDMA_BYTES_TRANSFERRED_HEADER: &str = cuobject::BYTES_TRANSFERRED_HEADER;
/// Reply code for a declined transfer (the client then errors or falls back).
pub const RDMA_REPLY_NOT_IMPLEMENTED: u16 = cuobject::REPLY_NOT_IMPLEMENTED;
/// Largest single `handleGetObject` the library accepts.
pub const MAX_RDMA_OP_BYTES: usize = cuobject::MAX_TRANSFER_BYTES;
/// Default RDMA control port of the cuObject server.
pub const DEFAULT_PORT: u16 = cuobject::DEFAULT_PORT;
/// Most ranges one scatter-gather transfer may carry (the library's `max_sge`).
pub const MAX_SG_RANGES: usize = cuobject::DEFAULT_MAX_SG_ENTRIES;
/// Version of the cuObject server headers the bindings were built against, as
/// (major, minor).
pub fn library_version() -> (u32, u32) {
cuobject::header_version()
}
#[derive(Debug, Error)]
pub enum CuObjectError {
#[error("cuobject {op} failed: {source}")]
Os {
op: &'static str,
#[source]
source: std::io::Error,
},
#[error("cuobject rdma token is malformed: {0}")]
Token(String),
#[error("cuobject rdma token buffer ({buffer} bytes) is smaller than the response ({needed} bytes)")]
TokenTooSmall { buffer: u64, needed: u64 },
#[error("cuobject rejected the request: {0}")]
Invalid(&'static str),
#[error("cuobject transfer failed with verbs status {verbs_status}: {source}")]
Transfer {
verbs_status: i32,
#[source]
source: std::io::Error,
},
#[error("cuobject transfer moved {moved} of {expected} bytes")]
ShortTransfer { moved: u64, expected: u64 },
#[error("cuobject staging pool exhausted")]
PoolExhausted,
#[error("cuobject library offers no completion descriptor for channel {0}")]
AsyncUnavailable(u16),
#[error("cuobject read source failed: {0}")]
Source(#[source] std::io::Error),
}
impl From<cuobject::Error> for CuObjectError {
fn from(error: cuobject::Error) -> Self {
match error {
cuobject::Error::Library { op, source } => Self::Os { op, source },
cuobject::Error::Transfer { verbs_status, source } => Self::Transfer { verbs_status, source },
cuobject::Error::InvalidArgument(reason) => Self::Invalid(reason),
cuobject::Error::InvalidToken(reason) => Self::Token(reason.to_owned()),
cuobject::Error::TokenTooSmall { buffer, needed } => Self::TokenTooSmall { buffer, needed },
cuobject::Error::AsyncUnavailable(channel) => Self::AsyncUnavailable(channel),
other => Self::Os {
op: "cuobject",
source: std::io::Error::other(other),
},
}
}
}
/// Parsed `x-amz-rdma-token`.
///
/// Wire format (cuObject RDMA descriptor protocol): colon-separated hex
/// fields `addr:size:rkey:lid:dctn:gid_present:gid`; only the leading
/// address and size are interpreted here, the library consumes the rest.
#[derive(Debug, Clone)]
pub struct RdmaToken {
inner: cuobject::RdmaToken,
/// Start of the client buffer (a copy of the parsed field).
pub remote_addr: u64,
/// Length of the client buffer (a copy of the parsed field).
pub buffer_len: u64,
}
impl RdmaToken {
pub fn parse(raw: &str) -> Result<Self, CuObjectError> {
let inner = cuobject::RdmaToken::parse(raw)?;
Ok(Self {
remote_addr: inner.remote_addr(),
buffer_len: inner.buffer_len(),
inner,
})
}
pub fn as_str(&self) -> &str {
self.inner.as_str()
}
}
/// Tuning for one server instance.
#[derive(Debug, Clone)]
pub struct CuObjectConfig {
pub port: u16,
pub num_dcis: i32,
pub cq_depth: u32,
/// Size of one staging buffer; one RDMA write moves at most this much.
pub chunk_bytes: usize,
/// Total registered staging memory per server instance.
pub pool_bytes: usize,
/// Complete transfers through the library's asynchronous mode (submit,
/// then wait on the channel's completion descriptor) instead of one
/// blocking library call per transfer.
pub async_transfers: bool,
}
impl Default for CuObjectConfig {
fn default() -> Self {
Self {
port: DEFAULT_PORT,
num_dcis: 128,
cq_depth: 640,
chunk_bytes: 64 * 1024 * 1024,
pool_bytes: 1024 * 1024 * 1024,
async_transfers: true,
}
}
}
struct Staging {
buffer: RegisteredBuffer<Arc<Server>>,
free: parking_lot::Mutex<VecDeque<usize>>,
permits: Semaphore,
chunk_bytes: usize,
}
/// One cuObject server bound to the RDMA device that owns an IP address.
pub struct CuObjectServer {
server: Arc<Server>,
staging: Staging,
addr: IpAddr,
port: u16,
/// One permit per library channel (DC initiator): a transfer waits here
/// instead of failing with `EBUSY` when every channel is in use.
channels: Arc<Semaphore>,
async_transfers: bool,
}
impl std::fmt::Debug for CuObjectServer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("CuObjectServer")
.field("addr", &self.addr)
.field("port", &self.port)
.field("chunk_bytes", &self.staging.chunk_bytes)
.finish()
}
}
/// Several byte ranges of one [`RegisteredArena`], sent in order as one
/// RDMA operation (at most [`MAX_SG_RANGES`]).
pub struct ArenaRanges {
pub arena: Arc<RegisteredArena>,
/// `(local_offset, len)` pairs, in stream order.
pub ranges: Vec<(usize, usize)>,
}
impl ArenaRanges {
pub fn total_len(&self) -> usize {
self.ranges.iter().map(|(_, len)| *len).sum()
}
}
/// A byte range of a [`RegisteredArena`] to send from.
pub struct ArenaRange {
pub arena: Arc<RegisteredArena>,
pub local_offset: usize,
pub len: usize,
}
/// Host memory registered with one server, owned by the caller (see
/// [`CuObjectServer::register_arena`]).
pub struct RegisteredArena {
buffer: RegisteredBuffer<Arc<Server>>,
server: Arc<CuObjectServer>,
id: u64,
}
impl RegisteredArena {
/// Length of the registered memory.
pub fn len(&self) -> usize {
self.buffer.len()
}
pub fn is_empty(&self) -> bool {
self.buffer.is_empty()
}
/// Process-unique id of this registration.
pub fn id(&self) -> u64 {
self.id
}
/// The server this memory is registered with.
pub fn server(&self) -> &Arc<CuObjectServer> {
&self.server
}
/// Raw pointer to the start of the registered memory. The memory lives as
/// long as this arena; callers hand out disjoint ranges of it and never
/// touch a range while a transfer sends it.
pub fn as_mut_ptr(&self) -> *mut u8 {
self.buffer.as_mut_ptr()
}
}
impl std::fmt::Debug for RegisteredArena {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("RegisteredArena")
.field("id", &self.id)
.field("len", &self.len())
.finish()
}
}
/// A reserved staging chunk; returned to the pool on drop.
///
/// The chunk's index is unique while it is reserved, so its range of the
/// staging buffer has one owner. Transfers take the chunk by value, so no
/// Rust borrow of the range overlaps an RDMA operation on it.
pub struct StagingChunk {
server: Arc<CuObjectServer>,
index: usize,
len: usize,
}
impl StagingChunk {
pub fn capacity(&self) -> usize {
self.server.staging.chunk_bytes
}
/// Bytes currently staged.
pub fn len(&self) -> usize {
self.len
}
pub fn is_empty(&self) -> bool {
self.len == 0
}
/// Copies `data` at `self.len()` and advances the staged length.
pub fn append(&mut self, data: &[u8]) -> Result<(), CuObjectError> {
if self.len + data.len() > self.capacity() {
return Err(CuObjectError::Invalid("staging append exceeds chunk"));
}
self.spare_mut()?[..data.len()].copy_from_slice(data);
self.len += data.len();
Ok(())
}
pub fn clear(&mut self) {
self.len = 0;
}
pub fn is_full(&self) -> bool {
self.len >= self.capacity()
}
/// The bytes staged so far.
#[allow(unsafe_code)]
pub fn staged(&self) -> Result<&[u8], CuObjectError> {
// SAFETY: the range lies in this chunk (see the type docs); `&self`
// excludes `spare_mut`, and transfers own the chunk they move.
Ok(unsafe { self.server.staging.buffer.slice(self.start(), self.len)? })
}
/// The unfilled remainder of the chunk, for a reader to fill directly.
#[allow(unsafe_code)]
pub fn spare_mut(&mut self) -> Result<&mut [u8], CuObjectError> {
let len = self.capacity() - self.len;
// SAFETY: the range lies in this chunk (see the type docs) and
// `&mut self` excludes every other borrow of it.
Ok(unsafe { self.server.staging.buffer.slice_mut(self.start() + self.len, len)? })
}
/// Marks `n` more bytes of the spare area as staged.
pub fn advance(&mut self, n: usize) -> Result<(), CuObjectError> {
if self.len + n > self.capacity() {
return Err(CuObjectError::Invalid("staging advance exceeds chunk"));
}
self.len += n;
Ok(())
}
fn start(&self) -> usize {
self.index * self.capacity()
}
}
impl Drop for StagingChunk {
fn drop(&mut self) {
self.server.staging.free.lock().push_back(self.index);
self.server.staging.permits.add_permits(1);
}
}
/// The local side of one GET, owned by the transfer until the DMA finished.
enum Local {
/// A range of the staging buffer, inside a chunk the transfer also owns.
Staging { offset: usize, len: usize },
/// A range of a caller-owned arena.
Arena {
arena: Arc<RegisteredArena>,
offset: usize,
len: usize,
},
/// Several ranges of a caller-owned arena, sent as one operation.
ArenaList {
arena: Arc<RegisteredArena>,
entries: Vec<SgEntry>,
},
}
impl Local {
fn len(&self) -> usize {
match self {
Self::Staging { len, .. } | Self::Arena { len, .. } => *len,
Self::ArenaList { entries, .. } => entries.iter().map(|entry| entry.size).sum(),
}
}
}
/// Why an asynchronous GET did not complete.
enum AsyncFailure {
/// The library has no completion descriptor; nothing was submitted.
Unavailable,
/// The GET failed, or was never submitted; nothing is in flight.
Failed(CuObjectError),
/// Waiting failed after submission: the NIC may still read the memory.
InFlight(CuObjectError),
}
impl From<cuobject::Error> for AsyncFailure {
fn from(error: cuobject::Error) -> Self {
Self::Failed(error.into())
}
}
/// The handle of the one GET in flight on an asynchronous channel.
const ASYNC_HANDLE: NonZeroUsize = NonZeroUsize::MIN;
impl CuObjectServer {
/// Creates the server and its registered staging pool.
pub fn new(addr: IpAddr, config: &CuObjectConfig) -> Result<Arc<Self>, CuObjectError> {
if config.chunk_bytes == 0 || config.chunk_bytes > MAX_RDMA_OP_BYTES || config.pool_bytes < config.chunk_bytes {
return Err(CuObjectError::Invalid("invalid staging configuration"));
}
let mut options = ServerOptions::default();
// A non-positive count keeps the library default.
options.num_dcis = u32::try_from(config.num_dcis).unwrap_or(0);
options.cq_depth = config.cq_depth;
let server = Arc::new(Server::new(addr, config.port, &options)?);
let chunks = config.pool_bytes / config.chunk_bytes;
let buffer = RegisteredBuffer::new(Arc::clone(&server), chunks * config.chunk_bytes)?;
Ok(Arc::new(Self {
channels: Arc::new(Semaphore::new(usize::try_from(config.num_dcis).unwrap_or(1).max(1))),
async_transfers: config.async_transfers,
server,
staging: Staging {
buffer,
free: parking_lot::Mutex::new((0..chunks).collect()),
permits: Semaphore::new(chunks),
chunk_bytes: config.chunk_bytes,
},
addr,
port: config.port,
}))
}
pub fn addr(&self) -> IpAddr {
self.addr
}
pub fn port(&self) -> u16 {
self.port
}
pub fn chunk_bytes(&self) -> usize {
self.staging.chunk_bytes
}
/// Reserves one staging chunk, waiting up to `timeout` for a free one.
pub async fn reserve(self: &Arc<Self>, timeout: Duration) -> Result<StagingChunk, CuObjectError> {
let permit = tokio::time::timeout(timeout, self.staging.permits.acquire())
.await
.map_err(|_| CuObjectError::PoolExhausted)?
.map_err(|_| CuObjectError::PoolExhausted)?;
permit.forget();
let index = self.staging.free.lock().pop_front().ok_or(CuObjectError::PoolExhausted)?;
Ok(StagingChunk {
server: Arc::clone(self),
index,
len: 0,
})
}
/// Reserves one staging chunk without waiting; `None` when the pool is empty.
pub fn try_reserve(self: &Arc<Self>) -> Option<StagingChunk> {
let permit = self.staging.permits.try_acquire().ok()?;
permit.forget();
let index = self.staging.free.lock().pop_front()?;
Some(StagingChunk {
server: Arc::clone(self),
index,
len: 0,
})
}
/// Staging chunks currently free in the pool (a hint; another reservation may win).
pub fn available_chunks(&self) -> usize {
self.staging.permits.available_permits()
}
/// Registers `len` bytes of zero-filled, page-aligned host memory with this
/// server for callers that manage the memory themselves (a shard arena the
/// decoder reads into and the server RDMA-writes from without staging).
pub fn register_arena(self: &Arc<Self>, len: usize) -> Result<Arc<RegisteredArena>, CuObjectError> {
let buffer = RegisteredBuffer::new(Arc::clone(&self.server), len)?;
static NEXT_ID: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
Ok(Arc::new(RegisteredArena {
buffer,
server: Arc::clone(self),
id: NEXT_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed),
}))
}
/// RDMA-writes the staged bytes of `chunk` to `token.remote_addr +
/// remote_offset`. The transfer owns the chunk until the DMA finished, so
/// cancelling the awaiting future can never release the staging slot under
/// an in-flight write. Returns the chunk with the byte count for reuse.
pub async fn write_to_client_owned(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
chunk: StagingChunk,
) -> Result<(u64, StagingChunk), CuObjectError> {
if chunk.is_empty() {
return Ok((0, chunk));
}
let local = Local::Staging {
offset: chunk.start(),
len: chunk.len(),
};
self.get_owned(key, token, remote_offset, local, chunk).await
}
/// RDMA-writes `source.len` bytes at `source.local_offset` of
/// `source.arena` to `token.remote_addr + remote_offset`, keeping `keep`
/// alive until the DMA finished (the caller passes whatever owns the arena
/// bytes). The arena must have been registered with this server.
pub async fn write_to_client_from_arena_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
source: ArenaRange,
keep: T,
) -> Result<(u64, T), CuObjectError> {
if source.len == 0 {
return Ok((0, keep));
}
let local = Local::Arena {
arena: source.arena,
offset: source.local_offset,
len: source.len,
};
self.get_owned(key, token, remote_offset, local, keep).await
}
/// RDMA-writes all of `source.ranges` (in order, as one operation) to
/// `token.remote_addr + remote_offset`, keeping `keep` alive until the DMA
/// finished. The arena must have been registered with this server.
pub async fn write_to_client_sg_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
source: ArenaRanges,
keep: T,
) -> Result<(u64, T), CuObjectError> {
if source.ranges.len() > MAX_SG_RANGES {
return Err(CuObjectError::Invalid("too many scatter-gather ranges"));
}
if source.total_len() == 0 {
return Ok((0, keep));
}
let entries = source
.ranges
.iter()
.map(|&(offset, size)| SgEntry {
offset: offset as u64,
size,
})
.collect();
let local = Local::ArenaList {
arena: source.arena,
entries,
};
self.get_owned(key, token, remote_offset, local, keep).await
}
/// RDMA-reads `len` bytes from `token.remote_addr + remote_offset` into the
/// (empty) `chunk`, owning it for the whole DMA like
/// [`Self::write_to_client_owned`]. On success the chunk holds `len` staged
/// bytes.
pub async fn read_from_client_owned(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
len: usize,
mut chunk: StagingChunk,
) -> Result<StagingChunk, CuObjectError> {
chunk.clear();
if len == 0 {
return Ok(chunk);
}
if len > chunk.capacity() {
return Err(CuObjectError::Invalid("read exceeds the staging chunk"));
}
let server = Arc::clone(self);
let key = key.to_owned();
let token = token.inner.clone();
let channel_permit = self.channel_permit().await?;
tokio::task::spawn_blocking(move || {
let _channel_permit = channel_permit;
let mut channel = Channel::lease(Arc::clone(&server.server))?;
let moved = channel.put(&key, &token, remote_offset, &server.staging.buffer, chunk.start(), len)?;
check_moved(moved as u64, len as u64)?;
chunk.advance(len)?;
Ok(chunk)
})
.await
.map_err(|error| join_error("read task", error))?
}
async fn channel_permit(&self) -> Result<tokio::sync::OwnedSemaphorePermit, CuObjectError> {
Arc::clone(&self.channels)
.acquire_owned()
.await
.map_err(|_| CuObjectError::Invalid("server closed"))
}
/// RDMA-writes `local` to the client and hands `keep` back once the DMA
/// finished. The transfer runs in a task of its own that owns `local`,
/// `keep` and the channel, so dropping this future never releases memory
/// the NIC may still read.
async fn get_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
local: Local,
keep: T,
) -> Result<(u64, T), CuObjectError> {
let expected = local.len() as u64;
let server = Arc::clone(self);
let key = key.to_owned();
let token = token.inner.clone();
let channel_permit = self.channel_permit().await?;
tokio::spawn(async move {
if server.async_transfers {
match server.get_async(&key, &token, remote_offset, &local).await {
Ok(()) => {
drop(channel_permit);
return Ok((expected, keep));
}
Err(AsyncFailure::Unavailable) => {}
Err(AsyncFailure::Failed(error)) => return Err(error),
Err(AsyncFailure::InFlight(error)) => {
// The NIC may still read this memory: never release it.
std::mem::forget(local);
std::mem::forget(keep);
return Err(error);
}
}
}
tokio::task::spawn_blocking(move || {
let _channel_permit = channel_permit;
let mut channel = Channel::lease(Arc::clone(&server.server))?;
let mut list = None;
let source = server.source(&local, &mut list)?;
let moved = channel.get(&key, &token, remote_offset, source)?;
check_moved(moved as u64, expected)?;
// `keep` (the chunk or the arena owner) is released only here.
Ok((expected, keep))
})
.await
.map_err(|error| join_error("write task", error))?
})
.await
.map_err(|error| join_error("write task", error))?
}
/// Moves `local` with the library's asynchronous mode: the GET is
/// submitted on a channel of its own and the task waits on that channel's
/// completion descriptor instead of holding a blocking thread.
// SAFETY: the allowance covers only this function; its two unsafe calls
// each state the invariant they rely on.
#[allow(unsafe_code)]
async fn get_async(
&self,
key: &str,
token: &cuobject::RdmaToken,
remote_offset: u64,
local: &Local,
) -> Result<(), AsyncFailure> {
let mut channel = Channel::lease(Arc::clone(&self.server))?;
let fd = match channel.completion_fd() {
Ok(fd) => fd,
Err(cuobject::Error::AsyncUnavailable(_)) => return Err(AsyncFailure::Unavailable),
Err(error) => return Err(error.into()),
};
// SAFETY: the library owns the descriptor and keeps it open while the
// server lives; `fd` holds the server, and nothing here closes or
// replaces the descriptor while it is registered.
let completions = unsafe { AsyncFd::register_with_interest(fd, Interest::READABLE) }.map_err(|error| {
AsyncFailure::Failed(CuObjectError::Os {
op: "completion fd",
source: error.into_parts().1,
})
})?;
let mut list = None;
let source = self.source(local, &mut list).map_err(AsyncFailure::Failed)?;
// SAFETY: the memory behind `source` belongs to `local`, owned by the
// calling task together with the transfer's `keep`; staging chunks and
// arena ranges given to a transfer are not borrowed by anyone else.
// This function returns only after the completion carrying the handle
// was drained, except through `InFlight`, where the channel and the
// list are leaked here and the caller leaks `local` and `keep`.
unsafe { channel.get_async(key, token, remote_offset, source, ASYNC_HANDLE) }?;
let mut events = [Completion::default(); 8];
loop {
let mut ready = match completions.readable().await {
Ok(ready) => ready,
Err(source) => {
std::mem::forget(list);
std::mem::forget(channel);
return Err(AsyncFailure::InFlight(CuObjectError::Os {
op: "completion wait",
source,
}));
}
};
loop {
let landed = match channel.process_completions(&mut events) {
Ok(landed) => landed,
Err(error) => {
std::mem::forget(list);
std::mem::forget(channel);
return Err(AsyncFailure::InFlight(error.into()));
}
};
if let Some(done) = events[..landed].iter().find(|event| event.handle() == ASYNC_HANDLE.get()) {
// A completion carries no byte count: success moved the whole range.
return done
.result()
.map_err(|source| AsyncFailure::Failed(CuObjectError::Transfer { verbs_status: 0, source }));
}
if landed < events.len() {
break;
}
}
ready.clear_ready();
}
}
/// The library view of `local`; a scatter-gather list is built into `list`.
fn source<'a: 'l, 'l>(
&'a self,
local: &'a Local,
list: &'l mut Option<SgBuffer<'a, Arc<Server>>>,
) -> Result<Source<'l, Arc<Server>>, CuObjectError> {
Ok(match local {
Local::Staging { offset, len } => Source::Range {
buffer: &self.staging.buffer,
offset: *offset,
len: *len,
},
Local::Arena { arena, offset, len } => Source::Range {
buffer: &arena.buffer,
offset: *offset,
len: *len,
},
Local::ArenaList { arena, entries } => Source::List(list.insert(arena.buffer.sg_buffer(entries)?)),
})
}
}
fn check_moved(moved: u64, expected: u64) -> Result<u64, CuObjectError> {
if moved != expected {
return Err(CuObjectError::ShortTransfer { moved, expected });
}
Ok(moved)
}
fn join_error(op: &'static str, error: tokio::task::JoinError) -> CuObjectError {
CuObjectError::Os {
op,
source: std::io::Error::other(error),
}
}
/// Copies owned bytes into staging; convenience for callers holding `Bytes`.
pub fn fill_chunk(chunk: &mut StagingChunk, data: &Bytes) -> Result<(), CuObjectError> {
chunk.append(data)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn token_parses_leading_fields_and_rejects_garbage() {
let token =
RdmaToken::parse("0000f52eddc00000:00100000:001824d2:0000:002873:1:00000000000000000000ffff0a0d5410").expect("token");
assert_eq!(token.remote_addr, 0x0000_f52e_ddc0_0000);
assert_eq!(token.buffer_len, 0x0010_0000);
assert!(matches!(RdmaToken::parse("nothex:00100000:1:2:3:4:5"), Err(CuObjectError::Token(_))));
assert!(RdmaToken::parse("0000f52eddc00000:00000000:1:2:3:4:5").is_err());
assert!(RdmaToken::parse("0000f52eddc00000:00100000").is_err());
}
}
+3
View File
@@ -39,5 +39,8 @@ serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
rustfs-filemeta = { workspace = true }
[dev-dependencies]
serde_json.workspace = true
[lib]
doctest = false
+37 -3
View File
@@ -66,6 +66,12 @@ pub const LEGACY_DATA_USAGE_OBJECT_NAME: &str = ".usage.json";
/// stale tier names from growing persisted per-tier maps without bound.
pub const UNKNOWN_TIER: &str = "UNKNOWN_TIER";
/// Whether a remote tier name can be represented in scanner accounting.
/// Local storage classes are checked separately by the configuration owner.
pub fn is_scannable_tier_name(name: &str) -> bool {
!name.is_empty() && name.len() <= 256 && !name.bytes().any(|byte| byte.is_ascii_control()) && name != UNKNOWN_TIER
}
/// Returns true when `existing_last_update` is ahead of `now` by more than
/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
@@ -474,6 +480,12 @@ pub struct BucketTargetUsageInfo {
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct BucketUsageInfo {
pub size: u64,
/// Provenance of this value. Missing and unknown states remain unproven.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_value_state: Option<crate::BucketUsageValueState>,
/// Certification of this exact value, independent of aggregation time.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_certification: Option<crate::BucketUsageCertification>,
/// Bucket identity certified by every set contributing to this snapshot.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bucket_incarnation: Option<String>,
@@ -580,6 +592,18 @@ pub struct DataUsageInfo {
/// which last-known-good generation contributed to the view.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub usage_snapshot_set_states: Vec<DataUsageSnapshotSetState>,
/// Complete mixed observations for admin display only. Legacy bucket maps
/// and quota consumers continue to use the canonical values above.
#[serde(
default,
skip_serializing_if = "HashMap::is_empty",
deserialize_with = "crate::usage_observation::deserialize_bucket_observations"
)]
pub usage_snapshot_bucket_observations: HashMap<String, crate::DataUsageBucketObservation>,
/// Response-only marker: a projected admin view must never be persisted or
/// admitted as a quota/cache baseline, including after serialization.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub usage_snapshot_admin_projection: bool,
/// Buckets whose dirty-usage generation moved while this complete
/// snapshot was being walked. Their `buckets_usage` values are the walk
/// result, which the cycle could not certify as current; the next cycle
@@ -709,7 +733,8 @@ pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &Da
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
!observed.usage_snapshot_admin_projection
&& observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
@@ -2024,7 +2049,8 @@ impl DataUsageInfo {
/// Whether this snapshot authoritatively covers every reported bucket.
pub fn is_complete_bucket_usage_snapshot(&self) -> bool {
!self.usage_snapshot_bootstrap_pending
!self.usage_snapshot_admin_projection
&& !self.usage_snapshot_bootstrap_pending
&& self.usage_snapshot_complete
&& self.last_update.is_some()
&& u64::try_from(self.buckets_usage.len()).ok() == Some(self.buckets_count)
@@ -2034,7 +2060,8 @@ impl DataUsageInfo {
/// admin display. Partial data is accepted only with unique set states,
/// a plan digest for every state, and at least one usable generation.
pub fn is_valid_partial_snapshot(&self) -> bool {
if self.usage_snapshot_bootstrap_pending
if self.usage_snapshot_admin_projection
|| self.usage_snapshot_bootstrap_pending
|| !self.usage_snapshot_partial
|| self.usage_snapshot_converged != Some(false)
|| self.last_update.is_none()
@@ -2072,9 +2099,11 @@ impl DataUsageInfo {
Ok(name) => name,
Err(_) => return,
};
self.usage_snapshot_bucket_observations.remove(&bucket_name);
// Update bucket statistics
if let Some(bucket_usage) = self.buckets_usage.get_mut(&bucket_name) {
bucket_usage.mark_observed();
bucket_usage.size += meta_object.size as u64;
bucket_usage.objects_count += 1;
bucket_usage.versions_count += 1; // Simplified: assume 1 version per object
@@ -2127,6 +2156,7 @@ impl DataUsageInfo {
Ok(name) => name,
Err(_) => return,
};
self.usage_snapshot_bucket_observations.remove(&bucket_name);
// Calculate accurate statistics from all versions
let mut total_size = 0u64;
@@ -2155,6 +2185,7 @@ impl DataUsageInfo {
// Update bucket statistics
if let Some(bucket_usage) = self.buckets_usage.get_mut(&bucket_name) {
bucket_usage.mark_observed();
bucket_usage.size += total_size;
bucket_usage.objects_count += 1;
bucket_usage.versions_count += versions_count;
@@ -2335,6 +2366,9 @@ impl BucketUsageInfo {
/// Add size summary to this bucket usage
/// Merge another BucketUsageInfo into this one
pub fn merge(&mut self, other: &BucketUsageInfo) {
// Adding contributions produces a new value, not either input's
// whole-bucket certificate. The scanner may certify the final sum.
self.mark_observed();
if self.bucket_incarnation != other.bucket_incarnation {
self.bucket_incarnation = None;
}
+2
View File
@@ -13,5 +13,7 @@
// limitations under the License.
pub mod data_usage;
pub mod usage_observation;
pub use data_usage::*;
pub use usage_observation::*;
+265
View File
@@ -0,0 +1,265 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
//! Diagnostic provenance and response-only projections of complete bucket observations.
use crate::{BucketUsageInfo, DataUsageInfo, DataUsageSnapshotIdentity};
use serde::{Deserialize, Serialize};
use std::{collections::HashMap, time::SystemTime};
pub const DATA_USAGE_BUCKET_OBSERVATION_VERSION: u16 = 1;
/// Provenance of the returned bucket value, independent of quota authority.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum BucketUsageValueState {
Certified,
Carried,
Observed,
#[serde(other)]
Unknown,
}
/// A certification of this exact bucket value, never its publication timestamp.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct BucketUsageCertification {
pub epoch: u64,
pub cycle: u64,
pub certified_at: SystemTime,
}
/// Immutable provenance of one complete traversal contribution.
///
/// The scanner validates its storage-owned sweep identity before constructing
/// this public projection. Equal start/request plans are allowed when a fresh
/// set contributes alongside another set's mixed traversal.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct DataUsageBucketObservationSource {
pub pool_index: u64,
pub set_index: u64,
pub bucket_incarnation: String,
pub scanner_epoch: u64,
pub scanner_cycle: u64,
pub scan_plan_digest: [u8; 32],
pub started_plan: [u8; 32],
pub requested_plan: [u8; 32],
pub completed_at: SystemTime,
}
/// An additive companion; legacy bucket maps never contain this mixed value.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct DataUsageBucketObservation {
pub version: u16,
pub usage: BucketUsageInfo,
/// Exact enclosing candidate identity. Rewrapping preserves source times.
pub snapshot_identity: DataUsageSnapshotIdentity,
/// Oldest contribution time, so a newer set cannot hide an older value.
pub completed_at: SystemTime,
/// Strictly ordered, complete coverage of the enclosing live set inventory.
pub sources: Vec<DataUsageBucketObservationSource>,
}
/// Companion evolution must not make an otherwise usable canonical snapshot
/// unreadable. Unsupported or malformed diagnostic entries never grant proof.
pub(crate) fn deserialize_bucket_observations<'de, D>(
deserializer: D,
) -> Result<HashMap<String, DataUsageBucketObservation>, D::Error>
where
D: serde::Deserializer<'de>,
{
#[derive(Deserialize)]
#[serde(untagged)]
enum Entry {
Observation(Box<DataUsageBucketObservation>),
Ignored(serde::de::IgnoredAny),
}
Ok(HashMap::<String, Entry>::deserialize(deserializer)?
.into_iter()
.filter_map(|(bucket, entry)| match entry {
Entry::Observation(observation) if observation.version == DATA_USAGE_BUCKET_OBSERVATION_VERSION => {
Some((bucket, *observation))
}
Entry::Observation(_) | Entry::Ignored(_) => None,
})
.collect())
}
impl BucketUsageInfo {
/// Attach a certificate only after the caller proves complete bucket coverage.
pub fn certify(&mut self, certification: BucketUsageCertification) {
self.usage_value_state = Some(BucketUsageValueState::Certified);
self.usage_certification = Some(certification);
}
/// Preserve the certificate when carrying the exact prior bucket value.
pub fn mark_carried(&mut self) {
self.normalize_usage_provenance();
self.usage_value_state = Some(BucketUsageValueState::Carried);
}
/// The value changed without a complete bucket certificate.
pub fn mark_observed(&mut self) {
self.usage_value_state = Some(BucketUsageValueState::Observed);
self.usage_certification = None;
}
/// Future or legacy provenance cannot authorize a retained certificate.
pub fn normalize_usage_provenance(&mut self) {
if !matches!(
self.usage_value_state,
Some(BucketUsageValueState::Certified | BucketUsageValueState::Carried)
) {
self.usage_certification = None;
}
if self.usage_value_state == Some(BucketUsageValueState::Certified) && self.usage_certification.is_none() {
self.usage_value_state = Some(BucketUsageValueState::Unknown);
}
}
}
impl DataUsageBucketObservation {
/// Verify the public provenance against the enclosing candidate and current
/// storage-owned incarnation. This never grants publication or ACK authority.
pub fn matches_snapshot(&self, snapshot: &DataUsageInfo, incarnation: &str) -> bool {
let Some(root_generation) = snapshot.scanner_epoch.zip(snapshot.scanner_cycle) else {
return false;
};
if self.version != DATA_USAGE_BUCKET_OBSERVATION_VERSION
|| snapshot.usage_snapshot_admin_projection
|| self.snapshot_identity != snapshot.snapshot_identity()
|| incarnation.is_empty()
|| self.usage.bucket_incarnation.as_deref() != Some(incarnation)
|| self.sources.is_empty()
|| self.usage.usage_certification.is_some()
|| self.usage.usage_value_state != Some(BucketUsageValueState::Observed)
|| self.sources.iter().map(|source| source.completed_at).min() != Some(self.completed_at)
{
return false;
}
let mut previous = None;
let mut plan = None;
for state in &snapshot.usage_snapshot_set_states {
let key = (state.pool_index, state.set_index);
if previous.is_some_and(|previous| previous >= key)
|| state.scan_plan_digest.is_none()
|| plan.is_some_and(|plan| Some(plan) != state.scan_plan_digest)
{
return false;
}
previous = Some(key);
plan = state.scan_plan_digest;
}
let mut sources = self.sources.iter();
for state in snapshot.usage_snapshot_set_states.iter().filter(|state| !state.tombstone) {
let Some(source) = sources.next() else {
return false;
};
let Some(set_generation) = state.scanner_epoch.zip(state.scanner_cycle) else {
return false;
};
if (source.pool_index, source.set_index) != (state.pool_index, state.set_index)
|| source.bucket_incarnation != incarnation
|| Some(source.scan_plan_digest) != state.scan_plan_digest
|| (source.scanner_epoch, source.scanner_cycle) > set_generation
|| set_generation > root_generation
{
return false;
}
}
sources.next().is_none()
}
}
impl DataUsageInfo {
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
/// The projection is marked explicitly so persistence and quota boundaries
/// can reject accidental reuse. Invalid companions leave canonical values
/// unchanged. Counter overflow rejects the whole projection.
pub fn project_bucket_observations_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
if self.usage_snapshot_admin_projection
|| (!self.is_complete_bucket_usage_snapshot() && !self.is_valid_partial_snapshot())
{
return 0;
}
let observations =
self.usage_snapshot_bucket_observations
.iter()
.filter(|(bucket, observation)| {
if self.usage_snapshot_complete
&& (!self.usage_snapshot_pending_buckets.iter().any(|pending| pending == *bucket)
|| u64::try_from(self.usage_snapshot_pending_buckets.len()).ok()
!= Some(self.usage_snapshot_pending_bucket_count))
{
return false;
}
if self
.buckets_usage
.get(*bucket)
.and_then(|usage| usage.usage_certification)
.is_some_and(|certificate| {
observation.sources.iter().all(|source| {
(source.scanner_epoch, source.scanner_cycle) <= (certificate.epoch, certificate.cycle)
})
})
{
return false;
}
incarnations.get(*bucket).is_some_and(|incarnation| {
self.buckets_usage
.get(*bucket)
.and_then(|usage| usage.bucket_incarnation.as_ref())
.is_none_or(|canonical| canonical == incarnation)
&& observation.matches_snapshot(self, incarnation)
})
})
.collect::<HashMap<_, _>>();
if observations.is_empty() {
return 0;
}
let mut values = self
.buckets_usage
.iter()
.filter(|(bucket, _)| !observations.contains_key(bucket))
.map(|(_, usage)| usage)
.chain(observations.values().map(|observation| &observation.usage));
let totals = values.try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let Some([size, objects, versions, markers]) = totals else {
return 0;
};
let projected = observations.len();
for (bucket, observation) in observations {
self.buckets_usage.insert(bucket.clone(), observation.usage.clone());
}
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_observational = true;
self.usage_snapshot_admin_projection = true;
projected
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,323 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use crate::DataUsageSnapshotSetState;
use std::time::Duration;
const INCARNATION: &str = "11111111-1111-4111-8111-111111111111";
fn time(seconds: u64) -> SystemTime {
SystemTime::UNIX_EPOCH + Duration::from_secs(seconds)
}
fn usage(size: u64) -> BucketUsageInfo {
BucketUsageInfo {
size,
objects_count: size,
versions_count: size,
bucket_incarnation: Some(INCARNATION.to_string()),
..Default::default()
}
}
fn snapshot() -> DataUsageInfo {
let mut main = usage(10);
main.certify(BucketUsageCertification {
epoch: 2,
cycle: 3,
certified_at: time(30),
});
main.mark_carried();
let mut snapshot = DataUsageInfo {
last_update: Some(time(90)),
scanner_epoch: Some(2),
scanner_cycle: Some(9),
buckets_count: 1,
buckets_usage: HashMap::from([("bucket".to_string(), main)]),
bucket_sizes: HashMap::from([("bucket".to_string(), 10)]),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_pending_buckets: vec!["bucket".to_string()],
usage_snapshot_pending_bucket_count: 1,
usage_snapshot_set_states: (0..2)
.map(|set_index| DataUsageSnapshotSetState {
pool_index: 0,
set_index,
scanner_epoch: Some(2),
scanner_cycle: Some(9),
scan_plan_digest: Some([1; 32]),
complete: true,
..Default::default()
})
.collect(),
..Default::default()
};
snapshot.calculate_totals();
let mut observed = usage(25);
observed.mark_observed();
let observation = DataUsageBucketObservation {
version: DATA_USAGE_BUCKET_OBSERVATION_VERSION,
usage: observed,
snapshot_identity: snapshot.snapshot_identity(),
completed_at: time(40),
sources: (0..2)
.map(|set_index| DataUsageBucketObservationSource {
pool_index: 0,
set_index,
bucket_incarnation: INCARNATION.to_string(),
scanner_epoch: 2,
scanner_cycle: 4 + set_index,
scan_plan_digest: [1; 32],
started_plan: [2; 32],
requested_plan: [3; 32],
completed_at: time(40 + set_index),
})
.collect(),
};
snapshot
.usage_snapshot_bucket_observations
.insert("bucket".to_string(), observation);
snapshot
}
fn incarnations() -> HashMap<String, String> {
HashMap::from([("bucket".to_string(), INCARNATION.to_string())])
}
#[test]
fn provenance_is_additive_and_unknown_states_do_not_keep_certificates() {
let legacy = usage(10);
let bytes = serde_json::to_vec(&legacy).expect("encode a legacy bucket without provenance");
let text = std::str::from_utf8(&bytes).expect("JSON is UTF-8");
assert!(!text.contains("usage_value_state"));
assert!(!text.contains("usage_certification"));
let decoded: BucketUsageInfo = serde_json::from_slice(&bytes).expect("read legacy bucket");
assert_eq!(decoded.usage_value_state, None);
assert_eq!(decoded.usage_certification, None);
let mut wire = serde_json::to_value(legacy).expect("encode future bucket fixture");
wire["usage_value_state"] = serde_json::json!("future_state");
wire["usage_certification"] =
serde_json::json!({"epoch":2,"cycle":3,"certified_at":{"secs_since_epoch":30,"nanos_since_epoch":0}});
let mut future: BucketUsageInfo = serde_json::from_value(wire).expect("unknown diagnostic enum remains readable");
future.normalize_usage_provenance();
assert_eq!(future.usage_value_state, Some(BucketUsageValueState::Unknown));
assert_eq!(future.usage_certification, None);
}
#[test]
fn carrying_preserves_time_but_adding_a_source_invalidates_the_certificate() {
let mut bucket = snapshot().buckets_usage.remove("bucket").expect("canonical bucket");
bucket.mark_carried();
assert_eq!(bucket.usage_certification.map(|certification| certification.certified_at), Some(time(30)));
bucket.merge(&usage(1));
assert_eq!(bucket.size, 11);
assert_eq!(bucket.usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(bucket.usage_certification, None);
}
#[test]
fn direct_object_accumulation_does_not_reuse_the_old_certificate() {
let mut simple = snapshot();
simple.add_object("bucket/object", &rustfs_filemeta::MetaObject::default());
assert_eq!(simple.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(simple.buckets_usage["bucket"].usage_certification, None);
assert!(simple.usage_snapshot_bucket_observations.is_empty());
let mut versioned = snapshot();
versioned.add_object_from_file_meta("bucket/object", &rustfs_filemeta::FileMeta::default());
assert_eq!(versioned.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(versioned.buckets_usage["bucket"].usage_certification, None);
assert!(versioned.usage_snapshot_bucket_observations.is_empty());
}
#[test]
fn future_or_malformed_companions_do_not_break_the_canonical_reader() {
let canonical = snapshot();
for entry in [
serde_json::json!({"version": 2, "future_payload": {"new": true}}),
serde_json::json!({"version": 1, "usage": "broken"}),
] {
let mut wire = serde_json::to_value(&canonical).expect("canonical JSON");
wire["usage_snapshot_bucket_observations"]["bucket"] = entry;
let decoded: DataUsageInfo = serde_json::from_value(wire).expect("diagnostics cannot break canonical data");
assert_eq!(decoded.buckets_usage, canonical.buckets_usage);
assert!(decoded.usage_snapshot_bucket_observations.is_empty());
assert!(decoded.is_complete_bucket_usage_snapshot());
}
}
#[test]
fn complete_companion_projects_only_the_admin_copy_and_keeps_measurement_times() {
let canonical = snapshot();
let canonical_bytes = serde_json::to_vec(&canonical).expect("encode canonical candidate");
let mut response = canonical.clone();
assert_eq!(response.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(response.buckets_usage["bucket"].size, 25);
assert_eq!(response.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(response.buckets_usage["bucket"].usage_certification, None);
assert_eq!(response.objects_total_size, 25);
assert_eq!(response.bucket_sizes["bucket"], 25);
assert_eq!(response.last_update, canonical.last_update);
assert_eq!(response.usage_snapshot_bucket_observations["bucket"].completed_at, time(40));
assert!(response.usage_snapshot_admin_projection);
assert!(response.usage_snapshot_partial);
assert!(response.usage_snapshot_observational);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(
!response.is_valid_partial_snapshot(),
"response-only values cannot be a storage candidate"
);
assert_eq!(serde_json::to_vec(&canonical).expect("reencode original"), canonical_bytes);
assert_eq!(canonical.buckets_usage["bucket"].size, 10);
assert_eq!(
canonical.buckets_usage["bucket"]
.usage_certification
.map(|certificate| certificate.certified_at),
Some(time(30))
);
assert_eq!(response.project_bucket_observations_for_admin(&incarnations()), 0);
}
#[test]
fn invalid_companions_preserve_the_canonical_bucket() {
for invalid in 0..11 {
let mut candidate = snapshot();
let observation = candidate
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("companion");
match invalid {
0 => observation.version += 1,
1 => observation.snapshot_identity.scanner_cycle = Some(8),
2 => {
observation.sources.pop();
}
3 => observation.sources[1].set_index = 0,
4 => observation.sources[1].scanner_cycle = 10,
5 => observation.sources[1].scan_plan_digest = [9; 32],
6 => observation.sources[1].bucket_incarnation = "other".to_string(),
7 => observation.completed_at = time(41),
8 => observation.usage.certify(BucketUsageCertification {
epoch: 2,
cycle: 9,
certified_at: time(90),
}),
9 => candidate.usage_snapshot_set_states[1].scanner_cycle = None,
10 => {
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some("recreated".to_string())
}
_ => unreachable!("bounded fixture variants"),
}
let before = candidate.clone();
assert_eq!(
candidate.project_bucket_observations_for_admin(&incarnations()),
0,
"invalid case {invalid}"
);
assert_eq!(candidate, before, "invalid case {invalid} changed canonical data");
}
}
#[test]
fn a_fresh_set_can_contribute_without_refreshing_an_older_mixed_set() {
let mut candidate = snapshot();
let observation = candidate
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("companion");
observation.sources[1].started_plan = observation.sources[1].requested_plan;
observation.sources[1].completed_at = time(90);
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(candidate.usage_snapshot_bucket_observations["bucket"].completed_at, time(40));
}
#[test]
fn companion_only_partial_keeps_legacy_maps_empty_until_admin_projection() {
let mut candidate = snapshot();
candidate.buckets_usage.clear();
candidate.bucket_sizes.clear();
candidate.buckets_count = 0;
candidate.calculate_totals();
candidate.usage_snapshot_complete = false;
candidate.usage_snapshot_partial = true;
candidate.usage_snapshot_observational = true;
for state in &mut candidate.usage_snapshot_set_states {
state.complete = false;
}
assert!(candidate.is_valid_partial_snapshot());
let wire = serde_json::to_value(&candidate).expect("encode companion-only observation");
assert_eq!(wire["buckets_usage"], serde_json::json!({}));
assert_eq!(wire["bucket_sizes"], serde_json::json!({}));
assert_eq!(wire["objects_total_size"], 0);
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(candidate.objects_total_size, 25);
assert!(candidate.usage_snapshot_partial);
assert!(!candidate.usage_snapshot_complete);
}
#[test]
fn projection_rejects_counter_overflow_without_partial_mutation() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(u64::MAX));
candidate.buckets_count = 2;
let before = candidate.clone();
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 0);
assert_eq!(candidate, before);
}
#[test]
fn a_newer_canonical_certificate_and_missing_pending_prevent_old_overlay() {
for newer_certificate in [false, true] {
let mut candidate = snapshot();
if newer_certificate {
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.certify(BucketUsageCertification {
epoch: 2,
cycle: 9,
certified_at: time(90),
});
} else {
candidate.usage_snapshot_pending_buckets.clear();
candidate.usage_snapshot_pending_bucket_count = 0;
}
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 0);
assert_eq!(candidate.buckets_usage["bucket"].size, 10);
}
}
#[test]
fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let canonical = snapshot();
let original = serde_json::to_vec(&canonical).expect("encode candidate");
let mut changed = canonical.clone();
changed
.buckets_usage
.get_mut("bucket")
.expect("bucket")
.usage_certification
.as_mut()
.expect("certificate")
.certified_at = time(31);
assert_ne!(serde_json::to_vec(&changed).expect("encode changed certificate"), original);
let mut changed = canonical.clone();
changed
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("observation")
.sources[0]
.started_plan = [8; 32];
assert_ne!(serde_json::to_vec(&changed).expect("encode changed source"), original);
let named = rmp_serde::to_vec_named(&canonical).expect("encode named map");
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
+11 -10
View File
@@ -98,11 +98,8 @@ rustfs-madmin.workspace = true
rustfs-filemeta.workspace = true
bytes = { workspace = true, features = ["serde"] }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
aws-config = { workspace = true }
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
aws-smithy-types.workspace = true
async-compression = { workspace = true, features = ["tokio", "bzip2", "lz4", "xz"] }
async-trait = { workspace = true }
flate2.workspace = true
http.workspace = true
@@ -126,27 +123,31 @@ urlencoding.workspace = true
walkdir.workspace = true
base64-simd = { workspace = true }
rand = { workspace = true, features = ["serde"] }
chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
hex-simd = { workspace = true }
md-5 = { workspace = true }
opentelemetry-proto = { workspace = true }
prost.workspace = true
sha2 = { workspace = true }
rustfs-tokio-tar = { workspace = true }
rustfs-s3-types = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
zstd.workspace = true
time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
rcgen.workspace = true
local-ip-address.workspace = true
anyhow.workspace = true
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
russh = { workspace = true, features = ["serde"] }
russh-sftp = { workspace = true }
zip.workspace = true
clap = { workspace = true, features = ["derive", "env"] }
[dev-dependencies]
async-compression = { workspace = true, features = ["tokio", "bzip2", "lz4", "xz"] }
aws-config = { workspace = true }
aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
opentelemetry-proto = { workspace = true }
russh = { workspace = true, features = ["serde"] }
russh-sftp = { workspace = true }
suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
rustfs-tokio-tar = { workspace = true }
zip.workspace = true
same-file.workspace = true
aws-smithy-http-client = { workspace = true, default-features = false, features = ["test-util"] }
rustfs-heal.workspace = true
+1 -1
View File
@@ -29,7 +29,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set `RUSTFS_E2E_REFERENCE_BINARY` to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Both sides run the binary under test on its only (gateway) stack unless `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary, which then runs side A. A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. `RUSTFS_E2E_STACK_B_REGION` starts side B in another region to inject a difference. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
@@ -21,8 +21,8 @@ use crate::common::{
};
use http::header::{AUTHORIZATION, HOST};
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::{constants::UNSIGNED_PAYLOAD, sign_v4};
use s3s::Body;
use serde_json::Value;
use std::{error::Error, time::Duration};
+133 -1
View File
@@ -146,7 +146,11 @@ async fn create_service_account_for(
Some(serde_json::json!({ "targetUser": parent }).to_string()),
)
.await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
service_account_keys(&response)
}
fn service_account_keys(response: &str) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
let response: serde_json::Value = serde_json::from_str(response)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
@@ -344,6 +348,134 @@ async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestRe
Ok(())
}
#[tokio::test]
async fn test_delegated_service_account_creation_through_console_sessions() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_without_cleanup(vec![]).await?;
let bucket = "delegated-backups";
let target = "backup-user";
let operator = "backup-operator";
let operator_secret = "operatorSecret123";
let wildcard_admin = "backup-admin";
let wildcard_secret = "adminSecret123";
let s3 = env.create_s3_client();
s3.create_bucket().bucket(bucket).send().await?;
for (name, secret) in [
(target, "backupSecret123"),
(operator, operator_secret),
(wildcard_admin, wildcard_secret),
("other-user", "otherUserSecret123"),
] {
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={name}"),
Some(serde_json::json!({"secretKey":secret,"status":"enabled"}).to_string()),
)
.await?;
}
let delegation = serde_json::json!({
"Version":"2012-10-17",
"Statement":[
{"Effect":"Allow","Action":"admin:CreateServiceAccount"},
{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:s3:::*"},
{"Effect":"Allow","Action":"admin:CreateServiceAccountForUser","Condition":{"StringEquals":{"svc:TargetUser":target}}}
]
});
let wildcard = serde_json::json!({
"Version":"2012-10-17",
"Statement":[
{"Effect":"Allow","Action":"admin:*"},
{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:s3:::*"}
]
});
for (user, name, policy) in [
(wildcard_admin, "backup-admin-wildcard", wildcard.to_string()),
(target, "backup-writer", bucket_rw_policy(bucket)),
(operator, "backup-key-manager", delegation.to_string()),
] {
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={name}"),
Some(policy),
)
.await?;
admin_ok(
&env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({"policies":[name],"user":user}).to_string()),
)
.await?;
}
for (name, secret) in [
(operator, operator_secret),
(wildcard_admin, wildcard_secret),
(env.access_key.as_str(), env.secret_key.as_str()),
] {
let session = build_test_sts_client(&env.url, name, secret, None, "delegated-create")
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/console")
.role_session_name("console")
.send()
.await?;
let caller = session.credentials().ok_or("missing console session")?;
let create = |user: &str| {
admin_request_with_session_token(
&env.url,
http::Method::PUT,
"/rustfs/admin/v3/add-service-account",
Some(serde_json::json!({"targetUser":user}).to_string()),
caller.access_key_id(),
caller.secret_access_key(),
Some(caller.session_token()),
)
};
let (status, response) = create(target).await?;
assert_eq!(status, StatusCode::OK, "create for target as {name}");
let (key, secret) = service_account_keys(&response)?;
let client = build_s3_client(&env.url, &key, &secret);
wait_for_s3_put(&client, bucket, "backup", Duration::from_secs(10)).await?;
let denied = client
.create_bucket()
.bucket("outside-backup-policy")
.send()
.await
.expect_err("target cannot create unrelated buckets");
assert_eq!(denied.as_service_error().and_then(|e| e.code()), Some("AccessDenied"));
if name != env.access_key {
let (status, _) = create(&env.access_key).await?;
assert_eq!(status, StatusCode::FORBIDDEN, "delegated targets exclude root");
}
if name == operator {
let (status, _) = create("other-user").await?;
assert_eq!(status, StatusCode::FORBIDDEN, "outside delegated target");
}
}
let restricted = build_test_sts_client(&env.url, &env.access_key, &env.secret_key, None, "restricted-root")
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/console")
.role_session_name("restricted-console")
.policy(r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"admin:CreateServiceAccount"}]}"#)
.send()
.await?;
let caller = restricted.credentials().ok_or("missing restricted root session")?;
let (status, _) = admin_request_with_session_token(
&env.url,
http::Method::PUT,
"/rustfs/admin/v3/add-service-account",
Some(serde_json::json!({"targetUser":target}).to_string()),
caller.access_key_id(),
caller.secret_access_key(),
Some(caller.session_token()),
)
.await?;
assert_eq!(status, StatusCode::FORBIDDEN, "restricted root session requires delegated permission");
Ok(())
}
/// Full user -> policy -> service-account lifecycle, proving each management
/// call takes effect on the data plane, not just that the endpoint answers 200.
#[tokio::test]
+1 -1
View File
@@ -37,9 +37,9 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use hmac::{Hmac, KeyInit as _, Mac};
use http::header::HOST;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use sha1::Sha1;
use std::error::Error;
use std::time::{SystemTime, UNIX_EPOCH};
+1 -1
View File
@@ -15,9 +15,9 @@
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde::Deserialize;
use std::error::Error;
@@ -17,9 +17,9 @@ use aws_sdk_s3::primitives::ByteStream;
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_madmin::{ITEM_OFFLINE, InfoMessage, StorageInfo};
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde::Deserialize;
use std::error::Error;
use tokio::time::{Duration, sleep, timeout};
@@ -21,9 +21,9 @@ mod tests {
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use http::header::{CONTENT_TYPE, HOST};
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::{pre_sign_v4, sign_v4};
use s3s::Body;
use sha2::{Digest, Sha256};
use std::error::Error;
use std::io::{Cursor, Write};
+17 -2
View File
@@ -37,9 +37,9 @@
use crate::common::{RustFSTestEnvironment, local_http_client};
use http::header::{CONTENT_TYPE, HOST};
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::error::Error;
@@ -170,6 +170,15 @@ impl DiskFaultHarness {
/// Start (or restart after [`Self::kill_server`]) the RustFS server with
/// all volume directories and the same address.
pub async fn start_server(&mut self) -> ChaosResult<()> {
self.start_server_with_readiness(false).await
}
/// Start a process once its node read-readiness probe passes.
pub async fn start_server_at_read_quorum(&mut self) -> ChaosResult<()> {
self.start_server_with_readiness(true).await
}
async fn start_server_with_readiness(&mut self, read_quorum_only: bool) -> ChaosResult<()> {
// The common helper always appends `env.temp_dir` as the final storage
// path: point it at the last disk for startup and pass the remaining
// disks explicitly, then restore it to the root so Drop cleans up
@@ -181,7 +190,13 @@ impl DiskFaultHarness {
let arg_refs: Vec<&str> = disk_args.iter().map(String::as_str).collect();
let env_refs: Vec<(&str, &str)> = self.extra_env.iter().map(|(k, v)| (k.as_str(), v.as_str())).collect();
let result = self.env.start_rustfs_server_with_env(arg_refs, &env_refs).await;
let result = if read_quorum_only {
self.env
.start_rustfs_server_with_env_at_read_quorum(arg_refs, &env_refs)
.await
} else {
self.env.start_rustfs_server_with_env(arg_refs, &env_refs).await
};
self.env.temp_dir = self.root.to_string_lossy().to_string();
result
}
+36 -6
View File
@@ -29,9 +29,9 @@ use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
use reqwest::Client as HttpClient;
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde_json;
use std::fs as stdfs;
use std::io::{self, ErrorKind};
@@ -722,7 +722,7 @@ impl RustFSTestEnvironment {
cleanup_existing: bool,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
self.start_rustfs_server_inner_with_binary(&binary_path, extra_args, extra_env, cleanup_existing)
self.start_rustfs_server_inner_with_binary(&binary_path, extra_args, extra_env, cleanup_existing, false)
.await
}
@@ -732,6 +732,7 @@ impl RustFSTestEnvironment {
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
cleanup_existing: bool,
read_quorum_only: bool,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
if cleanup_existing {
self.cleanup_existing_processes().await?;
@@ -756,7 +757,7 @@ impl RustFSTestEnvironment {
self.process = Some(process);
// Wait for server to be ready
self.wait_for_server_ready().await?;
self.wait_for_server_ready_with_probe(read_quorum_only).await?;
Ok(())
}
@@ -770,7 +771,7 @@ impl RustFSTestEnvironment {
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, true)
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, true, false)
.await
}
@@ -782,7 +783,7 @@ impl RustFSTestEnvironment {
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, false)
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, false, false)
.await
}
@@ -800,6 +801,18 @@ impl RustFSTestEnvironment {
self.start_rustfs_server_inner(extra_args, extra_env, true).await
}
/// Start a degraded read-serving process using the node readiness probe.
/// ListBuckets retains its namespace write-quorum requirement.
pub async fn start_rustfs_server_with_env_at_read_quorum(
&mut self,
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
self.start_rustfs_server_inner_with_binary(&binary_path, extra_args, extra_env, true, true)
.await
}
/// Start RustFS server without cleaning up other running RustFS processes.
///
/// This is useful for tests that need multiple independent RustFS instances
@@ -828,8 +841,16 @@ impl RustFSTestEnvironment {
/// Fails immediately if the spawned server process has already exited
/// (e.g. a port bind failure) instead of polling out the full timeout.
pub async fn wait_for_server_ready(&mut self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.wait_for_server_ready_with_probe(false).await
}
async fn wait_for_server_ready_with_probe(
&mut self,
read_quorum_only: bool,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
info!("Waiting for RustFS server to be ready on {}", self.address);
let client = self.create_s3_client();
let health_client = HttpClient::builder().no_proxy().timeout(Duration::from_secs(2)).build()?;
for i in 0..60 {
if let Some(process) = self.process.as_mut()
@@ -838,7 +859,16 @@ impl RustFSTestEnvironment {
return Err(format!("RustFS server process exited before becoming ready: {status}").into());
}
if TcpStream::connect(&self.address).await.is_ok() && client.list_buckets().send().await.is_ok() {
let ready = if read_quorum_only {
health_client
.get(format!("{}/minio/health/ready", self.url))
.send()
.await
.is_ok_and(|response| response.status().is_success())
} else {
TcpStream::connect(&self.address).await.is_ok() && client.list_buckets().send().await.is_ok()
};
if ready {
info!("✅ RustFS server is ready after {} attempts", i + 1);
return Ok(());
}
+36 -10
View File
@@ -13,8 +13,8 @@
// limitations under the License.
//! Over-the-wire smoke net for the embedded console listener (backlog#1154
//! peri-4). The console is a second HTTP listener that serves the full S3/admin
//! surface plus the unauthenticated console routes under `/rustfs/console`;
//! peri-4). The console is a second HTTP listener for the control-plane and
//! unauthenticated console routes under `/rustfs/console`;
//! before this test its behaviour was only covered by in-process router tests
//! (`rustfs/src/admin/console.rs`), so a misconfiguration that exposed
//! credentials through the public console endpoints or fail-opened the
@@ -26,13 +26,13 @@
//! * `/rustfs/console/license` exposes only the coarse `licensed` flag.
//! * the console SPA prefix dispatches to the static handler, never to the
//! S3 API (no S3 error XML), whether or not console assets are embedded.
//! * unauthenticated requests to the admin API and the S3 root on the console
//! listener are denied (403 AccessDenied) — the extra listener does not
//! fail-open the protected surface.
//! * unauthenticated admin API requests are denied with 403 AccessDenied.
//! * S3 root/object requests on the console listener return an empty 404,
//! including authenticated requests, before S3 dispatch.
//! * a listener with the console disabled (the main S3 port here) does not
//! serve the unauthenticated console endpoints at all.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client, signed_request_with_client};
use std::error::Error;
use tokio::time::{Duration, sleep};
@@ -166,13 +166,39 @@ async fn test_console_over_the_wire_smoke() -> TestResult {
let admin_body = admin_response.text().await?;
assert!(admin_body.contains("AccessDenied"), "admin denial must be AccessDenied: {admin_body}");
let s3_root_response = client.get(format!("{console_base}/")).send().await?;
let signed_data_root = signed_request_with_client(
&client,
http::Method::GET,
&format!("{}/", env.url),
&env.access_key,
&env.secret_key,
None,
None,
)
.await?;
assert_eq!(
s3_root_response.status(),
reqwest::StatusCode::FORBIDDEN,
"unauthenticated S3 root on the console listener must be denied"
signed_data_root.status(),
reqwest::StatusCode::OK,
"the same credentials must reach S3 on the data listener"
);
for path in ["/", "/console-object-bucket/key"] {
let url = format!("{console_base}{path}");
for signed in [false, true] {
let response = if signed {
signed_request_with_client(&client, http::Method::GET, &url, &env.access_key, &env.secret_key, None, None).await?
} else {
client.get(&url).send().await?
};
assert_eq!(
response.status(),
reqwest::StatusCode::NOT_FOUND,
"S3 route {path}, signed={signed} must be isolated from the console listener"
);
assert!(response.bytes().await?.is_empty(), "isolated S3 routes must not reach a handler");
}
}
// --- console disabled on a listener means no console surface at all ------
// The main S3 listener of this same process runs with the console disabled,
// so its console paths must not answer with the unauthenticated console
@@ -44,9 +44,9 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::primitives::ByteStream;
use http::header::HOST;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
/// Signed raw `PUT` copy request with an explicit copy-source conditional
@@ -135,6 +135,72 @@ mod tests {
Ok(())
}
/// A comma-separated region list widens the regions `CreateBucket`
/// accepts without changing the server's own region (the first entry): a
/// listed region creates the bucket, an unlisted one is still rejected,
/// and `GetBucketLocation` keeps reporting the site region.
#[tokio::test]
async fn test_create_bucket_accepts_listed_regions() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec!["--region", "us-east-1,ap-southeast-2,us-west-2"])
.await?;
for region in ["us-east-1", "ap-southeast-2", "us-west-2"] {
let client = Client::from_conf(
env.create_s3_client()
.config()
.to_builder()
.region(Region::new(region))
.build(),
);
let bucket = format!("additional-region-{region}");
client
.create_bucket()
.bucket(&bucket)
.create_bucket_configuration(
CreateBucketConfiguration::builder()
.location_constraint(BucketLocationConstraint::from(region))
.build(),
)
.send()
.await
.expect("a listed region must create the bucket");
let location = client.get_bucket_location().bucket(&bucket).send().await?;
assert_eq!(
location.location_constraint().map(|value| value.as_str()),
Some(""),
"GetBucketLocation keeps reporting the site region (us-east-1)"
);
}
let client = env.create_s3_client();
let error = client
.create_bucket()
.bucket("unlisted-region")
.create_bucket_configuration(
CreateBucketConfiguration::builder()
.location_constraint(BucketLocationConstraint::from("eu-west-1"))
.build(),
)
.send()
.await
.expect_err("a region missing from the region list must be rejected");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(400));
assert_eq!(
error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("InvalidLocationConstraint")
);
let missing = client
.head_bucket()
.bucket("unlisted-region")
.send()
.await
.expect_err("rejected bucket must not exist");
assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
Ok(())
}
#[tokio::test]
async fn test_create_bucket_uses_configured_region() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -21,9 +21,9 @@ mod tests {
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use http::header::HOST;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -26,7 +26,7 @@ use aws_sdk_s3::types::{
DefaultRetention, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
ObjectLockRule,
};
use chrono::{Duration as ChronoDuration, Utc};
use jiff::{SignedDuration, Timestamp};
fn delete_denied(error: &SdkError<DeleteObjectError>, context: &str) -> TestResult {
let code = error.as_service_error().and_then(ProvideErrorMetadata::code);
@@ -66,7 +66,7 @@ async fn four_node_four_drive_object_lock_worm_blocks_delete() -> TestResult {
.send()
.await?;
let retain_until = Utc::now() + ChronoDuration::days(1);
let retain_until = Timestamp::now() + SignedDuration::from_hours(24);
let compliance_key = "compliance.bin";
let compliance_version = put_object_with_retention(
@@ -176,7 +176,10 @@ async fn four_node_default_retention_is_visible_and_non_lock_bucket_rejects_conf
let retain_until = retention
.retain_until_date()
.ok_or("default retention omitted retain-until date")?;
assert!(retain_until.secs() > Utc::now().timestamp(), "default retention is not in the future");
assert!(
retain_until.secs() > Timestamp::now().as_second(),
"default retention is not in the future"
);
let versioning = reader.get_bucket_versioning().bucket(&bucket).send().await?;
assert_eq!(versioning.status().map(|status| status.as_str()), Some("Enabled"));
@@ -38,8 +38,8 @@ use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
VersioningConfiguration,
};
use chrono::{DateTime, Utc};
use http::{Method, StatusCode};
use jiff::Timestamp;
use serde_json::Value;
use std::collections::{BTreeMap, BTreeSet};
use std::path::{Path, PathBuf};
@@ -526,8 +526,8 @@ async fn put_multipart(client: &Client, bucket: &str, key: &str) -> TestResult<V
Ok(body)
}
fn retention_deadline() -> TestResult<DateTime<Utc>> {
DateTime::<Utc>::from_timestamp(Utc::now().timestamp() + 24 * 3600, 0).ok_or_else(|| "retention deadline out of range".into())
fn retention_deadline() -> TestResult<Timestamp> {
Ok(Timestamp::now() + jiff::SignedDuration::from_hours(24))
}
/// Seed the deterministic inventory into the seed pool and record it.
@@ -654,7 +654,7 @@ pub(crate) async fn seed_manifest(run: &TopologyRun) -> TestResult<TopologyManif
let mut entry = ManifestEntry::body(&locked, &key, "compliance", Some(version_id.clone()), Some(version_id), &body);
entry.retention = Some(RetentionRecord {
mode: "COMPLIANCE".to_string(),
retain_until_unix: retain_until.timestamp(),
retain_until_unix: retain_until.as_second(),
});
entries.push(entry);
}
@@ -12,14 +12,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//! Differential: two `rustfs` processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A runs `RUSTFS_S3_STACK=legacy` of the binary under test, unless `RUSTFS_E2E_REFERENCE_BINARY`
//! names a released binary: then A is that binary, started without `RUSTFS_S3_STACK` (older
//! releases do not know it), and a bad path fails the run instead of falling back (see
//! `reference`). The report names which side is the reference and the `--version` each side ran. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! Side B is the binary under test. Side A is the binary under test too, unless
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released binary: then A is that binary, and a bad path
//! fails the run instead of falling back (see `reference`). The report names which side is the
//! reference and the `--version` each side ran. `RUSTFS_E2E_STACK_B_REGION`, when set, starts B
//! in another region so that a difference can be injected by hand. Neither side is given the
//! retired stack selector: the binary under test refuses to start with it, and the gateway stack
//! is its only S3 stack. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
@@ -37,20 +38,18 @@ use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
/// Label of a side that runs the binary under test.
const UNDER_TEST: &str = "under-test";
/// Label of side A in the report when it is the released reference binary.
const STACK_A_REFERENCE: &str = "reference";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
const DEFAULT_STACK_B: &str = "legacy";
/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
/// report its own name and would add CreateBucket differences to the control.
const CONTROL_REGION_A: &str = "us-west-2";
const CONTROL_REGION_B: &str = "eu-west-1";
/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
/// `normal` cases side A must answer with a 2xx. The comparison alone cannot tell a
/// script that exercises real behavior from one in which both stacks fail the same way. Five of
/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
@@ -60,7 +59,6 @@ const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
side_a: SideA,
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
@@ -69,7 +67,6 @@ impl PairConfig {
fn from_env() -> Result<Self, BoxError> {
Ok(Self {
side_a: SideA::from_env()?,
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
})
@@ -77,16 +74,16 @@ impl PairConfig {
fn stack_a_label(&self) -> &'static str {
match self.side_a {
SideA::SameBinary => STACK_A,
SideA::SameBinary => UNDER_TEST,
SideA::Reference(_) => STACK_A_REFERENCE,
}
}
}
/// Starts one server of the binary under test on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
/// Starts one server of the binary under test, overriding its region only when asked.
async fn start_under_test(region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
let mut vars = Vec::new();
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
@@ -94,8 +91,7 @@ async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvi
Ok(env)
}
/// Starts one server of a released binary. `RUSTFS_S3_STACK` is not passed: older releases do not
/// know it and have exactly one stack.
/// Starts one server of a released binary.
async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = Vec::new();
@@ -111,11 +107,11 @@ async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTe
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
let side_a = async {
match &config.side_a {
SideA::SameBinary => start_stack(STACK_A, config.region_a.as_deref()).await,
SideA::SameBinary => start_under_test(config.region_a.as_deref()).await,
SideA::Reference(binary) => start_reference(binary, config.region_a.as_deref()).await,
}
};
tokio::try_join!(side_a, start_stack(&config.stack_b, config.region_b.as_deref()))
tokio::try_join!(side_a, start_under_test(config.region_b.as_deref()))
}
fn report_directory() -> PathBuf {
@@ -160,7 +156,7 @@ async fn run_differential(label: &str, config: &PairConfig) -> Run {
schema: 2,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: config.stack_a_label().to_owned(),
stack_b: config.stack_b.clone(),
stack_b: UNDER_TEST.to_owned(),
reference_side: matches!(config.side_a, SideA::Reference(_)).then_some("a"),
version_a,
version_b,
@@ -235,7 +231,6 @@ mod tests {
init_logging();
let config = PairConfig {
side_a: SideA::SameBinary,
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
};
@@ -15,7 +15,7 @@
//! Which binary side A of the differential runs, and what it reports about itself.
//!
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary to use as the reference. Unset,
//! side A is the binary under test on `RUSTFS_S3_STACK=legacy`. Set, it must resolve to an
//! side A is the binary under test. Set, it must resolve to an
//! executable file or the run fails: a reference that was asked for and silently replaced by the
//! binary under test would report an agreement with itself as agreement with the release.
@@ -28,9 +28,9 @@ pub(super) const ENV_REFERENCE_BINARY: &str = "RUSTFS_E2E_REFERENCE_BINARY";
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) enum SideA {
/// Both sides are the binary under test; A runs `RUSTFS_S3_STACK=legacy`.
/// Both sides are the binary under test.
SameBinary,
/// A is this released binary, started without `RUSTFS_S3_STACK`.
/// A is this released binary.
Reference(PathBuf),
}
@@ -14,7 +14,42 @@
//! Differential cases of the `bucket_access` batch (see `super`).
use super::super::Recorder;
use super::super::{Recorder, none};
use aws_sdk_s3::Client;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
const BUCKET: &str = "a2-bucket-access";
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
s.step("CreateBucket", "a2-setup", c.create_bucket().bucket(BUCKET).send(), none)
.await;
versioning(s, c).await;
}
async fn versioning(s: &mut Recorder, c: &Client) {
let status = |o: &aws_sdk_s3::operation::get_bucket_versioning::GetBucketVersioningOutput| {
vec![
("status", format!("{:?}", o.status())),
("mfa_delete", format!("{:?}", o.mfa_delete())),
]
};
s.step(
"GetBucketVersioning",
"unconfigured",
c.get_bucket_versioning().bucket(BUCKET).send(),
status,
)
.await;
let suspended = VersioningConfiguration::builder()
.status(BucketVersioningStatus::Suspended)
.build();
let put = c.put_bucket_versioning().bucket(BUCKET).versioning_configuration(suspended);
s.step("PutBucketVersioning", "a2-suspended", put.send(), none).await;
s.step(
"GetBucketVersioning",
"suspended",
c.get_bucket_versioning().bucket(BUCKET).send(),
status,
)
.await;
}
@@ -14,7 +14,59 @@
//! Differential cases of the `bucket_core` batch (see `super`).
use super::super::Recorder;
use super::super::{Recorder, none};
use aws_sdk_s3::Client;
use aws_sdk_s3::types::{BucketLocationConstraint, CreateBucketConfiguration};
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
/// Names RustFS creates although the AWS grammar reserves them, and one it refuses.
const RESERVED_NAMES: [(&str, &str); 5] = [
("reserved-sthree", "sthree-dual-stack"),
("reserved-s3alias", "dual-stack-s3alias"),
("reserved-ol-s3", "dual-stack--ol-s3"),
("reserved-x-s3", "dual-stack--x-s3"),
("reserved-xn", "xn--dual-stack"),
];
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
let located = |bucket: &str, location: &str| {
let configuration = CreateBucketConfiguration::builder()
.location_constraint(BucketLocationConstraint::from(location))
.build();
c.create_bucket().bucket(bucket).create_bucket_configuration(configuration)
};
s.step(
"CreateBucket",
"foreign-location",
located("dual-stack-foreign", "ap-south-1").send(),
none,
)
.await;
s.step("CreateBucket", "eu-alias", located("dual-stack-eu", "EU").send(), none)
.await;
for (case, name) in RESERVED_NAMES {
s.step("CreateBucket", case, c.create_bucket().bucket(name).send(), none)
.await;
}
for (case, bucket, header) in FORCE_DELETES {
// Each case deletes a bucket of its own; the create is setup, not a recorded step.
let _ = c.create_bucket().bucket(bucket).send().await;
let delete = c
.delete_bucket()
.bucket(bucket)
.customize()
.mutate_request(move |request| {
request.headers_mut().insert(header.0, header.1);
})
.send();
s.step("DeleteBucket", case, delete, none).await;
}
}
/// Force-delete headers under both spellings, with values each parser reads differently.
const FORCE_DELETES: [(&str, &str, (&str, &str)); 5] = [
("force-rustfs-true", "dual-stack-force-a", ("x-rustfs-force-delete", "true")),
("force-rustfs-invalid", "dual-stack-force-b", ("x-rustfs-force-delete", "maybe")),
("force-minio-true", "dual-stack-force-c", ("x-minio-force-delete", "true")),
("force-minio-numeric", "dual-stack-force-d", ("x-minio-force-delete", "1")),
("force-minio-invalid", "dual-stack-force-e", ("x-minio-force-delete", "maybe")),
];
@@ -13,8 +13,124 @@
// limitations under the License.
//! Differential cases of the `bucket_replication_lifecycle` batch (see `super`).
//!
//! Replication has no case of its own: storing a replication configuration needs a remote target
//! registered through the admin API, which this script does not drive.
use super::super::Recorder;
use super::super::{Fields, RETAIN_UNTIL_SECS, Recorder, none};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::DateTime;
use aws_sdk_s3::types::{
AbortIncompleteMultipartUpload, BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, LifecycleExpiration,
LifecycleRule, LifecycleRuleAndOperator, LifecycleRuleFilter, NoncurrentVersionExpiration, ObjectLockConfiguration,
ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, Tag,
};
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
/// A name RustFS creates and serves that the AWS bucket grammar reserves.
const RESERVED_BUCKET: &str = "sthree-dual-stack-a3";
const LOCK_BUCKET: &str = "dual-stack-a3-lock";
fn lifecycle_rules(
output: &aws_sdk_s3::operation::get_bucket_lifecycle_configuration::GetBucketLifecycleConfigurationOutput,
) -> Fields {
vec![("rules", format!("{:?}", output.rules()))]
}
fn object_lock(output: &aws_sdk_s3::operation::get_object_lock_configuration::GetObjectLockConfigurationOutput) -> Fields {
vec![("configuration", format!("{:?}", output.object_lock_configuration()))]
}
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
s.step("CreateBucket", "a3-reserved-name", c.create_bucket().bucket(RESERVED_BUCKET).send(), none)
.await;
let get_lifecycle = || c.get_bucket_lifecycle_configuration().bucket(RESERVED_BUCKET);
s.step("GetBucketLifecycleConfiguration", "a3-reserved-absent", get_lifecycle().send(), none)
.await;
let rule = LifecycleRule::builder()
.id("members")
.status(ExpirationStatus::Enabled)
.filter(
LifecycleRuleFilter::builder()
.and(
LifecycleRuleAndOperator::builder()
.prefix("logs/")
.tags(Tag::builder().key("tier").value("cold").build().expect("tag"))
.object_size_greater_than(1)
.build(),
)
.build(),
)
.expiration(LifecycleExpiration::builder().expired_object_delete_marker(true).build())
.noncurrent_version_expiration(
NoncurrentVersionExpiration::builder()
.noncurrent_days(7)
.newer_noncurrent_versions(2)
.build(),
)
.abort_incomplete_multipart_upload(AbortIncompleteMultipartUpload::builder().days_after_initiation(3).build())
.build()
.expect("lifecycle rule");
// A stored date is answered through a different conversion on each edge of the migration.
let dated = LifecycleRule::builder()
.id("dated")
.status(ExpirationStatus::Disabled)
.filter(LifecycleRuleFilter::builder().prefix("archive/").build())
.expiration(
LifecycleExpiration::builder()
.date(DateTime::from_secs(RETAIN_UNTIL_SECS))
.build(),
)
.build()
.expect("dated lifecycle rule");
let lifecycle = BucketLifecycleConfiguration::builder()
.rules(rule)
.rules(dated)
.build()
.expect("lifecycle configuration");
let put = c
.put_bucket_lifecycle_configuration()
.bucket(RESERVED_BUCKET)
.lifecycle_configuration(lifecycle);
s.step("PutBucketLifecycleConfiguration", "a3-every-member", put.send(), none)
.await;
s.step(
"GetBucketLifecycleConfiguration",
"a3-every-member",
get_lifecycle().send(),
lifecycle_rules,
)
.await;
let get_lock = |bucket: &str| c.get_object_lock_configuration().bucket(bucket);
s.step("GetObjectLockConfiguration", "a3-reserved-absent", get_lock(RESERVED_BUCKET).send(), none)
.await;
let create = c.create_bucket().bucket(LOCK_BUCKET).object_lock_enabled_for_bucket(true);
s.step("CreateBucket", "a3-lock", create.send(), none).await;
s.step(
"GetObjectLockConfiguration",
"a3-enabled-without-rule",
get_lock(LOCK_BUCKET).send(),
object_lock,
)
.await;
let lock = ObjectLockConfiguration::builder()
.object_lock_enabled(ObjectLockEnabled::Enabled)
.rule(
ObjectLockRule::builder()
.default_retention(
DefaultRetention::builder()
.mode(ObjectLockRetentionMode::Compliance)
.years(1)
.build(),
)
.build(),
)
.build();
let put = c
.put_object_lock_configuration()
.bucket(LOCK_BUCKET)
.object_lock_configuration(lock);
s.step("PutObjectLockConfiguration", "a3-years", put.send(), none).await;
s.step("GetObjectLockConfiguration", "a3-years", get_lock(LOCK_BUCKET).send(), object_lock)
.await;
}
@@ -12,9 +12,99 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Differential cases of the `listing_multipart` batch (see `super`).
//! Differential cases of the `listing_multipart` batch (see `super`): stored keys that need
//! encoding, with and without `encoding-type=url`, ListObjects' marker, ListObjectsV2 pagination,
//! and the listings' parameter refusals.
use super::super::Recorder;
use super::super::{Fields, Recorder, none};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::EncodingType;
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
const BUCKET: &str = "differential-listing";
const KEYS: [&str; 3] = ["dir a/file+b%.txt", "markup &<>\"'", "~tilde"];
fn v2_fields(o: &aws_sdk_s3::operation::list_objects_v2::ListObjectsV2Output) -> Fields {
let keys: Vec<_> = o.contents().iter().map(|object| object.key()).collect();
vec![
("keys", format!("{keys:?}")),
("encoding_type", format!("{:?}", o.encoding_type())),
("key_count", format!("{:?}", o.key_count())),
("is_truncated", format!("{:?}", o.is_truncated())),
("has_next_token", format!("{}", o.next_continuation_token().is_some())),
]
}
fn v1_fields(o: &aws_sdk_s3::operation::list_objects::ListObjectsOutput) -> Fields {
let keys: Vec<_> = o.contents().iter().map(|object| object.key()).collect();
vec![
("keys", format!("{keys:?}")),
("marker", format!("{:?}", o.marker())),
("next_marker", format!("{:?}", o.next_marker())),
("encoding_type", format!("{:?}", o.encoding_type())),
]
}
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
s.step("ListObjectsV2", "setup-bucket", c.create_bucket().bucket(BUCKET).send(), none)
.await;
for (key, case) in KEYS.into_iter().zip(["setup-object-1", "setup-object-2", "setup-object-3"]) {
let put = c
.put_object()
.bucket(BUCKET)
.key(key)
.body(ByteStream::from_static(b"x"))
.send();
s.step("ListObjectsV2", case, put, none).await;
}
let v2 = || c.list_objects_v2().bucket(BUCKET);
s.step("ListObjectsV2", "raw-keys", v2().send(), v2_fields).await;
s.step("ListObjectsV2", "url-encoded", v2().encoding_type(EncodingType::Url).send(), v2_fields)
.await;
let first = s
.step("ListObjectsV2", "first-page", v2().max_keys(2).send(), v2_fields)
.await;
let token = first.and_then(|page| page.next_continuation_token().map(str::to_owned));
s.step("ListObjectsV2", "next-page", v2().set_continuation_token(token).send(), v2_fields)
.await;
s.step("ListObjectsV2", "negative-max-keys", v2().max_keys(-1).send(), none)
.await;
s.step("ListObjectsV2", "undecodable-token", v2().continuation_token("%%%").send(), none)
.await;
let v1 = || c.list_objects().bucket(BUCKET).max_keys(1);
s.step("ListObjects", "raw-marker", v1().send(), v1_fields).await;
s.step(
"ListObjects",
"url-encoded-marker",
v1().encoding_type(EncodingType::Url).send(),
v1_fields,
)
.await;
s.step("ListObjects", "request-marker", v1().marker(KEYS[0]).send(), v1_fields)
.await;
s.step("ListObjects", "negative-max-keys", v1().max_keys(-1).send(), none)
.await;
let create = c.create_multipart_upload().bucket(BUCKET).key(KEYS[0]).send();
s.step("ListMultipartUploads", "setup-upload", create, none).await;
let uploads = || c.list_multipart_uploads().bucket(BUCKET);
s.step("ListMultipartUploads", "raw-keys", uploads().send(), upload_fields)
.await;
let encoded = uploads().prefix("dir a/").encoding_type(EncodingType::Url).send();
s.step("ListMultipartUploads", "url-encoded", encoded, upload_fields).await;
s.step("ListMultipartUploads", "zero-max-uploads", uploads().max_uploads(0).send(), none)
.await;
let foreign = uploads().prefix("dir").key_marker("other").send();
s.step("ListMultipartUploads", "foreign-key-marker", foreign, none).await;
}
fn upload_fields(o: &aws_sdk_s3::operation::list_multipart_uploads::ListMultipartUploadsOutput) -> Fields {
let keys: Vec<_> = o.uploads().iter().map(|upload| upload.key()).collect();
vec![
("keys", format!("{keys:?}")),
("prefix", format!("{:?}", o.prefix())),
("encoding_type", format!("{:?}", o.encoding_type())),
]
}
@@ -14,7 +14,106 @@
//! Differential cases of the `object_meta` batch (see `super`).
use super::super::Recorder;
use super::super::{Recorder, grants, none, tags};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{AccessControlPolicy, Grant, Grantee, Owner, Permission, Type};
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
const BUCKET: &str = "diff-object-meta";
const TAGGED: &str = "tagged";
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
s.step("CreateBucket", "object-meta", c.create_bucket().bucket(BUCKET).send(), none)
.await;
let put = c
.put_object()
.bucket(BUCKET)
.key(TAGGED)
.tagging("k=&odd%20key=v%2Fw")
.body(ByteStream::from_static(b"object meta"));
s.step("PutObject", "object-meta-tagged", put.send(), none).await;
let get_tags = |key: &str, version: Option<&str>| {
c.get_object_tagging()
.bucket(BUCKET)
.key(key)
.set_version_id(version.map(str::to_owned))
};
s.step("GetObjectTagging", "empty-and-encoded-tags", get_tags(TAGGED, None).send(), |o| {
vec![("tags", tags(o.tag_set())), ("version", format!("{:?}", o.version_id()))]
})
.await;
s.step(
"GetObjectTagging",
"malformed-version",
get_tags(TAGGED, Some("not-a-version")).send(),
none,
)
.await;
let absent_version = "00000000-0000-0000-0000-000000000001";
s.step("GetObjectTagging", "absent-version", get_tags(TAGGED, Some(absent_version)).send(), none)
.await;
s.step("GetObjectTagging", "nul-key", get_tags("a\0b", None).send(), none)
.await;
s.step(
"DeleteObjectTagging",
"nul-key",
c.delete_object_tagging().bucket(BUCKET).key("a\0b").send(),
none,
)
.await;
s.step(
"DeleteObjectTagging",
"then-empty",
c.delete_object_tagging().bucket(BUCKET).key(TAGGED).send(),
none,
)
.await;
s.step("GetObjectTagging", "after-delete", get_tags(TAGGED, None).send(), |o| {
vec![("tags", tags(o.tag_set()))]
})
.await;
let grantee = Grantee::builder()
.r#type(Type::CanonicalUser)
.id("owner")
.build()
.expect("grantee");
let policy = AccessControlPolicy::builder()
.owner(Owner::builder().id("owner").build())
.grants(Grant::builder().grantee(grantee).permission(Permission::Read).build())
.build();
let xml_grants = c.put_object_acl().bucket(BUCKET).key(TAGGED).access_control_policy(policy);
s.step("PutObjectAcl", "xml-grants", xml_grants.send(), none).await;
s.step(
"GetObjectAcl",
"after-xml-grants",
c.get_object_acl().bucket(BUCKET).key(TAGGED).send(),
|o| vec![("grants", grants(o.grants()))],
)
.await;
s.step("GetObjectAcl", "nul-key", c.get_object_acl().bucket(BUCKET).key("a\0b").send(), none)
.await;
s.step(
"GetObjectLegalHold",
"no-object-lock",
c.get_object_legal_hold().bucket(BUCKET).key(TAGGED).send(),
none,
)
.await;
s.step(
"GetObjectRetention",
"no-object-lock",
c.get_object_retention().bucket(BUCKET).key(TAGGED).send(),
none,
)
.await;
s.step(
"GetObjectTorrent",
"nul-key",
c.get_object_torrent().bucket(BUCKET).key("a\0b").send(),
none,
)
.await;
}
@@ -14,7 +14,50 @@
//! Differential cases of the `object_misc` batch (see `super`).
use super::super::Recorder;
use super::super::{Recorder, none};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
pub(super) async fn cases(_s: &mut Recorder, _c: &Client) {}
const BUCKET: &str = "dual-stack-object-misc";
const KEY: &str = "doomed.txt";
/// DeleteObject on a versioned bucket: the delete marker and its headers, the delete of that
/// marker by version, and the refusals the delete raises before storage.
pub(super) async fn cases(s: &mut Recorder, c: &Client) {
s.step("CreateBucket", "object-misc-setup", c.create_bucket().bucket(BUCKET).send(), none)
.await;
let enabled = VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build();
let versioning = c.put_bucket_versioning().bucket(BUCKET).versioning_configuration(enabled);
s.step("PutBucketVersioning", "object-misc-setup", versioning.send(), none)
.await;
let put = c.put_object().bucket(BUCKET).key(KEY).body(ByteStream::from_static(b"e2"));
s.step("PutObject", "object-misc-setup", put.send(), none).await;
let marker = s
.step(
"DeleteObject",
"creates-delete-marker",
c.delete_object().bucket(BUCKET).key(KEY).send(),
none,
)
.await
.and_then(|output| output.version_id().map(str::to_owned));
if let Some(marker) = marker {
let delete = c.delete_object().bucket(BUCKET).key(KEY).version_id(marker);
s.step("DeleteObject", "removes-delete-marker", delete.send(), none).await;
}
let delete = c.delete_object().bucket(BUCKET).key(KEY).version_id("not-a-version");
s.step("DeleteObject", "malformed-version", delete.send(), none).await;
let delete = c
.delete_object()
.bucket(BUCKET)
.key(KEY)
.customize()
.mutate_request(|request| {
request.headers_mut().insert("x-rustfs-force-delete", "maybe");
});
s.step("DeleteObject", "invalid-force-delete", delete.send(), none).await;
}
+3 -2
View File
@@ -29,10 +29,11 @@ use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::ServerSideEncryption;
use base64_simd::STANDARD as BASE64;
use http::header::{CONTENT_TYPE, HOST};
use jiff::Timestamp;
use md5::{Digest as Md5Digest, Md5};
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde_json;
use std::process::{Child, Command};
use std::time::Duration;
@@ -386,7 +387,7 @@ pub async fn create_key_with_specific_id(key_dir: &str, key_id: &str) -> Result<
"usage": "EncryptDecrypt",
"status": "Active",
"metadata": HashMap::<String, String>::new(),
"created_at": format!("{}[UTC]", chrono::Utc::now().to_rfc3339()),
"created_at": format!("{}[UTC]", Timestamp::now().strftime("%Y-%m-%dT%H:%M:%S%.fZ")),
"rotated_at": serde_json::Value::Null,
"created_by": "e2e-test",
"encrypted_key_material": BASE64.encode_to_string(key_data),
+2 -2
View File
@@ -31,8 +31,8 @@ pub mod chaos;
#[cfg(test)]
pub mod fake_s3_target;
// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
// Differential (rustfs/backlog#2740): two processes, the binary under test against
// itself or a released reference binary, the same SDK script for every S3 operation,
// compared on status, error code, allowlisted headers and typed body fields.
#[cfg(test)]
mod dual_stack_differential;
@@ -24,9 +24,9 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use http::header::HOST;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
/// Sends a SigV4-signed `GET` where the signature is computed over `sign_path`
@@ -21,9 +21,9 @@ mod tests {
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use tracing::info;
@@ -20,9 +20,9 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use tracing::info;
+4 -4
View File
@@ -23,17 +23,17 @@ use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use chrono::{Duration as ChronoDuration, Utc};
use flate2::{
Compression,
write::{GzEncoder, ZlibEncoder},
};
use http::HeaderValue;
use http::header::{CONTENT_TYPE, HOST};
use jiff::{SignedDuration, Timestamp};
use md5::{Digest as Md5Digest, Md5};
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::collections::HashMap;
use std::error::Error;
use std::io::Cursor;
@@ -42,8 +42,8 @@ use tokio::io::AsyncWriteExt;
use uuid::Uuid;
fn encode_post_policy(conditions: Vec<serde_json::Value>) -> String {
let expiration = (Utc::now() + ChronoDuration::hours(1))
.format("%Y-%m-%dT%H:%M:%S.000Z")
let expiration = (Timestamp::now() + SignedDuration::from_hours(1))
.strftime("%Y-%m-%dT%H:%M:%S.000Z")
.to_string();
let policy = serde_json::json!({
"expiration": expiration,
@@ -283,6 +283,7 @@ async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> Te
#[tokio::test]
async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
assert_single_node_cold_start_with_missing_volume().await?;
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
@@ -430,6 +431,138 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
Ok(())
}
async fn assert_single_node_cold_start_with_missing_volume() -> TestResult {
for drive_count in [2, 3, 4] {
let mut server = crate::chaos::DiskFaultHarness::new(drive_count).await?;
server.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{}", drive_count / 2));
server.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
server.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
server.set_env("RUSTFS_HEALTH_ENDPOINT_ENABLE", "true");
server.set_env("RUST_LOG", "warn");
server.start_server_at_read_quorum().await?;
let client = server.env.create_s3_client();
let client = Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
);
let bucket = format!("missing-volume-{drive_count}");
client.create_bucket().bucket(&bucket).send().await?;
let small = vec![17; 4096];
let large = vec![73; 8 * 1024 * 1024];
for (key, body) in [("small", &small), ("large", &large)] {
client
.put_object()
.bucket(&bucket)
.key(key)
.body(Bytes::copy_from_slice(body).into())
.send()
.await?;
for disk in 0..drive_count {
let census = crate::chaos::census_object_version_on_disk(server.disk_path(disk), &bucket, key, None)?;
assert!(census.is_complete(), "seed shard must exist before restart: {census:?}");
}
}
server.kill_server();
let missing = drive_count - 1;
server.take_disk_offline(missing)?;
server.start_server_at_read_quorum().await?;
for (key, body) in [("small", &small), ("large", &large)] {
assert_quorum_object_body(&client, &bucket, key, body).await?;
let head = client.head_object().bucket(&bucket).key(key).send().await?;
assert_eq!(head.content_length(), Some(i64::try_from(body.len())?));
}
let listing = client.list_objects_v2().bucket(&bucket).send().await?;
let keys: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(keys, ["large", "small"]);
let put = client
.put_object()
.bucket(&bucket)
.key("degraded-write")
.body(Bytes::from_static(b"degraded write").into())
.send()
.await;
if drive_count == 2 {
let error = put.expect_err("one of two drives cannot commit a new object");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
let error = client
.delete_object()
.bucket(&bucket)
.key("large")
.send()
.await
.expect_err("one of two drives cannot commit an object deletion");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_quorum_object_body(&client, &bucket, "large", &large).await?;
} else {
put?;
assert_quorum_object_body(&client, &bucket, "degraded-write", b"degraded write").await?;
client.delete_object().bucket(&bucket).key("degraded-write").send().await?;
}
assert!(
!server.disk_path(missing).exists(),
"degraded startup and S3 operations must leave the root absent"
);
if drive_count == 2 {
server.kill_server();
std::fs::create_dir(server.disk_path(missing))?;
server.start_server_at_read_quorum().await?;
for (key, body) in [("small", &small), ("large", &large)] {
assert_quorum_object_body(&client, &bucket, key, body).await?;
client.head_object().bucket(&bucket).key(key).send().await?;
}
assert_eq!(client.list_objects_v2().bucket(&bucket).send().await?.contents().len(), 2);
let error = client
.put_object()
.bucket(&bucket)
.key("empty-root-write")
.body(Bytes::from_static(b"must remain unavailable").into())
.send()
.await
.expect_err("an empty replacement cannot restore write quorum");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert!(
!server
.disk_path(missing)
.join(crate::storage_api::RUSTFS_META_BUCKET)
.join("format.json")
.exists(),
"read admission must not format the empty replacement"
);
}
server.kill_server();
server.bring_disk_online(missing)?;
server.start_server_at_read_quorum().await?;
for (key, body) in [("small", &small), ("large", &large)] {
assert_quorum_object_body(&client, &bucket, key, body).await?;
}
for key in ["degraded-write", "empty-root-write"] {
let error = client
.head_object()
.bucket(&bucket)
.key(key)
.send()
.await
.expect_err("rejected or deleted degraded write must stay absent after recovery");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(404));
}
client
.put_object()
.bucket(&bucket)
.key("recovery-write")
.body(Bytes::from_static(b"recovered write").into())
.send()
.await?;
assert_quorum_object_body(&client, &bucket, "recovery-write", b"recovered write").await?;
}
Ok(())
}
#[tokio::test]
async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
+1 -1
View File
@@ -17,9 +17,9 @@ use aws_sdk_s3::primitives::ByteStream;
use http::header::{CONTENT_TYPE, HOST};
use reqwest::StatusCode;
use rustfs_config::{ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, ENV_NOTIFY_ENABLE};
use rustfs_s3_types::Body;
use rustfs_signer::pre_sign_v4;
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
use s3s::Body;
use std::collections::HashMap;
use std::error::Error;
use time::OffsetDateTime;
+7 -7
View File
@@ -29,7 +29,7 @@ use aws_sdk_s3::types::{
DefaultRetention, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockLegalHold, ObjectLockLegalHoldStatus, ObjectLockMode,
ObjectLockRetention, ObjectLockRetentionMode, ObjectLockRule,
};
use chrono::{DateTime, Duration, Utc};
use jiff::{SignedDuration, Timestamp};
use tracing::info;
use crate::common::RustFSTestEnvironment;
@@ -126,10 +126,10 @@ pub async fn put_object_with_retention(
key: &str,
data: &[u8],
mode: ObjectLockRetentionMode,
retain_until: DateTime<Utc>,
retain_until: Timestamp,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
// AWS SDK requires UTC time without timezone offset (e.g., "2026-01-24T11:20:14Z")
let retain_until_str = retain_until.format("%Y-%m-%dT%H:%M:%SZ").to_string();
let retain_until_str = retain_until.strftime("%Y-%m-%dT%H:%M:%SZ").to_string();
let retain_until_datetime =
aws_sdk_s3::primitives::DateTime::from_str(&retain_until_str, aws_sdk_s3::primitives::DateTimeFormat::DateTime)?;
@@ -181,10 +181,10 @@ pub async fn put_object_retention(
key: &str,
version_id: Option<&str>,
mode: ObjectLockRetentionMode,
retain_until: DateTime<Utc>,
retain_until: Timestamp,
bypass_governance: bool,
) -> Result<(), Box<SdkError<PutObjectRetentionError>>> {
let retain_until_datetime = aws_sdk_s3::primitives::DateTime::from_secs(retain_until.timestamp());
let retain_until_datetime = aws_sdk_s3::primitives::DateTime::from_secs(retain_until.as_second());
let retention = ObjectLockRetention::builder()
.mode(mode.clone())
@@ -253,6 +253,6 @@ pub async fn delete_object_with_bypass(
}
/// Calculate a future retain_until date
pub fn future_retain_until(days: i64) -> DateTime<Utc> {
Utc::now() + Duration::days(days)
pub fn future_retain_until(days: i64) -> Timestamp {
Timestamp::now() + SignedDuration::from_hours(days * 24)
}
@@ -34,7 +34,7 @@ use aws_sdk_s3::types::{
CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
ObjectLockMode, ObjectLockRetentionMode,
};
use chrono::{DateTime, Duration, Utc};
use jiff::{SignedDuration, Timestamp};
use tracing::info;
/// Initialize test logging
@@ -67,7 +67,7 @@ async fn put_bucket_deny_policy(
}
fn retention_timestamp(days: i64) -> aws_sdk_s3::primitives::DateTime {
let retain_until = future_retain_until(days).format("%Y-%m-%dT%H:%M:%SZ").to_string();
let retain_until = future_retain_until(days).strftime("%Y-%m-%dT%H:%M:%SZ").to_string();
aws_sdk_s3::primitives::DateTime::from_str(&retain_until, aws_sdk_s3::primitives::DateTimeFormat::DateTime)
.expect("retention timestamp should parse")
}
@@ -118,13 +118,14 @@ where
);
}
fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc> {
fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> Timestamp {
let formatted = value
.fmt(DateTimeFormat::DateTime)
.expect("S3 timestamp should format as RFC3339");
DateTime::parse_from_rfc3339(&formatted)
.expect("S3 timestamp should parse as RFC3339")
.with_timezone(&Utc)
let normalized = formatted
.strip_suffix('Z')
.map_or(formatted.clone(), |prefix| format!("{prefix}+00:00"));
Timestamp::strptime("%Y-%m-%dT%H:%M:%S%.f%:z", normalized).expect("S3 timestamp should parse as RFC3339")
}
// ============================================================================
@@ -470,7 +471,7 @@ async fn test_get_object_retention_returns_configured_values() {
let bucket = "test-get-retention";
let key = "retained-object";
let retain_until = future_retain_until(30);
let retain_until_expected = retain_until.format("%Y-%m-%dT%H:%M:%SZ").to_string();
let retain_until_expected = retain_until.strftime("%Y-%m-%dT%H:%M:%SZ").to_string();
env.create_object_lock_bucket(bucket).await.unwrap();
@@ -1924,7 +1925,7 @@ async fn test_copy_object_retention_uses_destination_policy() {
.await
.unwrap();
let copy_started = Utc::now();
let copy_started = Timestamp::now();
client
.copy_object()
.copy_source(format!("{src_bucket}/{src_key}"))
@@ -1947,7 +1948,7 @@ async fn test_copy_object_retention_uses_destination_policy() {
assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
assert!(
retain_until < copy_started + Duration::days(3),
retain_until < copy_started + SignedDuration::from_hours(3 * 24),
"destination default retention should not inherit the source's longer retention"
);
@@ -2008,7 +2009,7 @@ async fn test_copy_object_retention_uses_destination_policy() {
.expect("explicit retain-until date should be present"),
);
assert!(
explicit_retain_until > Utc::now() + Duration::days(20),
explicit_retain_until > Timestamp::now() + SignedDuration::from_hours(20 * 24),
"explicit retention should override the shorter bucket default"
);
@@ -2088,7 +2089,7 @@ async fn test_multipart_default_retention_fixed_at_create() {
.await
.unwrap();
let create_started = Utc::now();
let create_started = Timestamp::now();
let create_output = client.create_multipart_upload().bucket(bucket).key(key).send().await.unwrap();
let upload_id = create_output.upload_id().unwrap();
@@ -2131,7 +2132,7 @@ async fn test_multipart_default_retention_fixed_at_create() {
assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
assert!(
retain_until < create_started + Duration::days(3),
retain_until < create_started + SignedDuration::from_hours(3 * 24),
"CompleteMultipartUpload should keep the default retention calculated at create time"
);
}
@@ -693,7 +693,7 @@ impl OdmTestEnv {
.uri(uri)
.header(http::header::HOST, authority)
.header("x-amz-content-sha256", rustfs_signer::constants::UNSIGNED_PAYLOAD)
.body(s3s::Body::empty())?;
.body(rustfs_s3_types::Body::empty())?;
let signed = rustfs_signer::sign_v4(request, 0, &self.rustfs.access_key, &self.rustfs.secret_key, "", "us-east-1");
let mut builder =
@@ -20,9 +20,9 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use http::header::{CONTENT_LENGTH, HOST, TRANSFER_ENCODING};
use rustfs_s3_types::Body;
use rustfs_signer::sign_v4;
use rustfs_utils::hash::EMPTY_STRING_SHA256_HASH;
use s3s::Body;
use std::error::Error;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -17,7 +17,7 @@
use async_trait::async_trait;
use rustfs_lock::{
LockClient, LockError, LockId, LockInfo, LockRequest, LockResponse, LockStats, LockStatus, LockType, Result,
CacheFenceStamp, LockClient, LockError, LockId, LockInfo, LockRequest, LockResponse, LockStats, LockStatus, LockType, Result,
types::{LockMetadata, LockPriority},
};
use rustfs_protos::proto_gen::node_service::{BatchGenerallyLockRequest, GenerallyLockRequest, PingRequest};
@@ -64,6 +64,7 @@ impl GrpcLockClient {
deadlock_detection: false,
suppress_contention_logs: false,
refresh_interval: None,
cache_fence_stamp_requested: false,
}
}
@@ -124,10 +125,15 @@ impl LockClient for GrpcLockClient {
// Check if the lock acquisition was successful
if resp.success {
Ok(LockResponse::success(
Self::build_lock_info(request, resp.lock_info),
std::time::Duration::ZERO,
))
let info = Self::build_lock_info(request, resp.lock_info);
match resp
.cache_fence_stamp
.as_deref()
.and_then(|encoded| CacheFenceStamp::decode(encoded).ok())
{
Some(stamp) => Ok(LockResponse::success_with_cache_fence_stamp(info, std::time::Duration::ZERO, stamp)),
None => Ok(LockResponse::success(info, std::time::Duration::ZERO)),
}
} else {
// Lock acquisition failed
Ok(LockResponse::failure(
@@ -160,7 +166,15 @@ impl LockClient for GrpcLockClient {
.enumerate()
.map(|(idx, request)| match resp.results.get(idx) {
Some(result) if result.success => {
LockResponse::success(Self::build_lock_info(request, result.lock_info.clone()), std::time::Duration::ZERO)
let info = Self::build_lock_info(request, result.lock_info.clone());
match result
.cache_fence_stamp
.as_deref()
.and_then(|encoded| CacheFenceStamp::decode(encoded).ok())
{
Some(stamp) => LockResponse::success_with_cache_fence_stamp(info, std::time::Duration::ZERO, stamp),
None => LockResponse::success(info, std::time::Duration::ZERO),
}
}
Some(result) => LockResponse::failure(
result
@@ -22,10 +22,11 @@ use rustfs_protos::{
models::PingBodyBuilder,
proto_gen::node_service::{
BatchGenerallyLockRequest, BatchGenerallyLockResponse, BatchReadVersionRequest, BatchReadVersionResponse,
GenerallyLockRequest, GenerallyLockResponse, GenerallyLockResult, PingRequest, PingResponse,
ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseReleaseResponse, ScannerPublicationLeaseRequest,
ScannerPublicationLeaseResponse, SnapshotLeaseMutationResponse, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
SnapshotLeaseRequest, SnapshotLeaseResponse, node_service_server::NodeService,
GenerallyLockRequest, GenerallyLockResponse, GenerallyLockResult, MrfLifecycleRequest, MrfLifecycleResponse, PingRequest,
PingResponse, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseReleaseResponse,
ScannerPublicationLeaseRequest, ScannerPublicationLeaseResponse, SnapshotLeaseMutationResponse,
SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse,
node_service_server::NodeService,
},
};
use std::pin::Pin;
@@ -42,6 +43,7 @@ fn lock_result_from_response(response: rustfs_lock::LockResponse) -> GenerallyLo
success: response.success,
error_info: response.error,
lock_info: response.lock_info.and_then(|info| serde_json::to_string(&info).ok()),
cache_fence_stamp: response.cache_fence_stamp.map(|stamp| stamp.encode().to_vec().into()),
..Default::default()
}
}
@@ -179,6 +181,7 @@ impl NodeService for MinimalLockNodeService {
success: result.success,
error_info: result.error,
lock_info: lock_info_json,
cache_fence_stamp: result.cache_fence_stamp.map(|stamp| stamp.encode().to_vec().into()),
..Default::default()
}))
}
@@ -438,6 +441,13 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn write_format_control(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::WriteFormatControlRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::WriteAllResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn compare_and_update_file(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::CompareAndUpdateFileRequest>,
@@ -563,6 +573,15 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
type RenameDataBatchStream = ResponseStream<rustfs_protos::proto_gen::node_service::RenameDataBatchResult>;
async fn rename_data_batch(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RenameDataBatchRequest>,
) -> Result<Response<Self::RenameDataBatchStream>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn rename_multipart_data(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RenameDataRequest>,
@@ -995,6 +1014,10 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn mrf_lifecycle(&self, _request: Request<MrfLifecycleRequest>) -> Result<Response<MrfLifecycleResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn background_heal_status(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::BackgroundHealStatusRequest>,
@@ -27,9 +27,9 @@ use http::Version;
use http::header::HOST;
use rcgen::generate_simple_self_signed;
use reqwest::{Certificate, Client, Response, StatusCode};
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use std::path::Path;
use std::process::Command;
+49 -6
View File
@@ -412,6 +412,7 @@ struct ManualTransitionRunReport {
skipped_delete_marker: u64,
skipped_directory: u64,
skipped_replication: u64,
skipped_already_transitioned: u64,
skipped_already_in_flight: u64,
skipped_queue_full: u64,
skipped_queue_closed: u64,
@@ -1214,9 +1215,16 @@ async fn test_manual_transition_run_black_box_semantics() -> TestResult {
assert!(!due.report.dry_run);
assert!(due.report.lifecycle_config_found);
assert_eq!(due.report.scanned, 1, "due report: {:#?}", due.report);
assert_eq!(due.report.eligible, 1, "due report: {:#?}", due.report);
// PUT can enqueue this object before the manual scan. Account for each
// terminal outcome without depending on how fast the tier worker runs.
assert_eq!(
due.report.enqueued + due.report.skipped_already_in_flight,
due.report.eligible + due.report.skipped_already_transitioned,
1,
"due report: {:#?}",
due.report
);
assert_eq!(
due.report.enqueued + due.report.skipped_already_in_flight + due.report.skipped_already_transitioned,
1,
"due report: {:#?}",
due.report
@@ -1225,12 +1233,34 @@ async fn test_manual_transition_run_black_box_semantics() -> TestResult {
assert_eq!(due.report.skipped_delete_marker, 0);
assert_eq!(due.report.skipped_directory, 0);
assert_eq!(due.report.skipped_replication, 0);
assert_eq!(due.report.skipped_queue_full, 0);
assert_eq!(due.report.skipped_queue_closed, 0);
assert_eq!(due.report.skipped_queue_timeout, 0);
assert_eq!(due.report.tier_failure, 0);
assert!(!due.report.cancelled);
assert!(!due.report.truncated_by_limit);
assert!(!due.report.truncated_by_duration);
wait_for_transition(&hot_client, MANUAL_DUE_BUCKET, MANUAL_DUE_KEY, StdDuration::from_secs(90)).await?;
let remote_count_after_due = cold_tier_object_count(&cold_client).await?;
assert_eq!(remote_count_after_due, 1, "due object must have exactly one remote copy");
// Once HEAD observes the committed transition, a repeated manual scan must
// report the completed object and leave both local reads and the tier intact.
let repeated = manual_transition_run(&hot, MANUAL_DUE_BUCKET, "manual-due/", false).await?;
assert_eq!(repeated.state, "completed", "repeated report: {:#?}", repeated.report);
assert_eq!(repeated.report.scanned, 1, "repeated report: {:#?}", repeated.report);
assert_eq!(repeated.report.eligible, 0, "repeated report: {:#?}", repeated.report);
assert_eq!(repeated.report.enqueued, 0, "repeated report: {:#?}", repeated.report);
assert_eq!(repeated.report.skipped_already_transitioned, 1);
assert_eq!(repeated.report.skipped_already_in_flight, 0);
assert_eq!(cold_tier_object_count(&cold_client).await?, remote_count_after_due);
let due_object = hot_client
.get_object()
.bucket(MANUAL_DUE_BUCKET)
.key(MANUAL_DUE_KEY)
.send()
.await?;
assert_eq!(due_object.body.collect().await?.into_bytes().as_ref(), b"manual due object");
hot_client.create_bucket().bucket(MANUAL_DRY_RUN_BUCKET).send().await?;
enable_bucket_versioning(&hot_client, MANUAL_DRY_RUN_BUCKET).await?;
@@ -2286,6 +2316,14 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
)
.await?;
// Configuration updates enqueue existing due objects. Observe their commit
// before paging so eligibility stays stable across the source restart.
for idx in 0..2 {
let key = format!("{MANUAL_CONTINUATION_PREFIX}obj-{idx:02}");
wait_for_transition(&hot_client, MANUAL_CONTINUATION_BUCKET, &key, StdDuration::from_secs(90)).await?;
}
assert_eq!(cold_tier_object_count(&cold_client).await?, 2);
let first = manual_transition_run_with_max(&hot, MANUAL_CONTINUATION_BUCKET, MANUAL_CONTINUATION_PREFIX, true, 1).await?;
assert_eq!(first.state, "partial", "first continuation page: {first:#?}");
assert_eq!(first.mode, "enqueue_only");
@@ -2293,8 +2331,10 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
assert_eq!(first.report.prefix, MANUAL_CONTINUATION_PREFIX);
assert!(first.report.dry_run);
assert_eq!(first.report.scanned, 1, "first continuation page: {first:#?}");
assert_eq!(first.report.eligible, 1, "first continuation page: {first:#?}");
assert_eq!(first.report.dry_run_eligible, 1, "first continuation page: {first:#?}");
assert_eq!(first.report.eligible, 0, "first continuation page: {first:#?}");
assert_eq!(first.report.dry_run_eligible, 0, "first continuation page: {first:#?}");
assert_eq!(first.report.skipped_already_transitioned, 1);
assert_eq!(first.report.enqueued, 0);
assert_eq!(first.report.tier_failure, 0);
assert!(!first.report.cancelled);
assert!(first.report.truncated_by_limit);
@@ -2321,12 +2361,15 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
.await?;
assert_eq!(second.state, "completed", "second continuation page: {second:#?}");
assert_eq!(second.report.scanned, 1, "second continuation page: {second:#?}");
assert_eq!(second.report.eligible, 1, "second continuation page: {second:#?}");
assert_eq!(second.report.dry_run_eligible, 1, "second continuation page: {second:#?}");
assert_eq!(second.report.eligible, 0, "second continuation page: {second:#?}");
assert_eq!(second.report.dry_run_eligible, 0, "second continuation page: {second:#?}");
assert_eq!(second.report.skipped_already_transitioned, 1);
assert_eq!(second.report.enqueued, 0);
assert_eq!(second.report.tier_failure, 0);
assert!(!second.report.cancelled);
assert!(!second.report.truncated_by_limit);
assert!(second.report.continuation_token.is_none());
assert_eq!(cold_tier_object_count(&cold_client).await?, 2);
Ok(())
}
@@ -59,7 +59,6 @@ use rustfs_madmin::{
AddServiceAccountReq, ListServiceAccountsResp, PeerInfo, PeerSite, ReplicateAddStatus, ReplicateEditStatus,
ReplicateRemoveStatus, SRRemoveReq, SRResyncOpStatus, SRStatusInfo, SiteReplicationInfo, SyncStatus,
};
use s3s::header::{X_AMZ_REPLICATION_STATUS, X_AMZ_TAGGING};
use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
use std::convert::Infallible;
@@ -78,6 +77,8 @@ use tokio::task::JoinHandle;
use tokio::task::JoinSet;
use tokio::time::{Duration, sleep, timeout};
const X_AMZ_REPLICATION_STATUS: http::HeaderName = http::HeaderName::from_static("x-amz-replication-status");
const X_AMZ_TAGGING: http::HeaderName = http::HeaderName::from_static("x-amz-tagging");
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
type BacklogMetricPoints = Arc<Mutex<BTreeMap<String, BTreeMap<String, (u64, f64)>>>>;
+1 -1
View File
@@ -32,9 +32,9 @@ mod tests {
use aws_sdk_s3::primitives::ByteStream;
use http::StatusCode;
use http::header::HOST;
use rustfs_s3_types::Body;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use tracing::{debug, info};

Some files were not shown because too many files have changed in this diff Show More