fix(s3): populate CopyObject response checksums and persist them (#4996)
A CopyObject that requested ChecksumAlgorithm=SHA256 applied the copy but returned no CopyObjectResult.ChecksumSHA256, and a later checksum-mode HEAD/GET on the destination returned nothing: execute_copy_object never read the requested algorithm, never gave the destination write a checksum, and built CopyObjectResult with ..Default::default() (all checksum fields None).
Give the destination object a checksum on the copy path. When the caller requests an algorithm, compute it fresh over the copied plaintext (the hasher sits on the innermost reader so it digests plaintext, before compression/encryption wrap it) so it persists into the object's checksum and a checksum-mode HEAD/GET returns it. When no algorithm is requested, carry the source object's stored checksum over unchanged — the copy does not transform the plaintext, so re-hashing would be wasted work and would flatten a multipart composite value. Fill CopyObjectResult from the persisted destination checksum decoded exactly the way GetObject/HeadObject do, so the response value is identical to a later checksum-mode HEAD/GET.
Add e2e regression tests: requested SHA256 is computed, returned, and HEAD-consistent; a no-algorithm copy preserves and reports the source SHA256; and a requested CRC32 over a SHA256 source is computed fresh (equals a reference CRC32 PUT), overrides the source algorithm, and is not inherited.
Fixes#4996
Follow-up to backlog#1191 (optional sub-item). The main accept loop
spawned one task per socket with no global bound, so a connection flood
could exhaust file descriptors and memory and take existing traffic
down with it.
- New RUSTFS_API_MAX_CONNECTIONS (default 0 = unlimited, no semaphore
constructed, accept loop unchanged).
- When set, the loop acquires an owned semaphore permit BEFORE
accept(): at saturation it simply stops accepting and lets the
kernel backlog absorb bursts (TCP-native backpressure) instead of
accept-then-close churn. The permit moves into the connection task
and is released by RAII on any exit path, including TLS handshake
failures.
- Saturation is observable via the
rustfs_http_server_connection_cap_saturated_total counter, a
connection_cap_state startup event, and a per-wait debug event;
shutdown stays responsive while parked on the semaphore.
- The cap covers everything on the main listener (S3, admin, console,
internode gRPC); the constant docs carry sizing guidance.
- E2E coverage: ten sequential Connection-close requests against cap 2
prove permits never leak; two stalled connections saturating cap 2
leave a third unserved until their permits are released, after which
the queued request is accepted and answered.
test(e2e): add socket-level network fault-injection proxy (backlog#1325)
Add `FaultProxy`, an in-process async TCP proxy for black-box cluster E2E tests, under `crates/e2e_test/src/fault_proxy.rs`. It binds a random local port, forwards every accepted connection to a fixed target, and lets a test flip the wire behaviour at runtime.
Fault modes: `Pass` (transparent), `Latency(Duration)` (delay each forwarded chunk), `Blackhole` (accept but forward nothing either way and never respond), and `Partition(Direction)` (block exactly one direction while the other keeps flowing). Modes are switchable at runtime via `set_mode`, and `shutdown` stops the accept loop and drops the listener so the bound port is released.
This is the black-box counterpart to the in-process white-box hooks (rename barrier, disk call counters), which cannot reach across the process boundary into a spawned RustFS server. It serves the lock-plane one-way partition and accept-then-blackhole-peer acceptance for #1312/#1319 and the cross-process replay/tamper seam for #1327. Wiring it into the cluster harness (expose a node port through a proxy) plus the 5GiB / nightly CI-lane budget are follow-ups, since the harness multi-drive / 2-pool work lands separately (rustfs#4937); this block stays independent and self-tested.
Self-tests run against a loopback echo server that records received bytes, covering: pass round-trip identity, latency lower-bound delay, blackhole no-response-no-panic, one-way partition in both directions (request-path vs return-path), runtime mode switching on a live connection, and port release after shutdown. Timing assertions use loose lower bounds and bounded read timeouts only, never fixed-sleep absolute-value checks, to stay non-flaky. No product code changes; the module is `#[cfg(test)]`-gated.
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
backlog#1183 tracks flipping the default GET data path from the legacy
tokio::io::duplex double-copy to the zero-duplex codec-streaming fast path.
That flip is gated behind RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED /
..._HEADER_COMPAT_CONFIRMED, which need empirical evidence the two paths are
byte- and header-identical.
Add an e2e regression net that runs the same object matrix twice against the
same on-disk EC shards, changing only the codec-streaming env gates: phase A
(default) takes the legacy duplex path, phase B (gates opened) takes codec
streaming. It asserts byte-for-byte (sha256) and header-for-header equality
across inline / small / multi-block (1.5M/3M/5M+) / multipart objects, plus a
ranged GET (which falls back to legacy by gate design). Path confirmation is
not assumed: the legacy path logs "Created duplex pipe ..." per full GET, so
the test counts that marker per phase and asserts the codec phase created zero
duplex pipes, proving the fast path actually ran rather than silently falling
back to the path it is compared against.
To capture the child server's logs for that assertion, add an optional
RustFSTestEnvironment.capture_log_path (default None = inherit stdio,
backward compatible) that redirects the spawned server's stdout+stderr to a file.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
test(admin-auth): unit + e2e coverage for the admin authorization gate
Adds the first tests for the central admin authorization gate
`rustfs/src/admin/auth.rs`, which previously had zero coverage
(backlog#1151 sec-4, master plan #1155).
Unit tests (rustfs/src/admin/auth.rs):
- Refactor the two `validate_admin_request*` entry points to share a
single generic decision core `evaluate_admin_actions<S: Store>` /
`check_admin_request_auth<S: Store>` (removes the duplicated
action-loop and makes the gate testable without a running cluster).
- Cover: owner/root credential allowed; authenticated non-admin
credential denied with AccessDenied; missing credential denied; and
the multi-action loop grants on any permitted action, denies when
none pass. Backed by an in-memory empty IamSys so the owner path
short-circuits to allow and every other principal resolves to deny.
E2E (crates/e2e_test/src/admin_auth_test.rs, raw SigV4 over HTTP, no
awscurl dependency):
- non_admin_credential_denied_on_admin_api: a limited IAM user gets
403 AccessDenied on GET /rustfs/admin/v3/info while root succeeds.
- root_credential_rotation_takes_effect: restart with rotated
--access-key/--secret-key; new credential works and the stale one is
rejected, on both the admin plane and the S3 data plane.
- default_credentials_emit_startup_warning: booting with the default
rustfsadmin credentials emits the default-credentials warning.
Refs rustfs/backlog#1151 (sec-4), #1155.
test(e2e): add negative header-SigV4 rejection suite (backlog#1151 sec-1)
RustFS delegates SigV4 verification to the s3s dependency, so nothing in
this repo pins OUR end-to-end wiring of it. Add negative_sigv4_test.rs
sending REJECTED header-SigV4 requests against a live RustFSTestEnvironment
and asserting the HTTP status plus the S3 error code XML:
- valid_header_sigv4_request_succeeds (positive control so the negatives
cannot pass for the wrong reason)
- tampered_signature_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- wrong_secret_key_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- tampered_payload_is_rejected (body != signed x-amz-content-sha256) -> not 200
- skewed_date_returns_request_time_too_skewed (>15min) -> 403 RequestTimeTooSkewed
- malformed_authorization_header_returns_clean_4xx (present-not-missing) -> 4xx, never 5xx
Signatures are hand-built via rustfs_signer::request_signature_v4 primitives
(get_signing_key/get_signature/get_scope) so the test controls the timestamp,
secret, signed payload hash, and final signature bytes. Missing-credential
negatives are intentionally not duplicated (covered by multipart_auth_test /
anonymous_access_test).
Refs backlog#1151 (sec-1), master plan backlog#1155.
Adds an S3-API-level e2e regression net proving that a large-object GET on a
degraded EC set never returns a silently truncated body under a full
Content-Length -- the historical "unexpected EOF" bug fixed on main by
rustfs#4594 (short-body GetObject stream -> UnexpectedEof), rustfs#4560
(in-place per-part legacy degradation for the lazy multipart reader), and
rustfs#4585 (DARE package-boundary truncation detection). Those fixes each
ship a *unit* regression; this covers the layer they do not -- the full HTTP
GET path streaming a real body reconstructed from real on-disk EC shards.
New file crates/e2e_test/src/degraded_read_eof_regression_test.rs, single-node
4-disk (EC 2+2) DiskFaultHarness, three scenarios:
(a) one disk offline: a 6 MiB single object (>=2 EC stripes) and a 3x5 MiB
multipart object GET back byte-identical with the correct Content-Length.
(b) mid-stream bitrot within quorum: 2 of 4 shards corrupted mid-file on a
large multipart object still reconstructs the full, hash-matching body.
(c) beyond read quorum (the heart of the net): 3 of 4 shards corrupted
mid-file -- the read MUST fail cleanly (non-2xx or a mid-stream body
error), NEVER close with a truncated body under the full Content-Length.
The shared get_checked() helper panics on the forbidden outcome (a clean 2xx
whose collected body is shorter than the advertised Content-Length), so the
truncation bug can never be silently tolerated.
CI placement: these spawn a 4-disk server per test and are resource-heavy, so
they stay OUT of the fast PR e2e-smoke filter. A new e2e-reliability nextest
test-group (max-threads=1) serializes them (and the existing
reliability_disk_fault_test) across nextest's process boundary; ci-7's nightly
full-e2e run picks them up. Visible via `cargo nextest list -p e2e_test`.
Refs rustfs/backlog#1150 (dist-13), rustfs/backlog#1155, rustfs#4594,
rustfs#4560, rustfs#4585, rustfs#2955.
Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.
Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:
- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
restart
Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cxymds