* fix(admin): bound IAM import archive expansion
MAX_IAM_IMPORT_SIZE caps the compressed upload at 10 MB, but every member of the
archive was then read with read_to_end into an unbounded Vec. Deflate ratios well
above 100:1 are easy to construct, so a small authorized upload could expand
without limit across the seven members ImportIam reads.
Add a shared expansion budget (MAX_IAM_IMPORT_EXPANDED_SIZE, 10x the compressed
cap) drawn down by every member, and route all seven reads through one helper
that reads a byte past the remaining budget to detect overrun. Sharing the budget
bounds the archive as a whole rather than letting each member spend the full
limit independently.
Covers R03-CAN-024 through R03-CAN-030 plus R04-CAN-077 (backlog #1471) — one
fix rather than seven, since all seven call sites were byte-identical.
* fix(kms): confine local key paths and refuse silent key replacement
Local KMS key identifiers arrive from request input — the `name` tag on CreateKey,
the `keyId` body field or query parameter on DeleteKey — and were joined onto
`key_dir` with no validation. An identifier such as `../../tmp/evil` escaped the
configured directory, making key creation a constrained arbitrary-file write and
`DeleteKey` with `force_immediate` a cross-directory delete.
Validate in `master_key_path` and make it fallible, so every filesystem path in
this backend inherits the guard: decode_stored_key, load_master_key,
save_master_key, create_key and delete_key all derive their paths there. The rule
is containment rather than a character allowlist, so identifiers already in use
keep resolving; only separators, NUL, absolute paths and non-single-component
forms are refused. Note `.` and `..` are contained rather than refused — the
`.key` suffix turns them into the ordinary filenames `..key` and `...key`.
Separately, `LocalKmsBackend::create_key` had no existence check, while the
sibling `KmsClient::create_key` has always had one. Since `save_master_key`
renames over its destination, creating a key under an existing name silently
replaced its material and destroyed the ability to decrypt everything wrapped
under it — and the backend path is the one the admin API uses. It now returns
KeyAlreadyExists, matching StaticKmsBackend.
Covers R03-CAN-072, R03-CAN-073 and R07-CAN-103 (backlog #1475). R03-CAN-073
needed no separate change: delete_key routes both its load and its remove_file
through master_key_path.
* fix(swift): bound SLO manifest reads to the 2 MiB manifest limit
The three Swift SLO handlers that load a stored manifest (handle_slo_get,
handle_slo_get_manifest, handle_slo_delete) read the `<object>.slo-manifest`
object to EOF with AsyncReadExt::read_to_end. That key is predictable and
writable through the ordinary object PUT path, so a tenant can replace the
manifest with an arbitrarily large object and then make the server allocate
its full size on every SLO GET, multipart-manifest=get, or
multipart-manifest=delete request - a memory amplification bounded only by
the stored object size (CWE-400 / CWE-770). The 2 MiB manifest limit that
handle_slo_put enforces was not applied on the read side.
Introduce MAX_SLO_MANIFEST_SIZE (the existing 2 MiB PUT limit, now a named
constant) and a shared read_manifest_bytes helper that reads through a
`take(limit + 1)` and rejects anything larger, so an oversized manifest is
refused instead of being buffered first. All three call sites go through the
helper. handle_slo_put now checks the size before parsing the JSON.
Regression tests: test_read_manifest_bytes_rejects_oversized_manifest and
test_read_manifest_bytes_stops_reading_oversized_manifest (which asserts the
reader is not consumed past the limit), plus a boundary test that a manifest
at exactly 2 MiB is still accepted.
* fix(protocols): authorize every object in FTPS/WebDAV recursive deletes
The FTPS and WebDAV gateways authorized only the container before a
recursive delete and then destroyed everything inside it without a
further check:
- FTPS RMD (and DELE on a bucket path ending in '/') cleared
s3:DeleteBucket, then delete_bucket_recursively listed the bucket and
deleted every object.
- WebDAV DELETE on a bucket did the same via its own
delete_bucket_recursively.
- WebDAV DELETE on a directory cleared s3:DeleteObject for the directory
marker key ("dir/") only, then listed that prefix and deleted every
child under it.
A principal holding s3:DeleteBucket (or s3:DeleteObject on a single
marker key) could therefore erase objects it had no s3:DeleteObject
permission for, and the operation reported success.
Deletion stays recursive - that is the expected behaviour for these
protocols - but each object now clears s3:DeleteObject on its own key
before it is removed, and the enumeration clears s3:ListBucket. A denial
aborts the whole operation with access denied rather than being skipped,
so the caller can never be told the delete succeeded while objects were
left behind or removed without authorization.
The test double gained shared-state cloning, delete_object/delete_bucket
call logs, and list/delete queue helpers so the regression tests can
observe that nothing is deleted once a deny lands.
* fix(server,ecstore): bound TLS handshakes and remote volume RPC waits
Three call sites let an unauthenticated client or a misbehaving peer hold
server resources with no deadline.
TLS listener (R03-CAN-035): process_connection awaited
`acceptor.accept(socket)` with no bound. A client that opens a TCP
connection and never finishes the handshake parks a Tokio task and a socket
forever, and the connection cap (RUSTFS_API_MAX_CONNECTIONS) is unlimited by
default, so nothing else sheds it. The handshake now runs under
accept_tls_with_deadline(), reusing the existing HTTP/1 header-read budget —
the established slow-client bound for the pre-request phase — and the
expiry is recorded through the same log/metric path as a handshake error,
under a new TIMEOUT failure kind.
Remote disk RPCs (R03-CAN-049, R03-CAN-050): list_volumes and delete_volume
passed Duration::ZERO, which execute_with_timeout treats as "no deadline",
so a peer that accepts the request and never answers stalls the coordinator
(and, for delete_volume, the bucket-deletion workflow). Both now pass
get_max_timeout_duration(), matching every sibling method in the file.
Regression tests: a silent TLS peer must be shed by the handshake deadline;
list_volumes/delete_volume against a peer that completes the TCP connect and
then goes silent must fail with DiskError::Timeout instead of hanging.
* fix(security): stop leaking signed headers and bound OIDC/KMS credentials
Three independent hygiene fixes found by the security review.
R03-CAN-018 (crates/signer): try_get_canonical_headers and get_signed_headers
logged the complete header map at DEBUG before signing. Runtime callers pass
session credentials and SSE-C key material through these headers, so anyone
able to raise the log level (or read DEBUG logs) recovered
X-Amz-Security-Token and SSE-C keys verbatim. The statements were debugging
leftovers with no operational value and are deleted rather than redacted.
R03-CAN-014 (crates/iam): the OIDC HTTP adapter buffered provider responses
with an unbounded Response::bytes(), so a configured, compromised or
attacker-pointed IdP endpoint could stream an arbitrarily large or endless
body into memory (the ValidateOidcConfig admin handler lets a ServerInfo
caller choose the endpoint). Responses are now read incrementally and fail
closed past MAX_OIDC_RESPONSE_SIZE, and the already SSRF-hardened client
builder gains request and connect timeouts so a stalled provider cannot pin
the calling task indefinitely.
R07-CAN-105 (helm): the Vault KMS token was serialized into the chart
ConfigMap, exposing it to every subject allowed to get ConfigMaps in the
namespace. It now renders into a dedicated Secret that the Deployment and
StatefulSet consume via envFrom; the Secret is separate from the main
credentials Secret so it also works when secret.existingSecret is set.
Regression tests:
- rustfs-signer: signing_never_logs_signed_header_material
- rustfs-iam: oidc_response_body_past_the_limit_is_rejected,
oidc_response_body_at_the_limit_is_accepted
- scripts/test_helm_templates.sh: KMS token must never render in plaintext
* fix(webdav): enforce body limit, request timeout and connection cap
The configured WebDAV maximum body size was enforced from Content-Length, so a
chunked request declared no length and bypassed it entirely. The configured
request timeout was never applied to the connection at all, and the accept loop
spawned a task per connection with no bound, so an unauthenticated client could
hold resources indefinitely and in unbounded number.
Enforce the limit on bytes actually read rather than the declared length, apply
the configured timeout to the request, and bound accepted connections with a new
RUSTFS_WEBDAV_MAX_CONNECTIONS (default 1024) surfaced in the config report.
Covers R03-CAN-051, R03-CAN-052, R03-CAN-067, R04-CAN-089, R05-CAN-094 and
R05-CAN-097 (backlog #1471, #1474).
* fix(security): stop STS credentials from crossing the parent trust boundary
Two related credential-boundary holes let a short-lived STS credential act
with the full, unrestricted authority of the long-term user it was minted
from.
AddUser (R03-CAN-021, CWE-269/863): should_check_deny_only relaxes the admin
policy check to deny-only when a Console/STS session targets the IAM user it
represents. Nothing then stopped that session from calling AddUser with its
own parent's access key, so the handler wrote an attacker-chosen secret key
and status over the parent's stored Credentials via create_user ->
save_user_identity. A session that expires in minutes became permanent
control of the account. AddUser now rejects any temp or service-account
requester whose resolved parent equals the target access key, resolving the
parent the same way should_check_deny_only does (parent_user field, else the
JWT `parent` claim, since some stores persist the parent only in the token).
FTPS/SFTP/WebDAV password auth (R04-CAN-086, CWE-287/862): these protocols
looked the access key up with check_key, which falls back to the STS account
cache, and then compared only the stored secret. An STS access key plus
secret therefore authenticated with no session token presented and no
session-policy claims applied - the holder got the parent's full permissions.
Password authentication now rejects temporary credentials before the secret
comparison. The discriminator is is_temp() && !is_service_account(), the same
one IamCache::update_user_with_claims uses to route an identity into the STS
cache, so service accounts - which resolve policy from stored IAM state
rather than a client-presented token - keep working over these protocols.
Regression tests cover both predicates and pin the guards to their call
sites so neither can be dropped without a test failure.
docs/testing/security-regressions.md requires every fixed advisory to map to a
named, greppable regression test. Rename the tests added with the fixes to carry
their advisory id so `rg -i ghsa` finds them, and add the four rows to the
advisory -> test map.
Adds the FTPS MKD regression test that was missing. It primes the dummy backend
with a successful create_bucket, so the assertion distinguishes "denied at the
authorization boundary" from "backend refused" — without the queued success an
unconfigured create_bucket fails on its own and the test would pass even with
the authorization check removed. Verified it fails when the check is reverted.
DummyBackend gains a Debug impl (FtpsDriver's trait bounds require it) and a
queue_create_bucket_ok helper.
Records in the CI-execution map why ghsa_g3vq_* runs in the default pass despite
sitting behind the ftps feature: the rustfs crate defaults to ["ftps", "webdav"]
and cargo unifies features across the workspace build, so the test executes
there even though `cargo test -p rustfs-protocols` alone would skip it.
* fix(policy): quantify negated string conditions per value
ForAllValues:/ForAnyValue: negated string operators computed the positive
quantified match and then negated the aggregate. That yields NOT(all match)
and NOT(any match), which is the semantics of the *other* quantifier, so
ForAllValues:StringNotEquals and ForAnyValue:StringNotEquals were exactly
transposed. The same applied to StringNotEqualsIgnoreCase, StringNotLike,
ArnNotEquals and ArnNotLike.
Introduce an explicit Quantifier and push negation into the per-value
predicate for the qualified forms, so ForAllValues requires every request
value to satisfy the operator and ForAnyValue requires at least one.
Unqualified operators keep negating the aggregate, preserving AWS
single-valued-key semantics. Absent keys now follow AWS: ForAllValues is
vacuously satisfied, ForAnyValue is not.
A request value set that is fully contained in or fully disjoint from the
policy set cannot distinguish the two quantifiers, which is why the existing
cases missed this; the new tests use partially overlapping sets.
* fix(auth): keep request headers out of server-derived condition keys
get_condition_values folded every remaining request header into the policy
condition map. HeaderMap lowercases header names, and the server-derived keys
userid, username, principaltype, versionid and signatureversion are lowercase
too, so a header of the same name collided with them. The collision branch used
extend(), and non-quantified string operators match if ANY value in the vector
matches, so sending `userid: admin` was enough to satisfy a condition on
aws:userid. The jwt:/ldap: claim keys were reachable the same way whenever the
credential carried no such claim.
groups was worse than an append: the header loop ran before the cred.groups
block, which is gated on !args.contains_key("groups"), so a `groups:` header
both injected a value and suppressed the credential's real group list.
Resolve claims and group membership before merging headers, then skip any header
naming a key the server already derived or a well-known identity/context key.
The reserved set comes from KeyName so it tracks the key registry; s3:x-amz-*
keys stay mergeable because they mirror request headers by design.
* fix(access): gate the ListBucketVersions fallback on public-access checks
An anonymous request for ListObjectVersions that the bucket policy does not
grant directly falls back to re-checking the grant as s3:ListBucket. That
fallback returned Ok(()) straight away, skipping the two gates the direct grant
passes through: deny_anonymous_table_data_plane_if_needed and the
RestrictPublicBuckets check on the bucket's public-access block.
So a bucket whose policy allows anonymous s3:ListBucket kept serving anonymous
version listings after an operator enabled RestrictPublicBuckets, even though
the equivalent GetObject was correctly denied.
Fold the fallback into policy_allowed so both routes reach the same gates.
* fix(ftps): authorize MKD against the CreateBucket boundary
FTPS MKD creates a bucket but ran no authorization check, so any principal that
could open an FTPS session could create buckets regardless of policy. Every
other operation in this driver authorizes first — LIST, RETR, STOR, DELE and
RMD all call authorize_operation — and the WebDAV gateway checks
S3Action::CreateBucket on the equivalent path.
Add the matching check so MKD clears the same boundary as an S3 CreateBucket.
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
* chore(deps): tighten crate dependency features
Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): trim hyper-rustls features
Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* fix(swift): replace assert!(false) with panic! in expiration_worker test
clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.
* fix(ecstore): add missing ctx field to multipart lock test store
The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
The FTPS and WebDAV authentication handlers compared the client-supplied secret
key against the stored secret with `String::eq`, which short-circuits on the
first differing byte. A network attacker who knows (or enumerates) a valid
access key can recover the secret key byte-by-byte via response-timing analysis;
neither path is rate limited.
Switch both to a constant-time comparison using `subtle::ConstantTimeEq`, the
same primitive the SFTP handler and `rustfs/src/auth.rs::constant_time_eq`
already use. `subtle` is added to the `ftps` and `webdav` feature dependency
sets (it was previously gated on `sftp` only).
Addresses GHSA-3p3x-734c-h5vx.
fix(protocols): allow clippy type_complexity in test mock struct
The MockExpirationObjectBackend test struct uses a nested generic type
that triggers clippy::type_complexity. Add #[allow(clippy::type_complexity)]
since this is test-only code where the type is inherent to the mock design.
Co-authored-by: houseme <housemecn@gmail.com>
fix(protocols): factor swift expiration mock result type into alias
The swift feature clippy matrix on main fails with clippy::type_complexity on the MockExpirationObjectBackend test struct introduced with the expiration worker tests, blocking CI for every open PR. Introduce a MetadataResult type alias in the test module; no behavior change.
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
security(swift): remove placeholder SSE module (backlog#646)
The Swift `encryption` module was a non-functional stub: `encrypt_data`
returned the plaintext unchanged while labeling it AES-256-GCM in the
object metadata, and `generate_iv` derived the IV from a timestamp
rather than a CSPRNG. It had no production caller (only a `pub mod`
declaration and one integration test), so wiring it in as-is would have
silently shipped plaintext advertised as ciphertext.
We are not supporting Swift server-side encryption for now, so remove
the module outright rather than keep a dangerous stub around:
- delete crates/protocols/src/swift/encryption.rs
- drop `pub mod encryption;` from swift/mod.rs
- remove the encryption case (and unused import) from the swift
integration test
The module reached main dubiously: it was introduced together with the
whole Swift API in commit 86e93624 ("fix(heal): canonicalize scanner
object-dir repairs (#3864)"), a 1665-file squash whose PR description
only covered the heal change and never mentioned Swift or SSE.
Verified: cargo fmt; cargo test -p rustfs-protocols --features swift
--test swift_simple_integration (10 passed); arch guardrail scripts pass.
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
Zhengchao An