The Test and Lint lane intermittently stalls until the inner 75m timeout
kills the cargo process group (issue #5394). The evidence collected since
#5402 shows the stall can begin in the build phase before any test runs
(run 30449339653: last output at minute 4 of the nextest step, then 71
silent minutes until SIGTERM), but the post-mortem pgrep always reports
nothing because GNU timeout has already terminated the whole process
group by the time it runs.
Add a background sampler to the nextest step that appends system and
process snapshots (loadavg, PSI, memory, disk, top-RSS processes,
cargo/rustc/linker/build-script processes, D-state processes) to the
existing test-and-lint artifact every 60 seconds, and record kernel
OOM/kill events in the post-mortem diagnostics. The last samples before
a timeout identify the wedged process or the resource pressure that
caused the stall. This only instruments the failure mode where the
runner survives; jobs whose runner disappears entirely still upload no
artifacts and need runner-pool-side logs.
Refs #5394
* fix: keep hotpath gate csv paths clean
Redirect the enhanced bench driver's verbose output away from command substitution so run_hotpath_warp_ab.sh records only the candidate cell path before passing baseline_compare.csv files to the gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(utils): reduce highway hash key setup
Cache the parsed HighwayHash keys as compile-time constants and add a criterion benchmark for the PUT hash hotpath.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The LoadBucketMetadata peer-notification handler loaded bucket metadata
with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and
unconditionally cached the result. On a transient read-quorum dip during
a reload notification, a peer cached an authoritative "no Object Lock"
default for a lock-enabled bucket, disabling the batch-delete retention
gate (object_lock_delete_check_required) on that node until the next
refresh, and wiping its bucket-target/durability sync state.
Production changes:
- New BucketMetadataSys::reload_from_store (metadata_sys::
reload_bucket_metadata): the peer reload path uses the presence-aware
loader and installs only metadata actually read from persisted
storage. A load miss returns an error (surfaced to the notifying peer
as success=false) and leaves the cache untouched; deletion still
propagates only through the dedicated DeleteBucketMetadata
notification.
- The reload runs under the outer metadata-sys write guard, load
included, mirroring update(): every other cache installer holds that
lock, so a reload snapshot can never land after - and roll back - a
newer concurrent install (the stale-load lost-update from the
review), and the install-plus-registry-sync sequence stays atomic
against concurrent removes and reloads (previously only the set call
was write-guarded, with the load outside any lock).
- The peer-visible miss error is a fixed string: the notifying peer
substring-matches error text against network-failure needles
(is_network_like_error), so interpolating a bucket name (e.g. a legal
bucket literally named "unavailable") could mark a healthy peer
offline.
- get_config's lazy insert routes through set(), picking up the
negative-cache invalidation.
An earlier draft instead guarded set() with a per-config updated_at
freshness comparison. Adversarial validation rejected it (three roles
independently): update_config stamps with the handling node's wall
clock, so within the skew the cluster already tolerates (+/-300s RPC
auth window) a config rewritten with an earlier stamp - e.g. revoking a
public-read policy through a second node, or any same-field rewrite
after an NTP step-back - would be skipped by every peer forever,
silently pinning the revoked permissive config with no re-convergence
path (the 15-minute refresh also routed through the guard). Race
staleness is second-scale while skew is minute-scale, so no tolerance
bound can separate them; the write-guard serialization closes the same
race without clocks and preserves the refresh loop's unconditional
converge-to-disk property, which is the cluster's self-healing
mechanism.
Startup audit (BucketMetadataSys::init): concurrent_load's
insert-if-vacant still installs a fabricated default when a transient
miss hits at boot - indistinguishable from a legacy bucket without a
metadata file at this layer - bounded by the next successful persisted
load. Making the object-lock gate fail closed on such entries is filed
as a follow-up, alongside the bare "unavailable" needle in
is_network_like_error and the Swift cache-only metadata writes.
Tests:
- bucket::metadata_sys::tests::
peer_reload_never_caches_fabricated_defaults_as_authoritative:
miss installs nothing / miss keeps the existing entry intact
(asserting the dedicated non-persisted error) / persisted reload
converges the cache over a stale entry.
- node_service::tests::
test_load_bucket_metadata_failure_skips_scanner_maintenance:
a failed reload reports failure and does not advance scanner
maintenance activity (previously recorded even on a miss).
- The handler success path stays uncovered at the RPC layer (needs an
isolated global object layer, like the pre-existing ignored test);
the composition is pinned at the sys level instead.
Verification:
- cargo fmt --check and cargo clippy --lib --tests clean on
rustfs-ecstore and rustfs.
- Targeted suites green; full cargo test -p rustfs-ecstore --lib:
3198/3200 with two parallelism-sensitive lock-test flakes from the
known baseline (pass in isolation; a different pair flakes per run).
- Adversarial validation (high-risk tier, all seven roles as
independent parallel reviewers) run per AGENTS.md; all findings
fixed or rebutted with evidence, three out-of-scope findings filed
as follow-up tasks.
Co-authored-by: houseme <housemecn@gmail.com>
Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.
Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.
Persisting these writes required hardening the paths that now produce
durable state:
- Account metadata writes validate account ownership. This metadata
holds the account's TempURL signing key, so an unauthenticated write
for someone else's account would have become a durable, cluster-wide
takeover of that account's pre-signed URLs. Reads stay open because
TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
metadata loader's "no metadata on disk" default would be persisted,
creating an orphan metadata file and caching a fabricated default as
authoritative.
- Container and account metadata are size- and count-limited, reusing
the Swift limits object metadata already enforces; these tags land in
the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
unreadable, instead of merging onto an empty set and wiping the
container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
now carry real disk and quorum detail.
The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation
* feat: enhance local SSE DEK handling with JSON envelope format and versioning
Align inline fast path cluster tests with the current tier mutation and manual transition contracts while keeping msgpack compat assertions focused on observed traffic.
Co-authored-by: heihutu <heihutu@gmail.com>
DeleteObjects rebuilt ObjectOptions inside its per-key phase-1 loop, and
del_opts re-fetched the bucket versioning configuration on every call: a
1000-key request paid 1001 identical metadata-sys lookups, each taking the
global bucket-metadata read lock and cloning a VersioningConfiguration.
Split del_opts into the existing async entry point plus a synchronous
del_opts_with_versioning that takes an already-resolved configuration, and
reuse the request-level version_cfg the handler had already fetched.
This also removes a snapshot inconsistency. The skip-stat decision
(can_skip_delete_objects_pre_stat) and the post-delete accounting already
derived from the request-level snapshot while del_opts re-read the config
per key, so a PutBucketVersioning committing mid-request could give a key
opts.versioned=true while the batch delete still ran with versioned=false:
the object was deleted outright with replication state attached to a delete
marker that was never created, and the advisory object-lock pre-check was
skipped on a false premise. All three now derive from one snapshot.
GET, HEAD and DeleteObject additionally establish bucket existence before
building ObjectOptions, so a request naming a nonexistent bucket no longer
performs bucket-metadata work first. GET keeps its cheap request-shape
validations (key, range, partNumber) ahead of the existence check so
InvalidArgument still wins over NoSuchBucket for malformed requests.
Bucket validation moves to validate_bucket_exists, which takes an explicit
store and shares the existing 5s-TTL cache. GET, PUT, HEAD and DeleteObject
now resolve the store through the request-bound server context
(backlog#1052 S6) instead of the process-global handle: get_validated_store
resolves the first-published global AppContext, so in an embedded
multi-instance process a request to the second server operated on the first
server's store. Remaining global-handle call sites in bucket_usecase and
select_object are left for a follow-up.
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Add a reusable evidence collector for manual transition tier-failure attribution samples and wire the runbook tests to cover its dry-run and argument guards.
Co-authored-by: heihutu <heihutu@gmail.com>
Recover accepted manual-transition tasks from the durable task journal when worker-result markers are missing after a restart.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a script-test entry for the manual transition rollout and soak runbook generators. The test covers ratio fail-fast behavior, generated status/metrics/journal checks, and runnable dry-run artifacts.
Verification:
- ./scripts/test_manual_transition_runbooks.sh
- make script-tests
Co-authored-by: heihutu <heihutu@gmail.com>
Add a reusable offline rebuild path for pinned paired ABBA benchmark evidence so corrected summaries can be regenerated from raw warp logs after parser fixes.
Record ACK-contract comparability explicitly in the rebuilt product comparison output, preventing relaxed RustFS ACK results from being treated as direct strict MinIO comparisons.
Co-authored-by: heihutu <heihutu@gmail.com>
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
* fix(admin): bound IAM import archive expansion
MAX_IAM_IMPORT_SIZE caps the compressed upload at 10 MB, but every member of the
archive was then read with read_to_end into an unbounded Vec. Deflate ratios well
above 100:1 are easy to construct, so a small authorized upload could expand
without limit across the seven members ImportIam reads.
Add a shared expansion budget (MAX_IAM_IMPORT_EXPANDED_SIZE, 10x the compressed
cap) drawn down by every member, and route all seven reads through one helper
that reads a byte past the remaining budget to detect overrun. Sharing the budget
bounds the archive as a whole rather than letting each member spend the full
limit independently.
Covers R03-CAN-024 through R03-CAN-030 plus R04-CAN-077 (backlog #1471) — one
fix rather than seven, since all seven call sites were byte-identical.
* fix(kms): confine local key paths and refuse silent key replacement
Local KMS key identifiers arrive from request input — the `name` tag on CreateKey,
the `keyId` body field or query parameter on DeleteKey — and were joined onto
`key_dir` with no validation. An identifier such as `../../tmp/evil` escaped the
configured directory, making key creation a constrained arbitrary-file write and
`DeleteKey` with `force_immediate` a cross-directory delete.
Validate in `master_key_path` and make it fallible, so every filesystem path in
this backend inherits the guard: decode_stored_key, load_master_key,
save_master_key, create_key and delete_key all derive their paths there. The rule
is containment rather than a character allowlist, so identifiers already in use
keep resolving; only separators, NUL, absolute paths and non-single-component
forms are refused. Note `.` and `..` are contained rather than refused — the
`.key` suffix turns them into the ordinary filenames `..key` and `...key`.
Separately, `LocalKmsBackend::create_key` had no existence check, while the
sibling `KmsClient::create_key` has always had one. Since `save_master_key`
renames over its destination, creating a key under an existing name silently
replaced its material and destroyed the ability to decrypt everything wrapped
under it — and the backend path is the one the admin API uses. It now returns
KeyAlreadyExists, matching StaticKmsBackend.
Covers R03-CAN-072, R03-CAN-073 and R07-CAN-103 (backlog #1475). R03-CAN-073
needed no separate change: delete_key routes both its load and its remove_file
through master_key_path.
* fix(swift): bound SLO manifest reads to the 2 MiB manifest limit
The three Swift SLO handlers that load a stored manifest (handle_slo_get,
handle_slo_get_manifest, handle_slo_delete) read the `<object>.slo-manifest`
object to EOF with AsyncReadExt::read_to_end. That key is predictable and
writable through the ordinary object PUT path, so a tenant can replace the
manifest with an arbitrarily large object and then make the server allocate
its full size on every SLO GET, multipart-manifest=get, or
multipart-manifest=delete request - a memory amplification bounded only by
the stored object size (CWE-400 / CWE-770). The 2 MiB manifest limit that
handle_slo_put enforces was not applied on the read side.
Introduce MAX_SLO_MANIFEST_SIZE (the existing 2 MiB PUT limit, now a named
constant) and a shared read_manifest_bytes helper that reads through a
`take(limit + 1)` and rejects anything larger, so an oversized manifest is
refused instead of being buffered first. All three call sites go through the
helper. handle_slo_put now checks the size before parsing the JSON.
Regression tests: test_read_manifest_bytes_rejects_oversized_manifest and
test_read_manifest_bytes_stops_reading_oversized_manifest (which asserts the
reader is not consumed past the limit), plus a boundary test that a manifest
at exactly 2 MiB is still accepted.
* fix(protocols): authorize every object in FTPS/WebDAV recursive deletes
The FTPS and WebDAV gateways authorized only the container before a
recursive delete and then destroyed everything inside it without a
further check:
- FTPS RMD (and DELE on a bucket path ending in '/') cleared
s3:DeleteBucket, then delete_bucket_recursively listed the bucket and
deleted every object.
- WebDAV DELETE on a bucket did the same via its own
delete_bucket_recursively.
- WebDAV DELETE on a directory cleared s3:DeleteObject for the directory
marker key ("dir/") only, then listed that prefix and deleted every
child under it.
A principal holding s3:DeleteBucket (or s3:DeleteObject on a single
marker key) could therefore erase objects it had no s3:DeleteObject
permission for, and the operation reported success.
Deletion stays recursive - that is the expected behaviour for these
protocols - but each object now clears s3:DeleteObject on its own key
before it is removed, and the enumeration clears s3:ListBucket. A denial
aborts the whole operation with access denied rather than being skipped,
so the caller can never be told the delete succeeded while objects were
left behind or removed without authorization.
The test double gained shared-state cloning, delete_object/delete_bucket
call logs, and list/delete queue helpers so the regression tests can
observe that nothing is deleted once a deny lands.
* fix(server,ecstore): bound TLS handshakes and remote volume RPC waits
Three call sites let an unauthenticated client or a misbehaving peer hold
server resources with no deadline.
TLS listener (R03-CAN-035): process_connection awaited
`acceptor.accept(socket)` with no bound. A client that opens a TCP
connection and never finishes the handshake parks a Tokio task and a socket
forever, and the connection cap (RUSTFS_API_MAX_CONNECTIONS) is unlimited by
default, so nothing else sheds it. The handshake now runs under
accept_tls_with_deadline(), reusing the existing HTTP/1 header-read budget —
the established slow-client bound for the pre-request phase — and the
expiry is recorded through the same log/metric path as a handshake error,
under a new TIMEOUT failure kind.
Remote disk RPCs (R03-CAN-049, R03-CAN-050): list_volumes and delete_volume
passed Duration::ZERO, which execute_with_timeout treats as "no deadline",
so a peer that accepts the request and never answers stalls the coordinator
(and, for delete_volume, the bucket-deletion workflow). Both now pass
get_max_timeout_duration(), matching every sibling method in the file.
Regression tests: a silent TLS peer must be shed by the handshake deadline;
list_volumes/delete_volume against a peer that completes the TCP connect and
then goes silent must fail with DiskError::Timeout instead of hanging.
* fix(security): stop leaking signed headers and bound OIDC/KMS credentials
Three independent hygiene fixes found by the security review.
R03-CAN-018 (crates/signer): try_get_canonical_headers and get_signed_headers
logged the complete header map at DEBUG before signing. Runtime callers pass
session credentials and SSE-C key material through these headers, so anyone
able to raise the log level (or read DEBUG logs) recovered
X-Amz-Security-Token and SSE-C keys verbatim. The statements were debugging
leftovers with no operational value and are deleted rather than redacted.
R03-CAN-014 (crates/iam): the OIDC HTTP adapter buffered provider responses
with an unbounded Response::bytes(), so a configured, compromised or
attacker-pointed IdP endpoint could stream an arbitrarily large or endless
body into memory (the ValidateOidcConfig admin handler lets a ServerInfo
caller choose the endpoint). Responses are now read incrementally and fail
closed past MAX_OIDC_RESPONSE_SIZE, and the already SSRF-hardened client
builder gains request and connect timeouts so a stalled provider cannot pin
the calling task indefinitely.
R07-CAN-105 (helm): the Vault KMS token was serialized into the chart
ConfigMap, exposing it to every subject allowed to get ConfigMaps in the
namespace. It now renders into a dedicated Secret that the Deployment and
StatefulSet consume via envFrom; the Secret is separate from the main
credentials Secret so it also works when secret.existingSecret is set.
Regression tests:
- rustfs-signer: signing_never_logs_signed_header_material
- rustfs-iam: oidc_response_body_past_the_limit_is_rejected,
oidc_response_body_at_the_limit_is_accepted
- scripts/test_helm_templates.sh: KMS token must never render in plaintext
* fix(webdav): enforce body limit, request timeout and connection cap
The configured WebDAV maximum body size was enforced from Content-Length, so a
chunked request declared no length and bypassed it entirely. The configured
request timeout was never applied to the connection at all, and the accept loop
spawned a task per connection with no bound, so an unauthenticated client could
hold resources indefinitely and in unbounded number.
Enforce the limit on bytes actually read rather than the declared length, apply
the configured timeout to the request, and bound accepted connections with a new
RUSTFS_WEBDAV_MAX_CONNECTIONS (default 1024) surfaced in the config report.
Covers R03-CAN-051, R03-CAN-052, R03-CAN-067, R04-CAN-089, R05-CAN-094 and
R05-CAN-097 (backlog #1471, #1474).
* fix(security): stop STS credentials from crossing the parent trust boundary
Two related credential-boundary holes let a short-lived STS credential act
with the full, unrestricted authority of the long-term user it was minted
from.
AddUser (R03-CAN-021, CWE-269/863): should_check_deny_only relaxes the admin
policy check to deny-only when a Console/STS session targets the IAM user it
represents. Nothing then stopped that session from calling AddUser with its
own parent's access key, so the handler wrote an attacker-chosen secret key
and status over the parent's stored Credentials via create_user ->
save_user_identity. A session that expires in minutes became permanent
control of the account. AddUser now rejects any temp or service-account
requester whose resolved parent equals the target access key, resolving the
parent the same way should_check_deny_only does (parent_user field, else the
JWT `parent` claim, since some stores persist the parent only in the token).
FTPS/SFTP/WebDAV password auth (R04-CAN-086, CWE-287/862): these protocols
looked the access key up with check_key, which falls back to the STS account
cache, and then compared only the stored secret. An STS access key plus
secret therefore authenticated with no session token presented and no
session-policy claims applied - the holder got the parent's full permissions.
Password authentication now rejects temporary credentials before the secret
comparison. The discriminator is is_temp() && !is_service_account(), the same
one IamCache::update_user_with_claims uses to route an identity into the STS
cache, so service accounts - which resolve policy from stored IAM state
rather than a client-presented token - keep working over these protocols.
Regression tests cover both predicates and pin the guards to their call
sites so neither can be dropped without a test failure.
Add a deterministic four-hot AddTier regression for the reporter path where one peer has already durably committed prepare replay and the remaining peers still need commit fanout before local publish.
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(storage): resolve staged dead code in store module (#730)
Drop the store module's blanket #![allow(dead_code)] and resolve every
unit it was masking, instead of keeping them staged:
- write_persistent_key_only_index: the keys-only writer wrapper only had
test callers; the in-process rebuild flow already exists and uses the
_with_metadata variant (prepare -> rebuild, triggered lazily from the
opt-in listing path). Tests call _with_metadata directly now.
- ListIndexLifecycle::recover_after_restart / mark_corrupt: production
derives index health per request from the persisted artifacts (index
file + namespace mutation journal), and restart recovery already lives
in load_persistent_key_only_index's journal restore. The persisted-
lifecycle design these transitions served was superseded, and Corrupt
had no detector, so the never-constructed Corrupt state/reason
variants go with them.
- record_list_objects_index_opt_in_fallback (+ helpers): superseded by
the inline per-site fallback recording in the opt-in listing path; the
recorder recomputed health with a hardcoded None provider, so it could
only ever report a disabled-based reason.
- Provider-contract traits (Generation/Page/PageIterator/KeyLookup):
provider dispatch went the ListObjectsIndexProviderKind enum route;
only ListMetadataIndexHealth is live (dyn in source-mode selection)
and stays, minus its unused is_healthy default.
- Delete the unused list_quorum_from_env/RUSTFS_API_LIST_QUORUM pair,
fold should_resume_local_decommission and
should_auto_start_rebalance_after_recovered_meta into their surviving
primitives, drop get_disk_via_endpoint, and cfg(test) the remaining
test-only conveniences.
* fix(ilm): fail closed on unsafe manual job recovery
Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(ilm): format manual recovery imports
Apply rustfmt import ordering after merging main into the manual recovery branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): deduplicate manual transition worker results
Persist per-task manual transition worker result markers before applying job counters so duplicate worker completion reports are no-ops. Reconcile persisted markers during drained-queue lease renewal and recovery to restore marker-before-record crash windows. Fail closed when persisted worker result markers are corrupt.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ilm): fail closed on unsafe manual job recovery
Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.
Co-authored-by: heihutu <heihutu@gmail.com>
Cover manual transition terminal job records for queue closed and queue send timeout outcomes so backpressure summaries remain partial with the expected counters and queue snapshots.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a black-box e2e for concurrent manual transition durable jobs with the same bucket and prefix. The test asserts that exactly one request is accepted, the other returns 409 with the active job status and cancel endpoint, and the accepted job reaches a completed terminal status after queued transitions settle.
Verification:
- cargo fmt --all --check
- git diff --check
- CARGO_TARGET_DIR=/private/tmp/rustfs-target-1481-same-scope cargo test -p e2e_test --lib reliant::tiering::test_manual_transition_async_same_scope_conflict_reports_active_job -- --exact --nocapture
Co-authored-by: heihutu <heihutu@gmail.com>
Add a focused ecstore regression that exercises active manual transition cancellation through the existing cancel_check hook after scan progress has been made. The test verifies the cancelled report, dry-run counters, and opaque resume cursor without relying on wall-clock timing.
Co-authored-by: heihutu <heihutu@gmail.com>
Extend the internode gRPC benchmark driver with P2 request-only, canary, after, and rollback phases. The request-only phase proves that requesting msgpack-only without fleet confirmation keeps compatibility JSON fields, while canary and rollback materialize the operator states needed for mixed-version convergence without claiming real fleet soak evidence.
Verification:
- scripts/test_internode_grpc_ab_bench.sh
- bash -n scripts/run_internode_grpc_ab_bench.sh scripts/test_internode_grpc_ab_bench.sh
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
test(perf): add pinned paired abba bench
Add a backlog#1432 paired benchmark orchestrator that runs MinIO/RustFS in an A1/B1/B2/A2 schedule while requiring immutable server identity, source revision or release, and ACK contract labels. The runner delegates actual workload execution to run_object_batch_bench_enhanced.sh so node metrics, resource captures, and provenance stay in the shared artifact format.
Verification:
- scripts/test_pinned_paired_abba_bench.sh
- bash -n scripts/run_pinned_paired_abba_bench.sh scripts/test_pinned_paired_abba_bench.sh
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
Fail startup when the configured console listener resolves to the same listener port covered by the S3 address. This keeps SO_REUSEPORT from hiding an intra-process S3/console endpoint collision.
Co-authored-by: heihutu <heihutu@gmail.com>
Ensure cancelled durable manual transition jobs expose a cancelled report both when worker results drain after a cancel request and when legacy persisted records are decoded.
Co-authored-by: heihutu <heihutu@gmail.com>
The background tier free-version recovery walk pinned a hardcoded 60s
total wall-clock timeout that overrides every operator knob, so any
bucket whose healthy full walk exceeds 60s fails forever; the failed
run's duration was also subtracted from the next 60s tick, restarting
the walk immediately and pinning CPU and disk I/O.
- Drop the total wall-clock budget on the recovery walk
(walkdir_timeout: Duration::ZERO) and inherit the operator-tunable
drive stall budget (RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS) for
per-call progress, so hung disks still fail fast.
- Back off failed runs from completion time: 60s doubling to a 600s
cap, reset on success; never subtract the failed run's duration.
- Add RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED (default true) to opt
out of the recovery worker on deployments with no remote tiers;
invalid values warn and fail open.
Fixes#5130
Co-authored-by: claude <claude@ehdtn.com>
Extend the inline fast-path mixed-msgpack E2E collector to capture msgpack/json decode-error counters by direction, message, and codec. Assert those counters remain unchanged for the multipart, part-number, SSE, compression, and transition fallback-control scenarios.
Co-authored-by: heihutu <heihutu@gmail.com>
When endpoints include remote nodes and no internode RPC secret is
resolvable, every remote format read fails client-side with "No valid
auth token" and startup dies ~2 minutes later with the misleading
"store init failed to load formats after 10 retries: erasure read
quorum" error (issues #4939, #5153).
Add a preflight to ECStore init that resolves the RPC secret once
before any disk opens: if any endpoint is non-local and resolution
fails, abort immediately with the operator-facing remediation message.
The preflight also emits the "RPC auth secret resolution failed" log
line so the log-analyzer rpc-secret-resolution rule keeps firing for
this scenario. Single-node (all-local) topologies never invoke the
resolver.
scripts/check_logging_guardrails.sh flags any lowercase secret-named
identifier interpolated into a format string. The static-secret-file test
added in #5245 interpolates two fixture bindings (file_secret, env_secret)
into format! when constructing the secret file and env var, tripping the
heuristic and breaking make pre-commit on every branch based on main.
Rename the bindings to file_key_b64 / env_key_b64 so they no longer match
the heuristic. The fixtures are dummy base64 key material written to a
temp file and env var, not log output, and the test coverage is unchanged.
The guard script itself is untouched.
docs/testing/security-regressions.md requires every fixed advisory to map to a
named, greppable regression test. Rename the tests added with the fixes to carry
their advisory id so `rg -i ghsa` finds them, and add the four rows to the
advisory -> test map.
Adds the FTPS MKD regression test that was missing. It primes the dummy backend
with a successful create_bucket, so the assertion distinguishes "denied at the
authorization boundary" from "backend refused" — without the queued success an
unconfigured create_bucket fails on its own and the test would pass even with
the authorization check removed. Verified it fails when the check is reverted.
DummyBackend gains a Debug impl (FtpsDriver's trait bounds require it) and a
queue_create_bucket_ok helper.
Records in the CI-execution map why ghsa_g3vq_* runs in the default pass despite
sitting behind the ftps feature: the rustfs crate defaults to ["ftps", "webdav"]
and cargo unifies features across the workspace build, so the test executes
there even though `cargo test -p rustfs-protocols` alone would skip it.
* fix(policy): quantify negated string conditions per value
ForAllValues:/ForAnyValue: negated string operators computed the positive
quantified match and then negated the aggregate. That yields NOT(all match)
and NOT(any match), which is the semantics of the *other* quantifier, so
ForAllValues:StringNotEquals and ForAnyValue:StringNotEquals were exactly
transposed. The same applied to StringNotEqualsIgnoreCase, StringNotLike,
ArnNotEquals and ArnNotLike.
Introduce an explicit Quantifier and push negation into the per-value
predicate for the qualified forms, so ForAllValues requires every request
value to satisfy the operator and ForAnyValue requires at least one.
Unqualified operators keep negating the aggregate, preserving AWS
single-valued-key semantics. Absent keys now follow AWS: ForAllValues is
vacuously satisfied, ForAnyValue is not.
A request value set that is fully contained in or fully disjoint from the
policy set cannot distinguish the two quantifiers, which is why the existing
cases missed this; the new tests use partially overlapping sets.
* fix(auth): keep request headers out of server-derived condition keys
get_condition_values folded every remaining request header into the policy
condition map. HeaderMap lowercases header names, and the server-derived keys
userid, username, principaltype, versionid and signatureversion are lowercase
too, so a header of the same name collided with them. The collision branch used
extend(), and non-quantified string operators match if ANY value in the vector
matches, so sending `userid: admin` was enough to satisfy a condition on
aws:userid. The jwt:/ldap: claim keys were reachable the same way whenever the
credential carried no such claim.
groups was worse than an append: the header loop ran before the cred.groups
block, which is gated on !args.contains_key("groups"), so a `groups:` header
both injected a value and suppressed the credential's real group list.
Resolve claims and group membership before merging headers, then skip any header
naming a key the server already derived or a well-known identity/context key.
The reserved set comes from KeyName so it tracks the key registry; s3:x-amz-*
keys stay mergeable because they mirror request headers by design.
* fix(access): gate the ListBucketVersions fallback on public-access checks
An anonymous request for ListObjectVersions that the bucket policy does not
grant directly falls back to re-checking the grant as s3:ListBucket. That
fallback returned Ok(()) straight away, skipping the two gates the direct grant
passes through: deny_anonymous_table_data_plane_if_needed and the
RestrictPublicBuckets check on the bucket's public-access block.
So a bucket whose policy allows anonymous s3:ListBucket kept serving anonymous
version listings after an operator enabled RestrictPublicBuckets, even though
the equivalent GetObject was correctly denied.
Fold the fallback into policy_allowed so both routes reach the same gates.
* fix(ftps): authorize MKD against the CreateBucket boundary
FTPS MKD creates a bucket but ran no authorization check, so any principal that
could open an FTPS session could create buckets regardless of policy. Every
other operation in this driver authorizes first — LIST, RETR, STOR, DELE and
RMD all call authorize_operation — and the WebDAV gateway checks
S3Action::CreateBucket on the equivalent path.
Add the matching check so MKD clears the same boundary as an S3 CreateBucket.
* fix(kms): do not include secret value in static KMS format error
The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().
* chore(security): add guardrails against secret values in error strings
Follow-up to the PR #5222 review finding fixed in PR #5243: a config value
that fails secret-format parsing is typically the raw secret itself, and
error messages are log content — they propagate via ? and are printed by
startup/error logging far from the construction site.
Three layers so this class of leak is caught earlier next time:
- security-advisory-lessons: state explicitly that error/panic messages are
log content; parse-failure hints must name the env var and expected
format, never echo the input. Add a matching review prompt.
- adversarial-validation: add a security-reviewer probe that greps
error-construction sites on secret-bearing config paths for interpolation
of the raw value, including the duplicated-parse trap where the leak hid.
- check_logging_guardrails.sh: mechanical backstop — flag inline format
interpolation of secret-named identifiers (secret/password/passwd/
private_key) in checked files; add crates/kms/src/config.rs (the twin
parse site) to the checked list. Verified the check passes on the fixed
tree and catches the original leak at rustfs/src/init.rs:399.
The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().
docs.rustfs.com dropped the .html suffix from page URLs, so
https://docs.rustfs.com/integration/tls-configured.html now returns 404.
Point both READMEs at the live canonical URL instead.
Supersedes #5154, which replaced the links with Wayback Machine
snapshots even though the docs site is alive and the page still exists.
* fix(ilm): evaluate complete version groups
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): log replication expiry blocks
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): diagnose incomplete noncurrent chains
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover purge-pending version groups
Add regression coverage for lifecycle-only version listing so purge-pending versions stay present for ILM evaluation while the public ListObjectVersions projection remains filtered.
Refs rustfs/backlog#1500
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): log lifecycle evaluation failures
Emit structured lifecycle_evaluation_failed events when version group loading or evaluation fails during immediate and existing-object expiry scans.
Refs rustfs/backlog#1503
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover incomplete noncurrent chains
Pin the fail-closed behavior for noncurrent expiration when successor_mod_time is missing and reuse a stable structured event name for that skip path.
Refs rustfs/backlog#1502
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover one-day noncurrent expiry boundary
Add a runtime regression test proving NoncurrentVersionExpiration Days=1 stays inactive before expected_expiry_time and deletes exactly at the computed due boundary.
Refs rustfs/backlog#1504
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): keep transition checks after incomplete expiry
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Retry peer tier config reloads after committed mutations so recovered nodes converge without requiring a second admin change.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures
Binds a domain-separated, length-prefixed canonical request-body digest into the
v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path
attacker on the default-plaintext internode channel can no longer tamper with a
mutation payload (or strip the msgpack `_bin` field to force the JSON fallback
decode) without invalidating the signature.
Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions,
WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile,
RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the
msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by
default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so
rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now
configurable and overflow fails closed with a metric.
Refs https://github.com/rustfs/backlog/issues/1327
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rpc): satisfy architecture-migration compat-marker guard
Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and
stop backticking env-var/metric names in the cleanup-register entry so the
guard's id extractor only sees the task-id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Cover async manual transition jobs that finish as partial because maxObjects truncates the scan, including terminal cancel idempotence and unchanged report counters.
Co-authored-by: heihutu <heihutu@gmail.com>
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env
Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before.
Refs https://github.com/rustfs/backlog/issues/1327
* ci: fix internode auth test lint failures
* perf(ecstore): cache internode sig strict env
---------
Co-authored-by: houseme <housemecn@gmail.com>
Add test coverage for the msgpack-only request/fleet confirmation truth table and exact request JSON policy manifest.
Pin request and response rollback behavior so removing either gate restores JSON compatibility while both gates enter msgpack-only for eligible fields.
Co-authored-by: heihutu <heihutu@gmail.com>
Update the internode gRPC benchmark P2 env output to require both the msgpack-only request flag and the fleet-confirmed gate before measuring a true msgpack-only phase.
Document the dual-gate rollback semantics and add a dry-run script test so the benchmark driver cannot silently regress to the single-flag flow.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards.
Co-authored-by: heihutu <heihutu@gmail.com>
test(internode): pin msgpack compat send sites
Add a checked test manifest for internode dual-encoded request and response payloads so node.proto _bin fields must be explicitly classified as msgpack-only eligible or always dual-write.
The manifest also pins the current JSON encoder call sites for request and response send paths, making future send-site drift visible before rollout.
Co-authored-by: heihutu <heihutu@gmail.com>
Cache the internode msgpack-only env gate after the first read so metadata RPC send paths avoid repeated environment parsing. Keep a reset hook for tests that intentionally change the env in-process.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep internode JSON compatibility fields unless operators explicitly confirm fleet-wide msgpack-only readiness. This prevents a single legacy rollout flag from emptying JSON fields in mixed-version clusters where older peers may still read the legacy JSON payload.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): bound manual transition duration
Add maxDurationSeconds handling for manual transition runs so operators can bound long scoped scans without changing the existing enqueue_only job contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ilm): document manual transition run limits
Document the enqueue_only manual transition contract, secret-handling guidance, and best-effort duration budget while pinning the new duration flag in the e2e response model.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): report manual transition backfill outcomes
Add scoped lifecycle transition backfill reporting for backlog #1478 and expose enqueue outcomes needed by #1479 without changing the existing scanner/compensation bool API.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(admin): add manual transition run endpoint
Add a bounded POST /rustfs/admin/v3/ilm/transition/run API for backlog #1477 and cover the route, policy, query parsing, and partial status contract needed by #1481. Console operations from #1480 are intentionally left for a later client integration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): allow manual transition resume markers
Accept additive marker and versionMarker parameters on the bounded manual transition run API so clients can continue from a partial report without changing the existing default scan behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): harden manual transition partial reports
Preserve the null-version cursor contract, stop manual scans on enqueue pressure without skipping the failed object, and keep raw resume markers out of admin JSON responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): add manual transition e2e coverage
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ilm): keep transition enqueue hot path direct
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A federated OIDC / AssumeRoleWithWebIdentity login derives parent_user from the external identity via session_identity() (username -> email -> sub -> oidc-user-unknown). At IAM request time a temporary credential whose parent_user equals the root access key is treated as owner (see rustfs/src/admin/auth.rs and the rustfs_iam owner resolution used by the policy engine and service-account paths). A federated identity whose display name happens to equal the root access key would therefore be silently granted full owner access on a string match alone.
Reject such issuance up front: immediately after parent_user is derived, and before credential generation, set_temp_user, and the site-replication hook, deny the binding when parent_user equals the root access key. Both federated flows funnel through the single DefaultFederatedSessionBinding::bind -> issue_credentials chokepoint, so this covers browser OIDC callback and AssumeRoleWithWebIdentity alike.
- rustfs/src/admin/service/federated_identity.rs: issue_credentials resolves the root access key via current_action_credentials() and rejects with FederatedSessionBindingError::InvalidRequest on collision. Added the pure helper parent_user_is_reserved(parent_user, root_access_key) so the collision decision is unit-testable without process globals; the comparison is exact and case-sensitive, mirroring the request-time owner comparison.
This is the federated-principal / root isolation pre-work (batch 0A, the immediate issuance-time block) from the OIDC review in rustfs/backlog#1437. Layer 2 (a signature-protected federated credential origin that forces is_owner=false at request time regardless of the parent_user string) is a separate follow-up. Already-issued colliding credentials continue on their existing TTL and should be revoked proactively. Legitimate federated identities are unaffected; only an exact collision with the configured root access key is denied.
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend
Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.
* fix(ecstore): guard encrypted multipart range seeks
* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix
Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
Stop the OIDC subsystem from writing credential-grade secrets into logs and returned errors.
- crates/iam/src/oidc.rs: redact sensitive header values (authorization, proxy-authorization, cookie, set-cookie) in format_http_headers, emitting only name and length; drop the raw request/response body from the DEBUG events (keep byte length); stop logging and stop splicing the raw token response body into the error returned on token_response_parse_failed (the TokenResponseBodyShape summary and length are retained); remove the now-unused format_http_body helper.
- rustfs/src/admin/handlers/oidc.rs: stop logging the raw authorization code and state on the code-exchange error path (code_len/state_len are kept).
This is the OIDC log/error redaction pre-work (batch 0B) from the OIDC review in rustfs/backlog#1437. It changes diagnostic content only; HTTP/STS status codes and legitimate request results are unchanged.
* fix(admin): confine service-account parent to caller scope (GHSA-5354)
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* test(admin): pin GHSA-5354 scope guard for derived credentials
The GHSA-5354 fix (#5141) added `add_service_account_parent_within_scope` and a
named-boundary test, but that test only covers the self-scoped case with
`req_user == req_parent_user`. A derived credential — a service account or STS
token holding CreateServiceAccountAdminAction — has `req_user` (its own key)
distinct from `req_parent_user` (its parent), which is the realistic attacker
shape and was left unexercised.
Add a test that pins the guard's allow set to exactly `owner || is_svc_acc`
across that derived-credential shape, including a derived non-owner aiming at the
root credential, so a later change to either the guard or the `is_svc_acc`
rewrite cannot silently let a derived non-owner escape its own parent's scope.
Also note at the check site that the guard is evaluated on the original
`target_user`, before the derived-credential rewrite to `req_parent_user`.
Refs GHSA-5354-r3w2-34m8.
PRs #5141 and #5142 fixed GHSA-5354 (service-account parent-scope enforcement) and GHSA-3ppv (versioned object-read authorization) but did not add the advisories to the security-regressions inventory that repo policy requires for every fixed GHSA. Add both rows to the advisory -> test mapping (advisory, class, fix PR, named regression test, layer) and list ghsa_5354_* / ghsa_3ppv_* under the default-CI unit-test execution map, since both tests live in the rustfs lib target and run in the default `cargo nextest run --profile ci --all --exclude e2e_test` pass.
get_object, the CopyObject source, and the UploadPartCopy source authorized reads that name an explicit versionId against s3:GetObject rather than s3:GetObjectVersion. A principal holding s3:GetObject but not s3:GetObjectVersion could therefore read historical object versions. get_object_attributes was already version-aware; these three paths were not.
Add a shared versioned_read_action helper that selects s3:GetObjectVersion when a version is named and s3:GetObject otherwise, and apply it to the three read/copy-source authorization sites. Add named regression test ghsa_3ppv_versioned_read_selects_get_object_version_action.
The ActionSet::is_match GetObjectVersion->GetObject compatibility mapping is intentionally retained for now; its removal is gated on a follow-up audit of the remaining version-aware read paths (HeadObject, GetObjectAcl, tagging). It does not affect this fix, which only prevents a GetObject grant from satisfying a versioned read.
Refs GHSA-3ppv-fx5m-m749.
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* fix(site-replication): replicate object-lock retention and legal hold
Object Lock did not survive site replication, so a WORM-protected object was
either missing from the peer entirely or present there unprotected. Two
independent defects caused this.
1. The site-replicator service account policy granted no object-lock actions.
Any replicated object carrying x-amz-object-lock-mode /
x-amz-object-lock-retain-until-date was rejected by the peer with
AccessDenied, so an object created under a retention rule never replicated
at all. Grant s3:GetObjectRetention, s3:PutObjectRetention,
s3:GetObjectLegalHold and s3:PutObjectLegalHold. s3:BypassGovernanceRetention
is deliberately NOT granted: replication must never be able to erase a
retained version on the peer.
2. PutObjectRetention and PutObjectLegalHold only rewrote object metadata. The
object PUT path is what normally computes a replication decision and
schedules replication, so a lock applied after upload stayed local and the
peer kept its previous, unprotected state. Schedule replication explicitly
after both metadata writes.
Verified live on a two-site cluster: an object created under a COMPLIANCE
retention rule now replicates, the peer copy reports COMPLIANCE, a
version-scoped delete on the peer is refused, and the content matches by MD5.
Retention and legal hold applied after upload propagate to the peer within
about 15s, and the replicated copy is likewise undeletable. No AccessDenied
replication errors remain.
Adds a regression test asserting the policy allows the four object-lock actions
and still denies governance bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(site-replication): persist pending marker before object-lock metadata commit
Review follow-up on the object-lock replication fix.
PutObjectRetention and PutObjectLegalHold scheduled replication but did not
persist a pending replication marker first, unlike the canonical object PUT
path. If the process restarted between the metadata commit and the replication
worker write-back, the in-memory task was lost and the object carried no
Pending/Failed marker, so the scanner heal skipped it and the peer silently
stayed unprotected.
Both handlers now mirror the PUT path ordering: compute the replication
decision once BEFORE the commit, fold SUFFIX_REPLICATION_STATUS and
SUFFIX_REPLICATION_TIMESTAMP into the eval_metadata passed to
put_object_metadata, then schedule after the commit. eval_metadata is merged
into the object metadata, so the lock fields are preserved. When the object
info cannot be read the decision is empty and nothing is scheduled.
Also aligns the two handlers to build their options the same way, and notes at
both schedule sites that this triggers a full object re-upload because
metadata-only replication is not implemented yet.
Adds an integration test asserting both handlers compute a replication decision
exactly once, so a regression that drops the scheduling is caught.
Verified live on a two-site cluster: applying retention while the peer is down
now leaves Replication Status: PENDING on disk at commit time, and once the
peer returns the lock propagates in about 15s and the status becomes COMPLETED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On macOS with DNS interception services, .invalid TLD domains resolve
to an interception address (e.g. 198.18.16.173) instead of failing DNS
resolution. The health probe then classifies the error as Unreachable
rather than DnsFailure. Accept both outcomes since both are correct
non-reachable error classifications.
Keep remote peer topology slots observable when peer client construction cannot build a dialing client, and publish admin server_info cache/failure state only after a complete probe round.
Covers rustfs/backlog#1426 and rustfs/backlog#1430.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(admin): use raw topology host for peer mapping
Backlog: rustfs/backlog#1427
Keep endpoint topology membership keyed by raw host:port when mapping peers to grid hosts, while preserving DNS resolution only for the PeerRestClient dialing host.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): reconcile v3 info with topology
Backlog: rustfs/backlog#1425
Backlog: rustfs/backlog#1428
Synthesize additive unknown server rows from configured endpoint topology before v3 backend counters are computed, and add exact drive-identity coverage tests so balanced totals cannot hide a missing member.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(admin): collapse topology reconciliation
Build a single endpoint topology index for v3 admin server reconciliation and completeness reporting, and reuse the raw peer/grid mapping when constructing peer clients.
Backlog: rustfs/backlog#1424
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Update selected workspace dependencies and lockfile entries.
Keep async-nats and rcgen on explicit feature sets while preserving the RustFS targets and TLS test surfaces.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): gate exact remote version consumption
Reject non-empty remote tier versions before transitioned GET and remote delete backend I/O when the tier backend does not support exact version operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(tier): split mock remote version validation fault
Separate one-shot mock remote version validation failures from persistent unsupported-backend behavior so cleanup durability tests can still verify exact-version recovery while #1358 fail-closed gate tests keep asserting no backend I/O.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Advance transition transactions to UploadOutcomeUnknown before remote tier PUT so response-loss windows can be recovered through provider-authoritative probing.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep coordinator Prepared intents durable when peer abort recovery fails, then make reload retry abort or commit based on the persisted tier config digest.
Co-authored-by: heihutu <heihutu@gmail.com>
A single instance with a malformed `enable` value (e.g. the typo `enable`
instead of `enabled`/`on`) made every configured notification/audit target
of every type fail to load. `create_targets_from_config_with_store_mode`
collected instance configs through the strict `try_collect_target_configs`,
whose `.collect::<Result<Vec, _>>()` short-circuits on the first error and
drops all siblings; the surrounding `?` then aborted the loop over every
plugin type.
This contradicted the function's own documented contract ("Creation is
fault-isolated per instance") and was inconsistent: target construction
failures were already isolated into the `failures` accumulator, only config
collection failures were fatal.
Add `collect_target_config_results[_from_env]`, which returns the enabled
`(id, config)` pairs plus a per-instance failure summary, and use it in the
create path. Per-instance collection errors now flow into the same
`failures`/`creation_failures` accumulator as construction failures: healthy
siblings and other target types still load, while the malformed instance is
surfaced so the notify lifecycle stays non-converged/retryable and an Admin
write cannot report a false success. Audit likewise keeps logging through
its valid targets instead of going fully Stopped.
Follow-up to #5088.
* feat(tier): add transition candidate probe contract
Add a fail-closed WarmBackend probe contract for provider-authoritative transition candidate state. Default providers report Unsupported, while the shared mock backend can now model missing, unversioned, and exact-version candidates for follow-up recovery tests.
This is a forward-compatible foundation for #1352/#1358 recovery work and does not change production cleanup behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(tier): probe transition candidates from providers
Implement provider-authoritative transition candidate probing for S3-family warm backends by querying ListObjectVersions with exact-key filtering and fail-closed classification for delete markers, multiple versions, truncation, and unknown versioning state.
This keeps non-S3 providers on the default Unsupported probe result and forwards MinIO, RustFS, and R2 through the S3 probe implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A metacache listing that misses quorum purely because the volume or path
is absent on a quorum of drives (every drive reports VolumeNotFound or
FileNotFound) is a benign, expected outcome: list_path_raw already returns
VolumeNotFound/FileNotFound and lets the caller decide how to react.
The common trigger is a startup race where the system bucket (.rustfs.sys)
is not yet created on every drive when an early reader such as the IAM
config loader lists config/iam/. Logging that at error prints a scary
message during normal boot and, in #5076, misled a user into blaming it
for unrelated upload failures.
Add is_benign_not_found_listing_failure() and demote the pure not-found
case to debug (state = "quorum_not_found"), keeping error for listings
that failed for a real reason (I/O, timeout, corruption). Return value and
control flow are unchanged. Add a unit test for the classifier.
Refs: #5076
Bind provider exact GET and DELETE capability to tier operation leases and reject non-empty remote versions before committing transition metadata when the provider cannot address exact versions.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep LocalCommitStarted transition transaction records retained when the local source cannot prove the commit, so recovery does not classify the record as failed or delete the remote candidate without cleanup proof.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep recovered tier mutation blocks installed until the local publish transition has atomically established draining for the affected tiers, so old-generation leases cannot slip in after peer commit replay and before local publish.
Delay committed intent cleanup until local publish succeeds. If peer replay succeeds but local publish fails, the durable committed intent remains available for retry and the runtime block stays fail-closed.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix: fail closed legacy tier mutations without etag
Reject non-add tier config mutations when the loaded durable snapshot has no current config ETag. This keeps legacy JSON/no-ETag paths from being saved as creation-style updates before coordinator intent state can bind an old config revision.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): allow empty clear without config etag
Allow empty tier clears to initialize the binary config under the namespace coordinator lock while keeping non-empty legacy config mutations fail-closed when no current ETag exists.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation control client
Add signed tier mutation prepare, commit, and abort client calls for peer fanout while preserving the existing protobuf and RPC contract.
Verify response proofs before interpreting peer outcomes, fail closed on invalid states, and reject oversized payloads before dialing peers.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Add machine-readable storage, listing, usage, and workload component diagnostics to the admin cluster snapshot response while keeping the external S3 API untouched.
Co-authored-by: heihutu <heihutu@gmail.com>
The peer health snapshot now reports CapabilityState::Supported for
local nodes (no probing needed), but the cluster_snapshot test still
expected Unknown. Update the assertion to match the new behavior
introduced in commit 7805cf5ae.
Add idempotent terminal transition handling for tier mutation intents and a compare-and-swap record update helper backed by config-object ETags. Cover commit and abort retries, conflicting terminal updates, stale ETag rejection, and exact-limit scan pagination.
Co-authored-by: heihutu <heihutu@gmail.com>
Add canonical record-object naming and crate-private save, load, and idempotent delete helpers for tier mutation intents. Cover malformed record keys, mismatched persisted mutation IDs, and an ECStore config-object round trip under test-util.
Co-authored-by: heihutu <heihutu@gmail.com>
Add an internal durable tier mutation intent model for the #1357 distributed fencing work. The new model validates schema, checksum, canonical targets, mutation-specific target identity shape, config ETags, expiry, and prepared-to-terminal state transitions without changing the production mutation path yet.
Co-authored-by: heihutu <heihutu@gmail.com>
Retry cleanup-pending transition transaction recovery after restart, keeping committed remote bodies when local metadata already references them and preserving records when remote cleanup fails.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): accept null version transition sources
Treat nil/null source version IDs as null-version transition sources when building transition transaction source identity, while preserving fail-closed validation for truly missing IDs on versioned sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): preserve multipart source versions
Propagate bucket versioning into CompleteMultipartUpload, assign a concrete version ID for versioned multipart completions, and classify unversioned FileInfo sources as null-version transition sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): normalize multipart completion versions
Treat nil staged version IDs as missing for versioned multipart completion and clear staged version IDs when completion publishes a null-version object under suspended or unversioned semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): refresh workspace dependencies
Refresh compatible workspace dependencies while preserving the requested
version pins. Update async-nats and Hyper, and replace the temporary Hyper
Git patch with the released crate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): bump hotpath to 0.21.5
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): inject local deployment into replicate-add payload when omitted
The web console's "Set Up Site Replication" flow posts only the remote peer(s) to
/rustfs/admin/v3/site-replication/add and omits the local deployment. The add
preflight requires the local deployment to be present
(validate_add_preflight_topology), so the console's request failed with
"site replication add request must include the local deployment" and no
replication was configured from the web UI.
Inject the local site into the sites list when the payload does not already
include it (matched by endpoint identity), before validation. `mc admin
replicate add` always sends every site, so this is a no-op for the CLI; the
local site carries no credentials (validate_add_sites already skips credential
checks for it).
Adds unit tests for inject-when-missing and no-op-when-already-present.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Restore the workspace reqwest default feature stack for RustFS outbound HTTPS clients, while keeping per-crate extra APIs such as json, stream, and multipart explicit. Lazily initialize the notification runtime from admin target access when RUSTFS_NOTIFY_ENABLE=true is already effective, and add regression coverage for HTTPS webhook custom CA handling and target-list visibility.
Fixes#5052.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): count replicated buckets as in-sync in replicate status
`mc admin replicate status` reported "0/N Buckets in sync" with a cross mark for
every bucket even when replication was healthy and objects had propagated.
build_sr_info stores each bucket's replication_config in the wire form as
base64-encoded XML (raw_config_to_base64), but site_replication_config_mismatch
XML-parsed that string directly without base64-decoding. The parse always
failed, so every bucket with a replication config was reported as a config
mismatch (replication_cfg_mismatch = true), which the status endpoint and mc
render as out-of-sync.
Decode the wire value (tolerant base64 decode, falling back to raw bytes) before
XML-parsing. Adds a regression test that feeds the base64 wire form used in
production; the existing tests only exercised raw XML, which masked the bug.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
fix(site-replication): report peer sync state in replicate info and surface pre-existing back-fill failures
BUG 1 — blank/Unknown Sync in `mc admin replicate info` and the console:
The info endpoint (SiteReplicationInfoHandler) serializes the persisted peer
map, whose sync_state is constructed as Unknown and never updated; the health
derivation added earlier only runs in build_status_info (the status endpoint).
Persist sync_state = Enable for peers on the add and join enable paths
(promote only Unknown, never clobber an explicit Disable). build_status_info
still refines live health for the status endpoint, so info/status/console and
peer_states all report a real sync state for a healthy peer.
BUG 2 — pre-existing-bucket back-fill failures were silently swallowed:
backfill_existing_buckets_after_add returned () and the add handler reported
success with an initial_sync_error_message that only ever carried
metadata-bootstrap errors, so a pre-existing bucket that failed to propagate
was invisible. Return per-bucket failures (including the previously silent
Ok(false) runtime-unavailable no-op and the versioning-failure drop), fold
them into the add response via compose_initial_sync_error_message, and log
reverse-direction gaps on the join path.
Adds unit tests for sync_state promotion (and Disable preservation) and for
back-fill failure surfacing.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Point the cla-bot config and the PR template at cla/v2.md so new
contributions are checked against the v2 Individual CLA.
- .github/cla.yml: document.version v1 -> v2, url -> cla/v2.md
- .github/pull_request_template.md: CLA link -> cla/v2.md
Commit aec2ee9ec (#5005) enforced that RPC secrets cannot be derived
from the public default credentials. This broke all multi-node cluster
E2E tests that relied on the constructor's DEFAULT_ACCESS_KEY /
DEFAULT_SECRET_KEY defaults.
Move the non-default credential and RUSTFS_RPC_SECRET blanking into
the RustFSTestClusterEnvironment constructor so every cluster test
starts with a valid RPC secret derivation base. Remove the per-test
overrides in cluster_multidrive_pool_test that were added as a partial
fix in #5005.
Add a deterministic transition matrix case for a stale prepared metadata snapshot followed by a committed transition, duplicate transition, and late GET. The test proves the stale metadata generation is physically reclaimed, duplicate transition does not upload a second remote candidate, and the late GET reads the committed remote body byte-for-byte instead of reopening the released local source.
Refs rustfs/backlog#1355
Co-authored-by: heihutu <heihutu@gmail.com>
Generate a unique Pulsar producer name for each producer instance while preserving the target type and target id context. This avoids reload-time collisions when an old producer is still connected while a new destination is initialized.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): add fallible erasure construction
(cherry picked from commit bd148b20f7)
* fix(ecstore): resolve storage parity per pool
(cherry picked from commit c05c2cb24b)
* fix(ecstore): keep carved per-pool parity core self-contained on main
Fixups so the cherry-picked fallible-erasure + per-pool-parity core builds standalone on current main without the excluded scope-creep commits:
- runtime/sources.rs: re-add backend_storage_class_parities (removed by the per-pool commit; its rebalance caller was updated in an unrelated reporting commit that was left out). Reimplemented over the snapshot API, behavior-identical.
- config/mod.rs: rename the storage-class publish test module (main independently added a mod tests, so the cherry-pick collided).
- rustfs storage_api.rs + startup_storage.rs: route the storage-class ENV consts through the startup storage facade and use a local const for the erasure-set-drive-count env name, satisfying the layer/facade guardrail (main's guardrail is stricter than when the core was authored).
* fix(ecstore): use struct-init in erasure test helper to satisfy clippy field_reassign_with_default
The cherry-picked fallible-erasure commit's `erasure_with_invalid_dimensions` test helper built `Erasure` via `default()` then reassigned fields, which trips `clippy::field_reassign_with_default` under `-D warnings` (only surfaced by `--all-targets`, which lints test code). #4977 fixed this in a later commit that was not part of the carved core. Use struct-init with `..Default::default()`, matching #4977's final form.
* fix(rustfs): gate the test-only storage-class ENV facade re-export behind cfg(test)
The ENV constants (INLINE_BLOCK_ENV/OPTIMIZE_ENV/RRS_ENV/STANDARD_ENV) re-exported through the startup storage facade are only consumed by a #[cfg(test)] test in startup_storage.rs, so in a non-test lib build the re-export is unused and trips -D unused-imports under clippy --all-targets. Gate it with #[cfg(test)], matching #4977's final form.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
The normative erasure-coding spec stated several target behaviors and known baseline defects as active invariants, and cited two symbols that do not exist on main. Each correction below was verified against the code.
- §4.1 shard size: legacy even-padding is RustFS-legacy-only and is NOT MinIO's sizing. MinIO (and the modern GF(2⁸) path) use plain div_ceil; for 1 MiB / 6 data shards that is 174763 bytes versus the legacy even-padded 174764. MinIO-migrated data is decoded by the modern path, so the legacy formula never applies to it.
- §6.3 / §11 MTime: the MTime key is always written (a None mod_time encodes as UNIX_EPOCH = 0), not omitted. Round-trip safety to None is enforced on the decode side, not by write omission. Only the legacy StatInfo.ModTime field is omitted-when-None.
- §7 commit: rollback after a missed write quorum is best-effort — undo failures are counted and warn!-logged, never propagated or retried — so a partially-failed rollback can leave shards behind; softened "never left partially committed" accordingly.
- §7 data_dir: reduce_common_data_dir votes over each disk's old_data_dir (a GC input for reclaiming the superseded dir), not the newly committed data_dir.
- §7 convergence: classify_rename_convergence is consumed only by the multipart-complete path (convergence.needs_heal()); the regular put_object path discards it.
- §7 write layout: removed the WriteLayout / resolve_write_layout citation — neither symbol exists on main (the write layout is computed inline), which violated the spec's own cite-real-symbols rule.
- §6.6 inline: inline presence is read from the meta_sys[inline-data] body marker alone; the header InlineData flag is written but not consulted on read, and disagreement is tolerated (MinIO may write the flag unset with inline data present). Removed the false "must agree" invariant.
- §11 UUID tolerance: version_id / data_dir (ID/DDir) decode as a fixed 16-byte Uuid::from_bytes and fail closed on a present-but-non-16-byte value; only transitioned-versionID uses the tolerant Uuid::from_slice(...).ok().filter(!nil). Split the previously-conflated bullet.
- §11 decode blanket: narrowed "everything else must degrade to a tolerant default" to enumerate the legitimate structural/geometry/version/length fail-closed guards (header array length, unknown header_ver, version > max, versions_len / bin_len bounds, non-16-byte ID/DDir).
- §8 / §13 codec guard: has_valid_dimensions() is a &self method, so it runs after construction and reliably covers only block_size == 0; a data_blocks == 0 geometry with parity > 0 panics in the .expect constructor before the guard runs. Noted the fallible-constructor fix.
Refuted and intentionally left unchanged: the "accepts container major == 1 with any minor" statement is correct — the reader compares only major (rejects major > 1) and never gates on minor, so minor 4 is accepted.
A duplicate transition task admitted after the winner released its
in-flight claim (#4839) re-reads the version before uploading, but on
unversioned buckets that read could hit a stale pre-transition entry in
the 2s-TTL GET metadata cache: transition_object never invalidated the
cache after delete_object_version persisted transition_status=complete
and freed the local data. The stale hit defeated the TRANSITION_COMPLETE
early-return, so the duplicate streamed the already-deleted local data
to the remote tier (NotFound reader errors + rejected duplicate tier
PUT with UnexpectedContent).
Invalidate the cache right after the transitioned metadata is persisted,
matching the other metadata-mutating paths, and add a regression test
that runs a duplicate transition against an already-transitioned version
and asserts no second tier upload and unchanged remote object metadata.
Fixes#4827
* docs(architecture): add normative erasure-coding algorithm and on-disk compatibility contract
Adds docs/architecture/erasure-coding.md as the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and the on-disk (xl.meta) / decode compatibility contract every future change must preserve. Grounded in the baseline (main) implementation with file:line anchors; cross-links (does not duplicate) the existing placement, MinIO-format-compat, layout-boundary, decommission, and tier-ILM docs, and the AGENTS.md cross-cutting invariants.
Covers: Reed-Solomon over GF(2^8) modern vs GF(2^16) legacy backends and how each is selected; pool/set/drive geometry with the 2..=16 set-size and per-pool parity invariants; the key-derived distribution permutation (1..=N); 1 MiB block size and the modern/legacy shard-size formulas; HighwayHash256S interleaved bitrot layout; the full xl.meta container/header/version-body schema and internal dual-key convention; write/read/heal quorum rules; and, newly codified as a first-class contract, the decode-tolerance invariants (nil-UUID/epoch-mod_time to None, skip unknown fields, hard-guard only length-critical arrays, tolerate the negative actual_size compressed sentinel, tolerate a malformed transitioned-versionID). Linked from the architecture README under "Contracts & invariants". Docs-only; check_doc_paths.sh passes.
* docs(architecture): anchor erasure spec by symbol name, not line numbers
Line numbers rot as code changes and are not validated by check_doc_paths.sh, so they would silently mislead the very code changes this normative spec is meant to guide. Replace all file:line citations with file-path + symbol-name references (functions/consts/types are greppable and rename only on deliberate changes; format byte offsets are kept). Add an explicit "references work here" note and a §13 rule that governed changes must update this spec in the same PR.
* docs(architecture): correct erasure spec after multi-expert adversarial review
Four independent adversarial reviewers fact-checked every claim against the code. Fixes:
- CRITICAL (found independently by two reviewers): §1 mislabeled the GF(2^16) reed-solomon-simd "legacy" backend as the reader for "older MinIO-lineage format". It is the opposite — that backend serves RustFS's own older main-branch (rmp_serde, uses_legacy_checksum) objects; MinIO-migrated data uses the same rs-vandermonde GF(2^8) scheme and is decoded by the modern backend. The old wording contradicted §1's own MinIO-interop line, §12, minio-file-format-compat.md, and the source comments, and would have misled the highest-stakes decode-routing decision.
- §2.1/§12: set size 2..=16 holds for multi-drive layouts; single-drive deployments run at N=1 (parity 0) outside is_valid_set_size.
- §2.2: validate_parity_inner enforces parity <= N/2 only for N > 2 (user storage-class parity flows through it); the standalone validate_parity is unconditional but only applied to the resolved default.
- §6.2/§12: header format is dispatched by header_ver; array length (4/5/7) is a per-version validation, not the discriminator.
- §6.3: the part-array length guard applies on the all_parts decode path.
- §6.5: get_bytes matches only the two canonical lowercase keys; only is_internal_key/get_str are case-insensitive.
- §6.5: transitioned-versionID — state the load-bearing invariant (non-16-byte decodes to None, never fatal); string-form recovery is optional, and transitioned-xl.meta interop is out of scope.
- §11: typo RustSF -> RustFS.
All other claims across §1-§14 were verified accurate against code (distribution formula, quorum formulas, on-disk key set/endianness/markers, decode-tolerance invariants, version anchors, standard references).
fix(s3): populate CopyObject response checksums and persist them (#4996)
A CopyObject that requested ChecksumAlgorithm=SHA256 applied the copy but returned no CopyObjectResult.ChecksumSHA256, and a later checksum-mode HEAD/GET on the destination returned nothing: execute_copy_object never read the requested algorithm, never gave the destination write a checksum, and built CopyObjectResult with ..Default::default() (all checksum fields None).
Give the destination object a checksum on the copy path. When the caller requests an algorithm, compute it fresh over the copied plaintext (the hasher sits on the innermost reader so it digests plaintext, before compression/encryption wrap it) so it persists into the object's checksum and a checksum-mode HEAD/GET returns it. When no algorithm is requested, carry the source object's stored checksum over unchanged — the copy does not transform the plaintext, so re-hashing would be wasted work and would flatten a multipart composite value. Fill CopyObjectResult from the persisted destination checksum decoded exactly the way GetObject/HeadObject do, so the response value is identical to a later checksum-mode HEAD/GET.
Add e2e regression tests: requested SHA256 is computed, returned, and HEAD-consistent; a no-algorithm copy preserves and reports the source SHA256; and a requested CRC32 over a SHA256 source is computed fresh (equals a reference CRC32 PUT), overrides the source algorithm, and is not inherited.
Fixes#4996
test(1306): pin Some(0) usage serialization, snapshot cache invalidation, and listing send classification
Follow-up test hardening for the merged admin-usage-snapshot work
(#4979/#4980/#4981/#4982, rustfs/backlog#1306). Tests only; no production
behavior change.
- B-1 madmin: pin that a scanned-but-empty bucket (Some(0)) serializes usage
stats as zeros, staying distinct from the no-snapshot (None) omitted case.
- B-2 gating: revert detector proving save_data_usage_in_backend invalidates
the 30s snapshot cache so a fresh save is visible to the next cached read.
- A-1 list_objects: pin that a successful gather_results send is never
misclassified as ConsumerGone (correct state + err sentinel delivered), and
document the wrapper Err arm invariant. A full wrapper-level producer-error
integration test is deferred as it needs the fake-disk list harness.
DeleteRestoredAction is supposed to demote a restored object back to its
pure transitioned state: remove only the local restored copy, strip the
x-amz-restore headers, and leave the version (and the remote tier data)
untouched. expire_transitioned_object set
opts.transition.expire_restored accordingly, but no delete path ever
read the flag, so delete_object ran an ordinary delete: on unversioned
buckets the whole object vanished and the free-version record scheduled
remote tier cleanup (tier data loss); on versioned buckets the latest
version got a spurious delete marker that replication propagated.
Route expire_restored explicitly in SetDisks::delete_object before
delete-marker resolution and replication dispatch: target the found
version with FileInfo.expire_restored=true and return early. The
FileMeta::delete_version layer already implements the semantics (strip
restore headers, keep the version, hand back the local data dir); this
wires it up.
Also fix the action matching in expire_transitioned_object (extracted
into transitioned_object_delete_opts): DeleteRestoredVersionAction
previously fell through to the full transitioned-object delete, which
removed the remote tier data of a noncurrent restored version. It now
routes through the same restored-copy cleanup with the exact version id,
matching MinIO's Action.DeleteVersioned()/DeleteRestored() dispatch.
Re-enable test_restore_chain_local_read_expiry_keeps_remote_and_allows_
re_restore in the ILM Integration (serial) lane; add unit tests pinning
the event->options routing and the filemeta expire_restored branch.
Closesrustfs/backlog#1302
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Replace stale dependency-depth and line-count snapshots with a domain overview based on the current Cargo workspace. This preserves the high-level architecture while avoiding references to removed crates and fragile size estimates.
Constraint: Workspace membership changes independently of architecture documentation updates
Rejected: Refresh the old numeric snapshots | they would immediately become stale again
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Cargo.toml as the source of truth for workspace membership
Tested: git diff --check; scripts/check_doc_paths.sh; cargo metadata --no-deps --format-version 1
Not-tested: make pre-commit (documentation-only change)
* fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
* fix(auth): POST-object lock fields and signed metadata=true listing 403s
Two authorization surfaces returned 403 for allowed requests (rustfs#4845): the PutObject access hook demanded PutObjectRetention/PutObjectLegalHold IAM actions for POST-object form uploads whose lock fields are governed by the validated POST policy, and the metadata=true listing route never resolved SigV4-verified credentials into ReqInfo so signed requests were evaluated as anonymous. Skip the PUT-header lock actions for POST form uploads and resolve credentials in the metadata route; re-admit the four e2e tests to the e2e-full profile.
Follow-up to backlog#1191 (optional sub-item). The main accept loop
spawned one task per socket with no global bound, so a connection flood
could exhaust file descriptors and memory and take existing traffic
down with it.
- New RUSTFS_API_MAX_CONNECTIONS (default 0 = unlimited, no semaphore
constructed, accept loop unchanged).
- When set, the loop acquires an owned semaphore permit BEFORE
accept(): at saturation it simply stops accepting and lets the
kernel backlog absorb bursts (TCP-native backpressure) instead of
accept-then-close churn. The permit moves into the connection task
and is released by RAII on any exit path, including TLS handshake
failures.
- Saturation is observable via the
rustfs_http_server_connection_cap_saturated_total counter, a
connection_cap_state startup event, and a per-wait debug event;
shutdown stays responsive while parked on the semaphore.
- The cap covers everything on the main listener (S3, admin, console,
internode gRPC); the constant docs carry sizing guidance.
- E2E coverage: ten sequential Connection-close requests against cap 2
prove permits never leak; two stalled connections saturating cap 2
leave a third unserved until their permits are released, after which
the queued request is accepted and answered.
POSIX lets rmdir report a non-empty directory as either ENOTEMPTY or EEXIST. Linux/macOS/Windows use ENOTEMPTY (ErrorKind::DirectoryNotEmpty); illumos/Solaris return EEXIST (errno 17), which Rust surfaces as ErrorKind::AlreadyExists and which a DirectoryNotEmpty match never catches.
LocalDisk::delete_file removes xl.meta and then recurses to rmdir the object directory, tolerating only NotFound and DirectoryNotEmpty. Since #4300 (transactional delete rollback-staging, new in beta9) the object directory still holds the rollback backup dir when that rmdir runs — the caller removes it only after write quorum is confirmed — so the rmdir legitimately reports "not empty". On Linux that is tolerated; on Solaris it is EEXIST, which fell through to the catch-all arm and became FileAccessDeniedWithContext. That failed the delete commit, rolled the metadata back, and left the object undeletable, so the client retried indefinitely with a spurious FileAccessDenied and no EACCES anywhere (rustfs/rustfs#4978). The same Linux-errno assumption also broke non-force DeleteBucket on a populated bucket on Solaris.
Add a portable is_dir_not_empty_error classifier (DirectoryNotEmpty kind plus raw ENOTEMPTY/EEXIST), mirroring MinIO's isSysErrNotEmpty, and use it at the two directory-removal sites via is_benign_object_rmdir_error (delete_file) and classify_delete_volume_error (delete_volume). The classifier is applied only at rmdir/remove_dir_all sites, where EEXIST unambiguously means "not empty", so EEXIST keeps its normal meaning everywhere else. rmdir never returns EEXIST on Linux/macOS/Windows, so the new raw-errno branch is unreachable there and the change is a strict no-op off illumos/Solaris.
Adds unit tests for the classifier (DirectoryNotEmpty/ENOTEMPTY/EEXIST match, EACCES/ENOENT reject, real non-empty rmdir against the host errno) and call-site decision tests that make a Solaris EEXIST regression detectable on Linux CI.
Fixes#4978
For a CopyObject whose source carries versioning, the response must echo the exact source version copied via x-amz-copy-source-version-id (SDK CopySourceVersionId), kept distinct from the newly created destination x-amz-version-id. RustFS set the destination version_id but left copy_source_version_id unset, so AWS SDK clients could not prove which source version was copied — the same field UploadPartCopy already populates.
Populate CopyObjectOutput.copy_source_version_id from the version actually read from the source, gated on the source bucket carrying versioning (enabled or suspended) and rendering the null version as "null", mirroring the GET/HEAD convention. Add an e2e regression test that copies a non-latest source version and asserts the source-version header equals the requested version, the destination header is a distinct new version, the destination bytes/size match the copied version, and the source version remains present.
Fixes#4976
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303)
The excluded test's 'missing xl.meta ... on disk2' was NOT an EC
metadata-distribution issue: after a transition all four shard disks hold
a fully consistent xl.meta (verified by decoding each shard). The panic
came from the tier test util's open_disk, which hardcoded disk_index 0
for every disk path; LocalDisk::new validates the endpoint's
(set_idx, disk_idx) against the disk's own format.json and rejected every
non-slot-0 disk with InconsistentDisk, which read_transition_meta
collapsed into 'missing xl.meta'. Derive the real indices from
format.json instead. This also un-breaks free_version_count /
wait_for_free_version_absence for non-first disks (silently 0 before).
With that fixed, the test advanced to the #4877 restore self-deadlock,
whose main paths #4886 already fixed. Complete that fix on the one path
it missed: update_restore_metadata (the restore-failure metadata
rewrite) still rebuilt copy_object options with no_lock=false and would
re-acquire the object write lock the restore handler already holds.
Propagate the caller's no_lock there too.
Remove the test from the serial-lane exclusion list; the four remaining
exclusions are unrelated known issues and stay.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP
dimension gives per-client fairness; this adds a collective per-bucket
budget so one hot bucket cannot monopolize the server regardless of how
many client IPs the traffic is spread across.
- Generalize RateLimiter over its key type with a Borrow-based check()
so &str lookups against String bucket keys stay allocation-free on
the hit path; client-IP call sites are unchanged in behavior.
- RateLimitLayer now carries optional client and bucket limiters; a
request must pass every configured dimension, and rejections report
which one tripped via a new 'dimension' metric label.
- Bucket extraction mirrors s3s host routing: virtual-hosted-style
resolves the Host/authority prefix against the same expanded domain
set (with port variants) the s3s router uses; otherwise the first
path segment. Admin and table-catalog namespaces are never buckets.
- New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 =
dimension off) under the existing enable switch; bucket-only
configurations (client RPM 0) are supported.
- Bucket names are attacker-chosen, so the bounded-shards design
(100k keys, lossless idle sweeps, most-idle eviction) is the memory
defense; a test floods 10k random names and asserts the cap holds.
- Unit tests for extraction, shared bucket budgets across IPs, both-
dimensions interaction, and the extended env matrix; e2e test proves
bucket-only throttling on the real server with an unrelated bucket
unaffected.
fix(api): return descriptive InvalidArgument reason for Windows-unsupported object keys
On Windows hosts object keys containing NTFS-reserved characters or
Win32-unaddressable path segments are rejected up front, but the client
only saw a bare "Invalid argument" (issue #3299). Attach an explicit
reason to these rejections and surface non-empty InvalidArgument reasons
through the S3 error message.
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
Add Reuse Before You Write and Necessary Code Only sections to AGENTS.md,
promote quality probes into a dedicated seventh adversarial-validation
role (simplicity adversary), extend rust-code-quality checks, and dedupe
code-change-verification's restated Rust checklist into a pointer.
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back
Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.
- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
(opaque, purpose-scoped object write lock with an is_lock_lost fence);
handle_restore_transitioned_object no longer locks the copy-back, so
HEAD/get_object_info stay non-blocking during a restore and the inner
put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
read-check-write (no_lock inside the scope), drops it before spawning
the copy-back; SELECT-type restores keep the plain read-locked path;
the copy-back pins the resolved version so a concurrent PUT cannot
strand the flagged version at ongoing=true; concurrent restores are
rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
integration test to the final semantics, and add a concurrent
double-POST test asserting exactly one acceptance and one tier GET.
PUT admission since #4928 rejected any request whose Content-Encoding declares
aws-chunked but lacks x-amz-decoded-content-length with 400 UnexpectedContent.
Whether the body is actually chunk-framed is signalled by a STREAMING-*
x-amz-content-sha256, not by the declared encoding: the s3s auth layer only
de-frames streaming payloads and already requires the decoded length for them,
so a declared-only aws-chunked request (issue #1857 clients) carries an
unframed body whose wire Content-Length is the authoritative object size.
Admit it against that length; keep failing closed for genuinely framed bodies
without a decoded length, and use the decoded length for streaming payloads
even when Content-Encoding is absent.
Refs: https://github.com/rustfs/backlog/issues/1336
Issue #4810's production incident showed the drive timeout knobs are
undiscoverable from symptoms: a listing that outruns the walk budget
gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs
existed only as code constants in crates/config/src/constants/drive.rs
with no operator-facing documentation.
Add docs/operations/drive-timeout-tuning.md covering the per-operation
drive timeout knobs, resolution precedence, the high_latency profile,
and a dedicated section on listing truncation and the walk stall
budget - including the correction that since the stall-budget rework
the foreground listing path is governed by
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob
the issue suggested. Link the guide from the README.
Ref #4810
docs(skills): single-commit release flow — preview and final tags share the validated commit
Version files are bumped once, directly to the final target version; the -preview.N suffix now exists only in tag names. The binary self-reports build::TAG and build.yml derives artifact naming and prerelease classification from the tag, so the preview tag and the final tag can point at the exact same commit — eliminating the post-validation version-bump commit that previously separated the validated hash from the released tag.
rustfs-release-version-bump gains a guard rejecting any -preview. target version.
test(e2e): add socket-level network fault-injection proxy (backlog#1325)
Add `FaultProxy`, an in-process async TCP proxy for black-box cluster E2E tests, under `crates/e2e_test/src/fault_proxy.rs`. It binds a random local port, forwards every accepted connection to a fixed target, and lets a test flip the wire behaviour at runtime.
Fault modes: `Pass` (transparent), `Latency(Duration)` (delay each forwarded chunk), `Blackhole` (accept but forward nothing either way and never respond), and `Partition(Direction)` (block exactly one direction while the other keeps flowing). Modes are switchable at runtime via `set_mode`, and `shutdown` stops the accept loop and drops the listener so the bound port is released.
This is the black-box counterpart to the in-process white-box hooks (rename barrier, disk call counters), which cannot reach across the process boundary into a spawned RustFS server. It serves the lock-plane one-way partition and accept-then-blackhole-peer acceptance for #1312/#1319 and the cross-process replay/tamper seam for #1327. Wiring it into the cluster harness (expose a node port through a proxy) plus the 5GiB / nightly CI-lane budget are follow-ups, since the harness multi-drive / 2-pool work lands separately (rustfs#4937); this block stays independent and self-tested.
Self-tests run against a loopback echo server that records received bytes, covering: pass round-trip identity, latency lower-bound delay, blackhole no-response-no-panic, one-way partition in both directions (request-path vs return-path), runtime mode switching on a live connection, and port release after shutdown. Timing assertions use loose lower bounds and bounded read timeouts only, never fixed-sleep absolute-value checks, to stay non-flaky. No product code changes; the module is `#[cfg(test)]`-gated.
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
test(ecstore): add rename/commit fan-out pause barrier and background-task introspection
Second white-box test-infra block for https://github.com/rustfs/backlog/issues/1325 (the first block landed the per-disk call counters in PR#4914). Adds a `#[cfg(test)]` awaitable pause barrier plus in-flight background-task introspection to the rename/commit fan-out in `crates/ecstore/src/set_disk/core/io_primitives.rs`, following the same dual-cfg seam style as the existing `disk_call_counters` and `cleanup_fault_injection` seams.
A test arms a barrier for `(object, disk_index, phase)`; the matching spawned fan-out task parks at its checkpoint until the test releases it, and the test awaits the pause through a deterministic `tokio::sync::Notify` handshake (no sleeps). A separate object-keyed task tracker reports how many rename/cleanup background disk tasks are still in flight, so a test can assert "a background disk write is still running" while paused and "no background disk write remains" once the fan-out drains. Both mechanisms live in one process-global registry keyed by object name, so concurrent tests using distinct object names stay isolated. Barriers are placed on the real `rename_data` fan-out (phase `rename`) and the `commit_rename_data_dir` old-data-dir cleanup fan-out (phase `cleanup`).
In production the barrier compiles to an immediately-ready `#[inline(always)]` no-op future and the task guard to `()`, so the fan-out control-flow shape and behavior are unchanged; only the `#[cfg(test)]` variants touch the registry. Coordinator lock-holding is asserted by the test at the store/coordinator layer via the guard it already holds; io_primitives has no handle to that namespace lock. Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool) remains a later cluster-harness block.
Serves the barrier-style white-box acceptances of #1312 (commit fencing: abort at the first-disk rename barrier, assert no background disk write remains after release), #1319, and #1313. Three demo tests drive the real fan-out functions and double as regression guards: neutralizing the barrier seam makes the pause await time out, and neutralizing the task guard pins the in-flight count at zero, so reverting either seam fails the demos.
Refs https://github.com/rustfs/backlog/issues/1317
Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously.
This replaces the permit-less bypass with a bounded overflow lane and a hard cap:
- A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token.
- Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`.
- A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged.
Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned.
Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here.
Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked.
Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail.
The PutObject usecase computed `must_replicate_object` twice with the same inputs: once before commit to persist the pending replication metadata, and again after commit to drive `schedule_object_replication`. Besides repeating the versioning/config/target traversal on the hot path, the two computations read the replication configuration independently, so a config hot update landing between them could split the two phases into a pending-without-schedule or schedule-without-pending divergence.
Reuse the single immutable `ReplicateDecision` computed before commit for both the pending metadata and the post-commit schedule. The two former computations were already equivalent for a stable config (`must_replicate` reads only `opts.replication_request` from the options, which the pending-suffix insertion does not touch), so this preserves replication semantics while removing the redundant traversal and closing the config-race window. The decision carries only stable target/rule/status identifiers (arn, replicate, synchronous, id) and no secrets.
Add a white-box regression test that drives a real PutObject through the usecase and asserts, via a test-only invocation counter on `must_replicate_object`, that a single PUT computes the decision exactly once; reverting to the pre-commit + post-commit recompute makes the counter observe 2 and fails the test.
Refs: https://github.com/rustfs/backlog/issues/1320
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope
Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315).
This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics:
- Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order.
- Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes.
- Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged.
Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention.
Ref: https://github.com/rustfs/backlog/issues/1315
fix(quota): admit bucket hard quota against authoritative decoded size and fail closed on checker faults
Bucket-quota admission for PUT/POST previously ran before the authoritative object length was known and used the raw wire Content-Length: for an aws-chunked upload that length counts chunk framing (overcounting) and can be absent entirely, in which case the check was silently skipped. Separately, any quota-checker fault (bucket-config read, config parse, usage lookup) degraded to allow, which silently bypasses a configured hard quota.
Resolve the authoritative decoded/plain object length first — rejecting negative and unknown lengths, and requiring x-amz-decoded-content-length for aws-chunked instead of falling back to the framed wire length — then run quota admission exactly once against that size. This is the same basis the settle phase records via ObjectInfo.size, so admission and accounting agree. When no quota is configured the QuotaChecker keeps its zero-extra-I/O fast path; once a hard quota is set, checker faults now fail closed with a retryable ServiceUnavailable, increment rustfs_bucket_quota_check_failed_total, and keep the client-facing message generic so internal config/usage details are not leaked.
Size resolution and quota-outcome mapping are extracted into pure functions (resolve_put_object_authoritative_size, map_quota_check_outcome) with unit tests covering aws-chunked decoded-vs-wire, missing/negative/unknown lengths, plain PUT, the exact/over-limit admission split, and fail-closed on checker error. QuotaCheckResult is re-exported through the ecstore api::bucket::quota surface for the app layer. Cross-node reservation and overwrite-delta accounting remain out of scope (sibling issue). Also corrects one stale doc path (set_disk/core/local.rs -> disk/local.rs) flagged by the doc-paths guard.
Refs: https://github.com/rustfs/backlog/issues/1311
CompleteMultipartUpload enqueued a normal-priority heal whenever
`rename_data` returned a `Some(versions)` signature. But the per-disk
signature is produced for every object with <=10 versions, and a healthy
quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value
conflated two distinct facts — "a version signature exists" and "the
committed replicas need heal". The result: nearly every healthy MPU
completion self-enqueued a heal, while >10-version objects (signature
`None`) did not — an algorithmic heal amplification on the healthy path
(rustfs/backlog#1321).
Replace the overloaded `Option<Vec<u8>>` second element of
`SetDisks::rename_data` with an explicit `RenameConvergence` classification
computed after the write-quorum gate:
- AllSuccessIdentical — every attempted disk committed with an identical,
known signature (no heal).
- PartialCommit — write quorum met but a disk failed/offline; a committed
replica is missing or stale (heal).
- SignatureDivergent — all committed but signatures diverge, or mix signed
(<=10-version) with unsigned (>10-version) disks (heal).
- Unknown — all committed, no signature produced (>10 versions); latent
divergence is left to the scanner backstop, not self-enqueued.
`RenameConvergence::needs_heal()` is the single decision point. The version
signature is now only comparison material; it no longer doubles as a heal
flag. The old `select_rename_data_versions` / `reduce_common_versions` /
`rename_data_versions_key` machinery that carried the conflation is removed.
The heal submission in `complete_multipart_upload` moves off the ACK
critical path into a detached task: it runs after the object lock is
dropped and after the durable `rename_data` commit, survives cancellation
of the completion future, and coalesces through the existing bounded /
deduplicated / observable heal-channel admission (one submit per degraded
completion, at most). A completion cancelled in the narrow window between
the durable commit and reaching the enqueue is scanner-backstopped, as is
the Unknown (>10-version) case.
The PUT path (`object.rs`) binds the second element as `_` and is
unchanged. The change is orthogonal to and composes with the #1312 commit
fence on the same `rename_data` path (epoch rejection is a commit-gate
failure surfaced through `Result::Err`, convergence is a post-commit
signal); documented in docs/architecture/unified-object-generation.md.
Tests: `classify_rename_convergence` white-box cases cover the full
acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline
disk, no-common-quorum split, >10-version all-success and with-failure,
mixed signed/unsigned) and fail on revert to the old "signature exists =>
heal" semantics. The decision function is tested directly rather than
through the process-global heal channel, whose receiver is owned
exclusively by the blackbox serial test (init_heal_channel is once per
binary).
Refs: https://github.com/rustfs/backlog/issues/1321
The legacy erasure-decode overlap path drove the speculative next-stripe read and the current-stripe emit with `tokio::join!`, which runs both futures to completion. When the current stripe's emit terminated the loop — a client disconnect or any emit error — the join still waited for the prefetch read, so a `Stop` could stall for a full shard-read deadline on a slow or wedged remote shard before the GET could fail.
Drive the two futures with a biased `select!` instead and, the moment emit reports `Stop`, drop the in-flight read future. Because the entire read pipeline is structured async (a `FuturesUnordered` of `read_shard` futures inside `ParallelReader::read`/`read_lockstep`, with no `tokio::spawn`), dropping the read future is a real cancellation: it drops every in-flight shard read and propagates cancellation down to the RemoteDisk/HTTP reader, leaving no background read behind. The `select!` is scoped so both pinned futures drop before `reader`/`shards` are reused, which is what performs the cancellation in the `Stop` case.
This only affects the overlap-enabled path. The default remains OFF (`prefetch_count == 1` and bitrot-decode overlap disabled), and the strictly-serial read -> reconstruct -> emit default branch is untouched and byte-for-byte identical. The `Continue` path preserves offset, short-tail, buffer recycle, and bitrot/reconstruction error ordering exactly as before.
Scope: cancel-safety only. The rollout decision (whether to enable overlap by default) still requires the Linux multi-node high-RTT three-size A/B from https://github.com/rustfs/backlog/issues/1310 and is deferred; this change does not flip the default or introduce any behavior that A/B must adjudicate.
White-box test `test_legacy_prefetch_cancels_next_read_on_emit_failure` drives the real `Erasure::decode` path with overlap enabled, a writer that fails emit, and shards that serve the first stripe then stall the next-stripe read far beyond the assertion window. Under the paused clock the read future is dropped and decode returns at virtual t~=0; reverting cancel-safety makes it wait out the shard-read timeout, so the test fails closed.
Refs: https://github.com/rustfs/backlog/issues/1310
fix(deps): pin tokio to 1.52.3 to fix dial9-tokio-telemetry build
tokio 1.52.4 made private, breaking
dial9-tokio-telemetry 0.3.14 which references it as public.
Pin tokio back to 1.52.3 until the upstream dependency fixes
compatibility.
Also fix a stale doc path reference in unified-object-generation.md
where is a planned file that does not exist yet.
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
test(ecstore): add per-disk call-counter registry for metadata fan-out
First landable piece of the backlog#1325 test-infrastructure work: a test-only, per-disk call-counter registry that can observe `read_version` RPC counts recorded inside `tokio::spawn` tasks. This unblocks the RPC-count assertions in backlog #1309 / #1314 / #1315, which the thread-local `CapturingRecorder` cannot serve because it is blind to metrics emitted from spawned tasks.
The new `#[cfg(test)]` module `disk_call_counters` (modeled on the existing `cleanup_fault_injection` seam) is a process-global registry keyed by object name; an RAII `observe(object)` scope collects per-disk counts and clears only its own on drop, so parallel tests using distinct object names stay isolated. A dual-`cfg` `SetDisks::record_read_version_call` seam records from inside both metadata-fanout `read_version` spawn sites for online disks only; the `#[cfg(not(test))]` variant is an empty `#[inline(always)]` fn, so production runtime behavior is unchanged.
Two demo/regression tests prove the facility works across worker threads and is revert-detecting (neutralizing the recorder makes them fail).
Refs: https://github.com/rustfs/backlog/issues/1325
* fix(get): enforce strict exact-length materialization for in-memory GET bodies (#1324)
The streaming GET path already fails a short read with UnexpectedEof, but several in-memory materialization branches only WARNed on a length mismatch and then served the body anyway: the encrypted-buffer path, the direct-memory/cache buffered-body path, and the seek-support buffer. Most dangerously, the seek-support path fell through to streaming the *same* reader after read_to_end had already drained K bytes on error, shipping a body missing its prefix (prefix-misaligned data — the closest this code came to returning wrong bytes rather than merely a short body).
Because the HTTP response commits to Content-Length == expected before the body is produced, any other body length is an unrecoverable, broken response. This change gives every in-memory source the same exact-length contract that the ODC materialize-fill path already had.
- Add strict_materialize_object_body, a shared helper that bounds the read to expected+1 (so an over-long stream is detected without buffering it unbounded), requires bytes_read == expected, and on any read error returns without reusing the partially consumed reader. The encrypted, seek, and ODC materialize branches now all route through it.
- The buffered-body branch hard-fails a length mismatch before headers instead of WARN-and-serve.
- MemoryTrackedBytesStream gains a defense-in-depth guard: a buffer whose length disagrees with the declared content length yields a stream error on first poll instead of a clean short body or a silently truncated over-long body. This backstops the cache-hit paths that hand bytes straight to the blob.
The compatibility boundary is preserved: expected is derived from get_actual_size(), the same value used for the committed Content-Length, so a legitimate object (including a legacy/backfilled-size object) whose decoded bytes equal its declared size still serves cleanly — only genuine short, over-long, or errored reads fail. This matches the streaming path, which already hard-fails short reads, so no new large-object behavior is introduced.
Tests: each source (encrypted/seek via the shared helper, cache via build_get_object_body_with_cache, buffered/memory via MemoryTrackedBytesStream) is exercised with expected N and actual N-1 / N / N+1 plus a read-K-then-error injection; only the exact-length read succeeds. Reversal is guarded: restoring WARN-and-serve or a partial fallback flips the short/over-long/error assertions from Err to Ok. Existing streaming UnexpectedEof tests are unchanged.
Refs: https://github.com/rustfs/backlog/issues/1324
* fix(app): reword WARNed comment to satisfy typos check
s3s parses a `Range` suffix length as `u64`, but the GET and HEAD handlers cast it straight to `i64` and bypass any satisfiability check. This truncates deterministically: `bytes=-18446744073709551615` wraps to `-1` and is then read as "last 1 byte", and `bytes=-0` produces a 0-length 206 instead of a 416.
Both handlers now share a single `range_to_http_range_spec` conversion that rejects a zero-length suffix with `InvalidRange` (416), clamps any suffix above `i64::MAX` to `i64::MAX` (such a suffix always covers the whole object, which `HTTPRangeSpec::get_length` then clamps to the real size), and keeps the int branch as a checked cast (s3s already caps `first`/`last` at `i64::MAX`). The scattered `as i64` casts are removed.
Note on ordering: a zero-length suffix is now rejected at conversion time, so `bytes=-0` on a missing object returns 416 rather than 404. This matches the handler's existing behavior of validating range shape (range + partNumber -> 400) before object existence.
Adds a table-driven unit test covering suffix `0/1/size/size+1/i64::MAX/i64::MAX+1/u64::MAX` over empty, 1-byte, and normal objects, asserting the InvalidRange (416) mapping, full-object return for over-size suffixes, and no regression of int first-last / open-ended ranges.
Refs: https://github.com/rustfs/backlog/issues/1322
Remove hard line wrapping from rustfs-release-publish prose (one logical line per sentence/paragraph, soft wrap for display) and link SemVer 2.0.0 spec including spec item 11 for prerelease precedence.
Require explicit target-version confirmation (AskUserQuestion with concrete semver candidates derived from the latest tag) before any release phase runs, and document SemVer 2.0.0 precedence and bump-type rules.
Adds a project skill that orchestrates the full release flow: preview tag first, CI/artifact verification, local run with console checks, rc client command validation, then final version bump and tag cut from the validated preview hash instead of latest main.
Add the shared design/contract document for backlog #1326: a single
per-object generation authority (the #1312 fencing epoch) that spans
commit fencing, read lease, prepared pool read, quota reservation, and
old-dir GC. Pins the five cross-cutting constraints once - RPC signature
binding with server-side nonce enforcement, xl.meta encoding contract
(no meta_ver bump, no positional FileInfo field, internal metadata map
under the dual-key contract), proto3 optional presence, mixed-version
fallback direction, and cluster-level capability negotiation - so the
implementation sub-issues follow them rather than each re-deciding.
No product code changes.
fix(docker): upgrade base-image packages in runtime stages
Trivy code scanning reports 40 open alerts against the published
container images, all from OS packages frozen at the base-image tag:
- musl image (alpine:3.23.4): openssl libssl3/libcrypto3 3.5.6-r0,
including HIGH CVE-2026-45447; fixed in 3.5.7-r0
- glibc image (ubuntu:24.04): tar, gzip, perl-base, ncurses CVEs with
fixes already published in the Ubuntu archive
The runtime stages only ever installed packages and never upgraded the
ones shipped with the base image, so distro security fixes could not
reach released images until the base tag itself moved. Run
`apk upgrade` / `apt-get upgrade -y` in the runtime stages of
Dockerfile, Dockerfile.glibc and Dockerfile.source so each build picks
up current security fixes.
Verified against the exact base tags: after upgrade, alpine 3.23.4
resolves libssl3/libcrypto3 3.5.7-r0 and ubuntu 24.04 resolves
tar 1.35+dfsg-3ubuntu0.2, gzip 1.12-1ubuntu3.2,
perl-base 5.38.2-3.2ubuntu0.3, ncurses 6.4+20240113-1ubuntu2.1 —
matching every fixed version demanded by the open alerts.
PR #4582 gated the update-latest-version job to stable release tags so prereleases could not overwrite the stable version pointer. But the project currently ships prerelease tags only (beta previews), so the job never runs and latest.json has been stale since. Run the job for every release tag again, and keep the honest half of the #4582 fix by writing release_type from the actual build type instead of hardcoding "stable".
* helm chart: one data drive per node - fix1
* refactor(helm): prevent negative or 0 replicaCount
Co-authored-by: Copilot <copilot@github.com>
* refactor(helm): remove env REPLICA_COUNT
* feat(helm): default no logs directory to force stdout
Co-authored-by: Copilot <copilot@github.com>
* feat(helm): add drivesPerNode
* feat(helm): correct default parity with 4 nodes / 1 drive per node
* conditional render of RUSTFS_OBS_LOG_DIRECTORY
* feat(chart): add table doc for parity
* fix(chart): handle invalid annotation objects
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* feat(chart): better table doc for parity
* fix(chart): leave RUSTFS_OBS_LOG_DIRECTORY empty, or the defaults will attempt to write to readonly fs
* fix(chart): chart defaults as the previous version: 4 nodes with 4 drives per node
* feat(chart): render RUSTFS_STORAGE_CLASS_STANDARD in configmap
* minor fix for pvcAnnotations
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
* fix(chart): better drivesPerNode handling
* fix(chart): obs_log_directory default non empty again to preserve previous deployments compatibility
* fix(chart): proper drivesPerNode impl
* fix(chart): values clarification for empty vs unset `obs_log_directory`
* feat(chart): add service externalIPs
* feat(helm): add optional service labels
* fix(helm): merge service labels
* fix(helm): storageclass pvcAnnotations
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* fix(helm): remove duplicate keys
* fix(helm): default drivesPerNode null, keep legacy chart behavior
* feat(helm): add template test for externalIPs
* fix(helm): per-pool drivesPerNode inference, restore regression tests
Repairs the drivesPerNode feature from #2693 so it renders and stays
upgrade-safe:
- define the missing drives inference (rustfs.poolDrives) and compute it
per pool inside rustfs.pools, so mixed 4x4 + 16x1 pool deployments keep
their exact legacy volumeClaimTemplates when drivesPerNode is unset
- restore the $poolsEnabled definition dropped in the rebase (chart failed
to render at all)
- fix .Values references inside the pool range (dot is the pool there) and
keep per-pool storageclass pvcAnnotations overrides working
- make rustfs.volumes derive the drive range from pool drives instead of
the pod count, and relax the pools.list 4-or-16 restriction to >= 2
- reject drivesPerNode=0 explicitly instead of silently inferring
- keep default rendering identical to main: storage_class_standard stays
unrendered by default, obs_endpoint.use_stdout stays false, clusterDomain
value restored
- restore the clusterDomain/mTLS SAN/explicit-volumes regression tests that
the branch deleted, keeping the new topology tests
---------
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Remove the Intel macOS entry from the Build and Release platform matrix so official release builds only publish the Apple Silicon macOS binary.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ilm): fix restore test object key to match transition filter
restore_object_usecase_reports_ongoing_conflict_and_completion used the
object key "restore/api-object.bin", but the shared
set_bucket_lifecycle_transition_with_tier helper only transitions objects
under the "test/" prefix. enqueue_transition_for_existing_objects therefore
matched nothing and wait_for_transition timed out at 15s, failing the test
deterministically.
The test was added in #4860 but its ILM Integration (serial) lane is
skipped on regular PRs, so it merged red and has failed on every main run
since. Move the object under the test/ prefix like every passing sibling
test in this file.
* ci(ilm): exclude broken RestoreObject API test from serial lane
restore_object_usecase_reports_ongoing_conflict_and_completion exposes a
real regression, not a test bug: the RestoreObject copy-back
(handle_restore_transitioned_object) now holds the object write lock added
in #4877 across the entire tier read-back, so it never releases in time and
the test's concurrent get_object_info times out with Lock(Timeout, 5s). The
failure is deterministic and independent of the mock tier's injected latency.
This is the same class of known-broken restore/transition failure already
tracked under backlog#1148 (three sibling scanner tests are excluded here by
name for the same reason), so exclude this one the same way until the restore
copy-back path is fixed or the #4877 lock scope is revisited. The prior commit
keeps its correct fix (the object key must live under the test/ transition
prefix); that was masking this deeper issue by never letting the object
transition in the first place.
Restore copy-back deadlock/hang under the #4877 lock is escalated separately
for a product-level decision (fix the copy-back vs. narrow/revert #4877).
* test(ilm): fix scanner restore test object keys to match transition filter
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore and
test_multipart_restore_preserves_parts_and_etag (both added in #4860) keyed
their objects under restore/ instead of the test/ prefix that
set_bucket_lifecycle_transition_with_tier filters on, so the objects never
transitioned and wait_for_transition timed out at 15s.
These surfaced only after the prior commit excluded the rustfs-side restore
API test: nextest runs -j1 fail-fast, so that earlier failure stopped the run
before these scanner tests executed. Unlike the excluded API test, both call
restore_transitioned_object().await sequentially and only read afterwards, so
they don't hit the concurrent-read-vs-#4877-write-lock timeout; the key
prefix was their only problem.
* ci(ilm): exclude the two remaining #4877-broken restore tests
test_multipart_restore_preserves_parts_and_etag and
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore both
call restore_transitioned_object().await, which since #4877 acquires the
object write lock and deterministically times out (Lock Timeout, 5s) against
an already-held lock, so restore never completes. They surfaced one at a time
because nextest runs -j1 fail-fast. The earlier prefix fix was necessary but
only advanced them from the transition wait to this restore-lock timeout.
Exclude both by name alongside their already-excluded sibling
test_transition_and_restore_flows (same root cause, tracked under
backlog#1148) so the ILM Integration (serial) lane goes green. The #4877 lock
scope still needs a product fix before any of these re-enable.
* docs(ilm): describe the excluded restore tests' symptom as a lock timeout, not a deadlock
The #4877 write lock is held across the tier read-back and outlives the 5s
lock timeout; nothing proves a true deadlock. Wording flagged by Copilot
review.
* fix(ecstore): stop restore copy-back self-deadlocking on the #4877 write lock
#4877 made handle_restore_transitioned_object hold the object write lock for
the whole restore and forward no_lock=true so the set layer would not
reacquire it. But the set-level copy-back rebuilds its own options
(put_restore_opts -> ropts, and the complete_multipart_upload opts) that
default no_lock=false, so the inner put_object / new_multipart_upload /
complete_multipart_upload each re-acquire this object's write lock in their
commit phase and block on the lock the restore already holds -> Lock(Timeout,
5s), and restore never completes.
Confirmed via RUSTFS_OBJECT_LOCK_DIAG_ENABLE: restore_transitioned_object
acquires the write lock, then holds it ~10.5s across two nested 5s acquire
timeouts before failing. This is a real product deadlock: a RestoreObject on
any transitioned object (multipart especially) hangs, not just the tests.
Propagate no_lock into the copy-back options so the inner writes inherit the
already-held lock. Use opts.no_lock (not a hardcoded true) so a caller that
restores without the outer lock still locks correctly. put_object_part is left
as-is: it locks the multipart upload-id resource, not the object key, so it
does not conflict. Verified test_multipart_restore_preserves_parts_and_etag
now passes (3.6s, was a 15s+ hang).
* ci(ilm): re-enable multipart restore test; scope remaining exclusions
The prior commit fixes the #4877 restore self-deadlock, so
test_multipart_restore_preserves_parts_and_etag passes again - drop it from
the serial-lane exclusion list and remove its 'currently excluded' note.
The other restore/transition tests still fail, but each on a DIFFERENT,
independent issue unrelated to the (now-fixed) lock, verified locally:
- test_restore_chain_...: DeleteRestoredAction sets expire_restored but no
delete path reads it, so cleanup deletes the whole object (unimplemented
semantics), not the local restored copy only.
- test_transition_and_restore_flows: transition xl.meta missing on one drive
(EC metadata distribution), not restore.
- restore_object_usecase_reports_ongoing_conflict_and_completion: asserts a
concurrent mid-restore ongoing=true read that #4877's read-vs-restore
serialization rules out (backlog#1148 ilm-8 criterion 1, an API-semantics
decision).
Comments and #[ignore] reasons updated to reflect each real cause. All remain
tracked under backlog#1148.
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
* chore(deps): tighten crate dependency features
Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): trim hyper-rustls features
Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`restore_object_usecase_reports_ongoing_conflict_and_completion` failed
deterministically in the "ILM Integration (serial)" lane, panicking at
its setup step ("object should transition before the restore API runs"):
the object never reached transition status `complete` within the 15s
wait, before the restore logic under test even ran.
Root cause: the test object key was `restore/api-object.bin`, but
`set_bucket_lifecycle_transition_with_tier` scopes the transition rule to
`<Filter><Prefix>test/</Prefix>`. An object outside that prefix never
matches the rule, so `enqueue_transition_for_existing_objects` never
enqueues it (confirmed: the mock tier saw zero puts and the object's
transitioned status stayed empty) and `wait_for_transition` times out.
Every other transition test in this file already keys its objects under
`test/`. The break shipped in #4860 because that PR's ILM serial lane was
skipped on the merge run, so the test never actually executed in CI.
Fix: key the object as `test/restore/api-object.bin` so it matches the
rule. The restore API, tier copy-back, and local-GET assertions are
unchanged (they address the object by bucket+key; the remote tier name is
generated internally).
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
Avoid compiling the mimalloc JSON parsing helpers on non-test Windows builds so the platform-specific dead_code warnings disappear while tests keep coverage through #[cfg(test)].
Co-authored-by: heihutu <heihutu@gmail.com>
`test_heal_resume_across_page_boundary_e2e` panicked whenever the
erasure-set healer deferred a single object version to a later heal
cycle. Under load a per-version `heal_object` can hit a transient error;
`ErasureSetHealer::heal_bucket_with_resume` then persists its
resume/checkpoint state and returns a terminal `Failed { .. "retry
scheduled" }`, expecting a fresh heal run to finish the job. In
production the background scanner is that next run — the e2e had no such
follow-up, so the first task's `Failed` state failed the test. This is a
pre-existing rare flake (the wiped-disk heal is otherwise correct); it is
unrelated to any policy/proptest work that happened to surface it in CI.
Drive the heal to a genuine `Completed` instead: on a `Failed` carrying
the `retry scheduled` marker (retry budget still remaining) re-submit the
idempotent heal (`force_start`), mirroring the production scanner, up to a
small bound. A `Failed` without that marker (e.g. `exhausted retries`) or
any other non-`Completed` terminal state still fails the test, and the
strict per-version data-restoration assertions still run only after a
real `Completed`. `wait_for_task` is refactored onto the shared
`await_terminal_status` poller; its panic-on-failure semantics for the
unversioned-bucket heal are unchanged.
Test-only change; no production heal code is touched.
Co-authored-by: heihutu <heihutu@gmail.com>
crates/policy had zero property tests: the wildcard Action/Resource matching
and Deny-first evaluation in Policy::is_allowed — the algebra authorization
safety rests on — were guarded only by example-based tests.
Add crates/policy/tests/policy_eval_proptest.rs with three generated-input
invariants (pure evaluation, no IO or global state, parallel-safe):
- explicit_deny_anywhere_denies: a Deny matching the request denies it no
matter how many broad Allow statements co-exist or at which index the Deny
sits; a built-in sanity assertion first proves the Allows alone would permit,
so the Deny is what flips the decision.
- wildcard_superset_implies_concrete_match: any request allowed by an
exact-action/exact-resource policy is also allowed after widening to s3:* on
bucket/* and to * on arn:aws:s3:::*, checked both on the built grant and on
random probe requests (implication form).
- empty_policy_denies_everything: a statement-less policy and Policy::default()
deny every non-owner request.
Statements are built from JSON exactly as production policies arrive via
PutPolicy. Generated names avoid wildcard metacharacters so the only wildcards
in play are the ones the properties introduce deliberately. proptest joins
crates/policy dev-deps following the filemeta/ecstore precedent.
Refs: backlog#1151 (sec-9)
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the
latest main to pull the newest compatible versions.
Manifest bumps (Cargo.toml):
- redis 1.3.0 -> 1.4.0
- xxhash-rust 0.8.16 -> 0.8.17
Lockfile-only bumps: simd-adler32 0.3.10, syn 2.0.119, and
toml_edit 0.25.13. `ratelimit` is held at 0.10.1 as requested.
Verification: cargo check --workspace --all-targets, cargo clippy
--all-targets -D warnings (plus the rio-v2 feature variant), cargo nextest
run --all --exclude e2e_test (8964 passed), and cargo deny check
advisories/sources/bans/licenses all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap
backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:
- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
/tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
init_bucket_metadata (default true; the iam bootstrap test opts out
to preserve its historical semantics). TestECStoreEnv exposes
temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
the sanctioned test-compat boundary pattern (mirrors
crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
for heal_env{,_n} wrappers; the tests/storage_api.rs integration
surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
ecstore_test_compat fixture shrinks to SetupType +
update_erasure_type.
rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.
* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move
* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
refactor(ecstore): make client base64 standard everywhere, drop the split
Self-review of the transition-client fixes: the Content-MD5 fix had only
converted the two transition call sites to a parallel base64_encode_standard,
leaving the same URL-safe, unpadded base64 bug on every other outbound value —
the SigV2/no-length multipart Content-MD5, the x-amz-checksum-* headers on the
parallel and no-length paths, api_remove's multi-delete Content-MD5, and the
checksum encode/decode helpers. Every base64 value this client emits or parses
is S3 wire format, which is standard base64; none is ever URL-safe.
Encode and decode both use base64_simd::STANDARD now, and the parallel
base64_encode_standard is gone. This fixes those seven latent call sites at the
root and removes the duplicate function. The transition path is functionally
unchanged (it already produced standard base64 via the helper), so the
end-to-end byte-for-byte result is preserved.
Also drop a stray Vec::with_capacity(part_size) that was allocated (up to
128 MiB) and immediately overwritten by read_multipart_part's own buffer.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-authored-by: heihutu <heihutu@gmail.com>
* ci: add e2e-full merge-gate job (single-node full e2e via nextest)
Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in
ci.yml (push main + merge_group + workflow_dispatch) that runs the
never-automated user-visible e2e suites (KMS, object_lock, multipart_auth,
quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke
subset skips.
The filter is the whole e2e_test crate minus the sets other lanes own:
protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites
(ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly
lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are
serialized, mirroring the ci profile. Reuses the downloaded debug binary and
uploads junit. backlog#1149 ci-5.
* ci: exclude characterized known-failures from e2e-full with tracked issues
First-ever automated run of the full suite (dispatch 29381309848: 341 ran /
32 failed / 460s) surfaced five real product-bug families, each now tracked:
rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit
vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845
(object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum
misclassified as timeout under load). Deterministic failures cannot be
retried away, so each family is excluded with its OPEN issue cited and the
fixing PR contract to delete the exclusion — passing negative-path siblings
stay in as regression guards.
xl.meta roundtrip coverage was one fixed example test plus byte-level no-panic
fuzz properties; nothing generated STRUCTURED version graphs, so a lossy
encoding bug in version headers, ordering, delete markers, or the dual
internal-metadata-key handling would only surface if the fixed example happened
to hit it.
Add crates/filemeta/tests/version_graph_roundtrip_proptest.rs with two
generated-input properties over multi-version graphs (1-7 versions, ~30% delete
markers, deliberately colliding mod_times to exercise tie-break ordering, user
metadata, and the AGENTS.md dual x-rustfs-internal-*/x-minio-internal-* keys
written via the real insert_bytes helper):
- version_graph_marshal_load_roundtrip: marshal -> load preserves version
count, meta_ver, the exact (id, type) sequence, full headers, fully decoded
per-version content, delete-marker payload shape, and both internal metadata
key forms with identical values.
- version_graph_marshal_is_idempotent: marshal(load(marshal(x))) is
byte-identical to marshal(x), catching encoders that mutate or reorder state
on the way out.
Complements the existing byte-level no-panic properties: those prove hostile
bytes cannot crash the decoder; these prove honest graphs are encoded
losslessly.
Refs: backlog#1151 (sec-10)
Register the webhook target while its endpoint is reachable, then drop
the listener before the PUT: registering against a dead endpoint stalls
behind the reachability probe timeout and flaked the ARN wait on the
rustfs#4821 merge run. Also widen the registration wait to 20s.
A single PUT can enqueue the same object for transition twice — immediately
(enqueue_transition_after_write) and again from the startup/lifecycle
compensation backfill. transition_object's own namespace lock is commented out,
so the two attempts are not serialized as same-object work: the winner
transitions the object and removes its local data, and the loser then reads that
already-removed data via get_object_fileinfo(read_data = true) and logs a
spurious "get_object_with_fileinfo err ... No such file" /
lifecycle_tier_operation_failed. For a large (multipart) object both attempts
can also race the source read, corrupting the transferred copy.
Guard the transition queue with an in-flight set keyed by
(bucket, object, version). queue_transition_task claims the key before sending;
a duplicate enqueue is reported handled without queueing a second task. The
worker releases the claim once the transition finishes, so a later lifecycle
pass can still re-transition the object, and a failed enqueue releases it
immediately. This is the sole path into the transition channel, so every
enqueue is covered.
Surfaced while validating the >128 MiB transition fix end-to-end in Docker
(rustfs/rustfs#4811); tracked as its own defect since it is unrelated to the
checksum/multipart-client bugs.
Tests: same-version enqueues dedupe (direct reserve and via queue_transition_task
with spare capacity), a distinct object still hits the full-queue path, and a
released claim can be re-acquired.
Refs: rustfs/backlog#1268
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed
ILM transition of any object larger than 128 MiB to a RustFS-native tier
(rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the
built-in TransitionClient) failed with "unsupported checksum type", while
objects <=128 MiB transitioned fine.
Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured
checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of
the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The
128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single
PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path,
`put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`,
disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which
returns the "unsupported checksum type" error. The single-PUT path hit the same
misjudgement but never calls `hasher()`, so it silently succeeded (without a
checksum), which is why only >128 MiB objects failed.
Fix:
- `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject`
flag, which has no base algorithm). This is the sole callers' intended
meaning: a concrete algorithm with a real hasher is selected.
- Defense in depth: guard the multipart checksum branch on
`auto_checksum.is_set()` so an unset mode uploads the part without a per-part
checksum header instead of hard-failing in `hasher()`.
Only the TransitionClient consumes this `ChecksumMode::is_set()`; the
server-side data path uses the unrelated `rustfs_rio::ChecksumType`.
Tests: is_set()/set_default semantics, hasher parity for every set mode, and a
`build_transition_put_options` invariant (checksum unset + Content-MD5 on).
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): read exactly one part per multipart chunk in transition uploads
Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811),
uncovered while verifying the checksum fix.
`put_object_multipart_stream_optional_checksum` read each part with
`read_all()` / `to_vec()`, which drained the entire source into the first part
and left every later part empty. Any multipart upload of a streamed
(`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the
single-part path and were unaffected; a 128 MiB + 1 byte object splits into a
128 MiB part plus a 1 byte part, so the first part received the whole object and
its declared Content-Length (part_size) did not match the body.
Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts,
and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes,
leaving 0 for part 2.
Fix:
- Add `read_multipart_part`, which reads exactly the requested part size (or
less at EOF) and advances the reader, for both `Body` (in-memory) and
`ObjectBody` (streamed) sources.
- Upload each part with the bytes actually read (`length`) as its size, and
account uploaded size by actual bytes, so a short read is detected instead of
masked.
The concurrent (`put_object_multipart_stream_parallel`) and SigV2
(`put_object_multipart`) paths share the same `read_all()` pattern but are not
exercised by transition; left untouched here and noted for follow-up.
Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for
both streamed and in-memory bodies, consumes the source fully, and stops at EOF
without overrun.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): complete the >128 MiB ILM transition multipart client
Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a
128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the
multipart transition path, each masked by the previous one. With the checksum
and part-splitting fixes in place the transition now failed later and later,
and finally produced a 0-byte object with no error at all. Fixed together:
- initiate_multipart_upload discarded the CreateMultipartUpload response and
returned an empty UploadId, so the first UploadPart failed with "UploadID
cannot be empty". Parse the response XML (InitiateMultipartUploadResult now
derives Deserialize with PascalCase).
- Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64,
which the remote rejected as "Invalid content MD5: Base64Error". Add
base64_encode_standard and use it for those outbound header values.
- PutObjectOptions::default() set legalhold to OFF, so header() attached
x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was
rejected with "does not accept object lock or governance bypass headers".
Default to an empty (unset) status.
- CompleteMultipartUpload / CompletePart had no serde renames, so the request
body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed
zero <Part> elements and completed a 0-byte object while returning 200. Emit
S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields.
Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote
tier and reads back (transparently restored) byte-for-byte identical
(sha256 match), with none of the four prior errors in the logs.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The individual policy layers were tested in isolation, but the cross-layer
resolution — which layer wins when IAM and bucket policy disagree — had no
coverage. A priority error there is a data-exposure or data-lockout bug.
Add crates/policy/tests/bucket_iam_authz_matrix.rs, a pure table-driven test
(no globals, no IAM store, no server) that drives the real Policy::is_allowed
and BucketPolicy::is_allowed through a helper modeling the request-layer
orchestration in rustfs::storage::access::authorize_request (bucket explicit-Deny
gate -> IAM allow -> bucket allow fallback; anonymous = bucket policy alone). It
pins all four Allow/Deny quadrants plus the anonymous case, and adds intra-policy
Deny-beats-Allow checks for both layers.
The matrix surfaces the resolution invariant explicitly: a BUCKET explicit Deny
is a hard gate and always wins, but an IAM explicit Deny is SOFT — a bucket Allow
fallback overrides it, which diverges from AWS "an explicit Deny in any policy
always wins". The test characterizes the current (MinIO-lineage) behavior; if IAM
Deny is later hardened into a gate, iam_deny_x_bucket_allow flips to false and
must be updated deliberately.
Refs: backlog#1151 (sec-8)
Co-authored-by: houseme <housemecn@gmail.com>
test(security): pin GHSA-m77q STS root-secret token signing (sec-7)
GHSA-m77q-r63m-pj89 (intentionally UNFIXED) is that STS session tokens are
signed with the shared root secret: crates/iam/src/root_credentials.rs
token_signing_key() returns the root secret_key, so anyone holding the root
secret can forge STS session tokens. No test named the advisory, and the
existing test_created_sts_credentials_authorize_with_session_token_claims uses
token_signing_key() for both signing and verifying, so it pins "same key signs
and verifies" but not the m77q-specific "signing key IS the root secret" — a
future fix that decouples the STS key from the root secret would pass it
silently.
Add a flow-level pin, test_ghsa_m77q_sts_session_token_signed_with_root_secret,
that captures the advisory's exact signature: (1) token_signing_key() == the
root secret; (2) an AssumeRole-style session token issued with that key decodes
with the root secret and NOT with any other secret; (3) it authorizes through
the STS path. All three assert CURRENT (by-design-vulnerable) behavior, so a
real m77q fix (a dedicated STS signing key) turns them red and forces a
red -> green regression update.
Also GHSA-name the existing characterization test and token_signing_key() with
doc comments and the advisory URL, and update the m77q row in
docs/testing/security-regressions.md. No production behavior changes.
Refs: backlog#1151 (sec-7)
Co-authored-by: houseme <housemecn@gmail.com>
chore(scripts): index scripts/ and archive 29 one-shot scripts
backlog#1153 infra-13. scripts/ had 80+ unlabelled top-level entries
mixing CI gates with finished one-shot issue-validation scripts.
- scripts/README.md — one index row per entry with status
(ci-gate / dev-tool / archived), purpose, and wiring; subdirectories
get one row each. run_scanner_benchmarks.sh is annotated
"disposition owned by backlog perf-10" and deliberately untouched.
- git mv 29 confirmed-stale one-shot entries to scripts/archive/:
11 issue-scoped validation/perf-capture scripts, the 5-script
backlog#706 large-PUT breakdown family, the 4-file GET-optimization
stress suite, 2 gt1g one-shots, and 7 other orphaned one-shots.
Evidence: a whole-tree boundary-aware reference census showed zero
references from CI/Makefiles/docs/code for every moved entry (or
references only from other scripts inside the same archived set);
re-run after the move shows zero dangling references.
- docs/testing/README.md links the index.
Moves only — no script content changed.
bench(rustfs): remove the fake s3_operations benchmark
rustfs/benches/s3_operations.rs never measured S3: all three groups
(put_object/get_object/list_objects) only black_box(data.len())/black_box(count)
with a 'In a real benchmark, this would call the actual S3 client' comment. It
gave a false 'S3 operations have benchmark coverage' signal and was dead weight
on rustfs/Cargo.toml.
Delete the file and its [[bench]] entry. S3-face performance is covered solely
by the warp A/B gate (performance-ab.yml); the runbook scope note now says so
explicitly. Micro-benchmarks stay at the function level (perf-8).
Refs rustfs/backlog#1152 (perf-9).
The script cannot run on any machine: WORKSPACE_ROOT is hardcoded to a personal
path (/home/dandan/code/rust/rustfs) and it targets the rustfs-ahm crate, which
no longer exists in the workspace (scanner is crates/scanner, heal is
crates/heal, neither ships benches/). It gave a false impression that scanner
performance automation existed.
There is no current scanner-bench requirement, so retire it (option a). Nothing
references the script anywhere else in the repo.
Refs rustfs/backlog#1152 (perf-10).
ci(coverage): weekly cargo-llvm-cov workspace baseline (non-blocking)
Add the weekly line-coverage report (backlog#1153 infra-5):
- .github/workflows/coverage.yml — Sunday 07:00 UTC + workflow_dispatch;
runs `cargo llvm-cov nextest --workspace --exclude e2e_test` under
NEXTEST_PROFILE=ci (same scope and profile as the ci.yml test gate),
writes a per-crate line-coverage table to the job summary, uploads
lcov + JSON as a 90-day artifact, and routes scheduled failures
through the shared schedule-failure-issue action (ci-8). Runs only on
schedule/dispatch, so it can never become a required PR check.
- scripts/coverage_per_crate.py — stdlib-only aggregation of the
llvm-cov JSON export into the per-crate markdown table (worst-first,
TOTAL row), shared by the workflow and the local target.
- make coverage (.config/make/coverage.mak) — local equivalent with the
same command sequence; fails with install hints when cargo-llvm-cov
or cargo-nextest are missing. Listed in make help.
- docs/testing/README.md — Coverage section: cadence, where the table
and artifacts live, the trend-comparison method, and what is not
measured (doctests, e2e_test).
Every push to main now builds the release binary once and stores it in the
actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job
restores it for origin/main and passes --baseline-bin; on the nightly, where
the candidate commit equals the baseline, one cached binary serves both phases
with --skip-build and the run does zero source builds. A cache miss falls back
to the source double-build via --baseline-ref origin/main.
This removes the ~32min-per-side double build that pushed the 24-cell nightly
past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout).
Also:
- alert-on-failure now fires on cancelled/timed-out, not just failure, so a
timed-out nightly is no longer silent (the composite action already reports
cancelled jobs); removed the stale perf-2 TODO now that the alert job exists.
- run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and
candidate SHAs + binary source, runner, warp version, matrix params) via a
repeatable --provenance-note; the gate exit code is preserved.
Refs rustfs/backlog#1152 (perf-3).
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
ci(mint): bypass local physical-disk-independence guard so mint boots
The mint container maps four RUSTFS_VOLUMES data dirs onto a single
runner device, so the startup physical-disk-independence guard aborts
with a FATAL before mint can run. The scheduled run 29183544431
(2026-07-12) crashed at 'Wait for RustFS ready' with 'local erasure
endpoints must use distinct physical disks'.
Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true on the container, the CI use
the guard explicitly sanctions -- mirroring the e2e-s3tests harness
fixed in #4768. Completes the ci-2 acceptance (a mint run that actually
produces log.json and the per-suite summary).
docs(testing): populate testing pyramid, naming, serial/nextest rules
Fill the docs/testing/README.md skeleton (backlog#1153 infra-11):
- Test taxonomy table for all eight layers (unit / ecstore black-box /
e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench) with a
verified entry command and a qualitative "when it runs" per layer; the
event x timeout x required-status matrix stays owned by
docs/testing/ci-gates.md (ci-15), linked not duplicated.
- Naming conventions section with the migration-gate reserved substrings
(data_movement / rebalance / decommission / source_cleanup /
delete_marker) linking the infra-12 count-floor guard, closing that
task's docs cross-link.
- Serial execution & nextest rules: nextest as a hard dependency with the
RUSTFS_ALLOW_CARGO_TEST_FALLBACK escape hatch and the runner-semantics
difference (folds the infra-14 README half), why #[serial] is a no-op
under nextest, and the default/ci/e2e-smoke/e2e-repl-nightly profiles.
- A time-control placeholder for infra-4 to fill.
The pre-existing flake-policy section (ci-10) is preserved verbatim.
Add pointers from CLAUDE.md and CONTRIBUTING.md.
* test(ecstore): assert decrypted_size for MinIO SSE interop round-trip
The ignored MinIO interop round-trip tests asserted `ObjectInfo.size`
against the plaintext length. For SSE objects `size` is the on-disk
DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), so the
assertion can never hold once real fixtures are present — the two
`#[ignore]` tests failed the moment a real MinIO-written fixture was fed
in, even though the decoded data was byte-identical.
The client-visible object size comes from `decrypted_size()` /
`get_actual_size()`, which correctly reads MinIO's
`x-*-internal-actual-size` metadata (verified: both SSE-S3 and SSE-KMS
8 MiB multipart fixtures now report 8388608). Assert against that
instead and keep the plaintext length and SHA-256 data checks.
With real 4-drive MinIO fixtures (RELEASE.2025-09-07) all four tests
pass, confirming RustFS reads MinIO erasure-coded SSE objects with
byte-identical data and correct logical size.
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): nightly MinIO interop check + dockerized fixture capture
Wire the ignored MinIO on-disk interop reader tests into a nightly,
non-required CI job, and make their fixtures reproducible without a host
MinIO install.
- Dockerfile + capture_via_docker.sh: build a throwaway image carrying
the official MinIO server binary (pinned RELEASE.2025-09-07) plus the
fixture lab on a small Python base, then run `lab.py capture-matrix` to
write the SSE-S3 / SSE-KMS multipart fixtures the tests consume. lab.py
drives MinIO's S3 API directly, so no `mc` is needed.
- .github/workflows/minio-interop.yml: nightly + manual workflow on
GitHub-hosted ubuntu-latest (reliable Docker + Python, unlike the
self-hosted fleet — see e2e-s3tests.yml infra note). Regenerates the
gitignored fixtures each run and executes the #[ignore] reader tests.
Not a PR gate.
- README: document the Docker capture path.
Validated end to end: the script builds the image, captures the two
multipart cases, and `cargo nextest run --run-ignored ignored-only`
passes all four interop tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(replication): lock outbound checksum consistency for new algorithms (T3)
Adds a consistency test at the replication put-options boundary confirming that
the AWS 2026-04 additional checksum algorithms (XXHash3/64/128, SHA-512, MD5) are
forwarded into replication user_metadata identically to the classic five. The
outbound replication path routes a stored object checksum through the
algorithm-agnostic decrypt_checksums -> user_metadata flow, so the new algorithms
(already covered by rustfs-rio read_checksums) need no new-algorithm-specific
handling. This locks that behavior against regressions.
Investigation summary (no code change needed on the outbound side): the
per-algorithm ChecksumMode selection path is dormant (opts.checksum is never set
to a specific algorithm; tiering uses Content-MD5; the add_crc bool is dead
code), so extending ChecksumMode was unnecessary.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)
Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).
The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.
Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)
Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.
Refs rustfs/backlog#1255rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)
Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.
Refs rustfs/backlog#1260rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)
HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.
Refs rustfs/backlog#1257rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(checksums): fail-closed on unknown checksum algorithm (S7)
A. Harden unknown/unsupported checksum algorithms to fail closed instead of
panicking. ChecksumMode::base() in the outbound S3 client
(crates/ecstore/src/client/checksum.rs) previously did
`panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
unit tests proving base() never panics and hasher() returns Err for
unsupported modes. rustfs-checksums FromStr already returns Err on unknown
names; added a regression test asserting garbage/unknown names fail closed.
B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
into_impl, name constants, the x-amz-checksum-* header constants and the
HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
digest against a direct library computation. MD5 stays intentionally
rejected (PR #4513) and is left untouched.
C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
with 7 variants already consuming 7 bits; adding the 4 new algorithms would
overflow u8 and require a breaking repr change, so ChecksumMode is left
unchanged. The new algorithms are available through the rustfs-checksums
ChecksumAlgorithm path.
Refs rustfs/backlog#1259rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)
Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:
- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
from build_get_object_checksums through GetObjectOutputContext into
finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
captured at the want_checksum set points before opts is moved.
Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.
The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.
Refs rustfs/backlog#1257rustfs/backlog#1256rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)
Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):
- complete_part_checksum previously returned the outer None for any algorithm
outside the five typed ones, which failed CompleteMultipartUpload with
InvalidPart. It now accepts any valid base type with no double-check value
(Some(None)) — mirroring the missing-value path of the typed algorithms —
since s3s CompletePart has no field to carry a client-supplied per-part
value and the part was already verified server-side at UploadPart. Genuinely
unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
concatenated per-part raw digests; full_object_requested() is false so
add_part() is correctly bypassed) already works for these algorithms via the
S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
their responses via the shared inject_additional_checksum_headers helper
(now pub(crate)), since s3s has no typed output field.
Refs rustfs/backlog#1261rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)
Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.
Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.
Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).
Refs rustfs/backlog#1258rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)
from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.
The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.
Refs rustfs/backlog#1263rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(checksums): collapse 5 duplicated response-checksum loops into one
Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.
Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
performs the typed/extra split once. All five call sites now destructure its
result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
hand-rolled five-way comparison.
Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(checksums): unit tests for classifier/echo helpers + fix unused import
Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.
Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)
The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.
Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)
Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.
Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).
Refs rustfs/backlog#1262rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(get): allow too_many_arguments on finalize_get_object_response
The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <housemecn@gmail.com>
The pulsar config validation rejected any non-`pulsar+ssl` broker whenever
`tls_allow_insecure` was set or `tls_hostname_verification` was disabled.
Both toggles are inert on a plaintext `pulsar://` broker — the Pulsar client
only applies them for `pulsar+ssl` — so treating a non-default value as fatal
is wrong.
This surfaced as issue #4796: the console persists `tls_hostname_verification`
as `false` (the checkbox defaults to unchecked while the server default is
`on`). At creation time the value is not present in the unmerged request and
falls back to the safe default, so the connectivity check passes and the
target comes online. On restart the persisted config is merged and validated,
the `false` is read back, and the target is rejected — permanently offline
even though Pulsar is reachable.
Relax both the loader-side (`validate_pulsar_broker_config`) and runtime-side
(`PulsarArgs::validate`) checks so only a `tls_ca` bundle — real TLS trust
material that is never defaulted to a non-empty value — requires a
`pulsar+ssl` scheme. The inert toggles no longer fail the target. This also
heals configs already persisted with the bad value. Add regression coverage
for both paths.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF
hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response
bytes are still buffered (a prior poll_flush() returned Poll::Pending and the
result was discarded), so a backpressured/slow-reading peer receives a graceful
FIN before the full Content-Length body is flushed. Standard S3 clients
(minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET
under load (rustfs/backlog#1232). This is the transport-layer bug from
Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF
already fixed via lockstep decode; here the app layer delivers the full body and
truncation is purely hyper->socket.
Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush
buffered data before shutdown"), which is not in any crates.io release yet as of
hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant
of the fix that still declares version 1.10.1).
Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the
server connection is driven by the transitive hyper-util (conn::auto /
GracefulShutdown), and a git source would not unify with the crates.io hyper
hyper-util resolves, leaving two hyper copies with the server path still on the
buggy one. The patch rewrites the crates.io source globally, so the lockfile
holds a single git-sourced hyper.
Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard
(mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped
or hyper is downgraded below the fix. Its load-bearing assertion is a
synchronously-set flag, so a slow runner can only under-detect, never false-red.
Closesrustfs/backlog#1232.
* build(deny): allow the hyperium/hyper git source for the flush-before-shutdown pin
cargo-deny's [sources] check denies unknown git sources. The hyper
[patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium
git source, so add it to allow-git with an owner/review note and a removal
condition (drop once a released hyper > 1.10.1 carries commit 72046cc7).
* fix(obs): open cleaner compression source with O_NOFOLLOW
The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.
Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): recompress instead of trusting leftover cleaner archives
archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.
Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.
Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): cap memory-gate reservation at cache growth headroom
The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.
Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.
The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.
Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): assert native O_DIRECT path runs in uring read test
uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.
Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.
The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.
Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads
classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.
Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.
Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document data-blocks-first default and its tail-latency cost
DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.
The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.
No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.
Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document wide-directory walk stall hazard and tuning
list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).
This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.
Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document consumer-peek vs producer-stall coupling
In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.
Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.
Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles
Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.
Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey
The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.
Add two genuinely-failable scenarios reusing the existing harness and
fixtures:
- Parity reconstruction A/B: take one data disk offline and re-run the
full object matrix on both phases while the EC 2+2 set rebuilds each
large object from the surviving shards. Assert codec == legacy
byte-for-byte (sha256) and header-for-header, and assert the codec
phase served the reconstructed objects with zero duplex-pipe fallback
(the reader gate is drive-health-independent, so the codec fast path
is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
missing-key GET across the legacy and codec phases and require them to
be identical (404/NoSuchKey), guarding against the codec env
perturbing the error path.
Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.
Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).
Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback
The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.
Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.
Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.
Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.
Refs #3231, #3191, #4356.
Co-authored-by: heihutu <heihutu@gmail.com>
Follow-up to #4772. After forcing every-cycle ILM evaluation the Days=1
plateau is reliable, but test_lifecycle_expiration / test_lifecyclev2_expiration
still flaked intermittently on the *second* assertion (`assert 4 == 2`):
the Days=5 (expire3) objects were not expired within their poll window.
Cause: _wait_for_lifecycle_count for expire3 starts its N*lc_interval
deadline only after the Days=1 poll returns (~1 debug-day in). With N=5
and lc_interval == debug_day, the 5*debug_day terms cancel and the slack
past the Days=5 due time is only ~1 debug-day (~9s) -- which a single slow
scanner cycle (observed ~13s spacing under CI load) can exceed, leaving the
count stalled at 4. Bumping the two expire3 windows to 8*lc_interval raises
the slack to ~39s, comfortably above the observed cycle jitter.
Test-only, lane-scoped: does not touch RUSTFS_ILM_DEBUG_DAY_SECS or the
4*lc_interval < 5*debug_day plateau invariant. The expire3 target count (2)
is terminal (nothing expires after it), so a wider window can never
over-expire. Also documents the #4772 RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1
knob in lifecycle_behavior_tests.txt.
chore(ci): guard against committed planning docs; remove re-added ones
PR #4771 removed the docs/superpowers planning-doc archive and added a rule,
but #4765 force-added two more (git add -f bypasses .gitignore):
docs/superpowers/plans/2026-07-12-observability-single-writer.md and
docs/superpowers/specs/2026-07-12-observability-single-writer-design.md — an
agentic implementation plan and its design spec. Delete both.
Close the enforcement gap so this cannot recur:
- New scripts/check_no_planning_docs.sh fails if anything is tracked under
docs/superpowers/, regardless of how it was added.
- Wire it into make pre-commit/pre-pr/dev-check.
- Run it in CI: ci.yml for code/mixed PRs, and ci-docs-only.yml (which was a
green stub) so docs-only PRs can no longer slip a planning doc past the
required 'Test and Lint' check.
- Document the guard in AGENTS.md and the arch-checks skill.
* fix(obs/cleaner): fsync archive and log dir before deleting source logs
OLC-01: the compression path flushed the BufWriter but never synced the
archive data or the parent directory before renaming, and the source was
then unlinked with no durability barrier. A crash after rename but before
the page cache reached disk could leave a truncated/zero-length archive
while the source was already gone — permanent log/audit data loss. Because
the archive is always renamed to a brand-new name (guaranteed by the
existing exists() guard), ext4 auto_da_alloc does not mask this.
Hand the underlying File back from the writer closure, sync_all() it before
rename, fsync the parent directory, and fsync the log directory after the
unlinks so a delete cannot be reordered ahead of the archive it justified.
Guard the temp file with an RAII cleanup so an early return or panic cannot
leak a *.tmp orphan.
Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW
OLC-03: the temp archive was opened with File::create at a predictable
`<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write
access to the log directory could pre-plant that path as a symlink and have
the compressor follow it — truncating and overwriting an arbitrary external
file, then chmod-ing it to the source log's mode. This mirrors the symlink
refusal already enforced on the deletion path (secure_delete).
Route temp creation through create_tmp_archive(), which uses create_new
(O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to
refuse a symlink at the final path component.
Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with restrictive mode
OLC-06: File::create left the temp archive world-readable (0644 & ~umask)
for the entire duration of compressing a large log, exposing the full
plaintext of a possibly-0600 audit log on shared hosts until the mode was
copied only after the write completed. Pass the source mode into
create_tmp_archive and open the temp file with it (default 0600) so it is
restrictive from creation; the post-write chmod still tightens/matches the
source mode exactly.
Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): validate existing archive before skipping recompression
OLC-02: the idempotency guard used Path::exists() (which follows symlinks)
and trusted whatever it found, then the caller deleted the source. A planted
`<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed
run (OLC-01), would green-light deleting the source with no valid backup —
data loss / log destruction.
Replace exists() with symlink_metadata (no follow) and only treat the entry
as a completed prior result when it is a regular, non-empty file whose header
matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls
through to recompression, whose atomic create_new+rename replaces the bad
entry (a symlink is replaced, never followed or deleted through).
Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression
OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so
projected_freed_bytes = input and delete_files reported the full input as
freed. A real run keeps the archive on disk (freed = input - archive), so
dry-run overstated reclaim by the whole archive footprint. Estimate the
archive with a deliberately conservative ratio so the projection never
exceeds what a real run reclaims.
Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): make freed-byte accounting resilient; document steal metric
OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which
silently reports 0 on failure and skews freed-byte metrics (input - 0 = full
input, overstating reclaim). Use the copy() byte count as the authoritative
input size and, when the archive metadata read fails, conservatively assume
no savings instead of 0. Also document that the steal_success_rate counts
only victim steals (batch = one success), so it reads as a relative
rebalancing signal, not absolute task acquisition.
Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels
OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21],
silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1
and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those
meanings survive, and echo the effective (post-clamp) levels in the startup
log via new effective_gzip_level()/effective_zstd_level() getters so the log
matches what actually runs.
Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0
OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so
retention=0 disabled it entirely while compression kept producing archives,
and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk
growth. Replace select_expired_compressed with select_archives_to_delete,
which applies age expiry (when retention is on) and, regardless of retention,
trims the oldest archives until the set fits under max_total_size_bytes. Also
warn at startup when compression is on with retention=0 so the "keep forever"
semantics are not mistaken for "delete immediately".
Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently
OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so
a typo (or a literal comma splitting a char-class in the config string) turned
"protect this file" into "delete this file" with no signal. Log a warning per
rejected pattern with the raw string and parse error so the misconfiguration
is visible.
Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor
OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used
yield_now on the empty path, burning CPU during redistribution windows;
worker_count had no upper bound so a mis-set parallel_workers over a directory
of thousands of logs could spawn thousands of threads; and
default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use
crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp
worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1.
Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn when active-file guard is disabled by empty filename
OLC-13 (defense-in-depth): the scanner protects the live log purely by exact
filename equality against active_filename. An empty active_filename silently
disables that protection, so a non-empty file_pattern could make the live log
a deletion candidate via the public builder. Warn in build() when that unsafe
combination is configured. The audit's "never delete the newest match"
structural guard is intentionally not implemented: it would conflict with the
legitimate keep_files=0 semantics (purge all rotated logs). The naming
contract is instead locked by regression tests (OLC-14).
Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode
OLC-07: from_config_str silently fell back to defaults for unrecognized
compression algorithm and match mode (any non-"prefix" value became Suffix),
hiding operator typos like "prefixx" that could make the cleaner match no
rotated logs. Warn on a non-empty unrecognized value in both parsers, and
echo the resolved match_mode in the startup log alongside the algorithm.
Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age
OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds
(default 3600), so crash-left orphans lingered up to an hour, and the tmp
suffix list was hardcoded rather than derived from compressed_suffixes() — a
new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes.
Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and
gate orphan removal on a small fixed grace window instead of min_file_age
(orphans are never live-written after the rename that would promote them).
Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases
OLC-14: add regression tests for the previously-untested safety/correctness
branches — symlink rejection (external target never deleted), archive age
expiry vs fresh retention, archive byte-cap trim with retention disabled,
gz/zst classification, max_single_file_size selection, min_age protecting a
fresh non-empty log, active-file exclusion when the active name also matches
the pattern, invalid exclude glob not aborting build, dry-run + compression
creating no archive, gzip round-trip validity, and the idempotent-archive
branch trusting a valid prior archive.
Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(obs/cleaner): apply rustfmt and collapse nested if (clippy)
Formatting-only cleanup over the audit fix series: rustfmt normalization of the
multi-line expressions introduced in compress.rs/core.rs, plus collapsing the
delete_files directory-fsync into a single let-chain to satisfy
clippy::collapsible_if. No behavior change.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(obs/cleaner): collapse redundant source stats in compression path
The per-issue fixes to compress_with_writer accumulated three metadata()
syscalls on the source in the real compression path: an input_bytes read that
was immediately shadowed by the copied byte count (a dead read), a source_mode
read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode.
Collapse to a single fd-based read — move the dry-run input_bytes read into
the dry-run branch, read source_mode from the already-open fd (no path stat,
no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged
(same inode's mode, written for input size); 3 source stats -> 1.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured)
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): prevent rolling log stdout aliasing
* fix(ecstore): bound hot-path tracing payloads
* test(logging): guard service and disk log invariants
* fix(obs): silence useless_conversion on st_dev for Linux clippy
rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op
that trips clippy::useless_conversion under -D warnings. The conversion is
still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the
lint on that line rather than dropping the portable fallible conversion.
* fix(logging): keep stdout sink validation portable (#4769)
* fix(obs): keep stdout device conversion portable
* docs(logging): update single-writer plan status
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
The s3-tests lifecycle expiration cases (test_lifecycle_expiration,
test_lifecyclev2_expiration, test_lifecycle_deletemarker_expiration)
flaked with counts stalling one plateau behind (e.g. `assert 6 == 4`).
Root cause: a compacted directory is only re-descended -- and its
objects re-evaluated against ILM rules -- once every
DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the lane's
accelerated RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so
a Days=1 object due at debug_day (10s) is not actually expired until the
next ~32s boundary (~42s) -- just past the test's 4*lc_interval (40s)
poll window, so the list still returns the pre-expiry count.
Set RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 for this debug-only lane so
compacted directories are re-descended every cycle and ILM fires within
~2s of the due time, comfortably inside the poll window. This does not
touch the 4*lc_interval < 5*debug_day plateau invariant.
docs: remove agent-generated planning docs, forbid committing them
Delete one-shot planning/progress artifacts that were checked into the tree:
the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus
issue-scoped implementation plans, optimization conclusions, and dated
benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans,
get-small-file conclusion, issue824/issue829 benchmark results, issue-713
>1GiB GET baseline summary and ops guide).
Codify the rule so they do not come back:
- .gitignore drops the docs/superpowers whitelist, so anything new under
docs/ stays ignored unless force-added.
- AGENTS.md gains an explicit 'do not commit planning-type documents' rule
scoping version control to the durable architecture/operations/testing sets.
- docs/architecture/README.md, overview.md, arch-checks SKILL.md, and
check_doc_paths.sh drop their references to the removed archive.
The scheduled e2e-s3tests sweep failed at "Wait for RustFS ready" in both
topologies because the server never started (issue #4762).
Two independent startup faults, both surfaced now that the local
physical-disk-independence guard is enforced:
- single: RUSTFS_VOLUMES=/data/rustfs{0...3} all live on one physical
device on the runner, so the guard aborts startup. Set the
CI-sanctioned RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true (what the guard's own
error message and the e2e tests already use).
- multi: the entrypoint's process_data_volumes skipped every non-absolute
entry, so the distributed URL form
"http://rustfs{1...4}:9000/data/rustfs{0...3}" never created
/data/rustfs0..3. LocalDisk init then aborts with VolumeNotFound because
resolve_local_disk_root no longer auto-creates the disk root. This also
broke the shipped .docker/compose/docker-compose.cluster.yaml for real
distributed docker deployments.
entrypoint.sh now:
1. Expands multiple {N...M} ranges per token (the URL form carries two).
The previous single-pass expander collapsed it to "http://rustfs1"
and dropped the disk path; it now re-scans until no ranges remain,
operating on only the first brace to keep multi-range tokens intact.
2. Creates the local filesystem path for URL-form endpoints (stripping
scheme://host:port) without appending them as CLI args — rustfs reads
the distributed form from RUSTFS_VOLUMES directly.
Absolute-path and default (/data) inputs expand byte-identically to before.
The multi compose also gets the CI disk-check bypass, since the four disks
share one device inside each node's container.
fix(cache): drain pending removals until entry_count reaches zero in clear()
The previous clear() implementation used a single run_pending_tasks()
call after invalidate_all(), which was insufficient under concurrent
fill pressure — moka processes invalidations lazily in batches, so
entries can linger after a single maintenance pass.
Replace the fixed single call with a drain loop (up to 256 rounds) that
calls run_pending_tasks() and yields between iterations until
entry_count() reaches zero. This ensures the concurrency storm test
(moka_backend_concurrency_storm_leaves_no_leaked_state) passes reliably.
The earlier fix (#4759) added a pre-invalidate_all drain and an 8-pass
loop but reordered operations in a way that introduced a new race. This
commit keeps the original invalidate_all-first ordering and only adds
the drain loop after the initial run_pending_tasks() call.
The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com>
Six doc-comments on `OtelConfig` fields stated defaults that disagreed
with the constants the runtime actually applies in
`extract_otel_config_from_env`. Correct them to match:
- profiling_export_enabled: false -> true (DEFAULT_OBS_PROFILING_EXPORT_ENABLED)
- sample_ratio: 0.1 -> 1.0 (SAMPLE_RATIO)
- meter_interval: 15 -> 30 (METER_INTERVAL)
- logger_level: info -> error (DEFAULT_LOG_LEVEL)
- log_rotation_time: daily -> hourly (DEFAULT_LOG_ROTATION_TIME), and
document the full minutely/hourly/daily set plus the daily fallback
- log_cleanup_interval_seconds: 21600 -> 1800 (DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS)
Doc-comment only; no behavior change.
fix(test): use canonicalized disk root for file_sync_probe in rename_data test
On macOS, tempfile::tempdir returns /var/folders/... while LocalDisk
resolves the root to /private/var/folders/... via dunce::canonicalize.
The file_sync_probe::enter() path check uses starts_with(), so passing
the non-canonical tempdir path caused the probe to never activate,
making wait_for_active() hang indefinitely.
Use disk.root (already canonicalized) for the probe instead.
`report_metrics` runs on every metrics collection cycle and did three things it
did not need to, for every metric, every cycle:
- interned `metric.name`/`metric.help` through a `Mutex<HashMap>` even when the
`Cow` was already `Borrowed(&'static str)` (the common case for statically
named metrics);
- re-ran `describe_*!` (which re-locks the recorder's metadata map) although the
metadata never changes;
- allocated a fresh `Vec<(String, String)>`, cloning every label key and value,
even though `metric.labels` is already `[(&'static str, Cow<'static, str>)]`.
Now: names/help resolve to `&'static str` without touching the intern cache when
already borrowed; each metric is described once (tracked in a `HashSet`); and the
recorder is fed `&metric.labels` directly, removing the per-cycle label clone.
No metric names, help, label keys/values, or emitted values change. Verified by
building rustfs-obs and running the report unit tests (the `metrics` macro
accepts the borrowed label slice directly).
Addresses rustfs/backlog#1185 (P3, report path).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1183, staged rollout step. Simplify enabling the zero-duplex GET
codec-streaming fast path to a single switch — RUSTFS_GET_CODEC_STREAMING_ROLLOUT
(default "off") — now that body/header parity is proven (parity e2e net + bench
A/B on backlog#1183).
- Add a clean production rollout token "on" (aliases "full"/"production"); the
legacy "internal"/"benchmark" tokens remain accepted.
- Flip DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE and the two ..._COMPAT_CONFIRMED
defaults to true. They are retained as emergency kill-switches (set any to
false to force the fast path off) but no longer gate enablement — the rollout
switch does.
No production behavior change: with no env set the rollout switch defaults to
"off", so GET stays on the legacy duplex path exactly as before. Flipping the
hard default to on is deferred to a follow-up after a production soak.
Note (intentional semantics change): with the rollout switch opted in
("on"/"internal"/"benchmark"), codec streaming now activates without also
setting the two ..._COMPAT_CONFIRMED vars — compatibility is confirmed, so those
confirmations are baked in.
Co-authored-by: heihutu <heihutu@gmail.com>
Bound large shard-directory syncs per disk and process while preserving small-directory and durability behavior.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The `metrics` macros re-run the recorder's `register_*` on every emission — a
`RwLock` read, a name-key hash, and an `Arc` clone — even for a metric that
never varies its key. For the hot, label-less recorders on the per-IO path
(`record_data_transfer`, `record_io_latency`, `record_io_latency_p95/p99`,
`record_io_queue_congestion`) that lookup is pure overhead once observability is
on.
Add `counter_increment_cached!` / `gauge_set_cached!` / `histogram_record_cached!`
that resolve the handle once via `LazyLock` in production and reuse it. Under
`cfg(test)` they re-resolve on every call, because the `metrics` crate resolves
against a thread-local recorder that `with_local_recorder` swaps per test — a
process-global cached handle would bind to whichever recorder was active first
and break test capture. The macros only wrap FIXED (label-less) keys, and the
`metrics_enabled()` gate still short-circuits before any emission when disabled.
Verified: the only callers of these functions are the collector (io-metrics'
own cfg(test) tests, which re-resolve) and production code; no cross-crate test
captures them. rustfs-io-metrics builds on both cfg paths, 147 unit + 4 doctests
pass, clippy clean.
Addresses rustfs/backlog#1185 (P3, per-emission handle caching).
Co-authored-by: heihutu <heihutu@gmail.com>
`record_uring_fallback` is called at multiple sites in the io_uring read
path whenever a read falls back to `StdBackend`. It formatted
`self.root.display().to_string()` on every call, heap-allocating a
`String` from a `Path` that never changes after construction — pure
per-read waste when io_uring is degraded.
Cache the label once in `UringBackend::try_new` as a `String` field
(`root_label`) and clone it per emission. The metric name and the
`"root"` label value are unchanged; only the redundant `Path` formatting
is removed. The clone is a single alloc of an already-short string.
Refs: rustfs/backlog#1185
Co-authored-by: heihutu <heihutu@gmail.com>
`cluster_peer_should_bypass` is called before every internode RPC when the
offline-bypass feature is enabled (remote disk `get_client`, remote locker), and
it took a single process-global `Mutex<HashMap>` even for the overwhelmingly
common case of an online or unknown peer, which is read-only. That serialized
all concurrent internode client acquisitions on one lock.
Switch `CLUSTER_PEER_HEALTH` to a `RwLock`:
- the hot check takes a shared read lock and returns immediately for unknown or
online peers, so concurrent RPCs no longer serialize;
- only an offline peer (rare) drops to the write lock to record a re-probe, with
a re-fetch/re-check because the state can flip back online between releasing the
read lock and taking the write lock;
- the write paths (dial reachable/unreachable) and the read-only
`cluster_peer_is_offline` move to `write()`/`read()` accordingly.
Behavior is unchanged on every path (online/unknown -> not bypassed; offline ->
bypass with one re-probe per interval); `Instant::now()` now runs only on the
offline path. Poison recovery is preserved. All internode unit tests pass.
Addresses rustfs/backlog#1185 (P2).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1183 tracks flipping the default GET data path from the legacy
tokio::io::duplex double-copy to the zero-duplex codec-streaming fast path.
That flip is gated behind RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED /
..._HEADER_COMPAT_CONFIRMED, which need empirical evidence the two paths are
byte- and header-identical.
Add an e2e regression net that runs the same object matrix twice against the
same on-disk EC shards, changing only the codec-streaming env gates: phase A
(default) takes the legacy duplex path, phase B (gates opened) takes codec
streaming. It asserts byte-for-byte (sha256) and header-for-header equality
across inline / small / multi-block (1.5M/3M/5M+) / multipart objects, plus a
ranged GET (which falls back to legacy by gate design). Path confirmation is
not assumed: the legacy path logs "Created duplex pipe ..." per full GET, so
the test counts that marker per phase and asserts the codec phase created zero
duplex pipes, proving the fast path actually ran rather than silently falling
back to the path it is compared against.
To capture the child server's logs for that assertion, add an optional
RustFSTestEnvironment.capture_log_path (default None = inherit stdio,
backward compatible) that redirects the spawned server's stdout+stderr to a file.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6)
The tier/lifecycle integration tests carried two byte-for-byte copies of an
in-memory WarmBackend mock — one in crates/scanner/tests and one in
rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both
implemented the same ecstore WarmBackend trait.
Consolidate them into ecstore behind a new `test-util` feature, exposed via the
`rustfs_ecstore::api::tier::test_util` facade:
- MockWarmBackend: in-memory WarmBackend with an operation log (for ordering
assertions such as "local delete precedes remote remove") and fault injection
(FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency,
plus external_remove to simulate an out-of-band remote deletion.
- register_mock_tier / register_mock_tier_backend: register the mock into any
TierConfigMgr handle (the global manager used by scanner tests or a
per-instance one used by the app tests).
- xl.meta transition assertion helpers: read_transition_meta,
assert_transition_meta_consistent (cross-shard consistency of the
status/tier/remote-key/remote-version-id tuple plus free-version count), and
free_version_count.
- polling helpers: wait_for_remote_absence, wait_for_object_count,
wait_for_free_version_absence.
Both existing copies now consume this single definition; `rg 'struct
MockWarmBackend'` collapses to one. The feature is enabled only from
[dev-dependencies], so it never links into the production binary (resolver 3).
Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault
injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the
scanner mock — that op-logging is now part of this shared surface, so #4706
should rebase onto it.
Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155.
* test(ecstore): fix shared MockWarmBackend usage after main merge
- Access stored objects via MockWarmBackend::contains() instead of the now
private inner objects map (fixes E0609 after the shared test-util refactor).
- Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused
transition_api test re-exports the mock extraction left behind.
- Reword the scanner/rustfs test-util dependency comments so they no longer
embed the literal rustfs_ecstore:: path that trips the ECStore
architecture-migration guard.
`MetricsCollector::record_io_operation` recomputed P50/P95/P99 on every disk IO
by taking a read lock, collecting up to `max_latency_samples` (1000) into a
`Vec<u128>` and fully sorting it — an O(n log n) alloc+sort per operation on the
GET read path (gated by stage metrics). Both consumers sample only periodically:
the autotuner reads `avg_io_latency_us` on its tuning tick, and the P95/P99
values are never read internally — they only feed OTEL export. Nothing needs
per-IO freshness.
Split the two costs:
- The window mean (stored in `avg_io_latency_us`, read by the autotuner) is now
maintained in O(1) via a running sum kept in step with the window's push/pop,
and refreshed on every op — no sort, no warm-up regression.
- The P95/P99 sort is throttled to once per `PERCENTILE_RECOMPUTE_INTERVAL`
(128) operations. A bounded lag is invisible to the periodic autotuner tick and
OTEL export while the per-op sort cost is amortised ~128x away.
Semantics are otherwise unchanged: same sliding window, same percentile indices,
same mean value. Tests updated — the percentile test forces a recompute to
exercise the math directly, and a new test asserts the mean tracks every op while
the percentiles only recompute at the interval boundary.
Addresses rustfs/backlog#1185 (P1).
Co-authored-by: heihutu <heihutu@gmail.com>
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
* perf(metrics): drop needless per-emission work on hot metric paths
Audit of the metrics hot paths surfaced four low-risk wins where emission did
work it did not need to:
- `record_file_cache_reclaim_success/error` (disk/local.rs) called `.to_string()`
on `kind` (already `&'static str`) and on the `"ok"`/`"err"` literals, heap-
allocating up to four `String`s per page-cache reclaim window — which runs per
read-stream reclaim. The `metrics` macros accept `&'static str` label values
directly, so pass them as-is.
- `record_read_repair_dedup` (set_disk/core/io_primitives.rs) likewise
`.to_string()`-ed an already-`&'static str` `reason`.
- `SetDisks::get_object_reader` (set_disk/ops/object.rs) captured `Instant::now()`
and emitted the `rustfs.lock.acquire.*` counter and histogram unconditionally
on every GET, right beside an already-gated stage timer. Gate them behind
`get_stage_metrics_enabled()` too, so an inactive observability config pays no
per-GET clock read or recorder lookups.
- The per-response-body-chunk counter in server/http.rs re-ran the `counter!`
registry lookup on every chunk (a streamed GET emits many). Resolve the
label-less handle once into a `LazyLock<metrics::Counter>`; the global recorder
is installed at startup before any response streams, so the cached handle binds
to the final recorder.
No metric names or label values change. The only behavior change is that the
`rustfs.lock.acquire.*` GET-path metrics now follow the GET stage-metrics flag,
consistent with the neighbouring stage timings.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): gate page-cache reclaim metrics behind metrics_enabled()
`record_file_cache_reclaim_success/error` run per read-stream reclaim window on
large-object reads and emitted unconditionally. When general metrics are
disabled the `counter!`/`histogram!` macros still construct three metric keys
per call for nothing. Skip the emission behind `rustfs_io_metrics::metrics_enabled()`,
matching how the io-metrics free functions self-gate. The serial reclaim-metrics
test now enables the flag (save/restore) alongside the existing stage gate.
Left ungated deliberately: `record_read_repair_dedup` (rare read-repair path,
and its non-serial test would need a global-flag toggle), and the HTTP body-chunk
counter (its cached handle already makes the disabled case a no-op increment).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`full_object_plaintext_len` decides whether a body-cache hook hit may serve
bytes in place of the erasure read. It is a fail-closed allow-list: it excludes
every read whose `ReadPlan::build` applies some other transform (ranged/part,
raw/data-movement, restore, encrypted, remote) with an early `return None`,
then returns a `Some(..)` length only for the whole-plaintext cases. A newly
added `ReadPlan` branch that nobody teaches this gate about falls through to
`None` and safely bypasses the cache. Flip it to a deny-list and the same new
branch silently serves bytes in the wrong representation — the backlog#1108 /
#1109 / #1146 class of bug.
The existing unit and e2e tests only cover the branches that exist today. This
adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into
pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate
and a `return None` still precede the first `Some(..)`. Reordering a predicate,
dropping one, moving the positive return ahead of the gate, or renaming/removing
the function all fail; wording, formatting, and adding a new exclusion in the
same gate do not. Mutation-tested against all four regression shapes.
This machine-enforces the structural invariant that backlog#1146 was kept open
to guard by hand.
Co-authored-by: heihutu <heihutu@gmail.com>
Close the three test-completeness items in rustfs/backlog#1180 that the earlier
hardening (rustfs/rustfs#4726, #4729) left unpinned; the other two of the five
(sharded cancel routing, bailout-handle error) already landed in rustfs/uring.
- rename_data end-to-end invalidation: drive the real `LocalDisk::rename_data`
commit (non-inline part, so `invalidate_part_paths` is non-empty) and assert
the destination part descriptor is dropped — not merely `rename_file`/`delete`.
Both production `rename_data` call sites share this invalidation, so a
"fix one copy, miss the other" regression is now caught. The test is
non-vacuous: it seeds the cache, removes the on-disk data dir out of band (the
cached fd keeps the old inode alive and clears the path for the directory
rename `rename_data` performs), and asserts a read still returns the OLD bytes
before the commit — which fails outright if the cache is off, so it cannot pass
without a live cache.
- FD_CACHE_TTL backstop: an injected short TTL proves the cache self-evicts a
descriptor with no explicit invalidation; a static check pins the 5s value.
- zero-length read bounds parity on the cache-HIT path: a `length == 0` read
past EOF must be rejected identically to the miss path and StdBackend, pinning
the #1173 fix against regression.
Refactors `FdCache::new` to delegate to a private `with_ttl(ttl)` helper so the
TTL backstop can be exercised with a short TTL instead of a multi-second wait.
Verified: `cargo test -p rustfs-ecstore` on Linux with real io_uring
(seccomp=unconfined, RLIMIT_NOFILE raised, RUSTFS_URING_TESTS_MUST_RUN=1 so a
degraded skip fails rather than passing vacuously).
Co-authored-by: heihutu <heihutu@gmail.com>
The io-metrics crate already had per-stage GET/PUT switches, but the request
headers/summaries and ~40 general recorders (I/O scheduler, bytes-pool,
zero-copy, bandwidth, system-resource, error/timeout/retry) emitted
unconditionally, paying their label allocations and arithmetic even when no
metric recorder is installed.
Add `METRICS_ENABLED: AtomicBool` (default false) with `set_metrics_enabled()`
/ `metrics_enabled()`, isomorphic to the existing stage switches, and gate:
- GET headers/summaries via `get_stage_metrics_enabled()`:
record_get_object_request_start / _request_started / _request_result /
_timeout / _completion / _total_duration_with_path, plus legacy
record_get_object.
- PUT headers via `put_stage_metrics_enabled()`:
record_put_object_request_start / _request_result, plus legacy
record_put_object.
- 41 general recorders via `metrics_enabled()`.
Startup enables all three switches together under
`observability_metric_enabled()` (startup_observability.rs), with a passthrough
in startup_runtime_sources.rs. The global recorder is itself only installed
when observability metric export is on, so gating off is pure cost savings when
export is disabled and a no-op when it is enabled.
Deliberately NOT gated (would break function, not just observation): the EC
encode in-flight and GET buffered-bytes accounting guards
(add/remove_ec_encode_inflight_bytes, track_get_object_buffered_bytes) whose
values are read back by current_*; and the stateful sibling modules
(lock/deadlock/backpressure/autotuner/adaptive_ttl/collector/internode). The
console realtime metrics endpoint (/admin/v3/metrics) samples system state and
internode metrics directly, so it does not depend on the gated recorders.
Adds a toggle test exercising both the enabled and disabled paths, serialized
on the existing METRICS_FLAG_LOCK to stay robust under nextest's shared-process
model.
Co-authored-by: heihutu <heihutu@gmail.com>
Two behavior-preserving cleanups to the io_uring local read backend that
accumulated as later work layered onto it:
- Reunite the `UringBackend` struct doc comment. The backlog#1145 fd-cache
constants were inserted into the middle of the struct's doc block, so its
opening sentences were orphaned onto `ENV_RUSTFS_IO_URING_FD_CACHE` and the
struct itself was documented by a sentence fragment starting mid-clause with
"through rustfs-uring's...". Move the opening back onto the struct and give
the constant its own one-line doc.
- Fold the fd-cache handle and its lookup key into a single
`Option<(&FdCache, FdKey)>` in `pread_uring`, so the get and the
insert_if_fresh sites stop re-deriving `self.fd_cache.as_ref()` and
re-matching presence. Semantics are identical, including the backlog#1176
generation guard (`gen_at_open` snapshot before open, insert only when the
generation is unchanged).
No functional change. The borrow/move pattern was validated against a
host-compilable reduction (the cfg(linux) path cannot be built from macOS);
CI compiles the Linux backend.
Co-authored-by: heihutu <heihutu@gmail.com>
chore(deps): bump rustfs-uring to 0.2.1 from crates.io
rustfs-uring 0.2.1 routes the driver's runtime diagnostics through `tracing`
with structured fields instead of `eprintln!` (rustfs/uring#13). No public API
change — UringDriver::probe_and_start_sharded, read_at, and read_at_direct keep
their signatures — and the read path / cancel-safety ownership model are
untouched, so this is a drop-in patch bump of the version requirement plus the
lockfile entry.
0.2.1 adds a `tracing` dependency; it is recorded in the rustfs-uring lock
entry. tracing is already in the workspace graph, so nothing new is pulled in.
The Cargo.lock change is scoped to the rustfs-uring package only; unrelated
lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176)
pread_uring's miss path opened a descriptor on the blocking pool and only then
inserted it into the moka cache. moka's invalidations cover only entries present
at call time, so a heal/delete commit that invalidated between the open and the
insert could not stop the just-opened stale inode from being cached afterwards —
serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal.
Add an invalidation generation to FdCache, bumped by invalidate_exact and
invalidate_under before they touch moka. The read path snapshots the generation
before opening and inserts via insert_if_fresh, which refuses the insert if the
generation moved during the open and, with a post-insert re-check, removes the
entry if an invalidation raced the insert itself. Reads that never miss are
unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175)
The fd-cache invalidation contract was only wired into DiskAPI::delete,
rename_file and rename_data, but object deletion almost never goes through
LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions ->
delete_versions_internal, and delete_paths, all of which remove a version's data
dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring
descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in
that window could still return deleted data.
Invalidate every cached fd under the removed data dir at each site: in
delete_version and delete_versions_internal the data_dir uuid and object path are
in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths
invalidates under each removed path. A later rollback that restores a data dir
just causes the next read to re-open it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177)
Three residual paths could keep serving a stale descriptor:
- delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no
invalidation, and the cache-hit read path skips the volume-access check, so a
cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume
(a per-volume moka predicate) and call it after the bucket is removed.
- A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept
its populated cache alive while still referenced by in-flight ops, so
invalidations through the new instance never reached it. close() now clears the
backend's cache via clear_cached_fds.
- rename_data's post-commit rollback (a commit-metadata fsync failure under
strict durability) restored the old data dir without dropping fds cached during
the committed window; the streaming branch now invalidates the dst part fds on
those rollback paths. The inline branch's rollback runs inside spawn_blocking
and is left to the TTL backstop.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171)
The runtime degradation classification reused the probe-time restriction errnos,
which the driver's C7 contract explicitly warns against, so a single per-file
error could latch a whole disk off io_uring:
- is_io_uring_unsupported no longer includes EACCES: at read time on an
already-open fd it is per-file (an LSM hooks security_file_permission on every
read) and StdBackend hits the same denial, so falling back masks nothing and a
full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after
startup) remain. EOPNOTSUPP is now classified per-path by the caller.
- pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape
error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads
take StdBackend's aligned path, instead of over-latching the whole io_uring
backend or never latching a read-side EINVAL at all.
- try_new only negative-caches genuine restriction-class probe failures in
URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now
falls back without latching, so the next reconnect re-probes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172)
A probe-gated gray release was flying blind: the permanent per-disk `active`
latch flipped with no log and no metric, so the only message operators ever saw
was the startup "io_uring read backend enabled" line — which stayed true on
dashboards even after the very first read latched the disk back to StdBackend
forever.
Add latch_active_off, which flips the latch with `swap` and logs the true->false
transition exactly once at warn with a dedicated event constant, disk root, and
errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric
counter and periodic export of the driver StatsSnapshot (cq_overflow,
cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181)
The dependency-review allow said rustfs-uring is "pulled as a git dependency",
but ecstore now pins it from crates.io. Update the rationale and scope the allow
to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version
bump forces a conscious re-review of the license/provenance claim instead of
being waved through on an outdated justification.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170)
UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk
reference dropped in async context (disk reconnect via renew_disk, or shutdown),
UringDriver's own Drop ran on that thread — sending Shutdown and joining each
shard thread, which can block up to the bounded-drain timeout (5s) on a hung /
D-state disk, stalling a tokio worker.
Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are
unchanged) and add a Drop that takes the Arc and, when a runtime is present,
drops it on a blocking thread so the potentially-blocking join never runs on a
runtime worker. Off-runtime it drops inline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173)
Two byte-for-byte parity breaks against StdBackend on the io_uring read paths:
- A zero-length read on an fd-cache hit returned Ok(empty) without any bounds
check, while StdBackend and the uring miss path return FileCorrupt for an
offset past EOF. Fstat the cached descriptor on the length==0 path and match.
- reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len)
range, but fadvise only drops fully-covered pages, so the head partial page
stayed resident — whereas StdBackend's mmap path reclaims the page-aligned
superset. Bitrot shards' 32-byte block headers keep offsets off page
boundaries, so this diverged on the common case. Page-align the reclaim window
to match the mmap path exactly.
(The third parity item from the audit — a failed reclaim fadvise failing the
read — is already parity: StdBackend's mmap path propagates the same fadvise
error with `?`, so no change is needed.)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174)
The driver's backpressure permits count operations, not bytes, and it zero-fills
a full-size buffer per op, so a single unbounded read could pin ~length bytes per
permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard
range as one pread_bytes with no upstream chunking.
On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into
sequential chunks, awaited one at a time, so worst-case in-flight memory is
bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough
that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The
O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178)
The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but
try_new cannot know the disk count and nothing checked the process fd budget. On
a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks
would already exhaust fds with EMFILE surfacing on reads and probes.
Check the soft limit at try_new: enable the cache only with ample headroom
(>= 16384), otherwise log a warning once and fall back to open-per-read. The
packaged systemd unit sets 1,048,576, so tuned deployments are unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179)
The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable
(bare `return`s or plain eprintlns), so a CI leg on a restricted runner never
exercises the real UringBackend/FdCache/latch paths yet still goes green — an
integration regression could merge unseen.
Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when
RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a
seccomp=unconfined container), panics instead of skipping. Route the silent-skip
sites through it. Wiring a dedicated CI leg that sets that env on a capable
runner is tracked in the issue; this provides the enforcement mechanism.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180)
Add an end-to-end test that seeds the descriptor cache with a read, removes the
part via disk.delete_paths (one of the primary object-delete entry points that
does not go through LocalDisk::delete), and asserts the next read no longer
returns the removed inode — pinning the invalidation added in #1175. The sharded
cancel-routing half of #1180 is covered in the rustfs-uring PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729)
* fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174)
The buffered read path already splits reads above URING_MAX_OP_LEN into
sequential chunks; do the same for the O_DIRECT path, which was left for a
follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk
sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one
block. Extract classify_direct_read_error so the single-op and chunked paths
share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather
than duplicating it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177)
The streaming rename_data branch invalidates cached part fds on its post-commit
rollback paths, but the inline branch runs its commit and rollback inside a
single spawn_blocking closure where the async invalidate cannot be called, so it
was left to the TTL backstop.
Capture the closure's result instead of `??`-propagating it: on error (a
commit-metadata fsync failure under strict durability rolls the committed rename
back), invalidate the dst part paths at the async level before returning. Inline
objects keep their data in xl.meta rather than separate part inodes, so this is
largely defensive, but it removes the caveat and keeps the two branches
consistent.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172)
Complete the gray-release observability. Beyond the warn log added earlier, emit
metrics so a dashboard can answer "how much traffic is on io_uring vs falling
back, and is any disk degrading":
- rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime.
- rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read
fallback (latched-off short-circuit, O_DIRECT error, buffered error).
- a low-frequency per-disk exporter of the driver StatsSnapshot as gauges
(in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a
Weak reference so it never keeps the driver alive, and drops any temporary
strong reference on the blocking pool so a last-reference UringDriver::Drop
join never runs on an async worker (rustfs/backlog#1170).
submit_errors is deliberately not exported yet: it is a field added in the
unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the
dependency is bumped (rustfs/backlog#1181).
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179)
The existing self-hosted sm-standard runners cannot guarantee io_uring is
available (a container seccomp filter can block io_uring_setup), so the ecstore
uring tests degrade to a silent skip and never exercise the real
UringBackend/FdCache/latch paths in CI.
Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no
container seccomp filter, running the uring-named ecstore tests with
RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the
non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable,
so an integration regression can no longer merge green behind a vacuous pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(docker): warn instead of rejecting default or missing credentials
The entrypoint hard-reject from #4278 broke the container-first UX and
the repo's own scheduled e2e lanes (e2e-s3tests, mint boot rustfs-ci
images with default credentials and died at startup). Maintainer
decision: ship no baked-in credentials, warn instead of block.
- missing credentials: warn and start; wording accounts for the
RUSTFS_ROOT_*/MINIO_* alias sources the entrypoint does not inspect
- default rustfsadmin via env/CLI/file: warn and start; the warning
notes all-default pairs cannot derive an internode RPC secret
- malformed config stays fatal: source conflicts, unreadable files,
empty or whitespace-only values, flags missing their argument
- present-but-empty env vars now hit the empty-value hard failure
instead of running the binary with an empty root credential
- empty/default checks trim CR and blanks like the binary; files
without a trailing newline are no longer falsely rejected as empty
- the no-baked-credentials guard covers all four Dockerfiles, and the
test harness refuses hosts where /usr/bin/rustfs exists
- e2e-s3tests/mint move to non-default credentials (rustfsadmin-ci),
which also restores RPC-secret derivation for the multi-node lane
GHSA-68cw fail-closed RPC derivation (#4402) is untouched; helm stays
fail-closed by design.
* chore(docker): reword entrypoint comment flagged by typos check
chore(deps): bump rustfs-uring to 0.2.0 from crates.io
rustfs-uring 0.2.0 is published on crates.io. It carries the read-driver
hardening from the backlog#1160 adversarial audit (cancel-safety and
graceful-degradation fixes on the Linux io_uring read path). The public
API ecstore consumes is unchanged — UringDriver::probe_and_start_sharded,
read_at, read_at_direct all keep their 0.1.0 signatures — so this is a
drop-in bump of the version requirement plus the lockfile entry.
The Cargo.lock change is intentionally scoped to the rustfs-uring package
only; unrelated lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
test(admin-auth): unit + e2e coverage for the admin authorization gate
Adds the first tests for the central admin authorization gate
`rustfs/src/admin/auth.rs`, which previously had zero coverage
(backlog#1151 sec-4, master plan #1155).
Unit tests (rustfs/src/admin/auth.rs):
- Refactor the two `validate_admin_request*` entry points to share a
single generic decision core `evaluate_admin_actions<S: Store>` /
`check_admin_request_auth<S: Store>` (removes the duplicated
action-loop and makes the gate testable without a running cluster).
- Cover: owner/root credential allowed; authenticated non-admin
credential denied with AccessDenied; missing credential denied; and
the multi-action loop grants on any permitted action, denies when
none pass. Backed by an in-memory empty IamSys so the owner path
short-circuits to allow and every other principal resolves to deny.
E2E (crates/e2e_test/src/admin_auth_test.rs, raw SigV4 over HTTP, no
awscurl dependency):
- non_admin_credential_denied_on_admin_api: a limited IAM user gets
403 AccessDenied on GET /rustfs/admin/v3/info while root succeeds.
- root_credential_rotation_takes_effect: restart with rotated
--access-key/--secret-key; new credential works and the stale one is
rejected, on both the admin plane and the S3 data plane.
- default_credentials_emit_startup_warning: booting with the default
rustfsadmin credentials emits the default-credentials warning.
Refs rustfs/backlog#1151 (sec-4), #1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
Three health handler tests assert on payload fields (degradedReasons,
details) that are absent when RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE is
true. The minimal-mode sibling test sets that env var via
temp_env::with_var, which leaks across parallel test threads.
Wrap the three affected tests with their own with_var guard pinning the
env var to false so they are deterministic regardless of thread order.
ci: pull replication tests out of e2e-smoke — loopback targets fail every PR
repl-1 (#4712) added 20 bucket-replication admin-path tests to the
e2e-smoke PR lane. They set a remote replication target at a loopback
endpoint (127.0.0.1, a second local server), which RustFS's target SSRF
guard rejects by default ('outbound URL host 127.0.0.1 is not allowed:
loopback address'). repl-1's local verification was inconclusive under
machine load, so this shipped broken and failed End-to-End Tests on
every PR based on post-repl-1 main (#4707 etc).
Move all replication e2e tests to the e2e-repl-nightly lane (now the full
replication_extension_test set, no allowlist split) to un-block PRs. The
nightly job needs loopback-target-allow server config for these to pass;
tracked as a repl-1 follow-up (backlog#1147). e2e-smoke returns to the
17 functional modules from ci-4 (63 tests).
The fill gate compared each request against a snapshot refreshed at most every
5 s, with no accounting for what it had already let through. A burst arriving
while the snapshot still read high therefore all passed the same check-then-act
test and over-allocated far past the real headroom before the next refresh — a
gap the burst stress test could expose but not close.
Track admitted bytes since the last refresh in the shared snapshot cell and
subtract them from available memory in `allows_fill`, reserving the request's
size on each admission. Cumulative admission is now bounded to the real budget
even though every fill reads the same stale snapshot; the refresh resets the
counter because the fresh reading already reflects those allocations. The
`min_free_memory_percent == 0` opt-out still short-circuits first, and the
already-low-memory path is unchanged.
New test `moka_backend_gate_reservation_bounds_burst_under_stale_snapshot`:
20 concurrent 40 KiB fills against a 500 KiB stale-high snapshot (300 KiB
budget) admit only a bounded handful, not the whole storm. Passed 10/10 runs;
mutation-verified — dropping the reservation admits all 20 and fails the test.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ecstore): unit-test TierConfigMgr add/edit/remove/verify state machine (backlog#1148 ilm-4)
Covers the tier config state machine and persistence paths that previously
had only 4 codec tests and none for tier_config.rs:
- add: non-uppercase name, duplicate name, unsupported type, missing
backend payload, and a regression anchor documenting that AWS-reserved
names (STANDARD) are not currently rejected.
- edit: unknown tier, missing-credentials rejection for RustFS and MinIO.
- remove: idempotent unknown-tier no-op, in-use rejection, empty-backend
success, force skips the in_use probe, and probe-error surfacing.
- verify: unknown tier, healthy backend, unhealthy backend.
- pure query helpers (empty/is_tier_valid/tier_type/get/list_tiers).
- persistence: JSON marshal/unmarshal roundtrip, external tier-config.bin
roundtrip for Azure and GCS payload mapping, truncated/unknown-format/
unknown-version rejection, legacy v1 version-word acceptance, and encode
failure on missing payload.
Tests are hermetic: error paths return before backend construction, and a
MockWarmBackend injected into driver_cache exercises remove/verify without
any real remote. Refs backlog#1155.
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(tier): reject reserved names STANDARD/RRS in TierConfigMgr::add (backlog#1148 ilm-4) (#4721)
test(security): add GHSA-named regression tests for 3p3x and r5qv (backlog#1151 sec-6)
Anchor the two fixed advisories to discoverable, named regression tests so
`rg -i "ghsa|3p3x|r5qv"` finds a guard for each, and future fixes are forced
to update pinned behavior (red -> green).
GHSA-3p3x-734c-h5vx (constant-time WebDAV/FTPS secret comparison, rustfs#4403):
- ftps_core.rs: new `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` drives
the `ct_eq` reject branch in FtpsAuthenticator::authenticate; asserts wrong
password and unknown user are both rejected (530) and indistinguishable.
- webdav_core.rs: the auth-failure block now sends a valid access key with a
wrong secret (exercising the `ct_eq` branch, not just the unknown-access-key
path) plus an unknown user, asserting both 401 and indistinguishable.
- Module doc comments map advisory -> tests -> fix PR on both files.
GHSA-r5qv-rc46-hv8q (internode RPC fail-closed, rustfs#4402):
- http_auth.rs: renamed the default-fallback rejection test to
`ghsa_r5qv_resolve_shared_secret_rejects_default_fallback` (and broadened it
to cover default env secret + blank secrets), and added
`ghsa_r5qv_verify_rpc_signature_fails_closed_on_missing_or_invalid_auth`
pinning the exact advisory scenario (missing/forged/cross-URL signature is
rejected; a correctly signed request still passes). File-level doc maps the
advisory.
Docs: new docs/testing/security-regressions.md with the advisory -> test
mapping table and where each layer runs; linked from docs/testing/README.md.
sec-14 will formalize the written policy in AGENTS.md.
The unit-level ghsa_r5qv_* tests run in the default CI pass. The WebDAV/FTPS
e2e live in the protocols suite (fixed ports, --test-threads=1); they cannot
join the e2e-smoke profile and are wired into CI by sec-5.
Refs: rustfs/backlog#1151 (sec-6), rustfs/backlog#1155
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
test(e2e): add negative header-SigV4 rejection suite (backlog#1151 sec-1)
RustFS delegates SigV4 verification to the s3s dependency, so nothing in
this repo pins OUR end-to-end wiring of it. Add negative_sigv4_test.rs
sending REJECTED header-SigV4 requests against a live RustFSTestEnvironment
and asserting the HTTP status plus the S3 error code XML:
- valid_header_sigv4_request_succeeds (positive control so the negatives
cannot pass for the wrong reason)
- tampered_signature_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- wrong_secret_key_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- tampered_payload_is_rejected (body != signed x-amz-content-sha256) -> not 200
- skewed_date_returns_request_time_too_skewed (>15min) -> 403 RequestTimeTooSkewed
- malformed_authorization_header_returns_clean_4xx (present-not-missing) -> 4xx, never 5xx
Signatures are hand-built via rustfs_signer::request_signature_v4 primitives
(get_signing_key/get_signature/get_scope) so the test controls the timestamp,
secret, signed payload hash, and final signature bytes. Missing-credential
negatives are intentionally not duplicated (covered by multipart_auth_test /
anonymous_access_test).
Refs backlog#1151 (sec-1), master plan backlog#1155.
Adds an S3-API-level e2e regression net proving that a large-object GET on a
degraded EC set never returns a silently truncated body under a full
Content-Length -- the historical "unexpected EOF" bug fixed on main by
rustfs#4594 (short-body GetObject stream -> UnexpectedEof), rustfs#4560
(in-place per-part legacy degradation for the lazy multipart reader), and
rustfs#4585 (DARE package-boundary truncation detection). Those fixes each
ship a *unit* regression; this covers the layer they do not -- the full HTTP
GET path streaming a real body reconstructed from real on-disk EC shards.
New file crates/e2e_test/src/degraded_read_eof_regression_test.rs, single-node
4-disk (EC 2+2) DiskFaultHarness, three scenarios:
(a) one disk offline: a 6 MiB single object (>=2 EC stripes) and a 3x5 MiB
multipart object GET back byte-identical with the correct Content-Length.
(b) mid-stream bitrot within quorum: 2 of 4 shards corrupted mid-file on a
large multipart object still reconstructs the full, hash-matching body.
(c) beyond read quorum (the heart of the net): 3 of 4 shards corrupted
mid-file -- the read MUST fail cleanly (non-2xx or a mid-stream body
error), NEVER close with a truncated body under the full Content-Length.
The shared get_checked() helper panics on the forbidden outcome (a clean 2xx
whose collected body is shorter than the advertised Content-Length), so the
truncation bug can never be silently tolerated.
CI placement: these spawn a 4-disk server per test and are resource-heavy, so
they stay OUT of the fast PR e2e-smoke filter. A new e2e-reliability nextest
test-group (max-threads=1) serializes them (and the existing
reliability_disk_fault_test) across nextest's process boundary; ci-7's nightly
full-e2e run picks them up. Visible via `cargo nextest list -p e2e_test`.
Refs rustfs/backlog#1150 (dist-13), rustfs/backlog#1155, rustfs#4594,
rustfs#4560, rustfs#4585, rustfs#2955.
* test(object-data-cache): add concurrency stress tests for the fill machinery
The race coverage so far pins specific interleavings with an injected barrier.
These add sustained, real-parallelism contention to catch what pinned tests
cannot — a leaked singleflight leader, a stranded semaphore permit, an
index/cache divergence that only surfaces under volume:
- moka_backend_concurrency_storm_leaves_no_leaked_state: 48 tasks x 400 mixed
fill/lookup/invalidate ops on a 6-key pool over 4 worker threads, then
asserts inflight_fills == 0 (no leaked in-flight fill), a clean post-storm
fill/hit/invalidate sequence still works (state not corrupted), and clear()
drains the cache.
- moka_backend_gate_bounds_admission_under_concurrent_burst: a low-memory
snapshot must reject an entire 32-fill burst — admission is bounded, not a
check-then-act race that lets the burst through. (This does not cover the
separate 5s-stale-snapshot overshoot, which needs byte-based reservation.)
Both are mutation-verified: neutering invalidate_object fails the storm's
"invalidation must win" assertion, and making allows_fill always-true fails the
burst's full-rejection assertion. The storm passed 20/20 runs. `rt-multi-thread`
is added to dev-deps so the tasks run on real worker threads.
Refs: backlog#1107
Co-Authored-By: heihutu <heihutu@gmail.com>
* bench(object-data-cache): measure the GET-path cost the audit argued statically
Adds a criterion benchmark driving the public ObjectDataCache facade, so the
"hot path is clean / high throughput" claim rests on numbers, not analysis:
- plan_get ~104-112 ns (per-GET planning + metric label-set hash)
- lookup_hit ~143-147 ns (a resident-body hit: moka get + Bytes clone + metric)
- fill_distinct ~2.2-2.4 us (cold fill: plan + singleflight + index + moka + inner spawn)
A hit at ~143 ns against a multi-millisecond erasure read is the ~1000x saving
the cache exists for; the per-GET metric cost is real but ~100 ns, not a
bottleneck; the fill cost (incl. the cancellation-safety spawn kept on purpose)
is negligible on a background task. Run with
`cargo bench -p rustfs-object-data-cache`.
Refs: backlog#1107
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Adds a serial-lane ILM integration regression test that pins the
local-first ordering contract established by rustfs#3491, where
`expire_transitioned_object` deletes local metadata BEFORE any remote
tier cleanup so a concurrent GET can never observe live local metadata
pointing at an already-removed remote tier version (user-visible
`NoSuchVersion`).
`serial_tests::test_expire_transitioned_object_never_races_concurrent_get`
in `crates/scanner/tests/lifecycle_integration_test.rs`:
- Transitions an object to a mock warm tier, then runs a tight
concurrent GET loop while calling `expire_transitioned_object`; every
GET must return a full, correct body or a clean object/version-not-found
-- never a tier-fetch failure.
- Deterministic ordering assertion (revert-proof): immediately after
expiry the remote tier object is still present and the mock recorded
zero remote `remove` calls, proving remote cleanup is deferred to
free-version recovery. Reverting to remote-first ordering turns this
assertion red.
Supporting changes:
- Re-export `expire_transitioned_object` from `api::bucket::lifecycle`.
- Record remote-tier mutating ops in the scanner test `MockWarmBackend`.
- Update tier-ilm-debugging.md to reference the new regression test.
Runs in the CI ILM Integration (serial) lane (ci.yml
test-ilm-integration-serial), picked up by the existing
binary(lifecycle_integration_test) filter.
Refs: rustfs/backlog#1148 (ilm-2), rustfs/backlog#1155
Activate the 36 dormant replication e2e tests in
crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before).
Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane:
- PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast
bucket-replication tests (target-registration / replication-check / list /
remove / delete admin paths) that validate config synchronously and never
wait for async convergence. Each spawns its own single-node rustfs server(s)
on random ports with isolated temp dirs, so parallel-safe by construction
(serial_test's #[serial] is a no-op under nextest's process-per-test model;
no test-group needed).
- Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml):
the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1
_real_single_node. Defined as 'replication module MINUS the PR allowlist' so
new replication tests default to nightly and are never silently unrun.
The nightly workflow builds the binary once, installs awscurl so the STS
dual-node test runs (skips gracefully with a visible log line otherwise), and
routes scheduled failures through .github/actions/schedule-failure-issue
(ci-8). Explicit division of labor with ci-5 e2e-full: these run only here.
Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16.
Docs updated: e2e-suite-inventory.md, e2e_test/README.md.
Refs backlog#1147 repl-1, backlog#1155.
docs(e2e): write contributor guide for e2e_test crate (backlog#1153 infra-10)
Turn the e2e_test README skeleton into a dense, link-heavy contributor guide:
crate overview + grouped module map, how to run (whole crate / one module /
e2e-smoke profile / ILM serial lane / protocols suite), #[ignore] semantics as
a pointer to live sources, how to add a test (RustFSTestEnvironment +
RustFSTestClusterEnvironment, common.rs/chaos.rs helper inventory, isolation
rules, #[serial]-vs-nextest reality), a CI subset map table, and a
troubleshooting section (stale binary / RUSTFS_TEST_PORT / orphan processes).
Keeps the ci-4 "CI smoke subset" section intact and restructures around it.
Adds a reciprocal link from crates/e2e_test/AGENTS.md.
Refs rustfs/backlog#1153 (infra-10), #1155.
* refactor(object-data-cache): derive Default for ObjectDataCacheGetRequest
The GET request literal is hand-listed field-by-field across ~13 test sites in
two crates. Adding `mod_time_unix_nanos` in backlog#1111 had to touch every one
and still missed a literal, producing a compile error caught only in a later
CI lane. Derive `Default` and spread the engine-crate literals so the next
field addition is absorbed rather than fanned out.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(cache): key the object body cache on data_dir for write-uniqueness
The cache key's correctness rested on being write-unique, but its only
write-scoped component was `mod_time` — a wall-clock timestamp that is not
monotonic and can be absent. Two writes that collide on MD5 (same etag+size)
and land on an equal mod_time (clock skew, same-tick, or absent) derived the
same key, so a node that never saw the overwrite could serve the previous body
for up to the TTL, and the same collision turned the fill-after-invalidation
race into a serving bug. This is the store's strong-read-after-write guarantee
leaning on a probabilistic argument.
Add `data_dir` — the xl.meta directory UUID ecstore regenerates on every body
write — as the primary write-unique anchor:
- surface `data_dir: Option<Uuid>` on ObjectInfo, copied from FileInfo in the
single GET-path constructor `from_file_info`;
- carry it into ObjectDataCacheKey as `data_dir_u128` (held as u128 to keep the
engine crate free of a uuid dependency), derived in the one planner site both
the ecstore hook and the usecase layer share, so both produce an identical
key by construction;
- keep `mod_time` as a second anchor and `etag+size` as belt-and-braces; an
absent data_dir falls back to the prior behavior — strict improvement, no
regression.
Two writes distinct only by data_dir now derive different keys even under an
MD5 collision with identical mod_time — the case mod_time alone cannot cover.
Blast radius is compiler-guarded: ObjectInfo derives Default and every real
construction site uses `..Default::default()`, so only from_file_info and one
full-literal test needed the field. The three P0 body_cache_hook_e2e
regressions, engine (80), and app (36) suites pass unchanged; a mutation that
severs the planner wiring fails planner_key_changes_with_data_dir.
Refs: backlog#1111, backlog#1118
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(cache): thread data_dir field through the merged mutation-hook test
Merging main (which landed the object-mutation-hook work, backlog#1131) brought
in a GetRequest test literal that predates the data_dir field. Spread it via
`..Default::default()` — the derive(Default) added here means this is the last
such hand-listed literal to need touching.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The recent addition of ObjectDataCacheGetRequest.mod_time_unix_nanos
(backlog#1111 / ODC-06) missed the test helper in mutation_hook.rs,
causing a compile error under clippy --all-targets.
ci: quarantine walk_dir stall-budget flake under the ci profile (rustfs#4690)
First application of the flake policy from docs/testing/README.md: the
test failed on a zero-Rust-diff PR (#4674, run 29099382470) — timing
windows in the stall-budget accounting stretch under CI load. retries=2
under profile.ci only; local default profile still never retries.
Tracked by rustfs#4690 (30-day fix-or-delete deadline).
* feat(object-data-cache): close write/delete-side invalidation gaps
The object data cache exposed only a single per-(bucket,object)
invalidation primitive and no write-side ecstore hook, so several
delete paths left dead bodies resident until TTL (hygiene/capacity, not
stale-serving: lookups follow a fresh metadata quorum and cannot serve a
gone object). This adds the missing primitives and wires them in.
ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET
body hook, registered next to it at startup, and call it from the
ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`,
`expire_transitioned_object` including the restored-copy branch, and
`delete_object_versions`). The app impl is one `invalidate_object` call
under a new `AfterLifecycleExpiry` reason.
ODC-27 (backlog#1132): force prefix delete now invalidates the whole
prefix, not just the prefix string. `store.delete_object(delete_prefix)`
returns no deleted-name list, so this uses a new prefix primitive rather
than the batch path.
ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new
bucket-scope primitive (covers force and non-force, which share the
delete_bucket call).
ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin
handlers (GET stats, POST flush) routed through admin runtime_sources.
The starshard identity index gains a single `remove_matching` full-scan
API backing prefix/bucket/clear; it is documented as admin/delete-path
only and never runs on the GET or fill hot path. New invalidation
reasons and metric labels added; outcome (removed/noop) labelling kept
correct for every new primitive.
Also fixes a pre-existing broken intra-doc link in memory.rs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(ecstore): extract the shared HookSlot behind both cache hooks
This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs,
which left two process-global registration slots whose register/get/clear
bodies were line-for-line identical except the trait type and the WARN string:
a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the
poison-recovery closure, and the same read-lock-and-clone read. Two copies of
the same swap-vs-warn logic can drift apart under maintenance.
Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook
module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged
public wrappers (register_/get_/clear_), so the crate's public surface and
every call site are untouched — this is an internal consolidation, not a
contract change.
The load-bearing #1126 guarantee (newest registration wins, so a rebuilt
AppContext is never stranded on a first-wins slot) previously had no direct
test — the hook tests only covered register-then-notify. HookSlot now has its
own unit tests including re_registration_swaps_to_the_latest_instance;
mutation-testing confirms a first-wins regression fails exactly that test.
No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e
regressions, and the app-layer mutation-hook tests all pass unchanged.
Refs: backlog#1126, backlog#1131
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): register the object-data-cache routes in the policy inventory
This PR added GET /object-data-cache/stats and POST /object-data-cache/flush
but did not list them in the two registries that must account for every
admin route: the route-policy inventory (route_policy.rs) and the route
matrix (route_registration_test.rs). Their coverage tests —
route_policy_inventory_covers_registered_routes and
test_admin_route_matrix_matches_registered_routes — failed on CI because a
registered route had no policy/matrix entry.
These two tests are not part of `make pre-commit` (which runs fmt + arch +
quick-check, not the full suite), so the gap passed local pre-commit and
only surfaced in the CI Test-and-Lint lane.
stats is a read (ServerInfoAdminAction, Sensitive); flush mutates
(ConfigUpdateAdminAction, High) — matching the actions the handlers already
enforce. The MinIO-alias matrix test is unaffected: these are native rustfs
endpoints with no MinIO equivalent.
Refs: backlog#1143
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(ecstore): switch rustfs-uring from the git rev to crates.io 0.1.0
Prepared ahead of the rustfs-uring 0.1.0 crates.io release (rustfs/uring):
flip ecstore's dependency from the pinned git revision to the published
`0.1.0`, and drop the now-unneeded git-source allowance in deny.toml.
DO NOT MERGE until rustfs-uring 0.1.0 is on crates.io. Until then cargo
cannot resolve `rustfs-uring = "0.1.0"`, so this branch will not build and
CI will be red — that is expected, not a defect in the change.
After the crate is published:
1. `cargo update -p rustfs-uring --precise 0.1.0` to regenerate Cargo.lock
(deliberately not touched here — the registry entry, with its checksum,
cannot be written before the crate exists);
2. push the Cargo.lock;
3. CI goes green; merge.
No code change. The guard `scripts/check_no_tokio_io_uring.sh` is unaffected
(it bans only tokio's io-uring runtime feature, not an explicit rustfs-uring
dependency). `audit.yml`'s `allow-dependencies-licenses` for rustfs-uring is
left in place — it is a first-party Apache-2.0 crate either way.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: refresh rustfs-uring lockfile
---------
Co-authored-by: heihutu <heihutu@gmail.com>
#4663 added a full S3 upload path (config fields + env parsing + wiring), then
reverted the wiring when its dependency turned out to carry RUSTSEC advisories
(D9-14). The revert stopped at enabled.rs and left the config layer half torn
down: `Dial9Config` still carried `s3_bucket`/`s3_prefix` with no consumer, plus
a dead `s3_upload_requested()`, and `Dial9SessionGuard::config()` was likewise
never called. All three are zero-consumer dead code.
Since S3 upload cannot be built at all, the config type should not model it.
Drop the two fields and both accessors. `from_env` still reads the S3 env vars
and warns about them (an operator who set them deserves to know why they do
nothing — that warning is deliberate and stays), but drops the values instead
of storing configuration nothing reads.
While here, collapse `from_env`'s two `record_config` calls into one by
extracting `from_env_enabled`, so the enabled/disabled paths share a single
publish point.
No behavior change: the S3 warning fires identically, and every other field is
untouched. Net -13 lines, and `Dial9Config` drops from 8 fields to 6.
Co-authored-by: heihutu <heihutu@gmail.com>
docs(object-data-cache): state the correctness boundary and the timing channel
The crate doc still described itself as "a minimal skeleton for the initial
rollout phase", which stopped being true several releases ago, and neither
the crate nor the operator-facing env constant said anything about what the
cache does or does not guarantee.
Record two things a reader has to know.
The correctness boundary: a hit is sound because the key matched metadata the
caller just resolved from a read quorum, not because invalidation ran.
Process-local invalidation is hygiene that frees capacity; the key is what
keeps a stale body from being served. Anyone tempted to weaken the key should
meet this sentence first.
The timing side channel: a hit skips the erasure read, bitrot verify and
decode, so it is reliably faster than a miss. A principal authorized to read
an object can time a single GET and learn whether someone read that object
within the entry's lifetime. This crosses no authorization boundary — the
probe needs read access to that exact object, checked before the cache is
consulted — but it does disclose a co-tenant's recent access pattern. Say so
where operators look: on RUSTFS_OBJECT_DATA_CACHE_ENABLE. Timing noise would
cost precisely the latency the cache exists to save, so the mitigation is to
leave the cache off for buckets where access-pattern confidentiality matters.
Refs: backlog#1139
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com>
The rename_data crash-consistency harness (backlog#935/#878) only armed
pre-commit crash points (AfterDataRename, AfterBackupBeforeMetaCommit),
which assert the *old* version survives. The other half of the
"partial commit -> old or new, never mixed" invariant — a hard power loss
*after* the xl.meta commit rename must leave the *new* version — had no
crash-point coverage. The existing after-metadata-commit failpoint only
exercises the graceful in-process rollback (restores old), a different
path from a hard crash (no rollback runs, commit stays on disk).
Add a RenameDataCrashPoint::AfterMetaCommit injection right after the
commit rename (cfg(test), compiles to a const-false no-op in production,
like the existing points) and overwrite/fresh x strict/relaxed tests
asserting the object reads back as the new version. The fresh case pins
that the point is genuinely post-commit: a pre-commit misplacement would
leave no readable object and fail the assertion.
cargo test -p rustfs-ecstore crash_consistency: 9 passed. clippy -D
warnings clean; fmt and arch guards clean.
P-A (`read_appending`) and P-C (the in-memory fast path) each copied the
hashed-read logic, so `BitrotReader` ended up with the hash verification, the
short-read error, and the scratch-buffer fill written three times across
`read` and `read_appending`'s two branches. That is patch-on-patch: a change
to the bitrot contract would have to be made in three places and kept in
sync by hand.
Collapse the duplication onto three single-source pieces:
- `split_and_verify` — a free function that splits `[hash][data]`, verifies
(unless skip_verify), and returns the data slice plus the hash time. Free
rather than a method so it can run while `self` is borrowed for the block.
- `read_scratch_block` — the single-pass fill of `self.buf` with the
short-read-to-UnexpectedEof contract.
- `short_shard_read` / `begin_read` — the shared error and preamble.
`read` and `read_appending` now differ only in what they must: how the block
is acquired (caller's slice vs `try_take_block` vs scratch fill) and where the
verified shard lands (`copy_from_slice` vs `extend_from_slice`).
This also fixes a latent inconsistency the duplication hid: the old `read`
did `copy_from_slice` *before* verifying, so on a hash mismatch it left the
corrupt bytes in the caller's buffer before returning the error, while
`read_appending` verified first. Both now verify before writing, so a shard
that fails the hash never reaches the caller's buffer on either method — the
stronger of the two behaviors.
Net -19 lines; behavior otherwise unchanged. Verified: `erasure::` 215
passed, 0 failed (including the fast-path equivalence and corrupt-shard tests,
and the existing `test_bitrot_read_hash_mismatch`); clippy --all-targets
-D warnings clean.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): remove the dial9 task-dump switch that could never do anything
Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.
dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:
tokio::spawn -> 0 dumps
dial9::spawn -> 14709 dumps, all with callchains
Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.
Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.
Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.
Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): replace guessed dial9 retention numbers with measured ones
Three corrections, all to claims I wrote in #4663 without measuring them.
"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.
dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.
What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.
Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).
Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`pread_uring` and `pread_uring_direct` accumulated across seven rounds
(#4632/#4635/#4645/#4649/#4653/#4658/#4662) and each carried its own copy of
the same open preamble: resolve the bucket path, run the volume access check,
resolve the object path, and check the path length. The only real difference
is what happens after — one opens buffered and errors as `DiskError`, the
other opens `O_DIRECT` and errors as `DirectOpenError` so it can latch an
O_DIRECT refusal.
Extract that shared sequence into `resolve_uring_object_path`, a blocking
helper called inside both `spawn_blocking` closures. Each caller keeps exactly
what differs — its open flags, its error type, its metadata/bounds/align
handling — and no longer restates the resolution. No behavior change: the same
checks run in the same order and produce the same errors (the O_DIRECT path
still maps them through `DirectOpenError::Disk`).
Net -12 lines. Verified on a real Linux host (16-core, real io_uring):
clippy -p rustfs-ecstore --all-targets -D warnings clean; disk::local tests
144 passed, 0 failed — including the io_uring, O_DIRECT, fd-cache, and
page-cache-reclaim cases that exercise both read paths.
Co-authored-by: heihutu <heihutu@gmail.com>
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
CI runs `cargo clippy --all-targets -- -D warnings` without
`--all-features`, but the Makefile's clippy-check used `--all-features`.
After PR #4663 made the dial9 telemetry feature opt-in and removed
`--cfg tokio_unstable` from .cargo/config.toml, `--all-features`
activated dial9 (which requires tokio_unstable) and dial9-taskdump
(which requires tokio/taskdump, Linux-only), breaking local
`make pre-pr` on macOS.
Align the Makefile with CI by dropping `--all-features` from clippy-check.
* perf(ecstore): slice in-memory shards instead of copying them twice (backlog#1159)
The GET path reads a shard out of the page cache into a `Bytes`, then
`open_disk_reader` erased it behind `Box<dyn AsyncRead>` by wrapping it in
a `Cursor`. Downstream, `BitrotReader` could only get it back by copying:
once out of the `Cursor` into its scratch buffer, and once from there into
the caller's buffer. CPU profiling of a cached 1 MiB GET (device reads = 0)
attributed 8.23% of the whole server to `Cursor::poll_read` alone — a copy
of data that was already sitting in memory.
Keep the source concrete instead of erasing it. `ShardReader` is an enum of
`InMemory(Cursor<Bytes>)` and `Stream(Box<dyn AsyncRead ...>)`, and the new
`ShardSource::try_take_block(n)` lets an in-memory source hand over the
`[hash][data]` block as a slice. `read_appending` uses it to verify the hash
on the slice and `extend_from_slice` the shard straight into the caller's
buffer: one copy instead of two.
`try_take_block` defaults to `None`, so a streaming source keeps the old
path byte for byte, along with its short-read and EOF semantics. A source
that cannot serve `n` bytes declines rather than truncating, so a partial
block still becomes UnexpectedEof rather than a short shard. The hash is
still checked before anything is appended, so a corrupt shard never reaches
the caller's buffer on either path. The deferred parity reader opens its
source lazily and stays on the streaming path; parity is only read when a
data shard fails.
Tests gate equivalence and non-vacuity:
* `try_take_block` fires for `Cursor<Bytes>`, advances the position exactly
as a read of the same length would, declines when fewer than `n` bytes
remain, and returns `None` for a non-`Bytes` source — without this the
equivalence test below would silently compare one path against itself;
* both paths return identical bytes for the same shard;
* a corrupt shard fails on the fast path too, appending nothing.
Verified: clippy --tests -D warnings clean; `erasure::` 215 passed, 0 failed;
`set_disk::core::io_primitives` 49 and `io_support::` 22 pass.
Stacked on #4681 (`read_appending`), which this builds on.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): implement ShardSource for Cursor<&[u8]> used by the erasure bench
`crates/ecstore/benches/erasure_benchmark.rs` builds
`BitrotReader<Cursor<&[u8]>>`, which the new `ShardSource` bound on
`ParallelReader`/`decode` does not accept. `cargo clippy --tests` does not
compile bench targets, so this only surfaced in CI's `--all-targets` run.
A borrowed slice carries no `Bytes` to hand out, so it takes the default
`try_take_block` and keeps the old streaming copy path — no behavior change.
Verified with the same target set CI uses:
`cargo clippy -p rustfs-ecstore --all-targets -- -D warnings` clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
ci: count migration-gate tests via nextest JSON, not human-format indentation
The count-floor guard (#4673) parsed nextest's human list format with
grep -c '^ '. CI's newer nextest emits a different indentation, so the
count collapsed to 0 and the gate failed on every PR based on current
main (first observed on #4683, then #4674/#4677/#4680). Count via
--message-format json + jq instead, which is stable across versions, and
fail loudly if the JSON shape ever changes.
git worktree add prints 'HEAD is now at …' on stdout, which polluted the
path captured from build_ref_binary and made the rig exec a two-line
string as the baseline binary (dispatch run 29087503110 died 1s after
spawn with 'No such file or directory'). Route command output to stderr,
verify the baseline binary exists before returning (exempt in dry-run),
and fix the 'mis-reports' typo flagged by the Typos check.
Introduce RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS, RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS
and RUSTFS_REPL_RESYNC_POLL_MAX_MS so tests and operators can shorten the
replication background loops. Defaults are unchanged; invalid values fall
back with a warn and values below 10ms are clamped to avoid busy-spin.
Add replication_fast_env() e2e helper (backlog#1147 repl-4).
feat(ecstore): add RUSTFS_ILM_DEBUG_DAY_SECS lifecycle time-acceleration for tests (backlog#1148 ilm-5)
Model a Ceph rgw_lc_debug_interval-style switch: when RUSTFS_ILM_DEBUG_DAY_SECS
is set to a positive integer N, lifecycle Days-based rule evaluation treats one
'day' as N seconds instead of 86400, so Days>=1 expiration/transition/noncurrent
rules become exercisable in seconds. Unset => byte-identical to production.
All Days->deadline conversions funnel through expected_expiry_time(); the new
ilm_day_secs() helper (OnceLock-cached in production, env-read under cfg(test))
rescales both the day offset and the default rounding boundary. An explicit
RUSTFS_ILM_PROCESS_TIME still wins for the rounding boundary. Absolute Date-based
rules are deliberately NOT rescaled. Activation emits a WARN; test/debug only.
Refs rustfs/backlog#1148 (ilm-5), rustfs/backlog#1155.
docs: fix 'mis-reports' typo failing the Typos check on every PR
Merged via #4669 into docs/operations/hotpath-warp-ab-runbook.md:132; the
Typos CI check now fails on all PRs based on current main.
Implements ilm-1 from the ILM test-strategy plan: the 33 #[ignore]d ILM
integration tests (14 in crates/scanner/tests/lifecycle_integration_test.rs,
19 in rustfs/src/app/lifecycle_transition_api_test.rs) never ran anywhere.
This adds a dedicated serialized CI job and repairs the app-layer suite,
activating 29 of them.
- ci.yml: new test-ilm-integration-serial job running the two ILM test
surfaces with 'cargo nextest run -j1 --run-ignored ignored-only'. The
tests share process-global singletons and fixed ports (9002/9003);
serial_test's #[serial] does not serialize across nextest's
process-per-test boundary, so -j1 is the serialization mechanism.
- app suite repair: the 19 app tests rotted after the InstanceContext
migration (usecases resolve the store via AppContext; the test env never
installed one, so every store-touching call failed with InternalError
'Not init'). Add a test-only install_test_app_context helper behind the
sanctioned context/runtime_sources boundaries and switch the tests from
without_context() to from_global(). All 19 now pass.
- delete test_lifecycle_transition_basic (+3 orphaned helpers): required
an external MinIO at localhost:9000 and slept 1200s; superseded by the
mock-tier tests in the same file.
- fix a timing flake: poll free-version metadata removal instead of
asserting a single read right after remote-object absence.
- 3 scanner tests fail on main for product reasons (multipart restore
UnexpectedEof; noncurrent transition/expiry after immediate compensation
transition on versioned buckets); they keep #[ignore] with a backlog
reference and are excluded from the lane by name.
Lane: 29 tests, ~43s test time, green twice locally.
Refs rustfs/backlog#1148 (ilm-1), rustfs/backlog#1155.
The migration-proof step in ci.yml selects tests by name substring
(data_movement / rebalance / decommission / source_cleanup /
delete_marker), so a rename can silently thin the gate to zero with no
CI signal. Move the filter expression into
scripts/check_migration_gate_count.sh as its single source of truth:
the script counts the selected tests with cargo nextest list, fails if
the count drops below the committed floor in
.config/migration-gate-floor.txt, and then runs the gate with the same
filter so the check and the run cannot drift.
The floor equals the exact selected-test count at landing (571), so any
reduction fails CI and intentional shrinks must edit the floor file in
the same PR, keeping gate changes visible in diffs.
Refs rustfs/backlog#1153 (infra-12), rustfs/backlog#1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
build(make): require cargo-nextest for make test with explicit escape hatch (backlog#1153 infra-14)
cargo-nextest changes test semantics — it runs each test in its own process
(so serial_test's #[serial] mutex does not serialize across tests) and is the
only runner that honours .config/nextest.toml [test-groups] (e.g. the
ecstore-serial-flaky serialization guard). CI installs and runs nextest, but
`make test` only warned when it was missing and then silently fell back to
`cargo test`, which runs with different serialization behaviour than CI and can
mask or invent flakes.
Make cargo-nextest a hard dependency of `make test`:
- Missing nextest now fails `make test` with a clear message and install
instructions (`cargo install cargo-nextest --locked` or a prebuilt binary
via https://nexte.st/docs/installation/).
- Set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to opt into the plain `cargo test`
fallback anyway; it prints a loud warning that serialization semantics differ
from CI and .config/nextest.toml test-groups will NOT apply.
- The check stays cheap (command -v). The warn-only test-deps prerequisite is
dropped from check.mak in favour of recipe-level gating.
Install docs live in the make-layer error message plus a header comment in
tests.mak, with a single-line note in CONTRIBUTING.md (kept minimal to avoid
conflicting with #4667 / infra-9, which rewrites the verification sections).
No CI change: .github/workflows/ci.yml already runs cargo nextest and
.github/actions/setup/action.yml already installs it.
Refs rustfs/backlog#1153 (infra-14), master plan #1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
Scheduled workflow failures previously went unnoticed (15 consecutive red
weekly s3-tests sweeps, silent perf nightly failures). Add a reusable
composite action that opens a tracking issue titled
"[scheduled-failure] <workflow name>" — or appends a comment to the
existing open one (dedupe key = workflow name) — with the run URL, run
attempt, and failed job names, labeled "infrastructure".
Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and
performance-ab via a final alert-on-failure job gated by
"always() && github.event_name == 'schedule' &&
contains(needs.*.result, 'failure')", with issues:write scoped to that
job only. e2e-s3tests and mint previously had no permissions key; they now
get workflow-level contents:read, reducing every other job's token.
Add a dispatch-only drill workflow that forces a job failure and runs the
action through the exact consumer wiring, for end-to-end verification.
The ci-7 e2e nightly does not exist yet; it is recorded as a pending
consumer in the action README.
Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155
Both nightly runs of the warp A/B rig failed at server bring-up:
`endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s
after start. The server binds its public listener only after startup converges
(erasure-format load + IAM); its own budget
(RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s
health poll gave up before a slow cold start on the shared sm-standard-2 runner
finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/),
so the root cause was invisible in the artifact.
Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh):
- Health poll is configurable via --health-timeout, default 180s (> the 120s
server startup budget). Fails fast if the local server process exits before
becoming healthy instead of polling out the full window.
- Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they
ride along in the CI artifact. On a health failure the rig dumps the last 50
log lines to the job log.
Workflow (.github/workflows/performance-ab.yml):
- On every run, write gate.md (or, on a pre-gate failure, the failing phase's
server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention.
- Short measurement matrix (--duration/--rounds/--cooldown) so the expanded
matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3
caches the baseline binary (the ~65min double build dominates).
- TODO(ci-8/perf-2) hook comment for the failure-alert composite action.
Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync
regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical
large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads
x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the
budget until perf-3.
Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.
The mint multi-SDK compatibility pipeline had exactly one recorded run
(28730530597, 2026-07-05), which died at "Enable buildx": the self-hosted
sm-standard-4 pod it landed on had no /var/run/docker.sock, so
`docker buildx create` failed to connect to the Docker API before any test
ran. Same heterogeneous-fleet root cause ci-1 fixed for the weekly s3-tests
sweep.
- Pin the job to GitHub-hosted ubuntu-latest, which reliably ships a running
Docker daemon + buildx + python3. Header note records the diagnosis and the
dind-sm-standard-2 tradeoff.
- Pin MINT_IMAGE default to the linux/amd64 digest for minio/mint:edge
resolved 2026-07-10 (was the rolling :edge tag); workflow_dispatch can still
override. Header comment records the refresh command.
- Quote shellcheck-flagged vars and drop an unused loop var so actionlint
passes with zero findings.
- Left report-only (ci-3 adds baseline gating later, per adjudication G4) and
a TODO(ci-8) alerting hook on the scheduled job.
Refs: rustfs/backlog#1149 (ci-2), rustfs/backlog#1155
`ShardBufferPool::take` handed out a `resize(len, 0)`-ed buffer, and the
reader then overwrote every byte of it. CPU profiling of a cached 1 MiB
GET (device reads = 0, so all cost is CPU) attributed 4.81% of the whole
server to that memset — a buffer pool exists to reuse an allocation, and
memsetting it gives the saving straight back.
The zeroing was load-bearing only because `BitrotReader::read` takes
`&mut [u8]`, which must be initialized. But the reader never reads what
the caller put there, and never returns a partially filled buffer: both
the hashed and the no-hash path either fill the whole shard or fail with
UnexpectedEof, and a hash mismatch is an error rather than a short read.
So the initialization bought nothing observable.
Add `BitrotReader::read_appending(&mut Vec<u8>, want)`, which appends into
the buffer's spare capacity instead of demanding initialized bytes:
* hashed path — unchanged single copy, `extend_from_slice(data)` in place
of `copy_from_slice` into a pre-zeroed buffer, and only after the hash
verifies, so corrupt bytes never reach the caller's buffer;
* no-hash path — `read_buf` writes straight into the spare capacity and
advances the length only over bytes the reader actually wrote, so an
uninitialized tail can never be exposed.
`ShardBufferPool::take` now yields an empty buffer with capacity, and
`read_shard` no longer needs to `truncate`. `read` keeps its old signature
for the remaining callers.
Four tests gate the contract rather than the call:
* `read_appending` is byte-for-byte identical to `read` on both paths;
* a truncated shard is UnexpectedEof, never a partially filled buffer;
* bytes that fail the bitrot hash never reach the caller's buffer;
* `want > shard_size` is rejected;
plus the pool test now asserts the allocation is reused (same pointer) and
never zeroed.
Verified: `erasure::` 213 passed, 0 failed; on a real Linux host
`erasure::` 209 and `disk::local::` 143 pass serially, and the failures
seen in a parallel full-suite run reproduce identically on unmodified main
(they are ENOSPC from a full root filesystem plus pre-existing flakes).
Not claimed: an end-to-end throughput number. The A/B on the bench host was
too noisy to attribute (one rep pair was not fully cached, and its root
filesystem filled mid-run); what is measured is that the removed memset was
4.81% of GET CPU in the pre-change profile.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): roll back delete on disks that staged then errored (backlog#1158)
#4300 rolls back a failed delete only on disks that returned Ok, skipping any
disk that staged its rollback backup, applied the delete, and then errored --
leaving that disk deleted while its peers are restored. On rollback, fan the
undo out to every online disk instead; the disk-side restore_delete_rollback
is already idempotent (Ok no-op when nothing was staged), so unstaged disks
are unaffected. The err.is_some() skip now applies only to the success/cleanup
path. Covers both single-object and batch delete.
Refs backlog#1158.
* fix(object-data-cache): make cache metrics one-increment-per-GET
Rework the object data cache observability so each counter carries one
clear meaning, and drop dead per-entry state.
ODC-17 (backlog#1122): split requests_total, which was incremented by
both plan_get and lookup_body, into plan_total{decision,reason,size_class}
(emitted only by plan_get) and lookup_total{result,size_class} (emitted
only by lookup_body). Each is now incremented exactly once per GET per
layer; help text states this.
ODC-18 (backlog#1123): add a JoinedInflightFill result (label
joined_inflight) so a singleflight waiter is no longer counted as an
insert. record_fill_result now always counts the outcome but records
fill bytes only when non-zero and the duration histogram only when
present, so joined_inflight / skipped_by_mode / skipped_size_mismatch
no longer inflate fill_bytes_total or add non-fill duration samples.
The waiter->JoinedInflightFill mapping lives in MokaBackend (owned by a
concurrent branch); cache.rs handles the variant already.
ODC-29 (backlog#1134): stop refreshing the cache-state gauge on every
lookup, and debounce it on fill/invalidate to at most once per second
via an AtomicU64 millis timestamp, since moka's entry_count is a
settling approximation.
ODC-36 (backlog#1141): give invalidations_total an outcome label
(removed|noop) and skip the gauge refresh on the no-op path. Extend
ObjectDataCacheInvalidationResult with Removed{keys}/NoOp (Success kept
as a transitional variant until MokaBackend reports the removal count);
NoopBackend now reports NoOp.
ODC-30 (backlog#1141): drop the dead content_length/etag/inserted_at
fields (and getters) from ObjectDataCacheEntry, which are redundant with
the moka key identity; the constructor keeps its arity so the
out-of-scope MokaBackend caller still compiles. Gate is_null_version
behind cfg(test).
Tests use a thread-local metrics recorder to avoid the global-singleton
flakiness. Fill-enabled test configs set min_free_memory_percent=0 so
the cgroup gate does not refuse fills in CI pods.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): move fill off GET path, bound & harden the fill
Implements five object-data-cache audit findings.
ODC-15 (backlog#1120): the GET response no longer blocks on cache fill.
The buffered/materialized body is already in hand, so the fill now runs
in a detached task and the response is built immediately. Singleflight is
made non-blocking: `try_acquire` returns Leader or Busy, and a non-leader
skips its own fill (SkippedSingleflightBusy) instead of waiting on another
request's leader. The inner fill spawn is KEPT: once the index key is
registered, the recheck/undo must complete even if the enclosing fill
future is aborted, so removing it would weaken the cancellation-safety
guarantee (ODC-13 test) and the invalidation-race undo.
ODC-11 (backlog#1116): enforce the fill-concurrency knobs. MokaBackend
now holds a Semaphore sized min(per_cpu * parallelism, max), acquired
after winning leadership and before the memory gate. On saturation it
rejects (try_acquire_owned) with SkippedFillConcurrency rather than
queueing, so the fill path never reintroduces GET-latency coupling.
ODC-14 (backlog#1119): the memory gate no longer does a blocking sysinfo
refresh under a mutex on the async fill path. A dedicated periodic
refresher (tokio interval + spawn_blocking) updates an atomic snapshot
every 5s; allows_fill is now lock-free. The min_free_memory_percent == 0
short-circuit still runs before any snapshot read. No runtime at
construction (sync unit tests) simply keeps the seed snapshot.
ODC-32 (backlog#1137): singleflight no longer emits metrics while holding
the map mutex. try_acquire/remove_entry capture the map length under the
guard, drop it, then emit the gauge/counter.
ODC-07 (backlog#1112): the materialize read is bounded via
take(capacity + 1) so an over-long stream cannot grow the buffer past the
in-memory GET threshold, and any length mismatch is now a hard error
matching the direct-memory GET path, instead of warn-and-serve.
cache.rs is owned by a concurrent branch; this commit only appends the
SkippedFillConcurrency and SkippedSingleflightBusy variants + label arms.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): report JoinedInflightFill and invalidation outcome
Reconciles the moka_backend half of two metrics-semantics findings after
merging fix/odc-metrics-semantics (which landed the cache.rs half).
ODC-18 (backlog#1123): the singleflight non-leader path now returns
JoinedInflightFill instead of counting as an insert, so N concurrent GETs
of one cold key record one insert, not N. This composes with the ODC-15
redesign: the non-leader does NOT wait for the leader (it already owns the
body and never re-serves from the fill result), so no blocking wait is
reintroduced. Drops the redundant local SkippedSingleflightBusy variant in
favor of the canonical JoinedInflightFill (mapped to no bytes / no duration
by the facade). SkippedFillConcurrency (ODC-11) is retained.
ODC-36 (backlog#1141): MokaBackend::invalidate_object now returns
Removed { keys } / NoOp instead of the transitional Success, so the facade
labels invalidations_total{outcome=removed|noop} and skips the cache-state
gauge refresh on the no-op path.
Tests: duplicate-fill test asserts JoinedInflightFill (bites when reverted
to Inserted); new no-op invalidation test; matching-identity test asserts
Removed { keys: 1 }.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): drop unused mut on materialize stream binding
The ODC-07 change moves `final_stream` into `AsyncReadExt::take`, so the
`build_get_object_body_with_cache` parameter is no longer mutated in place.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(object-data-cache): close the cross-branch coordination seams
Merging the metrics and hot-path branches left four transitional shims that
only existed because each side could not edit the other's files. With both
sides present they are dead weight, and two of them actively misreport.
- ObjectDataCacheEntry::new took content_length and etag only to keep the
caller in moka_backend.rs compiling, then discarded both. Drop them; the
entry is a Bytes wrapper and the caller now passes only the body.
- SkippedSingleflightClosed lost its only producer when the waiter model was
replaced by Leader/Busy election. Remove the variant.
- The fill-accounting match ended in a wildcard that charged fill_bytes and a
duration to every non-Inserted outcome, so a fill rejected by the memory
gate or the concurrency semaphore — which never touched the backend and
wrote nothing — still inflated fill_bytes_total. Enumerate every variant
explicitly: only outcomes that wrote the body report bytes and duration.
Exhaustiveness also forces a future variant to state its accounting rather
than inherit a wrong default.
- InvalidationResult::Success mapped a no-op to outcome=removed, the exact lie
backlog#1141 set out to fix, and its doc comment claimed MokaBackend still
returned it after that backend had been widened. It has no producer left.
Remove it, and tighten the app-layer test from "any successful variant" to
the real contract: the pre-mutation call reports Removed{keys:1}, the
post-delete call NoOp.
Refs: backlog#1123, backlog#1141, backlog#1135
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): make body-cache hook re-registrable + add e2e regressions
ODC-21 (backlog#1126): the GET body-cache hook lived in a first-wins
OnceLock. When AppContext is rebuilt (config reload, test re-init) a fresh
ObjectDataCacheAdapter is constructed and re-registered, but the OnceLock
kept ecstore's GET probe pointed at adapter #1 while every usecase-layer
fill and invalidation targeted adapter #2 — silently degrading the feature
to a 0% hit rate with no error, log, or metric, and stranding entries in the
unreachable cache until their TTL.
Replace the slot with RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> so
re-registration atomically swaps to the newest adapter, and log at WARN when
a swap replaces a *different* instance (Arc::ptr_eq). RwLock over
ArcSwapOption because arc-swap's RefCnt is impl<T> (Sized, thin *mut T) and
cannot hold an Arc<dyn Trait> without a sized newtype wrapper; the probe
reads the slot once per full-object GET but only clones an Arc, negligible
next to the metadata quorum fan-out already done before the probe. Add a
test-only clear_get_object_body_cache_hook so tests register/unregister
deterministically.
With the hook now re-registrable, add true end-to-end regressions that drive
get_object_reader (not the full_object_plaintext_len predicate) against a
real erasure-coded, genuinely-compressed object via the blackbox
make_local_set_disks harness, with a stand-in hook playing the app-layer
cache (the injection point production uses; the adapter itself lives above
ecstore). These close the gap the predicate-only tests left — a caller that
opens a new shortcut serving the cached body directly, the original form of
both P0s:
- backlog#1108: a raw_data_movement_read must yield the STORED (compressed)
bytes, never the cached plaintext.
- backlog#1109: a compressed cache hit must publish the DECOMPRESSED length
as object_info.size (the UploadPartCopy invariant), with the streamed
length matching.
- backlog#1146: a restore read (restore_request.days) must serve STORED
bytes, not the cache.
Mutation-verified each e2e test bites: dropping the raw_data_movement_read
gate serves plaintext (fails #1108); removing the hit-site size republication
publishes 2972 vs 660000 (fails #1109); dropping the restore gate serves
plaintext (fails #1146).
Co-authored-by: heihutu <heihutu@gmail.com>
CONTRIBUTING.md and the Makefile HEADER advertised nonexistent targets
(make clippy, make check) and claimed 'make pre-commit' runs
format + clippy + check + test, while .config/make/pre-commit.mak
actually runs fmt-check + guard scripts + quick-check with no clippy
and no tests. CONTRIBUTING.md also claimed the repository ships an
automated pre-commit git hook, but no hook is versioned.
- Replace phantom targets with the real ones (clippy-check,
quick-check, compilation-check)
- Document exactly what pre-commit and pre-pr run, gate by gate,
with an explicit note that pre-commit runs no clippy and no tests
- Point contributors to 'make pre-pr' as the full pre-PR gate
- Rewrite the git-hooks section to say hooks are optional and not
versioned; setup-hooks only marks an existing hook executable
No make targets were added or changed; docs/help truth only.
Refs: rustfs/backlog#1153 (infra-9), rustfs/backlog#1155
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
The weekly full ceph/s3-tests sweep (e2e-s3tests.yml, schedule path) has
failed every recorded run. The 2026-07-05 run (rustfs/rustfs #28727691591)
died at "Install Python tools" with `No module named pip`: the self-hosted
`sm-standard-4` label is served by heterogeneous runners and the scheduled
job landed on a minimal pod with neither `pip` (bare python3) nor
`docker.sock`, so no test ever executed.
Make the infrastructure assumptions explicit instead of trusting drifting
self-hosted runner state:
- Pin the job to GitHub-hosted `ubuntu-latest`, which reliably ships Docker,
docker compose, python3 and pip.
- Add an explicit actions/setup-python step before any pip usage so the
workflow stays correct across runner-image drift.
- Document the fix and rationale in the workflow header comment.
Also quote the shell variables flagged by shellcheck (SC2086) in the
Docker build/run steps and drop the unused loop variable (SC2034) so
actionlint passes with zero findings on the changed file.
The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
via DEPLOY_MODE=binary and defers pip setup to run.sh's self-bootstrap.
Refs: rustfs/backlog#1149 (ci-1), rustfs/backlog#1155
Re-enable the Fuzz workflow (disabled_manually) with two changes that
address the congestion that likely caused it to be disabled:
- Narrow the PR trigger paths to fuzz/**, scripts/fuzz/** and
fuzz.yml only. The broad crate paths (crates/ecstore|filemeta|utils)
queued a ~45min fuzz-build on nearly every PR; coverage of those
crates moves to the nightly schedule, which fuzzes against main.
- Expand the smoke matrix, nightly matrix, run.sh default set and the
fuzz-build staging/upload steps from 3 to all 5 buildable targets:
archive_extract, bucket_validation, local_metadata, path_containment,
policy_ingress. archive_extract and policy_ingress were previously
in no matrix.
The two *_storage_api.rs files are `mod` submodules of their parent
targets (bucket_validation, path_containment), not standalone bins, so
fuzz/Cargo.toml registers exactly 5 fuzz bins.
Smoke jobs run one target per parallel matrix job (-max_total_time=60),
so wall-clock stays per-target, well under the 15min budget for a
fuzz-only PR. A TODO(ci-8) hook marks where scheduled-failure alerting
will attach on the nightly job.
Refs rustfs/backlog#1149 (ci-9, absorbed sec-13), rustfs/backlog#1155
Under feature sets where the dial9 session guard carries no Drop impl (e.g.
--features sftp, after #4663), `drop(dial9_guard)` in the startup entrypoint
trips clippy::drop_non_drop and fails the sftp Test and Lint job. The drop is
an intentional flush point where the guard does implement Drop; annotate it so
it compiles clean across all feature combinations.
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): io_uring reads must reclaim the page cache like StdBackend (backlog#1145)
`StdBackend::pread_bytes` calls `fadvise(DONTNEED)` over the range it just
read whenever `should_reclaim_file_cache_after_read(length)` holds —
`RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE` (on by default) above a 4 MiB
threshold. Large object reads are usually cold, and leaving them resident
evicts everything else, so this is a deliberate policy rather than a side
effect of how StdBackend happens to read.
`pread_uring` and `pread_uring_direct` never did it. Enabling io_uring on a
disk therefore turned the policy off silently: shard reads at or above the
threshold stayed resident and grew the page cache without bound. This is the
same class of mistake as the O_DIRECT loss fixed earlier — copying the read
itself while dropping the policy attached to it.
It also badly distorts any benchmark. An end-to-end warp GET A/B on a
16-core host (4 MiB objects, conc 64) showed io_uring at 8782 MiB/s against
StdBackend's 116 MiB/s. That is not a 76x speedup: with io_uring the run
issued *zero* device reads and grew the page cache, while StdBackend read
2950 MiB from the device and shrank it. The two legs were not running the
same policy. (Disabling the mmap read path changed nothing — 1.01x — so the
mmap-vs-pread difference was not the cause either.)
Both io_uring read paths now reclaim the range they read, with the same gate,
the same range, and the same error handling as StdBackend. O_DIRECT should
leave nothing resident anyway, but a filesystem that quietly buffered the read
still has to honour the policy, so `pread_uring_direct` reclaims too.
The test measures the policy, not the call: it reads an 8 MiB file through
each backend and asks `mincore(2)` how much stayed resident. Crucially it also
asserts the reclaim-disabled case leaves the range resident — without that
gate, a backend that never reclaimed would pass silently. Verified: with the
fix removed, the test fails with "uring: reclaim is on ... 2048/2048 pages
remain"; with the fix it passes.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 143 passed, 0 failed.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(object-data-cache): bound cache config and make memory container-aware
Harden the object data cache configuration path so a single bad input
degrades to the disabled adapter instead of OOM-killing pods, panicking
at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/
1127/1140/1130).
- ODC-05: resolve capacity and the fill gate from the effective
(container-aware) memory. Prefer sysinfo cgroup_limits() and use
min(host, cgroup) as the total plus cgroup-derived availability, falling
back to host values when no cgroup limit constrains. Log the resolved
capacity and its basis once at startup.
- ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded
Duration can no longer trip moka's ~1000-year builder assertion.
- ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so
MAX_ENTRY_BYTES can no longer inflate total capacity above the safety
clamp; reject an entry larger than the resolved capacity instead.
- ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the
weigher overhead so a fillable-but-unretainable cache is rejected.
- ODC-22: reject max_entry_bytes at/above the u32 weigher boundary.
- ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the
min(ttl, time_to_idle) expiry interaction.
- ODC-25: give numeric env overrides two-valued semantics via a new
rustfs_utils::get_env_parse_outcome; a malformed value now disables the
whole cache with one aggregated warning instead of silently keeping
defaults.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): require identity_keys_max of at least 2
The identity index admits a new key by evicting the oldest one, so a
budget of 1 evicts the previous key on every fill and can never hold two
live keys of one object at once — a versioned bucket alternating two
versions then hits a permanent 0% hit rate.
Reject the degenerate value at config validation time, where the adapter
turns it into a disabled cache with a warning, since the bounded-eviction
policy cannot rescue it at runtime.
Handed off from the identity-index batch (backlog#1128), which changed the
overflow policy but could not touch validate().
Refs: backlog#1115, backlog#1128
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): let a zero free-memory floor opt out of the gate
Making the memory gate container-aware turned it live in CI: the runners
are Kubernetes pods, so the gate now reads the pod's cgroup free memory,
finds it below the 20% floor, and refuses the fill. Tests that assert a
fill succeeds then failed on CI while passing on a developer host, which
has no cgroup and falls back to host memory.
The gate is behaving correctly — the tests were the ones depending on a
live memory reading. Treat min_free_memory_percent = 0 as a deliberate
opt-out rather than an invalid value: allows_fill returns early before it
touches any snapshot, so admission becomes independent of where the suite
runs. Operators gain the same escape hatch.
Every test that requires a fill to succeed now sets the floor to 0. The
tests that exercise the gate keep it enabled via memory_gated_config, so
the ODC-05 coverage they provide is preserved rather than short-circuited.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): opt the usecase fill tests out of the memory gate
The previous commit exempted the fill-dependent tests it could find, but
missed the six adapters built inside object_usecase.rs. Fixing the first
batch moved nextest's fail-fast point forward and CI surfaced them:
build_get_object_body_with_cache_materializes_once_and_hits_later and
..._uses_cached_body_without_reader_preread both assert a fill lands, and
both ran with the default 20% free-memory floor.
Set the floor to 0 on all six, matching the sibling test modules. Verified
by pinning the gate's snapshot to 0% available — harsher than any CI pod —
and confirming these tests still pass, which shows the exemption path never
reads memory at all.
An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the
tree now shows no remaining site: the only unexempted ones are
fill_enabled()'s matches! arm and two adapter tests that assert the adapter
is disabled and therefore never fill.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): run one io_uring ring per shard on each disk (backlog#1145)
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): cache part-file descriptors for io_uring reads (backlog#1145)
`pread_uring` opened the file on the blocking pool for every read, so each
read paid a `spawn_blocking` round trip — the very thread hop io_uring
exists to avoid. Sharding the driver (backlog#1145) removed the previous
ceiling and left this as the binding cost. Measured on a 16-core host with
a 4-shard driver, warm page cache:
64 KiB, conc 8: 143942 -> 263054 IOPS (+83%), p999 240 -> 65 us
64 KiB, conc 32: 150128 -> 204876 IOPS (+36%), p999 2508 -> 871 us
64 KiB, conc 128: 129172 -> 361287 IOPS (+180%), p999 15329 -> 3046 us
1 MiB, conc 32: 33875 -> 42301 IOPS (+25%)
At 64 KiB / conc 128 the open is what masked io_uring entirely: with it,
io_uring beat StdBackend by 3.5%; without it, by 189%.
Add a bounded per-disk descriptor cache used by the io_uring read path.
A hit takes no `open` and no `spawn_blocking`, so the read never leaves the
runtime worker.
Why caching a part-file descriptor is safe:
* only `<object>/<data_dir>/part.N` reaches this backend's `pread_bytes`;
`xl.meta` — the one path replaced in place — is read through `read_all`
/ `read_metadata` and never gets here;
* part files are never rewritten in place. A replacement is always
write-new-tmp then `rename`, which swaps the inode, so a cached
descriptor can never observe a torn shard.
Why invalidation is nevertheless REQUIRED: heal reuses the existing
version's `data_dir` and renames a rebuilt shard onto the SAME part path.
A cached descriptor would keep serving the pre-heal (corrupt) inode,
defeating the heal and eroding read quorum. `delete` likewise unlinks a
part that a cached descriptor would keep readable. So `rename_data`,
`rename_file`, and `delete` all call the new
`LocalIoBackend::invalidate_cached_fds` after they mutate, and a 5s TTL
bounds the blast radius should a future mutation path forget to.
Two preamble checks the miss path runs are not silently lost on a hit:
* bounds — the driver only short-reads at EOF (it resubmits otherwise),
so `bytes.len() != length` is exactly the old `meta.len() < end_offset`
check, and now yields the same `FileCorrupt`;
* volume access — skipped while an entry is live. An unreachable disk
keeps serving already-open descriptors for at most the TTL, after which
the re-open re-runs the check. Disk health is tracked independently of
this per-read probe.
Scope: buffered io_uring reads only. The O_DIRECT path keeps opening per
read (its reads are >= 4 MiB, so the open is a small fraction), and
StdBackend is untouched — it must take the blocking hop for the pread
regardless, so caching would buy it only 2-6% while carrying the same
invalidation risk. `RUSTFS_IO_URING_FD_CACHE=false` restores open-per-read.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 135 passed, 0 failed. The new
heal-staleness test first asserts a read still returns the PRE-heal bytes —
proving the cache is live and the hazard real — then that invalidation makes
the healed shard visible. A second test drives `rename_file` and `delete`
through `LocalDisk` to prove those paths actually invalidate, and a unit
test pins prefix invalidation to component boundaries (`a/b` must not drop
`a/bc`).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-data-cache): close identity-index races in fill/invalidate (backlog#1117/#1118/#1125/#1128/#1136)
The object body cache's fill-vs-invalidation contract (index.insert before
cache.insert, then re-check index membership and undo the fill on mismatch)
had several races where the identity index and the cache could drift, either
poisoning the race metric, breaking invalidation silently, or letting a
principal evict a hot body on demand.
ODC-12 (backlog#1117) — generation-aware index removal.
The eviction listener removed (identity, key) by key equality with no way to
tell an evicted old entry from a freshly refilled one under the same key.
Verified against moka 0.12.15: an insert over a TTL/TTI-expired but not yet
purged entry runs do_insert_with_hash inline during insert().await, and
do_post_update_steps awaits notify_upsert INLINE with cause Expired
(was_evicted() == true), so the listener deleted the index key the refill just
registered and the post-insert recheck then self-destructed the fresh entry
(SkippedInvalidationRace). A deferred Size notification for an old generation
had the same effect on a cleanly refilled key.
Chose Option A (generation token) over Option B (re-insert after
cache.contains_key): B removes the key first and only reconciles afterward,
which reopens a narrow window where a concurrent invalidate_object misses the
key while it is transiently absent from the index. A never removes a key whose
generation does not match, so the decision is atomic under the shard lock. The
token is the cache entry's Arc pointer captured at fill time (readable from the
value moka hands the listener), so no change to the entry type is needed.
ODC-13 (backlog#1118) — cancellation-safe insert/recheck/undo.
The recheck+undo sat after two awaits inside the S3 GET future; a client
disconnect could cancel the fill at the recheck await after cache.insert made
the entry visible, stranding a stale body with no index entry. The
register/insert/recheck/undo sequence now runs in a spawned task whose
JoinHandle the leader awaits: cancelling the await detaches the task, which
still finishes the recheck and undo. The dropped leader unblocks waiters as
before (SkippedSingleflightClosed), so no waiter is stranded.
ODC-20 (backlog#1125) — do not prune in-flight sibling keys.
prune_missing could remove another in-flight fill's index key (a different key
of the same identity that had registered but not yet published its cache
entry), making that fill's recheck fail and delete its own valid entry. Added a
read-only ObjectDataCacheSingleflight::has_inflight and consult it in the prune
predicate alongside cache.contains_key.
ODC-23 (backlog#1128) — bounded eviction instead of clear-and-reject.
Exceeding identity_keys_max drained the whole key set and rejected the incoming
key, so a principal able to GET identity_keys_max+1 distinct versions could
evict an object's hot body on demand, and identity_keys_max=1 on a versioned
bucket meant a permanent 0% hit rate. The key set now evicts only the oldest
key(s) (the Vec preserves insertion order) and inserts the new key. The insert
result carries evicted_keys so the backend removes exactly those from the cache
and still caches the newest body. The Overflow variant and the now-unused
SkippedIdentityOverflow construction are gone.
NOTE: the audit also asks for identity_keys_max >= 2 validation; that lives in
config.rs (out of scope here) and is deferred to the config batch.
ODC-31 (backlog#1136) — deterministic race coverage.
Added a #[cfg(test)] fill barrier between the index insert and the cache insert
so the fill-vs-invalidation recheck can be driven deterministically, plus a
companion test asserting the identity-budget eviction drops the evicted keys'
cache entries.
New tests:
- index: key_set_bounded_eviction_evicts_oldest_and_inserts_new,
key_set_removes_only_matching_generation_token,
key_set_duplicate_refreshes_generation_token
- starshard_index: identity_index_bounded_eviction_evicts_oldest,
identity_index_stale_eviction_preserves_refreshed_key,
identity_index_matching_eviction_removes_key
- moka_backend: moka_backend_refill_after_expiry_without_maintenance_inserts,
moka_backend_concurrent_fills_same_identity_both_insert,
moka_backend_identity_budget_evicts_oldest_and_caches_newest,
moka_backend_fill_loses_race_to_invalidation,
moka_backend_cancelled_fill_still_undoes_lost_race
cargo fmt --all, cargo check/clippy/test -p rustfs-object-data-cache all pass
(44 tests). Verified the refill-after-expiry and concurrent-fill tests fail
without their respective fixes.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(cache): gate GET body-cache hook to preserve ReadPlan output
The ecstore GET body-cache hook serves cached full-object plaintext
directly, bypassing ReadPlan/ReadTransform. That is only sound when the
normal read path returns that same plaintext byte-for-byte. Two probe
conditions were missing, plus two usecase-layer planner gaps.
ODC-01 (backlog#1108): raw/data-movement reads. ReadPlan::build returns
the STORED representation for raw_data_movement_read (e.g. compressed
bytes, length = oi.size), but the cache holds the post-decompression
body. Decommission (raw_data_movement_read: true) would receive
decompressed plaintext where raw compressed bytes are required, silently
corrupting the destination pool.
ODC-02 (backlog#1109): compressed objects. ReadTransform::Compressed
rewrites object_info.size to the decompressed length; on a hook hit
object_info is returned unchanged, so object_info.size is the compressed
size while the stream carries the decompressed body. UploadPartCopy then
uses src_info.size as the copy length and truncates the part.
Fix: gate the hook probe with should_probe_body_cache_hook, refusing
raw_data_movement_read, data_movement, and compressed objects, mirroring
the conditions get_small_object_direct_memory_decision already applies.
ODC-33 (backlog#1138): build_get_object_body_cache_plan lacked the
is_remote() exclusion the ecstore hook enforces; add it so transitioned
(remote-tier) objects are excluded uniformly.
ODC-C1 (backlog#1142): zero-length bodies save no I/O (ecstore returns an
empty body before the hook probe) yet the planner admitted them; change
the guard to response_content_length <= 0 so they plan Skip, mirroring
should_buffer_get_object_in_memory_with_threshold.
Tests: body_cache_hook_gate_tests (4) cover plain-probe plus
raw/data-movement/compressed skips; planner gains
plan_skips_remote_transitioned_objects and plan_skips_zero_length_objects.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(cache): allow compressed bodies via a fail-closed read allow-list
The body-cache hook probe was gated by a deny-list that refused compressed
objects outright, which cost the cache every compressed body — a growing
share of stored data. Replace it with an allow-list that returns the exact
plaintext length a hit may serve, or None.
full_object_plaintext_len() answers a single question: would the normal
ReadPlan produce this object's complete plaintext, and under which size?
Compressed objects now qualify, and the hit site publishes the returned
length as object_info.size, reproducing the contract ReadTransform::
Compressed establishes. A hit whose body length disagrees is refused and
falls through to the erasure read.
This also closes a gate the deny-list only covered by accident: a restore
read forces ReadPlan down the Plain branch, so a compressed object yields
STORED bytes under its compressed size. Refusing compressed objects hid
that; admitting them exposes it, so restore reads are refused explicitly.
Being fail-closed, a newly added ReadPlan branch bypasses the cache by
default rather than silently serving the wrong representation — the
structural defect behind both backlog#1108 and backlog#1109.
Refs: backlog#1108, backlog#1109
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-authored-by: heihutu <heihutu@gmail.com>
The listing path no longer has a wrapper-level total timeout, so any
filesystem call the walk makes without a stall bound would have no liveness
bound at all. The first pass covered list_dir and read_metadata but left four
reads unbounded: the volume access() probe and fs::metadata() in walk_dir, the
xl.meta read in its dir-object branch, and is_empty_dir() in scan_dir.
Split the helper so reads that do not yield a Result go through the same
budget, and route the four remaining calls through it. A hung drive now fails
those reads with DiskError::Timeout instead of parking the walk until the
merge consumer's own stall timeout aborts it.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads via native io_uring (rustfs/backlog#1102)
Wire ecstore's io_uring read backend to rustfs-uring's native aligned
O_DIRECT read (`read_at_direct`, merged in rustfs/uring#3), so an
O_DIRECT-eligible read keeps BOTH io_uring's async submission AND
O_DIRECT's page-cache bypass instead of trading one for the other.
`UringBackend::pread_bytes` now selects the read shape by a tiered,
per-disk-latched ladder — never a blanket downgrade:
1. io_uring latched off for this disk (backlog#1101) -> StdBackend.
2. O_DIRECT-eligible + native path usable -> pread_uring_direct: open
the file O_DIRECT, probe the device alignment once (cached), and
read via read_at_direct. Best path: async + no cache pollution.
3. O_DIRECT-eligible but the fs refused O_DIRECT earlier (latched)
-> StdBackend aligned path (which itself degrades to buffered).
4. non-O_DIRECT read -> buffered pread_uring.
Latching keeps a failure from being re-attempted per-read: an fs that
refuses O_DIRECT (EINVAL/EOPNOTSUPP on open) latches the native direct
path off for that disk; a restriction-class errno from the read latches
io_uring off wholesale (backlog#1101). Any other error falls back for
that one read without masking a genuine data problem as a permanent
downgrade. The alignment comes from the existing probe_direct_io_align
(statx STATX_DIOALIGN, default 4096) and is cached in a per-disk
OnceLock so the probe runs at most once.
This replaces the interim guard that routed every O_DIRECT-eligible read
to StdBackend (which disabled io_uring on the hottest shard reads); the
native path is now the default and StdBackend is only a fallback tier.
Bumps the rustfs-uring pin to the merged #3 commit.
Verified on a real Linux host (16-core, real io_uring + O_DIRECT):
cargo check --tests, clippy -D warnings, and disk::local tests all pass
(128 passed, 0 failed), including uring_preserves_o_direct_for_eligible_reads
across unaligned ranges.
Co-authored-by: heihutu <heihutu@gmail.com>
Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.
WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.
Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.
Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.
Fixes#4644
Refs #2999, #3001
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads when io_uring is enabled (rustfs/backlog#1102)
UringBackend::pread_uring opens the file buffered, so with io_uring enabled an
O_DIRECT-eligible read (RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE + length >= the
threshold) silently lost O_DIRECT and polluted the page cache — a behavior
change the moment io_uring was turned on for a disk that uses O_DIRECT. The
io_uring backend is gray-off by default, so no deployment is affected, but the
gap had to close before it can be enabled anywhere O_DIRECT is in use.
Route O_DIRECT-eligible reads back through StdBackend's aligned path (which
itself falls back to buffered when the filesystem rejects O_DIRECT). A native
aligned O_DIRECT read in the driver remains part of rustfs/backlog#1102.
Adds uring_preserves_o_direct_for_eligible_reads: with BOTH flags on and the
threshold at 1, unaligned ranges still return exactly the requested bytes.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
test(ecstore): cover the io_uring per-disk probe cache hit (rustfs/backlog#1101)
Follow-up to #4635. The per-disk negative probe cache was implemented but had
no dedicated test. Add uring_probe_cache_skips_known_unsupported_disk: a root
recorded in URING_UNSUPPORTED_DISKS makes try_new return None without a fresh
probe. Completes the #1101 acceptance ("cache hit does not re-probe").
Co-authored-by: heihutu <heihutu@gmail.com>
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the
latest main to pull the newest compatible versions.
Manifest bumps (Cargo.toml):
- md5 0.8.0 -> 0.8.1
- regex 1.12.4 -> 1.13.0
- sysinfo 0.39.5 -> 0.39.6
Lockfile-only bumps: der 0.8.1, dial9-tokio-telemetry 0.3.14, lru 0.18.1,
regex-automata 0.4.15, symbolic-common/symbolic-demangle 13.9.0, and
zlib-rs 0.6.6. Resolver re-selection moves crypto-common 0.1.7 -> 0.1.6
and generic-array 0.14.7 -> 0.14.9 (both semver-compatible). `ratelimit`
is held at 0.10.1 as requested; `pollster` 1.0.0 is an incompatible major
bump and is intentionally not taken.
Verification: cargo check --workspace --all-targets, cargo clippy
--all-features -D warnings, cargo nextest run --all --exclude e2e_test
(8467 passed), cargo test --all --doc, and cargo deny check
advisories/sources/bans/licenses all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
The startup topology convergence auto-detection (added in #4631) checked
Kubernetes before endpoint style, so a local path deployment (single or
multi-drive, non-distributed) running inside Kubernetes was classified as
orchestrated with an effectively unbounded wait window. Path-style
endpoints have no hostnames to resolve, so the wait never actually
triggers and runtime behavior is unchanged, but the startup log then
advertised mode=orchestrated / wait_timeout=MAX for a purely local
deployment, which is misleading and contradicts the documented policy
(local path endpoints -> fail-fast).
Order the auto-detection by endpoint style first: only distributed URL
endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or
bounded (otherwise); local path endpoints stay fail-fast regardless of
the platform. Update the doc comment to match and add a policy case
locking Kubernetes + local path to fail-fast.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): io_uring runtime degradation latch + per-disk probe cache (rustfs/backlog#1101)
Follow-up to #1104. Add the runtime half of the io_uring degradation contract:
- Runtime latch: UringBackend gains an `active` flag (mirrors
DirectIoReadState::supported). A read that returns a restriction-class errno
(EPERM/EACCES/ENOSYS/EOPNOTSUPP) — io_uring became unusable on this disk —
trips the latch, and every further read goes straight to StdBackend with no
more per-read io_uring attempts. Data errors (EIO), missing files, and
parameter errors do NOT latch, since StdBackend would hit them too.
- Per-disk probe cache: a process-wide negative cache of disks whose probe
failed, so try_new skips re-probing (creating a ring + driver thread) on
reconnect.
Tests: the errno classifier (restriction-only) and a latched-off read that
still returns correct bytes via StdBackend; the #1104 differential test still
passes under real io_uring.
Co-authored-by: heihutu <heihutu@gmail.com>
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
* feat(embedded): allow multiple embedded servers to coexist in one process
backlog#1052 S5: turn the embedded startup guard into a sequential lock
and make every write-once shared cell tolerate a second embedded start.
Two RustFSServers on different ports and volumes now start, run, and
shut down independently in the same process.
- EMBEDDED_SERVER_STARTED is released once each startup hands off; a
second startup that runs after the first is no longer rejected.
- The second embedded server constructs its own InstanceContext instead
of adopting the process bootstrap one, so region/endpoints/deployment
id land on that context (the first server keeps adopting bootstrap to
keep single-instance ambient facades unchanged).
- Startup-time tolerant paths:
- action_credentials publish treats AlreadyInitialized as success
(per-server ActionCredentialHandle already holds the real creds).
- GLOBAL_RUSTFS_PORT warns instead of panicking on a second set.
- Observability install returns Ok when the process subscriber is
already set — the second server reuses it.
- New acceptance test proves two servers start, respond on their own
ports, and one can be shut down without disturbing the other; the
survivor keeps serving S3 requests. IAM and root-credential lookup
still share a process domain (a second server whose creds differ from
the first will fail signature validation), tracked as a follow-up.
- Embedded doc rewritten: 'Limitations' → 'Multi-instance status'; the
AlreadyStarted error is now scoped to concurrent startups only.
The remaining work in #1052 is the auth path per-server dispatch and the
matching data-plane routing so two servers with different credentials
serve independent buckets end-to-end.
* feat(app): per-server auth and application context for multiple embedded servers (#4633)
backlog#1052 S6: each embedded server now authenticates against — and
its request path resolves — its OWN application context, so two servers
with different root credentials each accept their own access key and
reject the other's.
- AppContext is per-server: ensure_startup_after_iam constructs a fresh
context around this server's store + IAM + KMS and installs it into the
server's own ServerContextSlot, then publishes it as the process
default first-writer-wins (publish_global_app_context) for legacy
ambient readers. The old 'reuse the global if present' path is gone.
- FS::check (the S3 data-plane access gate) resolves auth against
self.server_ctx's context: check_key_valid gains a _with_context
variant that takes the root credentials and IAM system from an explicit
context (None = ambient, unchanged for all 140+ existing callers). The
region and the server context slot are published into the request
extensions for downstream handlers.
- Each embedded server seeds its own root credentials into its context
(ActionCredentialHandle.publish) at startup, so credential validation
no longer falls back to the first server's process-global identity.
- The bucket/object/multipart use-cases resolve their store from the
server's context (bucket_usecase_for/object_usecase_for/... take &FS).
New acceptance test: two servers with distinct credentials each
authenticate with their own key and reject the other's.
KNOWN FOLLOW-UP: full bucket-namespace isolation still requires threading
the instance context through the lower ecstore data plane (peer_sys /
disk registry / bucket-metadata reads still resolve via the process
GLOBAL_OBJECT_API), so the two servers do not yet present independent
bucket listings even though each holds its own store. That deeper pass —
a continuation of the #939 object-graph ctx threading — is the remaining
work on #1052.
Stacked on the S5 guard change.
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
The io_uring cancel-safety spike (imported in #4625) has been migrated to a
dedicated repository, https://github.com/rustfs/uring (crate `rustfs-uring`),
with the full per-issue commit history preserved.
The standalone repo has independent CI that this workspace cannot run: it
excludes the crate from the build graph, so its checks passed without ever
building or testing it. rustfs/uring CI is green on fmt + clippy, a native leg
that runs the suite with real io_uring and fails on any skip, and a docker leg
exercising both the graceful-degradation and real-io_uring paths.
Removing the in-tree copy keeps rustfs/rustfs clean; the crate will be wired in
later as a git dependency behind runtime probing (rustfs/backlog#1051).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fifth slice (also part of what the plan called S4).
ActionCredentialHandle was a unit struct forwarding get to the process
singleton (rustfs_credentials' GLOBAL_ACTIVE_CRED), so every context
served the same credentials regardless of which server it belonged to.
The handle now keeps its own copy and gains publish():
- publish() lands on the owning context and tries to publish the process
global; readers prefer the owned copy and fall back to the global while
the initial startup publish still lands there — ambient callers (iam
token signing, request-signature validation) keep working.
- Single instance: both cells are written together, so reads are
unchanged; two contexts that both published stay isolated even though
the global remembers only the first (covered by a new test).
- The publish return signals "took effect": true if this handle newly
holds credentials OR the global was newly initialized, matching the
pre-existing init_global_action_credentials fail-fast contract, now
scoped to the handle.
The interface trait gains a default-body-friendly signature; the test
double implements it as a no-op. The startup publish path (S4) still
calls init_global_action_credentials directly; wiring it through the
handle is a follow-up.
* chore(experiments): import io_uring cancel-safety spike as audit baseline (backlog#894)
Import the Spike 0 io_uring cancel-safety prototype from the closed PR #4381
branch (houseme/p2-spike0-uring-cancel-safety) as the baseline for the
backlog#1051 audit remediation. This crate is a standalone workspace and is
deliberately kept out of the main Cargo.lock/build graph (NOT production code).
Subsequent commits apply the fixes tracked in backlog#1051 sub-issues, one
commit per issue.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): drain probe SQE to its CQE before releasing buffer (rustfs/backlog#1053)
The probe path had no pending-table backstop: after pushing the read SQE,
any early return (`submit_and_wait` error, missing CQE) dropped the probe
buffer and file while the read could still be in flight in io-wq, and the
caller dropped/unmapped the ring on the error path. If the kernel then wrote
the 512-byte result into that freed heap block, it was a use-after-free — the
exact bug class this spike exists to prevent, living in its own probe path.
Fix: once the SQE is pushed, drain to its CQE via `drain_probe_cqe`, retrying
the WAIT on EINTR without re-pushing (the kernel consumed the SQE atomically
before the wait). A bounded attempt count prevents a probe against a hung
device from blocking forever; on any drain failure the buffer (and file) are
`mem::forget`-ed ("leak over UAF") so the kernel can never write into freed
memory. Unmapping the ring on its own is safe; only the user buffer must
survive.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): split probe/runtime/transient errno classes, guard offset (rustfs/backlog#1059)
`is_expected_restriction` folded EINVAL into the "environment restricted"
class, but at runtime EINVAL is triple-meaning — offset > i64::MAX (signed
loff_t), O_DIRECT misalignment, and setup entries over the cap. Implementing
the rustfs/backlog#1048 permanent-degradation latch literally against this
class would fault a healthy disk off io_uring on one alignment retry or
offset-arithmetic bug.
Document that the class is probe-time ONLY and that P2 must split errnos into
probe-restriction / runtime-parameter-error / transient (EINTR/EAGAIN). Add a
concrete guard: `submit` rejects offset > i64::MAX with an InvalidInput error
instead of letting it reach the kernel as a runtime EINVAL. The probe EINTR
half of this issue is already handled by the drain loop from rustfs/backlog#1053
(retry the wait, never re-push).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): abort on driver-thread panic instead of freeing in-flight buffers (rustfs/backlog#1054)
The ownership model's "CQE is the only reclamation point" invariant held only
while the driver thread never unwound. On a panic inside drive(), Rust drop
order freed the `pending` table (every in-flight buffer) before the ring,
while the kernel could still be writing into those buffers → mass UAF.
`catch_unwind` cannot fix this: the destructors run during the unwind, before
the catch boundary.
Move ring + pending + backlog into a `DriverState` whose `Drop` checks
`thread::panicking()` and calls `process::abort()` BEFORE any field destructor
runs — leaving the ring mapped and the buffers allocated (leak over UAF). The
capacity-overflow panic that made this reachable (caller-controlled `len`) is
closed at the source in the len-guard commit (rustfs/backlog#1057).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): reject reads above MAX_RW_COUNT to stop u32 truncation (rustfs/backlog#1057)
The SQE length field is `len as u32`, so len == 4 GiB became a 0-byte read the
kernel answered with res=0 → an Ok(empty) the caller decodes as a false EOF
(and len > 4 GiB read only the low 32 bits). Silent truncation (CWE-197),
forbidden by the repo's rust-code-quality rules.
`submit` now rejects len > MAX_RW_COUNT (2 GiB - 4 KiB) with InvalidInput; the
`len as u32` cast in the driver is consequently lossless. This also closes the
caller-controlled capacity-overflow panic feeding rustfs/backlog#1054. P2 must
chunk reads larger than the cap.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): resubmit short reads to satisfy the whole-range contract (rustfs/backlog#1058)
CQE res >= 0 was truncated and delivered as final with no resubmit loop, and
Pending did not even store the requested length. io_uring can legally
short-read a regular file (io-wq signal interruption, NOWAIT partial page
cache, O_DIRECT tail blocks), while LocalIoBackend::pread_bytes is a whole-
range contract — a short shard fed to EC bitrot verification surfaces as
intermittent, hard-to-attribute integrity/quorum errors.
Track offset/nread in Pending and drive a resubmit loop: a short non-EOF read
re-queues the remainder into buf[nread..], keeping the entry (and its buffer,
and in_flight) until the FINAL CQE of the logical read; res == 0 is treated as
a real EOF. The resubmitted SQE reuses the op's user_data, so a late
ASYNC_CANCEL from a dropped future still cancels the logical read cleanly.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): bound the shutdown drain and record cancel outcomes (rustfs/backlog#1055)
Shutdown made "drain to in_flight == 0" a hard precondition for unmapping the
ring, but ASYNC_CANCEL is best-effort: it cannot interrupt a regular-file read
already executing in io-wq, so on a D-state/NFS-hung disk the CQE may never
arrive and drain-to-zero never terminates — the driver loops forever and
shutdown()/Drop join blocks the caller (and any tokio worker) permanently.
This is an internal contradiction (safe unmap needs drain; a bad disk makes
drain unbounded) in the very environment io_uring exists to handle.
Add a bounded-drain escape hatch: after DRAIN_TIMEOUT with ops still in flight,
leak the whole DriverState (ring stays mapped, buffers stay allocated — leak
over UAF) and exit so shutdown() returns. Soften shutdown()'s hard assert to a
warning for that degraded path; clean-drain tests still assert in_flight == 0
themselves. Also record the ASYNC_CANCEL three-state result
(succeeded/not-found/already-executing) so the hung-disk signal is observable
instead of discarded.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): assert NODROP, monitor CQ overflow, handle EBUSY (rustfs/backlog#1056)
In-flight had no upper bound and could exceed CQ capacity (entries=64 → CQ=128)
with zero overflow handling: no NODROP check, no overflow read, no EBUSY
handling. A lost CQE means its pending entry is never reclaimed, drain never
completes and shutdown hangs — and the spike only avoided this by accidental
reliance on the io-uring crate's auto-flush + NODROP kernel + poll cadence,
all of which P2's eventfd/AsyncFd reaping removes.
Assert the NODROP feature at probe (degrade via ENOSYS otherwise), monitor the
kernel CQ-overflow counter each turn and surface a non-zero value as fatal, and
handle submit() EBUSY as CQ-overflow backpressure (keep the backlog, reap this
turn) instead of swallowing it. The hard in-flight bound (permits ≤ CQ capacity)
lands with the backpressure work in rustfs/backlog#1060.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): add backpressure with permits released at the CQE (rustfs/backlog#1060)
Submission was unbounded (unbounded mpsc + uncapped pending/backlog), so a
concurrent large-object read storm had no memory ceiling. The subtler trap:
the planned SQ-depth semaphore, implemented the natural RAII way (permit held
by the ReadHandle/future), would release permits at future drop while orphan
buffers stay resident in the pending table awaiting slow-disk CQEs —
decoupling the permit count from resident memory and reopening the DoS surface
exactly in the EC quorum-drop hot path.
Add a `Backpressure` semaphore sized to the SQ depth (entries < CQ capacity, so
CQ overflow is structurally unreachable). `submit` acquires before handing the
op to the driver; the driver releases the permit at the CQE (pending-table
removal), NOT at future drop, tying the in-flight/memory bound to actual kernel
residency. Permits are balanced on the shutting-down reject and send-failure
paths, and the driver wakes all waiters on exit.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): open the probe file via O_TMPFILE instead of a predictable path (rustfs/backlog#1061)
The probe wrote a predictable temp path (uring-spike-probe-{pid}-{seq}) with
std::fs::write (O_CREAT|O_TRUNC, no O_EXCL/O_NOFOLLOW): a local attacker could
pre-plant a symlink there and have the process — often root — truncate and
overwrite an arbitrary target (CWE-59/377), with a TOCTOU window between write
and open (CWE-367). This probe is the direct blueprint for P2's per-disk
startup probe, so copied verbatim it becomes a production vulnerability.
Open via O_TMPFILE (anonymous inode, no name → nothing to plant a symlink at,
no TOCTOU, no leftover), falling back to O_CREAT|O_EXCL|O_NOFOLLOW + 0600 +
per-process nonce + immediate unlink on filesystems without O_TMPFILE. P2's
per-disk probe should create inside the tested data-disk directory the same
way, which also validates that disk's filesystem + io_uring combination.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(experiments/uring): correct invariant 2 mechanism, add invariants 6/7/8 (rustfs/backlog#1063)
Invariant 2 (the spike's flagship finding) mis-described the fd-reuse hazard:
it claimed the danger window is submission→CQE and that the kernel would
"write into someone else's file". Both are wrong — a submitted op holds a
struct file reference and is immune to fd close/reuse; the real window is SQE
construction (as_raw_fd) → io_uring_enter (backlog residency), and for a READ
the consequence is reading the WRONG file, not writing. A P2 optimization
reasoning from the false premise (drop Arc<File> after submit / registered-file
table) would step straight into it.
Correct the mechanism and add the invariants this audit hardened: driver-thread
unwind safety (6), backpressure permit released at the CQE (7), reused-buffer
content hygiene (8, detailed in rustfs/backlog#1062), plus the errno three-class
contract, bounded-drain escape hatch, and short-read resubmit responsibility.
Mark the now-remediated items in the leftover list.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(experiments/uring): pin the reused-buffer content-hygiene invariant for P3 (rustfs/backlog#1062)
The spike leaks nothing today (fresh zeroed buffer per op + truncate to res),
but rustfs/backlog#1048's P3 constraint mandates a driver-owned aligned slab
whose buffers are reused across requests as dirty memory. Both the SPIKE
invariants and the #1048 constraint address only buffer LIFETIME (UAF), not
content hygiene: once buffers are reused, any path that forgets to bound the
caller-visible bytes to cqe.res (O_DIRECT full-block read sliced upstream,
error path returning the whole buffer) discloses a previous tenant's object
data (CWE-226) in an S3 store.
Pin invariant 8: reused-buffer bytes visible to the caller must be strictly
⊆ [0, cqe.res). Documented in SPIKE.md and marked at the delivery point in the
driver so P3 preserves it; needs a dirty-buffer + short-read regression test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): pin fd ownership and orphan-integrity directly (rustfs/backlog#1064)
The memory-safety assertions were all counter proxies, and invariant 2 (fd
owned by the pending table) had zero coverage — deleting Pending.file compiled
and left every test green because each test kept its own Arc<File> alive.
Add two direct observations: pending_table_owns_fd_after_caller_drop drops the
caller's Arc while the op is in flight and asserts F_GETFD still succeeds (only
the pending table's clone keeps the fd open; removing that field would close it
→ EBADF). orphan_in_flight_does_not_corrupt_delivered_reads keeps an orphaned
buffer in flight while 64 delivered reads must return byte-exact, asserting the
orphan buffer is not reclaimed early and its kernel writes corrupt nothing. A
driver-level poison/canary leg is noted as a P2 acceptance-matrix item (ASAN
cannot see a kernel write into a freed buffer).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): cover CQ-overflow safety and read boundaries (rustfs/backlog#1065)
The suite never approached CQ capacity and never touched EOF/len boundaries.
Add no_cq_overflow_under_load (300 ops through a CQ of 128 with backpressure
capping in-flight at 64, asserting cq_overflow stays 0 and all deliver),
boundary_reads (len==0, read at EOF, a cross-EOF short read delivered to a
live receiver exercising the positioned resubmit path, and the rejected
huge-len/huge-offset guards), and pipe_half_close_reads_eof (a closed write
end surfaces res==0 EOF).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): cover Drop-without-shutdown and de-flake cancel_stress (rustfs/backlog#1066)
All tests ended via explicit shutdown(), so the UringDriver Drop impl's
live-thread branch (send Shutdown before join) was never exercised; add
drop_without_shutdown_drains_and_cancels which drops the driver with ops in
flight and asserts the held futures resolve to ECANCELED (a join-first
regression or unbounded hang makes it hang).
Also de-flake cancel_stress: the exact assert delivered == OPS/2 raced the
driver — an even-i read can complete between read_at returning and drop(handle),
delivering to the still-live receiver and flipping the split. Relax to the
deterministic conservation identity plus delivered >= OPS/2.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): make run-docker.sh assert each leg's real path (rustfs/backlog#1067)
Both legs ran the identical cargo test and checked only the exit code, and a
skip is indistinguishable from a real pass at that level: leg 1 depended on the
host Docker's default seccomp "usually" blocking io_uring, and leg 2 printed
"both legs passed" even if every test skipped (vacuous pass — zero real
io_uring coverage).
Add an explicit seccomp profile (seccomp-block-uring.json) that returns EPERM
for io_uring_setup/enter/register so leg 1 deterministically hits the
graceful-degradation path regardless of host defaults, and assert leg 1
actually degraded (SKIP lines present) while leg 2 did NOT skip a single test
(io_uring really ran). Either violation now fails the harness.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(experiments/uring): apply repo rustfmt (max_width=130) to the audit changes
Normalize the formatting of the remediation code to the repo rustfmt.toml.
Pure formatting; no behavior change. clippy --all-targets -D warnings is clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fourth slice. The AppContext's IamHandle already owned an
Arc<IamSys>, but the value it held was read back from the process
singleton after init — so a future second server's context would silently
bind to the first server's IAM domain (credentials, policies, users all
belong to a specific store's .rustfs.sys).
- rustfs_iam gains build_iam_sys(store): construct an IAM system bound to
a store without touching the singleton. init_iam_sys wraps it (build,
publish first-wins, return the handle — previously Result<()>).
- Startup threads the built handle: the inline bootstrap path passes the
Arc it just created into ensure_startup_after_iam, which constructs the
AppContext from the passed value instead of re-resolving the global.
The deferred-recovery path resolves the freshly published global
directly (context-first resolution cannot be used there: the AppContext
does not exist yet — creating it is the finalizer's job; caught by the
embedded deferred-IAM e2e).
- IamHandle::is_ready() reports the held system's readiness instead of
consulting the process singleton; the dead global re-resolver is
removed. token_signing_key stays on the credentials global until S4.
157 iam tests + embedded e2e (basic + deferred recovery) green.
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(app): give each context's server-config handle its own copy
backlog#1052 S3, first slice: establish the per-context state pattern on
the smallest subsystem. ServerConfigHandle was a unit struct forwarding
both get and set to the process-global GLOBAL_SERVER_CONFIG, so every
AppContext shared one config regardless of which context a caller held.
The handle now keeps its own copy: a set lands on the owning context and
on the process global (ambient readers — the config loader path, the
scanner — keep working), and a get prefers the owned copy, falling back
to the global while the initial load still publishes there. Single
instance: the owned copy and the global are written together, so reads
are unchanged. Two contexts that both published stay isolated even though
the shared global only remembers the last write (covered by a new test).
The global write in set() is the transitional bridge for ambient readers;
it goes away when those readers migrate and multi-instance flips on
(backlog#1052 S5).
* refactor(notify): hand out the notification system as an owned Arc
backlog#1052 S3, second slice. GLOBAL_NOTIFICATION_SYS stored the
NotificationSys by value and every accessor returned
Option<&'static NotificationSys> — a process-lifetime borrow that a
per-server AppContext can never hold. The global now stores an Arc and
the accessor chain (ecstore runtime sources, ECStore::notification_system,
the iam forwarders, the rustfs shims, NotificationSystemInterface and the
AppContext resolvers) hands out owned Arcs instead.
Call sites are unchanged apart from two borrow adjustments in the
rebalance admin handler (pass &Arc where a reference is expected; borrow
with as_ref() so the handle survives to the second use). Behavior is
identical — same single global instance, first init still wins — but the
type now permits a future per-server context to own its notification
system (backlog#1052 S3 follow-up).
backlog#1052 S2, second slice: admin request dispatch.
Admin operations are registered as &'static dyn Operation, so unlike FS
they cannot carry per-server state. Instead, the (per-server) S3Router now
holds the server's ServerContextSlot and injects it into every request's
extensions at both dispatch points (check_access and call) — the same
extensions channel already used for ReqInfo/RemoteAddr.
- make_admin_route binds the slot; the HTTP builder passes the same slot
the S3 service (FS) uses, so both paths dispatch to the same server.
- admin/runtime_sources gains object_store_from_req / an extensions-based
variant for handlers that have already moved request fields out; both
fall back to the ambient process context when no slot was injected
(direct handler tests, non-router paths) — single-instance unchanged.
- 27 handler resolution sites across 12 files now resolve per-request;
helper functions without request access (11 sites: site_replication
state helpers, quota/table_catalog internals, object_zip_download
workers) stay on the ambient path until app subsystems become
per-server (backlog#1052 S3).
- One site (site-replication resync) resolves before the request body is
consumed, keeping the original error precedence.
New test: the router injects its slot into request extensions by pointer
identity. Embedded e2e (basic S3 + deferred-IAM recovery) still green.
Two follow-up fixes to #4607 (both verified real bugs, each with a regression
test):
1. The in-call server_info retry did not re-dial on network errors. A
network-like first failure runs through `finalize_result`, which sets the
client's `offline` gate; the retry then called `evict_connection` and
re-invoked `server_info`, but `get_client` short-circuits on that gate and
returns "temporarily offline" without dialing. Only the async recovery
monitor would clear it. So the retry re-dialed only in the timeout branch and
was a no-op in the transport/half-open branch it was meant to cover. Add
`PeerRestClient::prepare_retry` (evict + clear the offline gate) and use it
before the retry.
2. Synthesized/degraded drive lists went empty on hostname deployments.
`PeerRestClient::host` is an `XHost` that `hosts_sorted` builds via
`XHost::try_from` -> `to_socket_addrs`, so it is the resolved `IP:port`; but
`synthesized_disks`/`peer_disk_health` compared it against
`Endpoint::host_port()`, which is the raw `hostname:port`. On hostname
clusters the compare missed, the drive list came back empty, and
`unknownDisks` stayed 0 — reproducing the "drives vanish from the summary"
regression #4607 fixed (also affected the pre-existing offline path). Compare
through a shared `endpoint_host_matches` helper that canonicalizes the
endpoint side through the same `XHost` resolution.
Tests: `peer_rest_client_prepare_retry_clears_offline_gate`,
`endpoint_host_matches_direct_and_canonicalized` (uses localhost, no external
DNS).
Follow-up to rustfs/rustfs#4607; tracked in rustfs/backlog#1049.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): read the persisted SSE-C nonce in the GET decrypt resolver
#4576 moved SSE-C encryption to a random per-encryption nonce persisted
in object metadata and taught the rustfs-layer decrypt resolver to read
it back — but the byte-level GET decryption resolves its material in
crates/ecstore object_api/readers.rs, a duplicated twin that still
recomputed the deterministic legacy nonce. Encrypt with the random
nonce, decrypt with the deterministic one: the first AEAD block fails
and every SSE-C GET aborts its body after the response headers
(IncompleteRead(0 bytes) on clients; all 8 SSE-C cases in the
implemented s3-tests whitelist), which is what has been driving the
"S3 Implemented Tests" CI job into its 60-minute timeout since
2026-07-09.
Resolve the base nonce like the encrypt side: prefer the persisted IV
(x-rustfs-encryption-iv, then the case-insensitive MinIO interop key),
fall back to the deterministic nonce only for legacy objects with no
stored IV or an undecodable value. Full implemented s3-tests whitelist
is back to 451 passed / 0 failed against the fixed binary.
* feat(helm): support multiple server pools for capacity expansion
The server already understands multiple pools (RUSTFS_VOLUMES is split on
spaces, one pool expression each; rc admin pool/expand/rebalance/
decommission exist), but the chart could only render a single StatefulSet.
Add an opt-in pools mode:
- pools.enabled=false (default) renders byte-identical output to the
current chart (verified against five values variants)
- pools.list renders one StatefulSet per entry; entries may set
replicaCount (4 or 16) and a storageclass block, everything else is
inherited from the top-level values
- pool 0 keeps the legacy StatefulSet/pod/PVC names and its (immutable)
selector, so existing single-pool deployments can be expanded in place
without renaming or data loss; additional pools render as
<fullname>-poolN with a rustfs.com/pool label in their selector
- RUSTFS_VOLUMES and RUSTFS_SERVER_DOMAINS enumerate every pool
- pod anti-affinity is scoped per pool so pools may share nodes while
each pool still spreads its own pods across distinct nodes
- template validation fails fast on unsupported replicaCount, an empty
pools.list, or pools in standalone mode
- pools are append-only by design (list index = identity), documented in
values.yaml and the README
* fix(helm): truncate pool names DNS-safely, document PDB pool scope
Truncate <fullname>-poolN to 63 chars by shortening the base name rather
than the suffix, so the pool index always survives and two pools of a
max-length release cannot collide. Document that the single
PodDisruptionBudget deliberately spans all pools.
* fix(helm): pool-mode anti-affinity must be preferred, not required
Found by live-testing in-place expansion on a 5-node cluster (4-replica
pool 0 + 4-replica pool 1): with requiredDuringScheduling anti-affinity
the expansion deadlocks in a cycle that cannot self-heal -
1. the not-yet-rolled pool-0 pods still carry the unscoped required
rule and repel every rustfs pod from their nodes, so part of pool 1
stays Pending (no IP, no headless-DNS record);
2. every pod that already has the new RUSTFS_VOLUMES exits fatally on
'failed to lookup address information' for those Pending peers;
3. the StatefulSet rolling update is gated on the crashing pod going
Ready, so the remaining pool-0 pods never roll and keep repelling.
Because StatefulSets assign revisions per ordinal, even a subsequent
template fix cannot rescue a wedged rollout (Pending ordinals are only
recreated with the new template after the higher ordinals go Ready) -
the new pool's StatefulSet has to be recreated. Shipping the soft
affinity from the start avoids the state entirely; expansion was
re-tested end-to-end with it and converges.
Pool-mode rendering only - pools.enabled=false still renders the
existing required anti-affinity byte-identically.
---------
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
test(e2e): align object-lock overwrite tests with AWS new-version semantics
PR #3179 made put-like writes (PUT, CopyObject, multipart create/complete)
on versioned object-lock buckets create a new version instead of being
blocked by the current version's legal hold or COMPLIANCE retention, which
is the correct AWS behavior. Six e2e assertions in object_lock_test.rs
still encoded the old blocking semantics and, because e2e_test is excluded
from the CI test run, stayed red unnoticed since 2026-06-03.
Verified against a real server built from current main: all six failed at
their "should be blocked" assertions. Rewritten to assert the AWS
contract: the write succeeds with a distinct new version id, the new
content is current, the protected version keeps its content and lock
metadata, and version-targeted deletion of the protected version stays
blocked (for COMPLIANCE even with bypass_governance). Full object_lock
module is green again (33/33).
PR #4468 removed the last use of wildmatch (crates/notify swapped it for a
hand-rolled star-only glob matcher), but the workspace dependency declaration
in the root Cargo.toml was left behind as an orphan. cargo shear flags it as
not used by any workspace member. Drop the stray line.
Also refresh the comments in crates/notify/src/rules/pattern.rs that still
referred to the removed wildmatch crate, so they describe the current
hand-rolled matcher instead of a dependency that no longer exists.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(admin): report unreachable info members as `unknown` with balanced drive counts
`GET /rustfs/admin/v3/info` synthesizes an entry for every peer, but a peer
whose properties RPC failed below the offline threshold with no cached snapshot
was reported as `initializing` with an empty drive list. That drive list being
empty meant its drives disappeared from the backend summary entirely, so
`onlineDisks + offlineDisks` no longer equaled `totalDrivesPerSet` (an operator
sees `onlineDisks: 3, offlineDisks: 0` for a 4-drive set), and `initializing` is
a misleading state for a member that has been running for hours — it reads like
a node was ejected when nothing is wrong (upstream rustfs/rustfs#4566).
Changes:
- madmin: add `ItemState::Unknown` ("unknown") and an `unknownDisks` bucket on
`ErasureBackend` (serde `default`, so older payloads still deserialize).
- ecstore diagnostics: `get_online_offline_disks_stats` now returns a third
`unknown` bucket instead of folding those drives into `offline`, keeping
`online + offline + unknown == totalDrivesPerSet`.
- ecstore notification_sys: a probe miss below the failure threshold with no
cache is now reported as `unknown` (not `initializing`) and carries the host's
drives from the pool topology (tagged `unknown`) so the summary stays balanced;
drive synthesis is factored into `synthesized_disks(host, endpoints, state)`,
shared by the offline and unknown paths.
Tracked in rustfs/backlog#1049 (P1-A + P0-A).
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(utils): make DNS resolver cache TTL configurable via RUSTFS_DNS_CACHE_TTL_SECS
The resolver cache in crates/utils/src/net.rs had a hardcoded 300s TTL with
no way to tune or disable it. On orchestrated networks (Docker Swarm,
Kubernetes) a peer's DNS name is its only durable identity while its container
IP changes on reschedule, and the platform DNS always serves the current
answer. A fixed 300s application-side cache can keep a member dialing a dead IP
for up to five minutes after a peer restarts. Rust has no runtime DNS cache, so
on musl images this was the only cache in the stack and the only one that could
not be turned off.
Read the TTL from RUSTFS_DNS_CACHE_TTL_SECS (u64, default 300), matching the
existing RUSTFS_INTERNODE_* transport knobs. `0` disables caching entirely so
every get_host_ip consults the system resolver. The resolved value is logged
once at first use, removing an invisible cache from the triage surface. Change
is backward-compatible and confined to net.rs.
Also drop the two per-lookup info! logs on the resolve path: they would flood
the log when caching is disabled and added hot-path noise otherwise.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(helm): derive mTLS secret names from chart fullname (#4601)
* fix(helm): mtls certificate for server and client hardcode
* fix hardcode issue in deployment
* update typo yaml file
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
fix(ecstore): emit CommonPrefix for object with same-named prefix in non-recursive listings (backlog#1042)
On single-disk / consistent deployments a non-recursive scan_dir that finds a/xl.meta classifies `a` as an object and does not descend, so the source never emits the prefix `a/`, dropping the CommonPrefix from delimiter listings even when `a/b` exists. This is the single-disk follow-up to backlog#880 (the multi-disk merge path was fixed in #4563).
Fix: in the scan_dir Ok branch, for a non-recursive listing of a plain object, probe whether `a/` holds a listing entry other than the object itself (new object_prefix_has_sibling_listing_entry, reusing directory_has_visible_listing_entry and skipping the object's own xl.meta and data dirs); if so, additionally schedule the prefix dir `a/`. The upstream fold in from_meta_cache_entries_sorted_infos already emits object `a` as Contents and dir `a/` as a CommonPrefix, so nothing changes there. Leaf objects (the common case) pay a single list_dir and return false immediately.
Verification: new scan_dir unit test (positive: `a` + `a/b` coexist; negative: leaf `c` produces no spurious `c/`); new end-to-end e2e (object `a` + `a/b` with delimiter "/" -> Contents `a` + CommonPrefix `a/`); existing list_objects_duplicates e2e (#1797 / #2439 regressions) stays green; set_disk::read 114 passed.
Note: lib-test compilation depends on #4573 (now merged), which added the allow_inplace_legacy_fallback argument to the codec streaming arity tests after #4560 / backlog#879.
Co-authored-by: heihutu <heihutu@gmail.com>
The three store::bucket::tests::bucket_delete_* tests drive make_bucket /
delete_bucket through the process-global local-disk registry
(GLOBAL_LOCAL_DISK_MAP) and lock client, and through Sets::new which reads the
process-global erasure mode. Under `cargo test` (single process, many threads)
they race other global-touching tests and fail deterministically with
InsufficientWriteQuorum.
serial_test's #[serial] with the crate default key serializes them across the
in-process suite. Measured at --test-threads=16: baseline fails all three every
round; with #[serial] all three pass every round. nextest's per-test processes
are covered separately by the ecstore-serial-flaky test-group in
.config/nextest.toml (#4558). Full instance-level isolation remains a backlog
#939 (InstanceContext) concern and is not attempted here.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(rustfs): sanitize user-metadata in metadata=true object listing (#2743)
The console listing path (`list-type=2&metadata=true`) serializes each
object's user-metadata key as a raw XML *element name* and its value as
XML text without validation. User-metadata keys derived from HTTP
headers can legally contain characters that are illegal in an XML `Name`
(space, `$`, `%`, `#`, a leading digit, control bytes), and values can
contain C0 control characters that are illegal in XML 1.0 text.
A single object carrying such a key or value produced a malformed
`ListBucketResult` document, which the console's XML parser rejected
wholesale — so every object under that prefix vanished from the Web UI,
while plain `ListObjectsV2` (which never serializes user metadata) kept
working. The breakage appeared the moment any one object in a prefix had
XML-unsafe metadata and cleared once that object was removed, matching
the report.
Guard the serialization in `ObjectMetadataExtension::serialize_content`
(shared by the list-objects and list-versions metadata outputs): skip
entries whose key is not a valid XML element name, and strip
XML-1.0-illegal control characters from values. A single poison object
can no longer corrupt the whole listing document.
fix(ecstore): only evict remote lock channel on transport failures (#4567)
The remote lock RPC path evicted the cached gRPC channel on *any*
`tonic::Status`. Genuine transport failures (connect refused/reset,
keepalive/deadline) surface that way, but so do server-produced
application statuses on a perfectly healthy channel: `Unauthenticated`
/`PermissionDenied` from the signature interceptor, `Internal`
/`FailedPrecondition` when the peer lock service is not ready yet,
`InvalidArgument`, `ResourceExhausted`, `Unimplemented`, and so on.
Evicting the channel for those cannot help — the channel is fine — and
each spurious eviction re-dials the connection and advances the peer
toward the offline threshold via `record_peer_unreachable`, producing
background channel churn on an otherwise healthy cluster.
Classify the tonic status with `is_transport_failure` (underlying
transport source present, or one of `Unavailable`/`DeadlineExceeded`
/`Unknown`/`Cancelled`) and only evict the cached channel when it is a
real transport failure. The client-side timeout arm still evicts. This
mirrors the transport-vs-application distinction already used on the
remote-disk RPC path.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning +
http2_keep_alive_* in build_http_client) only takes effect when the connection
negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode
transport the connection is HTTP/1.1 and all the tuning is silently inert,
with no signal to the operator.
Make this honest without changing protocol behavior (no h2c, no
prior_knowledge, no opt-in flag):
- InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator
set a window size, a non-default tuning profile, or the keepalive env.
- Add pure predicate should_warn_h2_inert(negotiated_is_http2,
h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a
process-lifetime AtomicBool so the warn fires at most once.
- Call maybe_warn_h2_inert at both record_internode_http_version sites using
the actual negotiated resp.version().
- Document the TLS-ALPN requirement at apply_http_client_tuning and near the
ENV_INTERNODE_HTTP2_* constants.
Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit
reflects explicit windows and a non-default profile.
fix(server): count in-flight HTTP requests with an RAII guard (backlog#806-35)
The active-requests gauge was maintained with tower-http TraceLayer hooks:
on_request (+1), on_response (-1) and on_failure (-1). With the default
ServerErrorsAsFailures classifier a 5xx response fires BOTH on_response and
on_failure, so every 5xx decremented the gauge twice (net -1); a streaming
200 that failed mid-body did the same. The gauge drifted downward and
saturated at zero, corrupting the readiness busy-protection signal
(alias_busy_threshold_exceeded).
Replace the hook arithmetic with an InFlightGuard (RAII): InFlightLayer wraps
each request future, incrementing on entry and decrementing exactly once when
the future resolves (any status) or errors/cancels. Removed the +1 from both
on_request hooks and the -1 from trace_on_response and both on_failure hooks,
keeping their tracing and failure-metric work intact. Applied to both the
external and internode stacks.
Added a test driving the layer for 2xx, 5xx and a no-response service error,
asserting the gauge nets to zero in every case (the 5xx path is the fix).
fix(ecstore): age LastMinuteLatency samples individually (backlog#806-16)
The rolling one-minute latency window stored bare Duration samples plus a
single, never-updated start_time. Its retain predicate ignored the element
and evaluated the constant now.duration_since(start_time) < 60s, so once the
window had existed for 60s every add() dropped ALL samples and the average
degenerated to the latest single sample.
Each sample now carries its own Instant and is retained by its individual
age. The type backs only in-memory EpHealth + admin display (the wire shape
is target::LatencyStat with curr/avg/max), so Serialize/Deserialize is kept
only to satisfy the enclosing LatencyStat derive; the sample Instant is
serde(skip) and restarts aging on reload.
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
DeleteServiceAccount looked up the target service account and, on any
get_service_account error OTHER than not-found (e.g. a stored-credential decode
error), set svc_account = None and continued. The non-admin owner guard then
used svc_account.is_some_and(|v| v.parent_user != user), which is FALSE for a
None target, so the guard was skipped and the code fell through to the delete —
a non-owner could delete a service account whose stored credential failed to
decode (authz bypass, fail-open).
Extract a fail-closed helper non_admin_may_delete_service_account(caller, target)
that returns true only when the target loaded AND is owned by the caller; a None
target (unverifiable ownership) or an owner mismatch now denies. Admins (holding
RemoveServiceAccount) are unaffected. + unit test.
Refs rustfs/backlog#806
fix(rio): typed-first internode HTTP error classification (backlog#805)
classify_reqwest_error matched reqwest/hyper error text by English
substrings, brittle to library/OS-locale wording. Refactor to a pure,
testable classify_transport_error(err, is_timeout, is_connect, is_body)
that trusts structured signals first: caller timeout, then the io::Error
kind found anywhere in the source chain (typed wins over any string/body
signal so a real ConnectionRefused is never mislabeled DnsResolutionFailed),
then a DNS-only string heuristic gated behind is_connect, then body, then
Unknown.
Flip DnsResolutionFailed to retryable, mirroring MinIO IsNetworkOrHostDown
(*net.DNSError => network-or-host-down => retryable); RustFS already retries
transient DNS at endpoints.rs, and bounded retries cost at most one extra
attempt on permanent NXDOMAIN.
Swap the ecstore remote_disk non-retryable exemplar from DnsResolutionFailed
to Unknown accordingly.
Add a default-on multi-role adversarial validation policy to the root
AGENTS.md (risk tiers, six reviewer roles, findings protocol, exit
criteria), a shared adversarial-validation skill with repo-specific
attack probes grounded in shipped bugs, seven audit fixes to the root
instruction files, and an actionlint gate for workflow changes.
In the default (non-rio-v2) build, SSE-C encrypted with AES-256-GCM using a deterministic nonce derived from MD5("{bucket}-{key}"), and decrypt recomputed it. Overwriting the same object under the same SSE-C key reused an identical (key, nonce) pair, breaking AES-GCM confidentiality and integrity. The base nonce was never persisted.
Generate a fresh random 12-byte nonce for every single-PUT and multipart session, persist it under both x-rustfs-encryption-iv and the MinIO interop key, and resolve it from stored metadata on decrypt and multipart part encryption. Objects written before this change carry no stored IV and fall back to generate_ssec_nonce, so existing objects still decrypt.
Refs rustfs/backlog#1039
fix(object-capacity): harden clock handling in write window, background intervals and scope registry
Three low-severity robustness gaps (S32+S33+#35):
- WriteRecord keyed its 60-bucket write window by wall-clock unix
seconds while the debounce used Instant. An NTP step backwards marked
recent buckets as future and silently suppressed write-triggered
refreshes until the wall clock caught up; a forward step aged the
whole window at once. Bucket keys now derive from a monotonic
per-record epoch, immune to clock steps.
- Background refresh/metrics intervals were only clamped at zero;
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=u64::MAX overflowed
Instant + Duration and panicked the spawned task, silently killing
scheduled refreshes. Intervals now clamp into [1s, 30 days] via one
helper with a structured warn.
- record_capacity_scope merged new disks into an expired entry and
refreshed its recorded_at, resurrecting a scope take_capacity_scope
would have discarded (and the merge path never pruned). Expired
entries are now replaced, not merged into.
Ref: rustfs/backlog#1022 (S32+S33+#35 from audit rustfs/backlog#1010)
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once
rustfs_cluster_servers_offline_total is the count of distinct keys in the
addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by
endpoint.grid_host() (scheme://host:port) while the lock path keys it by
url::Url::to_string() (scheme://host:port/, trailing slash), so one physical
peer becomes two health entries and a single downed node reports 2 (2N for N).
Normalize the address (trim trailing slashes) at the map boundary in
record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/
cluster_peer_should_bypass so both forms collapse to one canonical key.
Pure observability fix; peer selection and quorum are unchanged.
Fixesrustfs/backlog#1043
Co-authored-by: heihutu <heihutu@gmail.com>
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
PR #4560 added a 15th parameter (allow_inplace_legacy_fallback) to
build_codec_streaming_part_reader while the negative arity tests from a
concurrently developed branch still call it with 14 arguments — the two
changes merged cleanly textually but broke the workspace test build on
main (E0061 in set_disk/read.rs), failing CI for every open PR.
Pass false at both call sites: these tests assert Err outcomes
(oversized part, missing quorum) that are independent of the fallback
mode, and false preserves the historical whole-request fallback
semantics used by eager first-part reads.
walkdir defaults to follow_root_links=true precisely so a root that is
itself a symlink still descends, but the scanner overrode it with the
general follow_symlinks flag (default false). For the common indirection
layout (/data/rustfs0 -> /mnt/vol0) the walk yielded a single skipped
symlink entry and returned used_bytes=0, file_count=0, is_estimated=false
with no error and no log — an exact 0 committed straight into the
per-disk baseline (S05).
Canonicalize the scan root before walking (also covering nested
indirection and keeping the dedicated-mount statvfs check on the real
path); resolution failures surface as scan errors instead of a silent
zero. Inner-symlink policy is unchanged.
Ref: rustfs/backlog#1015 (S05 from audit rustfs/backlog#1010)
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=0 made every file count as
overflow with an empty exact prefix, so disks with fewer files than the
sample rate committed 0 bytes as the cluster total; STAT_TIMEOUT=0 with
dynamic timeout disabled made every scan fail at the first entry. Both
were accepted unvalidated, unlike the already-clamped sample_rate and
interval values.
Zero values for the threshold and the stat/min/max timeouts now fall
back to their defaults with a single structured warn at config load
(S11). The symlink-depth knob is left untouched here: it is removed
entirely by the backlog#1018 fix.
Ref: rustfs/backlog#1019 (S11 from audit rustfs/backlog#1010)
Two ecstore test groups pass in isolation but flake under the full parallel
nextest suite, producing CI noise on every in-flight PR (backlog #937):
- store::bucket::tests::bucket_delete_* race make_bucket into
InsufficientWriteQuorum because they share global disk-registry/lock-client
state with other concurrently running tests.
- bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
exceeds its already-maxed 60s lock-acquire deadline only when the suite
saturates disk I/O.
serial_test's #[serial] does not serialize these across nextest's per-test
process boundary. Add a nextest test-group (max-threads = 1) that matches both
groups so they run serialized, removing the load-induced flake. No test or
production code changes; long-term global-singleton isolation is tracked
separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Progress/timeout checks only ran when file_count advanced, so trees of
directory-only or error-dense entries (dirs, traversal errors, symlinks
never increment file_count) bypassed the entire cooperative time budget
and held the blocking thread for unbounded time, while each error entry
logged its own warn — permission-dense trees flooded the log (S13).
The stall detector was structurally unreachable: it fired on 'no file
progress between checks', but checks themselves only ran when file
progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything
since its introduction (S09).
- Progress checks (timeout + early-sampling entry) are now driven by a
visited-entry counter that also advances on directories and errors, so
every tree shape reaches the budget checks.
- Remove the unreachable stall detector and its plumbing end to end
(ProgressMonitor fields, ScanLimits, config getter, env const, stall
metric). Genuine walker wedges are handled by the hard outer
wall-clock budget from backlog#1017; no decorative protection is left.
- Cap per-entry error warns at 10 per scan with an explicit suppression
notice; had_partial_errors still records the condition.
Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010)
perf(ecstore): wire legacy decode stripe prefetch / bitrot-decode overlap (backlog#930 HP-9 step 2)
The legacy GET decode duplex loop (default GET path + every Range request)
was strictly serial: read a stripe, reconstruct it, emit it, and only then
begin reading the next stripe. Two switches introduced by PR#3972 to overlap
this work — RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT and
RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE — were config-only with zero call
sites since introduction.
Wire both into the loop as a depth-1 stripe prefetch: while the current
stripe is reconstructed and emitted, the next stripe's shard reads (including
bitrot verification) run concurrently, hiding read latency under the emit /
duplex-backpressure stage. ParallelReader::read is inherently serial (it takes
&mut self and advances a shared stripe cursor), so at most one read can be in
flight; a prefetch count above 1 therefore collapses to the same
single-stripe-ahead pipeline rather than reading several stripes ahead. Both
switches feed one gate, legacy_stripe_prefetch_enabled().
Default (count == 1, overlap == false) keeps the loop on the pre-existing
strictly-serial path, byte for byte: the prefetch pipeline is a separate branch
and the serial branch is unchanged. The reconstruct/emit body is factored into
a shared emit_decoded_stripe helper used by both branches so error attribution,
read-quorum handling, reconstruction verification and stage metrics cannot
drift between them.
Correctness guarantees preserved under prefetch:
- A speculatively prefetched read for stripe N+1 is only consumed when the loop
reaches N+1; if stripe N stops the loop, the in-flight read is awaited and
dropped, so its errors never surface against stripe N.
- Bitrot (HighwayHash) verification runs inside each stripe read and is not
bypassed or reordered; a corrupt shard is still rejected and, when
unrecoverable, the read fails without emitting garbage.
- Shard buffers are recycled only after the overlapping next read has claimed
its own — one extra stripe of memory (double buffering) with buffer reuse
preserved at a one-stripe lag.
- Per-stripe exact length advance (backlog#799 B2), lockstep reconstruction
verification (backlog#832) and the hash_size == 0 pass-through are unchanged.
Adds regression tests exercising serial-default, count>1 and overlap configs:
full/range reads byte-exact, degraded (missing-shard) reconstruction, corrupt
shard rejected-but-recovered, unrecoverable corruption erroring with no output,
late-stripe failure attributed correctly with stripe 0 still emitted,
hash_size == 0 pass-through, and the gate defaulting off.
Co-authored-by: heihutu <heihutu@gmail.com>
Two report-only defects on the heal result-reporting surface (they do not
affect the data path or heal decisions):
- default_heal_result (set_disk/ops/heal.rs): the offline-disk branch pushed
an Offline record but fell through into the unconditional push, emitting a
second (Corrupt) record for the same disk. This grew before/after.drives to
disk_count + offline_count and misaligned every entry after the first offline
slot. Add `continue` after the offline push, drive `disk_len` and the loop
from a single `self.disks` snapshot, and assert `errs.len() == disk_len`.
- Sets::heal_format (core/sets.rs): the before/after drive lists were
pre-filled with N default placeholders and then N real entries were pushed,
yielding a 2N list whose healed status updates (indexed 0..N) landed on the
blank placeholder half. Assign the lists directly from formats_to_drives_info
(mirroring the set-level heal_format) so the healed updates hit the real
entries.
Add regression tests covering offline/online record alignment and the
NoHealRequired and heal paths of the pool-level heal_format.
Co-authored-by: heihutu <heihutu@gmail.com>
Sub-change A only: the two local-disk metadata read paths
(`read_metadata_with_dmtime`, `read_all_data_with_dmtime` in
crates/ecstore/src/disk/local.rs) previously dispatched open, fstat and the
xl.meta read as three separate async fs hops (three spawn_blocking round-trips
under tokio::fs). Each is now folded into a single tokio::task::spawn_blocking
closure using std::fs, cutting per-metadata-read dispatch from 3 to 1.
This does NOT touch sub-change B (removing the entry-point access() call).
Correctness first: this is the hottest metadata read path, so the refactor
preserves byte-for-byte-equivalent Results. Every error mapping is kept
identical:
- open failure -> to_file_error
- is_dir -> Error::FileNotFound (not to_file_error(EISDIR))
- metadata failure -> to_file_error
- xl.meta parse -> propagated verbatim from the parser
- try_reserve -> Error::other
- read_to_end -> to_file_error
For read_all_data_with_dmtime the async NotFound -> access(volume_dir) ->
VolumeNotFound fallback (and its warn! event) is preserved on the async side:
the closure returns the raw open error unmapped; only open() can yield ENOENT
once the fd is valid, so gating the fallback on the open error is equivalent to
the original open-arm-only fallback.
To run the parser inside a blocking closure, filemeta gains a synchronous twin
`read_xl_meta_no_data_sync` (+ `read_more_sync`) in
crates/filemeta/src/filemeta/version.rs that line-for-line mirrors the async
version, differing only in std vs tokio read_exact. A new equivalence test
(`read_xl_meta_sync_equivalence_tests`) feeds identical buffers to both the
async and sync readers and asserts equal Ok bytes / equal Err variants across:
v1.0; v1.1/v1.2/v1.3; large meta triggering read_more; header truncation ->
UnexpectedEof; CRC-trailer truncation -> FileCorrupt; unknown major/minor ->
InvalidData; and want boundaries (exact fit, inline-data drop, read_more EOF).
Co-authored-by: heihutu <heihutu@gmail.com>
Two disk-layer correctness fixes in crates/ecstore/src/disk/local.rs.
move_to_trash (#948, ECA-07) previously handled only DiskFull in its
tail error block and let every other rename failure (EIO/EACCES/ENOTDIR/
cross-device) fall through to `return Ok(())`, so a delete that never
landed on a faulty disk was reported as success and the drive fault was
hidden from heal/offline logic. It now keeps the DiskFull in-place-remove
fallback, treats a missing source (FileNotFound) as benign, and
propagates every other error (already mapped by to_file_error, e.g.
I/O -> FaultyDisk, permission -> FileAccessDenied), matching MinIO's
deleteFile.
rename_file and rename_part (#960, ECA-19) directory (trailing-slash)
branch unconditionally removed the destination before rename and
propagated the resulting NotFound, so renaming a directory to a new
location always failed. Both now tolerate ErrorKind::NotFound on that
pre-rename remove and continue, matching MinIO's RenameFile.
Adds regression tests covering benign-missing-source, real-error
propagation, the unchanged move happy path, and directory rename to a
missing destination for both rename_file and rename_part.
Co-authored-by: heihutu <heihutu@gmail.com>
An object transitioned to an unversioned remote tier legally records an
empty remote version_id (per CLAUDE.md: a tier version of `None`/`""`
means the tier bucket is unversioned, so remote GET/DELETE must omit the
versionId). Two recovery paths wrongly treated that empty sentinel as an
incomplete/unrecoverable record while the persist/encode and worker
paths accept it, producing permanent leaks in the crash-recovery window.
- tier_delete_journal::into_jentry rejected entries with an empty
version_id as "incomplete", so unversioned-tier WAL entries could never
be decoded during recovery: the remote object was orphaned and the
journal file leaked forever. Drop the version_id emptiness check; keep
the obj_name/tier_name checks. Truncated payloads are still rejected at
JSON deserialization (all fields are required, non-Option, no default,
with deny_unknown_fields).
- tier_free_version_recovery::is_recoverable_tier_free_version required a
non-empty version_id, silently filtering out free versions of
unversioned-tier objects so a first enqueue failure leaked the remote
object and local free version permanently. Drop the version_id check;
keep free_version + name + tier checks.
Both downstream paths already issue a versionless remote delete for an
empty version_id, so no further changes are needed. Adds regression
tests covering the empty-version_id case and the preserved guards.
Co-authored-by: heihutu <heihutu@gmail.com>
bitrot_shard_file_size only counts per-block checksum bytes for the two
streaming Highway variants, while BitrotWriter::write interleaves a hash
for any hash_algo.size() > 0 and bitrot_verify's read loop assumes an
interleaved hash per block. The three disagree for non-streaming
algorithms (SHA256/HighwayHash256/BLAKE2b512/Md5), but the divergence is
unreachable in production: every write path hardcodes HighwayHash256S and
ErasureInfo::get_checksum_info defaults to HighwayHash256S.
Per the audit decision (backlog#959), do NOT change the size formula:
it is a byte-for-byte port of MinIO's bitrotShardFileSize and its bare
return for non-streaming algorithms is correct for MinIO whole-file
bitrot; changing it would break legacy interop. Instead, document the
per-algorithm layout contract at bitrot_shard_file_size, BitrotWriter,
and bitrot_verify, and add regression tests that pin the invariants:
get_checksum_info defaults to HighwayHash256S, and the size formula
counts per-block hash bytes for streaming variants only while returning
the bare size for non-streaming ones. No disk layout or formula change.
Co-authored-by: heihutu <heihutu@gmail.com>
reduce_errs seeds best_err with a synthetic Error::other("nil") placeholder
via unwrap_or. When the error slice is empty or every error is ignored,
err_counts is empty and nil_count is 0, so both nil tie-break conditions are
false and the else branch returned (0, Some(Error::other("nil"))). That
placeholder leaked to build_write_quorum_failure_summary, where dominant_error
became Some("nil"), skipping the nil_dominated early return and falling back to
the "other_error" metric label. Quorum decisions were unaffected (max_count 0
still fails any positive quorum), but write-quorum failure metric labels and
tracing fields were misclassified.
Add an explicit short-circuit: when best_count == 0 and nil_count == 0, return
the (0, None) sentinel so the placeholder never escapes. Add regression tests
for the all-ignored slice, empty slice, and the summary label path, which the
existing test_build_write_quorum_failure_summary (nil_count > 0) did not cover.
Fixes#957
Co-authored-by: heihutu <heihutu@gmail.com>
Two dirty-mark lifecycle gaps (S19+S30):
- A commit cleared every dirty mark for the disks it scanned, even marks
recorded while the walker was already past the written prefix, so the
next dirty-subset refresh served stale cached bytes as exact until the
next full scan. Dirty marks now carry the instant they were last
recorded and a commit only clears marks that predate its scan start
(CapacityUpdate::scan_started_at, set by refresh_capacity_with_scope).
- The write side marks every disk of an EC set dirty, including remote
peers, while scans and clearing only cover local disks — remote or
removed disks stayed marked forever, keeping the dirty-disk gauge
permanently non-zero with bounded memory stuck behind it. The refresh
disk selection now prunes dirty entries outside the current local
topology before consulting them.
Ref: rustfs/backlog#1020 (S19+S30 from audit rustfs/backlog#1010)
complete_multipart_upload deleted every part.N.meta via
cleanup_multipart_path *before* the authoritative rename_data commit.
If rename_data then failed write quorum, the upload directory was left
with data files but no part metadata, so a retried CompleteMultipartUpload
could never read the parts again and the upload became permanently
uncompletable (client must abort and re-upload all parts).
Move cleanup_multipart_path to run only after rename_data returns Ok,
matching the existing "clean up only after commit" pattern already used
for the old data-dir GC and the upload-dir delete_all. On a failed commit
the staging part.N.meta now survive so the retry can complete.
Add a hermetic regression test that forces rename_data to fail on every
disk (destination bucket dir made read-only) and asserts the staging
part.N.meta are preserved for retry.
Fixes#946.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(io-core): decrement available_buffers gauge on pooled-buffer reuse (backlog#806)
return_buffer does a fetch_add(1) on push but take_or_allocate_buffer's
available.pop() reuse path had no matching fetch_sub, so the available_buffers
gauge only ever grew and never reflected the real pool size. Decrement it on
reuse; + regression test.
load_all fetched policies/users/user-policies by passing the FULL remaining
list to load_*_concurrent every iteration and then split_off(32), so each item
was re-fetched once per preceding chunk — O(n^2) redundant loads on startup for
>32 items. Replace the split_off loops with chunks(32) so each item is fetched
exactly once. Behavior-preserving (cache inserts were already idempotent).
Erasure heal set write_quorum to the count of available target writers,
so every replacement disk had to succeed writing every block or the whole
object heal aborted. A single flapping replacement disk failing one block
made MultiWriter::write return Err, heal propagate Err, and the ops layer
delete the entire tmp staging dir — leaving the other healthy replacement
disks unhealed and blocking redundancy recovery indefinitely.
Set the heal write_quorum to 1, matching upstream MinIO's writeQuorum=1
for heal. MultiWriter already marks a failed writer as None, and the ops
layer (set_disk/ops/heal.rs) already drops the failed writer while
committing the survivors; the read-side quorum check and parity
cross-checks that guarantee reconstruction correctness are unchanged.
Add regression tests: one healthy-and-one-failing target still heals the
healthy targets and drops the failing one; all-targets-failing still
returns Err so nothing is falsely committed.
Fixes#947
Co-authored-by: heihutu <heihutu@gmail.com>
Two defensive gaps left the refresh singleflight vulnerable to a
permanent wedge (running=true forever: scheduled refreshes silently
stop, admin capacity joiners hang until restart):
- spawn_refresh_if_needed's spawned task had no leader guard, unlike
refresh_or_join: a panic after the refresh future completes (commit,
metrics) killed the task before the reset block. The task now holds
the same RAII RefreshLeaderGuard, disarmed only after the state is
reset and the result published (S20).
- refresh_fn() was evaluated before AssertUnwindSafe wrapping in both
paths, so a panic while constructing the future escaped catch_unwind;
construction now happens inside the wrapped future.
- RefreshLeaderGuard::drop silently skipped the reset when try_lock was
contended and no tokio runtime was current; it now falls back to a
blocking reset, which is safe precisely because there is no executor
to stall in that context (S31).
Ref: rustfs/backlog#1021 (S20+S31 from audit rustfs/backlog#1010)
Close the ~10ms instrumentation residual on the PUT success path that the
existing coarse writer_setup/encode/rename stage metrics do not attribute.
Adds `hp_guard!` measurement scopes to the previously uninstrumented
sub-stages called out in backlog#935 item 2:
- SetDisks::acquire_read_lock / acquire_write_lock (namespace lock acquire)
- S3::put_object_prelookup (pre-write get_object_info lookup)
- MultiWriter::shutdown (bitrot writer flush/close)
- SetDisks::commit_rename_data_dir (old data-dir reclaim)
- S3Access::put_object (S3 authorization segment)
Instrumentation only: `hp_guard!` expands to nothing without the `hotpath`
feature, so this is a pure-observation change with zero behavior impact and
zero cost in default builds. The pre-lookup site wraps only the lookup call
in a scoped block so the guard measures that slice exactly while preserving
the existing match and control flow.
Verified: cargo check -p rustfs-ecstore (default and --features hotpath) and
cargo check -p rustfs (default and --features hotpath) all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): route rebalance delete markers to store and count expiries
Rebalance migrated delete markers via SetDisks::delete_object (single set,
no cross-pool routing), which silently rewrote the marker back onto the
source set. The subsequent source cleanup then deleted the whole entry,
losing the delete marker and reviving logically-deleted objects (#942, P0).
Route delete-marker migration through ECStore::delete_object via a store-
level closure that mirrors the existing transfer closure, so the marker
lands on the cross-pool target honouring data_movement/src_pool_idx/
delete_marker; on failure it is not counted as moved, so the source entry
is not cleaned up. The unused MigrationBackend::delete_object_for_migration
footgun is removed.
Rebalance source cleanup also ignored lifecycle-expired versions: the gate
used rebalanced == total_versions and passed an empty allowed_missing, so
any entry with an expired version could never be cleaned up, leaking the
migrated versions in the source pool (#950). Mirror decommission: gate on
rebalanced + expired == total_versions and feed expired version identities
into the cleanup preflight allowed_missing, plus a source_retained warning
for parity with decommission diagnostics.
Adds regression tests for delete-marker store routing and the expiry-aware
cleanup gate.
Co-authored-by: heihutu <heihutu@gmail.com>
Non-slash delimiter ListObjects/ListObjectVersions force a recursive walk and
defer common-prefix folding to from_meta_cache_entries_sorted_infos. When a page
of max_keys+1 raw candidates collapses into fewer than max_keys common prefixes,
list_objects_paginate left is_truncated=false with no next_marker even though the
walker had filled its raw candidate limit (disk_has_more=true). Clients were told
listing was complete and silently lost every key beyond the scan window, a
data-loss bug for backup/sync/mirror/reconcile consumers (backlog ECA-03 / #944).
Add a delimiter re-fold guard: when disk_has_more is true but the folded visible
entries are fewer than max_keys, set is_truncated=true and continue from the last
RAW scanned key (captured before folding via last_scanned_entry_name). Using the
last raw key rather than a folded prefix is required for correctness: forward_past
advances on name > marker, and a folded prefix such as "data-" is lexicographically
smaller than its own keys ("data-0001"), so it would re-list and re-fold the same
keys forever. A real scanned key strictly advances past the scanned window and
keeps pagination finite. The slash-delimiter and no-delimiter paths fold during
the walk and are unaffected.
Regression tests cover the direct re-fold case, the disk-exhausted no-marker case,
the no-delimiter path guard, and an end-to-end 5000 data-* + 100 other-* paging
loop asserting bounded termination and full common-prefix coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
is_all_not_found and is_all_volume_not_found delegated to
is_err_bucket_not_found, which matches StorageError::DiskNotFound. When
every erasure set in every pool was unreachable, the per-set aggregate
DiskNotFound slice was classified as "all not found", so list_merged
returned Ok(empty) and walk_result_from_set_errors was fed an
availability failure disguised as an empty listing. Clients saw a
successful empty ListObjects and mistook a full outage for an empty
bucket.
Introduce is_err_strict_not_found (FileNotFound / VolumeNotFound /
FileVersionNotFound / ObjectNotFound / VersionNotFound, excluding
DiskNotFound) for is_all_not_found, mirroring MinIO's isAllNotFound and
DiskError::is_all_not_found. Give is_all_volume_not_found its own
is_err_strict_volume_not_found (VolumeNotFound / BucketNotFound, minus
DiskNotFound) so genuine volume/bucket absence still surfaces while a
missing object under an existing volume is not escalated. Add
is_all_disk_not_found and a pre-check in walk_result_from_set_errors so
an all-offline slice surfaces DiskNotFound instead of being swallowed as
an empty listing, while partial outages (a healthy set present) stay
tolerated as before.
Regression tests cover all-DiskNotFound rejection, genuine not-found
acceptance, volume-not-found semantics, partial/empty short-circuit, and
the walk full-offline vs partial-offline paths.
Co-authored-by: heihutu <heihutu@gmail.com>
Two correctness defects on the opt-in codec-streaming GET path.
ECA-02 (#943): ErasureDecodeReader only decremented `remaining` for the
main fill buffer. Under the default DualInFlight policy each fill also
produces a queued stripe that is delivered to the client via
`prefetched_bufs.pop_front()` without touching `remaining`, so any object
larger than one erasure block finished with `remaining > 0` and the GET
terminated with LessData despite delivering all bytes. The inflated
`remaining` was also fed back into the fill worker, which used it to trim
the final stripe and to decide whether to read past EOF. Account for the
queued-stripe bytes when they enter the prefetch queue; queued buffers
come only from `Ok(true)` decodes so they are non-empty and bounded by
`remaining - main_buf.len()`, ruling out underflow.
ECA-04 (#945): the codec-streaming gate did not inspect `opts.part_number`.
A partNumber GET carries `range == None`, so it was not classified as a
Range request and reached the full-object codec-streaming reader, which
drops the storage offset/length returned by GetObjectReader::new. A
partNumber >= 2 request would then stream the whole object. Mirror the
direct-memory part_number fallback and route any partNumber request back
to the legacy duplex path, which applies the offset/length correctly.
Regression tests: DualInFlight read_to_end on a multi-block object and on
a non-block-aligned object; SingleInFlight vs DualInFlight byte-identical
output; gate fallback on partNumber requests.
Co-authored-by: heihutu <heihutu@gmail.com>
The whole disk traversal runs inside spawn_blocking with no timeout
anywhere on the async side; the in-scan ProgressMonitor checks are
cooperative and only run between walker entries. A stat/readdir blocked
on a dying disk or hung NFS mount therefore never returns: the blocking
thread leaks, the refresh singleflight stays running forever, every
subsequent scheduled refresh is skipped as inflight, and every admin
capacity query joins an unbounded wait until process restart (S02).
- Wrap each disk scan in tokio::time::timeout with a hard wall-clock
ceiling of 2x the cooperative budget (min 5s). On expiry the caller
fails the disk scan (releasing the singleflight through the normal
error path, where the degraded/partial machinery from backlog#1014
keeps the failed disk's last-known value) and a shared AtomicBool asks
the blocking walker to exit at its next entry, bounding the thread
leak to the single wedged syscall.
- Bound refresh_or_join joiner waits at 5 minutes so admin queries
degrade into a clear error instead of hanging if the leader wedges in
a way the drop/panic guards don't cover.
Ref: rustfs/backlog#1017 (S02 from audit rustfs/backlog#1010)
refactor(concurrency): remove zero-caller facade modules and fix no-default-features build (backlog#1025)
The audit in rustfs/backlog#1010 (consistent with #805) established that most of crates/concurrency was a decorative facade with zero production callers; the real runtime concurrency control lives in rustfs/src/storage/*. This deletes the dead facades and keeps only what the workspace actually consumes.
Deleted (zero callers verified by workspace-wide grep):
- manager.rs: ConcurrencyManager, lifecycle start/stop, misleading 'started' lifecycle logs
- config.rs: ConcurrencyConfig, ConcurrencyFeatures, from_env
- timeout.rs: TimeoutManager, TimeoutGuard, TimeoutManagerPolicy
- lock.rs: LockManager, LockScopeGuard, OptimizedLockGuard
- scheduler.rs: SchedulerManager, SchedulerPolicy, IoStrategy
- deadlock.rs facade: DeadlockManager, RequestTracker
- backpressure.rs facade: BackpressureManager, BackpressurePipe
- the prelude module, unused io-core re-exports, and all feature flags
Kept (real callers in ecstore/heal/rustfs):
- workload.rs admission contract types (unchanged)
- workers.rs Workers pool (unchanged, retained per #4498)
- GetObjectQueueSnapshot (moved from manager.rs to new queue.rs)
- PipeBackpressurePolicy (used by rustfs/src/storage/backpressure.rs)
- DeadlockMonitorPolicy (used by rustfs/src/storage/deadlock_detector.rs)
- OperationProgress re-export (used by rustfs/src/storage/timeout_wrapper.rs)
Removing the feature flags fixes the previously broken cargo check -p rustfs-concurrency --no-default-features (E0432). Docs and the logging guardrail file list are updated to match.
Ref: rustfs/backlog#1025
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
Make OptimizedNotify::wait_for_read/wait_for_write cancellation-safe with an
RAII decrement guard so the waiter counter is decremented even when the future
is dropped at the await (capped-wait timeout or task abort), preventing the
counter from leaking upward on a hot lock.
Fix cleanup_expired_batch so recycling into the object pool actually works:
Arc::try_unwrap on a clone could never succeed, so no state was ever recycled.
Collect keys during retain, remove them afterward to obtain the owned Arc, and
try_unwrap that before returning the state to the pool.
Refs rustfs/backlog#1035
Phase 5 Slice 12 (backlog#939): move the local disk registry — the
path->disk map, the disk-id->endpoint map, and the pool/set/drive layout —
out of the three GLOBAL_LOCAL_DISK_* process statics into the per-instance
InstanceContext.
This is the migration the owned-guard prerequisite (#4501) unblocked.
- InstanceContext gains three `Arc<RwLock<..>>` registry fields, constructed
eagerly (empty), plus pub(crate) handle accessors.
- The three runtime-source handle functions (`local_disk_map_handle` etc.) now
return the current instance's registry via `current_ctx()`; every direct
`GLOBAL_LOCAL_DISK_*` access in runtime::sources (~15 sites) routes through
those handles instead. Accessors that hold a write guard across `.await`
(`record_local_disks`, `initialize_local_disk_maps`, `replace_local_disk_id`)
bind the handle first, so the same locks are held across the same awaits.
- The three statics are removed; the test-reset helper clears the current
context's registry.
Construction window: the registry is populated during storage startup, before
the ECStore exists, via the bootstrap context that `ECStore::new` then adopts —
so startup writes and post-construction reads share one registry (no
split-brain). Single-instance behavior is byte-for-byte unchanged; the
GLOBAL_LOCAL_DISK_* boundary arch guard passes with the statics gone.
Verification: cargo test -p rustfs-ecstore (32 disk/instance tests green) and
-p rustfs-heal (201 green), cargo clippy -p rustfs-ecstore -p rustfs-heal
--all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 12; disk-registry migration)
In the self.last_sec > o.last_sec branch, merge forwarded a clone of o to
age out stale ring-buffer slots but then summed the un-forwarded o.totals
instead of the forwarded copy, so aged-out windows leaked into the result
and the forwarded clone was a dead write. Read both operands in forwarded
form and align the summation on wrapping_add to match AccElem::merge.
The UnknownChecksumAlgorithmError message omitted crc64nvme and advertised
the deprecated md5. Parsing "md5" also returned a Crc32 algorithm and its
into_impl produced a 4-byte CRC32, so a caller asking for MD5 silently got
the wrong digest.
Enumerate the accepted algorithms (crc32, crc32c, crc64nvme, sha1, sha256)
in the error message, remove the unused Md5 enum variant, and make parsing
"md5" fail loudly. Add tests covering the message contents and the md5
parse error.
Follow-up hardening for the audit control plane. The ABBA deadlock
(backlog#961), dispatch failure propagation (backlog#962), and credential
header redaction (backlog#963) already landed on main; this change addresses
the remaining audit-side findings.
- backlog#970: reload/commit now shuts down existing replay workers and closes
old targets *before* activating the replacement set, so old and new workers
never drain the same store concurrently and re-deliver entries. Extracted a
state-neutral `shutdown_runtime_targets` helper (registry-then-cancellers
lock order preserved).
- backlog#978: `start()` claims the `Starting` transition atomically under the
state lock, closing the check-then-act race that could double-activate;
`dispatch()` no longer returns Ok while paused, it surfaces an explicit
`AuditError::Paused` so the audit trail is not silently corrupted.
- backlog#984 (audit part): `dispatch_audit_log` performs a single state read
and interprets not-running/paused as a deliberate skip while surfacing real
delivery failures; `dispatch_batch` records the same observability signals as
single dispatch; documented the unordered cross-target fan-out.
Adds unit tests: paused dispatch returns Err, concurrent start does not hang or
double-activate, commit closes old targets before installing new, and
dispatch/dispatch_batch delivery-outcome coverage (all-fail -> Err, partial ->
Ok, all-success -> Ok).
Relates to rustfs/backlog#970
Relates to rustfs/backlog#978
Relates to rustfs/backlog#984
Co-authored-by: heihutu <heihutu@gmail.com>
fix(targets): harden Postgres/MySQL SQL backends (pools, error class, idempotency)
Postgres (postgres.rs):
- Add deadpool wait/create/recycle timeouts + tokio-postgres connect_timeout,
and wrap every pool.get() in a Tokio hard-limit timeout so an unreachable
broker/DB can no longer block send_body/probe_table/is_active forever;
checkout timeouts map to TargetError::Timeout to trigger store replay.
- namespace format now DELETEs the row on s3:ObjectRemoved:* events instead of
UPSERTing, keeping the table consistent with the object lifecycle.
- map_pg_error: SQLSTATE class 40 (serialization_failure 40001,
deadlock_detected 40P01, transaction rollback) is now transient (Timeout,
retryable) instead of a permanent Request. Extracted map_pg_sqlstate for
unit-testable classification.
MySQL (mysql.rs):
- Wrap pool.get_conn() in a Tokio checkout timeout across insert/init/liveness
paths so an unreachable server cannot block the delivery thread.
- Add an event_id idempotency key: tables created by the target now carry an
event_id VARCHAR(255) PRIMARY KEY, inserts use ON DUPLICATE KEY UPDATE, and
replays reuse the stable store key so a lost-ack replay no longer appends a
duplicate audit row. Legacy two-column tables are detected and fall back to
the non-idempotent insert with a warning (backward compatible).
- redact_mysql_dsn splits on the last '@' so a password containing '@' no
longer leaks its tail into Debug output.
- Disconnect the stale pool before dropping it on inline TLS reload instead of
leaking its connections.
- Cache TLS file mtimes and only recompute the inline fingerprint when a cert
file changes, avoiding a 3-file read+hash on every checkout.
Adds unit tests for SQLSTATE classification, namespace delete SQL, removal-event
detection, DSN redaction with '@' in the password, and the MySQL insert/DDL
builders.
Relates to rustfs/backlog#976
Relates to rustfs/backlog#973
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
The capacity scan's timeout fallback required the exact prefix
(threshold + sample_rate files, default ~200k) to be fully enumerated
before any sample existed. Cold-cache HDDs cannot stat that many files
inside the 15s budget, so large slow disks — exactly the disks sampling
was designed for — always timed out with sampled_count == 0, discarded
the accumulated exact-prefix work with a hard error, and re-ran 15s of
useless full-disk I/O every 120s (S03). When a sample did exist, the
estimate only extrapolated over files the walker had seen, silently
under-reporting disks whose tail was never reached (S10).
- Enter sampling early: once half the (possibly dynamic) time budget is
consumed and the exact prefix hasn't filled, freeze the threshold at
the current position so the remaining budget collects samples and the
timeout fallback is always reachable.
- Compensate the unseen tail: when the scan root is a dedicated mount
point (stat st_dev differs from the parent), take the max of the
seen-files extrapolation and the filesystem-level used bytes. On
shared filesystems (multi-disk dev layouts) statvfs would overcount,
so it is not trusted and behavior is unchanged.
- sampled_count == 0 at timeout no longer hard-fails when a dedicated
mount provides filesystem usage; the exact prefix is kept as a floor.
Without any estimator the original error still surfaces.
- Extract ScanLimits from env reading so the blocking scanner is
parameterizable in tests.
Ref: rustfs/backlog#1013 (S03+S10 from audit rustfs/backlog#1010)
versions_scanned for both the scanner and ILM collectors was read from
the Lifecycle work source's `checked` counter, which is never recorded on
the production scan path — so rustfs_scanner_versions_scanned_total and
rustfs_ilm_versions_scanned_total sat at zero even while objects_scanned
climbed. The two metrics also have distinct intended meanings that were
conflated: "versions scanned" (all versions, any bucket) vs "versions
checked for ILM actions" (lifecycle-configured buckets only).
Add a lifetime `versions_scanned` counter recorded for every version the
scanner walks (independent of ILM), and record the Lifecycle source's
`checked` counter from the ILM evaluator so the ILM metric reflects the
real checked subset. The scanner collector now reports total scanned
versions; the ILM collector keeps the ILM-checked subset.
Closes backlog#995 (OBS-09).
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): fsync new object's ancestor dirs to close a power-loss gap
reliable_mkdir_all creates an object's directory (and any missing prefix dirs)
with plain mkdir and never fsyncs the parent chain. The commit-point fsync in
rename_data persists the object dir's *contents* (its xl.meta and data dir),
but not the object dir's own entry in the bucket/prefix directory. So on the
first PUT of an object, a power loss after the write is acknowledged could drop
the whole object directory even though its contents were durable — an
acknowledged write silently lost (rustfs/backlog#922 step 4).
For a new object (no prior xl.meta) under a durability tier that syncs commit
metadata, fsync the ancestor chain from the object dir's parent up to and
including the bucket after the commit rename, so the newly created directory
entries survive power loss. A starts_with guard bounds the walk to the bucket
subtree. Overwrites already have a durable object dir and are unaffected;
relaxed/none accept the wider window like the existing commit fsync.
Durability regressions are invisible to ordinary behavior tests, so the new
tests assert directly (via the fsync_dir recorder) that a first PUT under a new
prefix fsyncs both the prefix and bucket dirs, and that relaxed does not.
Scope: the non-inline (erasure) commit path. The inline branch has the same
gap and is a separate follow-up.
Refs: rustfs/backlog#922 (HP-1 step 4), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): fsync new inline object's ancestor dirs too
Extend the backlog#922 step 4 mkdir-gap fix to the inline commit branch. Like
the non-inline path, a first PUT of an inline object creates its directory
(and any missing prefix dirs) whose entry in the parent chain reliable_mkdir_all
never fsynced; the commit fsync persists the object dir's contents, not its own
entry. For a new inline object under a commit-metadata-syncing tier, fsync the
ancestor chain up to and including the bucket after the commit rename, using the
same starts_with-bounded walk (via the synchronous os::fsync_dir_std inside the
inline spawn_blocking closure).
Adds a test asserting a new inline object under a new prefix fsyncs both the
prefix and bucket dirs. rename_data now closes the ack'd-write power-loss gap on
both the erasure and inline paths.
Refs: rustfs/backlog#922 (HP-1 step 4), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(targets): make persistent queue store crash-safe and replay lifecycle correct
Harden the target notification persistent queue (store.rs) and the replay
worker lifecycle (runtime) against data loss, silent truncation, ordering
drift, orphaned tasks, and a few low-risk robustness gaps.
store.rs
- Atomic, durable writes: write to a per-key temp file, fsync (sync_all),
then rename into place; best-effort parent-dir fsync. A crash mid-write
can no longer lose an acknowledged event or leave a half-written payload
that reads as a valid entry.
- open() now removes leftover .tmp residue and zero-byte files, and only
indexes files matching the queue extension, so ghosts/foreign files are
never replayed.
- FIFO ordering is derived from time-ordered UUIDv7 entry names instead of
coarse, clock-dependent file mtimes, so replay order is stable and
identical after a restart.
- Clamp HashMap/Vec pre-allocation derived from untrusted inputs
(entry_limit, batch item_count) to avoid capacity-overflow panics / giant
allocations.
target/mod.rs
- QueuedPayload::decode validates body length against the recorded
payload_len, rejecting torn/truncated writes instead of delivering a
silently truncated body.
- send_from_store purges a NotFound/empty entry (index + file) instead of
skipping it, so it cannot occupy a queue slot and be replayed forever.
- sanitize_queue_dir_component appends a stable hash suffix when the id was
lossy, so distinct target ids can no longer collapse onto the same queue
directory; path-safe ids are unchanged (no migration).
runtime
- Replay backoff, idle waits, and inter-scan pauses are now cancel-aware, so
reload/shutdown is not blocked for the full retry delay.
- ReplayWorkerManager::stop_all signals cancellation and then awaits each
worker's exit (bounded, with abort fallback), preventing orphaned tasks
and overlapping drain of the same store.
- Fix the always-true replay flush condition so batching is real
(size/timeout based, one semaphore permit per batch) rather than one
permit per entry; dedup keys already pending in the batch.
- clear_and_close aggregates and reports per-target close failures instead
of swallowing them; explicit shutdown surfaces them.
Relates to rustfs/backlog#966
Relates to rustfs/backlog#967
Relates to rustfs/backlog#975
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(targets): apply rustfmt to replay batch dedup guard
Fixes the Quick Checks rustfmt failure on the multi-line `.iter().any(...)`
closure in the replay batch dedup guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A full refresh with partial disk failures used to commit the surviving
subset's sum as a fresh exact cluster total (no field carried the
partial-failure fact), while the complete disk cache kept the failed
disk's old value — so the reported capacity oscillated between the
partial sum and the cache-merged total on alternating refreshes.
- Add a degraded flag to CapacityUpdate/CachedCapacity, set when the
scan behind the update had partial errors; expose it in refresh logs,
admin capacity logs and a new rustfs_capacity_degraded_readings_total
counter.
- On a degraded full refresh, surface only the disks whose own scan
fully succeeded and merge them over a complete disk cache, so failed
disks keep their last-known values and the published total no longer
dips and bounces back. The disk cache is never replaced from a
degraded refresh.
- Without a complete cache, keep the partial sum (unchanged #805
non-pollution behavior) but mark the reading degraded.
Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010)
Address a batch of correctness/security audit findings in the messaging
notification backends (mqtt/nats/kafka/amqp/redis/pulsar/webhook).
MQTT (backlog#971): wait for broker acknowledgement via publish_tracked +
wait_completion_async (PUBACK/PUBCOMP for QoS>=1, flush for QoS0) with a
30s timeout before reporting success, instead of treating the enqueue as
delivered. Replace substring error matching with typed ClientError /
PublishNoticeError classification.
NATS (backlog#971, #973, #983): flush() after publish to confirm the broker
received the message before the durable copy is deleted; classify publish
and flush failures by typed error kind; warn when credentials are sent
without TLS.
Kafka (backlog#973, #983): use Error::is_retriable() so transient broker
states (NotLeaderForPartition, LeaderNotAvailable, RequestTimedOut, ...) are
retried instead of dropped as permanent; add a bucket/object message key for
per-object partition ordering; build the producer without holding the cache
lock across the connect await.
AMQP (backlog#973, #980): classify permanent broker protocol errors
(404/403/406, NOT_ALLOWED, ...) as request-level errors rather than
connectivity errors to avoid reconnect storms; bound publish and
publisher-confirm waits with a timeout; check is_enabled in is_active; run
full close() cleanup even when the broker close fails; warn when
mandatory=false may silently drop unroutable messages.
Redis (backlog#982): warn when PUBLISH reaches 0 subscribers; reuse the
cached ConnectionManager for health probes instead of a fresh handshake;
warn when tls_allow_insecure disables certificate verification.
Pulsar (backlog#983): replace std::sync::Mutex + unwrap with parking_lot
Mutex so a panic while holding the guard cannot poison later accesses.
Webhook (backlog#983): drain the response body so the connection can be
reused (keep-alive). The #974 redirect-follow SSRF fix already landed.
Relates to rustfs/backlog#971
Relates to rustfs/backlog#973
Relates to rustfs/backlog#974
Relates to rustfs/backlog#980
Relates to rustfs/backlog#982
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
Land the remaining notify-crate audit fixes.
backlog#979(b): remove_target now enforces the same bucket-binding guard as
remove_target_config, refusing to delete a target still referenced by a bucket
rule so notification rules are not left orphaned.
backlog#984:
- event.rs: an unversioned object omits versionId entirely instead of
serializing versionId:"" (empty object/request versions treated as "no
version").
- notifier.rs: RUSTFS_NOTIFY_SEND_CONCURRENCY=0 coerces back to the default
instead of building a zero-permit semaphore that deadlocks every dispatch;
init_bucket_targets_shared closes the replaced targets instead of dropping
them without close() (connection leak).
- subscriber_index.rs: store_snapshot uses an atomic compute_if_absent upsert,
removing the get-then-insert TOCTOU that could clobber a concurrent
first-writer's snapshot cell.
- pipeline.rs: send_event assigns the history sequence and broadcasts to live
subscribers under one critical section so broadcast order matches recorded
sequence order.
- xml_config.rs: filter value length is bounded by character count, not byte
length, so valid multi-byte keys are no longer wrongly rejected.
- global.rs: a losing initialize() race shuts the just-initialized system down
instead of leaking its targets/replay workers.
backlog#970 (notify part): reload_config stops the running replay workers
before activating the new ones, so old and new workers do not concurrently
drain the same persisted stores. The full signal+join shutdown lives in the
targets crate under the same issue.
Tests: added regression coverage for each fix.
cargo build -p rustfs-notify, cargo test -p rustfs-notify --lib (98 passed),
cargo clippy -p rustfs-notify --all-targets (clean).
Relates to rustfs/backlog#979
Relates to rustfs/backlog#984
Relates to rustfs/backlog#970
Co-authored-by: heihutu <heihutu@gmail.com>
Addresses security/correctness audit findings in the target-plugin control
plane, TLS reload coordinator, and runtime extension registries.
control_plane (backlog#977):
- Install now preserves the currently installed revision as previous_revision
so Rollback actually restores the prior version instead of a no-op.
- Split the gate: circuit-breaker and runtime-activation checks apply only to
Install/Enable; Disable and Rollback stay available as break-glass
remediation while the breaker is open.
- Enforce the sidecar runtime protocol version for every external transport
(previously skippable by declaring a non-gRPC transport) and validate the
plugin api_compatibility_version at planning time.
- Download/signature/provenance host allowlisting now matches the full
host authority, so an allowlisted host never implicitly authorizes a
different host:port.
TLS reload coordinator/validate (backlog#981, #970-coordinator):
- Compute the fingerprint before building material in register() to remove the
TOCTOU that could permanently pin an old certificate; rotation self-heals.
- Always start a detection loop when reload is enabled; Watch mode no longer
returns success without any loop.
- validate_cert_key_pairing now verifies the private key matches the
certificate's public key instead of only parsing both files.
- Normalize a zero poll interval to a positive minimum to avoid a panic that
silently killed the poll loop.
- Serialize reload cycles with a per-target mutex; a first-step fingerprint
read failure now records last_error and a failure metric.
- Duplicate label registration stops and joins the previous loop before
publishing the replacement (stop-before-start), preventing orphaned loops.
runtime extension points (backlog#983 runtime subset):
- ops_profiler/ops_diagnostics authorize before probing the registry so an
unauthorized caller cannot learn whether a backend/surface exists.
- s3_hooks dispatch_post_auth actually traverses the registered hooks for the
point instead of unconditionally returning Continue.
- sidecar send_with_timeout redacts errors and uses the configurable failure
threshold via policy, and a successful send resets the breaker accounting.
Relates to rustfs/backlog#977
Relates to rustfs/backlog#981
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): isolate process sampler windows
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- add a reusable ProcessSampler so callers can own independent sysinfo refresh windows
- wire separate sampler instances for obs metrics scheduling and memory observability
- keep compatibility helpers while avoiding cross-task CPU and disk delta interference
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): import process sampler bundle helper
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- import collect_process_metric_bundle_with in the metrics scheduler
- drop the stale collect_process_metric_bundle import after switching scheduler sampling to independent process samplers
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): move process sampler into blocking task
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- move the memory observability process sampler into the spawn_blocking closure
- satisfy the closure static lifetime required by tokio while keeping the isolated sampler design intact
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(ecstore,heal): return the local disk map as an owned read guard (Phase 5 prep)
Prerequisite for the Phase 5 disk-registry migration (backlog#939): the disk
map cannot move from the process global into the per-instance InstanceContext
while callers depend on a `'static` read guard borrowed from the global.
Change `local_disk_map_read` to return an owned guard
(`OwnedRwLockReadGuard`) via `Arc::read_owned` instead of
`RwLockReadGuard<'static, _>`:
- ecstore `runtime::sources::local_disk_map_read` now returns
`OwnedRwLockReadGuard<..>` (holds an Arc clone of the lock), not a `'static`
borrow of `GLOBAL_LOCAL_DISK_MAP`.
- heal's forwarding accessor and its callers (which hold the guard across
`.await` while clearing/writing per-disk markers) keep identical behavior —
the owned guard derefs to the same map, so iteration is unchanged.
This decouples the heal crate from the global's `'static` lifetime so a later
PR can source the map from the current instance's context. Single-instance
behavior is byte-for-byte unchanged; the same read lock is held across the same
awaits.
Verification: cargo test -p rustfs-heal (201 tests green), cargo clippy -p
rustfs-ecstore -p rustfs-heal --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, disk-registry prerequisite)
shard_read_costs_for_empty_disk_set_are_empty was a plain sync #[test], but
shard_read_costs_for_disks consults process-global topology state
(local_endpoint_hosts_for_shard_costs) whose fast-lock manager lazily spawns a
background cleanup task on first access. When this test was the first in a
process to touch that global — as under nextest's per-test isolation — the
tokio::spawn panicked with a TryCurrentError because no runtime was present,
making the test order-dependent flaky in CI (it passes only when some sibling
tokio test initializes the manager first).
Run it under a Tokio runtime like the sibling reservation tests
(#[tokio::test]), so the lazy init's spawn always has a runtime. Test-only
change; no production behavior change.
Verified failing before the change and passing after under both
`cargo test` and `cargo nextest run` in isolation.
Co-authored-by: heihutu <heihutu@gmail.com>
Phase 5 Slice 11 (backlog#939): move the background replication pool and stats —
the last service handles, and the only async ones — out of the process statics
into the per-instance InstanceContext.
- InstanceContext gains `replication_stats: OnceCell<Arc<ReplicationStats>>` and
`replication_pool: OnceCell<Arc<DynReplicationPool>>` (tokio async OnceCell),
with sync read accessors (`replication_stats`/`replication_pool`/
`replication_initialized`) and pub(crate) cell accessors for the async init.
- `init_background_replication` initializes the current instance's cells via the
same `get_or_init(async {…}).await` (workers still spawned once on first
init). The lifecycle owner helpers and the runtime-source accessors keep their
signatures and route through the current instance's context; the two statics
(and the now-unused lazy_static import) are removed. The replication-boundary
arch guard still passes.
Single-instance: init materializes one shared pool/stats via the bootstrap
context — byte-for-byte the same as the eager statics.
Tests: replication state is None until set and independent across instances.
Verification: cargo test -p rustfs-ecstore (22 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 11). Stacked on Slice 10 (#4494).
refactor(ecstore): migrate the bucket bandwidth monitor into InstanceContext (Phase 5 Slice 10)
Phase 5 Slice 10 (backlog#939): move the bucket bandwidth monitor out of the
process static into the per-instance InstanceContext.
- InstanceContext gains `bucket_monitor: OnceLock<Arc<Monitor>>` with
`init_bucket_monitor(num_nodes)` (set-once; returns false if already set) and
`bucket_monitor() -> Option<..>`.
- global.rs `init_global_bucket_monitor` / `get_global_bucket_monitor` keep
their signatures and route through the current instance's context; the static
is removed. The ignore-and-warn-on-reinit behavior is preserved (the warn
stays in global.rs).
Tests: the monitor is None until initialized, set-once (second init is a no-op),
and independent across instances.
Verification: cargo test -p rustfs-ecstore (21 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 10).
The non-inline rename_data commit path made the tmp xl.meta durable and then
fdatasynced the shard files in two sequential awaits, each its own blocking
round-trip. The two operations touch disjoint paths (the tmp xl.meta under the
tmp bucket vs the shard data dir) and have no ordering constraint between them:
both only need to be durable before the commit renames that follow.
Run them concurrently with tokio::join!, dropping a blocking round-trip from
the PUT commit critical path (backlog#922 step 2). The commit ordering is
unchanged — both futures complete before any rename, and a failure in either
aborts before the rename exactly as the sequential version did (tmp-meta error
is still surfaced first). Payload and metadata durability semantics are
identical under strict and relaxed tiers.
Validated by the rename_data crash-consistency harness (backlog#935): a crash
at any pre-commit point still reopens as old-or-new, never mixed. Existing
rename_data / durability-tier / disk::os tests are unchanged and pass.
Refs: rustfs/backlog#922 (HP-1 step 2), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
---
# Adversarial Validation Playbooks
The policy — risk tiers, role list, protocol, exit criteria — lives in the
root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
this skill does not restate it. This file adds the RustFS-specific probe
playbook for each role: concrete attacks, where they apply, and the real
shipped bug or rule that earns each probe its place.
## How to run a role
1. Pick the tier and the applicable roles per the root `AGENTS.md`.
2. Run each role as an independent pass over the final diff — a parallel
reviewer agent where the tooling supports it, otherwise a fresh
sequential pass that starts from the diff and the nearest scoped
`AGENTS.md`, discarding the writing session's assumptions.
3. Within a role, execute the probes whose domain the diff touches, plus any
attack the diff obviously invites that no probe lists — the playbook is a
floor, not a ceiling.
4. Report findings (concrete failure scenario or named missing test, with
file:line) or the role's null report: "attacked X, Y, Z — no break
found". A bare pass is not a result.
## Role playbooks
### Correctness adversary
- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
- Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
- Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
- Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
- Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
- Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
- Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
- Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
- Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
- Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
- Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
- Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
- Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
### Simplicity adversary
- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
- Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
- Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
- Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
- Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
### Security reviewer
- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
- Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
- Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
- Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
- Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
- Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
- Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
- Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
- Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
- Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
- Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
- Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
### Concurrency/durability reviewer
- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
- Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
- Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
- Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
- Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
- Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
- Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
- Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
- Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
- Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
- Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
- Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
### Compatibility reviewer
- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
- Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
- Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
- Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
- Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
- Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
- Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
- Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
- Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
- Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
- Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
- Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
- Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
- Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
- Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
- Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
- Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
- Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
### Performance reviewer
- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
- Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
- Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
- Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
- Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
- Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
- Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
- Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
- Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
- Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
- Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
- Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
- Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
- Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
- Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
### Test-coverage skeptic
- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
- Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
- Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
- Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
- Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
- Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
- Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
- Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
- Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
- Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
- Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
- Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
- Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
- Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
- Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
- Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
- Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
- Where: All crates; especially concurrent-branch merges into crates/ecstore
- Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
## Sources and maintenance
Probes are distilled from shipped bugs in git history (commit/PR references
above), GitHub security advisories (see the security-advisory-lessons
skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
and `docs/operations/`. Line numbers drift; when a cited location no longer
matches, trust the invariant and re-locate the code. When a new bug class
ships, add a probe with its evidence here rather than growing the policy
@@ -43,16 +43,7 @@ Use this skill to review code changes consistently before merge, before release,
#### Rust-specific checks (apply to all Rust changes)
- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
@@ -55,8 +58,8 @@ For every Rust code change, verify:
- [ ] No `f64 as usize` without prior clamping
### Concurrency
- [ ] Lock acquisition order is documented when multiple locks are used
- [ ] No `tokio::sync`write guards held across `.await` without bounded hold time
- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
- [ ] No `tokio::sync`lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
- [ ]`std::sync::Mutex` in async context is held only briefly, never across `.await`
@@ -84,12 +87,19 @@ For every Rust code change, verify:
- [ ] No camelCase statics or Hungarian notation
- [ ] New string literals don't duplicate existing constants
### Reuse and Necessity
- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
## Severity Classification
- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs``SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
bump version files to <target> (final version, ONE commit) -> merge
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
## Semver gate — confirm the target version before touching anything
Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
Rules:
- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
## Preview tag naming
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
## Phase 0 — Preflight
-`git status --short` clean; `git fetch origin main --tags`.
-`gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
```bash
git fetch origin main
PREVIEW_HASH=$(git rev-parse origin/main)# must contain the bump PR
```
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
## Phase 2 — Publish the preview tag
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>""$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and artifact verification
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
## Phase 4 — Run the artifact locally, verify the console
Work inside the session scratchpad directory; never leave stray data dirs.
-`./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
rc admin user add preview/ relcheckuser relchecksecret12
rc admin user remove preview/ relcheckuser
rc alias remove preview
```
- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
## Phase 6 — Publish the final tag on the validated commit
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}"# must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>""$PREVIEW_HASH"
git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,6 +18,8 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
-`AGENTS.md` (root and nearest path-specific files).
- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, and console preview/auth code.
- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
### 2. Map to advisory classes
- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
@@ -59,10 +59,18 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### IAM and service accounts
- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
### STS, OIDC, and federation flows
- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
### S3 copy, multipart, and presigned POST
- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
@@ -91,6 +99,8 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
@@ -113,6 +123,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### Trusted proxy and network identity
- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
- Add tests for direct spoofed headers and trusted-proxy headers.
@@ -130,6 +141,11 @@ Use these prompts while reviewing a diff:
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
@@ -27,9 +27,18 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### IAM import, service accounts, and privilege boundaries
-`GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-`GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-`GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-`GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-`GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
### STS, OIDC, and federation flows
-`GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-`GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-`GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-`GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
### S3 copy, multipart, and upload policy validation
-`GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -50,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Secrets, defaults, and cryptographic misuse
-`GHSA-j59h-h7q5-q348`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, and console surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-`GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-`GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-`GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-`GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
planning-docs-check:## Check that no planning-type documents are committed
@echo "📄 Checking for committed planning docs..."
./scripts/check_no_planning_docs.sh
.PHONY:pre-commit
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkquick-check## Run fast pre-commit checks without clippy/full tests
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY:pre-pr
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkclippy-checktest## Run full pre-PR checks with clippy and tests
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkclippy-checktest## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY:dev-check
dev-check:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkquick-check## Run fast local development checks
dev-check:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast local development checks
| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
# Docs-only PRs skip the full code CI, but they are exactly where a
# planning-type document could be slipped in (git add -f bypasses
# .gitignore). Run the guard here so the required "Test and Lint" check
# stays meaningful for docs-only changes.
- name:Check no planning docs committed
run:./scripts/check_no_planning_docs.sh
- name:Satisfy required check for docs-only changes
run:echo "Docs-only change — code CI is skipped by paths-ignore; planning-docs guard passed, reporting success for the required 'Test and Lint' check."
# noncurrent transition/expiry after an immediate compensation transition.
- name:Run ignored ILM integration tests serially
run:|
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
@@ -14,12 +14,18 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
## Execution Discipline
- Read the relevant existing code, tests, and local guidance before changing behavior.
- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
- State assumptions when they affect the implementation or verification path.
- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
## Autonomy and Approval Boundaries
- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
## Communication and Language
@@ -41,14 +47,27 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Do not refactor existing code only to make it easier to unit test.
- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
## Constant and String Usage
## Reuse Before You Write
- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
-If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
-When changing existing behavior, keep naming and format consistency by aligning with established project constants.
Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
-**Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
-**Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
## Necessary Code Only
Net-new code — files, types, branches, comments — is cost to justify, not progress:
- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
## Sources of Truth
@@ -59,17 +78,35 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- High-level architecture and crate map: `ARCHITECTURE.md`
- Migration guardrails, readiness contracts, support matrices:
`docs/architecture/README.md` (routes by audience)
-Historical implementation plans and trackers: `docs/superpowers/plans/`
- Shared agent skills (all tools): `.agents/skills/` — Claude Code reads them
through the `.claude/skills` symlink; add new skills to `.agents/skills/`
only, never as separate copies per tool
-Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
a frontmatter `description` stating when it applies. Scan the descriptions
before starting a task and follow any skill that matches, even if your tool
issue-scoped benchmark-result snapshots or optimization conclusions, or
agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
produces). Keep that work in the issue tracker or your local worktree, not in
the repository. Only durable reference — the architecture set under
`docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
and the test-suite references under `docs/testing/` — belongs in version
control; `.gitignore` ignores everything else under `docs/` by default, so a new
plan file will not be tracked unless someone force-adds it — don't.
`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and
CI) fails the build if anything is committed under `docs/superpowers/`, even via
`git add -f`.
## Verification Before PR
Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
For code changes, run and pass the following before opening a PR:
@@ -94,19 +131,119 @@ Before pushing code changes, make sure formatting is clean:
- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any installed git pre-commit hooks (for example, from `make setup-hooks`) may still run on commit unless explicitly skipped.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
Do not open a PR with code changes when the required checks fail.
Make a failing check pass by fixing the cause, never by weakening the gate:
do not loosen or skip a guard script, add entries to a baseline or allowance
list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or
delete or relax a failing assertion to get green. If a check itself is wrong,
change it deliberately and state the rationale in the PR.
For flaky tests, do not paper over them with retries. Follow the flake policy
in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h,
quarantine with an issue link, fix or delete within 30 days); the local
`default` nextest profile never retries.
## Adversarial Validation (Default On)
Every non-exempt output (see Risk tiers) — code change, bug fix, or
design/solution proposal — passes multi-role adversarial review before it
counts as done.
Author confidence is not evidence: each role's job is to refute the change,
not to bless it.
### Risk tiers
Pick the tier from the riskiest file touched; when in doubt, pick the higher.
- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
no runtime surface. Skip this section.
- **Mechanical:** pure renames, file moves, test-only or tooling changes —
correctness and simplicity adversaries only.
- **Standard (the default):** any change that affects behavior.
- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
absent, empty, and nil all mean "no value", never `Uuid::nil()`.
- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
send **no**`versionId` on tier GET/DELETE.
## Naming Conventions
- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic).
@@ -9,8 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
### Added
- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
- Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
- Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window
- Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
- Operator guide at `docs/operations/nats-jetstream.md`
- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
- Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
- Task-local storage for async-safe credential passing between middleware and auth handlers
@@ -33,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
@@ -8,9 +8,9 @@ This guide covers the local development environment and the checks expected befo
**MANDATORY**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
#### Pre-commit Requirements
#### Verification Requirements
Before every commit, you **MUST**:
Before submitting your changes for review, you **MUST**:
1.**Format your code**:
@@ -47,33 +47,59 @@ make fmt
# Check if code is properly formatted
make fmt-check
# Run clippy checks
make clippy
# Run clippy checks (all targets, all features, -D warnings)
make clippy-check
# Run compilation check
make check
# Fast workspace compilation check (excludes e2e_test)
make quick-check
# Run tests
# Full compilation check (cargo check --all-targets)
# Run all pre-commit checks (format + clippy + check + test)
# Fast pre-commit gate — see below for exactly what it runs
make pre-commit
# Setup git hooks (one-time setup)
make setup-hooks
# Full pre-PR gate (pre-commit gates + clippy + tests)
make pre-pr
```
> `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.
> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md).
### 🔒 Automated Pre-commit Hooks
#### What `make pre-commit` and `make pre-pr` actually run
This project includes a pre-commit hook that automatically runs before each commit to ensure:
`make pre-commit` is the **fast** gate. It runs, in order
(see `.config/make/pre-commit.mak`):
- ✅ Code is properly formatted (`cargo fmt --all --check`)
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
-`RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
blocked unless their exact `scheme://host:port` origin is listed in
`RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
**NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
2.**Create a Bucket**: Use the console to create a new bucket for your objects.
3.**Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
### OIDC Roles Claim (Microsoft Entra ID)
@@ -335,12 +344,18 @@ If you have any questions or need assistance:
RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
[](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
[](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
- `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
- `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service.
## Browser redirect environment variables
- `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node.
## Scanner environment aliases
- `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Zhengchao An
