rename_all_missing_source_still_warns and rename_all_real_failure_still_warns
assert that `warn_reliable_rename_failure` emitted its WARN, but that is a
single production callsite shared with tests that call rename_all *without*
installing a subscriber — rename_all_missing_source_returns_file_not_found,
two tests above, is one of them.
tracing caches each callsite's Interest process-globally and the first thread
to reach a callsite fixes that value; while at most one dispatcher is
registered, tracing-core derives it from the registering thread's own
subscriber, and registration is once-only. When the subscriber-less sibling
wins, the callsite is cached as Interest::never() and the WARN never fires,
so the assertion sees empty output:
ordinary missing-source failures must keep the WARN, got:
Reproduced at 3/25 with `disk::os::tests::rename_all_missing_source` (both
tests), against 0/20 for the victim alone. Fixed by pinning callsite interest
inside warn_capture(), so every current and future user of that helper is
covered rather than just the two tests that happen to fail today.
pin_callsite_interest_for_test() moves from cluster::rpc::background_monitor
to a new crate-level test_tracing module: it is domain-neutral and now has
consumers in two unrelated subsystems, and disk::os should not have to reach
into a cluster::rpc test helper.
Verified: repro filter 0/30 (was 3/25); disk::os:: 0/12; cluster::rpc:: 0/12
and its poisoner pair 0/15, confirming the moved helper still holds.
Follow-up to #5438. Closes the last item in #5439.
Restore persisted bucket notification rules after the notification target runtime converges so restarted nodes rebuild their local rule engine without requiring an unchanged PUT bucket notification request.
Fixes#5428
Co-authored-by: heihutu <heihutu@gmail.com>
peer_rest_recovery_probe_logs_keep_request_id_span_context failed ~10% of
`cargo test -p rustfs-ecstore --lib -- cluster::rpc::` runs with
left: "request-span", right: "recovery-monitor". The recovery-monitor
info_span! was evaluating to Span::none(), so the probe's log line landed
under the caller's span.
tracing caches each callsite's Interest in process-global state, and the
first thread to reach a callsite fixes that value. While at most one
dispatcher is registered, tracing-core takes a fast path that derives the
interest from the registering thread's own subscriber, and registration is
once-only (CAS). Under libtest a sibling test reaches recovery_monitor_span
via mark_offline_and_spawn_recovery from a thread with no subscriber, so
the interest is derived from NoSubscriber and cached as Interest::never()
for the whole process.
Add pin_callsite_interest_for_test(): registering a second, inert
dispatcher rebuilds every registered callsite's interest against the live
dispatcher set (repairing a poisoned value) and keeps tracing-core off the
single-dispatcher fast path (preventing new ones). This also covers the
production marked_suspect / recovery_monitor_started event callsites that
remote_disk_network_error_starts_recovery_monitor_with_request_context
asserts on.
rename_data_response_accepts_legacy_json_without_decode_error is a
separate root cause: it snapshots the process-global internode metrics and
asserts the decode-error counter did not move, which siblings that record
decode errors (or reset the counters) invalidate. Put the 11 tests that
observe those counters in one #[serial(internode_metrics)] group.
Both races are impossible under nextest, which runs each test in its own
process, so neither test belongs in the ecstore-serial-flaky test-group
(that serializes across process boundaries) nor in the ci-profile
quarantine (they never redden CI).
Verified: cluster::rpc:: subset 0/30 failures under libtest (was 3/30);
target test paired with its poisoner 0/30 (was 4/20); 5/5 clean under
nextest at 179/179.
The Test and Lint lane intermittently stalls until the inner 75m timeout
kills the cargo process group (issue #5394). The evidence collected since
#5402 shows the stall can begin in the build phase before any test runs
(run 30449339653: last output at minute 4 of the nextest step, then 71
silent minutes until SIGTERM), but the post-mortem pgrep always reports
nothing because GNU timeout has already terminated the whole process
group by the time it runs.
Add a background sampler to the nextest step that appends system and
process snapshots (loadavg, PSI, memory, disk, top-RSS processes,
cargo/rustc/linker/build-script processes, D-state processes) to the
existing test-and-lint artifact every 60 seconds, and record kernel
OOM/kill events in the post-mortem diagnostics. The last samples before
a timeout identify the wedged process or the resource pressure that
caused the stall. This only instruments the failure mode where the
runner survives; jobs whose runner disappears entirely still upload no
artifacts and need runner-pool-side logs.
Refs #5394
* fix: keep hotpath gate csv paths clean
Redirect the enhanced bench driver's verbose output away from command substitution so run_hotpath_warp_ab.sh records only the candidate cell path before passing baseline_compare.csv files to the gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(utils): reduce highway hash key setup
Cache the parsed HighwayHash keys as compile-time constants and add a criterion benchmark for the PUT hash hotpath.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The LoadBucketMetadata peer-notification handler loaded bucket metadata
with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and
unconditionally cached the result. On a transient read-quorum dip during
a reload notification, a peer cached an authoritative "no Object Lock"
default for a lock-enabled bucket, disabling the batch-delete retention
gate (object_lock_delete_check_required) on that node until the next
refresh, and wiping its bucket-target/durability sync state.
Production changes:
- New BucketMetadataSys::reload_from_store (metadata_sys::
reload_bucket_metadata): the peer reload path uses the presence-aware
loader and installs only metadata actually read from persisted
storage. A load miss returns an error (surfaced to the notifying peer
as success=false) and leaves the cache untouched; deletion still
propagates only through the dedicated DeleteBucketMetadata
notification.
- The reload runs under the outer metadata-sys write guard, load
included, mirroring update(): every other cache installer holds that
lock, so a reload snapshot can never land after - and roll back - a
newer concurrent install (the stale-load lost-update from the
review), and the install-plus-registry-sync sequence stays atomic
against concurrent removes and reloads (previously only the set call
was write-guarded, with the load outside any lock).
- The peer-visible miss error is a fixed string: the notifying peer
substring-matches error text against network-failure needles
(is_network_like_error), so interpolating a bucket name (e.g. a legal
bucket literally named "unavailable") could mark a healthy peer
offline.
- get_config's lazy insert routes through set(), picking up the
negative-cache invalidation.
An earlier draft instead guarded set() with a per-config updated_at
freshness comparison. Adversarial validation rejected it (three roles
independently): update_config stamps with the handling node's wall
clock, so within the skew the cluster already tolerates (+/-300s RPC
auth window) a config rewritten with an earlier stamp - e.g. revoking a
public-read policy through a second node, or any same-field rewrite
after an NTP step-back - would be skipped by every peer forever,
silently pinning the revoked permissive config with no re-convergence
path (the 15-minute refresh also routed through the guard). Race
staleness is second-scale while skew is minute-scale, so no tolerance
bound can separate them; the write-guard serialization closes the same
race without clocks and preserves the refresh loop's unconditional
converge-to-disk property, which is the cluster's self-healing
mechanism.
Startup audit (BucketMetadataSys::init): concurrent_load's
insert-if-vacant still installs a fabricated default when a transient
miss hits at boot - indistinguishable from a legacy bucket without a
metadata file at this layer - bounded by the next successful persisted
load. Making the object-lock gate fail closed on such entries is filed
as a follow-up, alongside the bare "unavailable" needle in
is_network_like_error and the Swift cache-only metadata writes.
Tests:
- bucket::metadata_sys::tests::
peer_reload_never_caches_fabricated_defaults_as_authoritative:
miss installs nothing / miss keeps the existing entry intact
(asserting the dedicated non-persisted error) / persisted reload
converges the cache over a stale entry.
- node_service::tests::
test_load_bucket_metadata_failure_skips_scanner_maintenance:
a failed reload reports failure and does not advance scanner
maintenance activity (previously recorded even on a miss).
- The handler success path stays uncovered at the RPC layer (needs an
isolated global object layer, like the pre-existing ignored test);
the composition is pinned at the sys level instead.
Verification:
- cargo fmt --check and cargo clippy --lib --tests clean on
rustfs-ecstore and rustfs.
- Targeted suites green; full cargo test -p rustfs-ecstore --lib:
3198/3200 with two parallelism-sensitive lock-test flakes from the
known baseline (pass in isolation; a different pair flakes per run).
- Adversarial validation (high-risk tier, all seven roles as
independent parallel reviewers) run per AGENTS.md; all findings
fixed or rebutted with evidence, three out-of-scope findings filed
as follow-up tasks.
Co-authored-by: houseme <housemecn@gmail.com>
Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.
Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.
Persisting these writes required hardening the paths that now produce
durable state:
- Account metadata writes validate account ownership. This metadata
holds the account's TempURL signing key, so an unauthenticated write
for someone else's account would have become a durable, cluster-wide
takeover of that account's pre-signed URLs. Reads stay open because
TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
metadata loader's "no metadata on disk" default would be persisted,
creating an orphan metadata file and caching a fabricated default as
authoritative.
- Container and account metadata are size- and count-limited, reusing
the Swift limits object metadata already enforces; these tags land in
the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
unreadable, instead of merging onto an empty set and wiping the
container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
now carry real disk and quorum detail.
The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation
* feat: enhance local SSE DEK handling with JSON envelope format and versioning
Align inline fast path cluster tests with the current tier mutation and manual transition contracts while keeping msgpack compat assertions focused on observed traffic.
Co-authored-by: heihutu <heihutu@gmail.com>
DeleteObjects rebuilt ObjectOptions inside its per-key phase-1 loop, and
del_opts re-fetched the bucket versioning configuration on every call: a
1000-key request paid 1001 identical metadata-sys lookups, each taking the
global bucket-metadata read lock and cloning a VersioningConfiguration.
Split del_opts into the existing async entry point plus a synchronous
del_opts_with_versioning that takes an already-resolved configuration, and
reuse the request-level version_cfg the handler had already fetched.
This also removes a snapshot inconsistency. The skip-stat decision
(can_skip_delete_objects_pre_stat) and the post-delete accounting already
derived from the request-level snapshot while del_opts re-read the config
per key, so a PutBucketVersioning committing mid-request could give a key
opts.versioned=true while the batch delete still ran with versioned=false:
the object was deleted outright with replication state attached to a delete
marker that was never created, and the advisory object-lock pre-check was
skipped on a false premise. All three now derive from one snapshot.
GET, HEAD and DeleteObject additionally establish bucket existence before
building ObjectOptions, so a request naming a nonexistent bucket no longer
performs bucket-metadata work first. GET keeps its cheap request-shape
validations (key, range, partNumber) ahead of the existence check so
InvalidArgument still wins over NoSuchBucket for malformed requests.
Bucket validation moves to validate_bucket_exists, which takes an explicit
store and shares the existing 5s-TTL cache. GET, PUT, HEAD and DeleteObject
now resolve the store through the request-bound server context
(backlog#1052 S6) instead of the process-global handle: get_validated_store
resolves the first-published global AppContext, so in an embedded
multi-instance process a request to the second server operated on the first
server's store. Remaining global-handle call sites in bucket_usecase and
select_object are left for a follow-up.
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Add a reusable evidence collector for manual transition tier-failure attribution samples and wire the runbook tests to cover its dry-run and argument guards.
Co-authored-by: heihutu <heihutu@gmail.com>
Recover accepted manual-transition tasks from the durable task journal when worker-result markers are missing after a restart.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a script-test entry for the manual transition rollout and soak runbook generators. The test covers ratio fail-fast behavior, generated status/metrics/journal checks, and runnable dry-run artifacts.
Verification:
- ./scripts/test_manual_transition_runbooks.sh
- make script-tests
Co-authored-by: heihutu <heihutu@gmail.com>
Add a reusable offline rebuild path for pinned paired ABBA benchmark evidence so corrected summaries can be regenerated from raw warp logs after parser fixes.
Record ACK-contract comparability explicitly in the rebuilt product comparison output, preventing relaxed RustFS ACK results from being treated as direct strict MinIO comparisons.
Co-authored-by: heihutu <heihutu@gmail.com>
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
* fix(admin): bound IAM import archive expansion
MAX_IAM_IMPORT_SIZE caps the compressed upload at 10 MB, but every member of the
archive was then read with read_to_end into an unbounded Vec. Deflate ratios well
above 100:1 are easy to construct, so a small authorized upload could expand
without limit across the seven members ImportIam reads.
Add a shared expansion budget (MAX_IAM_IMPORT_EXPANDED_SIZE, 10x the compressed
cap) drawn down by every member, and route all seven reads through one helper
that reads a byte past the remaining budget to detect overrun. Sharing the budget
bounds the archive as a whole rather than letting each member spend the full
limit independently.
Covers R03-CAN-024 through R03-CAN-030 plus R04-CAN-077 (backlog #1471) — one
fix rather than seven, since all seven call sites were byte-identical.
* fix(kms): confine local key paths and refuse silent key replacement
Local KMS key identifiers arrive from request input — the `name` tag on CreateKey,
the `keyId` body field or query parameter on DeleteKey — and were joined onto
`key_dir` with no validation. An identifier such as `../../tmp/evil` escaped the
configured directory, making key creation a constrained arbitrary-file write and
`DeleteKey` with `force_immediate` a cross-directory delete.
Validate in `master_key_path` and make it fallible, so every filesystem path in
this backend inherits the guard: decode_stored_key, load_master_key,
save_master_key, create_key and delete_key all derive their paths there. The rule
is containment rather than a character allowlist, so identifiers already in use
keep resolving; only separators, NUL, absolute paths and non-single-component
forms are refused. Note `.` and `..` are contained rather than refused — the
`.key` suffix turns them into the ordinary filenames `..key` and `...key`.
Separately, `LocalKmsBackend::create_key` had no existence check, while the
sibling `KmsClient::create_key` has always had one. Since `save_master_key`
renames over its destination, creating a key under an existing name silently
replaced its material and destroyed the ability to decrypt everything wrapped
under it — and the backend path is the one the admin API uses. It now returns
KeyAlreadyExists, matching StaticKmsBackend.
Covers R03-CAN-072, R03-CAN-073 and R07-CAN-103 (backlog #1475). R03-CAN-073
needed no separate change: delete_key routes both its load and its remove_file
through master_key_path.
* fix(swift): bound SLO manifest reads to the 2 MiB manifest limit
The three Swift SLO handlers that load a stored manifest (handle_slo_get,
handle_slo_get_manifest, handle_slo_delete) read the `<object>.slo-manifest`
object to EOF with AsyncReadExt::read_to_end. That key is predictable and
writable through the ordinary object PUT path, so a tenant can replace the
manifest with an arbitrarily large object and then make the server allocate
its full size on every SLO GET, multipart-manifest=get, or
multipart-manifest=delete request - a memory amplification bounded only by
the stored object size (CWE-400 / CWE-770). The 2 MiB manifest limit that
handle_slo_put enforces was not applied on the read side.
Introduce MAX_SLO_MANIFEST_SIZE (the existing 2 MiB PUT limit, now a named
constant) and a shared read_manifest_bytes helper that reads through a
`take(limit + 1)` and rejects anything larger, so an oversized manifest is
refused instead of being buffered first. All three call sites go through the
helper. handle_slo_put now checks the size before parsing the JSON.
Regression tests: test_read_manifest_bytes_rejects_oversized_manifest and
test_read_manifest_bytes_stops_reading_oversized_manifest (which asserts the
reader is not consumed past the limit), plus a boundary test that a manifest
at exactly 2 MiB is still accepted.
* fix(protocols): authorize every object in FTPS/WebDAV recursive deletes
The FTPS and WebDAV gateways authorized only the container before a
recursive delete and then destroyed everything inside it without a
further check:
- FTPS RMD (and DELE on a bucket path ending in '/') cleared
s3:DeleteBucket, then delete_bucket_recursively listed the bucket and
deleted every object.
- WebDAV DELETE on a bucket did the same via its own
delete_bucket_recursively.
- WebDAV DELETE on a directory cleared s3:DeleteObject for the directory
marker key ("dir/") only, then listed that prefix and deleted every
child under it.
A principal holding s3:DeleteBucket (or s3:DeleteObject on a single
marker key) could therefore erase objects it had no s3:DeleteObject
permission for, and the operation reported success.
Deletion stays recursive - that is the expected behaviour for these
protocols - but each object now clears s3:DeleteObject on its own key
before it is removed, and the enumeration clears s3:ListBucket. A denial
aborts the whole operation with access denied rather than being skipped,
so the caller can never be told the delete succeeded while objects were
left behind or removed without authorization.
The test double gained shared-state cloning, delete_object/delete_bucket
call logs, and list/delete queue helpers so the regression tests can
observe that nothing is deleted once a deny lands.
* fix(server,ecstore): bound TLS handshakes and remote volume RPC waits
Three call sites let an unauthenticated client or a misbehaving peer hold
server resources with no deadline.
TLS listener (R03-CAN-035): process_connection awaited
`acceptor.accept(socket)` with no bound. A client that opens a TCP
connection and never finishes the handshake parks a Tokio task and a socket
forever, and the connection cap (RUSTFS_API_MAX_CONNECTIONS) is unlimited by
default, so nothing else sheds it. The handshake now runs under
accept_tls_with_deadline(), reusing the existing HTTP/1 header-read budget —
the established slow-client bound for the pre-request phase — and the
expiry is recorded through the same log/metric path as a handshake error,
under a new TIMEOUT failure kind.
Remote disk RPCs (R03-CAN-049, R03-CAN-050): list_volumes and delete_volume
passed Duration::ZERO, which execute_with_timeout treats as "no deadline",
so a peer that accepts the request and never answers stalls the coordinator
(and, for delete_volume, the bucket-deletion workflow). Both now pass
get_max_timeout_duration(), matching every sibling method in the file.
Regression tests: a silent TLS peer must be shed by the handshake deadline;
list_volumes/delete_volume against a peer that completes the TCP connect and
then goes silent must fail with DiskError::Timeout instead of hanging.
* fix(security): stop leaking signed headers and bound OIDC/KMS credentials
Three independent hygiene fixes found by the security review.
R03-CAN-018 (crates/signer): try_get_canonical_headers and get_signed_headers
logged the complete header map at DEBUG before signing. Runtime callers pass
session credentials and SSE-C key material through these headers, so anyone
able to raise the log level (or read DEBUG logs) recovered
X-Amz-Security-Token and SSE-C keys verbatim. The statements were debugging
leftovers with no operational value and are deleted rather than redacted.
R03-CAN-014 (crates/iam): the OIDC HTTP adapter buffered provider responses
with an unbounded Response::bytes(), so a configured, compromised or
attacker-pointed IdP endpoint could stream an arbitrarily large or endless
body into memory (the ValidateOidcConfig admin handler lets a ServerInfo
caller choose the endpoint). Responses are now read incrementally and fail
closed past MAX_OIDC_RESPONSE_SIZE, and the already SSRF-hardened client
builder gains request and connect timeouts so a stalled provider cannot pin
the calling task indefinitely.
R07-CAN-105 (helm): the Vault KMS token was serialized into the chart
ConfigMap, exposing it to every subject allowed to get ConfigMaps in the
namespace. It now renders into a dedicated Secret that the Deployment and
StatefulSet consume via envFrom; the Secret is separate from the main
credentials Secret so it also works when secret.existingSecret is set.
Regression tests:
- rustfs-signer: signing_never_logs_signed_header_material
- rustfs-iam: oidc_response_body_past_the_limit_is_rejected,
oidc_response_body_at_the_limit_is_accepted
- scripts/test_helm_templates.sh: KMS token must never render in plaintext
* fix(webdav): enforce body limit, request timeout and connection cap
The configured WebDAV maximum body size was enforced from Content-Length, so a
chunked request declared no length and bypassed it entirely. The configured
request timeout was never applied to the connection at all, and the accept loop
spawned a task per connection with no bound, so an unauthenticated client could
hold resources indefinitely and in unbounded number.
Enforce the limit on bytes actually read rather than the declared length, apply
the configured timeout to the request, and bound accepted connections with a new
RUSTFS_WEBDAV_MAX_CONNECTIONS (default 1024) surfaced in the config report.
Covers R03-CAN-051, R03-CAN-052, R03-CAN-067, R04-CAN-089, R05-CAN-094 and
R05-CAN-097 (backlog #1471, #1474).
* fix(security): stop STS credentials from crossing the parent trust boundary
Two related credential-boundary holes let a short-lived STS credential act
with the full, unrestricted authority of the long-term user it was minted
from.
AddUser (R03-CAN-021, CWE-269/863): should_check_deny_only relaxes the admin
policy check to deny-only when a Console/STS session targets the IAM user it
represents. Nothing then stopped that session from calling AddUser with its
own parent's access key, so the handler wrote an attacker-chosen secret key
and status over the parent's stored Credentials via create_user ->
save_user_identity. A session that expires in minutes became permanent
control of the account. AddUser now rejects any temp or service-account
requester whose resolved parent equals the target access key, resolving the
parent the same way should_check_deny_only does (parent_user field, else the
JWT `parent` claim, since some stores persist the parent only in the token).
FTPS/SFTP/WebDAV password auth (R04-CAN-086, CWE-287/862): these protocols
looked the access key up with check_key, which falls back to the STS account
cache, and then compared only the stored secret. An STS access key plus
secret therefore authenticated with no session token presented and no
session-policy claims applied - the holder got the parent's full permissions.
Password authentication now rejects temporary credentials before the secret
comparison. The discriminator is is_temp() && !is_service_account(), the same
one IamCache::update_user_with_claims uses to route an identity into the STS
cache, so service accounts - which resolve policy from stored IAM state
rather than a client-presented token - keep working over these protocols.
Regression tests cover both predicates and pin the guards to their call
sites so neither can be dropped without a test failure.
Add a deterministic four-hot AddTier regression for the reporter path where one peer has already durably committed prepare replay and the remaining peers still need commit fanout before local publish.
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(storage): resolve staged dead code in store module (#730)
Drop the store module's blanket #![allow(dead_code)] and resolve every
unit it was masking, instead of keeping them staged:
- write_persistent_key_only_index: the keys-only writer wrapper only had
test callers; the in-process rebuild flow already exists and uses the
_with_metadata variant (prepare -> rebuild, triggered lazily from the
opt-in listing path). Tests call _with_metadata directly now.
- ListIndexLifecycle::recover_after_restart / mark_corrupt: production
derives index health per request from the persisted artifacts (index
file + namespace mutation journal), and restart recovery already lives
in load_persistent_key_only_index's journal restore. The persisted-
lifecycle design these transitions served was superseded, and Corrupt
had no detector, so the never-constructed Corrupt state/reason
variants go with them.
- record_list_objects_index_opt_in_fallback (+ helpers): superseded by
the inline per-site fallback recording in the opt-in listing path; the
recorder recomputed health with a hardcoded None provider, so it could
only ever report a disabled-based reason.
- Provider-contract traits (Generation/Page/PageIterator/KeyLookup):
provider dispatch went the ListObjectsIndexProviderKind enum route;
only ListMetadataIndexHealth is live (dyn in source-mode selection)
and stays, minus its unused is_healthy default.
- Delete the unused list_quorum_from_env/RUSTFS_API_LIST_QUORUM pair,
fold should_resume_local_decommission and
should_auto_start_rebalance_after_recovered_meta into their surviving
primitives, drop get_disk_via_endpoint, and cfg(test) the remaining
test-only conveniences.
* fix(ilm): fail closed on unsafe manual job recovery
Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(ilm): format manual recovery imports
Apply rustfmt import ordering after merging main into the manual recovery branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): deduplicate manual transition worker results
Persist per-task manual transition worker result markers before applying job counters so duplicate worker completion reports are no-ops. Reconcile persisted markers during drained-queue lease renewal and recovery to restore marker-before-record crash windows. Fail closed when persisted worker result markers are corrupt.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ilm): fail closed on unsafe manual job recovery
Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.
Co-authored-by: heihutu <heihutu@gmail.com>
Cover manual transition terminal job records for queue closed and queue send timeout outcomes so backpressure summaries remain partial with the expected counters and queue snapshots.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a black-box e2e for concurrent manual transition durable jobs with the same bucket and prefix. The test asserts that exactly one request is accepted, the other returns 409 with the active job status and cancel endpoint, and the accepted job reaches a completed terminal status after queued transitions settle.
Verification:
- cargo fmt --all --check
- git diff --check
- CARGO_TARGET_DIR=/private/tmp/rustfs-target-1481-same-scope cargo test -p e2e_test --lib reliant::tiering::test_manual_transition_async_same_scope_conflict_reports_active_job -- --exact --nocapture
Co-authored-by: heihutu <heihutu@gmail.com>
Add a focused ecstore regression that exercises active manual transition cancellation through the existing cancel_check hook after scan progress has been made. The test verifies the cancelled report, dry-run counters, and opaque resume cursor without relying on wall-clock timing.
Co-authored-by: heihutu <heihutu@gmail.com>
Extend the internode gRPC benchmark driver with P2 request-only, canary, after, and rollback phases. The request-only phase proves that requesting msgpack-only without fleet confirmation keeps compatibility JSON fields, while canary and rollback materialize the operator states needed for mixed-version convergence without claiming real fleet soak evidence.
Verification:
- scripts/test_internode_grpc_ab_bench.sh
- bash -n scripts/run_internode_grpc_ab_bench.sh scripts/test_internode_grpc_ab_bench.sh
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
test(perf): add pinned paired abba bench
Add a backlog#1432 paired benchmark orchestrator that runs MinIO/RustFS in an A1/B1/B2/A2 schedule while requiring immutable server identity, source revision or release, and ACK contract labels. The runner delegates actual workload execution to run_object_batch_bench_enhanced.sh so node metrics, resource captures, and provenance stay in the shared artifact format.
Verification:
- scripts/test_pinned_paired_abba_bench.sh
- bash -n scripts/run_pinned_paired_abba_bench.sh scripts/test_pinned_paired_abba_bench.sh
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
Fail startup when the configured console listener resolves to the same listener port covered by the S3 address. This keeps SO_REUSEPORT from hiding an intra-process S3/console endpoint collision.
Co-authored-by: heihutu <heihutu@gmail.com>
Ensure cancelled durable manual transition jobs expose a cancelled report both when worker results drain after a cancel request and when legacy persisted records are decoded.
Co-authored-by: heihutu <heihutu@gmail.com>
The background tier free-version recovery walk pinned a hardcoded 60s
total wall-clock timeout that overrides every operator knob, so any
bucket whose healthy full walk exceeds 60s fails forever; the failed
run's duration was also subtracted from the next 60s tick, restarting
the walk immediately and pinning CPU and disk I/O.
- Drop the total wall-clock budget on the recovery walk
(walkdir_timeout: Duration::ZERO) and inherit the operator-tunable
drive stall budget (RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS) for
per-call progress, so hung disks still fail fast.
- Back off failed runs from completion time: 60s doubling to a 600s
cap, reset on success; never subtract the failed run's duration.
- Add RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED (default true) to opt
out of the recovery worker on deployments with no remote tiers;
invalid values warn and fail open.
Fixes#5130
Co-authored-by: claude <claude@ehdtn.com>
Extend the inline fast-path mixed-msgpack E2E collector to capture msgpack/json decode-error counters by direction, message, and codec. Assert those counters remain unchanged for the multipart, part-number, SSE, compression, and transition fallback-control scenarios.
Co-authored-by: heihutu <heihutu@gmail.com>
When endpoints include remote nodes and no internode RPC secret is
resolvable, every remote format read fails client-side with "No valid
auth token" and startup dies ~2 minutes later with the misleading
"store init failed to load formats after 10 retries: erasure read
quorum" error (issues #4939, #5153).
Add a preflight to ECStore init that resolves the RPC secret once
before any disk opens: if any endpoint is non-local and resolution
fails, abort immediately with the operator-facing remediation message.
The preflight also emits the "RPC auth secret resolution failed" log
line so the log-analyzer rpc-secret-resolution rule keeps firing for
this scenario. Single-node (all-local) topologies never invoke the
resolver.
scripts/check_logging_guardrails.sh flags any lowercase secret-named
identifier interpolated into a format string. The static-secret-file test
added in #5245 interpolates two fixture bindings (file_secret, env_secret)
into format! when constructing the secret file and env var, tripping the
heuristic and breaking make pre-commit on every branch based on main.
Rename the bindings to file_key_b64 / env_key_b64 so they no longer match
the heuristic. The fixtures are dummy base64 key material written to a
temp file and env var, not log output, and the test coverage is unchanged.
The guard script itself is untouched.
docs/testing/security-regressions.md requires every fixed advisory to map to a
named, greppable regression test. Rename the tests added with the fixes to carry
their advisory id so `rg -i ghsa` finds them, and add the four rows to the
advisory -> test map.
Adds the FTPS MKD regression test that was missing. It primes the dummy backend
with a successful create_bucket, so the assertion distinguishes "denied at the
authorization boundary" from "backend refused" — without the queued success an
unconfigured create_bucket fails on its own and the test would pass even with
the authorization check removed. Verified it fails when the check is reverted.
DummyBackend gains a Debug impl (FtpsDriver's trait bounds require it) and a
queue_create_bucket_ok helper.
Records in the CI-execution map why ghsa_g3vq_* runs in the default pass despite
sitting behind the ftps feature: the rustfs crate defaults to ["ftps", "webdav"]
and cargo unifies features across the workspace build, so the test executes
there even though `cargo test -p rustfs-protocols` alone would skip it.
* fix(policy): quantify negated string conditions per value
ForAllValues:/ForAnyValue: negated string operators computed the positive
quantified match and then negated the aggregate. That yields NOT(all match)
and NOT(any match), which is the semantics of the *other* quantifier, so
ForAllValues:StringNotEquals and ForAnyValue:StringNotEquals were exactly
transposed. The same applied to StringNotEqualsIgnoreCase, StringNotLike,
ArnNotEquals and ArnNotLike.
Introduce an explicit Quantifier and push negation into the per-value
predicate for the qualified forms, so ForAllValues requires every request
value to satisfy the operator and ForAnyValue requires at least one.
Unqualified operators keep negating the aggregate, preserving AWS
single-valued-key semantics. Absent keys now follow AWS: ForAllValues is
vacuously satisfied, ForAnyValue is not.
A request value set that is fully contained in or fully disjoint from the
policy set cannot distinguish the two quantifiers, which is why the existing
cases missed this; the new tests use partially overlapping sets.
* fix(auth): keep request headers out of server-derived condition keys
get_condition_values folded every remaining request header into the policy
condition map. HeaderMap lowercases header names, and the server-derived keys
userid, username, principaltype, versionid and signatureversion are lowercase
too, so a header of the same name collided with them. The collision branch used
extend(), and non-quantified string operators match if ANY value in the vector
matches, so sending `userid: admin` was enough to satisfy a condition on
aws:userid. The jwt:/ldap: claim keys were reachable the same way whenever the
credential carried no such claim.
groups was worse than an append: the header loop ran before the cred.groups
block, which is gated on !args.contains_key("groups"), so a `groups:` header
both injected a value and suppressed the credential's real group list.
Resolve claims and group membership before merging headers, then skip any header
naming a key the server already derived or a well-known identity/context key.
The reserved set comes from KeyName so it tracks the key registry; s3:x-amz-*
keys stay mergeable because they mirror request headers by design.
* fix(access): gate the ListBucketVersions fallback on public-access checks
An anonymous request for ListObjectVersions that the bucket policy does not
grant directly falls back to re-checking the grant as s3:ListBucket. That
fallback returned Ok(()) straight away, skipping the two gates the direct grant
passes through: deny_anonymous_table_data_plane_if_needed and the
RestrictPublicBuckets check on the bucket's public-access block.
So a bucket whose policy allows anonymous s3:ListBucket kept serving anonymous
version listings after an operator enabled RestrictPublicBuckets, even though
the equivalent GetObject was correctly denied.
Fold the fallback into policy_allowed so both routes reach the same gates.
* fix(ftps): authorize MKD against the CreateBucket boundary
FTPS MKD creates a bucket but ran no authorization check, so any principal that
could open an FTPS session could create buckets regardless of policy. Every
other operation in this driver authorizes first — LIST, RETR, STOR, DELE and
RMD all call authorize_operation — and the WebDAV gateway checks
S3Action::CreateBucket on the equivalent path.
Add the matching check so MKD clears the same boundary as an S3 CreateBucket.
* fix(kms): do not include secret value in static KMS format error
The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().
* chore(security): add guardrails against secret values in error strings
Follow-up to the PR #5222 review finding fixed in PR #5243: a config value
that fails secret-format parsing is typically the raw secret itself, and
error messages are log content — they propagate via ? and are printed by
startup/error logging far from the construction site.
Three layers so this class of leak is caught earlier next time:
- security-advisory-lessons: state explicitly that error/panic messages are
log content; parse-failure hints must name the env var and expected
format, never echo the input. Add a matching review prompt.
- adversarial-validation: add a security-reviewer probe that greps
error-construction sites on secret-bearing config paths for interpolation
of the raw value, including the duplicated-parse trap where the leak hid.
- check_logging_guardrails.sh: mechanical backstop — flag inline format
interpolation of secret-named identifiers (secret/password/passwd/
private_key) in checked files; add crates/kms/src/config.rs (the twin
parse site) to the checked list. Verified the check passes on the fixed
tree and catches the original leak at rustfs/src/init.rs:399.
The malformed-format error message in build_static_kms_config embedded the
raw env var value. The most likely misconfiguration is setting
RUSTFS_KMS_STATIC_SECRET_KEY to the bare base64 key without the <key-name>:
prefix, in which case the full secret key material would be written to
startup logs. Drop the value from the message, matching the equivalent
parsing error in KmsConfig::from_env().
docs.rustfs.com dropped the .html suffix from page URLs, so
https://docs.rustfs.com/integration/tls-configured.html now returns 404.
Point both READMEs at the live canonical URL instead.
Supersedes #5154, which replaced the links with Wayback Machine
snapshots even though the docs site is alive and the page still exists.
* fix(ilm): evaluate complete version groups
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): log replication expiry blocks
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): diagnose incomplete noncurrent chains
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover purge-pending version groups
Add regression coverage for lifecycle-only version listing so purge-pending versions stay present for ILM evaluation while the public ListObjectVersions projection remains filtered.
Refs rustfs/backlog#1500
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): log lifecycle evaluation failures
Emit structured lifecycle_evaluation_failed events when version group loading or evaluation fails during immediate and existing-object expiry scans.
Refs rustfs/backlog#1503
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover incomplete noncurrent chains
Pin the fail-closed behavior for noncurrent expiration when successor_mod_time is missing and reuse a stable structured event name for that skip path.
Refs rustfs/backlog#1502
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): cover one-day noncurrent expiry boundary
Add a runtime regression test proving NoncurrentVersionExpiration Days=1 stays inactive before expected_expiry_time and deletes exactly at the computed due boundary.
Refs rustfs/backlog#1504
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): keep transition checks after incomplete expiry
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Retry peer tier config reloads after committed mutations so recovered nodes converge without requiring a second admin change.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures
Binds a domain-separated, length-prefixed canonical request-body digest into the
v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path
attacker on the default-plaintext internode channel can no longer tamper with a
mutation payload (or strip the msgpack `_bin` field to force the JSON fallback
decode) without invalidating the signature.
Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions,
WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile,
RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the
msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by
default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so
rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now
configurable and overflow fails closed with a metric.
Refs https://github.com/rustfs/backlog/issues/1327
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rpc): satisfy architecture-migration compat-marker guard
Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and
stop backticking env-var/metric names in the cleanup-register entry so the
guard's id extractor only sees the task-id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Cover async manual transition jobs that finish as partial because maxObjects truncates the scan, including terminal cancel idempotence and unchanged report counters.
Co-authored-by: heihutu <heihutu@gmail.com>
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env
Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before.
Refs https://github.com/rustfs/backlog/issues/1327
* ci: fix internode auth test lint failures
* perf(ecstore): cache internode sig strict env
---------
Co-authored-by: houseme <housemecn@gmail.com>
Add test coverage for the msgpack-only request/fleet confirmation truth table and exact request JSON policy manifest.
Pin request and response rollback behavior so removing either gate restores JSON compatibility while both gates enter msgpack-only for eligible fields.
Co-authored-by: heihutu <heihutu@gmail.com>
Update the internode gRPC benchmark P2 env output to require both the msgpack-only request flag and the fleet-confirmed gate before measuring a true msgpack-only phase.
Document the dual-gate rollback semantics and add a dry-run script test so the benchmark driver cannot silently regress to the single-flag flow.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards.
Co-authored-by: heihutu <heihutu@gmail.com>
test(internode): pin msgpack compat send sites
Add a checked test manifest for internode dual-encoded request and response payloads so node.proto _bin fields must be explicitly classified as msgpack-only eligible or always dual-write.
The manifest also pins the current JSON encoder call sites for request and response send paths, making future send-site drift visible before rollout.
Co-authored-by: heihutu <heihutu@gmail.com>
Cache the internode msgpack-only env gate after the first read so metadata RPC send paths avoid repeated environment parsing. Keep a reset hook for tests that intentionally change the env in-process.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep internode JSON compatibility fields unless operators explicitly confirm fleet-wide msgpack-only readiness. This prevents a single legacy rollout flag from emptying JSON fields in mixed-version clusters where older peers may still read the legacy JSON payload.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): bound manual transition duration
Add maxDurationSeconds handling for manual transition runs so operators can bound long scoped scans without changing the existing enqueue_only job contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ilm): document manual transition run limits
Document the enqueue_only manual transition contract, secret-handling guidance, and best-effort duration budget while pinning the new duration flag in the e2e response model.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): report manual transition backfill outcomes
Add scoped lifecycle transition backfill reporting for backlog #1478 and expose enqueue outcomes needed by #1479 without changing the existing scanner/compensation bool API.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(admin): add manual transition run endpoint
Add a bounded POST /rustfs/admin/v3/ilm/transition/run API for backlog #1477 and cover the route, policy, query parsing, and partial status contract needed by #1481. Console operations from #1480 are intentionally left for a later client integration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): allow manual transition resume markers
Accept additive marker and versionMarker parameters on the bounded manual transition run API so clients can continue from a partial report without changing the existing default scan behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): harden manual transition partial reports
Preserve the null-version cursor contract, stop manual scans on enqueue pressure without skipping the failed object, and keep raw resume markers out of admin JSON responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): add manual transition e2e coverage
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ilm): keep transition enqueue hot path direct
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A federated OIDC / AssumeRoleWithWebIdentity login derives parent_user from the external identity via session_identity() (username -> email -> sub -> oidc-user-unknown). At IAM request time a temporary credential whose parent_user equals the root access key is treated as owner (see rustfs/src/admin/auth.rs and the rustfs_iam owner resolution used by the policy engine and service-account paths). A federated identity whose display name happens to equal the root access key would therefore be silently granted full owner access on a string match alone.
Reject such issuance up front: immediately after parent_user is derived, and before credential generation, set_temp_user, and the site-replication hook, deny the binding when parent_user equals the root access key. Both federated flows funnel through the single DefaultFederatedSessionBinding::bind -> issue_credentials chokepoint, so this covers browser OIDC callback and AssumeRoleWithWebIdentity alike.
- rustfs/src/admin/service/federated_identity.rs: issue_credentials resolves the root access key via current_action_credentials() and rejects with FederatedSessionBindingError::InvalidRequest on collision. Added the pure helper parent_user_is_reserved(parent_user, root_access_key) so the collision decision is unit-testable without process globals; the comparison is exact and case-sensitive, mirroring the request-time owner comparison.
This is the federated-principal / root isolation pre-work (batch 0A, the immediate issuance-time block) from the OIDC review in rustfs/backlog#1437. Layer 2 (a signature-protected federated credential origin that forces is_owner=false at request time regardless of the parent_user string) is a separate follow-up. Already-issued colliding credentials continue on their existing TTL and should be revoked proactively. Legitimate federated identities are unaffected; only an exact collision with the configured root access key is denied.
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend
Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.
* fix(ecstore): guard encrypted multipart range seeks
* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix
Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
Stop the OIDC subsystem from writing credential-grade secrets into logs and returned errors.
- crates/iam/src/oidc.rs: redact sensitive header values (authorization, proxy-authorization, cookie, set-cookie) in format_http_headers, emitting only name and length; drop the raw request/response body from the DEBUG events (keep byte length); stop logging and stop splicing the raw token response body into the error returned on token_response_parse_failed (the TokenResponseBodyShape summary and length are retained); remove the now-unused format_http_body helper.
- rustfs/src/admin/handlers/oidc.rs: stop logging the raw authorization code and state on the code-exchange error path (code_len/state_len are kept).
This is the OIDC log/error redaction pre-work (batch 0B) from the OIDC review in rustfs/backlog#1437. It changes diagnostic content only; HTTP/STS status codes and legitimate request results are unchanged.
* fix(admin): confine service-account parent to caller scope (GHSA-5354)
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* test(admin): pin GHSA-5354 scope guard for derived credentials
The GHSA-5354 fix (#5141) added `add_service_account_parent_within_scope` and a
named-boundary test, but that test only covers the self-scoped case with
`req_user == req_parent_user`. A derived credential — a service account or STS
token holding CreateServiceAccountAdminAction — has `req_user` (its own key)
distinct from `req_parent_user` (its parent), which is the realistic attacker
shape and was left unexercised.
Add a test that pins the guard's allow set to exactly `owner || is_svc_acc`
across that derived-credential shape, including a derived non-owner aiming at the
root credential, so a later change to either the guard or the `is_svc_acc`
rewrite cannot silently let a derived non-owner escape its own parent's scope.
Also note at the check site that the guard is evaluated on the original
`target_user`, before the derived-credential rewrite to `req_parent_user`.
Refs GHSA-5354-r3w2-34m8.
PRs #5141 and #5142 fixed GHSA-5354 (service-account parent-scope enforcement) and GHSA-3ppv (versioned object-read authorization) but did not add the advisories to the security-regressions inventory that repo policy requires for every fixed GHSA. Add both rows to the advisory -> test mapping (advisory, class, fix PR, named regression test, layer) and list ghsa_5354_* / ghsa_3ppv_* under the default-CI unit-test execution map, since both tests live in the rustfs lib target and run in the default `cargo nextest run --profile ci --all --exclude e2e_test` pass.
get_object, the CopyObject source, and the UploadPartCopy source authorized reads that name an explicit versionId against s3:GetObject rather than s3:GetObjectVersion. A principal holding s3:GetObject but not s3:GetObjectVersion could therefore read historical object versions. get_object_attributes was already version-aware; these three paths were not.
Add a shared versioned_read_action helper that selects s3:GetObjectVersion when a version is named and s3:GetObject otherwise, and apply it to the three read/copy-source authorization sites. Add named regression test ghsa_3ppv_versioned_read_selects_get_object_version_action.
The ActionSet::is_match GetObjectVersion->GetObject compatibility mapping is intentionally retained for now; its removal is gated on a follow-up audit of the remaining version-aware read paths (HeadObject, GetObjectAcl, tagging). It does not affect this fix, which only prevents a GetObject grant from satisfying a versioned read.
Refs GHSA-3ppv-fx5m-m749.
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* fix(site-replication): replicate object-lock retention and legal hold
Object Lock did not survive site replication, so a WORM-protected object was
either missing from the peer entirely or present there unprotected. Two
independent defects caused this.
1. The site-replicator service account policy granted no object-lock actions.
Any replicated object carrying x-amz-object-lock-mode /
x-amz-object-lock-retain-until-date was rejected by the peer with
AccessDenied, so an object created under a retention rule never replicated
at all. Grant s3:GetObjectRetention, s3:PutObjectRetention,
s3:GetObjectLegalHold and s3:PutObjectLegalHold. s3:BypassGovernanceRetention
is deliberately NOT granted: replication must never be able to erase a
retained version on the peer.
2. PutObjectRetention and PutObjectLegalHold only rewrote object metadata. The
object PUT path is what normally computes a replication decision and
schedules replication, so a lock applied after upload stayed local and the
peer kept its previous, unprotected state. Schedule replication explicitly
after both metadata writes.
Verified live on a two-site cluster: an object created under a COMPLIANCE
retention rule now replicates, the peer copy reports COMPLIANCE, a
version-scoped delete on the peer is refused, and the content matches by MD5.
Retention and legal hold applied after upload propagate to the peer within
about 15s, and the replicated copy is likewise undeletable. No AccessDenied
replication errors remain.
Adds a regression test asserting the policy allows the four object-lock actions
and still denies governance bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(site-replication): persist pending marker before object-lock metadata commit
Review follow-up on the object-lock replication fix.
PutObjectRetention and PutObjectLegalHold scheduled replication but did not
persist a pending replication marker first, unlike the canonical object PUT
path. If the process restarted between the metadata commit and the replication
worker write-back, the in-memory task was lost and the object carried no
Pending/Failed marker, so the scanner heal skipped it and the peer silently
stayed unprotected.
Both handlers now mirror the PUT path ordering: compute the replication
decision once BEFORE the commit, fold SUFFIX_REPLICATION_STATUS and
SUFFIX_REPLICATION_TIMESTAMP into the eval_metadata passed to
put_object_metadata, then schedule after the commit. eval_metadata is merged
into the object metadata, so the lock fields are preserved. When the object
info cannot be read the decision is empty and nothing is scheduled.
Also aligns the two handlers to build their options the same way, and notes at
both schedule sites that this triggers a full object re-upload because
metadata-only replication is not implemented yet.
Adds an integration test asserting both handlers compute a replication decision
exactly once, so a regression that drops the scheduling is caught.
Verified live on a two-site cluster: applying retention while the peer is down
now leaves Replication Status: PENDING on disk at commit time, and once the
peer returns the lock propagates in about 15s and the status becomes COMPLETED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On macOS with DNS interception services, .invalid TLD domains resolve
to an interception address (e.g. 198.18.16.173) instead of failing DNS
resolution. The health probe then classifies the error as Unreachable
rather than DnsFailure. Accept both outcomes since both are correct
non-reachable error classifications.
Keep remote peer topology slots observable when peer client construction cannot build a dialing client, and publish admin server_info cache/failure state only after a complete probe round.
Covers rustfs/backlog#1426 and rustfs/backlog#1430.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(admin): use raw topology host for peer mapping
Backlog: rustfs/backlog#1427
Keep endpoint topology membership keyed by raw host:port when mapping peers to grid hosts, while preserving DNS resolution only for the PeerRestClient dialing host.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): reconcile v3 info with topology
Backlog: rustfs/backlog#1425
Backlog: rustfs/backlog#1428
Synthesize additive unknown server rows from configured endpoint topology before v3 backend counters are computed, and add exact drive-identity coverage tests so balanced totals cannot hide a missing member.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(admin): collapse topology reconciliation
Build a single endpoint topology index for v3 admin server reconciliation and completeness reporting, and reuse the raw peer/grid mapping when constructing peer clients.
Backlog: rustfs/backlog#1424
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Update selected workspace dependencies and lockfile entries.
Keep async-nats and rcgen on explicit feature sets while preserving the RustFS targets and TLS test surfaces.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): gate exact remote version consumption
Reject non-empty remote tier versions before transitioned GET and remote delete backend I/O when the tier backend does not support exact version operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(tier): split mock remote version validation fault
Separate one-shot mock remote version validation failures from persistent unsupported-backend behavior so cleanup durability tests can still verify exact-version recovery while #1358 fail-closed gate tests keep asserting no backend I/O.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Advance transition transactions to UploadOutcomeUnknown before remote tier PUT so response-loss windows can be recovered through provider-authoritative probing.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep coordinator Prepared intents durable when peer abort recovery fails, then make reload retry abort or commit based on the persisted tier config digest.
Co-authored-by: heihutu <heihutu@gmail.com>
A single instance with a malformed `enable` value (e.g. the typo `enable`
instead of `enabled`/`on`) made every configured notification/audit target
of every type fail to load. `create_targets_from_config_with_store_mode`
collected instance configs through the strict `try_collect_target_configs`,
whose `.collect::<Result<Vec, _>>()` short-circuits on the first error and
drops all siblings; the surrounding `?` then aborted the loop over every
plugin type.
This contradicted the function's own documented contract ("Creation is
fault-isolated per instance") and was inconsistent: target construction
failures were already isolated into the `failures` accumulator, only config
collection failures were fatal.
Add `collect_target_config_results[_from_env]`, which returns the enabled
`(id, config)` pairs plus a per-instance failure summary, and use it in the
create path. Per-instance collection errors now flow into the same
`failures`/`creation_failures` accumulator as construction failures: healthy
siblings and other target types still load, while the malformed instance is
surfaced so the notify lifecycle stays non-converged/retryable and an Admin
write cannot report a false success. Audit likewise keeps logging through
its valid targets instead of going fully Stopped.
Follow-up to #5088.
* feat(tier): add transition candidate probe contract
Add a fail-closed WarmBackend probe contract for provider-authoritative transition candidate state. Default providers report Unsupported, while the shared mock backend can now model missing, unversioned, and exact-version candidates for follow-up recovery tests.
This is a forward-compatible foundation for #1352/#1358 recovery work and does not change production cleanup behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(tier): probe transition candidates from providers
Implement provider-authoritative transition candidate probing for S3-family warm backends by querying ListObjectVersions with exact-key filtering and fail-closed classification for delete markers, multiple versions, truncation, and unknown versioning state.
This keeps non-S3 providers on the default Unsupported probe result and forwards MinIO, RustFS, and R2 through the S3 probe implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A metacache listing that misses quorum purely because the volume or path
is absent on a quorum of drives (every drive reports VolumeNotFound or
FileNotFound) is a benign, expected outcome: list_path_raw already returns
VolumeNotFound/FileNotFound and lets the caller decide how to react.
The common trigger is a startup race where the system bucket (.rustfs.sys)
is not yet created on every drive when an early reader such as the IAM
config loader lists config/iam/. Logging that at error prints a scary
message during normal boot and, in #5076, misled a user into blaming it
for unrelated upload failures.
Add is_benign_not_found_listing_failure() and demote the pure not-found
case to debug (state = "quorum_not_found"), keeping error for listings
that failed for a real reason (I/O, timeout, corruption). Return value and
control flow are unchanged. Add a unit test for the classifier.
Refs: #5076
Bind provider exact GET and DELETE capability to tier operation leases and reject non-empty remote versions before committing transition metadata when the provider cannot address exact versions.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep LocalCommitStarted transition transaction records retained when the local source cannot prove the commit, so recovery does not classify the record as failed or delete the remote candidate without cleanup proof.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep recovered tier mutation blocks installed until the local publish transition has atomically established draining for the affected tiers, so old-generation leases cannot slip in after peer commit replay and before local publish.
Delay committed intent cleanup until local publish succeeds. If peer replay succeeds but local publish fails, the durable committed intent remains available for retry and the runtime block stays fail-closed.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix: fail closed legacy tier mutations without etag
Reject non-add tier config mutations when the loaded durable snapshot has no current config ETag. This keeps legacy JSON/no-ETag paths from being saved as creation-style updates before coordinator intent state can bind an old config revision.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): allow empty clear without config etag
Allow empty tier clears to initialize the binary config under the namespace coordinator lock while keeping non-empty legacy config mutations fail-closed when no current ETag exists.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation control client
Add signed tier mutation prepare, commit, and abort client calls for peer fanout while preserving the existing protobuf and RPC contract.
Verify response proofs before interpreting peer outcomes, fail closed on invalid states, and reject oversized payloads before dialing peers.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Add machine-readable storage, listing, usage, and workload component diagnostics to the admin cluster snapshot response while keeping the external S3 API untouched.
Co-authored-by: heihutu <heihutu@gmail.com>
The peer health snapshot now reports CapabilityState::Supported for
local nodes (no probing needed), but the cluster_snapshot test still
expected Unknown. Update the assertion to match the new behavior
introduced in commit 7805cf5ae.
Add idempotent terminal transition handling for tier mutation intents and a compare-and-swap record update helper backed by config-object ETags. Cover commit and abort retries, conflicting terminal updates, stale ETag rejection, and exact-limit scan pagination.
Co-authored-by: heihutu <heihutu@gmail.com>
Add canonical record-object naming and crate-private save, load, and idempotent delete helpers for tier mutation intents. Cover malformed record keys, mismatched persisted mutation IDs, and an ECStore config-object round trip under test-util.
Co-authored-by: heihutu <heihutu@gmail.com>
Add an internal durable tier mutation intent model for the #1357 distributed fencing work. The new model validates schema, checksum, canonical targets, mutation-specific target identity shape, config ETags, expiry, and prepared-to-terminal state transitions without changing the production mutation path yet.
Co-authored-by: heihutu <heihutu@gmail.com>
Retry cleanup-pending transition transaction recovery after restart, keeping committed remote bodies when local metadata already references them and preserving records when remote cleanup fails.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): accept null version transition sources
Treat nil/null source version IDs as null-version transition sources when building transition transaction source identity, while preserving fail-closed validation for truly missing IDs on versioned sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): preserve multipart source versions
Propagate bucket versioning into CompleteMultipartUpload, assign a concrete version ID for versioned multipart completions, and classify unversioned FileInfo sources as null-version transition sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): normalize multipart completion versions
Treat nil staged version IDs as missing for versioned multipart completion and clear staged version IDs when completion publishes a null-version object under suspended or unversioned semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): refresh workspace dependencies
Refresh compatible workspace dependencies while preserving the requested
version pins. Update async-nats and Hyper, and replace the temporary Hyper
Git patch with the released crate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): bump hotpath to 0.21.5
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): inject local deployment into replicate-add payload when omitted
The web console's "Set Up Site Replication" flow posts only the remote peer(s) to
/rustfs/admin/v3/site-replication/add and omits the local deployment. The add
preflight requires the local deployment to be present
(validate_add_preflight_topology), so the console's request failed with
"site replication add request must include the local deployment" and no
replication was configured from the web UI.
Inject the local site into the sites list when the payload does not already
include it (matched by endpoint identity), before validation. `mc admin
replicate add` always sends every site, so this is a no-op for the CLI; the
local site carries no credentials (validate_add_sites already skips credential
checks for it).
Adds unit tests for inject-when-missing and no-op-when-already-present.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Restore the workspace reqwest default feature stack for RustFS outbound HTTPS clients, while keeping per-crate extra APIs such as json, stream, and multipart explicit. Lazily initialize the notification runtime from admin target access when RUSTFS_NOTIFY_ENABLE=true is already effective, and add regression coverage for HTTPS webhook custom CA handling and target-list visibility.
Fixes#5052.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): count replicated buckets as in-sync in replicate status
`mc admin replicate status` reported "0/N Buckets in sync" with a cross mark for
every bucket even when replication was healthy and objects had propagated.
build_sr_info stores each bucket's replication_config in the wire form as
base64-encoded XML (raw_config_to_base64), but site_replication_config_mismatch
XML-parsed that string directly without base64-decoding. The parse always
failed, so every bucket with a replication config was reported as a config
mismatch (replication_cfg_mismatch = true), which the status endpoint and mc
render as out-of-sync.
Decode the wire value (tolerant base64 decode, falling back to raw bytes) before
XML-parsing. Adds a regression test that feeds the base64 wire form used in
production; the existing tests only exercised raw XML, which masked the bug.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
fix(site-replication): report peer sync state in replicate info and surface pre-existing back-fill failures
BUG 1 — blank/Unknown Sync in `mc admin replicate info` and the console:
The info endpoint (SiteReplicationInfoHandler) serializes the persisted peer
map, whose sync_state is constructed as Unknown and never updated; the health
derivation added earlier only runs in build_status_info (the status endpoint).
Persist sync_state = Enable for peers on the add and join enable paths
(promote only Unknown, never clobber an explicit Disable). build_status_info
still refines live health for the status endpoint, so info/status/console and
peer_states all report a real sync state for a healthy peer.
BUG 2 — pre-existing-bucket back-fill failures were silently swallowed:
backfill_existing_buckets_after_add returned () and the add handler reported
success with an initial_sync_error_message that only ever carried
metadata-bootstrap errors, so a pre-existing bucket that failed to propagate
was invisible. Return per-bucket failures (including the previously silent
Ok(false) runtime-unavailable no-op and the versioning-failure drop), fold
them into the add response via compose_initial_sync_error_message, and log
reverse-direction gaps on the join path.
Adds unit tests for sync_state promotion (and Disable preservation) and for
back-fill failure surfacing.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Point the cla-bot config and the PR template at cla/v2.md so new
contributions are checked against the v2 Individual CLA.
- .github/cla.yml: document.version v1 -> v2, url -> cla/v2.md
- .github/pull_request_template.md: CLA link -> cla/v2.md
Commit aec2ee9ec (#5005) enforced that RPC secrets cannot be derived
from the public default credentials. This broke all multi-node cluster
E2E tests that relied on the constructor's DEFAULT_ACCESS_KEY /
DEFAULT_SECRET_KEY defaults.
Move the non-default credential and RUSTFS_RPC_SECRET blanking into
the RustFSTestClusterEnvironment constructor so every cluster test
starts with a valid RPC secret derivation base. Remove the per-test
overrides in cluster_multidrive_pool_test that were added as a partial
fix in #5005.
Add a deterministic transition matrix case for a stale prepared metadata snapshot followed by a committed transition, duplicate transition, and late GET. The test proves the stale metadata generation is physically reclaimed, duplicate transition does not upload a second remote candidate, and the late GET reads the committed remote body byte-for-byte instead of reopening the released local source.
Refs rustfs/backlog#1355
Co-authored-by: heihutu <heihutu@gmail.com>
Generate a unique Pulsar producer name for each producer instance while preserving the target type and target id context. This avoids reload-time collisions when an old producer is still connected while a new destination is initialized.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): add fallible erasure construction
(cherry picked from commit bd148b20f7)
* fix(ecstore): resolve storage parity per pool
(cherry picked from commit c05c2cb24b)
* fix(ecstore): keep carved per-pool parity core self-contained on main
Fixups so the cherry-picked fallible-erasure + per-pool-parity core builds standalone on current main without the excluded scope-creep commits:
- runtime/sources.rs: re-add backend_storage_class_parities (removed by the per-pool commit; its rebalance caller was updated in an unrelated reporting commit that was left out). Reimplemented over the snapshot API, behavior-identical.
- config/mod.rs: rename the storage-class publish test module (main independently added a mod tests, so the cherry-pick collided).
- rustfs storage_api.rs + startup_storage.rs: route the storage-class ENV consts through the startup storage facade and use a local const for the erasure-set-drive-count env name, satisfying the layer/facade guardrail (main's guardrail is stricter than when the core was authored).
* fix(ecstore): use struct-init in erasure test helper to satisfy clippy field_reassign_with_default
The cherry-picked fallible-erasure commit's `erasure_with_invalid_dimensions` test helper built `Erasure` via `default()` then reassigned fields, which trips `clippy::field_reassign_with_default` under `-D warnings` (only surfaced by `--all-targets`, which lints test code). #4977 fixed this in a later commit that was not part of the carved core. Use struct-init with `..Default::default()`, matching #4977's final form.
* fix(rustfs): gate the test-only storage-class ENV facade re-export behind cfg(test)
The ENV constants (INLINE_BLOCK_ENV/OPTIMIZE_ENV/RRS_ENV/STANDARD_ENV) re-exported through the startup storage facade are only consumed by a #[cfg(test)] test in startup_storage.rs, so in a non-test lib build the re-export is unused and trips -D unused-imports under clippy --all-targets. Gate it with #[cfg(test)], matching #4977's final form.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
The normative erasure-coding spec stated several target behaviors and known baseline defects as active invariants, and cited two symbols that do not exist on main. Each correction below was verified against the code.
- §4.1 shard size: legacy even-padding is RustFS-legacy-only and is NOT MinIO's sizing. MinIO (and the modern GF(2⁸) path) use plain div_ceil; for 1 MiB / 6 data shards that is 174763 bytes versus the legacy even-padded 174764. MinIO-migrated data is decoded by the modern path, so the legacy formula never applies to it.
- §6.3 / §11 MTime: the MTime key is always written (a None mod_time encodes as UNIX_EPOCH = 0), not omitted. Round-trip safety to None is enforced on the decode side, not by write omission. Only the legacy StatInfo.ModTime field is omitted-when-None.
- §7 commit: rollback after a missed write quorum is best-effort — undo failures are counted and warn!-logged, never propagated or retried — so a partially-failed rollback can leave shards behind; softened "never left partially committed" accordingly.
- §7 data_dir: reduce_common_data_dir votes over each disk's old_data_dir (a GC input for reclaiming the superseded dir), not the newly committed data_dir.
- §7 convergence: classify_rename_convergence is consumed only by the multipart-complete path (convergence.needs_heal()); the regular put_object path discards it.
- §7 write layout: removed the WriteLayout / resolve_write_layout citation — neither symbol exists on main (the write layout is computed inline), which violated the spec's own cite-real-symbols rule.
- §6.6 inline: inline presence is read from the meta_sys[inline-data] body marker alone; the header InlineData flag is written but not consulted on read, and disagreement is tolerated (MinIO may write the flag unset with inline data present). Removed the false "must agree" invariant.
- §11 UUID tolerance: version_id / data_dir (ID/DDir) decode as a fixed 16-byte Uuid::from_bytes and fail closed on a present-but-non-16-byte value; only transitioned-versionID uses the tolerant Uuid::from_slice(...).ok().filter(!nil). Split the previously-conflated bullet.
- §11 decode blanket: narrowed "everything else must degrade to a tolerant default" to enumerate the legitimate structural/geometry/version/length fail-closed guards (header array length, unknown header_ver, version > max, versions_len / bin_len bounds, non-16-byte ID/DDir).
- §8 / §13 codec guard: has_valid_dimensions() is a &self method, so it runs after construction and reliably covers only block_size == 0; a data_blocks == 0 geometry with parity > 0 panics in the .expect constructor before the guard runs. Noted the fallible-constructor fix.
Refuted and intentionally left unchanged: the "accepts container major == 1 with any minor" statement is correct — the reader compares only major (rejects major > 1) and never gates on minor, so minor 4 is accepted.
A duplicate transition task admitted after the winner released its
in-flight claim (#4839) re-reads the version before uploading, but on
unversioned buckets that read could hit a stale pre-transition entry in
the 2s-TTL GET metadata cache: transition_object never invalidated the
cache after delete_object_version persisted transition_status=complete
and freed the local data. The stale hit defeated the TRANSITION_COMPLETE
early-return, so the duplicate streamed the already-deleted local data
to the remote tier (NotFound reader errors + rejected duplicate tier
PUT with UnexpectedContent).
Invalidate the cache right after the transitioned metadata is persisted,
matching the other metadata-mutating paths, and add a regression test
that runs a duplicate transition against an already-transitioned version
and asserts no second tier upload and unchanged remote object metadata.
Fixes#4827
* docs(architecture): add normative erasure-coding algorithm and on-disk compatibility contract
Adds docs/architecture/erasure-coding.md as the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and the on-disk (xl.meta) / decode compatibility contract every future change must preserve. Grounded in the baseline (main) implementation with file:line anchors; cross-links (does not duplicate) the existing placement, MinIO-format-compat, layout-boundary, decommission, and tier-ILM docs, and the AGENTS.md cross-cutting invariants.
Covers: Reed-Solomon over GF(2^8) modern vs GF(2^16) legacy backends and how each is selected; pool/set/drive geometry with the 2..=16 set-size and per-pool parity invariants; the key-derived distribution permutation (1..=N); 1 MiB block size and the modern/legacy shard-size formulas; HighwayHash256S interleaved bitrot layout; the full xl.meta container/header/version-body schema and internal dual-key convention; write/read/heal quorum rules; and, newly codified as a first-class contract, the decode-tolerance invariants (nil-UUID/epoch-mod_time to None, skip unknown fields, hard-guard only length-critical arrays, tolerate the negative actual_size compressed sentinel, tolerate a malformed transitioned-versionID). Linked from the architecture README under "Contracts & invariants". Docs-only; check_doc_paths.sh passes.
* docs(architecture): anchor erasure spec by symbol name, not line numbers
Line numbers rot as code changes and are not validated by check_doc_paths.sh, so they would silently mislead the very code changes this normative spec is meant to guide. Replace all file:line citations with file-path + symbol-name references (functions/consts/types are greppable and rename only on deliberate changes; format byte offsets are kept). Add an explicit "references work here" note and a §13 rule that governed changes must update this spec in the same PR.
* docs(architecture): correct erasure spec after multi-expert adversarial review
Four independent adversarial reviewers fact-checked every claim against the code. Fixes:
- CRITICAL (found independently by two reviewers): §1 mislabeled the GF(2^16) reed-solomon-simd "legacy" backend as the reader for "older MinIO-lineage format". It is the opposite — that backend serves RustFS's own older main-branch (rmp_serde, uses_legacy_checksum) objects; MinIO-migrated data uses the same rs-vandermonde GF(2^8) scheme and is decoded by the modern backend. The old wording contradicted §1's own MinIO-interop line, §12, minio-file-format-compat.md, and the source comments, and would have misled the highest-stakes decode-routing decision.
- §2.1/§12: set size 2..=16 holds for multi-drive layouts; single-drive deployments run at N=1 (parity 0) outside is_valid_set_size.
- §2.2: validate_parity_inner enforces parity <= N/2 only for N > 2 (user storage-class parity flows through it); the standalone validate_parity is unconditional but only applied to the resolved default.
- §6.2/§12: header format is dispatched by header_ver; array length (4/5/7) is a per-version validation, not the discriminator.
- §6.3: the part-array length guard applies on the all_parts decode path.
- §6.5: get_bytes matches only the two canonical lowercase keys; only is_internal_key/get_str are case-insensitive.
- §6.5: transitioned-versionID — state the load-bearing invariant (non-16-byte decodes to None, never fatal); string-form recovery is optional, and transitioned-xl.meta interop is out of scope.
- §11: typo RustSF -> RustFS.
All other claims across §1-§14 were verified accurate against code (distribution formula, quorum formulas, on-disk key set/endianness/markers, decode-tolerance invariants, version anchors, standard references).
fix(s3): populate CopyObject response checksums and persist them (#4996)
A CopyObject that requested ChecksumAlgorithm=SHA256 applied the copy but returned no CopyObjectResult.ChecksumSHA256, and a later checksum-mode HEAD/GET on the destination returned nothing: execute_copy_object never read the requested algorithm, never gave the destination write a checksum, and built CopyObjectResult with ..Default::default() (all checksum fields None).
Give the destination object a checksum on the copy path. When the caller requests an algorithm, compute it fresh over the copied plaintext (the hasher sits on the innermost reader so it digests plaintext, before compression/encryption wrap it) so it persists into the object's checksum and a checksum-mode HEAD/GET returns it. When no algorithm is requested, carry the source object's stored checksum over unchanged — the copy does not transform the plaintext, so re-hashing would be wasted work and would flatten a multipart composite value. Fill CopyObjectResult from the persisted destination checksum decoded exactly the way GetObject/HeadObject do, so the response value is identical to a later checksum-mode HEAD/GET.
Add e2e regression tests: requested SHA256 is computed, returned, and HEAD-consistent; a no-algorithm copy preserves and reports the source SHA256; and a requested CRC32 over a SHA256 source is computed fresh (equals a reference CRC32 PUT), overrides the source algorithm, and is not inherited.
Fixes#4996
test(1306): pin Some(0) usage serialization, snapshot cache invalidation, and listing send classification
Follow-up test hardening for the merged admin-usage-snapshot work
(#4979/#4980/#4981/#4982, rustfs/backlog#1306). Tests only; no production
behavior change.
- B-1 madmin: pin that a scanned-but-empty bucket (Some(0)) serializes usage
stats as zeros, staying distinct from the no-snapshot (None) omitted case.
- B-2 gating: revert detector proving save_data_usage_in_backend invalidates
the 30s snapshot cache so a fresh save is visible to the next cached read.
- A-1 list_objects: pin that a successful gather_results send is never
misclassified as ConsumerGone (correct state + err sentinel delivered), and
document the wrapper Err arm invariant. A full wrapper-level producer-error
integration test is deferred as it needs the fake-disk list harness.
DeleteRestoredAction is supposed to demote a restored object back to its
pure transitioned state: remove only the local restored copy, strip the
x-amz-restore headers, and leave the version (and the remote tier data)
untouched. expire_transitioned_object set
opts.transition.expire_restored accordingly, but no delete path ever
read the flag, so delete_object ran an ordinary delete: on unversioned
buckets the whole object vanished and the free-version record scheduled
remote tier cleanup (tier data loss); on versioned buckets the latest
version got a spurious delete marker that replication propagated.
Route expire_restored explicitly in SetDisks::delete_object before
delete-marker resolution and replication dispatch: target the found
version with FileInfo.expire_restored=true and return early. The
FileMeta::delete_version layer already implements the semantics (strip
restore headers, keep the version, hand back the local data dir); this
wires it up.
Also fix the action matching in expire_transitioned_object (extracted
into transitioned_object_delete_opts): DeleteRestoredVersionAction
previously fell through to the full transitioned-object delete, which
removed the remote tier data of a noncurrent restored version. It now
routes through the same restored-copy cleanup with the exact version id,
matching MinIO's Action.DeleteVersioned()/DeleteRestored() dispatch.
Re-enable test_restore_chain_local_read_expiry_keeps_remote_and_allows_
re_restore in the ILM Integration (serial) lane; add unit tests pinning
the event->options routing and the filemeta expire_restored branch.
Closesrustfs/backlog#1302
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Replace stale dependency-depth and line-count snapshots with a domain overview based on the current Cargo workspace. This preserves the high-level architecture while avoiding references to removed crates and fragile size estimates.
Constraint: Workspace membership changes independently of architecture documentation updates
Rejected: Refresh the old numeric snapshots | they would immediately become stale again
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Cargo.toml as the source of truth for workspace membership
Tested: git diff --check; scripts/check_doc_paths.sh; cargo metadata --no-deps --format-version 1
Not-tested: make pre-commit (documentation-only change)
* fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
* fix(auth): POST-object lock fields and signed metadata=true listing 403s
Two authorization surfaces returned 403 for allowed requests (rustfs#4845): the PutObject access hook demanded PutObjectRetention/PutObjectLegalHold IAM actions for POST-object form uploads whose lock fields are governed by the validated POST policy, and the metadata=true listing route never resolved SigV4-verified credentials into ReqInfo so signed requests were evaluated as anonymous. Skip the PUT-header lock actions for POST form uploads and resolve credentials in the metadata route; re-admit the four e2e tests to the e2e-full profile.
Follow-up to backlog#1191 (optional sub-item). The main accept loop
spawned one task per socket with no global bound, so a connection flood
could exhaust file descriptors and memory and take existing traffic
down with it.
- New RUSTFS_API_MAX_CONNECTIONS (default 0 = unlimited, no semaphore
constructed, accept loop unchanged).
- When set, the loop acquires an owned semaphore permit BEFORE
accept(): at saturation it simply stops accepting and lets the
kernel backlog absorb bursts (TCP-native backpressure) instead of
accept-then-close churn. The permit moves into the connection task
and is released by RAII on any exit path, including TLS handshake
failures.
- Saturation is observable via the
rustfs_http_server_connection_cap_saturated_total counter, a
connection_cap_state startup event, and a per-wait debug event;
shutdown stays responsive while parked on the semaphore.
- The cap covers everything on the main listener (S3, admin, console,
internode gRPC); the constant docs carry sizing guidance.
- E2E coverage: ten sequential Connection-close requests against cap 2
prove permits never leak; two stalled connections saturating cap 2
leave a third unserved until their permits are released, after which
the queued request is accepted and answered.
POSIX lets rmdir report a non-empty directory as either ENOTEMPTY or EEXIST. Linux/macOS/Windows use ENOTEMPTY (ErrorKind::DirectoryNotEmpty); illumos/Solaris return EEXIST (errno 17), which Rust surfaces as ErrorKind::AlreadyExists and which a DirectoryNotEmpty match never catches.
LocalDisk::delete_file removes xl.meta and then recurses to rmdir the object directory, tolerating only NotFound and DirectoryNotEmpty. Since #4300 (transactional delete rollback-staging, new in beta9) the object directory still holds the rollback backup dir when that rmdir runs — the caller removes it only after write quorum is confirmed — so the rmdir legitimately reports "not empty". On Linux that is tolerated; on Solaris it is EEXIST, which fell through to the catch-all arm and became FileAccessDeniedWithContext. That failed the delete commit, rolled the metadata back, and left the object undeletable, so the client retried indefinitely with a spurious FileAccessDenied and no EACCES anywhere (rustfs/rustfs#4978). The same Linux-errno assumption also broke non-force DeleteBucket on a populated bucket on Solaris.
Add a portable is_dir_not_empty_error classifier (DirectoryNotEmpty kind plus raw ENOTEMPTY/EEXIST), mirroring MinIO's isSysErrNotEmpty, and use it at the two directory-removal sites via is_benign_object_rmdir_error (delete_file) and classify_delete_volume_error (delete_volume). The classifier is applied only at rmdir/remove_dir_all sites, where EEXIST unambiguously means "not empty", so EEXIST keeps its normal meaning everywhere else. rmdir never returns EEXIST on Linux/macOS/Windows, so the new raw-errno branch is unreachable there and the change is a strict no-op off illumos/Solaris.
Adds unit tests for the classifier (DirectoryNotEmpty/ENOTEMPTY/EEXIST match, EACCES/ENOENT reject, real non-empty rmdir against the host errno) and call-site decision tests that make a Solaris EEXIST regression detectable on Linux CI.
Fixes#4978
For a CopyObject whose source carries versioning, the response must echo the exact source version copied via x-amz-copy-source-version-id (SDK CopySourceVersionId), kept distinct from the newly created destination x-amz-version-id. RustFS set the destination version_id but left copy_source_version_id unset, so AWS SDK clients could not prove which source version was copied — the same field UploadPartCopy already populates.
Populate CopyObjectOutput.copy_source_version_id from the version actually read from the source, gated on the source bucket carrying versioning (enabled or suspended) and rendering the null version as "null", mirroring the GET/HEAD convention. Add an e2e regression test that copies a non-latest source version and asserts the source-version header equals the requested version, the destination header is a distinct new version, the destination bytes/size match the copied version, and the source version remains present.
Fixes#4976
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303)
The excluded test's 'missing xl.meta ... on disk2' was NOT an EC
metadata-distribution issue: after a transition all four shard disks hold
a fully consistent xl.meta (verified by decoding each shard). The panic
came from the tier test util's open_disk, which hardcoded disk_index 0
for every disk path; LocalDisk::new validates the endpoint's
(set_idx, disk_idx) against the disk's own format.json and rejected every
non-slot-0 disk with InconsistentDisk, which read_transition_meta
collapsed into 'missing xl.meta'. Derive the real indices from
format.json instead. This also un-breaks free_version_count /
wait_for_free_version_absence for non-first disks (silently 0 before).
With that fixed, the test advanced to the #4877 restore self-deadlock,
whose main paths #4886 already fixed. Complete that fix on the one path
it missed: update_restore_metadata (the restore-failure metadata
rewrite) still rebuilt copy_object options with no_lock=false and would
re-acquire the object write lock the restore handler already holds.
Propagate the caller's no_lock there too.
Remove the test from the serial-lane exclusion list; the four remaining
exclusions are unrelated known issues and stay.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP
dimension gives per-client fairness; this adds a collective per-bucket
budget so one hot bucket cannot monopolize the server regardless of how
many client IPs the traffic is spread across.
- Generalize RateLimiter over its key type with a Borrow-based check()
so &str lookups against String bucket keys stay allocation-free on
the hit path; client-IP call sites are unchanged in behavior.
- RateLimitLayer now carries optional client and bucket limiters; a
request must pass every configured dimension, and rejections report
which one tripped via a new 'dimension' metric label.
- Bucket extraction mirrors s3s host routing: virtual-hosted-style
resolves the Host/authority prefix against the same expanded domain
set (with port variants) the s3s router uses; otherwise the first
path segment. Admin and table-catalog namespaces are never buckets.
- New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 =
dimension off) under the existing enable switch; bucket-only
configurations (client RPM 0) are supported.
- Bucket names are attacker-chosen, so the bounded-shards design
(100k keys, lossless idle sweeps, most-idle eviction) is the memory
defense; a test floods 10k random names and asserts the cap holds.
- Unit tests for extraction, shared bucket budgets across IPs, both-
dimensions interaction, and the extended env matrix; e2e test proves
bucket-only throttling on the real server with an unrelated bucket
unaffected.
fix(api): return descriptive InvalidArgument reason for Windows-unsupported object keys
On Windows hosts object keys containing NTFS-reserved characters or
Win32-unaddressable path segments are rejected up front, but the client
only saw a bare "Invalid argument" (issue #3299). Attach an explicit
reason to these rejections and surface non-empty InvalidArgument reasons
through the S3 error message.
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
Add Reuse Before You Write and Necessary Code Only sections to AGENTS.md,
promote quality probes into a dedicated seventh adversarial-validation
role (simplicity adversary), extend rust-code-quality checks, and dedupe
code-change-verification's restated Rust checklist into a pointer.
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back
Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.
- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
(opaque, purpose-scoped object write lock with an is_lock_lost fence);
handle_restore_transitioned_object no longer locks the copy-back, so
HEAD/get_object_info stay non-blocking during a restore and the inner
put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
read-check-write (no_lock inside the scope), drops it before spawning
the copy-back; SELECT-type restores keep the plain read-locked path;
the copy-back pins the resolved version so a concurrent PUT cannot
strand the flagged version at ongoing=true; concurrent restores are
rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
integration test to the final semantics, and add a concurrent
double-POST test asserting exactly one acceptance and one tier GET.
PUT admission since #4928 rejected any request whose Content-Encoding declares
aws-chunked but lacks x-amz-decoded-content-length with 400 UnexpectedContent.
Whether the body is actually chunk-framed is signalled by a STREAMING-*
x-amz-content-sha256, not by the declared encoding: the s3s auth layer only
de-frames streaming payloads and already requires the decoded length for them,
so a declared-only aws-chunked request (issue #1857 clients) carries an
unframed body whose wire Content-Length is the authoritative object size.
Admit it against that length; keep failing closed for genuinely framed bodies
without a decoded length, and use the decoded length for streaming payloads
even when Content-Encoding is absent.
Refs: https://github.com/rustfs/backlog/issues/1336
Issue #4810's production incident showed the drive timeout knobs are
undiscoverable from symptoms: a listing that outruns the walk budget
gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs
existed only as code constants in crates/config/src/constants/drive.rs
with no operator-facing documentation.
Add docs/operations/drive-timeout-tuning.md covering the per-operation
drive timeout knobs, resolution precedence, the high_latency profile,
and a dedicated section on listing truncation and the walk stall
budget - including the correction that since the stall-budget rework
the foreground listing path is governed by
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob
the issue suggested. Link the guide from the README.
Ref #4810
docs(skills): single-commit release flow — preview and final tags share the validated commit
Version files are bumped once, directly to the final target version; the -preview.N suffix now exists only in tag names. The binary self-reports build::TAG and build.yml derives artifact naming and prerelease classification from the tag, so the preview tag and the final tag can point at the exact same commit — eliminating the post-validation version-bump commit that previously separated the validated hash from the released tag.
rustfs-release-version-bump gains a guard rejecting any -preview. target version.
test(e2e): add socket-level network fault-injection proxy (backlog#1325)
Add `FaultProxy`, an in-process async TCP proxy for black-box cluster E2E tests, under `crates/e2e_test/src/fault_proxy.rs`. It binds a random local port, forwards every accepted connection to a fixed target, and lets a test flip the wire behaviour at runtime.
Fault modes: `Pass` (transparent), `Latency(Duration)` (delay each forwarded chunk), `Blackhole` (accept but forward nothing either way and never respond), and `Partition(Direction)` (block exactly one direction while the other keeps flowing). Modes are switchable at runtime via `set_mode`, and `shutdown` stops the accept loop and drops the listener so the bound port is released.
This is the black-box counterpart to the in-process white-box hooks (rename barrier, disk call counters), which cannot reach across the process boundary into a spawned RustFS server. It serves the lock-plane one-way partition and accept-then-blackhole-peer acceptance for #1312/#1319 and the cross-process replay/tamper seam for #1327. Wiring it into the cluster harness (expose a node port through a proxy) plus the 5GiB / nightly CI-lane budget are follow-ups, since the harness multi-drive / 2-pool work lands separately (rustfs#4937); this block stays independent and self-tested.
Self-tests run against a loopback echo server that records received bytes, covering: pass round-trip identity, latency lower-bound delay, blackhole no-response-no-panic, one-way partition in both directions (request-path vs return-path), runtime mode switching on a live connection, and port release after shutdown. Timing assertions use loose lower bounds and bounded read timeouts only, never fixed-sleep absolute-value checks, to stay non-flaky. No product code changes; the module is `#[cfg(test)]`-gated.
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
test(ecstore): add rename/commit fan-out pause barrier and background-task introspection
Second white-box test-infra block for https://github.com/rustfs/backlog/issues/1325 (the first block landed the per-disk call counters in PR#4914). Adds a `#[cfg(test)]` awaitable pause barrier plus in-flight background-task introspection to the rename/commit fan-out in `crates/ecstore/src/set_disk/core/io_primitives.rs`, following the same dual-cfg seam style as the existing `disk_call_counters` and `cleanup_fault_injection` seams.
A test arms a barrier for `(object, disk_index, phase)`; the matching spawned fan-out task parks at its checkpoint until the test releases it, and the test awaits the pause through a deterministic `tokio::sync::Notify` handshake (no sleeps). A separate object-keyed task tracker reports how many rename/cleanup background disk tasks are still in flight, so a test can assert "a background disk write is still running" while paused and "no background disk write remains" once the fan-out drains. Both mechanisms live in one process-global registry keyed by object name, so concurrent tests using distinct object names stay isolated. Barriers are placed on the real `rename_data` fan-out (phase `rename`) and the `commit_rename_data_dir` old-data-dir cleanup fan-out (phase `cleanup`).
In production the barrier compiles to an immediately-ready `#[inline(always)]` no-op future and the task guard to `()`, so the fan-out control-flow shape and behavior are unchanged; only the `#[cfg(test)]` variants touch the registry. Coordinator lock-holding is asserted by the test at the store/coordinator layer via the guard it already holds; io_primitives has no handle to that namespace lock. Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool) remains a later cluster-harness block.
Serves the barrier-style white-box acceptances of #1312 (commit fencing: abort at the first-disk rename barrier, assert no background disk write remains after release), #1319, and #1313. Three demo tests drive the real fan-out functions and double as regression guards: neutralizing the barrier seam makes the pause await time out, and neutralizing the task guard pins the in-flight count at zero, so reverting either seam fails the demos.
Refs https://github.com/rustfs/backlog/issues/1317
Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously.
This replaces the permit-less bypass with a bounded overflow lane and a hard cap:
- A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token.
- Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`.
- A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged.
Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned.
Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here.
Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked.
Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail.
The PutObject usecase computed `must_replicate_object` twice with the same inputs: once before commit to persist the pending replication metadata, and again after commit to drive `schedule_object_replication`. Besides repeating the versioning/config/target traversal on the hot path, the two computations read the replication configuration independently, so a config hot update landing between them could split the two phases into a pending-without-schedule or schedule-without-pending divergence.
Reuse the single immutable `ReplicateDecision` computed before commit for both the pending metadata and the post-commit schedule. The two former computations were already equivalent for a stable config (`must_replicate` reads only `opts.replication_request` from the options, which the pending-suffix insertion does not touch), so this preserves replication semantics while removing the redundant traversal and closing the config-race window. The decision carries only stable target/rule/status identifiers (arn, replicate, synchronous, id) and no secrets.
Add a white-box regression test that drives a real PutObject through the usecase and asserts, via a test-only invocation counter on `must_replicate_object`, that a single PUT computes the decision exactly once; reverting to the pre-commit + post-commit recompute makes the counter observe 2 and fails the test.
Refs: https://github.com/rustfs/backlog/issues/1320
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope
Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315).
This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics:
- Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order.
- Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes.
- Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged.
Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention.
Ref: https://github.com/rustfs/backlog/issues/1315
fix(quota): admit bucket hard quota against authoritative decoded size and fail closed on checker faults
Bucket-quota admission for PUT/POST previously ran before the authoritative object length was known and used the raw wire Content-Length: for an aws-chunked upload that length counts chunk framing (overcounting) and can be absent entirely, in which case the check was silently skipped. Separately, any quota-checker fault (bucket-config read, config parse, usage lookup) degraded to allow, which silently bypasses a configured hard quota.
Resolve the authoritative decoded/plain object length first — rejecting negative and unknown lengths, and requiring x-amz-decoded-content-length for aws-chunked instead of falling back to the framed wire length — then run quota admission exactly once against that size. This is the same basis the settle phase records via ObjectInfo.size, so admission and accounting agree. When no quota is configured the QuotaChecker keeps its zero-extra-I/O fast path; once a hard quota is set, checker faults now fail closed with a retryable ServiceUnavailable, increment rustfs_bucket_quota_check_failed_total, and keep the client-facing message generic so internal config/usage details are not leaked.
Size resolution and quota-outcome mapping are extracted into pure functions (resolve_put_object_authoritative_size, map_quota_check_outcome) with unit tests covering aws-chunked decoded-vs-wire, missing/negative/unknown lengths, plain PUT, the exact/over-limit admission split, and fail-closed on checker error. QuotaCheckResult is re-exported through the ecstore api::bucket::quota surface for the app layer. Cross-node reservation and overwrite-delta accounting remain out of scope (sibling issue). Also corrects one stale doc path (set_disk/core/local.rs -> disk/local.rs) flagged by the doc-paths guard.
Refs: https://github.com/rustfs/backlog/issues/1311
CompleteMultipartUpload enqueued a normal-priority heal whenever
`rename_data` returned a `Some(versions)` signature. But the per-disk
signature is produced for every object with <=10 versions, and a healthy
quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value
conflated two distinct facts — "a version signature exists" and "the
committed replicas need heal". The result: nearly every healthy MPU
completion self-enqueued a heal, while >10-version objects (signature
`None`) did not — an algorithmic heal amplification on the healthy path
(rustfs/backlog#1321).
Replace the overloaded `Option<Vec<u8>>` second element of
`SetDisks::rename_data` with an explicit `RenameConvergence` classification
computed after the write-quorum gate:
- AllSuccessIdentical — every attempted disk committed with an identical,
known signature (no heal).
- PartialCommit — write quorum met but a disk failed/offline; a committed
replica is missing or stale (heal).
- SignatureDivergent — all committed but signatures diverge, or mix signed
(<=10-version) with unsigned (>10-version) disks (heal).
- Unknown — all committed, no signature produced (>10 versions); latent
divergence is left to the scanner backstop, not self-enqueued.
`RenameConvergence::needs_heal()` is the single decision point. The version
signature is now only comparison material; it no longer doubles as a heal
flag. The old `select_rename_data_versions` / `reduce_common_versions` /
`rename_data_versions_key` machinery that carried the conflation is removed.
The heal submission in `complete_multipart_upload` moves off the ACK
critical path into a detached task: it runs after the object lock is
dropped and after the durable `rename_data` commit, survives cancellation
of the completion future, and coalesces through the existing bounded /
deduplicated / observable heal-channel admission (one submit per degraded
completion, at most). A completion cancelled in the narrow window between
the durable commit and reaching the enqueue is scanner-backstopped, as is
the Unknown (>10-version) case.
The PUT path (`object.rs`) binds the second element as `_` and is
unchanged. The change is orthogonal to and composes with the #1312 commit
fence on the same `rename_data` path (epoch rejection is a commit-gate
failure surfaced through `Result::Err`, convergence is a post-commit
signal); documented in docs/architecture/unified-object-generation.md.
Tests: `classify_rename_convergence` white-box cases cover the full
acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline
disk, no-common-quorum split, >10-version all-success and with-failure,
mixed signed/unsigned) and fail on revert to the old "signature exists =>
heal" semantics. The decision function is tested directly rather than
through the process-global heal channel, whose receiver is owned
exclusively by the blackbox serial test (init_heal_channel is once per
binary).
Refs: https://github.com/rustfs/backlog/issues/1321
The legacy erasure-decode overlap path drove the speculative next-stripe read and the current-stripe emit with `tokio::join!`, which runs both futures to completion. When the current stripe's emit terminated the loop — a client disconnect or any emit error — the join still waited for the prefetch read, so a `Stop` could stall for a full shard-read deadline on a slow or wedged remote shard before the GET could fail.
Drive the two futures with a biased `select!` instead and, the moment emit reports `Stop`, drop the in-flight read future. Because the entire read pipeline is structured async (a `FuturesUnordered` of `read_shard` futures inside `ParallelReader::read`/`read_lockstep`, with no `tokio::spawn`), dropping the read future is a real cancellation: it drops every in-flight shard read and propagates cancellation down to the RemoteDisk/HTTP reader, leaving no background read behind. The `select!` is scoped so both pinned futures drop before `reader`/`shards` are reused, which is what performs the cancellation in the `Stop` case.
This only affects the overlap-enabled path. The default remains OFF (`prefetch_count == 1` and bitrot-decode overlap disabled), and the strictly-serial read -> reconstruct -> emit default branch is untouched and byte-for-byte identical. The `Continue` path preserves offset, short-tail, buffer recycle, and bitrot/reconstruction error ordering exactly as before.
Scope: cancel-safety only. The rollout decision (whether to enable overlap by default) still requires the Linux multi-node high-RTT three-size A/B from https://github.com/rustfs/backlog/issues/1310 and is deferred; this change does not flip the default or introduce any behavior that A/B must adjudicate.
White-box test `test_legacy_prefetch_cancels_next_read_on_emit_failure` drives the real `Erasure::decode` path with overlap enabled, a writer that fails emit, and shards that serve the first stripe then stall the next-stripe read far beyond the assertion window. Under the paused clock the read future is dropped and decode returns at virtual t~=0; reverting cancel-safety makes it wait out the shard-read timeout, so the test fails closed.
Refs: https://github.com/rustfs/backlog/issues/1310
fix(deps): pin tokio to 1.52.3 to fix dial9-tokio-telemetry build
tokio 1.52.4 made private, breaking
dial9-tokio-telemetry 0.3.14 which references it as public.
Pin tokio back to 1.52.3 until the upstream dependency fixes
compatibility.
Also fix a stale doc path reference in unified-object-generation.md
where is a planned file that does not exist yet.
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
test(ecstore): add per-disk call-counter registry for metadata fan-out
First landable piece of the backlog#1325 test-infrastructure work: a test-only, per-disk call-counter registry that can observe `read_version` RPC counts recorded inside `tokio::spawn` tasks. This unblocks the RPC-count assertions in backlog #1309 / #1314 / #1315, which the thread-local `CapturingRecorder` cannot serve because it is blind to metrics emitted from spawned tasks.
The new `#[cfg(test)]` module `disk_call_counters` (modeled on the existing `cleanup_fault_injection` seam) is a process-global registry keyed by object name; an RAII `observe(object)` scope collects per-disk counts and clears only its own on drop, so parallel tests using distinct object names stay isolated. A dual-`cfg` `SetDisks::record_read_version_call` seam records from inside both metadata-fanout `read_version` spawn sites for online disks only; the `#[cfg(not(test))]` variant is an empty `#[inline(always)]` fn, so production runtime behavior is unchanged.
Two demo/regression tests prove the facility works across worker threads and is revert-detecting (neutralizing the recorder makes them fail).
Refs: https://github.com/rustfs/backlog/issues/1325
* fix(get): enforce strict exact-length materialization for in-memory GET bodies (#1324)
The streaming GET path already fails a short read with UnexpectedEof, but several in-memory materialization branches only WARNed on a length mismatch and then served the body anyway: the encrypted-buffer path, the direct-memory/cache buffered-body path, and the seek-support buffer. Most dangerously, the seek-support path fell through to streaming the *same* reader after read_to_end had already drained K bytes on error, shipping a body missing its prefix (prefix-misaligned data — the closest this code came to returning wrong bytes rather than merely a short body).
Because the HTTP response commits to Content-Length == expected before the body is produced, any other body length is an unrecoverable, broken response. This change gives every in-memory source the same exact-length contract that the ODC materialize-fill path already had.
- Add strict_materialize_object_body, a shared helper that bounds the read to expected+1 (so an over-long stream is detected without buffering it unbounded), requires bytes_read == expected, and on any read error returns without reusing the partially consumed reader. The encrypted, seek, and ODC materialize branches now all route through it.
- The buffered-body branch hard-fails a length mismatch before headers instead of WARN-and-serve.
- MemoryTrackedBytesStream gains a defense-in-depth guard: a buffer whose length disagrees with the declared content length yields a stream error on first poll instead of a clean short body or a silently truncated over-long body. This backstops the cache-hit paths that hand bytes straight to the blob.
The compatibility boundary is preserved: expected is derived from get_actual_size(), the same value used for the committed Content-Length, so a legitimate object (including a legacy/backfilled-size object) whose decoded bytes equal its declared size still serves cleanly — only genuine short, over-long, or errored reads fail. This matches the streaming path, which already hard-fails short reads, so no new large-object behavior is introduced.
Tests: each source (encrypted/seek via the shared helper, cache via build_get_object_body_with_cache, buffered/memory via MemoryTrackedBytesStream) is exercised with expected N and actual N-1 / N / N+1 plus a read-K-then-error injection; only the exact-length read succeeds. Reversal is guarded: restoring WARN-and-serve or a partial fallback flips the short/over-long/error assertions from Err to Ok. Existing streaming UnexpectedEof tests are unchanged.
Refs: https://github.com/rustfs/backlog/issues/1324
* fix(app): reword WARNed comment to satisfy typos check
s3s parses a `Range` suffix length as `u64`, but the GET and HEAD handlers cast it straight to `i64` and bypass any satisfiability check. This truncates deterministically: `bytes=-18446744073709551615` wraps to `-1` and is then read as "last 1 byte", and `bytes=-0` produces a 0-length 206 instead of a 416.
Both handlers now share a single `range_to_http_range_spec` conversion that rejects a zero-length suffix with `InvalidRange` (416), clamps any suffix above `i64::MAX` to `i64::MAX` (such a suffix always covers the whole object, which `HTTPRangeSpec::get_length` then clamps to the real size), and keeps the int branch as a checked cast (s3s already caps `first`/`last` at `i64::MAX`). The scattered `as i64` casts are removed.
Note on ordering: a zero-length suffix is now rejected at conversion time, so `bytes=-0` on a missing object returns 416 rather than 404. This matches the handler's existing behavior of validating range shape (range + partNumber -> 400) before object existence.
Adds a table-driven unit test covering suffix `0/1/size/size+1/i64::MAX/i64::MAX+1/u64::MAX` over empty, 1-byte, and normal objects, asserting the InvalidRange (416) mapping, full-object return for over-size suffixes, and no regression of int first-last / open-ended ranges.
Refs: https://github.com/rustfs/backlog/issues/1322
Remove hard line wrapping from rustfs-release-publish prose (one logical line per sentence/paragraph, soft wrap for display) and link SemVer 2.0.0 spec including spec item 11 for prerelease precedence.
Require explicit target-version confirmation (AskUserQuestion with concrete semver candidates derived from the latest tag) before any release phase runs, and document SemVer 2.0.0 precedence and bump-type rules.
Adds a project skill that orchestrates the full release flow: preview tag first, CI/artifact verification, local run with console checks, rc client command validation, then final version bump and tag cut from the validated preview hash instead of latest main.
Add the shared design/contract document for backlog #1326: a single
per-object generation authority (the #1312 fencing epoch) that spans
commit fencing, read lease, prepared pool read, quota reservation, and
old-dir GC. Pins the five cross-cutting constraints once - RPC signature
binding with server-side nonce enforcement, xl.meta encoding contract
(no meta_ver bump, no positional FileInfo field, internal metadata map
under the dual-key contract), proto3 optional presence, mixed-version
fallback direction, and cluster-level capability negotiation - so the
implementation sub-issues follow them rather than each re-deciding.
No product code changes.
fix(docker): upgrade base-image packages in runtime stages
Trivy code scanning reports 40 open alerts against the published
container images, all from OS packages frozen at the base-image tag:
- musl image (alpine:3.23.4): openssl libssl3/libcrypto3 3.5.6-r0,
including HIGH CVE-2026-45447; fixed in 3.5.7-r0
- glibc image (ubuntu:24.04): tar, gzip, perl-base, ncurses CVEs with
fixes already published in the Ubuntu archive
The runtime stages only ever installed packages and never upgraded the
ones shipped with the base image, so distro security fixes could not
reach released images until the base tag itself moved. Run
`apk upgrade` / `apt-get upgrade -y` in the runtime stages of
Dockerfile, Dockerfile.glibc and Dockerfile.source so each build picks
up current security fixes.
Verified against the exact base tags: after upgrade, alpine 3.23.4
resolves libssl3/libcrypto3 3.5.7-r0 and ubuntu 24.04 resolves
tar 1.35+dfsg-3ubuntu0.2, gzip 1.12-1ubuntu3.2,
perl-base 5.38.2-3.2ubuntu0.3, ncurses 6.4+20240113-1ubuntu2.1 —
matching every fixed version demanded by the open alerts.
PR #4582 gated the update-latest-version job to stable release tags so prereleases could not overwrite the stable version pointer. But the project currently ships prerelease tags only (beta previews), so the job never runs and latest.json has been stale since. Run the job for every release tag again, and keep the honest half of the #4582 fix by writing release_type from the actual build type instead of hardcoding "stable".
* helm chart: one data drive per node - fix1
* refactor(helm): prevent negative or 0 replicaCount
Co-authored-by: Copilot <copilot@github.com>
* refactor(helm): remove env REPLICA_COUNT
* feat(helm): default no logs directory to force stdout
Co-authored-by: Copilot <copilot@github.com>
* feat(helm): add drivesPerNode
* feat(helm): correct default parity with 4 nodes / 1 drive per node
* conditional render of RUSTFS_OBS_LOG_DIRECTORY
* feat(chart): add table doc for parity
* fix(chart): handle invalid annotation objects
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* feat(chart): better table doc for parity
* fix(chart): leave RUSTFS_OBS_LOG_DIRECTORY empty, or the defaults will attempt to write to readonly fs
* fix(chart): chart defaults as the previous version: 4 nodes with 4 drives per node
* feat(chart): render RUSTFS_STORAGE_CLASS_STANDARD in configmap
* minor fix for pvcAnnotations
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
* fix(chart): better drivesPerNode handling
* fix(chart): obs_log_directory default non empty again to preserve previous deployments compatibility
* fix(chart): proper drivesPerNode impl
* fix(chart): values clarification for empty vs unset `obs_log_directory`
* feat(chart): add service externalIPs
* feat(helm): add optional service labels
* fix(helm): merge service labels
* fix(helm): storageclass pvcAnnotations
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* fix(helm): remove duplicate keys
* fix(helm): default drivesPerNode null, keep legacy chart behavior
* feat(helm): add template test for externalIPs
* fix(helm): per-pool drivesPerNode inference, restore regression tests
Repairs the drivesPerNode feature from #2693 so it renders and stays
upgrade-safe:
- define the missing drives inference (rustfs.poolDrives) and compute it
per pool inside rustfs.pools, so mixed 4x4 + 16x1 pool deployments keep
their exact legacy volumeClaimTemplates when drivesPerNode is unset
- restore the $poolsEnabled definition dropped in the rebase (chart failed
to render at all)
- fix .Values references inside the pool range (dot is the pool there) and
keep per-pool storageclass pvcAnnotations overrides working
- make rustfs.volumes derive the drive range from pool drives instead of
the pod count, and relax the pools.list 4-or-16 restriction to >= 2
- reject drivesPerNode=0 explicitly instead of silently inferring
- keep default rendering identical to main: storage_class_standard stays
unrendered by default, obs_endpoint.use_stdout stays false, clusterDomain
value restored
- restore the clusterDomain/mTLS SAN/explicit-volumes regression tests that
the branch deleted, keeping the new topology tests
---------
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Remove the Intel macOS entry from the Build and Release platform matrix so official release builds only publish the Apple Silicon macOS binary.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ilm): fix restore test object key to match transition filter
restore_object_usecase_reports_ongoing_conflict_and_completion used the
object key "restore/api-object.bin", but the shared
set_bucket_lifecycle_transition_with_tier helper only transitions objects
under the "test/" prefix. enqueue_transition_for_existing_objects therefore
matched nothing and wait_for_transition timed out at 15s, failing the test
deterministically.
The test was added in #4860 but its ILM Integration (serial) lane is
skipped on regular PRs, so it merged red and has failed on every main run
since. Move the object under the test/ prefix like every passing sibling
test in this file.
* ci(ilm): exclude broken RestoreObject API test from serial lane
restore_object_usecase_reports_ongoing_conflict_and_completion exposes a
real regression, not a test bug: the RestoreObject copy-back
(handle_restore_transitioned_object) now holds the object write lock added
in #4877 across the entire tier read-back, so it never releases in time and
the test's concurrent get_object_info times out with Lock(Timeout, 5s). The
failure is deterministic and independent of the mock tier's injected latency.
This is the same class of known-broken restore/transition failure already
tracked under backlog#1148 (three sibling scanner tests are excluded here by
name for the same reason), so exclude this one the same way until the restore
copy-back path is fixed or the #4877 lock scope is revisited. The prior commit
keeps its correct fix (the object key must live under the test/ transition
prefix); that was masking this deeper issue by never letting the object
transition in the first place.
Restore copy-back deadlock/hang under the #4877 lock is escalated separately
for a product-level decision (fix the copy-back vs. narrow/revert #4877).
* test(ilm): fix scanner restore test object keys to match transition filter
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore and
test_multipart_restore_preserves_parts_and_etag (both added in #4860) keyed
their objects under restore/ instead of the test/ prefix that
set_bucket_lifecycle_transition_with_tier filters on, so the objects never
transitioned and wait_for_transition timed out at 15s.
These surfaced only after the prior commit excluded the rustfs-side restore
API test: nextest runs -j1 fail-fast, so that earlier failure stopped the run
before these scanner tests executed. Unlike the excluded API test, both call
restore_transitioned_object().await sequentially and only read afterwards, so
they don't hit the concurrent-read-vs-#4877-write-lock timeout; the key
prefix was their only problem.
* ci(ilm): exclude the two remaining #4877-broken restore tests
test_multipart_restore_preserves_parts_and_etag and
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore both
call restore_transitioned_object().await, which since #4877 acquires the
object write lock and deterministically times out (Lock Timeout, 5s) against
an already-held lock, so restore never completes. They surfaced one at a time
because nextest runs -j1 fail-fast. The earlier prefix fix was necessary but
only advanced them from the transition wait to this restore-lock timeout.
Exclude both by name alongside their already-excluded sibling
test_transition_and_restore_flows (same root cause, tracked under
backlog#1148) so the ILM Integration (serial) lane goes green. The #4877 lock
scope still needs a product fix before any of these re-enable.
* docs(ilm): describe the excluded restore tests' symptom as a lock timeout, not a deadlock
The #4877 write lock is held across the tier read-back and outlives the 5s
lock timeout; nothing proves a true deadlock. Wording flagged by Copilot
review.
* fix(ecstore): stop restore copy-back self-deadlocking on the #4877 write lock
#4877 made handle_restore_transitioned_object hold the object write lock for
the whole restore and forward no_lock=true so the set layer would not
reacquire it. But the set-level copy-back rebuilds its own options
(put_restore_opts -> ropts, and the complete_multipart_upload opts) that
default no_lock=false, so the inner put_object / new_multipart_upload /
complete_multipart_upload each re-acquire this object's write lock in their
commit phase and block on the lock the restore already holds -> Lock(Timeout,
5s), and restore never completes.
Confirmed via RUSTFS_OBJECT_LOCK_DIAG_ENABLE: restore_transitioned_object
acquires the write lock, then holds it ~10.5s across two nested 5s acquire
timeouts before failing. This is a real product deadlock: a RestoreObject on
any transitioned object (multipart especially) hangs, not just the tests.
Propagate no_lock into the copy-back options so the inner writes inherit the
already-held lock. Use opts.no_lock (not a hardcoded true) so a caller that
restores without the outer lock still locks correctly. put_object_part is left
as-is: it locks the multipart upload-id resource, not the object key, so it
does not conflict. Verified test_multipart_restore_preserves_parts_and_etag
now passes (3.6s, was a 15s+ hang).
* ci(ilm): re-enable multipart restore test; scope remaining exclusions
The prior commit fixes the #4877 restore self-deadlock, so
test_multipart_restore_preserves_parts_and_etag passes again - drop it from
the serial-lane exclusion list and remove its 'currently excluded' note.
The other restore/transition tests still fail, but each on a DIFFERENT,
independent issue unrelated to the (now-fixed) lock, verified locally:
- test_restore_chain_...: DeleteRestoredAction sets expire_restored but no
delete path reads it, so cleanup deletes the whole object (unimplemented
semantics), not the local restored copy only.
- test_transition_and_restore_flows: transition xl.meta missing on one drive
(EC metadata distribution), not restore.
- restore_object_usecase_reports_ongoing_conflict_and_completion: asserts a
concurrent mid-restore ongoing=true read that #4877's read-vs-restore
serialization rules out (backlog#1148 ilm-8 criterion 1, an API-semantics
decision).
Comments and #[ignore] reasons updated to reflect each real cause. All remain
tracked under backlog#1148.
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
* chore(deps): tighten crate dependency features
Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): trim hyper-rustls features
Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`restore_object_usecase_reports_ongoing_conflict_and_completion` failed
deterministically in the "ILM Integration (serial)" lane, panicking at
its setup step ("object should transition before the restore API runs"):
the object never reached transition status `complete` within the 15s
wait, before the restore logic under test even ran.
Root cause: the test object key was `restore/api-object.bin`, but
`set_bucket_lifecycle_transition_with_tier` scopes the transition rule to
`<Filter><Prefix>test/</Prefix>`. An object outside that prefix never
matches the rule, so `enqueue_transition_for_existing_objects` never
enqueues it (confirmed: the mock tier saw zero puts and the object's
transitioned status stayed empty) and `wait_for_transition` times out.
Every other transition test in this file already keys its objects under
`test/`. The break shipped in #4860 because that PR's ILM serial lane was
skipped on the merge run, so the test never actually executed in CI.
Fix: key the object as `test/restore/api-object.bin` so it matches the
rule. The restore API, tier copy-back, and local-GET assertions are
unchanged (they address the object by bucket+key; the remote tier name is
generated internally).
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
Avoid compiling the mimalloc JSON parsing helpers on non-test Windows builds so the platform-specific dead_code warnings disappear while tests keep coverage through #[cfg(test)].
Co-authored-by: heihutu <heihutu@gmail.com>
`test_heal_resume_across_page_boundary_e2e` panicked whenever the
erasure-set healer deferred a single object version to a later heal
cycle. Under load a per-version `heal_object` can hit a transient error;
`ErasureSetHealer::heal_bucket_with_resume` then persists its
resume/checkpoint state and returns a terminal `Failed { .. "retry
scheduled" }`, expecting a fresh heal run to finish the job. In
production the background scanner is that next run — the e2e had no such
follow-up, so the first task's `Failed` state failed the test. This is a
pre-existing rare flake (the wiped-disk heal is otherwise correct); it is
unrelated to any policy/proptest work that happened to surface it in CI.
Drive the heal to a genuine `Completed` instead: on a `Failed` carrying
the `retry scheduled` marker (retry budget still remaining) re-submit the
idempotent heal (`force_start`), mirroring the production scanner, up to a
small bound. A `Failed` without that marker (e.g. `exhausted retries`) or
any other non-`Completed` terminal state still fails the test, and the
strict per-version data-restoration assertions still run only after a
real `Completed`. `wait_for_task` is refactored onto the shared
`await_terminal_status` poller; its panic-on-failure semantics for the
unversioned-bucket heal are unchanged.
Test-only change; no production heal code is touched.
Co-authored-by: heihutu <heihutu@gmail.com>
crates/policy had zero property tests: the wildcard Action/Resource matching
and Deny-first evaluation in Policy::is_allowed — the algebra authorization
safety rests on — were guarded only by example-based tests.
Add crates/policy/tests/policy_eval_proptest.rs with three generated-input
invariants (pure evaluation, no IO or global state, parallel-safe):
- explicit_deny_anywhere_denies: a Deny matching the request denies it no
matter how many broad Allow statements co-exist or at which index the Deny
sits; a built-in sanity assertion first proves the Allows alone would permit,
so the Deny is what flips the decision.
- wildcard_superset_implies_concrete_match: any request allowed by an
exact-action/exact-resource policy is also allowed after widening to s3:* on
bucket/* and to * on arn:aws:s3:::*, checked both on the built grant and on
random probe requests (implication form).
- empty_policy_denies_everything: a statement-less policy and Policy::default()
deny every non-owner request.
Statements are built from JSON exactly as production policies arrive via
PutPolicy. Generated names avoid wildcard metacharacters so the only wildcards
in play are the ones the properties introduce deliberately. proptest joins
crates/policy dev-deps following the filemeta/ecstore precedent.
Refs: backlog#1151 (sec-9)
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the
latest main to pull the newest compatible versions.
Manifest bumps (Cargo.toml):
- redis 1.3.0 -> 1.4.0
- xxhash-rust 0.8.16 -> 0.8.17
Lockfile-only bumps: simd-adler32 0.3.10, syn 2.0.119, and
toml_edit 0.25.13. `ratelimit` is held at 0.10.1 as requested.
Verification: cargo check --workspace --all-targets, cargo clippy
--all-targets -D warnings (plus the rio-v2 feature variant), cargo nextest
run --all --exclude e2e_test (8964 passed), and cargo deny check
advisories/sources/bans/licenses all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap
backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:
- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
/tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
init_bucket_metadata (default true; the iam bootstrap test opts out
to preserve its historical semantics). TestECStoreEnv exposes
temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
the sanctioned test-compat boundary pattern (mirrors
crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
for heal_env{,_n} wrappers; the tests/storage_api.rs integration
surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
ecstore_test_compat fixture shrinks to SetupType +
update_erasure_type.
rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.
* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move
* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
refactor(ecstore): make client base64 standard everywhere, drop the split
Self-review of the transition-client fixes: the Content-MD5 fix had only
converted the two transition call sites to a parallel base64_encode_standard,
leaving the same URL-safe, unpadded base64 bug on every other outbound value —
the SigV2/no-length multipart Content-MD5, the x-amz-checksum-* headers on the
parallel and no-length paths, api_remove's multi-delete Content-MD5, and the
checksum encode/decode helpers. Every base64 value this client emits or parses
is S3 wire format, which is standard base64; none is ever URL-safe.
Encode and decode both use base64_simd::STANDARD now, and the parallel
base64_encode_standard is gone. This fixes those seven latent call sites at the
root and removes the duplicate function. The transition path is functionally
unchanged (it already produced standard base64 via the helper), so the
end-to-end byte-for-byte result is preserved.
Also drop a stray Vec::with_capacity(part_size) that was allocated (up to
128 MiB) and immediately overwritten by read_multipart_part's own buffer.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-authored-by: heihutu <heihutu@gmail.com>
* ci: add e2e-full merge-gate job (single-node full e2e via nextest)
Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in
ci.yml (push main + merge_group + workflow_dispatch) that runs the
never-automated user-visible e2e suites (KMS, object_lock, multipart_auth,
quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke
subset skips.
The filter is the whole e2e_test crate minus the sets other lanes own:
protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites
(ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly
lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are
serialized, mirroring the ci profile. Reuses the downloaded debug binary and
uploads junit. backlog#1149 ci-5.
* ci: exclude characterized known-failures from e2e-full with tracked issues
First-ever automated run of the full suite (dispatch 29381309848: 341 ran /
32 failed / 460s) surfaced five real product-bug families, each now tracked:
rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit
vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845
(object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum
misclassified as timeout under load). Deterministic failures cannot be
retried away, so each family is excluded with its OPEN issue cited and the
fixing PR contract to delete the exclusion — passing negative-path siblings
stay in as regression guards.
xl.meta roundtrip coverage was one fixed example test plus byte-level no-panic
fuzz properties; nothing generated STRUCTURED version graphs, so a lossy
encoding bug in version headers, ordering, delete markers, or the dual
internal-metadata-key handling would only surface if the fixed example happened
to hit it.
Add crates/filemeta/tests/version_graph_roundtrip_proptest.rs with two
generated-input properties over multi-version graphs (1-7 versions, ~30% delete
markers, deliberately colliding mod_times to exercise tie-break ordering, user
metadata, and the AGENTS.md dual x-rustfs-internal-*/x-minio-internal-* keys
written via the real insert_bytes helper):
- version_graph_marshal_load_roundtrip: marshal -> load preserves version
count, meta_ver, the exact (id, type) sequence, full headers, fully decoded
per-version content, delete-marker payload shape, and both internal metadata
key forms with identical values.
- version_graph_marshal_is_idempotent: marshal(load(marshal(x))) is
byte-identical to marshal(x), catching encoders that mutate or reorder state
on the way out.
Complements the existing byte-level no-panic properties: those prove hostile
bytes cannot crash the decoder; these prove honest graphs are encoded
losslessly.
Refs: backlog#1151 (sec-10)
Register the webhook target while its endpoint is reachable, then drop
the listener before the PUT: registering against a dead endpoint stalls
behind the reachability probe timeout and flaked the ARN wait on the
rustfs#4821 merge run. Also widen the registration wait to 20s.
A single PUT can enqueue the same object for transition twice — immediately
(enqueue_transition_after_write) and again from the startup/lifecycle
compensation backfill. transition_object's own namespace lock is commented out,
so the two attempts are not serialized as same-object work: the winner
transitions the object and removes its local data, and the loser then reads that
already-removed data via get_object_fileinfo(read_data = true) and logs a
spurious "get_object_with_fileinfo err ... No such file" /
lifecycle_tier_operation_failed. For a large (multipart) object both attempts
can also race the source read, corrupting the transferred copy.
Guard the transition queue with an in-flight set keyed by
(bucket, object, version). queue_transition_task claims the key before sending;
a duplicate enqueue is reported handled without queueing a second task. The
worker releases the claim once the transition finishes, so a later lifecycle
pass can still re-transition the object, and a failed enqueue releases it
immediately. This is the sole path into the transition channel, so every
enqueue is covered.
Surfaced while validating the >128 MiB transition fix end-to-end in Docker
(rustfs/rustfs#4811); tracked as its own defect since it is unrelated to the
checksum/multipart-client bugs.
Tests: same-version enqueues dedupe (direct reserve and via queue_transition_task
with spare capacity), a distinct object still hits the full-queue path, and a
released claim can be re-acquired.
Refs: rustfs/backlog#1268
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed
ILM transition of any object larger than 128 MiB to a RustFS-native tier
(rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the
built-in TransitionClient) failed with "unsupported checksum type", while
objects <=128 MiB transitioned fine.
Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured
checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of
the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The
128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single
PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path,
`put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`,
disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which
returns the "unsupported checksum type" error. The single-PUT path hit the same
misjudgement but never calls `hasher()`, so it silently succeeded (without a
checksum), which is why only >128 MiB objects failed.
Fix:
- `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject`
flag, which has no base algorithm). This is the sole callers' intended
meaning: a concrete algorithm with a real hasher is selected.
- Defense in depth: guard the multipart checksum branch on
`auto_checksum.is_set()` so an unset mode uploads the part without a per-part
checksum header instead of hard-failing in `hasher()`.
Only the TransitionClient consumes this `ChecksumMode::is_set()`; the
server-side data path uses the unrelated `rustfs_rio::ChecksumType`.
Tests: is_set()/set_default semantics, hasher parity for every set mode, and a
`build_transition_put_options` invariant (checksum unset + Content-MD5 on).
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): read exactly one part per multipart chunk in transition uploads
Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811),
uncovered while verifying the checksum fix.
`put_object_multipart_stream_optional_checksum` read each part with
`read_all()` / `to_vec()`, which drained the entire source into the first part
and left every later part empty. Any multipart upload of a streamed
(`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the
single-part path and were unaffected; a 128 MiB + 1 byte object splits into a
128 MiB part plus a 1 byte part, so the first part received the whole object and
its declared Content-Length (part_size) did not match the body.
Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts,
and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes,
leaving 0 for part 2.
Fix:
- Add `read_multipart_part`, which reads exactly the requested part size (or
less at EOF) and advances the reader, for both `Body` (in-memory) and
`ObjectBody` (streamed) sources.
- Upload each part with the bytes actually read (`length`) as its size, and
account uploaded size by actual bytes, so a short read is detected instead of
masked.
The concurrent (`put_object_multipart_stream_parallel`) and SigV2
(`put_object_multipart`) paths share the same `read_all()` pattern but are not
exercised by transition; left untouched here and noted for follow-up.
Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for
both streamed and in-memory bodies, consumes the source fully, and stops at EOF
without overrun.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): complete the >128 MiB ILM transition multipart client
Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a
128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the
multipart transition path, each masked by the previous one. With the checksum
and part-splitting fixes in place the transition now failed later and later,
and finally produced a 0-byte object with no error at all. Fixed together:
- initiate_multipart_upload discarded the CreateMultipartUpload response and
returned an empty UploadId, so the first UploadPart failed with "UploadID
cannot be empty". Parse the response XML (InitiateMultipartUploadResult now
derives Deserialize with PascalCase).
- Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64,
which the remote rejected as "Invalid content MD5: Base64Error". Add
base64_encode_standard and use it for those outbound header values.
- PutObjectOptions::default() set legalhold to OFF, so header() attached
x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was
rejected with "does not accept object lock or governance bypass headers".
Default to an empty (unset) status.
- CompleteMultipartUpload / CompletePart had no serde renames, so the request
body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed
zero <Part> elements and completed a 0-byte object while returning 200. Emit
S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields.
Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote
tier and reads back (transparently restored) byte-for-byte identical
(sha256 match), with none of the four prior errors in the logs.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The individual policy layers were tested in isolation, but the cross-layer
resolution — which layer wins when IAM and bucket policy disagree — had no
coverage. A priority error there is a data-exposure or data-lockout bug.
Add crates/policy/tests/bucket_iam_authz_matrix.rs, a pure table-driven test
(no globals, no IAM store, no server) that drives the real Policy::is_allowed
and BucketPolicy::is_allowed through a helper modeling the request-layer
orchestration in rustfs::storage::access::authorize_request (bucket explicit-Deny
gate -> IAM allow -> bucket allow fallback; anonymous = bucket policy alone). It
pins all four Allow/Deny quadrants plus the anonymous case, and adds intra-policy
Deny-beats-Allow checks for both layers.
The matrix surfaces the resolution invariant explicitly: a BUCKET explicit Deny
is a hard gate and always wins, but an IAM explicit Deny is SOFT — a bucket Allow
fallback overrides it, which diverges from AWS "an explicit Deny in any policy
always wins". The test characterizes the current (MinIO-lineage) behavior; if IAM
Deny is later hardened into a gate, iam_deny_x_bucket_allow flips to false and
must be updated deliberately.
Refs: backlog#1151 (sec-8)
Co-authored-by: houseme <housemecn@gmail.com>
test(security): pin GHSA-m77q STS root-secret token signing (sec-7)
GHSA-m77q-r63m-pj89 (intentionally UNFIXED) is that STS session tokens are
signed with the shared root secret: crates/iam/src/root_credentials.rs
token_signing_key() returns the root secret_key, so anyone holding the root
secret can forge STS session tokens. No test named the advisory, and the
existing test_created_sts_credentials_authorize_with_session_token_claims uses
token_signing_key() for both signing and verifying, so it pins "same key signs
and verifies" but not the m77q-specific "signing key IS the root secret" — a
future fix that decouples the STS key from the root secret would pass it
silently.
Add a flow-level pin, test_ghsa_m77q_sts_session_token_signed_with_root_secret,
that captures the advisory's exact signature: (1) token_signing_key() == the
root secret; (2) an AssumeRole-style session token issued with that key decodes
with the root secret and NOT with any other secret; (3) it authorizes through
the STS path. All three assert CURRENT (by-design-vulnerable) behavior, so a
real m77q fix (a dedicated STS signing key) turns them red and forces a
red -> green regression update.
Also GHSA-name the existing characterization test and token_signing_key() with
doc comments and the advisory URL, and update the m77q row in
docs/testing/security-regressions.md. No production behavior changes.
Refs: backlog#1151 (sec-7)
Co-authored-by: houseme <housemecn@gmail.com>
chore(scripts): index scripts/ and archive 29 one-shot scripts
backlog#1153 infra-13. scripts/ had 80+ unlabelled top-level entries
mixing CI gates with finished one-shot issue-validation scripts.
- scripts/README.md — one index row per entry with status
(ci-gate / dev-tool / archived), purpose, and wiring; subdirectories
get one row each. run_scanner_benchmarks.sh is annotated
"disposition owned by backlog perf-10" and deliberately untouched.
- git mv 29 confirmed-stale one-shot entries to scripts/archive/:
11 issue-scoped validation/perf-capture scripts, the 5-script
backlog#706 large-PUT breakdown family, the 4-file GET-optimization
stress suite, 2 gt1g one-shots, and 7 other orphaned one-shots.
Evidence: a whole-tree boundary-aware reference census showed zero
references from CI/Makefiles/docs/code for every moved entry (or
references only from other scripts inside the same archived set);
re-run after the move shows zero dangling references.
- docs/testing/README.md links the index.
Moves only — no script content changed.
bench(rustfs): remove the fake s3_operations benchmark
rustfs/benches/s3_operations.rs never measured S3: all three groups
(put_object/get_object/list_objects) only black_box(data.len())/black_box(count)
with a 'In a real benchmark, this would call the actual S3 client' comment. It
gave a false 'S3 operations have benchmark coverage' signal and was dead weight
on rustfs/Cargo.toml.
Delete the file and its [[bench]] entry. S3-face performance is covered solely
by the warp A/B gate (performance-ab.yml); the runbook scope note now says so
explicitly. Micro-benchmarks stay at the function level (perf-8).
Refs rustfs/backlog#1152 (perf-9).
The script cannot run on any machine: WORKSPACE_ROOT is hardcoded to a personal
path (/home/dandan/code/rust/rustfs) and it targets the rustfs-ahm crate, which
no longer exists in the workspace (scanner is crates/scanner, heal is
crates/heal, neither ships benches/). It gave a false impression that scanner
performance automation existed.
There is no current scanner-bench requirement, so retire it (option a). Nothing
references the script anywhere else in the repo.
Refs rustfs/backlog#1152 (perf-10).
ci(coverage): weekly cargo-llvm-cov workspace baseline (non-blocking)
Add the weekly line-coverage report (backlog#1153 infra-5):
- .github/workflows/coverage.yml — Sunday 07:00 UTC + workflow_dispatch;
runs `cargo llvm-cov nextest --workspace --exclude e2e_test` under
NEXTEST_PROFILE=ci (same scope and profile as the ci.yml test gate),
writes a per-crate line-coverage table to the job summary, uploads
lcov + JSON as a 90-day artifact, and routes scheduled failures
through the shared schedule-failure-issue action (ci-8). Runs only on
schedule/dispatch, so it can never become a required PR check.
- scripts/coverage_per_crate.py — stdlib-only aggregation of the
llvm-cov JSON export into the per-crate markdown table (worst-first,
TOTAL row), shared by the workflow and the local target.
- make coverage (.config/make/coverage.mak) — local equivalent with the
same command sequence; fails with install hints when cargo-llvm-cov
or cargo-nextest are missing. Listed in make help.
- docs/testing/README.md — Coverage section: cadence, where the table
and artifacts live, the trend-comparison method, and what is not
measured (doctests, e2e_test).
Every push to main now builds the release binary once and stores it in the
actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job
restores it for origin/main and passes --baseline-bin; on the nightly, where
the candidate commit equals the baseline, one cached binary serves both phases
with --skip-build and the run does zero source builds. A cache miss falls back
to the source double-build via --baseline-ref origin/main.
This removes the ~32min-per-side double build that pushed the 24-cell nightly
past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout).
Also:
- alert-on-failure now fires on cancelled/timed-out, not just failure, so a
timed-out nightly is no longer silent (the composite action already reports
cancelled jobs); removed the stale perf-2 TODO now that the alert job exists.
- run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and
candidate SHAs + binary source, runner, warp version, matrix params) via a
repeatable --provenance-note; the gate exit code is preserved.
Refs rustfs/backlog#1152 (perf-3).
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
ci(mint): bypass local physical-disk-independence guard so mint boots
The mint container maps four RUSTFS_VOLUMES data dirs onto a single
runner device, so the startup physical-disk-independence guard aborts
with a FATAL before mint can run. The scheduled run 29183544431
(2026-07-12) crashed at 'Wait for RustFS ready' with 'local erasure
endpoints must use distinct physical disks'.
Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true on the container, the CI use
the guard explicitly sanctions -- mirroring the e2e-s3tests harness
fixed in #4768. Completes the ci-2 acceptance (a mint run that actually
produces log.json and the per-suite summary).
docs(testing): populate testing pyramid, naming, serial/nextest rules
Fill the docs/testing/README.md skeleton (backlog#1153 infra-11):
- Test taxonomy table for all eight layers (unit / ecstore black-box /
e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench) with a
verified entry command and a qualitative "when it runs" per layer; the
event x timeout x required-status matrix stays owned by
docs/testing/ci-gates.md (ci-15), linked not duplicated.
- Naming conventions section with the migration-gate reserved substrings
(data_movement / rebalance / decommission / source_cleanup /
delete_marker) linking the infra-12 count-floor guard, closing that
task's docs cross-link.
- Serial execution & nextest rules: nextest as a hard dependency with the
RUSTFS_ALLOW_CARGO_TEST_FALLBACK escape hatch and the runner-semantics
difference (folds the infra-14 README half), why #[serial] is a no-op
under nextest, and the default/ci/e2e-smoke/e2e-repl-nightly profiles.
- A time-control placeholder for infra-4 to fill.
The pre-existing flake-policy section (ci-10) is preserved verbatim.
Add pointers from CLAUDE.md and CONTRIBUTING.md.
* test(ecstore): assert decrypted_size for MinIO SSE interop round-trip
The ignored MinIO interop round-trip tests asserted `ObjectInfo.size`
against the plaintext length. For SSE objects `size` is the on-disk
DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), so the
assertion can never hold once real fixtures are present — the two
`#[ignore]` tests failed the moment a real MinIO-written fixture was fed
in, even though the decoded data was byte-identical.
The client-visible object size comes from `decrypted_size()` /
`get_actual_size()`, which correctly reads MinIO's
`x-*-internal-actual-size` metadata (verified: both SSE-S3 and SSE-KMS
8 MiB multipart fixtures now report 8388608). Assert against that
instead and keep the plaintext length and SHA-256 data checks.
With real 4-drive MinIO fixtures (RELEASE.2025-09-07) all four tests
pass, confirming RustFS reads MinIO erasure-coded SSE objects with
byte-identical data and correct logical size.
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): nightly MinIO interop check + dockerized fixture capture
Wire the ignored MinIO on-disk interop reader tests into a nightly,
non-required CI job, and make their fixtures reproducible without a host
MinIO install.
- Dockerfile + capture_via_docker.sh: build a throwaway image carrying
the official MinIO server binary (pinned RELEASE.2025-09-07) plus the
fixture lab on a small Python base, then run `lab.py capture-matrix` to
write the SSE-S3 / SSE-KMS multipart fixtures the tests consume. lab.py
drives MinIO's S3 API directly, so no `mc` is needed.
- .github/workflows/minio-interop.yml: nightly + manual workflow on
GitHub-hosted ubuntu-latest (reliable Docker + Python, unlike the
self-hosted fleet — see e2e-s3tests.yml infra note). Regenerates the
gitignored fixtures each run and executes the #[ignore] reader tests.
Not a PR gate.
- README: document the Docker capture path.
Validated end to end: the script builds the image, captures the two
multipart cases, and `cargo nextest run --run-ignored ignored-only`
passes all four interop tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(replication): lock outbound checksum consistency for new algorithms (T3)
Adds a consistency test at the replication put-options boundary confirming that
the AWS 2026-04 additional checksum algorithms (XXHash3/64/128, SHA-512, MD5) are
forwarded into replication user_metadata identically to the classic five. The
outbound replication path routes a stored object checksum through the
algorithm-agnostic decrypt_checksums -> user_metadata flow, so the new algorithms
(already covered by rustfs-rio read_checksums) need no new-algorithm-specific
handling. This locks that behavior against regressions.
Investigation summary (no code change needed on the outbound side): the
per-algorithm ChecksumMode selection path is dormant (opts.checksum is never set
to a specific algorithm; tiering uses Content-MD5; the add_crc bool is dead
code), so extending ChecksumMode was unnecessary.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)
Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).
The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.
Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)
Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.
Refs rustfs/backlog#1255rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)
Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.
Refs rustfs/backlog#1260rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)
HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.
Refs rustfs/backlog#1257rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(checksums): fail-closed on unknown checksum algorithm (S7)
A. Harden unknown/unsupported checksum algorithms to fail closed instead of
panicking. ChecksumMode::base() in the outbound S3 client
(crates/ecstore/src/client/checksum.rs) previously did
`panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
unit tests proving base() never panics and hasher() returns Err for
unsupported modes. rustfs-checksums FromStr already returns Err on unknown
names; added a regression test asserting garbage/unknown names fail closed.
B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
into_impl, name constants, the x-amz-checksum-* header constants and the
HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
digest against a direct library computation. MD5 stays intentionally
rejected (PR #4513) and is left untouched.
C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
with 7 variants already consuming 7 bits; adding the 4 new algorithms would
overflow u8 and require a breaking repr change, so ChecksumMode is left
unchanged. The new algorithms are available through the rustfs-checksums
ChecksumAlgorithm path.
Refs rustfs/backlog#1259rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)
Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:
- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
from build_get_object_checksums through GetObjectOutputContext into
finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
captured at the want_checksum set points before opts is moved.
Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.
The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.
Refs rustfs/backlog#1257rustfs/backlog#1256rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)
Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):
- complete_part_checksum previously returned the outer None for any algorithm
outside the five typed ones, which failed CompleteMultipartUpload with
InvalidPart. It now accepts any valid base type with no double-check value
(Some(None)) — mirroring the missing-value path of the typed algorithms —
since s3s CompletePart has no field to carry a client-supplied per-part
value and the part was already verified server-side at UploadPart. Genuinely
unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
concatenated per-part raw digests; full_object_requested() is false so
add_part() is correctly bypassed) already works for these algorithms via the
S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
their responses via the shared inject_additional_checksum_headers helper
(now pub(crate)), since s3s has no typed output field.
Refs rustfs/backlog#1261rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)
Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.
Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.
Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).
Refs rustfs/backlog#1258rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)
from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.
The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.
Refs rustfs/backlog#1263rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(checksums): collapse 5 duplicated response-checksum loops into one
Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.
Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
performs the typed/extra split once. All five call sites now destructure its
result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
hand-rolled five-way comparison.
Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(checksums): unit tests for classifier/echo helpers + fix unused import
Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.
Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)
The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.
Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)
Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.
Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).
Refs rustfs/backlog#1262rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(get): allow too_many_arguments on finalize_get_object_response
The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <housemecn@gmail.com>
The pulsar config validation rejected any non-`pulsar+ssl` broker whenever
`tls_allow_insecure` was set or `tls_hostname_verification` was disabled.
Both toggles are inert on a plaintext `pulsar://` broker — the Pulsar client
only applies them for `pulsar+ssl` — so treating a non-default value as fatal
is wrong.
This surfaced as issue #4796: the console persists `tls_hostname_verification`
as `false` (the checkbox defaults to unchecked while the server default is
`on`). At creation time the value is not present in the unmerged request and
falls back to the safe default, so the connectivity check passes and the
target comes online. On restart the persisted config is merged and validated,
the `false` is read back, and the target is rejected — permanently offline
even though Pulsar is reachable.
Relax both the loader-side (`validate_pulsar_broker_config`) and runtime-side
(`PulsarArgs::validate`) checks so only a `tls_ca` bundle — real TLS trust
material that is never defaulted to a non-empty value — requires a
`pulsar+ssl` scheme. The inert toggles no longer fail the target. This also
heals configs already persisted with the bad value. Add regression coverage
for both paths.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF
hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response
bytes are still buffered (a prior poll_flush() returned Poll::Pending and the
result was discarded), so a backpressured/slow-reading peer receives a graceful
FIN before the full Content-Length body is flushed. Standard S3 clients
(minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET
under load (rustfs/backlog#1232). This is the transport-layer bug from
Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF
already fixed via lockstep decode; here the app layer delivers the full body and
truncation is purely hyper->socket.
Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush
buffered data before shutdown"), which is not in any crates.io release yet as of
hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant
of the fix that still declares version 1.10.1).
Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the
server connection is driven by the transitive hyper-util (conn::auto /
GracefulShutdown), and a git source would not unify with the crates.io hyper
hyper-util resolves, leaving two hyper copies with the server path still on the
buggy one. The patch rewrites the crates.io source globally, so the lockfile
holds a single git-sourced hyper.
Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard
(mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped
or hyper is downgraded below the fix. Its load-bearing assertion is a
synchronously-set flag, so a slow runner can only under-detect, never false-red.
Closesrustfs/backlog#1232.
* build(deny): allow the hyperium/hyper git source for the flush-before-shutdown pin
cargo-deny's [sources] check denies unknown git sources. The hyper
[patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium
git source, so add it to allow-git with an owner/review note and a removal
condition (drop once a released hyper > 1.10.1 carries commit 72046cc7).
description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
---
# Adversarial Validation Playbooks
@@ -53,14 +53,22 @@ shipped bug or rule that earns each probe its place.
- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
- Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
- Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
- Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
- Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
- Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
### Simplicity adversary
- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
- Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
- Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
- Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
- Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
- Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
- Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
- Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
- Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
@@ -243,7 +254,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
- Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
- Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
- Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
- Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
@@ -43,16 +43,7 @@ Use this skill to review code changes consistently before merge, before release,
#### Rust-specific checks (apply to all Rust changes)
- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
@@ -55,8 +58,8 @@ For every Rust code change, verify:
- [ ] No `f64 as usize` without prior clamping
### Concurrency
- [ ] Lock acquisition order is documented when multiple locks are used
- [ ] No `tokio::sync`write guards held across `.await` without bounded hold time
- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
- [ ] No `tokio::sync`lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
- [ ]`std::sync::Mutex` in async context is held only briefly, never across `.await`
@@ -84,12 +87,19 @@ For every Rust code change, verify:
- [ ] No camelCase statics or Hungarian notation
- [ ] New string literals don't duplicate existing constants
### Reuse and Necessity
- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
## Severity Classification
- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs``SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
bump version files to <target> (final version, ONE commit) -> merge
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
## Semver gate — confirm the target version before touching anything
Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
Rules:
- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
## Preview tag naming
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
## Phase 0 — Preflight
-`git status --short` clean; `git fetch origin main --tags`.
-`gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
```bash
git fetch origin main
PREVIEW_HASH=$(git rev-parse origin/main)# must contain the bump PR
```
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
## Phase 2 — Publish the preview tag
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>""$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and artifact verification
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
## Phase 4 — Run the artifact locally, verify the console
Work inside the session scratchpad directory; never leave stray data dirs.
-`./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
rc admin user add preview/ relcheckuser relchecksecret12
rc admin user remove preview/ relcheckuser
rc alias remove preview
```
- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
## Phase 6 — Publish the final tag on the validated commit
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}"# must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>""$PREVIEW_HASH"
git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,6 +18,8 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
-`AGENTS.md` (root and nearest path-specific files).
@@ -60,12 +60,14 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### IAM and service accounts
- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
### STS, OIDC, and federation flows
- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
@@ -97,6 +99,8 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
@@ -137,7 +141,10 @@ Use these prompts while reviewing a diff:
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
@@ -27,14 +27,15 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### IAM import, service accounts, and privilege boundaries
-`GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-`GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-`GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-`GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-`GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
### STS, OIDC, and federation flows
-`GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
-`GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
-`GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-`GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-`GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-`GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
@@ -58,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Secrets, defaults, and cryptographic misuse
-`GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and`GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-`GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-`GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-`GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-`GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checkdoc-paths-checkplanning-docs-checkclippy-checktest## Run full pre-PR checks with clippy and tests
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkclippy-checktest## Run full pre-PR checks with clippy and tests
@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
# noncurrent transition/expiry after an immediate compensation transition.
- name:Run ignored ILM integration tests serially
run:|
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
@@ -14,13 +14,19 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
## Execution Discipline
- Read the relevant existing code, tests, and local guidance before changing behavior.
- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
- State assumptions when they affect the implementation or verification path.
- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
## Autonomy and Approval Boundaries
- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
## Communication and Language
- Respond in the same language used by the requester.
@@ -41,14 +47,27 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Do not refactor existing code only to make it easier to unit test.
- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
## Constant and String Usage
## Reuse Before You Write
- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
-If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
-When changing existing behavior, keep naming and format consistency by aligning with established project constants.
Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
-**Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
-**Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
## Necessary Code Only
Net-new code — files, types, branches, comments — is cost to justify, not progress:
- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
## Sources of Truth
@@ -141,7 +160,7 @@ Pick the tier from the riskiest file touched; when in doubt, pick the higher.
- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
no runtime surface. Skip this section.
- **Mechanical:** pure renames, file moves, test-only or tooling changes —
correctness adversary only.
correctness and simplicity adversaries only.
- **Standard (the default):** any change that affects behavior.
- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
@@ -9,8 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
### Added
- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
- Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
- Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window
- Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
- Operator guide at `docs/operations/nats-jetstream.md`
- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
- Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
- Task-local storage for async-safe credential passing between middleware and auth handlers
@@ -33,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
> `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.
> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md).
### 🔒 Automated Pre-commit Hooks
#### What `make pre-commit` and `make pre-pr` actually run
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
-`RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
blocked unless their exact `scheme://host:port` origin is listed in
`RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
**NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
2.**Create a Bucket**: Use the console to create a new bucket for your objects.
3.**Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
### OIDC Roles Claim (Microsoft Entra ID)
@@ -335,12 +344,18 @@ If you have any questions or need assistance:
RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
[](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
[](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
/// Usage snapshot written by scanner implementations predating distributed
/// leadership fencing. It is read only when neither authoritative snapshot
/// copy exists.
// RUSTFS_COMPAT_TODO(scanner-usage-v2): keep .usage.json readable and removable during rolling upgrades from pre-v2 scanners. Remove after supported direct-upgrade sources all write .usage.v2.json.
returnErr("Node count must be greater than zero".into());
}
ifself.drives_per_node==0{
returnErr("drives_per_node must be greater than zero".into());
}
letpools=self.normalized_pools();
// Every referenced node index must be in range.
for(pool_idx,nodes)inpools.iter().enumerate(){
ifnodes.is_empty(){
returnErr(format!("pool {pool_idx} has no nodes").into());
}
for&ninnodes{
ifn>=self.node_count{
returnErr(format!("pool {pool_idx} references node {n} but node_count is {}",self.node_count).into());
}
}
}
// Each node must belong to exactly one pool.
letmutseen=vec![0usize;self.node_count];
fornodesin&pools{
for&ninnodes{
seen[n]+=1;
}
}
for(n,count)inseen.iter().enumerate(){
matchcount{
0=>returnErr(format!("node {n} is not assigned to any pool").into()),
1=>{}
_=>returnErr(format!("node {n} is assigned to more than one pool").into()),
}
}
// Multi-pool constraints imposed by the single-host RUSTFS_VOLUMES syntax.
ifpools.len()>1{
ifself.drives_per_node<2{
returnErr(
"multi-pool topology requires drives_per_node >= 2 (the server parser rejects a single-drive ellipses pool)"
.into(),
);
}
for(pool_idx,nodes)inpools.iter().enumerate(){
ifnodes.len()!=1{
returnErr(format!(
"pool {pool_idx} spans {} nodes; a pool striped across multiple localhost nodes is not expressible (needs host ellipses, which collides on shared paths). Use one node per pool, or the nightly multi-host lane (backlog #1313/#1314).",
nodes.len()
)
.into());
}
}
}
Ok(())
}
}
/// Represents a single RustFS server instance in a test cluster.
///
/// Each `ClusterNode` tracks the node's network address, base URL for
/// S3-compatible requests, on-disk data directory, and the underlying
/// S3-compatible requests, on-disk data directories, and the underlying
/// child process handle when the node is running.
pubstructClusterNode{
pubaddress: String,
puburl: String,
/// Primary data directory for the node. For single-drive nodes this is the
/// node's only drive; for multi-drive nodes it is the first drive. Kept as a
/// stable field so existing single-drive tests continue to compile.
pubdata_dir: String,
/// All data drives exposed by this node (`data_dirs[0] == data_dir`).
This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
"codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
overtrue
