Freeze the direct s3s dependency surface with a lower-only ratchet so it
cannot grow while the s3gate/gateway migration shrinks it
(rustfs/backlog#1677 review finding F1; acceptance criteria in
rustfs/backlog#1733). Baselines verified on 2026-08-05: 236 files
importing s3s, 1686 s3_error! invocation lines. Wired into make
pre-commit / pre-pr / dev-check and the Quick Checks job in ci.yml and
its ci-docs-only.yml mirror.
* chore(deps): refresh mimalloc revision
Update mimalloc and libmimalloc-sys to the requested git revision after running the dependency refresh flow.
Keep ratelimit excluded while accepting compatible dependency updates from cargo update and cargo upgrade.
Harden all-feature test compilation by giving heavy integration test crates their own recursion limit and avoiding a cross-thread spawn for the embedded startup barrier future.
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade version
---------
Co-authored-by: heihutu <heihutu@gmail.com>
PR #5719 fixed the issue #5716 per-object heal log amplification (per-object statements demoted, heal spans forced to TRACE, raw metadata dumps banned by guardrail) and superseded the demotion originally proposed here. This PR now carries only the residual cleanup on top of it:
- Convert the remaining bare-field and format-arg heal logs in crates/ecstore/src/set_disk/ops/heal.rs to the file's structured convention (event/component/subsystem + context fields): missing-object skip, disk-marked-for-healing, cannot-reconstruct errors, dangling-cleanup error, missing data_dir error, xl.meta regeneration warn, and orphan-reclaim failure warn.
- Demote the last remaining info! in the file — the per-set heal_format "set disk formats success, NoHealRequired" no-op message — to a structured debug! (error_count instead of a raw errs dump), and drop its whitelist exclusion in scripts/check_logging_guardrails.sh so the no-INFO check for set-disk heal files is strict.
No control flow or behavior changes.
Multipart upload ids embed the process-global deployment id at both create time and list time. Under plain cargo test (thread-parallel, shared process globals) a concurrently running test that re-initializes a store can swap the global between the two reads, making full-upload-id equality assertions fail spuriously (observed: core::sets::tests::list_multipart_uploads_merges_all_sets_without_pagination_loss failing when run concurrently with bucket::quota tests, passing in isolation).
Add a test-only upload_uuid_suffix helper next to deployment_upload_id and make the affected assertions compare only the decoded <uuid>x<timestamp> suffix. Where suffix normalization changes within-key ordering (base64 alphabet order is not byte order), both sides are sorted before comparison. nextest/CI is unaffected (process-per-test); this only hardens local plain cargo test runs.
initialize_local_disk_maps appended pool entries to local_disk_set_drives and inserted into local_disk_map without ever clearing previous state. Every caller (both production startup entry points and all tests) passes the FULL topology, so re-initializing the same InstanceContext left the pool/set vectors sized for the stale topology and panicked with index-out-of-bounds for wider disk indices.
This surfaced as deterministic cross-test contamination under single-process cargo test: in crates/heal heal_b920_subquorum_union_test, a 4-disk test initialized pool 0 as [None; 4] on the process-level default context, and the two 8-disk tests then panicked at disk_idx 4. Each test passed alone, and CI never caught it because cargo nextest isolates every test in its own process.
Fix: clear both registries at the start of initialize_local_disk_maps so initialization is idempotent and last-topology-wins. Add a regression unit test in ecstore (process-isolation-proof, unlike the heal integration binary) that re-initializes the same context with a wider topology; it fails with the pre-fix code.
The multipart staging namespace is one flat set of sha256(bucket/object) directories shared by every bucket, and the cross-set listing rewrite reads every upload's metadata. Two shapes poisoned the whole ListMultipartUploads response with InternalError: Corrupted format: a healthy in-flight upload belonging to another bucket (its stored owner bucket fails the guard and fell into the corrupted-format arm), and a single upload directory whose xl.meta was torn by an unclean shutdown. Docker Distribution calls ListMultipartUploads on every PATCH/commit, so either shape broke OCI registry pushes entirely (issue #5716).
Foreign-bucket uploads are now skipped silently, and directories whose metadata is affirmatively corrupt at quorum are skipped with a debug log, while every other decode failure (quorum loss from offline disks, timeouts, transport errors) keeps failing the listing so clients retry instead of silently losing entries. The degrade-vs-propagate decision is a named corrupt-family classifier with a unit test pinning both sides. FileMeta::check_xl2_v1 now classifies a missing or wrong XL2 magic as FileCorrupt instead of an anonymous io error so damage is distinguishable from transient IO faults.
Refs #5716
#5724 reclaimed the synthetic inline-rollback dir after a committed rename with delete_data_dir(recursive: true), which has no notion of object metadata: for unversioned objects the synthetic UUID is a fixed, publicly-known constant, so object/<rollback-dir> can simultaneously be a legitimate child key's directory, and recursively deleting it reopens the authorization bypass #5703 closed (PutObject on K destroying K/<uuid> without DeleteObject permission).
Replace the recursive pass with a file-precise one: after quorum commit, delete exactly object/<rollback>/xl.meta.bkp with a non-recursive delete on every disk whose rollback dir is not also the cleanup dir. The parent-rmdir walk removes the dir only when the backup was its sole content, so the BucketNotEmpty leak fix is preserved (#5724's regression test passes unchanged) while a child key at the same path keeps its metadata. The undo path's restore_metadata_backup now also reclaims the emptied synthetic dir, mirroring restore_delete_rollback.
The fallback future embeds the whole snapshot loader, and every object write nests a quota check several futures deep, so inlining it grew each write's state machine by the loader's full size — the debug-build 2MiB worker-stack overflow class fixed for bucket-config writes in #5648. Box the fallback at its call site; the allocation only happens on the degraded path.
fix(heal): demote per-object logs and cap erasure-set failure warns
Follow-up to rustfs/rustfs#5716. Per-object heal task kinds (Object/Metadata/MRF/ECDecode) queued by MRF/autoheal/scanner loops emitted info!/warn!/error! lines per object: task lifecycle (started/completed/timed_out/failed), the missing-object warn, queue admission full/drop/displacement warns, retry-admission decisions, and uncapped per-object warns in erasure-set sweeps.
Add a shared demote_to_debug_when! macro that keeps aggregate task kinds and admin/internal requests at operator-visible levels while demoting per-object occurrences to debug!, sample-cap the erasure-set transient_skip/failed warns per bucket via take_failure_log_sample (reusing the heal_bucket_objects precedent), demote the per-retry admission decision logs to debug! (covered by rustfs_heal_admission_total and the scheduler task_retrying/task_failed events), and record the previously unmetered duplicate-admission outcome.
Extend scripts/check_logging_guardrails.sh with injection-verified regression guards and run it in both quick-checks jobs (ci.yml and its ci-docs-only.yml mirror).
A failed target construction previously logged only reason=construction_failed and pushed an opaque "target construction failed" summary, discarding the underlying TargetError. Debugging rustfs#5115 showed the log repeated every 5s with no root cause, even though the error carried an actionable egress-policy rejection (RUSTFS_OUTBOUND_ALLOW_ORIGINS hint).
The Err branch now includes the error detail in both the error! log (detail field) and the returned failures summary. The detail is scrubbed against the instance's merged config via the loader's existing per-field redaction (secrets -> ***redacted***, endpoint URLs -> origin only, DSNs -> password masked) so credential-bearing values never reach logs or Admin-visible summaries.
FileInfo's derived Debug printed the full metadata map (including X-Rustfs/X-Minio-Internal-Server-Side-Encryption-Sealed-Key and -Iv values, i.e. KEK-wrapped DEK ciphertext) and the full inline data bytes (plaintext user content for non-SSE small objects), so any whole-struct log dump such as the heal_object dumps leaked user data and sealed key material into logs.
Replace the derive with a manual Debug impl that redacts encryption metadata values (keys stay visible, values print as redacted with length) under both internal prefixes, and elides data/checksum bytes to a length summary. The exhaustive destructuring forces every future field through an explicit show/redact decision. starts_with_ignore_ascii_case is made pub in rustfs-utils for reuse.
#5703 split rollback state from old-data-dir cleanup so the synthetic inline-rollback dir is reported only as rollback_data_dir and never reclaimed as if it were a real data dir. But nothing reclaims it after a successful commit either: every overwrite of an inline version by a non-inline one leaves <object>/<rollback-dir>/xl.meta.bkp behind. The residue is not referenced by any version, so it survives object deletion and DeleteBucket fails with BucketNotEmpty forever — the mass teardown cascade currently failing the S3 Implemented Tests CI lane.
Reclaim the synthetic dirs in SetDisks::rename_data once the commit holds write quorum. The quorum-failure undo inside the same function is the only consumer of the backup, so its window is closed at that point. Best-effort with the same anti-misdelete posture as commit_rename_data_dir: never touch the just-committed data dir, and residue must not fail a durable write (backlog#898).
* fix(replication): accept explicit STANDARD destination storage class
The replication engine never reads Rule.Destination.StorageClass (replica
placement comes from the bucket-target config or the source object), yet the
validator rejected any config carrying the field. The console's add-rule form
always sends StorageClass=STANDARD, so every rule created through it failed
with InvalidRequest.
Tolerate exactly STANDARD as a no-op — semantically identical to omitting
the field — and keep rejecting every other value, which would be silently
ignored rather than honored. Document the deliberate omission from the
replication capability contract.
* feat(admin): support MinIO-style partial updates for set-remote-target
set-remote-target?update=true previously replaced every stored field and
required complete credentials in the body, so flipping a target's sync mode
from the console forced operators to re-enter the secret key, and real
mc replicate update bodies (madmin Clone() strips the secret) failed to
deserialize at all.
Adopt MinIO's TargetUpdateType contract: query params creds/sync/bandwidth/
path name the field groups to overlay onto the stored target, everything
else keeps its persisted value, and unsupported groups (proxy, healthcheck,
edge, edgeSyncBeforeExpiry) fail loudly. Credentials updates are skipped for
site-replication peer targets — probed by both scheme derivations of the
stored endpoint and the stored deployment id — because an operator never
knows the site replicator's credentials, and a body-supplied deployment id
is ignored on update since it anchors peer identity. madmin JSON aliases
(bandwidthlimit, storageclass, resetID, deploymentID, sessionToken) let mc
bodies parse under deny_unknown_fields.
e2e: cover a credential-free sync-only update preserving the stored
connection and the zero-ops no-op contract; align the missing-arn assertion
with the earlier validation error.
* chore(scripts): add two-site replication lab manager
site_replication_smoke.py spawns and manages two local rustfs processes,
pairs them via the site-replication admin API (idempotent), and verifies
bidirectional object replication. Subcommands: up/down/restart/status/logs/
smoke/info/remove/clean. Stdlib-only; requests are SigV4-signed the same
way as crates/e2e_test.
* chore(scripts): rename direction-suffixed payload variables for typos check
The typos linter reads the _ba suffix in payload_ba as a misspelling of
"by"; use payload_a_to_b / payload_b_to_a instead.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Upgrading from a pre-v2 release leaves only the legacy .usage.json snapshot, which has no completeness marker and is demoted to non-authoritative, so every write to a quota-enabled bucket failed closed with a retryable 503 until the scanner's first complete cycle persisted .usage.v2.json — a production outage on large namespaces (issue #5716).
Quota admission now degrades to the last persisted per-bucket size: normalize_loaded_data_usage returns the pre-discard bucket sizes, the TTL-bounded snapshot cache retains them (carried forward through failed refreshes), and QuotaChecker::get_real_time_usage falls back to that baseline when the authoritative caches miss. The baseline is static between snapshot loads, so hard-quota enforcement is advisory for the duration of the degraded window — strictly tighter than beta.11 (usage treated as 0) and strictly more available than a blanket 503. Buckets absent from every persisted snapshot still fail closed, and removing a bucket's usage from the backend purges the baseline so a recreated bucket cannot inherit the dead incarnation's size.
Refs #5716
Both per-target replication methods assign the optimistic Completed status to rinfo before building put options, and the Err branch of replication_put_object_options returned rinfo unchanged. Since #5633 made the source encryption classification case-insensitive, managed SSE sources are rejected at this gate, and the rejection was reported as successful replication: the source object was marked COMPLETED, ObjectReplicationComplete was emitted, and nothing existed on the target.
Set replication_status = Failed (and record the error in the replicate_object branch) so the composite status, the OperationFailedReplication event, and MRF retries reflect the fail-closed outcome. This restores the contract pinned by test_bucket_replication_sse_kms_failure_contract, which timed out in the e2e-replication-nightly runs on 2026-08-03 and 2026-08-04.
* perf(observability): avoid cgroup stat key allocations
Replace the memory.stat HashMap parser with a fixed-field parser so the memory observability sampler does not allocate String keys or hash every cgroup field on each interval.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(observability): parse mimalloc stats without copying
Parse the mimalloc stats JSON while the mimalloc-owned buffer is still alive, then free it immediately. This avoids allocating an owned String on each allocator memory sample.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(time): migrate audit and notify timestamps to jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): initialize heal walk decode error
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(targets): parse MySQL event time with jiff
Preserve MySQL DATETIME(6) wall-time formatting for RFC3339 eventTime values while removing the direct chrono dependency from rustfs-targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): prune unused workspace dependencies
Apply cargo shear --fix to remove unused path-clean and s3select-api tempfile entries after the scoped jiff migration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): remove duplicate heal walk decode error init
Remove the duplicate decode_error field from the heal walk test collector initializer so lib-test clippy compiles on CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(policy): emit OPA timestamps with jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
When a returning node carries a stale object version that was deleted on
the quorum, the heal disk-walk partial callback used `resolve_union`
which picks only one entry from divergent disk entries. The minority
version was never enumerated and therefore never cleaned up.
Replace `resolve_union` + `ingest` with a new `ingest_merged` that
collects all unique versions from every partial entry across disks,
deduplicating by (name, version_id). This ensures stale data on a
returning node is surfaced for healing and can be deleted as dangling.
Fixes#5029
Classify expected metadata-missing errors separately from unknown get pipeline failures and attribute internal meta-bucket reader failures to an internal_meta path instead of legacy_duplex.
This keeps scanner/data-usage metadata probes from polluting user GET/mixed failure attribution while preserving the existing read error behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
When a fresh multi-node cluster starts, the first disk detects all disks
as unformatted and initializes the format. However, `should_init_erasure_disks`
and `quorum_unformatted_disks` only counted `UnformattedDisk` errors. Remote
peers that have not yet started their gRPC server return transient network
errors (connection refused, timeout) instead of `UnformattedDisk`, causing
the first disk to miss the "all unformatted" signal and creating a deadlock:
first disk retries endlessly while non-first disks wait for it.
Add `is_unformatted_or_transient_network` that treats transient network
errors as equivalent to `UnformattedDisk` for the bootstrap decision.
A remote disk that cannot be reached during fresh-cluster startup is
indistinguishable from an unformatted disk — the peer may simply not
have started its gRPC server yet.
Fixes#5655
Large buckets with millions of objects can take longer than the default
5-second walkdir timeout to produce the first page of listing results.
This causes timeouts in the web UI and mc CLI when opening or scanning
such buckets.
Skip the walkdir total timeout for S3 ListObjects operations when no
explicit walkdir_timeout is configured. The stall timeout (5s with no
forward progress) still protects against drives that stop responding.
This matches the scanner's existing behavior for the same reason.
Fixes#5647
Audit initialization requires the AppContext (server config + object
store) which is published by ensure_startup_after_iam inside
init_iam_runtime. Moving init_audit_runtime after init_iam_runtime
ensures the runtime sources are available when audit starts.
Fixes#5681
Co-authored-by: RustFS <hello@rustfs.com>
fix(ecstore): don't exclude pool when has_space_for is indeterminate
When `has_space_for` returns an error (not enough online disks to
reliably determine space), the old code treated this the same as
"definitely no space" via `unwrap_or_default()`, zeroing out the
pool's available capacity. During pool decommission this creates a
false "Disk full" (500) for S3 PUT requests:
- Pool 0 (decommissioning) is correctly suspended → available = 0
- Pool 1 (active) has some disks whose disk_info call fails under
heavy migration I/O → has_space_for returns Err → available forced
to 0
- get_available_pool_idx sees total = 0 → returns None → DiskFull
Fix: distinguish Ok(false) (genuinely full) from Err (indeterminate).
On Err, log a warning and fall through to compute available space from
whatever disks did respond. The actual write will enforce its own
quorum; a premature zero at the pool-selection layer is a false
rejection.
Closes#5497
Zhengchao An