* feat(scanner): expose checkpoint and source work status
* fix(scanner): count ignored checkpoints once per scan
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* fix(allocator): restore validated jemalloc target gating
Restrict the global allocator and jemalloc profiling paths to linux-gnu-x86_64 so Linux ARM64 builds fall back to mimalloc again.
Update the runtime profiling, allocator reclaim, and admin profiling handlers to use the same target gating and avoid exposing jemalloc-only code on unsupported targets.
Verification:
- cargo fmt --all
- cargo fmt --all --check
- cargo check -p rustfs
- make pre-commit
* fix(profiling): restore non-jemalloc platform behavior
Restore CPU profiling support on macOS while keeping jemalloc-backed memory profiling restricted to validated linux-gnu-x86_64 targets.
Also restore Unix log directory permission hardening so the allocator regression fix does not roll back unrelated observability behavior on other supported platforms.
Verification in progress:
- cargo fmt --all
- cargo check -p rustfs
- make pre-commit (running)
* fix(profiling): qualify periodic memory log macros
Use explicit tracing macro paths in periodic jemalloc memory profiling logging so builds do not fail when the local target configuration excludes those branches from the current import set.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(profiling): gate unsupported memory helper
Restrict the unsupported memory profiling helper to non-linux-gnu-x86_64 targets so Linux builds do not emit dead_code warnings for an unreachable fallback.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(bucket-encryption): populate default KMS key for SSE-KMS without key ID
When PutBucketEncryption receives SSE-KMS configuration without a specific
KMS key ID, query the KMS service for the default key and populate
KMSMasterKeyID before storing. This ensures GetBucketEncryption responses
include the key ID, which S3 clients like mc rely on to distinguish SSE-KMS
from SSE-S3 in their display logic.
Fixes#3039
* test(kms): cover empty bucket encryption key ID
* fix(bucket-encryption): require default KMS key
* refactor(ecstore): add accessor methods for service globals
Phase 3 of global singleton consolidation. Add ECStore accessor
methods that delegate to process-global service singletons,
providing a unified API through ECStore.
New methods:
- notification_system() — delegates to notification_sys global
- bucket_metadata_sys() — delegates to bucket metadata global
- endpoints() — delegates to endpoints global
- region() — delegates to region global
- tier_config_mgr() — delegates to tier config global
- server_config() — delegates to server config global
- storage_class() — delegates to storage class global
1161 tests pass, clippy clean, formatting clean.
* fix: use imported functions instead of fully-qualified paths
Address review comments: use imported functions instead of
crate:: paths to avoid unused import warnings.
1160 tests pass, clippy clean.
* fix(ecstore): remove unused service imports
* fix(ecstore): skip hidden metadata in walk limit
* fix(ecstore): match walk limit to visible versions
* fix(ecstore): avoid decoding all versions for limit
* refactor(ecstore): migrate config globals into ECStore struct fields
Phase 2 of global singleton consolidation. Add server_config and
storage_class fields to ECStore, sharing the same underlying data
as the process-global LazyLock statics.
New ECStore fields:
- server_config: RwLock<Option<Config>>
- storage_class: RwLock<storageclass::Config>
New async accessor methods:
- get_server_config() / set_server_config()
- get_storage_class() / set_storage_class()
Fields initialized with defaults in ECStore::new(), synced from
globals after init() completes. Global functions preserved for
backward compatibility.
1160 tests pass, clippy clean, full workspace compiles.
* fix: address PR #3219 review comments - delegate to globals
Accessors now delegate to process-global statics to avoid state drift:
- get_server_config() delegates to get_global_server_config()
- set_server_config() updates both global and local field
- get_storage_class() delegates to get_global_storage_class()
- set_storage_class() updates both global and local field
- Removed stale sync code from init()
- Changed fields to std::sync::RwLock for sync access
1160 tests pass.
* fix: correct import order for cargo fmt
* fix: simplify config accessors - delegate to globals without local state
Remove local server_config and storage_class fields from ECStore.
Accessors now purely delegate to process-global statics, eliminating
the state drift risk identified in review.
1160 tests pass, formatting clean.
* chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* chore(ecstore): add delete-objects lock batch diagnostics
Log DeleteObjects batch-lock request size, distributed lock quorum summary, and failed object keys behind RUSTFS_ISSUE3031_DIAG_ENABLE.
Include a Chinese operator checklist under docs/ that explains which logs to capture and how to interpret DeleteObjects lock_batch timeout incidents.
* chore: remove lock batch checklist from repo
Keep docs/issue-658-deleteobjects-lock-batch-checklist-zh.md only in the local checkout and drop it from the PR history.
* perf(erasure): remove UUID from clone + increase encode inflight budget
Two targeted optimizations for the erasure encoding hot path:
1. Erasure::clone() no longer generates Uuid::new_v4() per clone.
The _id field is unused in the hot path; reusing the original ID
eliminates a CSPRNG call per block encode (100 calls for a 100MB
object with 1MB blocks).
2. Default RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES raised from 8MB
to 32MB. This increases the encode pipeline depth from ~5 to ~20
blocks, allowing more read-ahead between the encoder and disk
writer stages. The per-request memory bound is still controlled
by the 8-block hard cap and the env var override.
3. Added encode_data_owned() utility method for zero-copy encoding
when the caller already owns a heap buffer (Vec<u8> → BytesMut
via Bytes::try_into_mut). Not used in the hot path yet but
available for future callers.
All 1157 ecstore tests pass. Criterion micro-benchmarks show no
regression (< 2% variance). Single-machine warp E2E tests were
inconclusive due to high variance; a dedicated multi-disk test
environment is needed for reliable E2E comparison.
Ref: https://github.com/rustfs/backlog/issues/659
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: update Cargo.lock
* fix(erasure): align encode inflight cap and tests
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* fix(http): reduce header timeout log noise
Log peer_addr for HTTP connection errors so idle or probe traffic can be traced back to the source.
Downgrade HeaderTimeout connection logs from warn to info because these timeouts are often caused by incomplete probe traffic rather than a service-side fault.
* fix(ecstore): suppress spurious warn logs for missing trash cleanup sources
Extract `reliable_rename_inner` with a `warn_on_failure` flag so that
`rename_all_ignore_missing_source` can skip the warn log when the source
is absent. This completes the idempotent cleanup fix — the previous
implementation still emitted a `reliable_rename failed` warning before
the NotFound was caught and silenced.
Also restore the `recursive` branching in `move_to_trash` so that
non-recursive deletions use `fs::rename` directly (preserving original
semantics), while recursive deletions use the idempotent helper.
`is_safe_claim_policy_name` rejected any character other than
`[a-zA-Z0-9_-]`, silently dropping policy names containing colons.
This breaks Kubernetes workload identity where `claim_name=sub`
resolves to `system:serviceaccount:<namespace>:<sa-name>` — a valid
policy name that can be created via the admin API but is then
unreachable during STS session authorization.
Add `:` and `.` to the allowed character set. These characters are:
- Used in K8s service account `sub` claims (colons)
- Used in Java/DNS-style group names from OIDC providers (dots)
- Already accepted by the `add-canned-policy` admin API endpoint
Require at least one alphanumeric character to prevent meaningless
names (`.`, `..`, `-`, `_`, `:`, etc.) from resolving.
Still rejected: `/`, `\`, whitespace, `$`, `;`, `{`, `}` and other
chars that could enable path traversal or injection.
Signed-off-by: Alexander Kharkevich <alex@mara.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: GatewayJ <835269233@qq.com>
* fix(ecstore): remove dead _buf field from Erasure struct
_buf was allocated in new(), clone(), and Default but never read or written.
Each clone() was allocating block_size bytes (typically 10MB) for nothing.
- Remove _buf from struct, Default, Clone, new_with_options
- 51 erasure tests pass
* fix: remove empty line after doc comment
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(iam): prevent transient IAM walk timeout from crashing startup
IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
When that distributed walk times out (e.g. disk pressure after cluster reboot),
the old code treated the failure as fatal and exited the process, causing a
systemd restart loop.
Changes:
- Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
- Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
- `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
`IamSysNotInitialized` (existing fail-closed behavior preserved)
- Fix admin path boundary matching: `/minio/administrator` no longer falsely
matches as admin prefix
- Normalize Content-Length: 0 for admin GET requests with empty body
Fixes#3175
* fix(iam): move constant assertion into const block
Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.
* fix(iam): address PR review comments
- Replace OnceLock with AtomicU64 sentinel for test isolation;
add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
finalize-retry phases so transient readiness failures are retried
instead of leaving the server permanently degraded
* fix(iam): gate test hooks behind debug_assertions
- reset_test_failure_counter() now stores sentinel (u64::MAX) to
correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds
* test(iam): cover deferred bootstrap recovery
Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.
Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.
* fix(startup): tighten deferred IAM recovery path
Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.
Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.
* refactor(startup): simplify IAM recovery task
Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.
Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.
* refactor(startup): trim IAM recovery test scaffolding
Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.
Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.
* fix: apply code review improvements from PR #3188 review
- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper
* style: merge has_path_prefix import into existing use block
* fix(iam): address final review follow-ups
- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes
* fix: isolate listing timeouts from drive health
Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.
Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.
* test(iam): gate deferred bootstrap test to debug
Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.
* test(ecstore): bound prior walk timeout regressions
- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable
* fmt
* fix(ecstore): retry transient walk dir stream errors
Retry RemoteDisk walk_dir once when the first failure looks transport-related so restart-time HttpReader stream errors do not immediately count as hard listing failures.
Improve HttpReader request and stream error messages with method and URL context, and add regression coverage for retry recovery and diagnostics.
* fix(ecstore): avoid retry after partial walk dir stream
Stop retrying walk_dir after copy_stream_with_buffer has already written partial bytes into the destination writer.
Keep the retry only for open_walk_dir failures and add a regression test that proves partial stream failures are returned without issuing a second stream.
Add `RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI[_<SUFFIX>]` setting that
removes a provider from the login page while keeping it fully
functional for STS AssumeRoleWithWebIdentity and site-replication.
Changes:
- Add `hide_from_ui: bool` to `OidcProviderConfig`
- Add `list_visible_providers()` that filters hidden providers
(used by console login and /v3/oidc/providers endpoint)
- Keep `list_providers()` unfiltered for site-replication/admin config
- Extract `normalize_provider_config(config) -> config` to deduplicate
field normalization (accepts the struct directly, not 18 parameters)
- Add `parse_enable_state()` helper for consistent EnableState parsing
- Plumb through admin API request structs (`#[serde(default)]`)
- Expose in `OidcConfigView` for admin GET config round-trip
- Persist via `upsert_persisted_provider_config()`
Note: adding `hide_from_ui` to the public `OidcProviderConfig` struct
is a source-level change for code constructing it with struct literals.
This is acceptable for the current pre-1.0 release cycle.
Signed-off-by: Alexander Kharkevich <alex@mara.com>
Co-authored-by: GatewayJ <835269233@qq.com>
* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
* fix(rio): surface incomplete put bodies
Propagate incomplete PUT request bodies as IncompleteBody instead of allowing erasure encode to treat truncated input as a normal EOF.\n\n- mark premature EOFs in HardLimitReader with an explicit IncompleteBody error\n- preserve EOF error chains through read_full and map them to S3 IncompleteBody\n- stop erasure encode from swallowing UnexpectedEof on truncated input\n- add regression tests for reader, erasure encode, and API error mapping\n\nRefs: rustfs/backlog#654
* fix(s3): honor decoded length for aws chunked put
Use x-amz-decoded-content-length for aws-chunked PutObject requests so trailer-checksum uploads are sized against the decoded payload instead of the wire-encoded content-length.\n\n- prefer decoded content length for aws-chunked put bodies\n- add a regression test covering the size selection logic\n- keeps the incomplete body fix working for truly truncated uploads while restoring checksum trailer compatibility\n\nRefs: rustfs/backlog#654
* fix(io): follow up review comments on incompletebody handling
Address PR review feedback by restoring read_full's existing EOF contract, adding a dedicated read_full_or_eof helper for erasure encoding, covering nested incomplete-body error chains, and documenting plus hardening aws-chunked size selection.\n\n- keep read_full returning early EOF on empty reads\n- use read_full_or_eof only in erasure encoding paths\n- detect aws-chunked via content-encoding or transfer-encoding\n- add nested error-chain and aws-chunked regression tests\n\nRefs: rustfs/backlog#654
* fix(rio): surface incomplete put bodies
Propagate incomplete PUT request bodies as IncompleteBody instead of allowing erasure encode to treat truncated input as a normal EOF.\n\n- mark premature EOFs in HardLimitReader with an explicit IncompleteBody error\n- preserve EOF error chains through read_full and map them to S3 IncompleteBody\n- stop erasure encode from swallowing UnexpectedEof on truncated input\n- add regression tests for reader, erasure encode, and API error mapping\n\nRefs: rustfs/backlog#654
* fix(s3): honor decoded length for aws chunked put
Use x-amz-decoded-content-length for aws-chunked PutObject requests so trailer-checksum uploads are sized against the decoded payload instead of the wire-encoded content-length.\n\n- prefer decoded content length for aws-chunked put bodies\n- add a regression test covering the size selection logic\n- keeps the incomplete body fix working for truly truncated uploads while restoring checksum trailer compatibility\n\nRefs: rustfs/backlog#654
* fix(io): follow up review comments on incompletebody handling
Address PR review feedback by restoring read_full's existing EOF contract, adding a dedicated read_full_or_eof helper for erasure encoding, covering nested incomplete-body error chains, and documenting plus hardening aws-chunked size selection.\n\n- keep read_full returning early EOF on empty reads\n- use read_full_or_eof only in erasure encoding paths\n- detect aws-chunked via content-encoding or transfer-encoding\n- add nested error-chain and aws-chunked regression tests\n\nRefs: rustfs/backlog#654
* fix(rio): reject bytes beyond hard limit
* fix(ecstore): reject zero-sized erasure blocks
Henry Guo