gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-04 04:17:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f49827fc58a010b4bec8eb0238833580a4aa2a39
rustfs/crates/config
T
History
houseme f49827fc58 fix(iam): prevent transient IAM walk timeout from crashing startup (#3188) 
* fix(iam): prevent transient IAM walk timeout from crashing startup

  IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
  When that distributed walk times out (e.g. disk pressure after cluster reboot),
  the old code treated the failure as fatal and exited the process, causing a
  systemd restart loop.

  Changes:
  - Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
    spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
  - Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
  - `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
    `IamSysNotInitialized` (existing fail-closed behavior preserved)
  - Fix admin path boundary matching: `/minio/administrator` no longer falsely
    matches as admin prefix
  - Normalize Content-Length: 0 for admin GET requests with empty body

  Fixes #3175

* fix(iam): move constant assertion into const block

Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.

* fix(iam): address PR review comments

- Replace OnceLock with AtomicU64 sentinel for test isolation;
  add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
  compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
  finalize-retry phases so transient readiness failures are retried
  instead of leaving the server permanently degraded

* fix(iam): gate test hooks behind debug_assertions

- reset_test_failure_counter() now stores sentinel (u64::MAX) to
  correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds

* test(iam): cover deferred bootstrap recovery

Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.

Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.

* fix(startup): tighten deferred IAM recovery path

Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.

Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.

* refactor(startup): simplify IAM recovery task

Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.

Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.

* refactor(startup): trim IAM recovery test scaffolding

Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.

Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.

* fix: apply code review improvements from PR #3188 review

- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper

* style: merge has_path_prefix import into existing use block

* fix(iam): address final review follow-ups

- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes

* fix: isolate listing timeouts from drive health

Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.

Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.

* test(iam): gate deferred bootstrap test to debug

Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.

* test(ecstore): bound prior walk timeout regressions

- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable

* fmt
2026-06-03 14:37:25 +00:00
..
src
fix(iam): prevent transient IAM walk timeout from crashing startup (#3188)
2026-06-03 14:37:25 +00:00
AGENTS.md
docs: scope AGENTS instructions by directory (#2083)
2026-03-05 17:25:37 +08:00
Cargo.toml
feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242)
2026-03-20 18:52:33 +08:00
README.md
fix(ecstore): send valid ping body in remote locker (#3112)
2026-05-29 08:02:50 +00:00

README.md

RustFS

RustFS Config - Configuration Management

Configuration management and validation module for RustFS distributed object storage

CI 📖 Documentation · 🐛 Bug Reports · 💬 Discussions

📖 Overview

RustFS Config provides configuration management and validation capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

  • Multi-format configuration support (TOML, YAML, JSON, ENV)
  • Environment variable integration and override
  • Configuration validation and type safety
  • Hot-reload capabilities for dynamic updates
  • Default value management and fallbacks
  • Secure credential handling and encryption

📚 Documentation

For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.

Environment Variable Naming Conventions

RustFS uses a flat naming style for top-level configuration: environment variables are RUSTFS_* without nested module segments.

Examples:

  • RUSTFS_REGION
  • RUSTFS_ADDRESS
  • RUSTFS_VOLUMES
  • RUSTFS_LICENSE
  • RUSTFS_LICENSE_PUBLIC_KEY

Current guidance:

  • Prefer module-specific names only when they are not top-level product configuration.
  • Renamed variables must keep backward-compatible aliases until before beta.
  • Alias usage must emit deprecation warnings and be treated as transitional only.
  • Deprecated example:
    • RUSTFS_ENABLE_SCANNER -> RUSTFS_SCANNER_ENABLED
    • RUSTFS_ENABLE_HEAL -> RUSTFS_HEAL_ENABLED
    • RUSTFS_DATA_SCANNER_START_DELAY_SECS -> RUSTFS_SCANNER_START_DELAY_SECS

License environment variables

  • RUSTFS_LICENSE contains the signed license token.
  • RUSTFS_LICENSE_PUBLIC_KEY contains the RSA public key used to verify signed license tokens.

CORS environment variables

  • RUSTFS_CORS_ALLOWED_ORIGINS defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set * for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
  • RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS defaults to * for the console service.

Scanner environment aliases

  • RUSTFS_SCANNER_SPEED (canonical, also accepts MINIO_SCANNER_SPEED)
  • RUSTFS_SCANNER_CYCLE (canonical, also accepts MINIO_SCANNER_CYCLE)
  • RUSTFS_SCANNER_START_DELAY_SECS (canonical)
  • RUSTFS_DATA_SCANNER_START_DELAY_SECS (deprecated alias for compatibility)
  • RUSTFS_SCANNER_IDLE_MODE (canonical)
  • RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS (canonical)

Health compatibility switches

  • RUSTFS_HEALTH_ENDPOINT_ENABLE
    • controls canonical /health, /health/live, and /health/ready endpoint exposure.
  • RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE
    • enables minimal payload mode for GET health responses (status, ready only).
  • RUSTFS_HEALTH_READINESS_CACHE_TTL_MS
    • TTL for readiness cache evaluation.
  • RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
    • enables busy protection behavior for health probes.
    • default is false.
  • RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS
    • max active HTTP requests; health probes return 429 when active requests reach or exceed this value.
    • 0 disables thresholding even if busy protection is enabled.
  • RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
    • enables KMS readiness enforcement for /health/ready.
    • default is false.

Drive timeout environment variables

  • RUSTFS_DRIVE_METADATA_TIMEOUT_SECS
  • RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS
  • RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS

Legacy compatibility fallback:

  • RUSTFS_DRIVE_MAX_TIMEOUT_DURATION This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.

Drive timeout health-action policy:

  • RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION
    • mark_failure (default): timeout marks failure and may transition drive runtime state.
    • ignore_scanner: timeout does not mark failure for scanner-sensitive operations (walk_dir, read_metadata, list_dir, disk_info).

Drive timeout profile preset:

  • RUSTFS_DRIVE_TIMEOUT_PROFILE
    • default (default): keep current timeout defaults.
    • high_latency: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (read_metadata, disk_info, list_dir, walk_dir, walk_dir_stall).
  • Precedence:
    • Explicit per-operation timeout env (RUSTFS_DRIVE_*_TIMEOUT_SECS) takes highest precedence.
    • Then RUSTFS_DRIVE_MAX_TIMEOUT_DURATION legacy fallback.
    • Then the profile-derived default (default or high_latency).

Startup filesystem boundary policy

  • RUSTFS_UNSUPPORTED_FS_POLICY controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
    • warn (default): log warning and continue startup.
    • fail: abort startup with an error.

RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example nfs, cifs, smb2, and fuse.*) are treated as unsupported by this startup guard.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Reference in New Issue View Git Blame Copy Permalink