gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-03 20:07:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f73880f3545ea1339d4bba7f1aa9461efe31ae42
rustfs/crates/config
T
History
houseme f73880f354 fix(startup): survive slow multi-node cold starts (#4357) 
* fix(server): make startup readiness wait configurable and raise default (#4264)

The startup runtime-readiness wait was a hardcoded 30s constant with no env
override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window
is shorter than the internal startup budgets it depends on — the endpoint
DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so
readiness times out and the node exits with
`startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false`
before storage/lock quorum can converge, feeding the restart storm reported in
the issue.

- Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in
  rustfs-config health constants.
- Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a
  value of `0` falls back to the default instead of timing out instantly.
- Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so
  there is a single source of truth, and cover the getter with unit tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264)

Long-lived peer health/recovery and remote-disk monitors are detached
`tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for
their whole lifetime. Nothing cancelled them at shutdown, so on the normal
return path the Tokio runtime was dropped while they were still alive and their
`Span`s were dropped during worker-thread thread-local-storage (TLS)
destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an
already-destroyed TLS slot and panic with
`cannot access a Thread Local Storage value during or after destruction`, which
escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash
reported on Synology in issue #4264, amplified by the restart storm.

- Add `cluster::rpc::background_monitor` with a process-global shutdown token,
  `spawn_background_monitor()` (races the monitor future against that token so
  its span drops while the runtime is alive), and public
  `shutdown_background_monitors()`.
- Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn
  through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`.
- Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the
  graceful shutdown sequence (right after `ctx.cancel()`, before runtime
  teardown) via the `storage_api` compatibility boundary.

Existing recovery-probe span-context tests still pass, confirming log
correlation is preserved.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-07 14:02:00 +08:00
..
src
fix(startup): survive slow multi-node cold starts (#4357)
2026-07-07 14:02:00 +08:00
AGENTS.md
docs: scope AGENTS instructions by directory (#2083)
2026-03-05 17:25:37 +08:00
Cargo.toml
refactor(config): extract server config model (#3351)
2026-06-11 16:11:17 +08:00
README.md
config: add mmap read env alias (#4021)
2026-06-28 23:24:24 +08:00

README.md

RustFS

RustFS Config - Configuration Management

Configuration management and validation module for RustFS distributed object storage

CI 📖 Documentation · 🐛 Bug Reports · 💬 Discussions

📖 Overview

RustFS Config provides configuration management and validation capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

  • Multi-format configuration support (TOML, YAML, JSON, ENV)
  • Environment variable integration and override
  • Configuration validation and type safety
  • Hot-reload capabilities for dynamic updates
  • Default value management and fallbacks
  • Secure credential handling and encryption

📚 Documentation

For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.

Environment Variable Naming Conventions

RustFS uses a flat naming style for top-level configuration: environment variables are RUSTFS_* without nested module segments.

Examples:

  • RUSTFS_REGION
  • RUSTFS_ADDRESS
  • RUSTFS_VOLUMES
  • RUSTFS_LICENSE
  • RUSTFS_LICENSE_PUBLIC_KEY

Current guidance:

  • Prefer module-specific names only when they are not top-level product configuration.
  • Renamed variables must keep backward-compatible aliases until before beta.
  • Alias usage must emit deprecation warnings and be treated as transitional only.
  • Deprecated example:
    • RUSTFS_ENABLE_SCANNER -> RUSTFS_SCANNER_ENABLED
    • RUSTFS_ENABLE_HEAL -> RUSTFS_HEAL_ENABLED
    • RUSTFS_DATA_SCANNER_START_DELAY_SECS -> RUSTFS_SCANNER_START_DELAY_SECS

License environment variables

  • RUSTFS_LICENSE contains the signed license token.
  • RUSTFS_LICENSE_PUBLIC_KEY contains the RSA public key used to verify signed license tokens.

CORS environment variables

  • RUSTFS_CORS_ALLOWED_ORIGINS defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set * for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
  • RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS defaults to * for the console service.

Scanner environment aliases

  • RUSTFS_SCANNER_SPEED (canonical, also accepts MINIO_SCANNER_SPEED)
  • RUSTFS_SCANNER_DELAY (canonical)
  • RUSTFS_SCANNER_MAX_WAIT_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE (canonical, also accepts MINIO_SCANNER_CYCLE)
  • RUSTFS_SCANNER_START_DELAY_SECS (canonical)
  • RUSTFS_DATA_SCANNER_START_DELAY_SECS (deprecated alias for compatibility)
  • RUSTFS_SCANNER_IDLE_MODE (canonical)
  • RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_OBJECTS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES (canonical)

Mmap read environment aliases

  • RUSTFS_OBJECT_MMAP_READ_ENABLE (canonical)
  • RUSTFS_OBJECT_ZERO_COPY_ENABLE (deprecated alias for compatibility)

Health compatibility switches

  • RUSTFS_HEALTH_ENDPOINT_ENABLE
    • controls canonical /health, /health/live, and /health/ready endpoint exposure.
  • RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE
    • enables minimal payload mode for GET health responses (status, ready only).
  • RUSTFS_HEALTH_READINESS_CACHE_TTL_MS
    • TTL for readiness cache evaluation.
  • RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
    • enables busy protection behavior for health probes.
    • default is false.
  • RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS
    • max active HTTP requests; health probes return 429 when active requests reach or exceed this value.
    • 0 disables thresholding even if busy protection is enabled.
  • RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
    • enables KMS readiness enforcement for /health/ready.
    • default is false.

Drive timeout environment variables

  • RUSTFS_DRIVE_METADATA_TIMEOUT_SECS
  • RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS
  • RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS

Legacy compatibility fallback:

  • RUSTFS_DRIVE_MAX_TIMEOUT_DURATION This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.

Drive timeout health-action policy:

  • RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION
    • mark_failure (default): timeout marks failure and may transition drive runtime state.
    • ignore_scanner: timeout does not mark failure for scanner-sensitive operations (walk_dir, read_metadata, list_dir, disk_info).

Drive timeout profile preset:

  • RUSTFS_DRIVE_TIMEOUT_PROFILE
    • default (default): keep current timeout defaults.
    • high_latency: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (read_metadata, disk_info, list_dir, walk_dir, walk_dir_stall).
  • Precedence:
    • Explicit per-operation timeout env (RUSTFS_DRIVE_*_TIMEOUT_SECS) takes highest precedence.
    • Then RUSTFS_DRIVE_MAX_TIMEOUT_DURATION legacy fallback.
    • Then the profile-derived default (default or high_latency).

Startup filesystem boundary policy

  • RUSTFS_UNSUPPORTED_FS_POLICY controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
    • warn (default): log warning and continue startup.
    • fail: abort startup with an error.

RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example nfs, cifs, smb2, and fuse.*) are treated as unsupported by this startup guard.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Reference in New Issue View Git Blame Copy Permalink