gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 09:08:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
769549dd2cc5259de98bbce301f0c65f9fc1da9c
rustfs/crates/config
T
History
houseme eebd16d8a4 feat(cache): add object data cache engine and app flow (#4187) 
* feat(cache): add object data cache engine

* feat(cache): wire app-layer object cache flow

* refactor(cache): streamline app-layer cache flow

* refactor(cache): tighten cache flow internals

* refactor: address final clippy cleanup

* chore(deps): update quick-xml to 0.41.0

* feat(cache): wire object data cache env config

* fix(cache): gate materialize fill by cache plan

* chore(cache): add object data cache benchmark gate

* fix(cache): guard object cache fill size mismatches

* refactor(cache): streamline object cache body planning

* fix(cache): align object cache rollout config

* test(cache): cover buffered object cache benchmark

* test(cache): isolate object cache benchmark metrics

* test(cache): mark materialize rollout experimental

* test(cache): tighten object cache benchmark gate

* fix(cache): address review findings for object data cache

- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test

- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)

- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space

- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window

- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry

* fix(cache): close invalidation races and drop full-cache scan on writes

- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test

- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)

- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)

- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(storage): fix media-dependent buffer clamp expectation

test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(storage): format buffer clamp assertion

* chore(logging): update tier guardrail path

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-03 18:11:14 +08:00
..
src
feat(cache): add object data cache engine and app flow (#4187)
2026-07-03 18:11:14 +08:00
AGENTS.md
docs: scope AGENTS instructions by directory (#2083)
2026-03-05 17:25:37 +08:00
Cargo.toml
refactor(config): extract server config model (#3351)
2026-06-11 16:11:17 +08:00
README.md
config: add mmap read env alias (#4021)
2026-06-28 23:24:24 +08:00

README.md

RustFS

RustFS Config - Configuration Management

Configuration management and validation module for RustFS distributed object storage

CI 📖 Documentation · 🐛 Bug Reports · 💬 Discussions

📖 Overview

RustFS Config provides configuration management and validation capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

  • Multi-format configuration support (TOML, YAML, JSON, ENV)
  • Environment variable integration and override
  • Configuration validation and type safety
  • Hot-reload capabilities for dynamic updates
  • Default value management and fallbacks
  • Secure credential handling and encryption

📚 Documentation

For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.

Environment Variable Naming Conventions

RustFS uses a flat naming style for top-level configuration: environment variables are RUSTFS_* without nested module segments.

Examples:

  • RUSTFS_REGION
  • RUSTFS_ADDRESS
  • RUSTFS_VOLUMES
  • RUSTFS_LICENSE
  • RUSTFS_LICENSE_PUBLIC_KEY

Current guidance:

  • Prefer module-specific names only when they are not top-level product configuration.
  • Renamed variables must keep backward-compatible aliases until before beta.
  • Alias usage must emit deprecation warnings and be treated as transitional only.
  • Deprecated example:
    • RUSTFS_ENABLE_SCANNER -> RUSTFS_SCANNER_ENABLED
    • RUSTFS_ENABLE_HEAL -> RUSTFS_HEAL_ENABLED
    • RUSTFS_DATA_SCANNER_START_DELAY_SECS -> RUSTFS_SCANNER_START_DELAY_SECS

License environment variables

  • RUSTFS_LICENSE contains the signed license token.
  • RUSTFS_LICENSE_PUBLIC_KEY contains the RSA public key used to verify signed license tokens.

CORS environment variables

  • RUSTFS_CORS_ALLOWED_ORIGINS defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set * for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
  • RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS defaults to * for the console service.

Scanner environment aliases

  • RUSTFS_SCANNER_SPEED (canonical, also accepts MINIO_SCANNER_SPEED)
  • RUSTFS_SCANNER_DELAY (canonical)
  • RUSTFS_SCANNER_MAX_WAIT_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE (canonical, also accepts MINIO_SCANNER_CYCLE)
  • RUSTFS_SCANNER_START_DELAY_SECS (canonical)
  • RUSTFS_DATA_SCANNER_START_DELAY_SECS (deprecated alias for compatibility)
  • RUSTFS_SCANNER_IDLE_MODE (canonical)
  • RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_OBJECTS (canonical)
  • RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES (canonical)

Mmap read environment aliases

  • RUSTFS_OBJECT_MMAP_READ_ENABLE (canonical)
  • RUSTFS_OBJECT_ZERO_COPY_ENABLE (deprecated alias for compatibility)

Health compatibility switches

  • RUSTFS_HEALTH_ENDPOINT_ENABLE
    • controls canonical /health, /health/live, and /health/ready endpoint exposure.
  • RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE
    • enables minimal payload mode for GET health responses (status, ready only).
  • RUSTFS_HEALTH_READINESS_CACHE_TTL_MS
    • TTL for readiness cache evaluation.
  • RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
    • enables busy protection behavior for health probes.
    • default is false.
  • RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS
    • max active HTTP requests; health probes return 429 when active requests reach or exceed this value.
    • 0 disables thresholding even if busy protection is enabled.
  • RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
    • enables KMS readiness enforcement for /health/ready.
    • default is false.

Drive timeout environment variables

  • RUSTFS_DRIVE_METADATA_TIMEOUT_SECS
  • RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS
  • RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS
  • RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS

Legacy compatibility fallback:

  • RUSTFS_DRIVE_MAX_TIMEOUT_DURATION This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.

Drive timeout health-action policy:

  • RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION
    • mark_failure (default): timeout marks failure and may transition drive runtime state.
    • ignore_scanner: timeout does not mark failure for scanner-sensitive operations (walk_dir, read_metadata, list_dir, disk_info).

Drive timeout profile preset:

  • RUSTFS_DRIVE_TIMEOUT_PROFILE
    • default (default): keep current timeout defaults.
    • high_latency: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (read_metadata, disk_info, list_dir, walk_dir, walk_dir_stall).
  • Precedence:
    • Explicit per-operation timeout env (RUSTFS_DRIVE_*_TIMEOUT_SECS) takes highest precedence.
    • Then RUSTFS_DRIVE_MAX_TIMEOUT_DURATION legacy fallback.
    • Then the profile-derived default (default or high_latency).

Startup filesystem boundary policy

  • RUSTFS_UNSUPPORTED_FS_POLICY controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
    • warn (default): log warning and continue startup.
    • fail: abort startup with an error.

RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example nfs, cifs, smb2, and fuse.*) are treated as unsupported by this startup guard.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Reference in New Issue View Git Blame Copy Permalink