mirror of https://github.com/rustfs/rustfs.git synced 2026-07-27 08:38:58 +00:00

Files

T

Zhengchao An 3674f5f56e fix(ecstore): bound remote shard writers with a progress deadline so one black-hole peer cannot pin write quorum (#4925 )

A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).

MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.

The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.

When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.

Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.

The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.

2026-07-16 16:41:38 +00:00

src

fix(ecstore): bound remote shard writers with a progress deadline so one black-hole peer cannot pin write quorum (#4925 )

2026-07-16 16:41:38 +00:00

AGENTS.md

docs: scope AGENTS instructions by directory (#2083 )

2026-03-05 17:25:37 +08:00

Cargo.toml

chore(deps): localize workspace dependency features (#4888 )

2026-07-16 03:55:27 +00:00

README.md

fix(admin): support external OIDC browser redirects (#4280 )

2026-07-09 18:07:49 +08:00

README.md

RustFS Config - Configuration Management

Configuration management and validation module for RustFS distributed object storage

📖 Documentation · 🐛 Bug Reports · 💬 Discussions

📖 Overview

RustFS Config provides configuration management and validation capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

Multi-format configuration support (TOML, YAML, JSON, ENV)
Environment variable integration and override
Configuration validation and type safety
Hot-reload capabilities for dynamic updates
Default value management and fallbacks
Secure credential handling and encryption

📚 Documentation

For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.

Environment Variable Naming Conventions

RustFS uses a flat naming style for top-level configuration: environment variables are RUSTFS_* without nested module segments.

Examples:

RUSTFS_REGION
RUSTFS_ADDRESS
RUSTFS_VOLUMES
RUSTFS_LICENSE
RUSTFS_LICENSE_PUBLIC_KEY

Current guidance:

Prefer module-specific names only when they are not top-level product configuration.
Renamed variables must keep backward-compatible aliases until before beta.
Alias usage must emit deprecation warnings and be treated as transitional only.
Deprecated example:
- RUSTFS_ENABLE_SCANNER -> RUSTFS_SCANNER_ENABLED
- RUSTFS_ENABLE_HEAL -> RUSTFS_HEAL_ENABLED
- RUSTFS_DATA_SCANNER_START_DELAY_SECS -> RUSTFS_SCANNER_START_DELAY_SECS

License environment variables

RUSTFS_LICENSE contains the signed license token.
RUSTFS_LICENSE_PUBLIC_KEY contains the RSA public key used to verify signed license tokens.

CORS environment variables

RUSTFS_CORS_ALLOWED_ORIGINS defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set * for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS defaults to * for the console service.

Browser redirect environment variables

RUSTFS_BROWSER_REDIRECT_URL sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example https://console.example.com. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC state is local to the RustFS node.

Scanner environment aliases

RUSTFS_SCANNER_SPEED (canonical, also accepts MINIO_SCANNER_SPEED)
RUSTFS_SCANNER_DELAY (canonical)
RUSTFS_SCANNER_MAX_WAIT_SECS (canonical)
RUSTFS_SCANNER_CYCLE (canonical, also accepts MINIO_SCANNER_CYCLE)
RUSTFS_SCANNER_START_DELAY_SECS (canonical)
RUSTFS_DATA_SCANNER_START_DELAY_SECS (deprecated alias for compatibility)
RUSTFS_SCANNER_IDLE_MODE (canonical)
RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_OBJECTS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES (canonical)

Mmap read environment aliases

RUSTFS_OBJECT_MMAP_READ_ENABLE (canonical)
RUSTFS_OBJECT_ZERO_COPY_ENABLE (deprecated alias for compatibility)

Health compatibility switches

RUSTFS_HEALTH_ENDPOINT_ENABLE
- controls canonical /health, /health/live, and /health/ready endpoint exposure.
RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE
- enables minimal payload mode for GET health responses (status, ready only).
RUSTFS_HEALTH_READINESS_CACHE_TTL_MS
- TTL for readiness cache evaluation.
RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
- enables busy protection behavior for health probes.
- default is false.
RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS
- max active HTTP requests; health probes return 429 when active requests reach or exceed this value.
- 0 disables thresholding even if busy protection is enabled.
RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
- enables KMS readiness enforcement for /health/ready.
- default is false.

Drive timeout environment variables

RUSTFS_DRIVE_METADATA_TIMEOUT_SECS
RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS
RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS
RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS

Legacy compatibility fallback:

RUSTFS_DRIVE_MAX_TIMEOUT_DURATION This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.

Drive timeout health-action policy:

RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION
- mark_failure (default): timeout marks failure and may transition drive runtime state.
- ignore_scanner: timeout does not mark failure for scanner-sensitive operations (walk_dir, read_metadata, list_dir, disk_info).

Drive timeout profile preset:

RUSTFS_DRIVE_TIMEOUT_PROFILE
- default (default): keep current timeout defaults.
- high_latency: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (read_metadata, disk_info, list_dir, walk_dir, walk_dir_stall).
Precedence:
- Explicit per-operation timeout env (RUSTFS_DRIVE_*_TIMEOUT_SECS) takes highest precedence.
- Then RUSTFS_DRIVE_MAX_TIMEOUT_DURATION legacy fallback.
- Then the profile-derived default (default or high_latency).

Startup filesystem boundary policy

RUSTFS_UNSUPPORTED_FS_POLICY controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
- warn (default): log warning and continue startup.
- fail: abort startup with an error.

RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example nfs, cifs, smb2, and fuse.*) are treated as unsupported by this startup guard.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.