mirror of https://github.com/rustfs/rustfs.git synced 2026-11-08 19:43:05 +00:00

Files

T

Chris An d6210ac17b fix(s3,ecstore,table-catalog): bound work driven by client input (#9172 )

* fix(ecstore,heal,select): bound storage-layer work driven by client input

Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:

- Multipart: upload ids are canonicalized before they name a staging path
or a lock key (backlog#3027); trailing-slash keys are encoded like every
other handler (backlog#3055); unlisted parts are removed at completion
(backlog#3106); per-bucket uploads are found through a signed per-disk
index instead of a scan of the shared staging area (backlog#3121);
encrypted part ETags carry a per-upload secret so a write-only user
cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
(backlog#3038); a delete marker on a key whose xl.meta path is a
directory, and sidecar names used as child keys, no longer wedge the
journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
counts delete markers against max-keys (backlog#3108), and rejects a
marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
bucket is skipped instead of failing the rebuild (backlog#3040), and
directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
modification time (backlog#3073, backlog#3074); Snowball transport
headers are not forwarded to the tier (backlog#3090); an epoch
modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
(backlog#3131); SelectObjectContent releases its namespace locks once
the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
(backlog#3129); a bucket-metadata reload timeout no longer marks the
peer offline (backlog#3124); replication target requests carry an
operation deadline (backlog#3102); the scanner resumes its interrupted
wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
batches, Parquet dictionary bounds and pre-planner projection checks
(backlog#3075, backlog#3076, backlog#3077, backlog#3078).

* test(e2e): seed legacy duplicate bucket tags offline for the OPA contract

* fix(s3): authorize what handlers store and bound request-driven work

Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.

Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
the tag set it copies (backlog#3034); HeadObject with the replication
check header still authorizes s3:GetObject (backlog#3047); the
x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
recursive force-delete with a REPLICA header authorizes every
descendant as a delete (backlog#3070); include-deleted listings require
s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
table isolation and uses the same prefix it authorizes (backlog#3053,
backlog#3054); If-Match probing needs read permission (backlog#3097);
Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
returns lock and tag headers only with permission (backlog#3113);
IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
(backlog#3049); bucket policies are parsed once and cached
(backlog#3032); keystone: access keys no longer verify with an empty
secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
(backlog#3023); PutObject takes buffer and foreground admission only
after body bytes arrive and bounds the wait (backlog#3031,
backlog#3084); xl.meta size is bounded on the write path
(backlog#3100); XML compat layers stream instead of buffering whole
bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
completion (backlog#3130); MinIO admin decryption runs Argon2id off the
async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
(backlog#3092, backlog#3093); control characters in keys and lifecycle
rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
Content-MD5 and admin/STS reject SigV2 replays with altered query or
body (backlog#3037, backlog#3026); request logs redact presigned
queries and handler spans skip credentials (backlog#3068,
backlog#3107); SSE-C listings and completions withhold plaintext
metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
keys (backlog#3126); notifications fire after a client disconnect
(backlog#3127); ListenNotification streams re-check access
(backlog#3119).

* fix(api): drop imports left unused by the base merge

* test(s3): move the KMS multipart denial test to the app layer

The test drives app-layer multipart use cases, so it belongs with them;
living in the SSE module it imported app code from the infra layer and
failed the layer dependency guard. The SSE test helpers it borrows are
exposed through the storage and app facades.

* fix(s3): keep the SSE-C key fingerprint on keyless multipart completion

The backlog#3050 fix withheld both the plaintext checksums and the
customer-key MD5 header from a CompleteMultipartUpload that did not
present the key. The checksums are a digest of the plaintext and stay
hidden; the key MD5 identifies the key only, and the S3 conformance suite
(test_copy_part_enc) expects it on a keyless completion, so it is echoed
again.

* fix(tier): stream GET bodies and document replication budgets

* fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes

Two lifecycle regressions surfaced once the storage and request batches
ran together against main:

- The multipart index owner cloned the disk's pinned publication root and
bootstrapped on a task of its own. That kept the mount alive past the
last LocalDisk owner (the disk lifecycle allows exactly three holders of
the root FD) and left a stray task behind, which the local-disk and
read-repair tests detect. The owner now keeps only the root path and
reopens it for identity checks, and the bootstrap scan rides the disk's
existing cleanup task, where stopping the disk abandons it and a
reconnect resumes it.
- The disk-read admission wrapper defers releasing its permit until an
in-flight read completes, so a cancelled client cannot widen disk
concurrency. The cold-fill producer, however, drops a read it has
already classified as stalled when its deadline fires, and that permit
stayed pinned for the lifetime of the stuck read. The producer now
abandons the admission explicitly on timeout.

* test(ecstore): pass the index owner to the Linux cleanup-loop test

* fix(s3): release cold-fill admission after watchdog handoff

* fix(s3,ecstore): close the round-1 adversarial review findings

Adversarial review of the combined request- and storage-layer batch found
fixes that fell short of their issues, two behaviour regressions for
legitimate clients and several unrequested costs:

- backlog#3038: a bulk DeleteObjects reserved the whole per-disk
inspection allowance up front for every key in flight and exhausted the
16 Mi request budget on any set wider than eight drives, failing healthy
keys with SlowDown. Calls are now admitted while budget remains and
charged for the entries they actually visited. Object-scoped discovery
charges plain child keys at one entry per 64 names instead of one each,
so deleting a missing key that names a folder prefix completes instead
of answering 503 forever while a hostile prefix stays bounded. A peer
still on the previous binary is reported
as an unsupported participant, the repository's rolling-upgrade
contract, instead of as a budget verdict.
- backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was
still replayable with another body; it is refused with MissingContentMD5
while SigV4 keeps the optional-MD5 contract.
- backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before
authorization like PUT and COPY, so a replica delete cannot remove a
version the policy never saw.
- backlog#3121: the multipart index no longer fsyncs a marker and four
directories per multipart metadata write (it is rebuilt on every start),
and ListMultipartUploads discovers roots concurrently.
- backlog#3077: Parquet SelectObjectContent decodes budget-sized batches
instead of one row per batch.
- backlog#3031: a full upload buffer pool hands out an unpooled buffer
instead of refusing the upload; the progress floor already bounds stalls.
- backlog#3023: the GET disk-read idle cut-off is configurable
(RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused
client is not truncated after 30 s.
- backlog#3075: JSON Lines schema inference keeps the sample collected at
the 4 MiB bound instead of failing the query.
- backlog#3024: compat admin decryption queues briefly for a slot instead
of refusing the third concurrent request.
- backlog#3113: HEAD skips its permission probes for the bucket owner.
- backlog#3117: control characters in a lifecycle rule ID are rejected on
PutBucketLifecycle as well.

The lock-crate publication FIFO serves backlog#3111 and moves to the
table-catalog PR. Also fixed: a lossy admin error message, an unused
s3-client function, two unused test-util features, a duplicated key
validation, the s3s footprint baseline, and a stale log event name in the
reverse-proxy runbook. The ODM multipart write-back test runs on the large
test stack like its sibling.

* fix(table-catalog): bound maintenance work and isolate table namespaces (#9174)

* fix(table-catalog): bound maintenance work and isolate table namespaces

Scan findings in the Iceberg table catalog:

- Metadata maintenance deleted every unreferenced *.metadata.json in a
table's directory without checking whether a renamed table in another
namespace still owned those files (backlog#3060); it now resolves the
owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
writer schema and no recursion depth limit; a self-referencing record
schema overflowed the stack (backlog#3080). Schemas are validated for
depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
memory; a decompression bomb OOM-killed the server (backlog#3085). Row
groups are streamed through a byte budget and oversized inputs are
refused.
- An S3 request for a key outside an indexed table prefix listed the
bucket's whole namespaces/ catalog prefix and failed once it held more
than 4096 objects (backlog#3086); a per-bucket directory index answers
index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
left the bucket's catalog entry locked at 503 (backlog#3088); the rename
runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
starving every object write (backlog#3111); commits now hold it under a
deadline and object writes are admitted between commits.

The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.

The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.

* fix(table-catalog): close the review gaps in the scan-fix batch

Adversarial review of the table-catalog batch found the fixes narrower
than the issues they close and two cost regressions:

- backlog#3085: the compaction footer check bounded nothing a page header
could not override; parquet sizes its page buffers from the header.
Walk every column chunk's page headers (Thrift compact, no decompression)
and hold each page and the chunk total to the budget before decoding.
The stored-input budget is per rewrite group, so a multi-group job no
longer fails after writing earlier outputs.
- backlog#3086: the default index-miss scan still counted maintenance
reports toward its 4096-object limit; only table entries count now, and
the listing itself is bounded to 64 pages. The directory index remains
an optional accelerator.
- backlog#3111: a create wrote its metadata file under the fence as the
first durable mutation, which disarmed the deadline before the unbounded
registration inventory. The write precedes publication now. The deadline
gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS,
default 30 s, never below the lock-acquire timeout) instead of reusing
the 5 s lock timeout, and the ownership inventory reads entries
concurrently.
- Directory candidates are written only while a directory exists; the
backfill covers earlier entries. A directory on the migration-fence
path is reported as a conflict instead of a parse error.
- Dead test hooks, a stale comment and the support-matrix text that did
not match the code are removed or corrected.

* fix(select): preserve complete samples and Parquet batch limits

* fix(s3,ecstore): close the round-2 adversarial review findings

DeleteObject resolved x-rustfs-source-version-id for authorization on
every request while storage consumes it only for a REPLICA request, so a
non-replica caller with s3:DeleteObjectVersion could trigger a plain
delete of the current version. The hook now resolves the header under
the same predicate as storage, and the regression test covers the
non-replica case.

HEAD no longer skips its permission probes for the bucket owner; an
explicit bucket-policy Deny on the metadata actions binds the owner too.

Multipart index pages verify each root against the staging area and
prune markers whose staging root is gone, under the marker's own lease
with a re-check, so the per-drive index stays proportional to live
uploads instead of every key ever multipart-initiated on the drive.

SigV2 configuration PUTs that declare an empty body need no Content-MD5:
there is no body to replay. The minimum-progress floor stays independent
of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now
says so instead of claiming 0 disables it. The quota bootstrap window
(503 until the background scan publishes the ledger) is kept, because
ledger admission must not come from scanner reporting; the compatibility
matrix now documents it.

The S3 compatibility matrix gains a section for the client-visible
validation bounds, and the drive-timeout tuning guide tabulates
RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS.

* fix(lock): queue table-bucket publication writers fairly

Table commits take the bucket publication fence as a write lock; under a
stream of commits the lock's try-and-retry acquisition let a new commit
jump ahead of object writers that had been waiting longer, which is part
of what starved them (backlog#3111). Requests for the publication lock
path now take a FIFO ticket per shard: a waiter is admitted only when it
is at the head of the queue, cancelled waiters release their turn, and
batch acquisitions join the same queue. Other lock paths are unchanged.

This moved here from the request-layer batch, where no cited issue
covered it; it serves the table-catalog issue this PR closes.

* fix(table-catalog): close the round-2 adversarial review findings

Parquet page-header walker: read the size fields by field id as parquet
does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so
a header encoding uncompressed_page_size as I64 can no longer bypass the
budget while parquet allocates it. Bound every list, set and map count by
the bytes remaining in the chunk, and reject bool collection elements
(parquet skips them with zero width, so either width would desynchronize
the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>.
Start each column-chunk walk at the dictionary page offset whenever the
footer declares one, exactly as parquet's byte_range does, instead of
min(dictionary, data).

Footer pre-walk: before parquet decodes a footer, walk FileMetaData with
the hardened cursor and charge the declared row_groups count times the
RowGroupMetaData size to the decode budget, because parquet 59 reserves
that Vec before reading one element.

Preparation deadline: clamp the
RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400
seconds so Instant + Duration cannot overflow.

Index-miss scan: after each listing page, jump the cursor past the
maintenance subtree the page ended in, and allow one extra page per table
entry already listed, so job-report history scales with the catalog while
a report flood under one table stays bounded.

Migration fence: read the fence-path object once and decide from its
stored kind whether it is directory state or a fence.

Expired or failed create: remove the initial metadata file when the
registration fails before any catalog write may have begun; nothing else
references it. Correct the comment and support matrix that claimed
maintenance reclaims it.

Tests cover each walker divergence, the footer row-group reservation, the
deadline ceiling, the paging budget for honest catalogs and for a report
flood, and the removal of the unpublished metadata file.

* test(admin): register the catalog directory routes in the route matrix

The table-catalog batch added GET, POST and DELETE
{warehouse}/catalog/directory on both the Iceberg and the compat admin
prefixes without listing them in the admin route matrix, so
test_admin_route_matrix_matches_registered_routes failed on CI.

* fix(s3,ecstore): close the round-3 adversarial review findings

Multipart index pages now treat the cursor as the last examined marker
rather than the last root, so a page that pruned a stale marker still
validates and ListMultipartUploads keeps using the index. A marker is
written before its staging root is published, so pruning spares markers
younger than a five-minute grace period and an existing marker is
touched on every new upload; the prune re-checks both under the marker
lease. Bounded delete-recovery inspection rides the durable-mutation
lane like the unbounded path instead of the one-worker foreground lane.
The on-demand-migration inline body budget scales with the object size
(at least 64 KiB/s) instead of a flat 120 s. The control-body digest
service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no
inactivity timeout" like every other reader of that knob.

Quota reconciliation throttle objects are deleted together with a
retired generation's ledger, so bucket turnover leaves no residue. The PUT guard
test shim is gone; tests call the shared guard. Bounded DiskAPI defaults
answer MethodNotAllowed like their siblings. Stale
budget docs, an unreachable CSV arm, an always-Some limit parameter,
commented-out constants and a misplaced doc comment are cleaned up;
weak test assertions name the expected error; lifecycle rule-ID control
characters gain a test. The compatibility matrix records the CopyObject
tag-copy permission, the exact SigV2 empty-body predicate, replica and
migration writes of control-character keys, and the SigV2 DeleteObjects
digest rule.

* fix(table-catalog,heal): close the round-3 review findings

A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job,
TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before
any output is written, and name the ceiling in the commit-path errors;
document the effective cap in the support matrix and scheduler runbook.
B. Keep TaskCancelled/TaskTimeout classification on replacement passes:
the namespace verification runs only for completed or storage-failed
passes, in the erasure healer and the task bucket prepass.
C. Compare the replacement bucket pass per bucket: a recorded identity
that is gone or replaced still errors or restarts the pass, buckets
created mid-pass are appended to the identities, pending_buckets and a
persisted positional order without rewinding progress or the
checkpoint; verification ignores additions.
D. Read the migration-fence object once in the object-backed write permit
and dispatch on its kind.
E. Drop the strong store's copy of maintenance_cleanup_roots.
F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and
pause_after_metadata; make compact_parquet_data_files and
PublicationTicket.state private.
G. Document the body-revision identity, the legacy revalidation budget
and its HTTP 503 in the support matrix.
H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it
from heal through the storage_api facade.

* fix(ecstore): cap the index pages one disk may answer

Pruned multipart index pages may carry few roots, so the page cursor no
longer bounds how many pages a peer can make a listing fetch. Cut a disk
off after 4096 pages (about one million examined markers) so it
contributes no coverage and the listing falls back to the staging scan.

* fix(ecstore,table-catalog): close the round-4 adversarial review findings

Multipart index markers dated in the future (a backwards clock step
between recording a marker and publishing its staging root) were judged
old and could be pruned while their upload was in flight; a future-dated
marker is now kept. The marker touch on EEXIST is best effort, so a
filesystem that refuses to set times no longer fails the upload commit.
Pruning runs on the durable-mutation lane instead of the one-worker
foreground lane.

Quota reconciliation keeps one throttle object per bucket incarnation (a
quota revision change replaces it instead of orphaning the previous
revision's object), the retirement cleanup deletes it ahead of the ledger
and also when the ledger is already gone, and a scan the retirement
already refused writes nothing back. Bounded DiskAPI defaults answer
UnsupportedDisk, the verdict the aggregate gives a peer without the
protocol. The compaction ceiling pre-check runs over every rewrite group
before any output is written, and the worker's ceiling error no longer
names a placeholder group.

Docs: the inline on-demand-migration client budget, the protocol-2
downgrade conflict of replacement passes, the exact footprint numbers.

2026-11-05 23:32:54 +08:00

src

fix(s3,ecstore,table-catalog): bound work driven by client input (#9172 )

2026-11-05 23:32:54 +08:00

AGENTS.md

docs: scope AGENTS instructions by directory (#2083 )

2026-03-05 17:25:37 +08:00

Cargo.toml

feat: extend hotpath coverage across crates (#5505 )

2026-07-31 05:42:19 +00:00

README.md

feat(s3): serve Veeam SOSAPI system and capacity XML (#9007 )

2026-10-28 22:04:49 +08:00

README.md

RustFS Config - Configuration Management

Configuration management and validation module for RustFS distributed object storage

📖 Documentation · 🐛 Bug Reports · 💬 Discussions

📖 Overview

RustFS Config provides configuration management and validation capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

Multi-format configuration support (TOML, YAML, JSON, ENV)
Environment variable integration and override
Configuration validation and type safety
Hot-reload capabilities for dynamic updates
Default value management and fallbacks
Secure credential handling and encryption

📚 Documentation

For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.

Environment Variable Naming Conventions

RustFS uses a flat naming style for top-level configuration: environment variables are RUSTFS_* without nested module segments.

Examples:

RUSTFS_REGION
RUSTFS_ADDRESS
RUSTFS_VOLUMES
RUSTFS_LICENSE
RUSTFS_LICENSE_PUBLIC_KEY

Current guidance:

Prefer module-specific names only when they are not top-level product configuration.
Renamed variables must keep backward-compatible aliases until before beta.
Alias usage must emit deprecation warnings and be treated as transitional only.
Deprecated example:
- RUSTFS_ENABLE_SCANNER -> RUSTFS_SCANNER_ENABLED
- RUSTFS_ENABLE_HEAL -> RUSTFS_HEAL_ENABLED
- RUSTFS_DATA_SCANNER_START_DELAY_SECS -> RUSTFS_SCANNER_START_DELAY_SECS

License environment variables

RUSTFS_LICENSE contains the signed license token.
RUSTFS_LICENSE_PUBLIC_KEY contains the RSA public key used to verify signed license tokens.

CORS environment variables

RUSTFS_CORS_ALLOWED_ORIGINS defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set * for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS defaults to * for the console service.

Console URL prefix

RUSTFS_CONSOLE_PREFIX changes the embedded console URL prefix. The default is /rustfs/console. For example, RUSTFS_CONSOLE_PREFIX=/console serves the UI at http://localhost:9001/console/. Nested prefixes such as /management/console are supported; one trailing slash is removed. Restart the server after changing it.

The prefix must be a non-root absolute path of at most 256 bytes, with nonempty segments containing only ASCII letters, digits, -, _, ., or ~. Dot segments, encoded characters, and overlaps with reserved admin, RPC, health, profiling, browser entry, and icon routes are rejected at startup. / is not supported. Choose a prefix that does not collide with S3 bucket paths.

The console routes, embedded frontend asset URLs, browser redirects, and OIDC console redirects use this prefix. Admin API paths and the identity provider's /rustfs/admin/v3/oidc/callback/... URL remain unchanged. RUSTFS_CONSOLE_ADDRESS continues to control only the listening address and port. The server adapts bundled console asset references from their build-time base path to the runtime prefix.

OEM builds can set RUSTFS_CONSOLE_BASE_PATH when compiling RustFS to embed a different default, such as /nuofans/console. Build the bundled console with the same NEXT_PUBLIC_BASE_PATH. An unset or empty build variable retains /rustfs/console. The build path must satisfy the validation rules above and must not have a trailing slash.

At startup, RUSTFS_CONSOLE_PREFIX takes precedence over the compiled default. Changing RUSTFS_CONSOLE_BASE_PATH when starting an existing binary has no effect; rebuild both components to change the embedded default. If a runtime prefix is configured, asset adaptation uses the compiled base path as its source, including when restoring /rustfs/console for a custom OEM build.

Browser redirect environment variables

RUSTFS_BROWSER_REDIRECT_URL sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example https://console.example.com. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC state is local to the RustFS node.

S3 API environment variables

RUSTFS_API_OBJECT_MAX_VERSIONS caps the number of retained versions for a single object. It defaults to 9223372036854775807, matching MinIO's practical-unlimited default. Set a positive integer to enforce a lower per-object metadata bound.
RUSTFS_VEEAM_SOSAPI_ENABLE serves Veeam SOSAPI system.xml and capacity.xml for an authenticated GET or HEAD of those exact bucket-root keys. It defaults to true. Explicit version reads use the normal object path. Capacity requests return ServiceUnavailable until the scanner publishes a complete snapshot certified for the current bucket incarnation and authoritative quota metadata can be read. Legacy or mixed-version snapshots without an incarnation proof cannot serve capacity until a compatible scanner publishes a certified snapshot. Capacity also returns ServiceUnavailable when cluster or disk observations are missing, stale, or degraded. Retry after scanner convergence and storage or metadata recovery. Capacity assumes shard paths have independent backing space; shared physical storage can make the estimate optimistic. Set the flag to false to restore the normal object path, including deletion of a real object previously stored at that key.
MINIO_API_OBJECT_MAX_VERSIONS is accepted as a compatibility alias when the canonical RustFS variable is not set.

Distributed endpoint locality

RUSTFS_LOCAL_ENDPOINT_HOST identifies this server's host in a distributed RUSTFS_VOLUMES topology without resolving every peer during startup. Set it to exactly one host, without a scheme, port, or path. It is accepted only for orchestrated URL topologies and must match at least one endpoint on the RustFS server port; invalid or unmatched values fail startup. Leave it unset to retain DNS-based locality discovery.

Scanner environment aliases

RUSTFS_SCANNER_SPEED (canonical, also accepts MINIO_SCANNER_SPEED)
RUSTFS_SCANNER_DELAY (canonical)
RUSTFS_SCANNER_MAX_WAIT_SECS (canonical)
RUSTFS_SCANNER_CYCLE (canonical, also accepts MINIO_SCANNER_CYCLE)
RUSTFS_SCANNER_START_DELAY_SECS (canonical)
RUSTFS_DATA_SCANNER_START_DELAY_SECS (deprecated alias for compatibility)
RUSTFS_SCANNER_IDLE_MODE (canonical)
RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS (canonical)
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_OBJECTS (canonical)
RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES (canonical)
RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS (canonical)
RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS (canonical)
RUSTFS_SCANNER_WALKERS_PER_DISK (canonical)

Scanner cycle budget controls:

When RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance.
An explicit 0 preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to 0.

Scanner recovery retry controls:

RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS caps the wait between automatic startup recovery retries. The persistent admin key is scanner.recovery_backoff_cap; the environment takes precedence over the persistent value, and the unset default remains 600 seconds (10 minutes). Values must be positive integer seconds within the platform timer range; 0, malformed values, and overflow are rejected. A valid environment override does not hide an invalid persistent value from admin config set validation.
The cap bounds the existing exponential retry schedule. Updating the persistent value applies to the next computed retry; it does not reschedule a wait already in progress. Retryable observations continue retrying after the short retry budget. Admin scanner status reports the effective value and source at runtime_config.recovery_backoff_cap_seconds.

Scanner walker controls:

RUSTFS_SCANNER_WALKERS_PER_DISK sets how many bucket walkers each local disk of an erasure set runs; the default 1 walks one bucket per disk at a time, which is the right value for rotational media. On flash, 2 to 4 lets one disk walk several buckets concurrently, which matters most for single-disk deployments with many buckets, where the walk is otherwise fully serial. Values outside 1 through 16 are rejected by admin config set; an out-of-range or non-numeric environment value is logged as a scanner runtime config error (state = "startup_apply_failed") and the scanner keeps the default. The walkers of one disk share that disk's scanner admission, so peers still see one scan session per physical disk, and RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS still caps the total.
The same value is available as the persistent scanner.walkers_per_disk key and is reported under runtime_config.walkers_per_disk in the admin scanner status; see Scanner Runtime Controls.
A timed-out cycle cancels cooperative scanner work, then fences its leader epoch before releasing the lease. An uncooperative I/O operation is dropped after the bounded shutdown window; its cursor is not claimed to be durable and the scanner reports recovery-required when the worker cannot stop cooperatively, the cycle state was not confirmed durable, or epoch fencing cannot be persisted.

Mmap read environment aliases

RUSTFS_OBJECT_MMAP_READ_ENABLE (canonical)
RUSTFS_OBJECT_ZERO_COPY_ENABLE (deprecated alias for compatibility)

Health compatibility switches

RUSTFS_HEALTH_ENDPOINT_ENABLE
- controls canonical /health, /health/live, and /health/ready endpoint exposure.
RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE
- enables minimal payload mode for GET health responses (status, ready only).
RUSTFS_HEALTH_READINESS_CACHE_TTL_MS
- TTL for readiness cache evaluation.
RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE
- withdraws readiness when bounded object read/write stages stop completing while requests remain active.
- default is true.
RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS
- maximum time without completion in a bounded object stage before readiness is withdrawn.
- default is 30000; 0 uses the default.
- the effective value is at least 5 seconds longer than RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT.
- this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
- enables busy protection behavior for health probes.
- default is false.
RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS
- max active HTTP requests; health probes return 429 when active requests reach or exceed this value.
- 0 disables thresholding even if busy protection is enabled.
RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
- enables KMS readiness enforcement for /health/ready.
- default is false.

Object lock admission environment variables

RUSTFS_PUT_COMMIT_NAMESPACE_LOCK_ACQUIRE_TIMEOUT_MS
- experimental same-object PUT commit namespace-lock admission budget.
- default is 0, which disables this override and keeps RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT behavior.
- when set, only put_object_commit write-lock acquisition is bounded by this millisecond budget; other namespace lock users keep the global object-lock timeout.
- timeout returns S3 SlowDown, so clients should use normal SDK retry handling.
- this is not a fdatasync or group-commit switch. Track fdatasync batching separately with rustfs_s3_put_object_rename_fdatasync_batch_files.

Foreground write admission environment variables

Large direct PutObject requests, multipart UploadPart requests, and server-side multipart UploadPartCopy requests share one per-process permit pool that bounds concurrent body ingest and storage writes. Small direct PUTs stay on the legacy path.

RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE
- enables the default-on pool; false keeps only the soft request counter.
- default is true.
RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT
- a positive value is an exact request-count limit shared by all gated writes.
- default is 0: derive half of RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS, clamped to 32, as large-write slots. Each slot has eight units. A gated direct PUT or unknown-size multipart part uses eight units; known-size parts use one unit per 8 MiB, rounded up, with a minimum of one and maximum of eight.
- stock settings therefore share 256 units across at most 32 large/unknown writes or 256 parts of up to 8 MiB. Mixed sizes consume the same budget. Admitted known parts of up to 64 MiB total at most 2 GiB of declared payload; larger streaming writes and queued bodies are separate. This is not a bound on total process or kernel memory.
- admission snapshots report allocated or reserved units in active and the unit budget in limit under automatic sizing. Explicit and strict limits still report request-count permits.
RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES
- smallest direct PutObject that takes a permit; unknown-size requests always do.
- default is 33554432 (32 MiB).
RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS
- how long a direct PutObject waits for a permit before returning S3 SlowDown.
- default is 250.
RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES
- smallest UploadPart that takes a permit; 0 gates every part.
- default is 0.
RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS
- how long an UploadPart or UploadPartCopy waits in the bounded queue for a permit before returning S3 SlowDown; 0 rejects immediately when the pool is full.
- default is 10000. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
- RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. UploadPartCopy waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s connect_timeout to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of SlowDown.
RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING
- maximum UploadPart and UploadPartCopy requests waiting for a permit at once; parts beyond it return SlowDown without waiting.
- default is 0, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE, RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT, RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS
- experimental strict gate that applies to every foreground write regardless of size and replaces the pool above when enabled.
- default is disabled; enabling it with limit 0 disables foreground write admission entirely.

HTTP listener socket environment variables

RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES
- fixed SO_RCVBUF for the API listener, inherited by every accepted socket; 0 leaves the receive buffer to kernel autotuning.
- default is 0. A fixed buffer disables receive autotuning. Linux doubles the requested value for socket-memory accounting, subject to kernel limits; that capacity is not a measurement of queued payload or allocated memory.
- with autotuning the receive ceiling is controlled by the kernel (net.ipv4.tcp_rmem on Linux); its defaults vary with kernel version and memory. A connection can retain a buffer enlarged by previous requests while its next request is queued. Tune the kernel ceiling first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
- the send buffer remains fixed at 4 MiB; this setting only controls the receive side.

Remote tier timeout environment variables

RUSTFS_TIER_REMOTE_CONNECT_TIMEOUT_SECS
- remote tier TCP connect timeout.
- default is 10.
- must be positive; zero fails tier client initialization, while an invalid integer is logged and falls back to the default.
RUSTFS_TIER_REMOTE_REQUEST_TIMEOUT_SECS
- remote tier request timeout through response headers.
- default is 86400 so large transition uploads keep a production-safe budget.
- must be positive; zero fails tier client initialization, while an invalid integer is logged and falls back to the default. Very large values are accepted and act as a correspondingly long budget.
RUSTFS_TIER_REMOTE_RESPONSE_BODY_IDLE_TIMEOUT_SECS
- maximum idle time between remote tier response-body chunks.
- default is 60; the timer resets only when non-empty body data keeps progressing.
- must be positive; zero fails tier client initialization, while an invalid integer is logged and falls back to the default.

Drive timeout environment variables

RUSTFS_DRIVE_METADATA_TIMEOUT_SECS
RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS
RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS
RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS

Legacy compatibility fallback:

RUSTFS_DRIVE_MAX_TIMEOUT_DURATION This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.

Drive timeout health-action policy:

RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION
- mark_failure (default): timeout marks failure and may transition drive runtime state.
- ignore_scanner: timeout does not mark failure for scanner-sensitive operations (walk_dir, read_metadata, list_dir, disk_info).

Drive timeout profile preset:

RUSTFS_DRIVE_TIMEOUT_PROFILE
- default (default): keep current timeout defaults.
- high_latency: use 60s default timeout for scanner-sensitive operations when no operation-specific override is set (read_metadata, disk_info, list_dir, walk_dir, walk_dir_stall, and object-capacity scan base/maximum budgets).
Precedence:
- Explicit per-operation timeout env (RUSTFS_DRIVE_*_TIMEOUT_SECS) takes highest precedence.
- Explicit object-capacity timeout env (RUSTFS_CAPACITY_STAT_TIMEOUT, RUSTFS_CAPACITY_MAX_TIMEOUT) takes precedence for capacity scans.
- Then RUSTFS_DRIVE_MAX_TIMEOUT_DURATION legacy fallback.
- Then the profile-derived default (default or high_latency).

Admin peer probe timeout

RUSTFS_ADMIN_PEER_PROBE_TIMEOUT_SECS
- total per-peer budget for the server_info/storage_info admin probe round; server_info may reconnect once and storage_info remains a single attempt.
- default is 10 seconds, preserving the previous two-attempt worst-case budget.
- values must be positive; 0 or an invalid value falls back to the default, and values above 60 are clamped to 60.
- the setting is read by the aggregating node only; it does not change the internode RPC wire contract. Any retry shares one round deadline rather than receiving a fresh timeout.

Startup filesystem boundary policy

RUSTFS_UNSUPPORTED_FS_POLICY controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
- warn (default): log warning and continue startup.
- fail: abort startup with an error.

RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example nfs, cifs, smb2, and fuse.*) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as \\server\share and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually NTFS) for them.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.