Switch IAM QR rendering from qrcode to qrcode-rs 2.0.0 while keeping only the std and svg feature path enabled.
Set the release profile to a single codegen unit and disable release debuginfo as requested.
Verification:
- cargo info qrcode-rs --registry crates-io
- cargo tree -p rustfs-iam -e features
- CARGO_TARGET_DIR=/private/tmp/rustfs-target-qrcode-rs-profile-tuning cargo test -p rustfs-iam --locked
- cargo fmt --all --check
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(signer): cache signing key to avoid redundant HMAC-SHA256
Cache the AWS4 signing key per (secret, region, date, service_type)
tuple. The signing key is derived from 4 HMAC-SHA256 calls and is
constant for a given user within the same UTC day, so caching it
eliminates ~0.5-1ms of redundant crypto per request.
The cache uses a LazyLock<Mutex<HashMap>> with automatic daily
rotation (cache entries naturally expire when the date component
of the key changes).
Refs: https://github.com/rustfs/backlog/issues/2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(signer): bound signing key cache
* fix(signer): satisfy cache lint
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Upgrade rustfs-mimalloc and rustfs-mimalloc-sys to 0.5.1, then call the new safe wrapper from Tokio worker thread startup so mimalloc can treat runtime threads as threadpool workers.
Keep the hint no-op on Windows, matching RustFS allocator platform boundaries.
Co-authored-by: heihutu <heihutu@gmail.com>
The INVENTORY_UID constant is only referenced inside
#[cfg(target_os = "linux")] test functions, so it appears unused on
macOS. Add a cfg_attr to allow dead_code on non-linux targets.
* ci(nightly): persist the nightly deb on Cloudflare R2
Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.
* test: add pool expansion / decommission E2E script and workflow
Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.
* ci(pool-test): run the pool expansion test on the smoke-testing runner
Backlog#1845 step 8 conclusion. The plan called for folding iam::Error into a policy::Error #[from] wrapper and deleting the hand-written mapping. Measurement rejected the fold: the duplicated variants have ~220 construction/match sites (about 140 in production) across iam and the admin handlers - all auth-critical - and the alias route is blocked by the orphan rule (iam's From<IamStorageError> and io conversions cannot be implemented for a foreign type). Meanwhile the drift risk the fold targeted is already compiler-covered: the From match is exhaustive with no catch-all, so any new policy variant fails the build until mapped.
What remains of the step, delivered: the six dead policy variants are gone (previous commit), the grouped lossy arm is down to the two variants actually produced, a doc comment on the From impl records the verdict with the evidence, and a new totality test constructs one representative of every policy::error::Error variant and asserts the conversion preserves the rendered message - so the mapping is now pinned loss-free in both directions the classifiers care about.
Ref rustfs/backlog#1845
The nightly GNU build now also packages the release binary as
rustfs-nightly-<YYYY-MM-DD>.deb (Asia/Shanghai date, matching the schedule
timezone) and uploads it as a workflow artifact. Packaging mirrors
package.yml: DEBIAN control/conffiles and the systemd service from
deploy/build/, built with fakeroot dpkg-deb.
refactor(heal): classify recoverability typed-first with documented needle fallback
Backlog#1845 step 6. Heal's retry decision leaned on substring matching of rendered messages; the typed information available in the error values now takes priority:
- Lock failures classify by LockError's own taxonomy instead of the blanket Lock(_) => recoverable: fatal variants (ResourceNotFound / PermissionDenied / Configuration) are terminal since retrying cannot fix them, while contention and transport variants (Timeout, Network, Internal, AlreadyLocked, QuorumNotReached, InsufficientNodes, ...) stay recoverable exactly as before.
- DiskError::RemoteClientUnavailable and its StorageError twin (typed in #6619) join the typed recoverable lists, so client-acquisition failures no longer depend on which needle happens to appear in the detail.
- task.rs is_transient_lock_or_timeout_error consults LockError::is_retryable / QuorumNotReached and the typed Timeout variants before falling back to needles.
- The substring list is demoted to a documented fallback: every needle now carries a producer census comment naming what emits it, with the shrink-only rule stated (delete the needle when its producer becomes typed end-to-end). heal rename incomplete remains the one needle with no typed producer.
heal gains a direct rustfs-lock dependency (already transitive via ecstore) to name LockError variants.
New tests pin each typed source: contention/transport lock variants recoverable, fatal lock variants terminal, RemoteClientUnavailable recoverable with a detail that avoids every needle. All existing recoverability tests stay green.
Ref rustfs/backlog#1845
Backlog#1845 step 8 prerequisite. policy::error::Error carried six variants with zero construction and zero match sites anywhere in the workspace: ErrCredMalformed, CredNotInitialized, NoAccessKey, InvalidToken, InvalidAccessKey, InvalidExpiration. Their only reference was the grouped fallthrough arm in iam's From<policy::error::Error>, whose own dead same-name twins were already removed in backlog#1831 (#6030).
Delete the variants and their display-message test rows; the iam mapping's grouped arm shrinks from eight variants to the two that are actually produced (InvalidServiceType from service_type parsing, JWTError via #[from]). This clears the way for folding the remaining 25-arm hand-written mapping (backlog#1845 step 8).
Ref rustfs/backlog#1845
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4.
This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore.
Three reverse dependencies from the client into engine internals are severed so the move can be pure:
- transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted.
- api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects).
- the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new.
Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore).
Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
Backlog#1845 step 7. The replication crate's hand-rolled, crate-generic Error type actually describes one thing: failures of the persisted resync/MRF state files. Rename it to ResyncStateError so the name says so, and stop collapsing io::Error into Other(String): a new Io(std::io::Error) variant keeps the kind and source chain, Display renders identically, and the ecstore boundary maps it to StorageError::Io so the kind survives into store-layer classification instead of degrading into a stringified other().
No thiserror introduced - the crate keeps its zero-internal-deps posture and hand-written impls.
Ref rustfs/backlog#1845
refactor(ecstore): make store-to-disk error narrowing a named fallible operation
Backlog#1845 step 4. The blanket impl From<StorageError> for DiskError let ? silently push store-only errors (locks, buckets, quotas) across the disk boundary into DiskError::other, where the rendered message fragments reduce_errs quorum buckets. Same story for the blanket From<StorageError> for rustfs_filemeta::Error and its other() catch-all.
Both impls are replaced by named, fallible methods: StorageError::narrow_to_disk() and StorageError::narrow_to_filemeta(). Variants with an identity on the far side map across unchanged - including the two documented lossy collapses (SlowDown -> TooManyOpenFiles, StorageFull -> DiskFull) that the round-trip tests pin - and everything else returns Err(self) so the call site decides what crossing the boundary means. Removing the impls let the compiler enumerate every conversion site; the census that scoped this issue had found 5, the compiler found 33.
Call sites keep their existing behavior: the io identity bridge and the generic sites fold Err into the io-backed other() exactly as the old catch-all did (identity still recoverable by downcast), listing paths use one shared to_filemeta_err helper, and the two sites that relied on the SlowDown collapse now construct DiskError::TooManyOpenFiles directly so the loss is visible where it happens. No behavior change intended anywhere; the io::Error bridge itself is untouched by design.
Ref rustfs/backlog#1845
The doc comment named RUSTFS_LOCK_ACQUIRE_TIMEOUT with a 30-second
default, but the function reads RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT
with a 5-second default. RUSTFS_LOCK_ACQUIRE_TIMEOUT is a real,
separate knob read by the lock and scanner crates, so tuning the
documented name silently has no effect on this path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(ecstore): type internode client-acquisition failures for stable quorum buckets
Backlog#1845 step 3, first typed family. The largest other(format!) message family in ecstore was 'can not get client, err: {detail}' (~50 production sites): every internode RPC that fails to acquire a client wrapped the dial/auth error with per-peer detail into DiskError::other / StorageError::other, whose Io equality compares the rendered message. N disks failing for this same cause therefore counted as N distinct errors in reduce_errs, starving quorum aggregation, and remote_disk call sites double-wrapped the message on top of get_client's own wrap.
Introduce DiskError::RemoteClientUnavailable(String) (wire code 0x2B) and its StorageError twin (StorageErrorCode 0x54): equality and hashing use the wire code alone, so same-cause failures land in one quorum bucket regardless of per-peer detail, while Display keeps the detail so substring classifiers (network needles, heal recoverability) keep reading it unchanged. Wire encoding carries the rendered detail in error_info and decode restores the typed variant; old peers fall back to the legacy string form gracefully.
Call sites: remote_disk get_client/get_bulk_client/offline-bypass/recovery-probe now construct the typed variant and the ~60 redundant double-wrap map_errs are gone; peer_rest_client's three client getters and offline gates, peer_s3_client, and admin_server_info follow. The tier-config-reload connection classifier's anchored 'can not get client' substring check becomes a typed match on the variant (the string form is retired and now classifies as Terminal, pinned by test).
Ref rustfs/backlog#1845
* chore(ci): refresh error other ratchet baseline
* fix(ecstore): classify typed client network failures
refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
refactor(common): move scanner/heal domain contracts into dedicated crates
crates/common carried ~5.6K lines of scanner/heal domain code (metrics.rs,
heal_channel.rs, last_minute.rs) parked there to break dependency cycles;
every scanner type change recompiled all 11 rustfs-common dependents.
Pure move, zero renames, zero shape changes (backlog#1843):
- New crate rustfs-heal-contracts receives heal_channel.
- New crate rustfs-scanner-contracts receives metrics, last_minute, and the
GLOBAL_INIT_TIME trio (metrics::report() reads it as the current-cycle
fallback, so it must live below the shim to avoid a dependency cycle).
- rustfs-common re-exports everything at the old paths as a transitional
shim; consumers migrate crate by crate, then the shims are deleted.
Backlog#1845 step 1 (pure tests, no behavior change): pin the current behavior of every conversion seam an error crosses before heal, replication, or quorum aggregation classifies it, so the later typed-variant and narrow_to_disk() refactors change these expectations deliberately rather than silently.
Covered seams: DiskError <-> StorageError, DiskError <-> node_service wire Error, DiskError/StorageError <-> io::Error (the by-design identity bridge), and StorageError <-> rustfs_filemeta::Error.
Documented lossy edges pinned as-is: SlowDown collapses to TooManyOpenFiles across the disk boundary (StorageFull to DiskFull likewise), the wire Io catch-all re-wraps the rendered message on every hop and drops the io::ErrorKind, and other(format!) messages with per-disk detail fragment reduce_errs quorum buckets while identical messages still bucket together.
Ref rustfs/backlog#1845
Backlog#1845 step 2. reduce_errs buckets per-disk errors by equality, and Io equality compares the rendered message, so an other(format!(..)) error embedding per-disk detail makes N same-cause failures count as N distinct errors during quorum aggregation. The census that opened the issue counted 1,609 such sites; the production count in crates/ecstore/src is 657 today and was still growing.
Freeze it: scripts/check_error_other_format_ratchet.sh counts ::other(format! sites per file (trailing #[cfg(test)] modules excluded) against a shrink-only per-file baseline, failing on any growth and on stale entries after a shrink, following the layer-dependency-baseline model. Wired into make pre-commit / pre-pr / dev-check and the CI Quick Checks job.
Ref rustfs/backlog#1845
test(obs): replace source-text logging tests with logging guardrail script coverage
The seven fs::read_to_string source-text tests in crates/obs/src/logging.rs asserted retired logging patterns and required structured-logging fields across 13 files in other crates, four of them reverse reads into the rustfs binary crate. Their patterns are now enforced by scripts/check_logging_guardrails.sh, which runs in pre-commit and CI, covers the same files through checked_files plus require_patterns, and does not silently lapse when a governed file moves.
Part of rustfs/backlog#1884.
Single-part encrypted objects in the legacy format could not serve range
reads without decrypting from byte 0: v1 frames are emitted per upstream
read, so no closed-form plaintext-to-ciphertext mapping exists. The v2
layout fixed the frame length (8218 ciphertext bytes per 8 KiB plaintext
frame), making the mapping closed-form.
Consume it:
- Single-part PUTs that encrypt locally under the v2 write switch stamp
the frame-layout marker, valued with the object's data_dir token -
ciphertext passthrough, data movement and copies mint a new data_dir
or strip the marker, so a re-homed marker disqualifies itself.
- The encrypted read plan seeks marked, uncompressed single-part objects
to frame_index * 8218 and decrypts from that frame: the frame index
rides the plan's sequence-number slot into DecryptReader::new_at_block,
whose nonce and AAD bind absolute indices. New metric path label
frame_seek.
- A lying marker fails closed: v2 authentication rejects bytes at a fake
frame boundary; plaintext is never served from the wrong offset.
Compressed objects and multipart sub-part seeks keep the conservative
paths (follow-up work); reading needs no switch - seekability follows
the marker.
The legacy rio v1 stream format authenticates only each frame's
ciphertext: the 8-byte header (length + plaintext CRC32) and the end
marker sit outside the AEAD, frames carry no position binding, and
nothing marks the last frame - header rewrites, frame reordering and
truncation of trailing frames are not cryptographically detected.
Add a v2 layout in the same format family, dispatched per frame by the
type byte:
- the header plus the frame's index are AEAD associated data (0x01), so
header tampering, reordering and cross-position splicing fail
authentication;
- the final frame carries its own authenticated type byte (0x02); a
clean EOF or an end marker before it is an error, every stream
(including the empty one) ends in an authenticated final frame, and a
v2 multipart stream fails if it ends before all listed part segments;
- the writer accumulates full 8 KiB blocks, so non-final frames are
fixed-length (8218 ciphertext bytes) and single-part objects gain a
closed-form offset mapping for the follow-up range seek.
Key hierarchy, nonce derivation, envelopes and metadata are unchanged;
v1 objects stay readable forever, while v2 frames reject the historical
nonce fallbacks and unknown frame types become a hard error.
Write side ships off by default (RUSTFS_ENCRYPTION_FRAME_V2): mixed
version clusters cannot read v2 frames, and encrypted ciphertext travels
verbatim through transition, decommission and SSE-C replication
passthrough. This release ships read support; the default flips in a
following release.
The dynamic-configuration flow persisted KmsConfig to cluster storage as
raw JSON, leaving inline authentication material - the Vault token, an
AppRole secret_id, the Local master key - in config/kms_config.json in
cleartext.
Add rustfs_kms::config_secret: with the per-node RUSTFS_KMS_CONFIG_SECRET
set, those field values are sealed in place before persistence (Argon2id
with the Local key store's parameters + AES-256-GCM, per-value random
salt, the field's logical label bound as AEAD associated data so sealed
values cannot be swapped between fields). Sealed values carry the
versioned prefix RUSTFS-KMS-ENC[v1]:.
Compatibility is warn-only by owner decision: an unset secret keeps the
plaintext format and warns naming the exposed fields; plaintext values
load forever and reseal on the next save. Sealed values fail closed on a
missing or wrong secret. The sealing secret must be an independent trust
root - reusing the Local master key or Static secret is refused,
mirroring the backup-KEK rule.
Range GETs on encrypted objects read the whole ciphertext from offset 0
and discarded the decrypted prefix, because the part-boundary seek
shipped behind RUSTFS_ENCRYPTED_RANGE_SEEK defaulted to false
(backlog#1316 Phase A).
Flip the default to true. Safety rests on the marker chain: MPUs created
without a candidate layout marker never become seek-eligible,
CompleteMultipartUpload promotes the candidate to the quorum marker only
after revalidating it against the object's data_dir under the uploadId
write lock, and reads seek only when the quorum marker matches the
current data_dir. Single-part, compressed and markerless objects keep
the full-read path; RUSTFS_ENCRYPTED_RANGE_SEEK=false remains the kill
switch.
The stale default-off regression test becomes
test_legacy_range_seek_defaults_enabled: the unset-env default must
match the explicit opt-in plan, seek past the leading parts, and not
span the whole ciphertext.
perf(sse): classify GET response headers without a second KMS unwrap
An SSE-KMS GET performed two backend Decrypt calls per request: the
object layer's encryption resolver unwraps the envelope to build the
decrypted stream, and the S3 layer then called sse_decryption again
purely to derive response headers, discarding the returned key bytes.
Replace the S3-layer call with classify_sse_read_response, which
reproduces that call's behavior from stored metadata alone: SSE-C
validation errors and precedence, per-key kms:Decrypt authorization
ahead of every other failure mode, and the request's KMS audit summary
fields. The success outcome stays honest because a failed unwrap aborts
the read in the object layer before response classification is reached.
Tests cover header parity against the unwrap-based path, audit-tag
parity for allowed and denied principals, SSE-C validation parity, and
prove classification needs no DEK provider at all.
houseme