refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
The dynamic-configuration flow persisted KmsConfig to cluster storage as
raw JSON, leaving inline authentication material - the Vault token, an
AppRole secret_id, the Local master key - in config/kms_config.json in
cleartext.
Add rustfs_kms::config_secret: with the per-node RUSTFS_KMS_CONFIG_SECRET
set, those field values are sealed in place before persistence (Argon2id
with the Local key store's parameters + AES-256-GCM, per-value random
salt, the field's logical label bound as AEAD associated data so sealed
values cannot be swapped between fields). Sealed values carry the
versioned prefix RUSTFS-KMS-ENC[v1]:.
Compatibility is warn-only by owner decision: an unset secret keeps the
plaintext format and warns naming the exposed fields; plaintext values
load forever and reseal on the next save. Sealed values fail closed on a
missing or wrong secret. The sealing secret must be an independent trust
root - reusing the Local master key or Static secret is refused,
mirroring the backup-KEK rule.
perf(sse): classify GET response headers without a second KMS unwrap
An SSE-KMS GET performed two backend Decrypt calls per request: the
object layer's encryption resolver unwraps the envelope to build the
decrypted stream, and the S3 layer then called sse_decryption again
purely to derive response headers, discarding the returned key bytes.
Replace the S3-layer call with classify_sse_read_response, which
reproduces that call's behavior from stored metadata alone: SSE-C
validation errors and precedence, per-key kms:Decrypt authorization
ahead of every other failure mode, and the request's KMS audit summary
fields. The success outcome stays honest because a failed unwrap aborts
the read in the object layer before response classification is reached.
Tests cover header parity against the unwrap-based path, audit-tag
parity for allowed and denied principals, SSE-C validation parity, and
prove classification needs no DEK provider at all.
* feat(madmin): add account and two-factor wire contract
Defines the self-service account and MFA API shapes in one place so the
console and the `rc` CLI decode identical payloads instead of each
carrying its own copy of the contract.
`AccountMutability` is part of the contract on purpose: a client needs to
know whether the server will accept a password change for this identity
before offering the control, rather than discovering it from a rejected
request.
* feat(s3-types): add IAM identity audit events
Adds `iam:Identity:CredentialChanged` and `iam:Identity:AuthChallenge`
so account and authentication activity reaches the audit pipeline in its
own namespace, the way the KMS events already do. Neither is reachable
from a bucket notification config.
Two variants for the whole surface rather than one per operation:
`mask()` gives every variant its own bit in a `u64`, and the budget is
nearly spent (63 of 64 used after this). The per-operation detail lives
in `AuditEntry::api.name` and the `iamOperation` tag, which is what a
SIEM filters on anyway. Splitting these further needs `mask()` widened
first.
* feat(iam): add two-factor authentication primitives
Implements the state machine behind TOTP enrollment and verification in
the IAM domain, so the admin handlers stay HTTP plumbing and the console
and CLI drive identical logic.
* `totp`: RFC 6238 over the workspace's existing hmac/sha1, pinned to the
published Appendix B vectors. SHA-1, 6 digits, 30s: the parameters every
mainstream authenticator app implements. Verification returns the
matched time step so the caller can burn it.
* `recovery`: ten single-use codes, 100 bits each, in a Crockford base32
alphabet without I/L/O/U. Stored as domain-separated SHA-256 digests —
a password KDF would have to run once per stored code on every attempt,
turning each guess into an attacker-controlled cost, and with uniform
100-bit input there is no dictionary for it to defend against.
* `challenge`: stateless HMAC tokens. A TTL cache would be node-local, so
a cluster without session affinity would issue on one node and verify
on another; nothing here needs replicating.
* `record`: two-phase enrollment, replay high-water mark, and lockout.
Pending enrollment never gates a login, so a mis-scanned QR cannot lock
an operator out, and re-configuring keeps the old factor working until
the new one is confirmed.
* `store`: one object per identity under `config/mfa/`, a sibling of
`config/iam/` so the IAM cache loader's startup walk does not sweep it
up. Optimistic `If-Match` writes; deliberately uncached, because a cache
would need cluster-wide invalidation to keep the replay mark and the
lockout counter honest.
* `qr`: server-side rendering, so neither client needs a QR encoder.
Enrollment is refused without `RUSTFS_IAM_MASTER_KEY`. A TOTP secret is
credential-equivalent, and one written in plaintext could be lifted off a
disk — worse than no second factor, because the user believes they have
one. IAM identities tolerate a missing master key for backward
compatibility; a new feature has no such history to honour.
Also adds `IamSys::revoke_sts_sessions_for_parent`, so a credential
rotation can invalidate the sessions minted under the old secret.
* feat(admin): add self-service account endpoints and the two-factor login gate
Adds the account surface (`/v3/account/*`), the second-factor endpoints,
the administrative reset (`/v3/user/mfa`), and `PUT
/v3/set-user-secret-key`, plus the gate on `AssumeRole`.
What the gate covers, and what it deliberately does not:
* `AssumeRole` is the only interactive login RustFS has, so it is where a
second factor can be enforced. With one enrolled it requires
`TokenCode`; without an enrollment the code path is unchanged, so
existing deployments are untouched.
* A request signed directly with a long-term access key stays ungated.
Gating it would break every script and CLI the moment a human enabled
2FA on their own account, and would add no protection: whoever holds
the secret key already has full access without presenting a code. This
is the division AWS draws; making 2FA meaningful for API access needs an
`aws:MultiFactorAuthPresent` policy condition, tracked separately.
`SerialNumber`/`TokenCode` are STS's own parameters, so an SDK or script
authenticates the same way the console does.
`caller_identity` resolves who a request acts as. The console signs with
a short-lived STS session, so "the caller" is almost never the key that
signed. It reports two separate capabilities: root cannot rotate its
secret (a process-wide `OnceLock` that also derives the internode RPC
secret) but *can* enroll a second factor — conflating the two would leave
the default deployment's console login unprotectable.
The self-service routes carry no admin action. Giving them one would be
wrong in both directions: it would stop an ordinary user from changing
their own password, and let any holder of that action change someone
else's. They gate on possession of the credential plus, for the
mutations, knowledge of the current secret — a signature only proves a
credential was used, so without that a hijacked tab could rewrite the
account's credentials or strip its second factor.
`set-user-secret-key` exists because the only prior way to change a
password was to re-POST the whole user through `add-user`, which rewrote
`status` and dropped the policy field — a password reset that silently
re-enabled a disabled account.
Wrong, replayed and malformed codes are indistinguishable on the wire;
the distinction survives only in the audit trail, where no submitted
value, secret or code is ever recorded.
* test(e2e): cover the two-factor lifecycle and its regressions
Unit tests cover the state machine at its edges; only an end-to-end test
proves the pieces are wired together and that the existing
authentication paths still behave.
Asserts, against a real server: enrollment is refused without a master
key; the full enroll/activate flow works with a genuine RFC 6238 code;
`AssumeRole` refuses without a factor and accepts a valid one; a recovery
code works exactly once; a direct SigV4 admin request keeps working with
a factor enrolled; `AssumeRole` for an unenrolled identity is unchanged;
and a password rotation invalidates the old secret.
The test computes TOTP codes itself rather than calling the server's
implementation — a shared helper could agree with a bug on both sides.
This suite caught a real defect during development: enrollment was
refused for root because its *password* is immutable, which would have
left the default deployment — an administrator signing into the console
as root — unable to protect the one login the feature exists for.
* docs(operations): document the two-factor authentication model
Records what the second factor protects and what it deliberately does
not, because several of the boundaries look like gaps until the
alternative is spelled out: why direct SigV4 access stays ungated, why
root credentials cannot be rotated at runtime, why secret keys cannot be
hashed in an S3 server, and why at-rest protection is mandatory for a
TOTP secret but optional for an IAM identity.
Also states the limitations plainly, including that GHSA-m77q-r63m-pj89
is unaffected: a holder of the root secret can still forge a session
token, 2FA claim included.
Placed alongside the other authentication and KMS security documents
rather than under a new `docs/security/`, which `.gitignore` excludes.
* fix(admin): route the new account handlers through the admin s3 facade
Two of the guardrails in the CI "Quick Checks" job rejected the previous
commits, so the required check would have gone red as soon as a maintainer
approved the workflow run.
`check_architecture_migration_rules.sh` requires everything under
`rustfs/src/admin` to reach `ECStore` through a domain module rather than
the root of `storage_api`. The MFA handler and the two `AssumeRole`
signatures now use `storage_api::runtime::ECStore`, which is where the
other ten admin handlers already take it from.
`check_s3s_footprint.sh` ratchets two counters that new code may not grow:
files referencing `s3s` and error-macro invocation lines. This branch added
four files and thirty-two lines to them. The ratchet is lower-only and its
header forbids raising a baseline to get green, so the construction moves
behind the facade instead: `storage_api::s3` now re-exports the request and
body types these handlers need and gains an `error` constructor over
`S3Error::with_message`. That is the same constructor the macro expands to
and the one `handlers/mod.rs`, `rebalance_internal_error` and
`invalid_object_lock_configuration` already call, so this is the existing
practice rather than a new one, and it keeps the `s3s` dependency in the
boundary file the s3gate migration replaces.
Every error code and message is carried over unchanged. In `sts.rs` only
the call site this branch added is converted; the sixteen that predate it
are left alone, because rewriting them would put unrelated churn in a
feature PR and push the counter below the baseline it is meant to hold.
Run replication resync target reconcile and follow-up resync recovery in a background startup task so bucket metadata transaction lock contention cannot keep a node from joining the cluster.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): handle metadata-less bucket residue
Diagnose metadata-less on-disk residue before non-force DeleteBucket reaches physical deletion, and keep scanner-discovered metadata-missing objects on a non-destructive heal path.
Add explicit heal --remove cleanup for unversioned metadata-less data directories, using the existing data-dir delete primitive and fail-closed shape checks so pre-commit or unknown residue is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): adapt offline array validator
Wrap the filesystem summary validator in a closure so Option::is_some_and can pass the concrete array reference accepted by serde_json::Value::as_array.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): remove redundant offline test clones
Move the temporary path into the swap closure after deriving the output path, keeping clippy's redundant-clone lint clean for offline bundle tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Keep liveness probes local by avoiding readiness collection and omitting readiness-only fields from liveness payloads. Readiness and MinIO cluster probes continue to report dependency and quorum state.
Co-authored-by: heihutu <heihutu@gmail.com>
Treat lock quorum as part of node readiness for both /health and /health/ready response bodies while preserving the /health liveness HTTP 200 contract.
Add focused regression coverage for lock-quorum-only degradation and make the public /health layer fixture independent from process-global readiness state.
Refs: rustfs/backlog#2011
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): update s3s revision
Pin the workspace s3s dependency to rustfs/s3s commit 39080d610e0560c55f068f6dd76b976e267b2f67 and refresh compatible dependencies with cargo update and cargo upgrade.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(s3): preserve SigV4 body validation errors
Map s3s upload stream body validation failures into existing RustFS client-error types before the PUT body readers consume them. This keeps tampered single-chunk payload hashes from surfacing as InternalError after the s3s revision update.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): use s3s 0.15.0 release
Switch the workspace dependency from the temporary s3s git revision to the published 0.15.0 crate and refresh the lockfile updates that come with the release.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(mimalloc): add arena diagnostics and configuration
Based on mimalloc maintainer feedback (microsoft/mimalloc#1372),
add diagnostics to check mimalloc arena configuration at runtime.
Changes:
- Add rustfs-mimalloc-sys to workspace dependencies
- Add log_mimalloc_diagnostics() function to check:
- arena_max_object_size
- pagemap_commit status
- mimalloc version
- Add memory_observability module with mimalloc diagnostics
This helps diagnose why allocations might be going outside arenas,
which is the suspected root cause of futex contention.
Ref: rustfs/backlog#2005
Ref: microsoft/mimalloc#1372
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): add Vec<u8> buffer pool for EC operations
Add a general-purpose buffer pool to reduce Vec<u8> allocations
in hot paths like EC encoding/decoding.
Changes:
- Add BufferPool struct in crates/ecstore/src/erasure/codec/buffer_pool.rs
- Thread-safe pool with capacity-based bucketing (power-of-two)
- Global EC_BUFFER_POOL instance with 16 buffers per bucket
- Add buffer_pool module to codec/mod.rs
Expected impact:
- Reduce heap allocations in EC encode/decode paths
- Avoid memzero overhead (proven 4.8% CPU saving in ShardBufferPool)
- Reduce mimalloc lock contention
Note: Main bottleneck remains mimalloc internal synchronization
(futex 98.64% time). Buffer pool provides modest improvement (+2-5%).
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and related files
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): add #[allow(dead_code)] to buffer pool
The BufferPool infrastructure is ready but not yet integrated
into the EC hot paths. Add #[allow(dead_code)] with clear
documentation about integration status.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): integrate BufferPool into bitrot verify path
Replace vec![0; shard_size] with get_ec_buffer() in the bitrot
verification hot path to reduce heap allocations and avoid memzero.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and bitrot changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(ecstore): clean up buffer pool code
- Remove unnecessary #[allow(dead_code)] attributes
- Update module documentation to reflect current integration status
- Simplify code structure
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(runtime): cap default worker threads at 16
Testing showed 16 worker threads outperforms 32+ for 1KiB PUT
workloads due to reduced mimalloc lock contention.
A/B test results (testing 4-node cluster, c=64):
- worker_threads=32: 740 obj/s (baseline)
- worker_threads=16: 785 obj/s (+6.1%)
The default was detect_cores() which returned 32 on our testing
nodes. Cap at 16 for optimal small-object performance.
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and runtime changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): remove unused BufferPool::new() function
The new() function was never used since EC_BUFFER_POOL
initializes directly with with_limits(16).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): update buffer_pool tests to use with_limits
Replace BufferPool::new() with BufferPool::with_limits(16) in tests
since new() was removed in favor of with_limits().
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: hector <hetor@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* perf: optimize cgroup resource detection with single System instance
Consolidate two sysinfo::System instantiations into one for CPU and
memory detection. Pre-compute the metrics basis string ("cgroup"/"host")
in ContainerResources to avoid per-snapshot String allocations in the
memory observability hot path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt formatting
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The /health endpoint (liveness) was returning a hardcoded `ready: true`
in its response body regardless of actual node readiness state. This
caused a semantic contradiction with /health/ready (readiness), which
correctly reported readiness based on storage, IAM, lock quorum, and
peer health.
This led to confusing behavior in Kubernetes deployments where:
- /health returned 200 with `ready: true` (liveness)
- /health/ready returned 503 (readiness)
- Pods remained Running but were removed from Service endpoints
Changes:
- readiness_source_for_probe(Liveness) now returns Node readiness source
- health_check_state() for Liveness reflects actual readiness in body
while keeping HTTP 200 status (process is alive)
- build_health_response_parts() for Liveness now includes dependency
details and degradedReasons when readiness report is available
This ensures the `ready` field in /health body is truthful while
maintaining backward compatibility for liveness probe behavior.
Refs: rustfs/backlog#2011
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(replication): tolerate orphaned resync intents at startup
Since #5215 (1.0.0-beta.12) startup reconciles every pending/started
resync intent in resync.bin against the bucket's configured targets and
aborts the whole server when an intent has no matching target ARN. A
resync whose remote target was later removed leaves exactly such an
orphan on disk, so every later start fails with "accepted replication
resync target ... is not configured" regardless of the binary version.
Skip orphaned intents with a warning instead of failing startup; the
resync routine already settles them to ResyncFailed. Cancel the intent
when its remote target is removed so the orphan is not created again.
Fixes#4784
* fix(replication): cancel removed-target resync under the admission lock
Canceling through this node's cached whole-bucket status map could
persist a map that predates another node's admission, erasing that
node's durable restart intent. Reload resync.bin under the bucket
admission lock, publish the fresh map, and only then mark the removed
target's intent canceled. Two-node regression covers the clobber.
* fix(replication): persist resync status via ETag CAS merge
mark_status, the periodic saver, admission, and removed-target
cancellation all persisted their node's cached whole-bucket map, so any
one node's stale cache could resurrect states another node had already
finalized (a canceled intent flipping back to Pending, an admission
vanishing). All resync.bin writers now go through update_resync_status_cas:
load the freshest document with its ETag, apply a per-target mutation
with staleness and canceled-is-terminal guards re-checked against the
persisted entry, and save conditionally, retrying on concurrent writes.
The periodic saver merges per target, letting terminal states and newer
admissions recorded elsewhere win. Cache convergence stays per-target so
locally running resyncs keep their authoritative progress counters.
Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent
(node B's pre-cancel cache marking its own run Started must not revive
node A's canceled intent) plus unit coverage for the periodic-save merge.
* test(ecstore): rename resync test helper off the guarded contract name
fn resync_target is on the architecture guard's reserved list for
crates/replication operation contracts; the merge-test helper now reads
resync_target_state.
* fix(replication): serialize resync status updates
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
唐小鸭