Commit Graph

1897 Commits

Author SHA1 Message Date
houseme 4b6b6f14bd feat(hotpath): use mimalloc in inner counting allocator (#5523)
* feat(hotpath): use mimalloc in counting allocator

* fix(hotpath): adapt mimalloc for allocation counting

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-01 10:43:52 +08:00
cxymds c09d11ff3b fix(config): fence persisted config updates and reloads (#5512)
* fix(config): fence persisted config updates and reloads

* fix(ci): unblock config and e2e checks

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-01 00:33:46 +00:00
houseme 04c5921850 fix(s3): allow CopyObject COPY request metadata (#5514)
* fix(s3): allow CopyObject COPY request metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(kms): remove stale KmsClient import

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 17:52:00 +00:00
houseme 40eee6177a test: add formal hotpath ABBA runner (#5507)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 11:33:10 +00:00
houseme 78d6918c52 feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.

Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 05:42:19 +00:00
houseme e06c9c02c6 feat: add hotpath primitive profiling coverage (#5492)
* chore(deps): refresh google cloud dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat: add hotpath primitive coverage

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat: extend hotpath profiling features

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: gate OPA hotpath client wrapping

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: avoid request-scoped hotpath primitive wrappers

Preserve the original bounded channel and stream semantics in EC and RIO request paths while keeping hotpath CPU profiling as a separate opt-in feature that implies base hotpath instrumentation.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 01:59:13 +00:00
Zhengchao An a2fe5d7d88 feat(admin): add KMS key lifecycle endpoints and deletion reference gate (#5496)
* feat(kms): add key lifecycle operations to the backend contract

Add enable_key/disable_key/rotate_key to KmsBackend with conservative
defaults returning the typed UnsupportedCapability error, mirroring
remove_expired_key. KmsManager gains matching pass-through methods and
drops cached key metadata after every successful state mutation so the
next describe observes backend truth. The local backend overrides
enable/disable, delegating to its state-machine-gated client methods;
rotation stays rejected, matching its advertised capabilities. New
dedicated policy actions kms:EnableKey and kms:DisableKey complete the
KMS action taxonomy alongside the existing kms:RotateKey.

* feat(admin): add KMS key enable/disable/rotate endpoints

POST /v3/kms/keys/enable, /v3/kms/keys/disable and /v3/kms/keys/rotate,
following the existing /v3/kms/keys handler conventions: key_id body
with keyId query fallback, {success, message, key_id, key_metadata}
responses, and 503 JSON while the KMS service is absent. Error mapping
keeps InvalidOperation/ValidationError at 400 like the sibling handlers
and surfaces UnsupportedCapability as 501 so a backend capability gap is
never mistaken for a missing key. Existing /v3/kms/keys handlers are
untouched apart from a visibility change on a private query helper.

* feat(kms): gate scheduled key deletion on bucket encryption references

Implement the DeletionReferenceChecker seam left by the deletion worker:
before any material is destroyed, every bucket's SSE configuration is
checked for a default KMS key reference and a hit blocks the removal.
The gate fails closed - an unpublished object store, a failed bucket
listing or an unreadable per-bucket encryption config all report a
blocking reference - because destroying key material is irreversible
while a blocked removal is simply retried on the next sweep. Registered
during init_kms_system before the service can start, so every worker
spawn observes it. Storage access goes through a new kms section of the
root storage facade.
2026-07-31 01:37:07 +00:00
Zhengchao An b4901abd17 fix(admin): keep data usage endpoint available (#5494) 2026-07-31 00:11:14 +00:00
Zhengchao An 342ee1df78 feat(kms): add backend capability discovery (#5485) 2026-07-31 06:09:43 +08:00
houseme 7051a5ce41 feat: add opt-in hotpath profiling (#5488)
* feat: add opt-in hotpath profiling

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: fix vault kms client construction

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 01:49:19 +08:00
Zhengchao An d4f2efa2ad fix(kms): report accurate Vault KV2 security contract and disable unsafe rotation (#5474) 2026-07-30 22:56:43 +08:00
Zhengchao An 19cdd806a2 fix(kms): fail closed on missing or corrupt key material (#5475) 2026-07-30 22:56:30 +08:00
唐小鸭 ad7663afd1 refactor(sse): decouple ecstore and harden KMS lifecycle (#5435)
* refactor(sse): decouple encryption from ecstore

* feat(kms): enhance KMS service manager with runtime state and persistence support

* feat(kms): add local key export functionality for SSE-S3 migration tests

* fix(kms): keep local key export narrowly scoped

* fix(sse): validate copy source customer algorithm

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-07-30 12:39:25 +08:00
Zhengchao An 2d4f77fd3b fix(iam): add correctly named policy APIs (#5457) 2026-07-30 00:55:23 +00:00
Zhengchao An 3a6212f597 fix(admin): fail closed for empty server context slots (#5452)
* fix(admin): fail closed for empty server context slots

* test(embedded): cover uninitialized context slot window

* test(embedded): authenticate startup probe as server B

* fix(admin): satisfy strict context-slot test lints
2026-07-30 07:25:33 +08:00
Zhengchao An d77439929c fix(server): preserve non-S3 trace context (#5450) 2026-07-30 07:24:23 +08:00
Henry Guo abc5f2e818 fix(scanner): persist portable usage cache keys (#5444)
* fix(scanner): persist portable usage cache keys

* fix(scanner): validate complete bucket cache graphs

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 23:22:30 +00:00
Zhengchao An 719c0d6ef0 feat(rpc): add replay-scoped internode authentication (#5455) 2026-07-29 23:12:38 +00:00
cxymds 0247c48ce0 fix(tiering): bind recovery to transaction metadata (#5409)
* fix(tiering): bind recovery to transaction metadata

* fix(tier): add operator transition reconciliation (#5410)

* fix(tier): add operator transition reconciliation

* fix(tiering): require live fleet capability proof (#5423)
2026-07-29 18:44:44 +00:00
Zhengchao An 88fa3877c1 fix(ecstore): serialize every bucket config write under the transaction lock (#5445)
Only replication-target writes took the bucket transaction lock. Every other
config write (policy, tagging, lifecycle, versioning, ...) went straight to
the process-local metadata-system guard, which serializes nothing across
nodes.

Each config write is a read-modify-write of one whole BucketMetadata blob:
load the blob, replace one field, save the blob back. The namespace locks
inside read_config/save_config are taken and released separately, so they do
not span that cycle. Two nodes updating different config files of the same
bucket therefore both load the same blob, each set their own field, and the
later save drops the other's -- with both clients already told 2xx. This is
not last-writer-wins on one document; an orthogonal config silently vanishes.

Route update(), delete() and update_config_with() through
acquire_config_write_guards(), which takes the cluster-wide transaction lock
first and the metadata-system write guard second. That order is load-bearing:
taking the process-local guard first would park every local reader and writer
of every bucket behind a lock whose holder may be another node, turning
remote contention into a local stall.

The lock is per bucket rather than per config file, since a per-file key
would let exactly the offending pair run concurrently. Rename the helper to
acquire_bucket_metadata_transaction_lock to match, but deliberately keep the
lock resource string as "bucket-targets/{bucket}/transaction.lock": the key
is what nodes agree on, so renaming it would leave a mixed-version cluster
with two disjoint keys and stop old and new nodes from excluding each other
on the very writes that are serialized today.

update_config_with() already narrowed its staleness window to a single load
and save, but its exclusion was explicitly process-local; it is now
cluster-wide, so its doc comment no longer disclaims cross-node races.

Also make update_and_parse load through self.api instead of the ambient store
handle, so the read and the write of one read-modify-write cannot resolve to
different instances.

The new tests drive two BucketMetadataSys instances over one ECStore -- the
in-process stand-in for two nodes, since they share no RwLock and can only be
serialized by the namespace lock. Verified the lost-update test has teeth by
removing the lock and confirming it fails on round 0, with the tagging config
clobbered to empty by the concurrent policy write.
2026-07-29 17:17:09 +00:00
GatewayJ 2dea4a9acf fix(s3): correlate server-owned request IDs (#5433)
* fix(s3): correlate server-owned request IDs

* fix(server): preserve trace context and Swift routing

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 23:50:59 +08:00
houseme 09157485aa fix(notify): reconcile persisted bucket rules (#5437)
Restore persisted bucket notification rules after the notification target runtime converges so restarted nodes rebuild their local rule engine without requiring an unchanged PUT bucket notification request.

Fixes #5428

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 15:00:13 +00:00
cxymds 55be5af661 fix(ecstore): fence bucket metadata generations (#5427) 2026-07-29 22:34:11 +08:00
houseme f7c1b13c0f refactor(deps): replace md5 crate with md-5 (#5432)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 11:29:37 +00:00
唐小鸭 f329d330df feat(kms): support safe local KMS evaluation workflows (#5418)
* feat(kms): enable safe local KMS evaluation workflow

* test(kms): align SSE reconfigure coverage

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-07-29 10:19:52 +00:00
Zhengchao An d48870df97 fix(rpc): authenticate non-disk mutation bodies (#5425) 2026-07-29 09:17:30 +00:00
cxymds b65210b1db fix(ilm): preserve restore source version mode (#5406)
* fix(ilm): preserve restore source version mode

* test(ilm): cover suspended null-version restore

* fix(ilm): reconcile worker results from journal
2026-07-29 16:16:40 +08:00
Zhengchao An a7f035a8c3 fix(ecstore): fail peer metadata reloads closed instead of caching fabricated defaults (#5396)
The LoadBucketMetadata peer-notification handler loaded bucket metadata
with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and
unconditionally cached the result. On a transient read-quorum dip during
a reload notification, a peer cached an authoritative "no Object Lock"
default for a lock-enabled bucket, disabling the batch-delete retention
gate (object_lock_delete_check_required) on that node until the next
refresh, and wiping its bucket-target/durability sync state.

Production changes:
- New BucketMetadataSys::reload_from_store (metadata_sys::
  reload_bucket_metadata): the peer reload path uses the presence-aware
  loader and installs only metadata actually read from persisted
  storage. A load miss returns an error (surfaced to the notifying peer
  as success=false) and leaves the cache untouched; deletion still
  propagates only through the dedicated DeleteBucketMetadata
  notification.
- The reload runs under the outer metadata-sys write guard, load
  included, mirroring update(): every other cache installer holds that
  lock, so a reload snapshot can never land after - and roll back - a
  newer concurrent install (the stale-load lost-update from the
  review), and the install-plus-registry-sync sequence stays atomic
  against concurrent removes and reloads (previously only the set call
  was write-guarded, with the load outside any lock).
- The peer-visible miss error is a fixed string: the notifying peer
  substring-matches error text against network-failure needles
  (is_network_like_error), so interpolating a bucket name (e.g. a legal
  bucket literally named "unavailable") could mark a healthy peer
  offline.
- get_config's lazy insert routes through set(), picking up the
  negative-cache invalidation.

An earlier draft instead guarded set() with a per-config updated_at
freshness comparison. Adversarial validation rejected it (three roles
independently): update_config stamps with the handling node's wall
clock, so within the skew the cluster already tolerates (+/-300s RPC
auth window) a config rewritten with an earlier stamp - e.g. revoking a
public-read policy through a second node, or any same-field rewrite
after an NTP step-back - would be skipped by every peer forever,
silently pinning the revoked permissive config with no re-convergence
path (the 15-minute refresh also routed through the guard). Race
staleness is second-scale while skew is minute-scale, so no tolerance
bound can separate them; the write-guard serialization closes the same
race without clocks and preserves the refresh loop's unconditional
converge-to-disk property, which is the cluster's self-healing
mechanism.

Startup audit (BucketMetadataSys::init): concurrent_load's
insert-if-vacant still installs a fabricated default when a transient
miss hits at boot - indistinguishable from a legacy bucket without a
metadata file at this layer - bounded by the next successful persisted
load. Making the object-lock gate fail closed on such entries is filed
as a follow-up, alongside the bare "unavailable" needle in
is_network_like_error and the Swift cache-only metadata writes.

Tests:
- bucket::metadata_sys::tests::
  peer_reload_never_caches_fabricated_defaults_as_authoritative:
  miss installs nothing / miss keeps the existing entry intact
  (asserting the dedicated non-persisted error) / persisted reload
  converges the cache over a stale entry.
- node_service::tests::
  test_load_bucket_metadata_failure_skips_scanner_maintenance:
  a failed reload reports failure and does not advance scanner
  maintenance activity (previously recorded even on a miss).
- The handler success path stays uncovered at the RPC layer (needs an
  isolated global object layer, like the pre-existing ignored test);
  the composition is pinned at the sys level instead.

Verification:
- cargo fmt --check and cargo clippy --lib --tests clean on
  rustfs-ecstore and rustfs.
- Targeted suites green; full cargo test -p rustfs-ecstore --lib:
  3198/3200 with two parallelism-sensitive lock-test flakes from the
  known baseline (pass in isolation; a different pair flakes per run).
- Adversarial validation (high-risk tier, all seven roles as
  independent parallel reviewers) run per AGENTS.md; all findings
  fixed or rebutted with evidence, three out-of-scope findings filed
  as follow-up tasks.

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 08:13:36 +00:00
cxymds d9efd6b853 feat(ecstore): add remote snapshot lease RPCs (#5389)
* feat(ecstore): add local snapshot leases

* feat(ecstore): add remote snapshot lease RPCs

* fix(rpc): keep snapshot lease checks CI-compatible
2026-07-29 15:06:18 +08:00
cxymds cb62079ba6 test(ilm): make active cancellation deterministic (#5403)
* ci: preserve timeout evidence for workspace tests

* test(ilm): make active cancellation deterministic

* ci: allow cold doctest compilation to finish

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 13:53:59 +08:00
cxymds 294c79c156 fix(tiering): gate remote version state safely (#5374)
* feat(tiering): model provider version capabilities

* feat(tiering): persist opaque remote versions

* fix(tiering): gate remote version state safely

* fix(tiering): preserve remote version state on delete

* fix(tiering): accept unversioned transition responses

* fix(tiering): replay exact cleanup journals

* test(tiering): pin empty exact cleanup guard

* test(tiering): accept strict missing journal errors

* test(tiering): exercise free-version identity guard

* test(tiering): reach destination identity guard

* test(tiering): persist version identity drift

* test(tiering): bind version drift fixture

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 02:43:28 +00:00
houseme 5af56cbb02 test(ci): stabilize lifecycle timeout coverage (#5404)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 01:37:29 +00:00
houseme 775279b6fd fix(notify): defer disabled bootstrap storage refresh (#5373)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 17:39:22 +00:00
cxymds eb755e2b97 fix(auth): compare sensitive tokens in constant time (#5371)
* fix(auth): compare sensitive tokens in constant time

* test(auth): scope constant-time source guard

* test(auth): ignore test source in comparison guard

* chore(deps): use constant-time s3s authentication
2026-07-28 17:56:22 +08:00
cxymds 6bc5fc77b5 fix(notify): emit noop event for missing deletes (#5381) 2026-07-28 17:05:10 +08:00
cxymds b432f31c2c fix(multipart): preserve retried parts on quorum failure (#5363)
* fix(multipart): preserve retried parts on quorum failure

* style(multipart): format transaction rollback

* fix(proto): regenerate multipart transaction RPCs

* fix(multipart): import rollback marker constant

* fix(multipart): export transaction action

* fix: import multipart transaction test requests
2026-07-28 17:04:46 +08:00
cxymds 4fb9b0dc7f fix(authz): fail closed on policy load errors (#5359)
* fix(authz): fail closed on policy load errors

* test(authz): cover policy failure precedence

* test: align policy failure expectations

* test: align upload part copy fail-closed expectation
2026-07-28 17:04:35 +08:00
唐小鸭 2216f00cfd fix(kms): unify persisted SSE data key envelopes (#5343)
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation

* feat: enhance local SSE DEK handling with JSON envelope format and versioning
2026-07-28 17:02:18 +08:00
cxymds fd2a87d47e fix(s3): reject empty multipart completions (#5352)
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-28 08:35:50 +00:00
GatewayJ 5cedab09ab fix(sts): return Query API-compatible responses (#5282)
* fix(sts): return Query API-compatible responses

* fix(sts): resolve review and CI failures

* fix(sts): harden query response conversion

* test(e2e): stabilize transition conflict coverage

* fix(ilm): retry vanished transition admission CAS

* test(e2e): narrow transition overlap coverage

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-28 10:39:17 +08:00
Henry Guo d5df66ac4f fix(scanner): require authoritative usage snapshots (#5333) 2026-07-28 07:29:57 +08:00
Zhengchao An 5a768d3a44 perf(s3): resolve bucket versioning once per DeleteObjects request (#5340)
DeleteObjects rebuilt ObjectOptions inside its per-key phase-1 loop, and
del_opts re-fetched the bucket versioning configuration on every call: a
1000-key request paid 1001 identical metadata-sys lookups, each taking the
global bucket-metadata read lock and cloning a VersioningConfiguration.

Split del_opts into the existing async entry point plus a synchronous
del_opts_with_versioning that takes an already-resolved configuration, and
reuse the request-level version_cfg the handler had already fetched.

This also removes a snapshot inconsistency. The skip-stat decision
(can_skip_delete_objects_pre_stat) and the post-delete accounting already
derived from the request-level snapshot while del_opts re-read the config
per key, so a PutBucketVersioning committing mid-request could give a key
opts.versioned=true while the batch delete still ran with versioned=false:
the object was deleted outright with replication state attached to a delete
marker that was never created, and the advisory object-lock pre-check was
skipped on a false premise. All three now derive from one snapshot.

GET, HEAD and DeleteObject additionally establish bucket existence before
building ObjectOptions, so a request naming a nonexistent bucket no longer
performs bucket-metadata work first. GET keeps its cheap request-shape
validations (key, range, partNumber) ahead of the existence check so
InvalidArgument still wins over NoSuchBucket for malformed requests.

Bucket validation moves to validate_bucket_exists, which takes an explicit
store and shares the existing 5s-TTL cache. GET, PUT, HEAD and DeleteObject
now resolve the store through the request-bound server context
(backlog#1052 S6) instead of the process-global handle: get_validated_store
resolves the first-published global AppContext, so in an embedded
multi-instance process a request to the second server operated on the first
server's store. Remaining global-handle call sites in bucket_usecase and
select_object are left for a follow-up.
2026-07-28 06:33:17 +08:00
houseme 6d3ce90c0f fix(lifecycle): harden manual transition rollout checks (#5344)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 17:29:06 +00:00
houseme 0e42a3d1d9 fix(audit): load env-only targets at startup (#5342)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 16:41:38 +00:00
abdullahnah92 c3aac2279f fix(site-replication): keep reverse direction after config broadcast (#5292)
* fix(site-replication): keep reverse direction after config broadcast

`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.

Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.

Four defects kept that invisible or unrecoverable:

- `update_all_targets` discarded target-client build errors silently, and
  `replicate_object` logged the resulting missing-target drop at debug
  while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
  destination named a remote site, so two sites holding identical
  configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
  encrypted with a previous root secret decode as "no such account".
  Startup now reconciles the account, reseeding from the secret every
  site-replication bucket target already stores, and refuses the
  delete-then-create sequence when the parent cannot back an account.
  Bucket rules are reconciled too, so an already-broken site heals on
  upgrade.
- A joined site never verified it could reach the initiator, whose
  endpoint is derived from the Host header of the admin request that
  created the topology. The join now probes each peer and reports through
  `initial_sync_error_message`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): report unreachable targets and reconcile on a timer

Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.

Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.

The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(site-replication): invert reconcile dependency to satisfy layers

`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.

Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.

Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.

The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): harden reconcile per review feedback

Addresses the automated review on #5292.

Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.

Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.

Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.

Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.

Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-27 22:39:23 +08:00
GatewayJ 24cf2cdb78 fix(iam): align OIDC parent IDs with MinIO (#5290)
* fix(iam): align OIDC parent IDs with MinIO

* test(iam): cover OIDC STS binding policy lookup

* test(admin): use runtime facade for AppContext setup

* test(ilm): wait for lifecycle backfill before manual failure

* test(ilm): make overlapping admission check concurrent
2026-07-27 07:16:22 +00:00
houseme 5af997ce5f test(ilm): cover manual transition e2e stress (#5320)
* test(e2e): cover distributed manual transition admission

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: cover manual transition restart recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover queue pressure status readback

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 18:46:15 +00:00
houseme 683ff52a7b feat(internode): track msgpack decode traffic (#5309)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-26 17:03:58 +00:00
Zhengchao An b2a376c2d2 Merge commit from fork
* fix(admin): bound IAM import archive expansion

MAX_IAM_IMPORT_SIZE caps the compressed upload at 10 MB, but every member of the
archive was then read with read_to_end into an unbounded Vec. Deflate ratios well
above 100:1 are easy to construct, so a small authorized upload could expand
without limit across the seven members ImportIam reads.

Add a shared expansion budget (MAX_IAM_IMPORT_EXPANDED_SIZE, 10x the compressed
cap) drawn down by every member, and route all seven reads through one helper
that reads a byte past the remaining budget to detect overrun. Sharing the budget
bounds the archive as a whole rather than letting each member spend the full
limit independently.

Covers R03-CAN-024 through R03-CAN-030 plus R04-CAN-077 (backlog #1471) — one
fix rather than seven, since all seven call sites were byte-identical.

* fix(kms): confine local key paths and refuse silent key replacement

Local KMS key identifiers arrive from request input — the `name` tag on CreateKey,
the `keyId` body field or query parameter on DeleteKey — and were joined onto
`key_dir` with no validation. An identifier such as `../../tmp/evil` escaped the
configured directory, making key creation a constrained arbitrary-file write and
`DeleteKey` with `force_immediate` a cross-directory delete.

Validate in `master_key_path` and make it fallible, so every filesystem path in
this backend inherits the guard: decode_stored_key, load_master_key,
save_master_key, create_key and delete_key all derive their paths there. The rule
is containment rather than a character allowlist, so identifiers already in use
keep resolving; only separators, NUL, absolute paths and non-single-component
forms are refused. Note `.` and `..` are contained rather than refused — the
`.key` suffix turns them into the ordinary filenames `..key` and `...key`.

Separately, `LocalKmsBackend::create_key` had no existence check, while the
sibling `KmsClient::create_key` has always had one. Since `save_master_key`
renames over its destination, creating a key under an existing name silently
replaced its material and destroyed the ability to decrypt everything wrapped
under it — and the backend path is the one the admin API uses. It now returns
KeyAlreadyExists, matching StaticKmsBackend.

Covers R03-CAN-072, R03-CAN-073 and R07-CAN-103 (backlog #1475). R03-CAN-073
needed no separate change: delete_key routes both its load and its remove_file
through master_key_path.

* fix(swift): bound SLO manifest reads to the 2 MiB manifest limit

The three Swift SLO handlers that load a stored manifest (handle_slo_get,
handle_slo_get_manifest, handle_slo_delete) read the `<object>.slo-manifest`
object to EOF with AsyncReadExt::read_to_end. That key is predictable and
writable through the ordinary object PUT path, so a tenant can replace the
manifest with an arbitrarily large object and then make the server allocate
its full size on every SLO GET, multipart-manifest=get, or
multipart-manifest=delete request - a memory amplification bounded only by
the stored object size (CWE-400 / CWE-770). The 2 MiB manifest limit that
handle_slo_put enforces was not applied on the read side.

Introduce MAX_SLO_MANIFEST_SIZE (the existing 2 MiB PUT limit, now a named
constant) and a shared read_manifest_bytes helper that reads through a
`take(limit + 1)` and rejects anything larger, so an oversized manifest is
refused instead of being buffered first. All three call sites go through the
helper. handle_slo_put now checks the size before parsing the JSON.

Regression tests: test_read_manifest_bytes_rejects_oversized_manifest and
test_read_manifest_bytes_stops_reading_oversized_manifest (which asserts the
reader is not consumed past the limit), plus a boundary test that a manifest
at exactly 2 MiB is still accepted.

* fix(protocols): authorize every object in FTPS/WebDAV recursive deletes

The FTPS and WebDAV gateways authorized only the container before a
recursive delete and then destroyed everything inside it without a
further check:

- FTPS RMD (and DELE on a bucket path ending in '/') cleared
  s3:DeleteBucket, then delete_bucket_recursively listed the bucket and
  deleted every object.
- WebDAV DELETE on a bucket did the same via its own
  delete_bucket_recursively.
- WebDAV DELETE on a directory cleared s3:DeleteObject for the directory
  marker key ("dir/") only, then listed that prefix and deleted every
  child under it.

A principal holding s3:DeleteBucket (or s3:DeleteObject on a single
marker key) could therefore erase objects it had no s3:DeleteObject
permission for, and the operation reported success.

Deletion stays recursive - that is the expected behaviour for these
protocols - but each object now clears s3:DeleteObject on its own key
before it is removed, and the enumeration clears s3:ListBucket. A denial
aborts the whole operation with access denied rather than being skipped,
so the caller can never be told the delete succeeded while objects were
left behind or removed without authorization.

The test double gained shared-state cloning, delete_object/delete_bucket
call logs, and list/delete queue helpers so the regression tests can
observe that nothing is deleted once a deny lands.

* fix(server,ecstore): bound TLS handshakes and remote volume RPC waits

Three call sites let an unauthenticated client or a misbehaving peer hold
server resources with no deadline.

TLS listener (R03-CAN-035): process_connection awaited
`acceptor.accept(socket)` with no bound. A client that opens a TCP
connection and never finishes the handshake parks a Tokio task and a socket
forever, and the connection cap (RUSTFS_API_MAX_CONNECTIONS) is unlimited by
default, so nothing else sheds it. The handshake now runs under
accept_tls_with_deadline(), reusing the existing HTTP/1 header-read budget —
the established slow-client bound for the pre-request phase — and the
expiry is recorded through the same log/metric path as a handshake error,
under a new TIMEOUT failure kind.

Remote disk RPCs (R03-CAN-049, R03-CAN-050): list_volumes and delete_volume
passed Duration::ZERO, which execute_with_timeout treats as "no deadline",
so a peer that accepts the request and never answers stalls the coordinator
(and, for delete_volume, the bucket-deletion workflow). Both now pass
get_max_timeout_duration(), matching every sibling method in the file.

Regression tests: a silent TLS peer must be shed by the handshake deadline;
list_volumes/delete_volume against a peer that completes the TCP connect and
then goes silent must fail with DiskError::Timeout instead of hanging.

* fix(security): stop leaking signed headers and bound OIDC/KMS credentials

Three independent hygiene fixes found by the security review.

R03-CAN-018 (crates/signer): try_get_canonical_headers and get_signed_headers
logged the complete header map at DEBUG before signing. Runtime callers pass
session credentials and SSE-C key material through these headers, so anyone
able to raise the log level (or read DEBUG logs) recovered
X-Amz-Security-Token and SSE-C keys verbatim. The statements were debugging
leftovers with no operational value and are deleted rather than redacted.

R03-CAN-014 (crates/iam): the OIDC HTTP adapter buffered provider responses
with an unbounded Response::bytes(), so a configured, compromised or
attacker-pointed IdP endpoint could stream an arbitrarily large or endless
body into memory (the ValidateOidcConfig admin handler lets a ServerInfo
caller choose the endpoint). Responses are now read incrementally and fail
closed past MAX_OIDC_RESPONSE_SIZE, and the already SSRF-hardened client
builder gains request and connect timeouts so a stalled provider cannot pin
the calling task indefinitely.

R07-CAN-105 (helm): the Vault KMS token was serialized into the chart
ConfigMap, exposing it to every subject allowed to get ConfigMaps in the
namespace. It now renders into a dedicated Secret that the Deployment and
StatefulSet consume via envFrom; the Secret is separate from the main
credentials Secret so it also works when secret.existingSecret is set.

Regression tests:
- rustfs-signer: signing_never_logs_signed_header_material
- rustfs-iam: oidc_response_body_past_the_limit_is_rejected,
  oidc_response_body_at_the_limit_is_accepted
- scripts/test_helm_templates.sh: KMS token must never render in plaintext

* fix(webdav): enforce body limit, request timeout and connection cap

The configured WebDAV maximum body size was enforced from Content-Length, so a
chunked request declared no length and bypassed it entirely. The configured
request timeout was never applied to the connection at all, and the accept loop
spawned a task per connection with no bound, so an unauthenticated client could
hold resources indefinitely and in unbounded number.

Enforce the limit on bytes actually read rather than the declared length, apply
the configured timeout to the request, and bound accepted connections with a new
RUSTFS_WEBDAV_MAX_CONNECTIONS (default 1024) surfaced in the config report.

Covers R03-CAN-051, R03-CAN-052, R03-CAN-067, R04-CAN-089, R05-CAN-094 and
R05-CAN-097 (backlog #1471, #1474).

* fix(security): stop STS credentials from crossing the parent trust boundary

Two related credential-boundary holes let a short-lived STS credential act
with the full, unrestricted authority of the long-term user it was minted
from.

AddUser (R03-CAN-021, CWE-269/863): should_check_deny_only relaxes the admin
policy check to deny-only when a Console/STS session targets the IAM user it
represents. Nothing then stopped that session from calling AddUser with its
own parent's access key, so the handler wrote an attacker-chosen secret key
and status over the parent's stored Credentials via create_user ->
save_user_identity. A session that expires in minutes became permanent
control of the account. AddUser now rejects any temp or service-account
requester whose resolved parent equals the target access key, resolving the
parent the same way should_check_deny_only does (parent_user field, else the
JWT `parent` claim, since some stores persist the parent only in the token).

FTPS/SFTP/WebDAV password auth (R04-CAN-086, CWE-287/862): these protocols
looked the access key up with check_key, which falls back to the STS account
cache, and then compared only the stored secret. An STS access key plus
secret therefore authenticated with no session token presented and no
session-policy claims applied - the holder got the parent's full permissions.
Password authentication now rejects temporary credentials before the secret
comparison. The discriminator is is_temp() && !is_service_account(), the same
one IamCache::update_user_with_claims uses to route an identity into the STS
cache, so service accounts - which resolve policy from stored IAM state
rather than a client-presented token - keep working over these protocols.

Regression tests cover both predicates and pin the guards to their call
sites so neither can be dropped without a test failure.
2026-07-27 00:22:50 +08:00
houseme 887868e7cd test(admin): expose manual transition job capabilities (#5306)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-27 00:17:40 +08:00