Commit Graph

2117 Commits

Author SHA1 Message Date
houseme f0c4fbd28f chore(deps): refresh mimalloc revision (#5736)
* chore(deps): refresh mimalloc revision

Update mimalloc and libmimalloc-sys to the requested git revision after running the dependency refresh flow.

Keep ratelimit excluded while accepting compatible dependency updates from cargo update and cargo upgrade.

Harden all-feature test compilation by giving heavy integration test crates their own recursion limit and avoiding a cross-thread spawn for the embedded startup barrier future.

Co-Authored-By: heihutu <heihutu@gmail.com>

* upgrade version

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-05 04:46:13 +00:00
唐小鸭 15b9c1f4e3 fix(replication): make bucket replication rules editable from clients (#5715)
* fix(replication): accept explicit STANDARD destination storage class

The replication engine never reads Rule.Destination.StorageClass (replica
placement comes from the bucket-target config or the source object), yet the
validator rejected any config carrying the field. The console's add-rule form
always sends StorageClass=STANDARD, so every rule created through it failed
with InvalidRequest.

Tolerate exactly STANDARD as a no-op — semantically identical to omitting
the field — and keep rejecting every other value, which would be silently
ignored rather than honored. Document the deliberate omission from the
replication capability contract.

* feat(admin): support MinIO-style partial updates for set-remote-target

set-remote-target?update=true previously replaced every stored field and
required complete credentials in the body, so flipping a target's sync mode
from the console forced operators to re-enter the secret key, and real
mc replicate update bodies (madmin Clone() strips the secret) failed to
deserialize at all.

Adopt MinIO's TargetUpdateType contract: query params creds/sync/bandwidth/
path name the field groups to overlay onto the stored target, everything
else keeps its persisted value, and unsupported groups (proxy, healthcheck,
edge, edgeSyncBeforeExpiry) fail loudly. Credentials updates are skipped for
site-replication peer targets — probed by both scheme derivations of the
stored endpoint and the stored deployment id — because an operator never
knows the site replicator's credentials, and a body-supplied deployment id
is ignored on update since it anchors peer identity. madmin JSON aliases
(bandwidthlimit, storageclass, resetID, deploymentID, sessionToken) let mc
bodies parse under deny_unknown_fields.

e2e: cover a credential-free sync-only update preserving the stored
connection and the zero-ops no-op contract; align the missing-arn assertion
with the earlier validation error.

* chore(scripts): add two-site replication lab manager

site_replication_smoke.py spawns and manages two local rustfs processes,
pairs them via the site-replication admin API (idempotent), and verifies
bidirectional object replication. Subcommands: up/down/restart/status/logs/
smoke/info/remove/clean. Stdlib-only; requests are SigV4-signed the same
way as crates/e2e_test.

* chore(scripts): rename direction-suffixed payload variables for typos check

The typos linter reads the _ba suffix in payload_ba as a misspelling of
"by"; use payload_a_to_b / payload_b_to_a instead.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-05 01:50:31 +00:00
Zhengchao An 4576c2e470 fix(iam): invalidate peer STS caches on revocation (#5718) 2026-08-05 01:17:36 +00:00
houseme 16c2928965 refactor(metrics): migrate scanner report timestamps to jiff (#5710)
* refactor(metrics): migrate scanner report timestamps to jiff

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(madmin): migrate admin timestamps to jiff (#5712)

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(storage): migrate RPC DTO timestamps to jiff (#5713)

Co-authored-by: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 18:35:38 +00:00
houseme 3dabac4a09 perf(observability): avoid cgroup path allocation (#5711)
* perf(observability): avoid cgroup path allocation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): satisfy regression test clippy

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 17:47:07 +00:00
houseme 3405b4e980 perf(observability): avoid sampler stat allocations (#5708)
* perf(observability): avoid cgroup stat key allocations

Replace the memory.stat HashMap parser with a fixed-field parser so the memory observability sampler does not allocate String keys or hash every cgroup field on each interval.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(observability): parse mimalloc stats without copying

Parse the mimalloc stats JSON while the mimalloc-owned buffer is still alive, then free it immediately. This avoids allocating an owned String on each allocator memory sample.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 23:20:49 +08:00
houseme 510b0350d6 refactor(time): migrate audit and notify timestamps to jiff (#5707)
* refactor(time): migrate audit and notify timestamps to jiff

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): initialize heal walk decode error

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(targets): parse MySQL event time with jiff

Preserve MySQL DATETIME(6) wall-time formatting for RFC3339 eventTime values while removing the direct chrono dependency from rustfs-targets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(deps): prune unused workspace dependencies

Apply cargo shear --fix to remove unused path-clean and s3select-api tempfile entries after the scoped jiff migration.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): remove duplicate heal walk decode error init

Remove the duplicate decode_error field from the heal walk test collector initializer so lib-test clippy compiles on CI.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(policy): emit OPA timestamps with jiff

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 23:20:44 +08:00
Zhengchao An f3eba31aee fix(ecstore): retain namespace read locks for streaming GETs to prevent quota-bypass (#5699)
* fix(ecstore): retain locks for streaming GETs

* fix(rpc): keep disabled snapshot leases lint-clean
2026-08-04 23:20:35 +08:00
Zhengchao An a43267160d fix(auth): enforce object-lock actions for POST uploads (#5701) 2026-08-04 23:20:24 +08:00
Zhengchao An 2039ba5f65 fix(s3select): enforce SSE-KMS read authorization (#5698)
* fix(s3select): enforce SSE-KMS read authorization

* fix(app): route select SSE auth through facade
2026-08-04 23:20:19 +08:00
Zhengchao An 3a6f630ff1 fix(api): bound bucket rate limiter keys (#5706) 2026-08-04 15:13:25 +00:00
Zhengchao An 1695873e55 fix(auth): isolate embedded IAM contexts (#5704) 2026-08-04 23:12:35 +08:00
cxymds eb87bb1faf fix(replication): harden resync and MRF recovery (#5694)
* fix(replication): harden resync and MRF recovery

* fix(replication): correct MRF validation regressions

* fix(replication): address CI validation failures

* fix(heal): initialize decode error in merge test
2026-08-04 13:40:50 +00:00
houseme 93fcd6b6b5 fix(observability): fallback mimalloc requested memory stats (#5695)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 21:10:29 +08:00
cxymds 31959b90db fix(heal): harden resumable set repair failures (#5693)
* fix(heal): enforce resumable task control

* fix(ecstore): surface bucket and metadata heal errors

* chore: refresh guardrail path references

---------

Signed-off-by: cxymds <cxymds@gmail.com>
2026-08-04 12:35:58 +00:00
Zhengchao An cfce7bd9b1 fix(filemeta): preserve FileInfo wire compatibility (#5689) 2026-08-04 09:25:28 +00:00
houseme b71483b1c8 fix(ecstore): split internal get metadata metrics (#5687)
Classify expected metadata-missing errors separately from unknown get pipeline failures and attribute internal meta-bucket reader failures to an internal_meta path instead of legacy_duplex.

This keeps scanner/data-usage metadata probes from polluting user GET/mixed failure attribution while preserving the existing read error behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-04 08:37:09 +00:00
Henry Guo d6e11cf018 refactor(table-catalog): modularize catalog implementation (#5678)
* refactor(table-catalog): split catalog foundations

* refactor(table-catalog): split REST handler modules

* refactor(table-catalog): split domain and store modules

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-04 11:48:00 +08:00
Zhengchao An e08cf474db fix(audit): move audit init after IAM bootstrap to fix startup ordering (#5685)
Audit initialization requires the AppContext (server config + object
store) which is published by ensure_startup_after_iam inside
init_iam_runtime. Moving init_audit_runtime after init_iam_runtime
ensures the runtime sources are available when audit starts.

Fixes #5681

Co-authored-by: RustFS <hello@rustfs.com>
2026-08-04 11:45:48 +08:00
Zhengchao An 98d3619613 fix: address rc.1 release blockers (#5648)
* fix: address rc.1 release blockers

* fix: route release guards through architecture boundaries

* fix: close remaining rc.1 regression gaps

* refactor: group multipart listing options

* fix: resolve rc.1 CI regressions

* fix(ecstore): keep bucket-config writes off the caller's stack

A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.

Two CI lanes aborted with SIGABRT on this:

  ILM Integration (serial)
    rustfs app::lifecycle_transition_api_test::
      compensation_driven_complete_multipart_upload_still_transitions
  Test and Lint (swift)
    rustfs-protocols::swift_metadata_persistence::
      swift_metadata_writes_are_durable

Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.

Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.

* fix(lifecycle): unbreak restore under the bucket generation fence

The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.

1. RestoreObject and ListMultipartUploads hard-required
   `opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
   deliberately leaves it unset when no guard extension is present — only the
   S3 access layer installs one. Every direct caller therefore got
   `InternalError: ... bucket generation guard is missing`. Resolve the
   current generation instead, the way the copy path already does. The fence
   is unaffected: RestoreObject still re-reads the incarnation from disk and
   compares before admitting the restore, and the multipart listing is
   filtered by the value it resolves.

2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
   restored-copy expiry whose `restore_expires` had not already elapsed.
   Whether the restored copy is due to expire is the ILM evaluator's
   decision, made when it emitted DeleteRestoredAction; re-deriving it in
   the set layer only adds a way for a legitimate action to be rejected.
   The stale-event risk it appears to guard is already covered by the
   surrounding snapshot match — a re-restore rewrites `restore_expires`,
   so a replayed event fails the equality check. Drop the clause; the
   fifteen identity clauses are unchanged.

Fixed:
  rustfs app::lifecycle_transition_api_test::
    restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
    restore_object_usecase_completes_suspended_null_version_in_place
    restore_object_usecase_reports_ongoing_conflict
  rustfs-scanner::lifecycle_integration_test serial_tests::
    test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore

Verification: the CI ILM lane filter now runs 53/53 green locally.

* chore: address review follow-ups on this branch

Four items from the adversarial review that were still open.

- Restore the assertion `test_bucket_replication_replayed_delete_marker_
  preserves_source_mtime_without_source_restart` is named for. The branch
  had replaced the backlog#867 mtime check with `assert_replication_
  converged`, which any successful replication satisfies, and deleted the
  two helpers it needed — so the regression the test exists to catch would
  now pass. This matters here specifically because the branch changes the
  flag feeding `replication_delete_remove_options` and routes replay
  through a new file and ordering.

- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
  one real consumer calls the `_with_metadata` variant directly). Its test
  stanza now exercises that variant, so the coverage moves to live code
  rather than being deleted.

- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
  2.7.0 before this branch and is untouched, so the caret range already
  resolved there. Nothing in the diff uses the crate.

- Split the AGENTS.md "Adversarial Validation" policy change out of this
  branch. The edit is defensible on its own, but it relaxes the review gate
  that this branch has to pass, so it should land as its own PR reviewed on
  its own merits rather than bundled with the change that benefits from it.
  The reverted hunks are unchanged and ready to re-apply.

Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.

* test(ci): serialize the new bucket-incarnation tests

The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.

Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.

Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.

* test(ecstore): pin that a drained bucket is actually deletable

`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.

That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.

Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.

The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.

* fix(ecstore): let an explicit null-version delete purge its delete marker

Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.

On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:

    explicit null-version purge of the null delete marker must succeed,
    got [Some(MethodNotAllowed)]

so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.

The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.

This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.

Normalize both sides through `delete_file_info_version_id`.

The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.

* fix(app): drop the now-needless struct update

Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.

Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.

* test(e2e): assert the fresh-volume classification

four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.

Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.

* fix(bucket): apply the requested Object Lock to existing buckets

Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.

Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.

An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.

Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".

cargo nextest run -p rustfs-ecstore --lib: 3633 passed.

* fix(ecstore): box the generation-checked config mutation paths too

The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.

* fix(restore): keep the nil-version normalization the strip removed

Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:

    -  self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
    +  self.version_id == fi.version_id

For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.

Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
2026-08-03 19:25:43 +00:00
ccccpj d8d22599fe fix(replication): preserve multipart source mtime (#5669) 2026-08-03 14:46:04 +00:00
Zhengchao An 975003d60a fix(s3): stop authorizing DeleteBucketWebsite with a read action (#5665)
`delete_bucket_website` authorized through `s3:GetBucketPolicy` while
`put_bucket_website` used `s3:PutBucketPolicy`. The handler is a real
mutation — `rustfs/src/storage/ecfs.rs` calls
`delete_bucket_metadata_config(bucket, BUCKET_WEBSITE_CONFIG)`, permanently
removing the persisted website configuration.

So a principal holding only

    {"Effect":"Allow","Action":["s3:GetBucketPolicy"],
     "Resource":"arn:aws:s3:::victim"}

— an ordinary read-only "may read my bucket policy" grant — could send
`DELETE /victim?website` and destroy the configuration. On a bucket whose
policy grants that to `Principal: "*"`, it is reachable anonymously.

AWS treats this as its own permission: "This DELETE action requires the
S3:DeleteBucketWebsite permission." RustFS has no dedicated
`s3:PutBucketWebsite` / `s3:DeleteBucketWebsite` action, so this keeps the
existing bucket-config convention (`s3:PutBucketPolicy`, the same one
`put_bucket_request_payment` and `put_bucket_accelerate_configuration` use)
rather than adding actions, which would silently invalidate deployed
policies that already grant website writes.

Rather than correcting one constant, both handlers now route through a
single `bucket_website_config_authorize_action()`, so the read/write pair
cannot drift apart again.

Swept the rest of the surface while here: `delete_bucket_website` was the
only mutation handler authorizing through a Get*/List* action.
`delete_bucket_ownership_controls`, `put_bucket_ownership_controls` and
`put_bucket_metrics_configuration` return `Ok(())` with no authorization,
but none of them is implemented outside the access hook, so there is no
operation to authorize — left alone.

Adding a dedicated `s3:DeleteBucketWebsite` for full AWS parity is a
separate change with a policy-compatibility impact; noted, not done here.

Verification: cargo fmt --all --check, git diff --check,
cargo check -p rustfs --all-targets, cargo clippy -p rustfs --all-targets
(clean), and the new regression test. Mutation-checked: restoring
`GetBucketPolicyAction` turns
`bucket_website_config_never_authorizes_through_a_read_action` red.
2026-08-03 16:11:52 +08:00
houseme 371a3529e5 chore(deps): refresh hotpath allocator support (#5660)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 06:52:44 +00:00
houseme a8574d0104 fix(metrics): close dimension review gaps (#5656)
* fix(metrics): close dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(metrics): cover dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(metrics): cover failed disk info UUID fallback

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 04:12:31 +00:00
cxymds 380ec74ece fix(replication): persist force-delete handoff state (#5641)
* fix(replication): persist force-delete handoff state

* fix(arch): route force-delete config access through boundary

* style: format force-delete imports

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-08-03 01:44:28 +00:00
houseme 035ce5d784 feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics

Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets.

Backlog: rustfs/backlog#1655

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): preserve suspect drive runtime state

Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): skip unknown drive inode samples

Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner source work detail metrics

Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add ilm action detail metrics

Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add delivery target server metrics

Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication target flow metrics

Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add request server metrics

Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): apply rustfmt to metrics changes

Apply rustfmt output to the metrics dimension changes without altering behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): reuse audit target label constant

Use the exported audit target_id label constant for legacy audit target metrics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): populate drive disk metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner bucket drive result metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication proxy server metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric liveness review

Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address additional metric review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): count drive calls at start

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metrics dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address scanner review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): reduce disk metric contention

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): retire stale dimension series

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 09:03:34 +08:00
cxymds 2ce670837c fix(ecstore): make transitioned deletes durable (#5644)
* fix(ecstore): make transitioned deletes durable

* fix(ecstore): journal force deletes

* fix(ecstore): journal force deletes
2026-08-02 18:00:11 +00:00
cxymds ec67884f8d fix(replication): preserve durable MRF delete admission (#5643) 2026-08-02 23:53:53 +08:00
Henry Guo e5cfa8e375 feat(table-catalog): paginate Iceberg REST listings (#5466)
* feat(table-catalog): paginate Iceberg REST listings

* test(table-catalog): remove redundant token clones

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-02 15:27:15 +00:00
cxymds 1fdcbd9225 fix(replication): fail closed on destination encryption (#5633)
* fix(replication): fail closed on destination encryption

* test(replication): avoid Debug bound in encryption assertion
2026-08-02 22:54:51 +08:00
cxymds 114b2420a2 feat(admin): expose versioned replication capabilities (#5631)
* feat(admin): expose replication capabilities

* fix(admin): route replication capabilities through facades
2026-08-02 22:54:38 +08:00
GatewayJ 6028dad2f4 test(iam): freeze OIDC federation behavior (#5627) 2026-08-02 22:33:20 +08:00
Zhengchao An 54d8c02a2f fix(targets): explain webhook outbound allowlist failures (#5616) 2026-08-02 22:33:05 +08:00
cxymds 779b5a49ea fix(replication): propagate metadata changes (#5635)
Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.

Refs #1616
2026-08-02 13:50:24 +00:00
cxymds 2cc7443067 feat(replication): bound DeleteObjects queue admission (#5637)
feat(replication): batch DeleteObjects queue admission
2026-08-02 21:06:54 +08:00
cxymds 378c9ba67f fix(replication): enforce bucket write contract (#5629) 2026-08-02 11:51:47 +00:00
Zhengchao An 4473c548be test(admin): audit KMS deletion guard outcomes (#5628) 2026-08-02 11:39:37 +00:00
cxymds 2cdba03dee fix(authz): gate replication-only PUT headers (#5625) 2026-08-02 19:28:54 +08:00
cxymds 885096d1de fix(replication): reject unsupported target options (#5622) 2026-08-02 19:28:30 +08:00
Zhengchao An 2698a03582 test(kms): pin admin KMS response shapes where they are served (#5626)
The snapshots in crates/kms/src/api_types.rs pinned DeleteKeyResponse,
ListKeysResponse, DescribeKeyResponse and CancelKeyDeletionResponse, none
of which is serialized by any handler: those endpoints answer with
DeleteKmsKeyResponse and siblings in rustfs/src/admin/handlers/kms_keys.rs,
separate types carrying different fields. A breaking change to an admin
response could not fail them. Tag, untag and update-description had the
same gap, where the handler discards the kms-side response and serves its
own KmsKeyMetadataResponse.

Pin the shapes in the crate that produces them, and delete the four kms
mirrors. They were never in the pub use api_types list, had no
constructors and no callers, and only looked live because those snapshots
named them.

Keep the api_types snapshots that pin something real: configure, start,
stop and status are served verbatim by kms_dynamic, and the tag family
are live ObjectEncryptionService return types whose snapshots pin this
crate's public API rather than a wire shape.
2026-08-02 11:20:17 +00:00
Zhengchao An b1ddda3bb2 fix(sse): rewrite data when a same-key copy changes encryption (#5618)
A same-name CopyObject marks the operation `metadata_only`, which lets the
store layer rewrite `xl.meta` in place and leave the data blocks untouched.
The handler independently strips the source encryption metadata and calls
`sse_encryption`, which mints a *fresh* DEK. On an unversioned bucket both
happen at once, so the object ends up with a new DEK sitting beside ciphertext
sealed under the old one, and can never be decrypted again.

The mirror case is silent: an encrypted source copied without any destination
SSE keeps its ciphertext while losing the key metadata, so GET returns raw
ciphertext as if it were plaintext, with HTTP 200 and no error anywhere.

Keep `metadata_only` off whenever either side of the copy is encrypted, so the
store layer performs a full read/write rewrite through `put_object`. This is
the same resolution the versioned historical-restore path already uses for
this risk (issue #4238), and it matches MinIO's
`isSourceEncrypted || isTargetEncrypted -> metadataOnly = false` guard in
CopyObjectHandler.

The target half of the predicate deliberately tests `effective_sse` rather
than the request headers MinIO inspects: `effective_sse` also resolves the
bucket default-encryption rule, and `sse_encryption` mints a DEK from that
resolved value. A header-only check would miss a same-key copy performed under
a bucket default rule. The source half reuses `ObjectInfo::is_encrypted` so a
future encryption flavour is covered here as soon as it is recognised there.

Versioned buckets were already safe: that path falls through to `put_object`
regardless of `metadata_only`. RestoreObject also sets `metadata_only` but
only appends restore keys and never re-derives a DEK, so it is unaffected.
2026-08-02 11:00:52 +00:00
Zhengchao An f440a14e53 fix(admin): preflight webhook outbound policy (#5619) 2026-08-02 18:31:52 +08:00
cxymds 7463655ae1 fix(authz): preserve direct delete usecase callers (#5614)
* fix(authz): preserve direct delete usecase callers

* test(ilm): provide owner context for forced deletes
2026-08-02 17:01:32 +08:00
cxymds f00f5fe776 fix(authz): restrict recursive force-delete scope (#5610) 2026-08-02 06:30:36 +00:00
Zhengchao An 2bf2ce1ff2 feat(admin): apply the requested key listing filters (#5608) 2026-08-02 14:03:57 +08:00
cxymds c1955a8498 fix(replication): harden live delete admission (#5599) 2026-08-02 12:52:11 +08:00
Zhengchao An 8e3e552576 fix(s3): strip every encryption marker from a copy destination (#5597)
fix(sse): strip inherited SSE key-id and algorithm on copy

strip_managed_encryption_metadata cleared the MinIO spellings of the
managed-SSE key id and seal algorithm but not their RustFS-native
counterparts, so a CopyObject destination kept the source object's
x-rustfs-encryption-key-id and x-rustfs-encryption-algorithm.

When the destination resolves to no server-side encryption, nothing
rewrites those keys. is_object_encryption_marker treats any remaining
x-rustfs-encryption-* key as proof the payload is encrypted, so the
plaintext destination reports ObjectInfo::is_encrypted, the reader takes
its encrypted branch, and the read fails closed with "encrypted object
metadata is incomplete" because the actual key material was stripped.

Add both constants to the strip list so a destination inherits no
encryption marker it has no material for.
2026-08-02 10:27:23 +08:00
Zhengchao An 557a616ae6 feat(kms): report the configuration references that block a key deletion (#5598)
* feat(kms): report configuration references that block a key deletion

Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.

Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.

* test(kms): cover the immediate-deletion reference refusal

* feat(kms): surface configuration references on the admin key endpoints

DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.

The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.

* test(kms): pin the impact wire shape and the unreferenced force-delete path

* fix(kms): make the DescribeKey impact section opt-in

Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.

A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.

* fix(kms): box the query-parse refusal now that responses carry impact

The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
2026-08-02 08:37:37 +08:00
Zhengchao An 7528a0b916 feat(kms): accept the AWS backend through KMS configuration (#5592)
* feat(kms): accept the AWS backend through KMS configuration

The AWS KMS backend could be constructed but not selected: the admin
configure API had no AWS variant and startup rejected the backend name.

The configure request pins the region rather than defaulting it, because
that configuration is persisted once and replayed on every node: leaving
the region to each node's ambient provider chain would let nodes address
different regions, and therefore different keys, while reporting an
identical configuration. The request accepts no credential fields, so
credentials stay with the aws-config provider chain on each node, and
`deny_unknown_fields` refuses attempts to submit them anyway.

* test(kms): cover AWS backend selection through the service manager

An end-to-end check that an admin configure request selects the AWS
backend, builds a client, and passes the startup health check. Marked
#[ignore]: it needs real AWS credentials, though it creates no key and
is therefore not billable on its own.
2026-08-02 03:58:20 +08:00
Zhengchao An f1a85c6a93 fix(admin): refuse immediate KMS key deletion through the query string (#5589)
fix(admin): retire the query-string form of immediate KMS key deletion

Immediate deletion destroys master key material outright, and every
object encrypted under that key becomes permanently unreadable. The
delete endpoint accepted that request as a query parameter, which is the
form most easily issued by accident and the one that made the waiting
window bypassable.

The query string can now only schedule a deletion: `force_immediate`
with any value other than `false`, or a `confirm_key_id` parameter, is
refused with 400 rather than downgraded to a scheduled deletion, so a
caller cannot read the answer as "destroyed". The JSON body form is
unchanged and remains the single way to reach the service gate that
enforces the server opt-in and the echoed confirmation.

Classify the route accordingly: `RouteRiskLevel` gains `Critical` for
routes whose worst case is permanent loss of user data, and the KMS key
deletion route is the only member, pinned in both directions by a matrix
test. Endpoint-level coverage for the 7-30 day window bound is added for
every configured backend.

Refs rustfs/backlog#1585 (part of rustfs/backlog#1562)
2026-08-02 03:46:11 +08:00