Commit Graph

5159 Commits

Author SHA1 Message Date
Zhengchao An 975003d60a fix(s3): stop authorizing DeleteBucketWebsite with a read action (#5665)
`delete_bucket_website` authorized through `s3:GetBucketPolicy` while
`put_bucket_website` used `s3:PutBucketPolicy`. The handler is a real
mutation — `rustfs/src/storage/ecfs.rs` calls
`delete_bucket_metadata_config(bucket, BUCKET_WEBSITE_CONFIG)`, permanently
removing the persisted website configuration.

So a principal holding only

    {"Effect":"Allow","Action":["s3:GetBucketPolicy"],
     "Resource":"arn:aws:s3:::victim"}

— an ordinary read-only "may read my bucket policy" grant — could send
`DELETE /victim?website` and destroy the configuration. On a bucket whose
policy grants that to `Principal: "*"`, it is reachable anonymously.

AWS treats this as its own permission: "This DELETE action requires the
S3:DeleteBucketWebsite permission." RustFS has no dedicated
`s3:PutBucketWebsite` / `s3:DeleteBucketWebsite` action, so this keeps the
existing bucket-config convention (`s3:PutBucketPolicy`, the same one
`put_bucket_request_payment` and `put_bucket_accelerate_configuration` use)
rather than adding actions, which would silently invalidate deployed
policies that already grant website writes.

Rather than correcting one constant, both handlers now route through a
single `bucket_website_config_authorize_action()`, so the read/write pair
cannot drift apart again.

Swept the rest of the surface while here: `delete_bucket_website` was the
only mutation handler authorizing through a Get*/List* action.
`delete_bucket_ownership_controls`, `put_bucket_ownership_controls` and
`put_bucket_metrics_configuration` return `Ok(())` with no authorization,
but none of them is implemented outside the access hook, so there is no
operation to authorize — left alone.

Adding a dedicated `s3:DeleteBucketWebsite` for full AWS parity is a
separate change with a policy-compatibility impact; noted, not done here.

Verification: cargo fmt --all --check, git diff --check,
cargo check -p rustfs --all-targets, cargo clippy -p rustfs --all-targets
(clean), and the new regression test. Mutation-checked: restoring
`GetBucketPolicyAction` turns
`bucket_website_config_never_authorizes_through_a_read_action` red.
2026-08-03 16:11:52 +08:00
Henry Guo b563230782 fix(ecstore): allow Windows renames under guarded parents (#5663)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-03 16:11:37 +08:00
cxymds 9b4a73f315 fix(replication): harden MRF replay durability (#5659)
* feat(replication): add MRF envelope capabilities

* fix(replication): retain failed MRF replay entries

* fix(replication): retain transient MRF source failures

* fix(replication): address MRF durability review feedback

* fix(replication): preserve MRF recovery handoff

* fix(replication): harden MRF recovery handoff
2026-08-03 15:35:00 +08:00
houseme 371a3529e5 chore(deps): refresh hotpath allocator support (#5660)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 06:52:44 +00:00
houseme a8574d0104 fix(metrics): close dimension review gaps (#5656)
* fix(metrics): close dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(metrics): cover dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(metrics): cover failed disk info UUID fallback

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 04:12:31 +00:00
Zhengchao An 2fb88d2c60 test(ci): serialize cross-node metadata writes (#5654) 2026-08-03 02:00:43 +00:00
cxymds 380ec74ece fix(replication): persist force-delete handoff state (#5641)
* fix(replication): persist force-delete handoff state

* fix(arch): route force-delete config access through boundary

* style: format force-delete imports

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-08-03 01:44:28 +00:00
houseme 035ce5d784 feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics

Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets.

Backlog: rustfs/backlog#1655

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): preserve suspect drive runtime state

Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): skip unknown drive inode samples

Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner source work detail metrics

Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add ilm action detail metrics

Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add delivery target server metrics

Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication target flow metrics

Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add request server metrics

Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): apply rustfmt to metrics changes

Apply rustfmt output to the metrics dimension changes without altering behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): reuse audit target label constant

Use the exported audit target_id label constant for legacy audit target metrics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): populate drive disk metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner bucket drive result metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication proxy server metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric liveness review

Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address additional metric review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): count drive calls at start

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metrics dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address scanner review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): reduce disk metric contention

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): retire stale dimension series

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 09:03:34 +08:00
cxymds 988cd8adbb fix(ci): keep PR e2e smoke lane from timing out (#5649)
fix(ci): prevent e2e smoke lane timeout
2026-08-03 06:13:46 +08:00
Zhengchao An 9dd0461f3e test(kms): exercise real Vault Raft failover (#5653) 2026-08-03 05:25:23 +08:00
Zhengchao An fbb6cebeb4 feat(kms): bound backend concurrency and failures (#5651) 2026-08-02 18:24:26 +00:00
cxymds 2ce670837c fix(ecstore): make transitioned deletes durable (#5644)
* fix(ecstore): make transitioned deletes durable

* fix(ecstore): journal force deletes

* fix(ecstore): journal force deletes
2026-08-02 18:00:11 +00:00
Zhengchao An 8a65017f36 fix(kms): bound persisted format parsing (#5652)
fix(kms): harden persisted format compatibility
2026-08-02 17:53:01 +00:00
Zhengchao An 3a5b6eb11d test(kms): verify AppRole against live Vault (#5650)
* test(kms): add ignored Vault AppRole live harness

* test(kms): tighten Vault AppRole live contract
2026-08-02 17:25:49 +00:00
Zhengchao An 0800f74874 fix(kms): version local key records safely (#5638) 2026-08-02 16:30:23 +00:00
cxymds a918f1a48a fix(replication): snapshot existing object admission targets (#5634) 2026-08-03 00:13:32 +08:00
cxymds ec67884f8d fix(replication): preserve durable MRF delete admission (#5643) 2026-08-02 23:53:53 +08:00
Henry Guo e5cfa8e375 feat(table-catalog): paginate Iceberg REST listings (#5466)
* feat(table-catalog): paginate Iceberg REST listings

* test(table-catalog): remove redundant token clones

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-02 15:27:15 +00:00
cxymds 1fdcbd9225 fix(replication): fail closed on destination encryption (#5633)
* fix(replication): fail closed on destination encryption

* test(replication): avoid Debug bound in encryption assertion
2026-08-02 22:54:51 +08:00
cxymds 114b2420a2 feat(admin): expose versioned replication capabilities (#5631)
* feat(admin): expose replication capabilities

* fix(admin): route replication capabilities through facades
2026-08-02 22:54:38 +08:00
Zhengchao An 52c70738eb fix(kms): cover concurrent Vault KV2 rotation races (#5632)
* fix(kms): handle concurrent Vault KV2 baseline races

* test(kms): keep concurrent rotation regression fail-closed
2026-08-02 22:53:40 +08:00
Zhengchao An 60ee86c835 test(kms): pin AWS timeout and contract divergence (#5636) 2026-08-02 22:52:54 +08:00
Zhengchao An 5206c82423 ci: require MinIO interop reader matrix (#5640) 2026-08-02 22:34:19 +08:00
cxymds 00324e6936 test(e2e): add replication acceptance matrix (#5642) 2026-08-02 22:34:00 +08:00
GatewayJ 6028dad2f4 test(iam): freeze OIDC federation behavior (#5627) 2026-08-02 22:33:20 +08:00
Zhengchao An 54d8c02a2f fix(targets): explain webhook outbound allowlist failures (#5616) 2026-08-02 22:33:05 +08:00
唐小鸭 3f716746cf fix(replication): honor target TLS in health checks (#5613) 2026-08-02 22:32:56 +08:00
cxymds 779b5a49ea fix(replication): propagate metadata changes (#5635)
Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.

Refs #1616
2026-08-02 13:50:24 +00:00
cxymds 2cc7443067 feat(replication): bound DeleteObjects queue admission (#5637)
feat(replication): batch DeleteObjects queue admission
2026-08-02 21:06:54 +08:00
cxymds 378c9ba67f fix(replication): enforce bucket write contract (#5629) 2026-08-02 11:51:47 +00:00
Zhengchao An 4473c548be test(admin): audit KMS deletion guard outcomes (#5628) 2026-08-02 11:39:37 +00:00
cxymds ac63808d3c fix(replication): make sync delivery target-granular (#5630) 2026-08-02 11:31:13 +00:00
cxymds 2cdba03dee fix(authz): gate replication-only PUT headers (#5625) 2026-08-02 19:28:54 +08:00
cxymds 885096d1de fix(replication): reject unsupported target options (#5622) 2026-08-02 19:28:30 +08:00
cxymds 921ddef2c7 fix(lifecycle): bind delete replication admission (#5621) 2026-08-02 19:27:39 +08:00
Zhengchao An f1a4588326 docs(kms): record CLI and console admin handoff matrix (#5639)
docs(kms): record client admin API handoff matrix
2026-08-02 19:27:19 +08:00
Zhengchao An 2698a03582 test(kms): pin admin KMS response shapes where they are served (#5626)
The snapshots in crates/kms/src/api_types.rs pinned DeleteKeyResponse,
ListKeysResponse, DescribeKeyResponse and CancelKeyDeletionResponse, none
of which is serialized by any handler: those endpoints answer with
DeleteKmsKeyResponse and siblings in rustfs/src/admin/handlers/kms_keys.rs,
separate types carrying different fields. A breaking change to an admin
response could not fail them. Tag, untag and update-description had the
same gap, where the handler discards the kms-side response and serves its
own KmsKeyMetadataResponse.

Pin the shapes in the crate that produces them, and delete the four kms
mirrors. They were never in the pub use api_types list, had no
constructors and no callers, and only looked live because those snapshots
named them.

Keep the api_types snapshots that pin something real: configure, start,
stop and status are served verbatim by kms_dynamic, and the tag family
are live ObjectEncryptionService return types whose snapshots pin this
crate's public API rather than a wire shape.
2026-08-02 11:20:17 +00:00
Zhengchao An b1ddda3bb2 fix(sse): rewrite data when a same-key copy changes encryption (#5618)
A same-name CopyObject marks the operation `metadata_only`, which lets the
store layer rewrite `xl.meta` in place and leave the data blocks untouched.
The handler independently strips the source encryption metadata and calls
`sse_encryption`, which mints a *fresh* DEK. On an unversioned bucket both
happen at once, so the object ends up with a new DEK sitting beside ciphertext
sealed under the old one, and can never be decrypted again.

The mirror case is silent: an encrypted source copied without any destination
SSE keeps its ciphertext while losing the key metadata, so GET returns raw
ciphertext as if it were plaintext, with HTTP 200 and no error anywhere.

Keep `metadata_only` off whenever either side of the copy is encrypted, so the
store layer performs a full read/write rewrite through `put_object`. This is
the same resolution the versioned historical-restore path already uses for
this risk (issue #4238), and it matches MinIO's
`isSourceEncrypted || isTargetEncrypted -> metadataOnly = false` guard in
CopyObjectHandler.

The target half of the predicate deliberately tests `effective_sse` rather
than the request headers MinIO inspects: `effective_sse` also resolves the
bucket default-encryption rule, and `sse_encryption` mints a DEK from that
resolved value. A header-only check would miss a same-key copy performed under
a bucket default rule. The source half reuses `ObjectInfo::is_encrypted` so a
future encryption flavour is covered here as soon as it is recognised there.

Versioned buckets were already safe: that path falls through to `put_object`
regardless of `metadata_only`. RestoreObject also sets `metadata_only` but
only appends restore keys and never re-derives a DEK, so it is unaffected.
2026-08-02 11:00:52 +00:00
Zhengchao An da531c8a97 docs(kms): guard outward FIPS wording (#5624) 2026-08-02 18:50:54 +08:00
Zhengchao An 40cd10c1d0 fix(scanner): surface per-tier usage in the data-usage snapshot (#5623)
SizeSummary::tier_stats was populated for every scanned object but
apply_scanner_size_summary dropped it, so per-tier usage never reached
DataUsageInfo. Wire it through the same merge chain repl_target_stats
already uses, up to DataUsageInfo::tier_stats.

DataUsageEntry used the derived MessagePack encoding, which serialises
structs as arrays: appending a field turns the whole cache into a decode
error for older readers, so mixed-version nodes would invalidate each
other's cache every scan cycle. Give it the same hand-written
map-encoded Serialize DataUsageCacheInfo already carries, and record the
invariant in AGENTS.md.

Widen TierStats counters from i32 to u64 so a tier past 2^31 versions
cannot make checked_merge reject an entire usage snapshot, and drop the
duplicate TierStats/AllTierStats definitions in the scanner crate in
favour of the data-usage ones.
2026-08-02 10:46:36 +00:00
Zhengchao An f440a14e53 fix(admin): preflight webhook outbound policy (#5619) 2026-08-02 18:31:52 +08:00
Zhengchao An c104ba23d4 chore(kms): remove dead key-management types from api_types (#5620) 2026-08-02 10:09:37 +00:00
cxymds 7463655ae1 fix(authz): preserve direct delete usecase callers (#5614)
* fix(authz): preserve direct delete usecase callers

* test(ilm): provide owner context for forced deletes
2026-08-02 17:01:32 +08:00
Zhengchao An 93b38b3fbd fix(lifecycle): count the first transition sample for each tier (#5612) 2026-08-02 15:35:12 +08:00
cxymds f00f5fe776 fix(authz): restrict recursive force-delete scope (#5610) 2026-08-02 06:30:36 +00:00
Henry Guo 6eddc04b0a fix(scanner): bound corrupt metadata traversal (#5609) 2026-08-02 06:07:06 +00:00
houseme 3d59c90371 chore(deps): update flake.lock (#5611) 2026-08-02 14:05:02 +08:00
Zhengchao An 2bf2ce1ff2 feat(admin): apply the requested key listing filters (#5608) 2026-08-02 14:03:57 +08:00
Zhengchao An bd834297da feat(kms): add the data key rewrap primitive (#5607)
* feat(kms): add data key rewrap and wrapping inspection primitives

Rewrap re-protects an existing data key envelope with the master key's
current version without touching the data key itself, which is the
precondition for ever retiring an older version: until every envelope a
version wrapped has been moved off it, destroying that version orphans
every object whose data key it wrapped.

Adds KmsBackend::rewrap_data_key and its read-only counterpart
describe_data_key_wrapping, both gated by a new BackendCapabilities::rewrap
flag and defaulting to UnsupportedCapability. Vault KV2 unwraps with the
frozen version record that wrapped the envelope and re-wraps with the
current material; Vault Transit uses the native transit/rewrap endpoint so
the data key never enters this process.

No read or write path changes: nothing calls these yet.

* test(kms): cover the rewrap primitive against a scripted Vault

* fix(kms): resolve both key materials before the data key is unwrapped

Keeps every fallible step out of the window in which the plaintext data
key exists, so no error path can drop it without zeroizing it first.
2026-08-02 05:51:12 +00:00
Zhengchao An c147afd19c fix(kms): fail closed on Local key records that cannot be interpreted (#5606)
* fix(kms): fail closed on Local key records this build cannot interpret

The Local backend's protection marker is the only version discriminator its
key records have, and three readers walked past it.

`ensure_missing_salt_can_be_generated` skipped every record it could not read
or parse, so a directory whose protection state is unknown still got a fresh
salt published before startup validation failed. That write is the
irreversible step: the next startup finds a salt file, never re-enters the
guard, and the evidence that the real salt was lost is gone. Every record the
guard now rejects already failed startup key validation a few lines later, so
no directory that initializes today stops initializing.

Backup export and restore folded an unknown marker into "material corrupt" /
"bundle corrupted". The record is intact and a newer build reads it fine, so
the operator response is a version change, not a disaster recovery. Both now
classify the marker before their schema parse, sharing one probe with the
backend reader.

`list_keys` dropped any record it could not decode from the page. Concurrent
removal stays a skip; anything else fails the listing rather than answering
"these are your keys" with a set that silently omits one.

* test(kms): cover every fail-closed path around the Local protection marker

Each test fails on the pre-fix code in the way the fix is about: the salt
cases because a replacement salt is published before startup validation
fails, the export and restore cases because the verdict comes back as
corruption, and the listing case because the record is edited out of the page.

The restore commit marker's unknown-version branch had no test at all,
unlike its Vault counterpart; it is now driven from the decoder, from the
restore entry point, and from backend startup.

Also states the widened salt guard in the Local backend operations doc,
including the operator recovery path for an unrecognized record.

* fix(kms): say 'not a readable JSON object' when the marker probe cannot parse

The probe now fails on any input that is not a JSON object, not only on
malformed JSON, so the message must cover both.

* test(kms): assert the salt file before the error variant

The replacement salt is written before the error the guard reports, so the
file assertion is the one that fails on a regression.

* test(kms): guard the new backup error variant's display string
2026-08-02 05:43:38 +00:00