Commit Graph

4797 Commits

Author SHA1 Message Date
Zhengchao An c5eb1c69ba test(rpc): cross-process replay/tamper acceptance for internode RPC signatures (#5236) 2026-07-26 04:04:16 +08:00
houseme 77b5e1b64c fix(ilm): report manual transition tier failures (#5238)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 18:02:37 +00:00
houseme 23a5db4012 fix(ilm): preserve lifecycle version groups (#5239)
* fix(ilm): evaluate complete version groups

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): log replication expiry blocks

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): diagnose incomplete noncurrent chains

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover purge-pending version groups

Add regression coverage for lifecycle-only version listing so purge-pending versions stay present for ILM evaluation while the public ListObjectVersions projection remains filtered.

Refs rustfs/backlog#1500

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): log lifecycle evaluation failures

Emit structured lifecycle_evaluation_failed events when version group loading or evaluation fails during immediate and existing-object expiry scans.

Refs rustfs/backlog#1503

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover incomplete noncurrent chains

Pin the fail-closed behavior for noncurrent expiration when successor_mod_time is missing and reuse a stable structured event name for that skip path.

Refs rustfs/backlog#1502

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ilm): cover one-day noncurrent expiry boundary

Add a runtime regression test proving NoncurrentVersionExpiration Days=1 stays inactive before expected_expiry_time and deletes exactly at the computed due boundary.

Refs rustfs/backlog#1504

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): keep transition checks after incomplete expiry

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 17:42:02 +00:00
houseme 516f7fecc1 fix(tier): converge config after peer recovery (#5240)
Retry peer tier config reloads after committed mutations so recovered nodes converge without requiring a second admin change.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 17:25:37 +00:00
houseme 1e95e6d311 fix(ilm): harden durable transition admission (#5235)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 15:46:55 +00:00
houseme 3cbe3d6b94 fix(tier): reconcile committed mutation replay (#5230)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 15:20:54 +00:00
Zhengchao An 61d4e04d65 feat(rpc): bind canonical body digest into internode mutating disk RPC signatures (#5234)
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures

Binds a domain-separated, length-prefixed canonical request-body digest into the
v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path
attacker on the default-plaintext internode channel can no longer tamper with a
mutation payload (or strip the msgpack `_bin` field to force the JSON fallback
decode) without invalidating the signature.

Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions,
WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile,
RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the
msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by
default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so
rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now
configurable and overflow fails closed with a metric.

Refs https://github.com/rustfs/backlog/issues/1327

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(rpc): satisfy architecture-migration compat-marker guard

Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and
stop backticking env-var/metric names in the cleanup-register entry so the
guard's id extractor only sees the task-id.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:56:27 +08:00
houseme 6974963e20 feat(ilm): add durable manual transition job store (#5229)
* feat(ilm): add manual transition job route contract

Refs #1479

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ilm): add durable manual transition job store

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ilm): harden durable transition job cancellation

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:54:14 +00:00
houseme 7ab0955f8b test(e2e): stabilize tier and storage class checks (#5228)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:21:52 +00:00
houseme 2cf5fd6bfc fix(getobject): avoid duplicate downstream-close metrics (#5231)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:13:47 +00:00
houseme de2c337fae test(e2e): cover async transition limit partial (#5233)
Cover async manual transition jobs that finish as partial because maxObjects truncates the scan, including terminal cancel idempotence and unchanged report counters.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 14:05:47 +00:00
houseme 5988606e68 test(internode): extend fallback and transition coverage (#5232)
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 13:26:23 +00:00
Hiroaki KAWAI e73ed4f2a1 fix(lock): jitter distributed lock retries (#5113)
* fix(lock): jitter distributed lock retries

* fix(lock): keep retry jitter within bounds

* fix(lock): qualify test size_of usage

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 20:26:37 +08:00
houseme ffde6c43ee feat(ilm): add manual transition job route contract (#5221)
Refs #1479

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 12:22:46 +00:00
cxymds f52dde87d1 fix(object): make version undo preconditions atomic (#5225)
* fix(object): make version undo preconditions atomic

* fix(object): fence metadata-only undo copies

* fix(object): order versions by commit time
2026-07-25 19:26:50 +08:00
houseme 8e83087ba4 fix(tier): handle mutation intent CAS races (#5227)
* test(scripts): expand internode grpc ab env coverage

* fix(tier): handle mutation intent CAS races

Make tier mutation intent advancement retry idempotently when an If-Match CAS loses to a matching committed peer update.

Expand four-node inline fallback E2E coverage for mixed msgpack controls, transition readiness evidence, and per-node environment overrides.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 11:10:13 +00:00
Henry Guo a63b79004c fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative

* fix(scanner): close distributed refresh races

* fix(config): align scanner reload integration

* fix(admin): scope config test helpers

* fix(scanner): harden distributed usage convergence

* fix(scanner): preserve rolling activity compatibility

* fix(admin): expose non-secret optional config values

* fix(scanner): acknowledge distributed dirty usage

* fix(ecstore): make bucket mutations cancellation safe

* fix(scanner): preserve pending dirty acknowledgements

* test(obs): account for superseded scanner metric

* fix(api): reject excess detached bucket mutations

* test: close scanner convergence coverage gaps

* fix(scanner): make path tracking cleanup one-shot

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 18:45:16 +08:00
GatewayJ 0364523dad fix(s3select): enforce query and resource limits (#5028)
* fix(s3select): enforce query and resource limits

* fix(s3select): close query resource limit gaps

* fix(s3select): preserve timeout and stream invariants

* fix(s3select): enforce staged query limits

* fix(s3select): preserve policy error compatibility

* fix(s3select): bound error source traversal
2026-07-25 18:44:53 +08:00
houseme 2dc4d0b651 refactor(getobject): scope downstream close tagging (#5224)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 09:36:19 +00:00
houseme 2ee111ad8b feat(ilm): add durable manual transition jobs (#5223)
Refs #1479

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 09:12:44 +00:00
houseme 9ddb30139d fix(getobject): distinguish downstream output closure (#5220)
fix(getobject): preserve internal broken pipe failures

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 08:22:20 +00:00
Zhengchao An ffd1b94e1f feat(rpc): add server-side internode v2 signature verification (fail-open, method-path binding groundwork) (#5160)
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env

Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before.

Refs https://github.com/rustfs/backlog/issues/1327

* ci: fix internode auth test lint failures

* perf(ecstore): cache internode sig strict env

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 08:17:49 +00:00
houseme ce41adfa9b test(ilm): cover manual transition e2e gaps (#5219)
* test(ilm): cover manual transition queue pressure

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(protos): serialize msgpack-only env tests

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-25 07:00:24 +00:00
cxymds 59361ed786 feat(site-replication): make repair operations durable (#5217)
* feat(site-replication): make repair operations durable

* fix(site-replication): serialize durable repair execution
2026-07-25 10:35:37 +08:00
houseme dfb0a20048 test(rpc): add corrupt-json decode coverage (#5216)
* test(rpc): add corrupt-json decode coverage

* test(ilm): assert manual transition status-less contract
2026-07-25 01:03:59 +00:00
cxymds 7320d7fab2 fix(replication): make resync starts atomic (#5215) 2026-07-25 08:58:06 +08:00
Zhengchao An 28d19db9fc fix(console): handle apple touch icon paths (#5214) 2026-07-25 08:06:48 +08:00
dependabot[bot] e257573962 chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 in /fuzz in the cargo group across 1 directory (#5210)
chore(deps): bump quinn-proto

Bumps the cargo group with 1 update in the /fuzz directory: [quinn-proto](https://github.com/quinn-rs/quinn).


Updates `quinn-proto` from 0.11.14 to 0.11.16
- [Release notes](https://github.com/quinn-rs/quinn/releases)
- [Commits](https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.14...quinn-proto-0.11.16)

---
updated-dependencies:
- dependency-name: quinn-proto
  dependency-version: 0.11.16
  dependency-type: indirect
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-25 08:05:02 +08:00
cxymds 45b675c641 fix(replication): report authoritative backlog metrics (#5209) 2026-07-25 01:32:50 +08:00
cxymds 05caec0bd5 feat(replication): structure active check results (#5208) 2026-07-25 01:28:03 +08:00
cxymds eaa17e0441 feat(admin): add encrypted diagnostic archives (#5207) 2026-07-25 01:19:47 +08:00
cxymds 2b6cc0ee08 fix(admin): redact and seal config history (#5206) 2026-07-24 23:54:24 +08:00
cxymds 938f7296f9 fix(admin): report truthful diagnostic capabilities (#5205) 2026-07-24 23:33:45 +08:00
cxymds 866ac5073d fix(kms): validate configured backend cleanup (#5204)
test(kms): validate configured backend cleanup
2026-07-24 23:24:40 +08:00
houseme 187a060919 test(internode): pin msgpack gate rollback coverage (#5203)
Add test coverage for the msgpack-only request/fleet confirmation truth table and exact request JSON policy manifest.

Pin request and response rollback behavior so removing either gate restores JSON compatibility while both gates enter msgpack-only for eligible fields.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 15:16:23 +00:00
houseme cda443bd81 feat(ilm): reject overlapping manual transition runs (#5202)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 14:56:42 +00:00
houseme 44b1916103 fix(getobject): enrich stream error context (#5201)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 14:47:56 +00:00
Henry Guo 9d1b10144f perf(ecstore): avoid redundant leaf object scans (#5176)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-24 22:25:03 +08:00
cxymds d7f30fe0a2 fix(replication): persist sanitized resync errors (#5200) 2026-07-24 22:15:51 +08:00
houseme 20c4ea864a fix(ecstore): fix wide-prefix listing stalls and decode downstream logging (#5198)
* fix(ecstore): handle list stall and downstream decode logs

* docs(ecstore): keep wide-directory stall context for list_dir
2026-07-24 14:05:27 +00:00
houseme d1c2c42c90 fix(internode): align msgpack benchmark gates (#5197)
Update the internode gRPC benchmark P2 env output to require both the msgpack-only request flag and the fleet-confirmed gate before measuring a true msgpack-only phase.

Document the dual-gate rollback semantics and add a dry-run script test so the benchmark driver cannot silently regress to the single-flag flow.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 13:32:44 +00:00
cxymds fc43b149c5 fix(admin): restore complete config history snapshots (#5196) 2026-07-24 21:28:12 +08:00
cxymds fa235e9018 fix(s3): list multipart uploads by bucket prefix (#5195)
* fix(s3): list multipart uploads by bucket prefix

* fix(s3): preserve exact-key crash reclamation
2026-07-24 21:28:06 +08:00
houseme dd46de0945 fix(heal): report no-parity bitrot as unrecoverable (#5192)
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 20:20:40 +08:00
houseme bf6f0e5e81 test(internode): pin msgpack compatibility send sites (#5193)
test(internode): pin msgpack compat send sites

Add a checked test manifest for internode dual-encoded request and response payloads so node.proto _bin fields must be explicitly classified as msgpack-only eligible or always dual-write.

The manifest also pins the current JSON encoder call sites for request and response send paths, making future send-site drift visible before rollout.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 12:07:11 +00:00
houseme beb807ae2b perf(protos): cache internode msgpack-only flag (#5191)
Cache the internode msgpack-only env gate after the first read so metadata RPC send paths avoid repeated environment parsing. Keep a reset hook for tests that intentionally change the env in-process.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:54:14 +00:00
houseme d8426dc459 feat(ilm): gate durable manual transition mode (#5190)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:53:19 +00:00
cxymds f46ea6e14f fix(s3): enforce SSE-C copy key validation (#5185) 2026-07-24 19:11:36 +08:00
houseme fa26b6730d test(ilm): deny manual transition for non-admin (#5188)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:10:31 +00:00
houseme 7876319811 test(ilm): cover manual transition queue pressure (#5189)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-24 11:10:04 +00:00