Record local read_version path resolution, path length check, xl.meta read, and metadata decode durations through the existing GET stage metrics channel. The new samples are gated by GET stage metrics so metrics-off reads avoid timer and recorder work.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* fix(kms): construct wrap_budget_reserved in the VaultKeyData deserializer
main does not compile: #6019 added VaultKeyData.wrap_budget_reserved on a base that predated #6003's hand-written Deserialize, so the visitor's struct literal never learned about the field. Each PR was green on its own base; the breakage only exists in their merge.
The field joins the other three lists the hand-written impl maintains (Field enum, match arm, struct literal, FIELDS) and defaults to 0 when absent — the value a record written before wrap accounting, or rewritten by an older build, carries; zero restarts the reservation rather than blocking a wrap.
vault_key_data_deserializer_covers_every_serialized_field turns this class of mistake into a test failure instead of a merge-order accident: it serializes a fully populated record and asserts the deserializer recognizes every emitted key (unknown-field counter stays zero) and reads every value back. Mutation-verified by dropping the new match arm.
* feat(kms): report a key as due for rotation once its wrap budget is spent
The rotation readiness verdict only knew about age; the wrap accounting landed by #6019 counted wraps and published an aggregate gauge but never fed the per-key verdict, leaving the criterion backlog#1636 asks for unimplemented.
RUSTFS_KMS_ROTATION_MAX_WRAPS adds the second, independent threshold, parsed with the same discipline as the age one: unset or unparsable leaves the verdict unreported rather than inventing a policy, and values below one million are raised to it because wraps are reserved in blocks of that size and a smaller threshold would trip on the first reservation regardless of how many wraps happened.
The wrap check runs before the age check so that a key crossing both reports 'wraps': the AES-GCM random-nonce ceiling is a cryptographic bound an operator cannot negotiate, while the age period is a policy they chose. Backends that report no count — Transit and AWS wrap externally, and pre-accounting records carry nothing — leave the wrap half silent instead of guessing, and a backend that cannot rotate is still never told to.
Refs rustfs/backlog#1636 (PR-3 acceptance criterion), rustfs/backlog#1562.
---------
Co-authored-by: houseme <housemecn@gmail.com>
Add a default-off inline-only data-read metadata early-stop gate that verifies inline plaintext before cancelling pending metadata tasks.
Keep non-inline, prepared, and request-shape-sensitive reads on full fanout, and record scheduled/completed/cancelled ReadVersion lifecycle metrics for normal fanout completion.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep S3 CopyObject's real outer owner task alive across caller cancellation so the source/destination bucket guards, same-key copy guard, storage commit, and post-commit publication hooks complete as one request-owned transaction boundary.
Co-authored-by: heihutu <heihutu@gmail.com>
The two table_catalog test files (27.5K lines combined) each maintained a parallel constructor stack for Iceberg metadata JSON and avro manifest-list/manifest bytes. Per the issue's adversarial ruling the parameterized admin variants are canonical (the store file hardcoded sequence 7 / snapshot 20); the two stacks were verified structurally identical first — schemas byte-equal, field lists and values aligned.
New #[cfg(test)] table_catalog/test_support.rs owns the seven constructors (metadata JSON, three manifest-list variants, two manifest variants, nullable_long). The admin tests import them under their old names; the store tests keep their historical signatures as thin delegates passing the fixed values explicitly — every produced byte is identical to the pre-extraction fixtures (the delegate's argument order was cross-checked against the canonical destructuring after an initial swap surfaced as five sequence-bound validation failures).
Ref rustfs/backlog#1837 (PR1).
Use BytesMut spare capacity for direct and pooled small PUT body reads while preserving exact-length validation.
Co-authored-by: heihutu <heihutu@gmail.com>
RemoteAddr, the DependencyReadiness family (DependencyReadiness, ReadinessDegradedReason, DependencyReadinessReport), and convert_ecstore_object_info (with its offset_date_time_to_timestamp helper) are consumed across app, infra, and interface layers but lived under server, so every lower-layer import was an upward app->interface or infra->interface edge the layer guard had to baseline.
They now live in a new layer-neutral rustfs/src/shared_types.rs (classified infra by the guard, making all consumer imports downward or lateral). server::readiness and server::event re-export for their own internals; the eight consumer sites (admin_usecase, bucket_usecase, object_usecase, cluster_snapshot, storage/access, storage/helper, plus the admin handler tests) import from the new home. Pure move: no type, impl, or behavior change.
The regenerated layer-dependency baseline shrinks by exactly eight lines with zero additions — the ratchet's intended direction. The two remaining readiness entries (collect/snapshot fn imports) need the collection machinery itself extracted from server and are left for the issue's PR5 scope.
Ref rustfs/backlog#1834 (PR4).
The 14 tests carried #[ignore = "requires isolated global object layer state"], and the only CI lane that runs ignored tests filters for lifecycle tests — so they executed nowhere. Under nextest, the authoritative runner, every test owns its process and the stale reason no longer applies; all 14 pass.
Ten of them assert the InternalError path taken while the global object layer is uninitialized, a premise a sibling test can destroy under the documented shared-process cargo test fallback. Those ten now start with an explicit premise guard: when a sibling already initialized the store the test skips with a message instead of asserting against a scenario it does not describe. Under nextest the guard never fires and the assertions always run.
Dual-runner evidence: nextest 79 passed; cargo test module-scoped 79 passed; the full storage-tree cargo test sweep returns to its pre-existing baseline (8 unrelated in-process failures, none introduced or worsened here). No test deleted.
Ref rustfs/backlog#1830 (PR1).
* perf(metrics): attribute PUT stage costs
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move PUT metadata during shuffle
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(s3): reuse PUT object lock state
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): trim PUT metadata fanout clones
Build per-disk PUT metadata only for committed writer slots, move the response metadata out of the fanout vector, and preserve fresh FileInfo shuffle semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): make PUT stage attribution opt-in
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): commit inline PUT shards directly
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): streamline rename staging cleanup
Use the directory-specific removal operation for rename_data staging parents. This avoids a guaranteed failed file-removal probe on Unix-like hosts and lets Windows remove the empty directory directly while preserving best-effort non-empty handling.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover inline PUT rename failures
Cache the detailed stage metrics gate once per PUT and exercise exact-quorum and quorum-minus-one failures after inline shard encoding.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Keep encoded shards in one contiguous Bytes buffer while they cross the streaming write queue, and materialize Vec<Bytes> only for the existing public APIs.
Co-authored-by: heihutu <heihutu@gmail.com>
init_background_expiry resolved its worker count through three env vars, none documented, none set in any known deployment: RUSTFS_MAX_EXPIRY_WORKERS, silently overridden by the underscore-prefixed _RUSTFS_ILM_EXPIRATION_WORKERS (a MinIO fossil, comment included), with a zero value then falling through to RUSTFS_DEFAULT_EXPIRY_WORKERS.
RUSTFS_MAX_EXPIRY_WORKERS stays as the canonical name per the rc constraint (no new env var names): the count is now resolved once — a set, parseable, non-zero value wins, anything else falls back to min(cpus, 16). The constant moves to rustfs-config's runtime constants alongside ENV_TRANSITION_WORKERS, the two dead names are gone repo-wide (rg-verified), and a serial four-state unit test (unset/zero/valid/garbage) pins the resolution, modeled on the transition-worker env harness.
Ref rustfs/backlog#1832 (PR2).
PR #6008 deleted crates/io-metrics/src/config.rs but the crate docs still taught the deleted API: both READMEs kept the Unified Configuration sections, module-tree entries and ./src/config.rs links, the example kept an orphaned numbered comment, and the io-core/io-metrics changelog had no removal record. Scrub all of it; keep cache_config.rs references, which are still real.
* test(e2e): fold seven identical POST-policy exact-mismatch tests into one table
The seven *_policy_mismatch tests in multipart_auth_test.rs were body-identical after literal normalization: the policy pins one field to an exact value, the form sends a different value, and the upload must be rejected with 400 InvalidPolicyDocument naming the field. Each test booted its own full server.
They fold into one table-driven test on the run_post_object_policy_case helper introduced by the PR1 fold. Every row keeps its original test's exact bucket, key, field name, policy value, mismatched form value, body bytes, and expected error strings — including the three rows that asserted the stronger <Code>InvalidPolicyDocument</Code> form. The pinned condition is built with an explicit serde_json::Map since the field name is now a table parameter.
cargo nextest list reports 97 tests for this module; the inventory row is updated in the same diff (103 -> 97).
Ref rustfs/backlog#1838 (PR2).
* test(e2e): fold the remaining POST-policy duplicate groups (15 tests) (#6018)
Completes the multipart_auth table-driven fold: the six remaining body-identical groups collapse onto the shared run_post_object_policy_case helper.
- Seven single-field exact-mismatch tests (cache-control, expires, tagging, storage-class, content-type, success_action_status, metadata-field-exact) join the existing exact-condition mismatch table as rows — same shape as the PR2 fold.
- The two object-lock mismatch tests become a two-row table (policy pins mode + retain-until-date, one form field mismatches).
- The three SSE-KMS parameter mismatch tests become a three-row table (policy pins the SSE mode plus one KMS parameter, form differs).
- The three SSE-KMS outside-policy tests become a three-row table pinning the distinct contract: an undeclared KMS parameter sails past policy validation and is rejected at runtime with 501 NotImplemented, not a policy error.
Every row keeps its original test's exact bucket, key, field names, values, body bytes, and expected status/code strings. cargo nextest list reports 85 tests for the module; the inventory row is updated in the same diff (97 -> 85).
Ref rustfs/backlog#1838 (PR3).
Zhengchao An