277 Commits

Author	SHA1	Message	Date
Zhengchao An	2f5db4e8f2	chore(release): merge release into main for rc.6 (#7638) ... * test(heal): cover MRF manifest CAS legacy transition (#7443) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): enable release branch checks and repair test imports (#7441) * test(scanner): structure heal release evidence lanes (#7442) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): require segment producer identities (#7444) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): expose recovery intent identity (#7445) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): summarize failed heal perf reports (#7446) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(common): retain unmatched MRF repair proofs (#7447) Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): route replication read plan through boundary (#7448) * test(scanner): cover flat-bucket quantum fairness (#7449) Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): reuse same admission request id (#7451) Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require perf summary evidence fields (#7453) Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind release evidence fields (#7452) Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): tag dirty usage producer identities (#7454) Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): gate release evidence bundles (#7457) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require hard evidence ABBA manifest (#7459) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(heal): add MRF committed snapshot writer (#7458) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover EC8+4 restart shard rebuild (#7461) Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): echo measured release evidence in ABBA fake adapter (#7460) Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require complete ABBA summary matrix (#7462) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: retain MRF replay journal after accepted/merged replays Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: reclaim superseded MRF snapshots after readback Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: require exact owner for object receipts Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: latch object receipt owner before repair Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: replay committed MRF checkpoints durably (#7465) Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind hard evidence bundle provenance (#7467) Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass. Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: reject dry-run object repair receipts Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: reject cancelled object repair receipts Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): reject failed object repair receipts Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): cover bucket object repair receipts (#7468) Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation. Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add EC8+4 heal restart evidence (#7469) Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology. Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: verify replacement pool metadata repair (#7471) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover quorum and mixed repair receipts Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: verify admin recreate pool metadata (#7474) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: preserve merged result for duplicate submits Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): preserve EC84 restart semantics (#7478) Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF snapshot torn successor recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): write EC84 distributed restart oracle (#7481) Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test. Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): harden MRF replay boundaries (#7483) Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records. Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bind MRF manifest CAS dirsync recovery (#7482) Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): derive evidence runner profile from registry (#7484) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require mixed-version evidence roles (#7485) Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role. Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval. The release gate remains blocked until measured field evidence is present. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): publish committed MRF runtime checkpoints (#7490) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind profile artifacts in release evidence (#7487) Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks. Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): harden measured ABBA evidence claims (#7491) Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash. Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values. Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(e2e): group scanner heal evidence payload (#7494) Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint. The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF replay bundle fields (#7486) Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): profile EC84 evidence case runs (#7495) Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions. Expose the runtime profile in plan-only output and cover every registry case in the script self-test. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): reject empty release evidence artifacts (#7496) Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them. Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require G14 same-window field coverage (#7489) Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window. Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require two-hour measured ABBA windows (#7493) Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF idle checkpoint cleanup (#7497) Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(e2e): ignore untracked files in build identity (#7498) Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty. This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF disk-full evidence fields (#7499) Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind release JSON artifact provenance (#7500) Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF rollback mirror filtering (#7501) Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require segment activation evidence (#7504) * test(scanner): require segment activation evidence Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): cover activation proof inputs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF crash matrix cases (#7505) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require release bundle domain evidence (#7502) Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF retention GC evidence (#7506) Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF cleanup GC soak evidence (#7507) Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF disk-full commit anchors (#7509) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): report port allocator bind errors (#7510) Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511) * fix(sse): resolve bucket default encryption per request PUT and the POST-object/extract path resolved a bucket's default encryption with a hard-coded "no explicit SSE-C" flag, so the default was layered onto a request that already carried an SSE-C header triple and then tripped that request's own mutual-exclusion check. Every bucket with default encryption refused SSE-C single PUTs with 400 InvalidArgument, while CreateMultipartUpload on the same bucket succeeded because it resolves SSE elsewhere. Both call sites now derive the flag from the request headers, as COPY already did. The bucket default's KMS key id was also inherited independently of the effective algorithm, so an explicit AES256 request against an aws:kms default bucket produced a self-contradictory algorithm/key-id pair and was rejected. The key id is now inherited only when the effective algorithm is aws:kms, matching the storage-layer resolver. Refs backlog#2368 B1, B2. * fix(sse): refuse SSE-KMS without a running KMS service A write requesting aws:kms on a node with no KMS service fell back to the node-local SSE-S3 provider: the data key was wrapped with RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded aws:kms and the requested KMS key id. The stored object claimed a KMS protection it never had, under a key that was never consulted, and no signal distinguished it from a genuine SSE-KMS object. The managed-encryption path now asks the resolved DEK provider whether it wraps with a node-local master key and refuses SSE-KMS in that case: InvalidRequest when KMS was never configured, ServiceUnavailable when a configured service is not running. The check sits after the per-key authorization gate so an unauthorized caller still receives AccessDenied whatever the KMS runtime state is, and asks the provider rather than a parallel availability signal because the provider is what actually wraps the key. A missing master key no longer answers an SSE-KMS request with an SSE-S3-worded configuration error. The SSE-S3 local fallback is unchanged. Refs backlog#2368 B4. * fix(ecstore): restore and archive tiers in stored coordinates Multipart restore addressed the remote tier in plaintext coordinates while the copy-back reads the stored representation. Each part received a misaligned slice of the remote object whose length still satisfied the range, the hash reader and the completion size check, so the restore reported success and silently replaced the object's bytes. Encrypted and compressed multipart objects were both affected. Restore now accumulates stored part sizes, passes the stored length to the hash reader alongside the plaintext length, and validates against the stored size. The copy-back digests stored bytes, so its computed MD5 is not the object's public ETag. Restore now preserves the object ETag on both the single-part and multipart paths, and gives each restored part its own recorded part ETag rather than the object-level value. Transition also handed the tier the object's SSE headers and its RustFS-wrapped data key as request headers. Any S3 target rejected an SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a second time under a key id it does not own, and the wrapped DEK left the cluster. The archive request now strips every SSE header and encryption marker with the predicate the replication path already uses; the local xl.meta keeps all of it, so read-through and restore are unaffected. Objects restored by an affected release are not detected or repaired retroactively and must be re-restored from the tier. Refs backlog#2368 B3, B5; backlog#2369 P7.1. * fix(rio): lock the v1 nonce layout within a segment Decrypting a v1 segment tried three historical nonce layouts per frame, independently for every frame. The last of them exists for streams written before 1.0.0-alpha.91, which reused a segment's part nonce for every block in it; because block zero's derived nonce equals that base nonce, a frame encrypted at index zero authenticated at any position. An attacker able to rewrite the underlying shards could replay it and have the forged plaintext returned with 200 and an unchanged length. Shard integrity uses a keyed-hash-free checksum, which such an attacker can recompute, so it is not a barrier. A segment now locks onto whichever layout decoded its first non-zero-index frame and rejects any later frame needing a different one. That leaves one residual shape: a stream built purely from repeats of frame zero has no later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK (default true, so pre-alpha.91 objects keep decrypting) drops the third layout entirely when set to false, which closes it. Turning it off refuses pre-alpha.91 objects, so migrate them first by rewriting in place. Refs backlog#2369 P2. * fix(kms): reload a service that failed to start POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted configuration matched the in-memory one byte for byte. A node whose KMS failed to start keeps that configuration and sits in Error, so the documented recovery call returned "reloaded successfully" while leaving the node down. Peers reached the same path through the reload broadcast, so a cluster that lost Vault during a rolling restart had no working recovery route other than the node-local start endpoint. Reload now short-circuits only for a service that is actually running, and otherwise reconfigures, which starts a service that is not running. The AWS backend also advertised key-version enumeration through kms/status, which its own documentation says it cannot do; the capability and its golden snapshot now say false. Refs backlog#2369 P1, P7.3. * docs: record the SSE and KMS changes for 1.0.0 The Unreleased changelog section carried no entry for any encryption work merged since 1.0.0-rc.5, including three items with operational impact: the config-secret variable whose absence persists secrets in cleartext with only a warning, the v2 frame write switch and its rolling-upgrade constraint, and per-key authorization making a public bucket incompatible with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS refusal as a breaking change with both routes out. Also corrects four places where documentation contradicted the code: the cleanup register still called encrypted range seek opt-in after its default flipped, the Helm README claimed vault_mount_path only applies to Transit while the template also feeds the KV2 mount, the disaster-recovery drill listed bundle contents for backends whose export is refused with 501, and the Chinese README capability table predated most of the feature set. Documents the SSE-S3 local master key as a first-class operational mode with its rotation dead end, and what the v1 frame layout does and does not authenticate. Refs backlog#2369 P5. * fix(kms): classify data-path KMS failures by what the caller can do Only "key not found" and a backend outage were classified; every other KMS failure that reached the S3 data path fell through to 500 InternalError with a generic message. A disabled or pending-deletion key, a denied KMS grant, an encryption-context mismatch, an unsupported algorithm, a credential or timeout failure, and a capability the configured backend does not have all looked identical to a server fault. SDKs therefore applied exponential backoff to configuration errors no retry can fix, and monitoring counted every one of them against the server's own error rate. Unusable-key and request-side failures now answer 400, a denied grant 403, transient backend failures 503, and a missing backend capability 501. Damaged, unreadable, or unknown-format key material keeps its 500: it is a server-side integrity fault, and existing tests pin it. The classifier is deliberately separate from the admin lifecycle mapping, which answers 404 for a missing key because there a key id is the resource being addressed; on the data path it arrives inside a request header or a bucket default. Messages either name what the caller asked for or stay generic, with deployment-side detail left on the error source the way the storage-IO mapping already does. Refs backlog#2368 B6. * fix(kms): track and renew static Vault tokens Token authentication hard-coded "this token carries no lease", so the renewal task never started, the remaining-TTL gauge was never published, and nothing looked wrong. `vault token create` grants a 768-hour TTL by default, so a cluster that had been healthy for a month turned every KMS call into a 403 and could not recover without a restart or a reconfigure. Production configuration validation only rejects the literal dev-token, so an ordinary expiring token reaches a whole cluster. The source now reads `auth/token/lookup-self` at login and adopts what Vault reports. A token with no expiry behaves exactly as before. An expiring renewable one is picked up by the existing renewal loop and renewed at half TTL like every other auth method. An expiring non-renewable one warns with its remaining lifetime and publishes the gauge, so the fail-closed window is visible before it arrives. The probe never fails the login: a policy that omits lookup-self, or a Vault that is briefly unreachable, warns and falls back to exactly the previous behaviour rather than taking down a deployment that works today. The scripted Vault test double answers the lookup out of band so existing scripts keep describing only the protocol under test. Refs backlog#2369 P3. * feat(sse): report SSE-C requests that arrive without TLS An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS accepted them on any transport: a plaintext hop hands the key to anyone on the path, and since the object cannot be read without that same key, the exposure lasts as long as the object does. Refusing outright is the correct end state but not a safe default to adopt inside a release window, because the project's own s3-tests and e2e lanes and most staging deployments speak plain HTTP. This release reports instead: each such request increments rustfs_ssec_plaintext_requests_total and logs one warning per process, so an operator can confirm nothing would break before the default flips. RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now. The verdict is per connection rather than per deployment: the layer is built with whether this listener terminated TLS, and additionally accepts an https protocol forwarded by a proxy the trusted-proxy configuration already vetted. It sits beside the rate limiter, after the layer that makes a forwarded protocol trustworthy and after the request context, so a rejection can echo the request id. Refs backlog#2369 P7.2. * fix(kms): say what a node-local backend means for a cluster The Local backend keeps key material on each node's own disk and generates its Argon2id salt per node, so two nodes derive different keys from the same master_key and an object encrypted on one node cannot be decrypted on another. Behind a load balancer that surfaces as intermittent 500s on reads that succeeded moments earlier, with nothing tying the symptom to the cause: the only signal was a generic "development, testing and demos only" positioning warning that says nothing about what actually breaks. Configuring or reconfiguring Local while the deployment is distributed now logs a dedicated event and appends the consequence to the configure response, so the operator who made the change sees it. The product decision to warn rather than refuse is unchanged. Refs backlog#2369 P7.4. * docs: record the remaining SSE and KMS changes for 1.0.0 Adds changelog entries for the KMS data-path status classification, the Vault static-token lease probe, the SSE-C plaintext-transport report and its switch, and the node-local backend warning. Documents two things the backend security guide never stated: that SSE-C belongs on a secure transport, with the counter and switch to plan the change around, and that the Local backend cannot be shared by a multi-node deployment because each node derives different keys from the same master key. Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4. * fix(kms): report an unreadable key store as an outage on the S3 path A backend now distinguishes a key store it could not read from a key that is genuinely absent, but the S3 boundary collapsed the first one back onto 500 InternalError through the fallthrough for integrity faults. The distinction was therefore invisible to the client: a temporary key-directory outage looked exactly like a permanently damaged key record, and neither the status nor the metric said the request was worth retrying. An unreadable key store joins the retryable class and answers 503, next to a backend error and a credential failure. Damaged, unreadable or unknown-format key material keeps its 500. Refs backlog#2368 B6; builds on rustfs/rustfs#7470. * test(storage): cover MRF cleanup delete ENOSPC anchor (#7513) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require heal retry stats evidence (#7514) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add release bundle dry-run fixture (#7515) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): confirm recovery intent accept readback (#7516) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF cleanup delete ENOSPC evidence (#7518) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require bounded retry window evidence (#7517) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): emit scanner heal G09 upgrade evidence (#7519) Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): persist MRF rollback mirror boundary (#7520) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add G09 upgrade evidence runner (#7522) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes. The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): implement V2 evidence preflights (#7523) * feat(scanner): wire dirty usage producer identities Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): add segment activation preflight proof Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): expose distributed invalidation evidence Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds. Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): assemble release evidence bundles (#7526) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add G09 upgrade evidence runner (#7524) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): replay recovery intents while disabled (#7521) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): preflight G09 evidence disk space (#7527) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): preflight G09 evidence disk space Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): respect cargo target dir in G09 runner (#7528) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * fix(scanner): respect cargo target dir in G09 runner Write the RustFS feature stamp under the resolved Cargo target directory so remote validation hosts with CARGO_TARGET_DIR set can reuse the built PR-head binary. Extend the runner plan/self-test path to cover relative target-dir resolution. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): add G09 upgrade evidence runner (#7525) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(e2e): add G09 upgrade evidence runner Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts. Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): emit G09 release bundle gate evidence (#7530) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): emit G09 release bundle gate evidence Write a bundle-ready G09 gate descriptor from the Linux upgrade evidence runner and validate the single G09 gate with the shared release-bundle rules without approving the full Scanner/Heal release. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): reject invalid G09 runner test selection (#7531) Validate the selected G09 evidence lane before process-substitution case expansion so --plan-only cannot turn an unknown --test value into an empty successful plan. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): keep G09 source binary path clean (#7532) Keep checksum verification output out of the command substitution that resolves the previous-release binary for the G09 runner. Also tolerate non-GNU sha256sum in local self-tests by falling back to shasum when GNU --check support is unavailable. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF replay proof cleanup (#7534) Exercise the committed MRF replay checkpoint path with a real ECStore-backed bucket incarnation, retain the replay anchor until an exact verified proof arrives, then delete the proof-discharged checkpoint and prove a subsequent restart does not resurrect the replayed work. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): carry segment activation preflight evidence (#7535) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): surface cold segment reuse oracle (#7536) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require hard evidence artifact payloads (#7537) Require Scanner/Heal release bundle JSON artifacts to carry hard-domain evidence fields for mixed-version, crash, capacity, disk-full, replica-loss, and MRF cleanup gates. This prevents a descriptor from approving a hard gate while pointing at a generic measured artifact that lacks the boundary-specific oracle fields. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): bind producer evidence to dirty generations (#7538) Record scanner segment producer identities with the dirty usage generation they invalidated, then expose a cycle-local producer evidence snapshot for segment reuse activation preflight. Production activation remains fail-closed until durable producer identity and restart-gap proof are available. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): persist segment invalidation proof metadata (#7539) * test(scanner): wait for EC8+4 heal control readiness (#7540) * test(scanner): list full P4 release evidence fields (#7542) Align the Scanner/Heal release requirement registry with the release bundle gate for P4 so the closure checklist advertises MRF scale, replay cost, retained responsibility, and cleanup/GC soak evidence together. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add W16 release evidence runner (#7543) Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors. Wire its shell self-test into script-tests and document the release evidence entry point. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: address release acceptance regressions (#7541) * feat(scanner): admit durable segment proof replay (#7544) * feat(scanner): admit durable segment proof evidence Validate complete set snapshot segment invalidation proof metadata against the current dirty usage generation window and scanner process epoch before clearing durable producer and restart-gap activation blockers. Production segment reuse remains gated by the explicit activation flag. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): replay segment proof from root snapshots Carry segment invalidation proof metadata into root snapshot set states so a complete published baseline can replay the durable producer evidence recorded by each set cache. Keep the field additive for older readers and leave incomplete or LKG set states unproven. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind MRF release artifacts to evidence kinds (#7545) Require Scanner/Heal G07/G08/P4 JSON artifacts to declare the exact MRF evidence kind expected for each release-bundle field. This keeps generic measured JSON from satisfying durable replay, ENOSPC, replica-loss, scale, replay-cost, retained-responsibility, or cleanup-GC soak gates. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): enable segment reuse activation gate (#7546) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): run scoped fallback cases on dedicated stack (#7550) Linux libtest defaults can abort these deep EC-store futures before the assertions run. Wrap the async scoped fallback cases in the repository's dedicated-stack test pattern so the default cargo test path exercises the oracle directly. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): roll out distributed e2e selection (#7549) Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add W13 MRF evidence runner (#7547) Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate. The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): fix segment proof fixtures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): reduce W13 evidence helper arguments Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci(test): enable local KMS for s3 compatibility tests (#7553) Configure the s3-tests harness with a local KMS key so SSE-KMS cases run in CI without relying on an external KMS service. Also move the anonymous POST default SSE-KMS regression onto the shared local KMS test environment. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind segment proof to typed dirty suffix (#7564) Track producer identity completeness per pending bucket and retain process coverage across durable acknowledgements. Admit a prior same-process segment proof only for an exact or strictly earlier fully typed generation window. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): validate scoped ACK clear receipts (#7563) Fail closed when a signed scoped ACK response claims more cleared entries than its request contained, while preserving partial-clear semantics and zero-mutation capability probes. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind cold scoped cache reuse (#7565) Record an explicit cold bucket reuse proof for scoped set scans and treat fully cold sets as valid cold-zero-walk reuse when the baseline and bucket incarnations are bound. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(s3): preserve delete marker read error headers (#7558) * chore(release): sync configurable nightly build workflow from main (#7567) feat(nightly): make the build branch configurable (NIGHTLY_BRANCH var + dispatch input) (#7557) * docs(obs): correct metric export interval and explain alert timing (#7571) * fix(heal): persist scoped admin admission handoff (#7569) * fix(heal): persist root admission handoff Persist administrator root heal ownership before queue publication so crash restart can replay an admitted control-plane start without relying on process-local replay cache state. Protect queued durable root owners from priority displacement; callers must use forceStart cancellation semantics to replace them. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): persist admin heal handoff types Extend durable admin heal admission handoff beyond cluster roots while preserving legacy root recovery records. Persist the heal type in schema v2, replay typed admin requests after restart, keep non-admin requests out of the durable control-plane handoff, and allow path cancellation to clear durable-only non-root records. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): preserve disjoint durable admin owners Limit admin forceStart durable-owner cleanup to overlapping heal requests after the admin recovery journal begins storing non-root work. Add fail-closed and displacement tests so durable bucket owners cannot be silently dropped by unrelated replacement or priority pressure. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): persist admin terminal receipts Persist terminal receipts for durable administrator heal starts before retiring their pending intent so restart can answer completed or cancelled scoped tasks without replaying them. Retain pending work when terminal publication fails and keep timeout owners on checkpoint-only replay semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): mark committed replication and tier mutations (#7568) * fix(scanner): mark committed replication and tier mutations Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): replay durable dirty producer records Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state. Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): recover expansion buckets and resume retirement (#7574) * fix(ecstore): recover partially created expansion bucket volumes * fix(ecstore): resume retirement after startup safety checks * fix(ci): align S3 KMS fixtures and default key expectations (#7573) * fix(heal): prune retained root terminal receipts (#7576) Bound durable root-heal terminal receipt retention to the completed-status TTL while keeping expired receipts authoritative enough to retire stale pending intents before deleting the receipt itself. Add a budgeted background GC pass and fail closed on corrupt or non-terminal receipt records. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(storage): preserve retryable publication guard errors (#7561) * fix(storage): preserve retryable publication guard errors * fix(scanner): satisfy scoped cache clippy lint Use bool::then_some for the scoped cold bucket reuse proof construction after merging the latest release branch. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(health): report storage quorum in node readiness (#7566) * fix: preserve scanner backlog during pool retirement (#7555) * fix: hand off native scanner backlog before pool retirement * test: initialize optional proof in data usage fixtures * perf: bound parallel V3 pool metadata phase writes * test: keep native backlog fault errors typed * test: skip unchanged pool metadata in crash barriers --------- Co-authored-by: cxymds <cxymds@gmail.com> * test(scanner): bind profile artifact provenance (#7577) * fix(scanner): persist dirty producer journal (#7580) Persist scanner dirty producer replay records from committed replication and tier mutations, hydrate journal state on restart, and clear durable records only after acknowledged dirty generations. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test: add scanner heal scheduler pressure evidence (#7584) Add measured G10/P1/P3 scheduler-pressure release descriptor generation, ABBA scheduler/profile provenance, and focused self-tests for the release bundle gate. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test: add scanner heal G14 multiset evidence (#7583) Add EC8+4 multi-set and multi-pool scanner/heal release evidence coverage, including e2e registry cases, oracle checks, and a G14 descriptor assembler for measured case artifacts. Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(nightly): push assets with plain git on release — the build fleet has no gh CLI (#7586) fix(nightly): push assets with plain git — the build fleet has no gh CLI The sm-standard-4 runners used by the nightly build have no gh binary (functional-chain workflows run elsewhere, on the jumpbox). The assets publish step died on 'gh: command not found' at the credential-helper setup, and the preceding clone failure had been masked by 2>/dev/null, misleading the step into the orphan path. Swap clone and remote setup to plain git with the token embedded in the URL; push semantics are unchanged. * fix(scanner): stabilize retired backlog survivor commits (#7585) Treat an authoritative scanner pause-backlog commit as stable when every member in that commit stores the matching stable and committed record. Retired source replicas can remain missing in the observed topology while survivor membership is already stable. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(metrics): prevent duplicate drive reporting across nodes (#7582) Co-authored-by: houseme <housemecn@gmail.com> * fix(ecstore): avoid Windows batch-delete cleanup sharing conflicts (#7579) Co-authored-by: houseme <housemecn@gmail.com> * feat(ilm): inspect legacy transition version state (#7581) * test(scanner): add release evidence ingestion gates (#7588) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: reuse admin error boundary for legacy tier reconciliation (#7587) * fix(test): stabilize release CI heal fixtures (#7589) Keep heal admission-only tests off durable root recovery so mock storage does not require local metadata disks. Align forceStart retry assertions with replacement semantics and fix the e2e metric clippy collapse. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592) Replace the assets-branch scheme with a proper GitHub Release on rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets are replaced in place on every build. This is the standard channel — visible on the repo's Releases page, stable download URLs, no git history growth (release assets live outside the repository). - New scripts/release/publish_nightly_assets.sh: resolves-or-creates the 'nightly' release via the REST API, deletes same-name assets, uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release body with the build provenance (ref@sha, run link, sizes, SHA256). Plain curl + python3, no gh CLI (the build fleet has none — #7586). - The workflow step shrinks to invoking the script; full flow exercised end-to-end against the real release with probe files (create / upload / overwrite / download round-trip / body update). * test(scanner): add release bundle evidence descriptors (#7595) * test(scanner): merge release evidence descriptors Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): add authority and legacy release descriptors Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): add scoped ack release descriptor Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): require all release evidence fields Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): add mrf release descriptor producer Add a standalone W13 MRF descriptor producer for measured G07/G08/P4 raw artifacts, with fail-closed self-tests for missing responsibility cases, disk-full ENOSPC observations, synthetic artifacts, and short P4 cleanup soaks. Also bind G12 quota path evidence to explicit case lists so W16 descriptors cannot pass without reset and settlement coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): reuse mrf release descriptor producer Route the W13 runner through the shared MRF descriptor producer so measured runs and offline re-packaging use the same fail-closed validation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(ilm): backfill proven legacy transition state (#7591) * feat(ilm): backfill proven legacy transition state * fix(ilm): remove redundant reconcile test clone Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(ecstore): remove redundant transition clone (#7597) Drop an unnecessary clone in the legacy transition state digest regression so release CI clippy lanes can run with warnings denied. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): seed current backlog members before committing (#7594) * fix(scanner): seed current backlog members before committing * test(app): run recursive force delete test on large stack Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind raw profile artifacts (#7598) Require Scanner/Heal release profile wrappers to carry bundled raw profile artifacts plus the matching profile cost metrics. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): surface first failed object in status (#7599) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover retry budget across attempts (#7600) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): harden durable producer evidence gate (#7601) Require Scanner/Heal G11 maintenance evidence to prove durable dirty producer journal replay, replay-state hydration, generation-bound ACK clearing, and invalid journal fail-closed cases before segment reuse can be activated. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): harden scoped ack evidence proof (#7602) Require Scanner/Heal G03 scoped ACK evidence fields to carry their own measured provenance and concrete ACK, capability, and mixed-peer observations before release-bundle gate verification. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind status outcome raw provenance (#7603) Require measured G05/G06/R-D raw status, compatibility, and disposition artifacts to share run identity before producing release bundle descriptors. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): retry fully recoverable batch failures (#7604) Allow recursive bucket and cluster heal attempts whose retained batch failure is entirely retryable to use the existing bounded manager retry budget. Permanent or mixed batch failures remain terminal so heal outcome semantics stay strict. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): avoid movement debt for stale remote lease (#7605) Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks. Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh workspace dependencies (#7606) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind G14 case artifacts (#7611) Require Scanner/Heal G14 release bundle JSON wrappers to mirror their outer evidence and carry self-contained case artifact provenance. Copy proof-json case artifacts into the generated G14 descriptor bundle so assembled release bundles can validate case file hashes after relocation. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry contended decommission cancel target locks (#7612) * test: align transition fixtures and KMS context expectations (#7610) * test(scanner): stabilize W13 release gate evidence (#7613) * feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593) feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592) Replace the assets-branch scheme with a proper GitHub Release on rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets are replaced in place on every build. This is the standard channel — visible on the repo's Releases page, stable download URLs, no git history growth (release assets live outside the repository). - New scripts/release/publish_nightly_assets.sh: resolves-or-creates the 'nightly' release via the REST API, deletes same-name assets, uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release body with the build provenance (ref@sha, run link, sizes, SHA256). Plain curl + python3, no gh CLI (the build fleet has none — #7586). - The workflow step shrinks to invoking the script; full flow exercised end-to-end against the real release with probe files (create / upload / overwrite / download round-trip / body update). * test(scanner): stabilize W13 release gate evidence Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays. Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bound native backlog fixture resources (#7609) * fix(scanner): bound native backlog fixture resources Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops. Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(tier): refresh release-merged failure fixtures Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge. Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards. Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): align release E2E gate and repair regression tests (#7607) * fix(scanner): harden heal release gates (#7614) * test(scanner): reject synthetic heal evidence inputs (#7617) Harden the Scanner/Heal checkpoint restart and G14 EC evidence producers so release descriptors cannot be assembled from marked fixture, dry-run, or synthetic case inputs. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): collect status outcome raw evidence (#7618) Normalize live Scanner/Heal status/outcome observations into the measured raw artifacts consumed by the G05/G06/R-D release descriptor producer. Reject synthetic observations, incomplete required cases, and reused run/window identities before raw artifacts can enter the release bundle flow. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind scoped ack confirmation to generation (#7619) Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response. Refs rustfs/backlog#2427 Refs rustfs/backlog#2281 Refs rustfs/backlog#2240 Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test: report tier reference proof setup failures without hanging (#7608) * test(scanner): plan Linux evidence validation (#7620) Add a Scanner/Heal Linux release-evidence planner that emits a machine-readable execution manifest for the remaining measured validation lanes. The planner can run only lightweight preflight checks and keeps plan-only output distinct from measured release evidence. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): select G14 replacement drive by evidence (#7622) Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): report Linux evidence plan status (#7624) Add a status mode for the Scanner/Heal Linux evidence planner so operators can identify missing or incomplete artifacts before final bundle assembly. The status output stays plan-only and never reports release approval. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add G14 multi-pool background heal evidence (#7628) test(scanner): align G14 multi-pool outage evidence Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): verify deferred multipool outage evidence (#7629) Validate the G14 multi-pool oracle field that records down-window outage PUT refusal and requires the deferred post-rejoin outage object to be accepted and checked through S3. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): allow metadata snapshots at read quorum (#7627) fix(ecstore): admit metadata snapshots at read quorum Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh release dependencies (#7632) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): use pool lock quorum for set writes (#7630) Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice. Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): keep G14 multi-set restart graceful (#7633) test(e2e): keep multi-set restart graceful Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): record deferred G14 outage writes (#7637) Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bound G14 multi-pool evidence size (#7640) Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation. Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>	2026-09-10 18:24:20 +08:00
Zhengchao An	8fc1c9281e	refactor(odm): move migration orchestration into application (#7226) ... * refactor(odm): move migration orchestration into application * style(odm): format relocated listing test imports	2026-09-06 01:31:58 +08:00
Zhengchao An	9863f4848d	feat(tier): report cluster tier stats and count tier requests (#7110) ... `GET /v3/tier-stats` answered from whichever process received the request, returning that node's rolling 24-hour transition counters as if they were cluster totals, and the `TierRequestsSuccess` and `TierRequestsFailure` metric names had no producer at all. The body now separates the two quantities a tier carries. Stored inventory comes from the persisted scanner usage snapshot, which is already cluster-wide; rolling activity is summed over every member through a new read-only `TierDailyStats` peer RPC. Rings are merged rather than added, so an idle node's expired hours age out, and each node counts only its own committed transitions, so a retry is counted once. Coverage travels with the numbers: `activity.status` names the reporting members and the ones that could not be asked, timed out, or answered with a ring this build refuses to merge, and per-tier inventory is absent rather than zero when the snapshot has no accounting. The version 1 body stays reachable at `?format=legacy`. Tier request counters are recorded at the two seams every remote request passes through, so a new provider is counted by construction, with a closed operation/outcome label set that can never grow a tier name, endpoint or object key. Closes rustfs/backlog#2207 Co-authored-by: cxymds <cxymds@gmail.com>	2026-09-04 09:40:03 +08:00
Zhengchao An	0713a723cd	fix(odm): close three on-demand migration follow-ups from the e2e matrix (#7101)	2026-09-03 19:40:54 +08:00
houseme	0e6ee3bf62	feat(scanner): coordinate usage and workload boundaries (#7093) ... * test(scanner): wire usage and heal rebuild gates * docs(scanner): define usage authority protocol * docs(heal): clarify scanner and ecstore boundaries * refactor(scanner): split metrics from contracts * feat(scanner): use shared workload snapshots * fix(ecstore): recheck capacity before decommission drain	2026-09-03 17:02:43 +08:00
Zhengchao An	74be040c62	feat(ecstore): add the on-demand migration backfill job (#7087) ... * feat(ecstore): add on-demand migration backfill job core Add the background backfill job for on-demand migration (rustfs/backlog#2159): a durable checkpoint under buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by every save, a recovery pass that takes over expired leases (or jobs this node owned before a restart) and cancels jobs whose config changed, and a main loop over the source ListObjectsV2 pages with the skip_existing policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing. The pull queue gains per-job completion reports so the job can count pulled/failed keys (hashes only), and pull permits become two-tier so an online miss is never queued behind a backfill pull. * feat(admin): expose on-demand migration backfill job Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators: POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and GET .../backfill return the checkpoint document, GET .../status gains a backfill summary, and the recovery loop plus the process-wide runner are installed at startup. Backfill control reuses Set/GetBucketOnDemandMigrationAction and is recorded in the route policy, the registration matrix and the admin route snapshot. Add the rustfs-madmin wire types and client methods with golden fixtures shared by the server tests, the backfill_* metric descriptors and their collector, and three e2e scenarios: a full backfill across list pages, cancellation, and resuming from the persisted continuation token after a server restart.	2026-09-03 08:52:58 +08:00
Zhengchao An	07dce1cab0	feat(obs): export on-demand migration metrics and status snapshot (#7081) ... * feat(obs): export on-demand migration bucket metrics Add the on_demand_migration metric subsystem: per-bucket request, pull, failure, inflight, queue depth, source latency distribution and breaker state series fed from the ODM runtime snapshot through the storage boundary, collected alongside bucket replication metrics, and retired once a bucket's config disappears. * feat(admin): report the full on-demand migration status snapshot Extend GET /v3/on-demand-migration/{bucket}/status with provider, endpoint host, breaker state, runtime counters, last source error, inflight and queue gauges and the config timestamp. served_by_source_ratio stays null: no per-bucket GET total exists to divide by. Update the madmin status type and golden fixture together.	2026-09-03 03:46:28 +08:00
houseme	c006f84461	feat(info): report all rustfs features (#6722) ... * feat(info): report all rustfs features Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): update s3s revision Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): adapt dial9 telemetry API Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-27 14:45:50 +08:00
Zhengchao An	a96dd7d289	refactor: migrate consumers off rustfs-common heal/scanner shims (#6623) ... * refactor(ecstore): import heal/scanner contracts crates directly (backlog#1843) * refactor(heal): import heal/scanner contracts crates directly (backlog#1843) * refactor(lifecycle): import heal/scanner contracts crates directly (backlog#1843) * refactor(obs): import heal/scanner contracts crates directly (backlog#1843) * refactor(protos): import heal/scanner contracts crates directly (backlog#1843) * refactor(scanner): import heal/scanner contracts crates directly (backlog#1843) * refactor(rustfs): import heal/scanner contracts crates directly (backlog#1843)	2026-08-26 21:13:18 +08:00
Zhengchao An	45c03ca37f	test(obs): move source-text logging tests into the logging guardrail script (#6610) ... test(obs): replace source-text logging tests with logging guardrail script coverage The seven fs::read_to_string source-text tests in crates/obs/src/logging.rs asserted retired logging patterns and required structured-logging fields across 13 files in other crates, four of them reverse reads into the rustfs binary crate. Their patterns are now enforced by scripts/check_logging_guardrails.sh, which runs in pre-commit and CI, covers the same files through checked_files plus require_patterns, and does not silently lapse when a governed file moves. Part of rustfs/backlog#1884.	2026-08-26 09:55:49 +08:00
Zhengchao An	5d820df79c	chore(obs): rename ReplicationStats to ReplicationMetricsSnapshot (#6344) ... * chore(obs): ReplicationStats -> ReplicationMetricsSnapshot, BucketReplicationStats -> BucketReplicationMetricsSnapshot Rename in-obs-crate ReplicationStats and BucketReplicationStats to ReplicationMetricsSnapshot and BucketReplicationMetricsSnapshot respectively. No serde impact (these types are Prometheus metric collectors, not serialized). No external consumers found outside the obs crate. * cleanup: remove #[serial] annotations from e2e_test, scanner, lifecycle, and object-capacity crates Remove no-op #[serial] attributes (nextest ignores serial_test) and the serial_test dependency from four crates. All tests already use temp_env for env-var isolation, making #[serial] purely redundant. Crates cleaned: - e2e_test (37 annotations, 9 imports, removed serial_test dep) - rustfs-scanner (115 annotations across 7 files, removed serial_test dep) - rustfs-lifecycle (46 annotations, removed serial_test dep) - rustfs-object-capacity (38 annotations, removed serial_test dep) Also converted scanner/tests/lifecycle_integration_test.rs with_forced_immediate_enqueue_timeout helper from unsafe raw env::set_var/remove_var to temp_env::async_with_vars for proper isolation, and added async_closure feature to scanner's temp-env dep. * fix(lifecycle): restore #[serial] on 2 tests that read env vars without temp_env eval_inner_expires_latest_object_after_days_due and eval_inner_does_not_panic_on_many_equal_due_events call eval_inner() which reads ENV_ILM_PROCESS_TIME via std::env::var(). Without #[serial] they race with other tests that set these vars via temp_env. * style: cargo fmt	2026-08-22 00:30:30 +00:00
Zhengchao An	d22cb5d07a	fix: resolve release-blocking integration failures (#6320) ... * fix: resolve release-blocking integration failures * fix: satisfy stable clippy lints * fix: satisfy Rust 1.98 CI lints	2026-08-21 06:39:29 +08:00
houseme	830e553a3c	feat(obs): complete metric dimension coverage (#6314) ... Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-20 22:43:40 +08:00
Henry Guo	24cfce12ed	fix(metrics): remove duplicate scanner counter producers (#6245) ... * fix(metrics): remove duplicate scanner counter producers * chore(deps): centralize metrics test dependencies * ci: avoid apt mirror for ripgrep setup * test(protos): track read-version encoder refactor --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>	2026-08-19 21:58:22 +08:00
Zhengchao An	7db3882777	chore(obs): adjudicate 19 bare dead_code allows (#6162) ... backlog#1823 step 10, batch 2. Eighteen of the nineteen suppress nothing and are deleted; one was real and keeps an allow that now says why. Rotation::Never is constructed only by the rolling-appender tests at rolling.rs:456, 477 and 498, so the lib target reports it as never constructed. Its allow is restored with that reason. Finding it corrected the method used for batch 1. Removing all nineteen and running cargo check -p rustfs-obs --tests reported zero warnings even after touching every source file, while clippy --lib --tests -D warnings caught Rotation::Never. cargo's warning output is not a reliable completeness check — it does not re-emit for cached compilations, and touching the sources did not cover the lib target here. Later batches should treat clippy -D warnings as the gate; batch 1's six crates were re-checked under clippy and are clean. Taken with #6086, which cleared this crate's 44 module-level blankets and left six real items, obs has now had 63 dead-code suppressions examined, of which seven were suppressing anything at all. The rest sat on items that are publicly reachable, where dead_code never applied — the same shape as the swift module and kms's dek.rs. Verification: clippy --lib --tests -D warnings clean in the default, gpu and pyroscope lanes; cargo nextest run -p rustfs-obs 324 passed; make pre-commit exit 0. Ref rustfs/backlog#1823 (step 10).	2026-08-17 11:32:23 +08:00
Zhengchao An	7c2b513613	chore(obs): drop 44 dead_code blankets from the metrics tree (#6086)	2026-08-14 08:10:49 +08:00
Henry Guo	4c5e73b2f2	fix(scanner): defer cycles during data movement (#5970) ... * fix(scanner): defer cycles during data movement * fix(scanner): distinguish deferred scan cycles --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>	2026-08-12 20:46:06 +08:00
anthonymartin	706a8b6061	fix(scanner): publish bounded observational usage (#5742) ... * fix(scanner): publish bounded observational usage * test(ci): serialize embedded integration ports * test(cache): isolate generation-change timeout * fix(scanner): address observational usage review Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-07 08:52:34 +08:00
anthonymartin	656a2f14bf	fix(logging): bound hot-path span amplification (#5763) ... * fix(logging): bound hot-path span amplification * refactor(logging): reuse HTTP log target constant Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-06 14:29:18 +00:00
houseme	16c2928965	refactor(metrics): migrate scanner report timestamps to jiff (#5710) ... * refactor(metrics): migrate scanner report timestamps to jiff Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(madmin): migrate admin timestamps to jiff (#5712) Co-authored-by: heihutu <heihutu@gmail.com> * refactor(storage): migrate RPC DTO timestamps to jiff (#5713) Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-04 18:35:38 +00:00
houseme	035ce5d784	feat(obs): add bounded metrics dimensions (#5645) ... * feat(obs): add drive topology detail metrics Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets. Backlog: rustfs/backlog#1655 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve suspect drive runtime state Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): skip unknown drive inode samples Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner source work detail metrics Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add ilm action detail metrics Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add delivery target server metrics Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication target flow metrics Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add request server metrics Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): apply rustfmt to metrics changes Apply rustfmt output to the metrics dimension changes without altering behavior. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): reuse audit target label constant Use the exported audit target_id label constant for legacy audit target metrics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): populate drive disk metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner bucket drive result metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication proxy server metrics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric liveness review Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address additional metric review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): count drive calls at start Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metrics dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address scanner review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): reduce disk metric contention Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): retire stale dimension series Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-03 09:03:34 +08:00
houseme	fbec33bd29	Expose target-scoped durable MRF backlog metrics (#5584) ... * feat(replication): expose target durable mrf backlog Add target ARN attribution to durable MRF entries and surface target-scoped durable backlog metrics without changing existing bucket-only metric labels. Keep legacy MRF files bucket-only by defaulting missing targetARNs to an empty list, and expose target snapshots through an additive API so existing DurableMrfBacklogSummary callers remain source-compatible. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(replication): expose runtime target backlog (#5586) Track runtime replication backlog by target ARN for regular, large, delete, and MRF admission paths while preserving the existing bucket-level backlog semantics. Add target-scoped current backlog metrics and merge them with durable target backlog snapshots for observability. Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-01 17:31:50 +00:00
houseme	29dedcc7fd	fix(obs): retire replication backlog metric series (#5574) ... Emit zero tombstones for removed bucket-level replication backlog series and retire the corresponding metric descriptors after the configured tombstone window, matching the existing replication bandwidth lifecycle. Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-01 13:21:41 +00:00
houseme	da389c0e21	fix(replication): harden backlog observability (#5564) ... Add RAII guards for replication runtime backlog tickets so active worker and queue counters unwind on every terminal path. Expose node-local MRF pending, dropped, missed, and flush-failure metrics through the bucket replication Prometheus collector while keeping the existing current backlog and durable MRF gauges additive. Update durable MRF summary maintenance to aggregate incrementally during the persister loop, avoiding repeated full-entry scans on each successful flush. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-08-01 11:12:58 +00:00
houseme	d5c6ba99d5	fix(ecstore): harden HotPath profiling boundaries (#5555) ... * test(hotpath): gate mimalloc heap test by platform Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): attribute HotPath CPU measurements to impls Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): trace raw shard I/O with HotPath Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): redact profiler and OPA endpoint diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): settle encoded queue accounting Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): abort encoder producer on cancellation Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-01 08:49:48 +00:00
houseme	62d44d10b8	Expose replication backlog gauges (#5557) ... * fix(replication): count backlog at queue admission Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): expose bucket replication backlog gauges Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): report recent backlog from queued work Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve legacy backlog metric semantics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): expose durable MRF backlog gauges Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs): cover replication backlog metric scope Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): keep backlog metrics API-compatible Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(obs): streamline replication backlog metrics Keep MRF backlog accounting and OBS metric collection on a single, cheaper path. Co-Authored-By: heihutu <heihutu@gmail.com> * test(kms): update aws capability snapshot Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-01 08:48:45 +00:00
houseme	78d6918c52	feat: extend hotpath coverage across crates (#5505) ... Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling. Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-31 05:42:19 +00:00
houseme	30dc04c94b	fix(obs): label node-local metrics by server (#5465) ... Add stable server labels to node-local Prometheus metrics and OTLP resource attributes so dashboards can distinguish per-node CPU, memory, host network, and internode traffic series. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-30 04:57:22 +00:00
Henry Guo	3f20fbd77b	fix(scanner): report active first-cycle status (#5397) ... * fix(scanner): report active first-cycle status * fix(scanner): publish cycle activity consistently * test(common): satisfy Rust 1.97 waker lint * fix(scanner): satisfy Rust 1.97 clippy --------- Co-authored-by: houseme <housemecn@gmail.com>	2026-07-29 22:33:25 +08:00
Henry Guo	d5df66ac4f	fix(scanner): require authoritative usage snapshots (#5333)	2026-07-28 07:29:57 +08:00
Henry Guo	a63b79004c	fix(scanner): make distributed usage convergence authoritative (#5151) ... * fix(scanner): make distributed usage cycles authoritative * fix(scanner): close distributed refresh races * fix(config): align scanner reload integration * fix(admin): scope config test helpers * fix(scanner): harden distributed usage convergence * fix(scanner): preserve rolling activity compatibility * fix(admin): expose non-secret optional config values * fix(scanner): acknowledge distributed dirty usage * fix(ecstore): make bucket mutations cancellation safe * fix(scanner): preserve pending dirty acknowledgements * test(obs): account for superseded scanner metric * fix(api): reject excess detached bucket mutations * test: close scanner convergence coverage gaps * fix(scanner): make path tracking cleanup one-shot --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>	2026-07-25 18:45:16 +08:00
Henry Guo	bb7bba3237	fix(obs): clarify cluster bucket usage metrics (#5081) ... Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>	2026-07-21 17:23:49 +08:00
cxymds	b0c6c4cbce	fix(storage): resolve erasure parity per pool (#4977) ... * fix(filemeta): add state-aware file info validation * fix(filemeta): validate shard arithmetic and delete paths * fix(ecstore): add fallible erasure construction * fix(ecstore): resolve storage parity per pool * fix(storage): report heterogeneous erasure layouts * fix(admin): publish prepared storage config atomically * fix(storage): harden per-pool parity boundaries * fix(storage): address pre-PR validation findings * test(ci): fix strict-topology validation fixtures * fix(heal): preserve delete markers during repair * refactor(filemeta): drop unused ValidatedFileInfo witness ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime. No behavior change. * fix(filemeta): keep compressed and MinIO-migrated tiered objects readable The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt: - Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked. - MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read. Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery. * fix(scanner): build the delete-marker test fixture without erasure geometry get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2. Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker. * docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit. * fix(heal): report parity from erasure geometry, not is_valid() heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change. * fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object `From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`. * test(filemeta): assert real historical xl.meta versions pass metadata-read validation Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt. Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red. * fix(ci): remove duplicate storage test re-exports --------- Co-authored-by: overtrue <anzhengchao@gmail.com>	2026-07-19 21:52:31 +08:00
houseme	3a4937367a	chore(deps): remove redundant obs tracing feature (#4892) ... Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-16 14:22:06 +08:00
houseme	56179210ab	chore(deps): simplify dependency features (#4890) ... * chore(deps): remove redundant dependency features Remove manifest feature entries that are implied by other requested features in the same dependency declaration. Verified that the resolved Cargo feature graph is unchanged after the cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): narrow tokio and reqwest features Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-16 05:20:43 +00:00
houseme	f3a7a4b0da	chore(deps): localize workspace dependency features (#4888) ... Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root. Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-16 03:55:27 +00:00
escapecode	a80699b6dd	feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634) ... feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme <housemecn@gmail.com>	2026-07-14 15:36:14 +08:00
houseme	1553dc3f62	Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783) ... * fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-12 16:03:28 +00:00
houseme	3b139e5267	fix(obs/cleaner): harden log cleaner durability, symlink safety, and retention (audit OLC-01..14) (#4776) ... * fix(obs/cleaner): fsync archive and log dir before deleting source logs OLC-01: the compression path flushed the BufWriter but never synced the archive data or the parent directory before renaming, and the source was then unlinked with no durability barrier. A crash after rename but before the page cache reached disk could leave a truncated/zero-length archive while the source was already gone — permanent log/audit data loss. Because the archive is always renamed to a brand-new name (guaranteed by the existing exists() guard), ext4 auto_da_alloc does not mask this. Hand the underlying File back from the writer closure, sync_all() it before rename, fsync the parent directory, and fsync the log directory after the unlinks so a delete cannot be reordered ahead of the archive it justified. Guard the temp file with an RAII cleanup so an early return or panic cannot leak a *.tmp orphan. Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW OLC-03: the temp archive was opened with File::create at a predictable `<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write access to the log directory could pre-plant that path as a symlink and have the compressor follow it — truncating and overwriting an arbitrary external file, then chmod-ing it to the source log's mode. This mirrors the symlink refusal already enforced on the deletion path (secure_delete). Route temp creation through create_tmp_archive(), which uses create_new (O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to refuse a symlink at the final path component. Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with restrictive mode OLC-06: File::create left the temp archive world-readable (0644 & ~umask) for the entire duration of compressing a large log, exposing the full plaintext of a possibly-0600 audit log on shared hosts until the mode was copied only after the write completed. Pass the source mode into create_tmp_archive and open the temp file with it (default 0600) so it is restrictive from creation; the post-write chmod still tightens/matches the source mode exactly. Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): validate existing archive before skipping recompression OLC-02: the idempotency guard used Path::exists() (which follows symlinks) and trusted whatever it found, then the caller deleted the source. A planted `<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed run (OLC-01), would green-light deleting the source with no valid backup — data loss / log destruction. Replace exists() with symlink_metadata (no follow) and only treat the entry as a completed prior result when it is a regular, non-empty file whose header matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls through to recompression, whose atomic create_new+rename replaces the bad entry (a symlink is replaced, never followed or deleted through). Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so projected_freed_bytes = input and delete_files reported the full input as freed. A real run keeps the archive on disk (freed = input - archive), so dry-run overstated reclaim by the whole archive footprint. Estimate the archive with a deliberately conservative ratio so the projection never exceeds what a real run reclaims. Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): make freed-byte accounting resilient; document steal metric OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which silently reports 0 on failure and skews freed-byte metrics (input - 0 = full input, overstating reclaim). Use the copy() byte count as the authoritative input size and, when the archive metadata read fails, conservatively assume no savings instead of 0. Also document that the steal_success_rate counts only victim steals (batch = one success), so it reads as a relative rebalancing signal, not absolute task acquisition. Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21], silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1 and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those meanings survive, and echo the effective (post-clamp) levels in the startup log via new effective_gzip_level()/effective_zstd_level() getters so the log matches what actually runs. Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0 OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so retention=0 disabled it entirely while compression kept producing archives, and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk growth. Replace select_expired_compressed with select_archives_to_delete, which applies age expiry (when retention is on) and, regardless of retention, trims the oldest archives until the set fits under max_total_size_bytes. Also warn at startup when compression is on with retention=0 so the "keep forever" semantics are not mistaken for "delete immediately". Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so a typo (or a literal comma splitting a char-class in the config string) turned "protect this file" into "delete this file" with no signal. Log a warning per rejected pattern with the raw string and parse error so the misconfiguration is visible. Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used yield_now on the empty path, burning CPU during redistribution windows; worker_count had no upper bound so a mis-set parallel_workers over a directory of thousands of logs could spawn thousands of threads; and default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1. Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn when active-file guard is disabled by empty filename OLC-13 (defense-in-depth): the scanner protects the live log purely by exact filename equality against active_filename. An empty active_filename silently disables that protection, so a non-empty file_pattern could make the live log a deletion candidate via the public builder. Warn in build() when that unsafe combination is configured. The audit's "never delete the newest match" structural guard is intentionally not implemented: it would conflict with the legitimate keep_files=0 semantics (purge all rotated logs). The naming contract is instead locked by regression tests (OLC-14). Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode OLC-07: from_config_str silently fell back to defaults for unrecognized compression algorithm and match mode (any non-"prefix" value became Suffix), hiding operator typos like "prefixx" that could make the cleaner match no rotated logs. Warn on a non-empty unrecognized value in both parsers, and echo the resolved match_mode in the startup log alongside the algorithm. Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds (default 3600), so crash-left orphans lingered up to an hour, and the tmp suffix list was hardcoded rather than derived from compressed_suffixes() — a new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes. Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and gate orphan removal on a small fixed grace window instead of min_file_age (orphans are never live-written after the rename that would promote them). Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases OLC-14: add regression tests for the previously-untested safety/correctness branches — symlink rejection (external target never deleted), archive age expiry vs fresh retention, archive byte-cap trim with retention disabled, gz/zst classification, max_single_file_size selection, min_age protecting a fresh non-empty log, active-file exclusion when the active name also matches the pattern, invalid exclude glob not aborting build, dry-run + compression creating no archive, gzip round-trip validity, and the idempotent-archive branch trusting a valid prior archive. Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs/cleaner): apply rustfmt and collapse nested if (clippy) Formatting-only cleanup over the audit fix series: rustfmt normalization of the multi-line expressions introduced in compress.rs/core.rs, plus collapsing the delete_files directory-fsync into a single let-chain to satisfy clippy::collapsible_if. No behavior change. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(obs/cleaner): collapse redundant source stats in compression path The per-issue fixes to compress_with_writer accumulated three metadata() syscalls on the source in the real compression path: an input_bytes read that was immediately shadowed by the copied byte count (a dead read), a source_mode read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode. Collapse to a single fd-based read — move the dry-run input_bytes read into the dry-run branch, read source_mode from the already-open fd (no path stat, no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged (same inode's mode, written for input size); 3 source stats -> 1. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured) Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-12 12:01:52 +00:00
cxymds	d8e69a3adf	fix(logging): enforce single-writer sinks and bound tracing (#4765) ... * fix(obs): prevent rolling log stdout aliasing * fix(ecstore): bound hot-path tracing payloads * test(logging): guard service and disk log invariants * fix(obs): silence useless_conversion on st_dev for Linux clippy rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op that trips clippy::useless_conversion under -D warnings. The conversion is still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the lint on that line rather than dropping the portable fallible conversion. * fix(logging): keep stdout sink validation portable (#4769) * fix(obs): keep stdout device conversion portable * docs(logging): update single-writer plan status --------- Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: houseme <housemecn@gmail.com>	2026-07-12 16:03:01 +08:00
Zhengchao An	631d93092e	docs(obs): align OtelConfig doc defaults with actual constants (#4760) ... Six doc-comments on `OtelConfig` fields stated defaults that disagreed with the constants the runtime actually applies in `extract_otel_config_from_env`. Correct them to match: - profiling_export_enabled: false -> true (DEFAULT_OBS_PROFILING_EXPORT_ENABLED) - sample_ratio: 0.1 -> 1.0 (SAMPLE_RATIO) - meter_interval: 15 -> 30 (METER_INTERVAL) - logger_level: info -> error (DEFAULT_LOG_LEVEL) - log_rotation_time: daily -> hourly (DEFAULT_LOG_ROTATION_TIME), and document the full minutely/hourly/daily set plus the daily fallback - log_cleanup_interval_seconds: 21600 -> 1800 (DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS) Doc-comment only; no behavior change.	2026-07-12 11:10:42 +08:00
houseme	5088a6cde4	perf(obs): trim per-collection-cycle waste in report_metrics (#4748) ... `report_metrics` runs on every metrics collection cycle and did three things it did not need to, for every metric, every cycle: - interned `metric.name`/`metric.help` through a `Mutex<HashMap>` even when the `Cow` was already `Borrowed(&'static str)` (the common case for statically named metrics); - re-ran `describe_*!` (which re-locks the recorder's metadata map) although the metadata never changes; - allocated a fresh `Vec<(String, String)>`, cloning every label key and value, even though `metric.labels` is already `[(&'static str, Cow<'static, str>)]`. Now: names/help resolve to `&'static str` without touching the intern cache when already borrowed; each metric is described once (tracked in a `HashSet`); and the recorder is fed `&metric.labels` directly, removing the per-cycle label clone. No metric names, help, label keys/values, or emitted values change. Verified by building rustfs-obs and running the report unit tests (the `metrics` macro accepts the borrowed label slice directly). Addresses rustfs/backlog#1185 (P3, report path). Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-11 17:46:13 +00:00
Zhengchao An	f63af3df63	chore: retire completed-migration scaffolding, wire orphaned boundary check (#4719) ... The ecstore/global-state migrations are done (backlog#815, #939, #1052 all closed). Review of every migration-era test/gate measure found three things actually retirable or broken — everything else is a live anti-regression guard and stays. Remove: - scripts/check_metrics_migration_refs.sh — guards a migration that finished: rustfs_metrics:: has zero hits, the metrics crate no longer exists, and the script was never wired into CI or make (only reference was one line in config-model-boundary-adr.md, also removed). - crates/obs init_metrics_collectors — the "backward-compatible alias kept during migration" the removed script was guarding. Zero callers; pure delegate to init_metrics_runtime. Archive (docs/superpowers/plans/, continuing the 2026-07 convention, with the standard archived banner): - startup-timeline.md, scheduler-baseline.md, profiling-numa-capability-inventory.md, kms-development-defaults-inventory.md — one-shot snapshots whose only consumer is the already-archived migration-progress ledger (their same-dir links there start resolving again after the move); zero script pins; fed the closed backlog#660/#665 architecture-review ledger. Fixed the one outbound link (startup-timeline -> readiness-matrix) that the move would have broken — check_doc_paths.sh deliberately does not scan plans/, so nothing else would have caught it. Wire (found orphaned by the same review): - scripts/check_extension_schema_boundaries.sh guards a live contract crate but was never invoked anywhere. Add lint-fmt.mak target, include in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job already installs ripgrep), sync the CONTRIBUTING.md enumerated list, and harden the script against a silently-passing rg probe when src/ is missing. Keep (verified live, documented so the next cleanup pass does not repeat this analysis): - scripts/check_architecture_migration_rules.sh — added a header stating it is a permanent boundary guard, not retirable migration scaffolding; 'migration' in the name is historical. - check_migration_gate_count.sh + floor, delete-marker e2e proof, all pinned docs, compat-cleanup-register sync, remaining inventories (referenced by live docs). Verification: all 7 guard scripts pass, actionlint clean, cargo check --workspace (excl e2e) clean, cargo fmt --check clean. Adversarially reviewed by two independent skeptic passes; their 7 findings (alias left behind, broken outbound link, missing banners, wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing header rationale) are all folded in.	2026-07-11 13:42:56 +08:00
houseme	48bdf87e53	refactor(obs): drop the dial9 S3 config left stranded by the upload revert (#4700) ... #4663 added a full S3 upload path (config fields + env parsing + wiring), then reverted the wiring when its dependency turned out to carry RUSTSEC advisories (D9-14). The revert stopped at enabled.rs and left the config layer half torn down: `Dial9Config` still carried `s3_bucket`/`s3_prefix` with no consumer, plus a dead `s3_upload_requested()`, and `Dial9SessionGuard::config()` was likewise never called. All three are zero-consumer dead code. Since S3 upload cannot be built at all, the config type should not model it. Drop the two fields and both accessors. `from_env` still reads the S3 env vars and warns about them (an operator who set them deserves to know why they do nothing — that warning is deliberate and stays), but drops the values instead of storing configuration nothing reads. While here, collapse `from_env`'s two `record_config` calls into one by extracting `from_env_enabled`, so the enabled/disabled paths share a single publish point. No behavior change: the S3 warning fires identically, and every other field is untouched. Net -13 lines, and `Dial9Config` drops from 8 fields to 6. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 19:44:07 +00:00
houseme	60ad15a7f9	fix(obs): remove the dial9 task-dump switch that could never work; correct measured claims (#4688) ... * fix(obs): remove the dial9 task-dump switch that could never do anything Measured on a bench host (Linux x86_64) against the code merged in #4663: with `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and `--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events. dial9 captures a task dump only for futures it wrapped itself — those spawned through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn` throughout. Same workload, same binary, only the spawner changed: tokio::spawn -> 0 dumps dial9::spawn -> 14709 dumps, all with callchains Upstream documents this (README line 151) and tracks the doc gap at dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663, and so shipped exactly the kind of lying configuration knob that PR set out to delete. Remove it: the two environment variables, the config fields, the `with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was to constrain the build to Linux while recording nothing. Re-adding it only makes sense together with migrating the paths under investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157. Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured: dumps are captured with and without it (14709 vs 14674, within noise), and upstream never asked for it. That requirement was mine, invented and untested. Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(obs): replace guessed dial9 retention numbers with measured ones Three corrections, all to claims I wrote in #4663 without measuring them. "Under a high poll rate that budget can wrap in minutes" was a guess. Measured on a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent): 13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108 minutes. Even at ten times the throughput that is ~11 minutes. State the measured rate and how to scale it instead. dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O on the blocking pool and through io_uring, never on an async worker, so a slow drive never lengthens a poll. Injecting 200 ms of latency on one of four drives cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms: 49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers does not help: sched events are per-worker only, and the CPU profiler samples on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at the `rustfs_io_*` metrics instead. What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no fault injected at all — real worker stalls nothing else in the obs stack surfaces. Lead with that. Also link the two upstream issues filed for the gaps we documented: dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs). Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18). Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 17:34:59 +00:00
houseme	8039a4ceae	test(cache): end-to-end regressions for the body-cache hook P0s (#4675) ... fix(ecstore): make body-cache hook re-registrable + add e2e regressions ODC-21 (backlog#1126): the GET body-cache hook lived in a first-wins OnceLock. When AppContext is rebuilt (config reload, test re-init) a fresh ObjectDataCacheAdapter is constructed and re-registered, but the OnceLock kept ecstore's GET probe pointed at adapter #1 while every usecase-layer fill and invalidation targeted adapter #2 — silently degrading the feature to a 0% hit rate with no error, log, or metric, and stranding entries in the unreachable cache until their TTL. Replace the slot with RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> so re-registration atomically swaps to the newest adapter, and log at WARN when a swap replaces a *different* instance (Arc::ptr_eq). RwLock over ArcSwapOption because arc-swap's RefCnt is impl<T> (Sized, thin *mut T) and cannot hold an Arc<dyn Trait> without a sized newtype wrapper; the probe reads the slot once per full-object GET but only clones an Arc, negligible next to the metadata quorum fan-out already done before the probe. Add a test-only clear_get_object_body_cache_hook so tests register/unregister deterministically. With the hook now re-registrable, add true end-to-end regressions that drive get_object_reader (not the full_object_plaintext_len predicate) against a real erasure-coded, genuinely-compressed object via the blackbox make_local_set_disks harness, with a stand-in hook playing the app-layer cache (the injection point production uses; the adapter itself lives above ecstore). These close the gap the predicate-only tests left — a caller that opens a new shortcut serving the cached body directly, the original form of both P0s: - backlog#1108: a raw_data_movement_read must yield the STORED (compressed) bytes, never the cached plaintext. - backlog#1109: a compressed cache hit must publish the DECOMPRESSED length as object_info.size (the UploadPartCopy invariant), with the streamed length matching. - backlog#1146: a restore read (restore_request.days) must serve STORED bytes, not the cache. Mutation-verified each e2e test bites: dropping the raw_data_movement_read gate serves plaintext (fails #1108); removing the hit-site size republication publishes 2972 vs 660000 (fails #1109); dropping the restore gate serves plaintext (fails #1146). Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 12:17:20 +00:00
houseme	00536da80c	refactor(obs): make dial9 telemetry opt-in and actually record events (#4663) ... * refactor(obs): make dial9 telemetry opt-in and actually record events The dial9 Tokio-runtime profiler was disabled by default, yet every build paid for it, and enabling it produced trace files with no events in them. Recorded empty traces --------------------- `build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9 only starts recording in `build_and_start*`. `build` still returns a live guard whose `is_enabled()` reports true, and still creates and seals segment files — they just contain a header and no events. It also skipped `with_trace_path`, so the background worker driving the segment pipeline was never spawned. Measured on the new smoke example: 310 bytes of bare segment header, against 5640 bytes for the same workload once recording actually starts. Switch to `with_trace_path(..).build_and_start(..)`. Cost was unconditional ---------------------- `--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs` depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never appends to) the config-file value, so any caller exporting their own RUSTFLAGS silently dropped the flag — the long comment in build.yml was a scar from that. dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`), the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the feature is on without the flag. Telemetry builds go through `make build-profiling`. Metrics that could not lie -------------------------- `rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}` were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened". Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1 even when the traced runtime failed and the process fell back to a standard runtime; it is replaced by `rustfs_dial9_supported` (compile-time), `rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality). No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its `Finished` state and stop writing, but exposes no way to observe that, so the gauge could only ever be hard-coded to 1. Documented as a known gap instead. Final events were lost ---------------------- The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so buffered events were never flushed at exit. `build_tokio_runtime` now returns the guard and `run_process` drops it before any exit path. Also ---- - `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics collection path. It is now sampled by a background task into an atomic. - `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind `dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed. - Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`. - Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the stub keeps the public API identical so callers need no `#[cfg]`. - Drop four print-only examples and the manual test bin that exercised the removed `init_session` scaffolding. Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`; build.rs correctly rejects `dial9` without `--cfg tokio_unstable`; `make pre-commit` passes. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(obs): document dial9 as an on-demand profiler scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9, and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce CPU overhead" on the strength of it. The knob is gone; the guidance built on it had to go too. Replace it with what is actually true: dial9 needs a `make build-profiling` binary, its disk budget evicts oldest-first (so a high poll rate can overwrite the incident you are chasing), and it cannot be toggled without a restart. Add docs/operations/dial9-runtime-profiling.md covering the build variants, an investigation walkthrough, the configuration table, how to read the three supported/configured/active_sessions gauges against each other, and the upstream gap that makes writer death only indirectly observable. Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs): add a dial9 smoke example that proves events are recorded The bug this guards against is invisible to every existing signal: with `build` instead of `build_and_start`, dial9 creates the trace file, seals segments, and reports `TelemetryGuard::is_enabled() == true` — it simply records no events. Only the segment's byte count tells the two apart. Measured on this workload: 5640 bytes when recording, 310 bytes (a bare segment header) when not. The example asserts >= 2048 bytes, and was verified to fail with the `build` call restored. Also correct the comment on the `is_enabled` check in `finish_traced_runtime`. It claimed to catch "recording silently off"; it does not. It only rejects the inert guard a lenient config yields after a build failure. Recording is guaranteed by `build_and_start`, not by that check. Co-Authored-By: heihutu <heihutu@gmail.com> * test(rustfs): accept Unsupported runtime telemetry capability A binary built without the `dial9` feature now reports the runtime-telemetry capability as `Unsupported` rather than `Disabled`. The distinction matters to operators: `Disabled` implies the capability can be switched on by setting an environment variable, which is not true here — telemetry needs a rebuild. Widen the assertion and pin the new semantics: when `dial9::is_supported()` is false, the state must be exactly `Unsupported`. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable CI's Dependency Review and `cargo deny` both reject the branch: dial9's `worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That webpki carries RUSTSEC-2026-0098, -0099 and -0104. 0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the smithy client, so there is nothing to upgrade to. Cargo's feature unification can add features but cannot drop a transitive dependency, so it cannot be worked around from here either — the rest of the workspace already resolves to the safe rustls-webpki 0.103 / hyper-rustls 0.27. Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3 environment variables stay parsed and warned about, now naming the real reason rather than a missing build feature. Trace segments are collected from the output directory instead. Tracked as D9-14 in rustfs/backlog#1157. With this, Cargo.lock is byte-identical to main: the PR no longer touches the dependency graph at all. Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on macOS, which is how the claim was found to be wrong. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 10:52:48 +00:00
houseme	e0c8b3dca6	fix(obs): evict retired metric cache entries (#4539) ... Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-08 17:41:10 +00:00
houseme	6ab8636734	fix(obs): count scanned versions independent of ILM (#4516) ... versions_scanned for both the scanner and ILM collectors was read from the Lifecycle work source's `checked` counter, which is never recorded on the production scan path — so rustfs_scanner_versions_scanned_total and rustfs_ilm_versions_scanned_total sat at zero even while objects_scanned climbed. The two metrics also have distinct intended meanings that were conflated: "versions scanned" (all versions, any bucket) vs "versions checked for ILM actions" (lifecycle-configured buckets only). Add a lifetime `versions_scanned` counter recorded for every version the scanner walks (independent of ILM), and record the Lifecycle source's `checked` counter from the ILM evaluator so the ILM metric reflects the real checked subset. The scanner collector now reports total scanned versions; the ILM collector keeps the ILM-checked subset. Closes backlog#995 (OBS-09). Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-08 16:50:51 +00:00
houseme	c9292688d3	fix(obs): wire dial9 runtime telemetry (#4491) ... Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-08 16:16:06 +00:00