diff --git a/.agents/skills/adversarial-validation/SKILL.md b/.agents/skills/adversarial-validation/SKILL.md index 4eddc3551..323be5fa1 100644 --- a/.agents/skills/adversarial-validation/SKILL.md +++ b/.agents/skills/adversarial-validation/SKILL.md @@ -1,277 +1,45 @@ --- name: adversarial-validation -description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done. +description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation. --- -# Adversarial Validation Playbooks +# RustFS Adversarial Validation -The policy — risk tiers, role list, protocol, exit criteria — lives in the -root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first; -this skill does not restate it. This file adds the RustFS-specific probe -playbook for each role: concrete attacks, where they apply, and the real -shipped bug or rule that earns each probe its place. +Use the risk tier and review shape defined in the root `AGENTS.md`. This skill +routes a review to RustFS-specific probes without loading unrelated domains. -## How to run a role +## Select Lenses -1. Pick the tier and the applicable roles per the root `AGENTS.md`. -2. Run each role as an independent pass over the final diff — a parallel - reviewer agent where the tooling supports it, otherwise a fresh - sequential pass that starts from the diff and the nearest scoped - `AGENTS.md`, discarding the writing session's assumptions. -3. Within a role, execute the probes whose domain the diff touches, plus any - attack the diff obviously invites that no probe lists — the playbook is a - floor, not a ceiling. -4. Report findings (concrete failure scenario or named missing test, with - file:line) or the role's null report: "attacked X, Y, Z — no break - found". A bare pass is not a result. +Read only the references required by the diff: -## Role playbooks +| Lens | When to read | +|---|---| +| [Correctness](references/correctness.md) | Every non-exempt adversarial review | +| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth | +| [Test coverage](references/test-coverage.md) | Behavior or test changes | +| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption | +| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues | +| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions | +| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out | -### Correctness adversary +Do not read all references as a precaution. A path name alone is insufficient; +the changed behavior must touch the lens's domain. -- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success? - - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal - - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.' -- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part. - - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding - - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535). -- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries. - - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage - - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563). -- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries. - - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers - - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN). -- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets. - - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs - - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552). -- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding. - - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage - - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546). -- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size. - - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math - - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness. -- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value. - - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates - - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class. -- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable. - - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers - - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here. +For a dedicated security audit or advisory analysis, use +`security-advisory-lessons` instead of loading it automatically during every +adversarial review. -Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found." +## Review Protocol -### Simplicity adversary +1. Freeze the exact final diff/head and list the selected lenses. +2. Run the review shape required by root `AGENTS.md`. +3. For each selected lens, either report a concrete finding or a null verdict + naming the attacks performed. +4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong + outcome, and a focused fix or missing regression check. +5. Fix or rebut every finding with code-path, test, or invariant evidence. +6. After a non-trivial edit, rerun only lenses affected by that edit against the + new exact diff. -- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence. - - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule - - Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack. -- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit. - - Where: Any diff adding helpers, branches on decoded/peer data, or tests - - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing). -- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff. - -Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break." - -### Security reviewer - -- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check. - - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites - - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'. -- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child. - - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs - - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'. -- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached. - - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites - - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.' -- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret. - - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification - - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403. -- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides. - - Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check) - - Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write'). -- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles. - - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup - - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402). -- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret. - - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling - - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods. -- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe. - - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies - - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse. -- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket. - - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream - - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only. -- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it. - - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement - - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'. -- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers. - - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths - - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill. -- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive. - - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing - - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806). -- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed. - - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication - - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'. -- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead. - - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling - - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'. - -Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found." - -### Concurrency/durability reviewer - -- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding. - - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff - - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers) -- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early. - - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/** - - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs -- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds. - - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs - - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff -- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write. - - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths - - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks' -- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target). - - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/** - - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions -- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever. - - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort - - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops -- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches. - - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/** - - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard -- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two. - - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/ - - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit -- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy. - - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers - - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path -- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key. - - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs - - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency -- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode. - - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs - - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption' - -Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff." - -### Compatibility reviewer - -- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec) silently loses mixed-case MinIO keys. - - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs - - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs -- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()). - - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs - - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552) -- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction. - - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/ - - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction -- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade. - - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers) - - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377) -- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff. - - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs) - - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md -- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption. - - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs - - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash -- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak. - - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/ - - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs -- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision. - - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer - - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons) -- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly. - - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/ - - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth -- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt. - - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/ - - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md -- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO() marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade. - - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path - - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573) - -Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found." - -### Performance reviewer - -- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs. - - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/ - - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric -- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes. - - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes - - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate -- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit. - - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/ - - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size -- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA). - - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths - - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370) -- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks. - - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path - - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit -- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item). - - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/ - - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape -- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances. - - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/ - - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse -- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff. - - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh - - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14) -- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding. - - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs - - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse -- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid. - - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh - - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect) - -Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found." - -### Test-coverage skeptic - -- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper. - - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal - - Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took. -- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works? - - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff - - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied. -- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch. - - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter - - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites. -- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err. - - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets - - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green. -- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk. - - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios - - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency. -- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377). - - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code - - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop. -- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps. - - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml - - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'. -- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map. - - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage - - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests. -- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered. - - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata - - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested. -- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here. - - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage - - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly. -- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default. - - Where: All crates; especially concurrent-branch merges into crates/ecstore - - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR. - -Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found." - -## Sources and maintenance - -Probes are distilled from shipped bugs in git history (commit/PR references -above), GitHub security advisories (see the security-advisory-lessons -skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/` -and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge -new incidents into an existing probe when they share a failure class; add a -new probe only for a distinct attack, rather than growing the root policy. +Do not turn a null verdict into a long checklist. Record concise evidence that +the relevant failure classes were attacked. diff --git a/.agents/skills/adversarial-validation/references/compatibility.md b/.agents/skills/adversarial-validation/references/compatibility.md new file mode 100644 index 000000000..6b2b9b73b --- /dev/null +++ b/.agents/skills/adversarial-validation/references/compatibility.md @@ -0,0 +1,24 @@ +# Compatibility Lens + +- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys, + including mixed casing and removal of both twins. +- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned + remote tiers receive no `versionId`; versioned purge requests retain the real + version ID. +- `xl.meta` changes preserve supported header/meta versions, recompute + signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO. +- Foreign/corrupt metadata validates parallel array lengths and missing fields; + it returns a decode error rather than indexing, panicking, or fabricating data. +- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence. + Bitrot framing, shard math, distribution, and inline prefixes are contracts. +- Client-visible metadata/events strip both internal prefixes + case-insensitively. +- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend + compatibly and absent new fields fail closed where authorization/quorum is + involved. +- Replay real client request shapes and exact pagination boundaries for S3 + handler changes. +- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO + fixtures and encrypted migration data. +- Compatibility shims use `RUSTFS_COMPAT_TODO()`, have a removal + condition, and default toward reading old data safely. diff --git a/.agents/skills/adversarial-validation/references/concurrency-durability.md b/.agents/skills/adversarial-validation/references/concurrency-durability.md new file mode 100644 index 000000000..ae09a7718 --- /dev/null +++ b/.agents/skills/adversarial-validation/references/concurrency-durability.md @@ -0,0 +1,23 @@ +# Concurrency and Durability Lens + +- For every changed lock, enumerate overlapping lock sets and construct the + ABBA interleaving. Multiple-lock order must be documented and consistent. +- Mark guard lifetimes and every `.await`, disk, and RPC call inside them. + Estimate contention and timeout behavior under concurrent requests. +- Object commits remain fenced if the distributed lock is lost after shard + writes and before metadata rename. +- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent + -> sync required ancestors`; simulate a crash after each step and honor the + configured durability gate. +- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success; + heal remains best-effort per target where that is the established contract. +- At every new cancellable await between mutation and cleanup/commit, drop the + future and inspect leftover files, counters, permits, and replay state. +- Multipart operations on the same upload ID are serialized where required; + abort/complete/list races cannot delete parts before durable commit. +- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already + committed write or delete the last surviving copy. +- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe + and duplicate delivery has an idempotency contract. +- Streaming reconstruction failures after partial output surface as errors, not + successful EOF. diff --git a/.agents/skills/adversarial-validation/references/correctness.md b/.agents/skills/adversarial-validation/references/correctness.md new file mode 100644 index 000000000..05b3d95f6 --- /dev/null +++ b/.agents/skills/adversarial-validation/references/correctness.md @@ -0,0 +1,29 @@ +# Correctness Lens + +Attack the changed behavior, not every subsystem in the repository. + +- Trace new error paths to the caller. Inject the ignored/wildcard variants and + verify they cannot become success, not-found, or a plausible default. +- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every + changed count, size, index, page limit, or optional value. +- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with + mixed disk errors and nil/placeholder entries. +- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding, + continuation markers, and object/prefix name collisions. +- For EC/read/streaming changes, inject failure after partial output and verify + the client receives an error rather than a clean truncated body. Assert exact + bytes and length. +- For multipart/object commits, fail before/after rename and cleanup; committed + data must remain readable and pre-commit cleanup must not destroy parts. +- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing + versions, and deterministic tie-breaking. +- For directory-object behavior, trace `__XLDIR__` at the store layer; branches + below the layer that sees trailing slashes are dead. +- For binary UUID metadata, absent, empty, and nil all mean no value. Never send + nil/empty `versionId` to an unversioned tier. +- For agent rules/skill routers, test a trigger matrix covering ordinary + inquiry, low-risk implementation, explicit review, high-risk code, PR + creation, release, and post-PR monitoring. Each case must select only the + intended workflow and retain required safety/authorization boundaries. + +Null verdicts name only the probes relevant to the diff. diff --git a/.agents/skills/adversarial-validation/references/performance.md b/.agents/skills/adversarial-validation/references/performance.md new file mode 100644 index 000000000..2b352ac2a --- /dev/null +++ b/.agents/skills/adversarial-validation/references/performance.md @@ -0,0 +1,20 @@ +# Performance Lens + +- For added clones/allocations on request/object/block paths, quantify copied + data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or + capacity reservation only for a concrete repeated cost. +- Route every new sync/flush through the durability-mode and bucket override + gates; mode `none` must not pay the new fsync. +- Keep blocking filesystem/CPU work off async runtime threads, but do not split + one small operation into many `spawn_blocking` round trips. +- Measure lock hold time across I/O and compare acquisition order for ABBA. +- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical + section when they need not be there. +- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans; + use a gate or bounded concurrency when the concrete fan-out warrants it. +- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge + balance and avoid repeated metadata decode/fetch per object. +- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths + require an existing gate. +- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or + A/B evidence, especially for 4 KiB objects. diff --git a/.agents/skills/adversarial-validation/references/security.md b/.agents/skills/adversarial-validation/references/security.md new file mode 100644 index 000000000..9490a09f5 --- /dev/null +++ b/.agents/skills/adversarial-validation/references/security.md @@ -0,0 +1,31 @@ +# Security Lens + +Use `security-advisory-lessons` only for a dedicated advisory/security audit. +For an ordinary matched diff, attack these boundaries: + +- Admin routes: route registration, whitelist, handler authn, and the exact + `AdminAction` must agree. Read-only diagnostics still require admin authz. +- IAM/service accounts: treat parent, claims, keys, groups, status, and policy + names as attacker-controlled; prove ownership/root authority before writes. +- Protocol frontends: every changed/sibling command authorizes the matching S3 + action before reaching storage. +- Secrets/signatures: use constant-time comparison, normalize public failures, + keep RPC/root/STS keys independent, and fail closed when secrets are absent. +- RPC: bind signatures to the exact method/path and timestamp; reject replay, + stale, malformed, truncated, and invalid-enum payloads without panic. +- Paths/object/archive entries: reject traversal, absolute/platform escapes, + and normalization differences between authz and storage. +- Copy/multipart/presigned POST: enforce source, destination, version-aware + actions, copy-source conditions, and every signed policy condition. +- Logging/errors: never expose credentials, tokens, expected signatures, raw + secret-bearing input, or merged configs—including via `Debug` and parse errors. +- Untrusted serde: reject unknown fields where compatible and validate + security-critical defaults/ranges before numeric conversion. +- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order; + isolate user content; never reflect credentialed arbitrary origins or trust + forwarded identity from direct clients. +- Object Lock: unreadable/fabricated/unparsable metadata fails closed across + foreground, lifecycle, scanner, and force-delete paths. + +Security findings distinguish unauthenticated compromise from a +low-privileged authenticated bypass. diff --git a/.agents/skills/adversarial-validation/references/simplicity.md b/.agents/skills/adversarial-validation/references/simplicity.md new file mode 100644 index 000000000..ca192f884 --- /dev/null +++ b/.agents/skills/adversarial-validation/references/simplicity.md @@ -0,0 +1,22 @@ +# Simplicity Lens + +- Compare the production diff with the smallest equivalent local edit. Fewer + lines alone are not evidence; the replacement must preserve correctness, + compatibility, readability, and real boundaries. +- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and + relevant dependencies for each new helper, constant, wrapper, or fixture. +- Reject forced reuse when normalization, error, deadline, or durability + semantics differ. +- Require a concrete trigger for every new defensive branch. Keep boundary + checks for disk/RPC/version data and checks immediately before destructive + actions. +- Flag one-caller helpers only when they merely forward or split a short linear + flow without adding domain naming, invariant isolation, or useful context. +- Ensure a replacement removes the superseded in-scope path or keeps one + canonical core behind a documented compatibility adapter. +- Remove narration/change-history comments; preserve concise safety, lock, + durability, and compatibility invariants. +- Treat tests, fixtures, generated code, and documentation separately from + production growth. Do not optimize away meaningful regression coverage. + +A finding must include a concrete smaller design, not a style preference. diff --git a/.agents/skills/adversarial-validation/references/test-coverage.md b/.agents/skills/adversarial-validation/references/test-coverage.md new file mode 100644 index 000000000..942295f88 --- /dev/null +++ b/.agents/skills/adversarial-validation/references/test-coverage.md @@ -0,0 +1,24 @@ +# Test-Coverage Lens + +- For every behavior claim, name the focused test/check that fails if the + changed hunk is reverted. If none is practical, require the reason and + residual risk. +- Confirm tests exercise the real production path and assert returned values, + exact bytes, stored state, or the specific error variant—not only success, + `is_err()`, or no panic. +- For new flags/modes, verify each branch and ask which test fails if the branch + is inverted. +- For new error propagation, inject the failure and assert the caller observes + it; mentally replacing `?`/`return Err` with success must break a test. +- Streaming GET tests assert the complete body and length under degraded reads. +- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code + round-trips are insufficient for compatibility. +- Concurrency tests use readiness polling, isolate global state, and avoid fixed + sleeps or unrealistically short timeouts. Use nextest groups when process-level + serialization is required. +- Internal metadata tests assert both RustFS and MinIO keys, not only read-back + through a helper that prefers one key. +- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and + absent vs empty vs nil UUID. +- A focused test proves only the targets/features it builds. Add compilation or + Clippy only for uncovered changed targets. diff --git a/.agents/skills/code-change-verification/SKILL.md b/.agents/skills/code-change-verification/SKILL.md index 553822c82..e295f975d 100644 --- a/.agents/skills/code-change-verification/SKILL.md +++ b/.agents/skills/code-change-verification/SKILL.md @@ -1,11 +1,12 @@ --- name: code-change-verification -description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release. +description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews. --- # Code Change Verification -Use this skill to review code changes consistently before merge, before release, and during incident follow-up. +Use this skill for an ordinary requested review. If the root policy or user calls +for adversarial validation, use `adversarial-validation` instead of running both. ## Quick Start @@ -79,4 +80,3 @@ Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — - Impact: ... - Fix suggestion: ... - Validation: ... - diff --git a/.agents/skills/code-change-verification/agents/openai.yaml b/.agents/skills/code-change-verification/agents/openai.yaml index 7e4566774..2f76eace4 100644 --- a/.agents/skills/code-change-verification/agents/openai.yaml +++ b/.agents/skills/code-change-verification/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Code Change Verification" short_description: "Prioritize risks and verify code changes before merge." - default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes." + default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings." diff --git a/.agents/skills/pr-creation-checker/SKILL.md b/.agents/skills/pr-creation-checker/SKILL.md index c2c24bd29..e83490713 100644 --- a/.agents/skills/pr-creation-checker/SKILL.md +++ b/.agents/skills/pr-creation-checker/SKILL.md @@ -1,97 +1,46 @@ --- name: pr-creation-checker -description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS. +description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review. --- # PR Creation Checker -Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR. +Use this skill only at the PR boundary. Reuse completed diff review and +verification evidence; do not reread the repository or rerun equivalent checks. -## Read sources of truth first +## Preflight -- Read `AGENTS.md`. -- Read `.github/pull_request_template.md`. -- Use `Makefile` and `.config/make/` for local quality commands. -- Use `.github/workflows/ci.yml` for CI expectations. -- Do not restate long command matrices or template sections from memory when the files exist. +1. Confirm the branch is based on current `origin/main` and contains only the + intended task diff. +2. Inspect `git diff --stat`, `git diff --check`, and changed file names for + secrets, logs, generated artifacts, or unrelated edits. +3. Confirm the checks selected by root `AGENTS.md` passed on the final diff. + Do not replace focused behavioral tests with a generic gate or rerun checks + already covered by an unchanged umbrella run. +4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`, + or CI only when the required command/current gate is uncertain. +5. Return `BLOCKED` for an unclean scope, missing required evidence, failed + required checks, or non-compliant metadata. -## Workflow +## Metadata -1. Collect PR context -- Confirm base branch, current branch, change goal, and scope. -- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness. -- Confirm whether the branch includes only intended changes. +- Title: English Conventional Commit, at most 72 characters, with no tool + prefix. +- Body: English, exact template headings, `N/A` where needed, concise rationale, + actual verification commands, and material risks/rollback notes. +- Use repository-relative paths; never include local absolute paths. +- Keep prose paragraphs on one logical line and never include the literal + sequence `\n`. +- Use a temporary body file with `gh pr create --body-file` or + `gh pr edit --body-file`; never pass multiline Markdown inline. -2. Inspect change scope -- Review the diff and summarize what changed. -- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas. -- Call out unrelated edits, generated artifacts, logs, or secrets as blockers. -- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes. -- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries. -- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core. -- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules. -- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body. +## Output -3. Verify readiness requirements -- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate. -- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body. -- If `make` is unavailable, use the equivalent commands from `.config/make/`. -- Add scope-specific verification commands when the changed area needs more than the baseline. -- If required checks fail, stop and return `BLOCKED`. +- Status: `READY` or `BLOCKED`. +- Title. +- Complete PR body. +- Verification commands and results. +- Risks or `N/A`. -4. Draft PR metadata -- Write the PR title in English using Conventional Commits and keep it within 72 characters. -- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead. -- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits. -- Keep the PR body in English. -- Use the exact section headings from `.github/pull_request_template.md`. -- Fill non-applicable sections with `N/A`. -- Include verification commands in the PR description. -- Do not include local filesystem paths in the PR body unless the user explicitly asks for them. -- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`. - -5. Prepare reviewer context -- Summarize why the change exists. -- Summarize what was verified. -- Call out risks, rollout notes, config impact, and rollback notes when applicable. -- Mention assumptions or missing context instead of guessing. - -6. Prepare CLI-safe output -- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown. -- Return a ready-to-save PR body plus a short title. -- If not ready, return blockers first and list the minimum steps needed to unblock. - -## Output format - -### Status -- `READY` or `BLOCKED` - -### Title -- `(): ` - -### PR Body -- Reproduce the repository template headings exactly. -- Fill every section. -- Omit local absolute paths unless explicitly required. - -### Verification -- List each command run. -- State pass/fail. - -### Risks -- List breaking changes, config changes, migration impact, or `N/A`. - -## Blocker rules - -- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed. -- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface. -- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged. -- Return `BLOCKED` if required template sections are missing. -- Return `BLOCKED` if the title/body is not in English. -- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule. -- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not. -- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence. - -## Reference - -- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR. +Immediately before the GitHub write, repeat only the five preflight checks above +against the final head. diff --git a/.agents/skills/pr-creation-checker/agents/openai.yaml b/.agents/skills/pr-creation-checker/agents/openai.yaml index 6a709cc7d..761e02936 100644 --- a/.agents/skills/pr-creation-checker/agents/openai.yaml +++ b/.agents/skills/pr-creation-checker/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "PR Creation Checker" short_description: "Draft RustFS-ready PRs with checks, template, and blockers." - default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status." + default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata." diff --git a/.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md b/.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md deleted file mode 100644 index 354571430..000000000 --- a/.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md +++ /dev/null @@ -1,16 +0,0 @@ -# PR Readiness Checklist - -- Confirm the branch is based on current `main`. -- Confirm the diff matches the stated scope. -- Confirm no secrets, logs, temp files, or unrelated refactors are included. -- Confirm the checks required by the `AGENTS.md` validation tier passed. -- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient. -- Confirm extra verification commands are listed for risky changes. -- Confirm the PR title uses Conventional Commits and stays within 72 characters. -- Confirm the PR title does not use tool-specific prefixes such as `[codex]`. -- Confirm the PR body is in English. -- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`. -- Confirm non-applicable sections are filled with `N/A`. -- Confirm the PR body does not include local absolute paths unless explicitly required. -- Confirm multiline GitHub CLI commands use `--body-file`. -- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception. diff --git a/.agents/skills/rust-code-quality/SKILL.md b/.agents/skills/rust-code-quality/SKILL.md index 3c85805bd..ef78c23fe 100644 --- a/.agents/skills/rust-code-quality/SKILL.md +++ b/.agents/skills/rust-code-quality/SKILL.md @@ -1,11 +1,12 @@ --- name: rust-code-quality -description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns. +description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit. --- # Rust Code Quality Gate -Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch. +Use this skill for a dedicated Rust review to cover rules that `cargo clippy` +does not catch. ## Quick Start @@ -45,7 +46,7 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' ## Manual Review Checklist -For every Rust code change, verify: +For the Rust diff under review, verify: ### Error Handling - [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment diff --git a/.agents/skills/rustfs-logging-governance/SKILL.md b/.agents/skills/rustfs-logging-governance/SKILL.md index ec716fca6..b00e57216 100644 --- a/.agents/skills/rustfs-logging-governance/SKILL.md +++ b/.agents/skills/rustfs-logging-governance/SKILL.md @@ -1,107 +1,34 @@ --- name: rustfs-logging-governance -description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`. +description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script. --- # RustFS Logging Governance -Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions. +Apply this skill only to changed logging sites; do not turn a local log edit into +a broad logging cleanup. -## Quick Start +## Workflow -1. Identify the files whose logs are changing. -2. Scan current `tracing` or `log` macros before editing. -3. Convert sentence-style logs to short event-style logs. -4. Demote hot-path success logs unless operators truly need them at `info`. -5. Preserve failure, fallback, and security-relevant diagnostics. -6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class. -7. Validate with formatting, targeted checks/tests, and the logging guardrail script. +1. Read the changed function/module context and classify the site as lifecycle, + request/hot path, fallback, external fetch, or summary. +2. Match neighboring structured events and reuse existing `EVENT_*`, + `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants. +3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`, + then context) and a short label last. +4. Select the level by operational meaning: + - `error`: behavior/security-affecting failure; + - `warn`: degraded/fallback/operator-actionable state; + - `info`: low-frequency lifecycle/mode change; + - `debug`: targeted diagnostics; + - `trace`: repetitive request/object/shard success paths. +5. Never log secrets, tokens, auth headers, credential payloads, raw + attacker-controlled bodies, or merged config dumps. Error strings and + `Debug` output are log surfaces too. +6. Prefer one aggregate summary over inventories or startup banners. +7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root + `AGENTS.md`. -## Core Workflow - -### 1. Scope the logging surface - -- Read the changed module in full before touching log lines. -- Classify the log site: - - lifecycle/startup - - request or validation path - - background loop or hot path - - fallback/degraded behavior - - cloud metadata or external fetch path - - metrics/config summary -- Do not rewrite business logic to make logging easier. - -### 2. Use the RustFS event shape - -- Prefer fields first, message second. -- Use short labels, not prose paragraphs. -- Default field shape: - - `event` - - `component` - - `subsystem` - - `state` or `result` - - key context fields -- Reuse stable field names and avoid inventing near-duplicates. - -See `references/logging-governance.md` for the event model, level policy, and anti-pattern list. - -### 3. Choose the right level - -- `error`: operation failure that affects behavior or security guarantees. -- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration. -- `info`: low-frequency lifecycle or mode selection. -- `debug`: targeted diagnostics and low-volume detail. -- `trace`: hot-path and repetitive success-path events. - -When in doubt, lower the verbosity of normal success paths and keep structured detail in fields. - -### 4. Preserve security and privacy boundaries - -- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps. -- Avoid logging raw forwarded headers or full trusted network inventories above `debug`. -- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily. - -### 5. Keep summaries aggregated - -- Replace multi-line startup banners or checklist logs with one structured event. -- If metrics already express a concept, avoid duplicating it with many `info!` lines. -- Prefer counts, modes, and sources over inventories unless debug detail is truly needed. - -### 6. Update guardrails when needed - -- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`. -- Add forbidden patterns only for styles the repo has intentionally retired: - - sentence-style lifecycle logs - - noisy hot-path `info!` - - checklist-style summary logs - - legacy fallback wording that has been replaced by structured fields -- Keep guardrails concrete and grep-friendly. - -### 7. Validate manually - -Use the smallest relevant set: - -```bash -cargo fmt --all --check -./scripts/check_logging_guardrails.sh -cargo check -p -cargo test -p -``` - -For broader Rust changes, add: - -```bash -./scripts/check_unsafe_code_allowances.sh -./scripts/check_architecture_migration_rules.sh -cargo clippy -p --all-targets -- -D warnings -``` - -## RustFS-Specific Notes - -- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`. -- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs. -- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning. - -## References - -- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist. +Read [logging-governance.md](references/logging-governance.md) only for a broad +logging audit, event-model migration, or guardrail expansion. Ordinary single- +site edits do not require the full workspace scope map. diff --git a/.agents/skills/rustfs-logging-governance/references/logging-governance.md b/.agents/skills/rustfs-logging-governance/references/logging-governance.md index 7d4b23893..4002932e3 100644 --- a/.agents/skills/rustfs-logging-governance/references/logging-governance.md +++ b/.agents/skills/rustfs-logging-governance/references/logging-governance.md @@ -1,285 +1,62 @@ -# RustFS Logging Governance Reference +# Logging Audit and Migration Reference -## Workspace Scope Map +Read this reference only for a broad logging audit, an event-model migration, +or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the +current workspace/crate list instead of maintaining one here. -Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role. +## Audit by Operational Role -### Core Server And Request Handling +- Server/protocol/admin: lifecycle, authorization failures, request boundaries, + and degraded subsystems; avoid normal request success at `info`. +- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle; + avoid per-object, per-shard, and folder iteration noise. +- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results; + never emit secrets, claims, payloads, or expected authenticators. +- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid + per-event success logs. +- Locking/concurrency/I/O foundations: contention anomalies and state changes; + prefer metrics for high-frequency worker/permit signals. +- Shared type/schema crates: log at the operational caller boundary unless the + crate itself owns the failure context. -- `rustfs` - - Role: top-level server, startup, auth, admin wiring, S3 request handling. - - Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems. -- `crates/protocols` - - Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers. - - Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures. -- `crates/madmin` - - Role: admin API contracts and management interfaces. - - Logging focus: admin action boundaries, validation failures, compatibility warnings. -- `crates/trusted-proxies` - - Role: forwarded IP trust, proxy chain validation, cloud metadata sources. - - Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries. -- `crates/keystone` - - Role: Keystone auth integration. - - Logging focus: integration enablement, upstream auth failures, config safety without credential leakage. +## Event Shape -### Storage, Healing, And Data Plane +Prefer stable fields in this order when available: -- `crates/ecstore` - - Role: erasure-coded storage implementation and peer/store initialization. - - Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise. -- `crates/heal` - - Role: healing orchestration and repair workflows. - - Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`. -- `crates/scanner` - - Role: data integrity scanning and health monitoring. - - Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`. -- `crates/object-capacity` - - Role: capacity scan and refresh core. - - Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter. -- `crates/filemeta` - - Role: file metadata parsing and helpers. - - Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads. -- `crates/storage-api` - - Role: storage contracts and shared data plane interfaces. - - Logging focus: contract mismatch and boundary diagnostics, usually low-volume. -- `crates/checksums` - - Role: checksum helpers and validation. - - Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs. -- `crates/zip` - - Role: ZIP handling and compression helpers. - - Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs. +1. `event` +2. `component` +3. `subsystem` +4. `state` or `result` +5. stable context such as mode, duration, reason, counts, safe identifiers, or + capacity/permit values +6. short message label -### Security, Identity, And Policy +Reuse the module's constants and neighboring field names. Do not create aliases +for the same concept. -- `crates/iam` - - Role: identity and access management. - - Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims. -- `crates/policy` - - Role: policy modeling and evaluation. - - Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps. -- `crates/credentials` - - Role: credential handling. - - Logging focus: never log secrets or tokens; only safe identifiers and redacted states. -- `crates/kms` - - Role: key management service integration. - - Logging focus: init/health/fallback, key-source availability, never log key material. -- `crates/crypto` - - Role: cryptographic helpers and security primitives. - - Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material. -- `crates/security-governance` - - Role: security governance contracts. - - Logging focus: policy/state transitions and enforcement diagnostics. -- `crates/signer` - - Role: request signing helpers. - - Logging focus: signature validation failures without expected-signature leakage. +## Patterns to Retire -### Notifications, Audit, And Targets +- sentence-style lifecycle announcements; +- startup banners and checklist lines; +- repetitive success logs at `info`/`debug`; +- raw inventories when an aggregate count is sufficient; +- fallback prose with values embedded in the message; +- `?value`/`Debug` output for credential-bearing or attacker-controlled data; +- logging a parse input when the malformed input may itself be a secret. -- `crates/notify` - - Role: notification dispatch, runtime facade, notifier implementations. - - Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam. -- `crates/audit` - - Role: audit target fan-out and audit pipeline management. - - Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose. -- `crates/targets` - - Role: target-specific configuration and utilities used by fan-out style systems. - - Logging focus: target selection, config validation, per-target degraded state. -- `crates/s3-types` - - Role: S3 event and type definitions. - - Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates. -- `crates/s3-ops` - - Role: S3 operation definitions and mapping. - - Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam. +## Guardrail Changes -### Concurrency, Locking, And Runtime Foundations +When expanding `scripts/check_logging_guardrails.sh`: -- `crates/concurrency` - - Role: timeout, locking, backpressure, and I/O scheduling facade. - - Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`. -- `crates/lock` - - Role: distributed locking implementation. - - Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics. -- `crates/tls-runtime` - - Role: shared TLS runtime foundation. - - Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps. -- `crates/obs` - - Role: observability helpers. - - Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries. -- `crates/io-core` - - Role: zero-copy I/O core primitives. - - Logging focus: keep very sparse; prefer metrics unless failures are actionable. -- `crates/io-metrics` - - Role: I/O metrics collection. - - Logging focus: typically minimal; metrics should carry the hot-path signal. -- `crates/rio` - - Role: Rust I/O utility layer. - - Logging focus: compatibility or runtime boundary failures, not fast-path internals. -- `crates/rio-v2` - - Role: next-generation I/O compatibility layer. - - Logging focus: migration/feature-mode differences and degraded fallback between I/O paths. -- `crates/utils` - - Role: shared helpers. - - Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary. -- `crates/common` - - Role: shared data structures and helpers. - - Logging focus: same principle as `utils`; prefer callers to log context-rich events. -- `crates/config` - - Role: configuration management. - - Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs. -- `crates/data-usage` - - Role: shared data usage models and algorithms. - - Logging focus: refresh lifecycle, summary stats, and degraded reads. +1. Add only files/patterns intentionally migrated in the same change. +2. Keep patterns concrete and grep-friendly. +3. Do not encode a style that remains valid elsewhere as a global ban. +4. Run the guardrail script and the root validation tier. +5. Treat the script as a floor; manually verify level, field shape, and privacy. -### Schema, Contracts, And API Support - -- `crates/protos` - - Role: protobuf definitions. - - Logging focus: usually none inside the crate; emit logs at decode/use boundaries. -- `crates/extension-schema` - - Role: extension schema contracts. - - Logging focus: schema validation and compatibility mismatches. -- `crates/s3select-api` - - Role: S3 Select API interfaces. - - Logging focus: request validation and unsupported feature boundaries. -- `crates/s3select-query` - - Role: S3 Select query engine. - - Logging focus: query parse/planning/execution failures, avoid row-level spam. -- `crates/protocols` - - Role: non-S3 protocol support. - - Logging focus: see core server section; keep per-request verbosity below `info`. - -### Testing And Non-Production Crates - -- `crates/e2e_test` - - Role: end-to-end tests. - - Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates. - -## Current Guardrail Coverage Map - -`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas: - -- `rustfs/src/main.rs` -- `rustfs/src/startup_iam.rs` -- `rustfs/src/auth.rs` -- `rustfs/src/protocols/client.rs` -- `crates/audit/src/pipeline.rs` -- `crates/audit/src/system.rs` -- `crates/audit/src/global.rs` -- `crates/notify/src/config_manager.rs` -- `crates/notify/src/runtime_facade.rs` -- `crates/notify/src/notifier.rs` -- `crates/ecstore/src/store/peer.rs` -- `crates/ecstore/src/store/init.rs` -- `crates/ecstore/src/tier/tier.rs` -- `crates/concurrency/src/workers.rs` -- `crates/concurrency/src/manager.rs` -- `crates/concurrency/src/lock.rs` -- `crates/concurrency/src/deadlock.rs` -- `crates/trusted-proxies/src/global.rs` -- `crates/trusted-proxies/src/config/loader.rs` -- `crates/trusted-proxies/src/proxy/metrics.rs` -- `crates/trusted-proxies/src/proxy/validator.rs` -- `crates/trusted-proxies/src/proxy/chain.rs` -- `crates/trusted-proxies/src/middleware/service.rs` -- `crates/trusted-proxies/src/cloud/detector.rs` -- `crates/trusted-proxies/src/cloud/ranges.rs` -- `crates/trusted-proxies/src/cloud/metadata/aws.rs` -- `crates/trusted-proxies/src/cloud/metadata/azure.rs` -- `crates/trusted-proxies/src/cloud/metadata/gcp.rs` - -When expanding coverage, prefer crates with: - -- repeated sentence-style lifecycle logs -- high-frequency success-path `info!` -- startup/config checklist banners -- security-sensitive fallback wording -- external fetch/retry/fallback flows - -That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`. - -## Event Model - -Prefer this structure when the fields are available: - -- `event` -- `component` -- `subsystem` -- `state` or `result` -- stable context fields such as: - - `enabled` - - `implementation` - - `validation_mode` - - `peer_ip` - - `client_ip` - - `proxy_hops` - - `duration_ms` - - `fallback` - - `reason` - - `range_count` - - `hold_time_ms` - - `available_slots` - - `total_slots` - - `permits_in_use` - -## Level Policy - -- `error`: the operation fails and callers or security guarantees are affected. -- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs. -- `info`: a low-frequency lifecycle or mode transition occurs. -- `debug`: useful diagnostics exist but normal operators do not need them all the time. -- `trace`: hot-path and repetitive success-path details occur. - -## Preferred Patterns - -- Use a short message label: - - `"trusted proxy validation failed"` - - `"concurrency manager state changed"` - - `"trusted proxy cloud metadata loaded"` -- Put key meaning into fields, not only the message text. -- Aggregate config or metrics summaries into one log event. - -## Retired Patterns - -These should usually be removed or replaced: - -- Sentence-style lifecycle logs: - - `info!("Concurrency manager stopped")` - - `info!("Trusted Proxies module initialized")` -- Checklist or banner logs: - - `info!("=== Application Configuration ===")` - - `info!("Available metrics:")` -- Hot-path noise: - - `info!("worker take, {}", *available)` - - `debug!("Proxy validation successful in {:?}", duration)` -- Legacy fallback prose: - - `"Request from private network but not trusted: ..."` - - `"Cloud metadata fetching is disabled"` - -## Guardrail Update Checklist - -When extending `scripts/check_logging_guardrails.sh`: - -1. Add the touched files to `checked_files`. -2. Add only legacy patterns that have been intentionally retired. -3. Keep patterns literal and grep-friendly. -4. Run the guardrail script after changes. -5. Avoid adding patterns for logs that are still valid elsewhere in the repo. - -## Validation Checklist - -For logging-only changes: +Useful search seeds for the changed surface: ```bash -cargo fmt --all --check -./scripts/check_logging_guardrails.sh -cargo check -p -cargo test -p -``` - -For broader Rust changes: - -```bash -./scripts/check_unsafe_code_allowances.sh -./scripts/check_architecture_migration_rules.sh -cargo clippy -p --all-targets -- -D warnings +rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' +rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' ``` diff --git a/.agents/skills/rustfs-release-publish/SKILL.md b/.agents/skills/rustfs-release-publish/SKILL.md index 9bf8ab297..e38cfc587 100644 --- a/.agents/skills/rustfs-release-publish/SKILL.md +++ b/.agents/skills/rustfs-release-publish/SKILL.md @@ -1,6 +1,6 @@ --- name: rustfs-release-publish -description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)." +description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)." --- # RustFS Release Publish (preview-validated pipeline) diff --git a/.agents/skills/rustfs-release-version-bump/SKILL.md b/.agents/skills/rustfs-release-version-bump/SKILL.md index 3c760f409..d184ea27a 100644 --- a/.agents/skills/rustfs-release-version-bump/SKILL.md +++ b/.agents/skills/rustfs-release-version-bump/SKILL.md @@ -1,6 +1,6 @@ --- name: rustfs-release-version-bump -description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation." +description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow." --- # RustFS Release Version Bump @@ -81,10 +81,7 @@ Only drop a file when the current repository release process clearly no longer r 4. Verify before shipping - Run: -- `cargo fmt --all` -- `cargo fmt --all --check` - `make pre-commit` -- If verification passes, run `cargo clean`. - If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks. 5. Commit strategy @@ -109,10 +106,7 @@ Only drop a file when the current repository release process clearly no longer r - `git diff --name-only origin/main...HEAD` - `git diff --stat origin/main...HEAD` - `rg -n "|" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec` -- `cargo fmt --all` -- `cargo fmt --all --check` - `make pre-commit` -- `cargo clean` ## Output contract diff --git a/.agents/skills/rustfs-release-version-bump/agents/openai.yaml b/.agents/skills/rustfs-release-version-bump/agents/openai.yaml index 3483dedcc..4bd3e0ff6 100644 --- a/.agents/skills/rustfs-release-version-bump/agents/openai.yaml +++ b/.agents/skills/rustfs-release-version-bump/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "RustFS Release Bump" short_description: "Prepare RustFS release branches like PR #2957." - default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow." + default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump." diff --git a/.agents/skills/security-advisory-lessons/SKILL.md b/.agents/skills/security-advisory-lessons/SKILL.md index 492669f44..99a43fcc7 100644 --- a/.agents/skills/security-advisory-lessons/SKILL.md +++ b/.agents/skills/security-advisory-lessons/SKILL.md @@ -1,170 +1,40 @@ --- name: security-advisory-lessons -description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required. +description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path. --- # RustFS Security Advisory Lessons -Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md). +Use this skill as the deep security lens. For a normal adversarial review with a +matched security surface, the concise security reference under +`adversarial-validation` is sufficient. -When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror: +## Workflow + +1. Freeze the exact diff/head and identify the changed trust boundaries. +2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only + the matching sections. Useful headings are + auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths, + secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde. +3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed, + and default-config cases. Security decisions must fail closed. +4. Require a focused negative regression test for the bypass/exploit form, not + only the intended success path. State residual risk when a test is impractical. +5. Report proven vulnerabilities separately from defense-in-depth hardening. + +When advisory currentness matters, fetch the live inventory instead of treating +the reference as a status mirror: ```bash gh api repos/rustfs/rustfs/security-advisories --paginate \ --jq '.[] | {ghsa_id,state,severity,summary,updated_at}' ``` -Fetch full advisory details only when the live summary suggests a new or changed lesson: +Fetch an individual advisory only when the live summary indicates a new or +changed lesson. -```bash -gh api repos/rustfs/rustfs/security-advisories/ -``` +## Finding Standard -For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md). - -## Workflow - -### 1. Scope the change -- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks. -- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code. - -### 2. Map to advisory classes -- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons. -- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening. - -### 3. Verify fail-closed behavior -- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly. -- Prefer exact action/permission checks over broad helper calls or inferred ownership. -- Confirm lower storage/RPC layers do not bypass checks done in upper layers. - -### 4. Require regression evidence -- For behavior changes, add focused negative tests that reproduce the advisory class. -- For sensitive fixes, include tests for the bypass form, not only the happy path. -- If a test is impractical, explain the residual risk and provide a manual verification command. - -### 5. Report clearly -- Lead with concrete findings and file/line evidence. -- Separate proven vulnerabilities from hardening risks. -- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass. - -## Advisory-Derived Guardrails - -### Auth and admin authorization -- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree. -- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class. -- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`. -- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough. -- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers. -- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action. - -### IAM and service accounts -- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups. -- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`. -- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed. -- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks. -- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities. - -### STS, OIDC, and federation flows -- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials. -- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks. -- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret. -- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials. -- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin. -- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks. - -### IAM policy conditions and plugins -- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims. -- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available. -- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs. -- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision. - -### S3 object actions, copy, multipart, and presigned POST -- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`. -- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions. -- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization. -- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow. -- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions. - -### Protocol frontends and IAM parity -- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization. -- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`. -- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver. -- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command. -- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints. - -### Paths, object keys, and filesystem access -- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks. -- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root. -- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths. -- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries. - -### Secrets, default credentials, and crypto -- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys. -- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary. -- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other. -- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity. -- Plan key rotation and key IDs when removing exposed keys. - -### Logging and debug output -- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials. -- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces. -- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string. -- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel. -- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies. - -### RPC, parsing, and panic safety -- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors. -- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures. -- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe. -- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call. - -### Browser, CORS, and console surfaces -- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured. -- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript. -- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`. -- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`. -- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps. - -### Profiling, debug, and health endpoints -- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses. -- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses. -- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics. - -### Trusted proxy and network identity -- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy. -- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`. -- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation. -- Add tests for direct spoofed headers and trusted-proxy headers. - -### SSE and storage invariants -- Encryption metadata is not proof that bytes were encrypted on disk. -- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests. -- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers. - -### Object Lock and retention invariants -- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes. -- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result. -- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry. - -## Review Prompts - -Use these prompts while reviewing a diff: - -- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object? -- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage? -- Does a public/default/empty config change security behavior from fail-closed to fail-open? -- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body? -- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization? -- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input? -- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority? -- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling? -- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority? -- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary? -- Is an archive entry, object key, or policy resource normalized differently between authorization and storage? -- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract? -- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path? -- Can a caller-controlled header populate a condition key that should be derived only by the server? -- Do condition tests include partially overlapping multi-value inputs for quantified operators? -- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear? -- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection? -- Does the test prove the exploit form is denied, or only that the intended form still works? +Each finding includes severity, `file:line`, attacker prerequisites, concrete +input/path, impact, smallest safe fix, and a regression check. Do not exaggerate +unauthenticated impact when the actual issue requires authenticated low privilege. diff --git a/.agents/skills/security-advisory-lessons/agents/openai.yaml b/.agents/skills/security-advisory-lessons/agents/openai.yaml index 8bfb41d7a..a90ec68ac 100644 --- a/.agents/skills/security-advisory-lessons/agents/openai.yaml +++ b/.agents/skills/security-advisory-lessons/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Security Advisory Lessons" short_description: "Apply advisory lessons in reviews." - default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes." + default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories." diff --git a/.config/e2e-full-selection.txt b/.config/e2e-full-selection.txt new file mode 100644 index 000000000..dfad0f0bd --- /dev/null +++ b/.config/e2e-full-selection.txt @@ -0,0 +1,2 @@ +sha256-darwin=9f767b37ed8b1c82da62ea441462d75487785c8086e56f08fb6f6cd89c6e2e52 +sha256-linux=fbdaf42b220958d4b1e8880e0f8b5a7992d38e21051bb60596dd4538424757d6 diff --git a/.config/e2e-nightly-selection.txt b/.config/e2e-nightly-selection.txt new file mode 100644 index 000000000..9d929e2ee --- /dev/null +++ b/.config/e2e-nightly-selection.txt @@ -0,0 +1 @@ +sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680 diff --git a/.config/e2e-protocols-selection.txt b/.config/e2e-protocols-selection.txt new file mode 100644 index 000000000..b0c07f708 --- /dev/null +++ b/.config/e2e-protocols-selection.txt @@ -0,0 +1,2 @@ +sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315 +sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17 diff --git a/.config/e2e-repl-nightly-selection.txt b/.config/e2e-repl-nightly-selection.txt new file mode 100644 index 000000000..d78fd611b --- /dev/null +++ b/.config/e2e-repl-nightly-selection.txt @@ -0,0 +1 @@ +sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7 diff --git a/.config/e2e-smoke-selection.txt b/.config/e2e-smoke-selection.txt new file mode 100644 index 000000000..7ca219cf1 --- /dev/null +++ b/.config/e2e-smoke-selection.txt @@ -0,0 +1 @@ +sha256=ec27cde6ce6400723c4b372bfbd2ac61709c744294e4810af765e8a808d8e31d diff --git a/.config/make/lint-fmt.mak b/.config/make/lint-fmt.mak index 55ed1efc6..dda47ae88 100644 --- a/.config/make/lint-fmt.mak +++ b/.config/make/lint-fmt.mak @@ -75,6 +75,11 @@ embedded-secrets-check: ## Check no private key material or credential literal i @echo "🔑 Checking embedded secret material guard..." ./scripts/check_embedded_secrets.sh +.PHONY: test-wiring-check +test-wiring-check: ## Check tests stay registered and selected by their intended runners + @echo "🧪 Checking test wiring..." + python3 ./scripts/check_test_wiring.py + .PHONY: log-analyzer-rules-check log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source @echo "🩺 Checking log-analyzer rule anchors..." diff --git a/.config/make/pre-commit.mak b/.config/make/pre-commit.mak index b4ba093b0..54b54e34c 100644 --- a/.config/make/pre-commit.mak +++ b/.config/make/pre-commit.mak @@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed ./scripts/check_no_planning_docs.sh .PHONY: pre-commit -pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests +pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests @echo "✅ All pre-commit checks passed!" .PHONY: pre-pr -pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests +pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests @echo "✅ All pre-PR checks passed!" .PHONY: dev-check -dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks +dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks @echo "✅ Fast development checks passed!" diff --git a/.config/make/tests.mak b/.config/make/tests.mak index 937a22aca..626df9b84 100644 --- a/.config/make/tests.mak +++ b/.config/make/tests.mak @@ -35,6 +35,9 @@ script-tests: ## Run shell script tests ./scripts/test_pinned_paired_abba_bench.sh ./scripts/test_manual_transition_runbooks.sh ./scripts/check_embedded_secrets.sh --self-test + python3 ./scripts/check_test_wiring.py --self-test + python3 ./scripts/check_scheduled_validation_freshness.py --self-test + python3 ./scripts/s3-tests/test_report_compat.py bash -n ./scripts/validate_object_data_cache_cold_stampede.sh python3 ./scripts/check_object_data_cache_follower_samples.py --self-test ./scripts/validate_object_data_cache_cold_stampede.sh --self-test diff --git a/.config/nextest.toml b/.config/nextest.toml index 4dd2d1d0e..097c79e3e 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -38,10 +38,11 @@ e2e-vault = { max-threads = 1 } # replacement_privileged_e2e_test when explicitly run as root on Linux). They # are correct in isolation but resource-heavy; serialize them under nextest's # process boundary (serial_test's #[serial] does not cross it) so several 4-disk -# servers never run at once. ci-7's nightly picks these up via the e2e suite; +# servers never run at once. The e2e-full merge/main lane picks these up; # they are deliberately NOT in the fast PR `e2e-smoke` filter. e2e-reliability = { max-threads = 1 } e2e-inline-boundaries = { max-threads = 1 } +e2e-cluster-nightly = { max-threads = 1 } # --- default profile (local): serialize the flaky groups, never retry -------- [[profile.default.overrides]] @@ -161,7 +162,7 @@ retries = 2 # Serialize the 4-disk reliability / degraded-read e2e tests under the ci # profile too (see the e2e-reliability test-group note near the top). Not a # quarantine: no retries, just single-threaded so several 4-disk servers never -# run concurrently when ci-7's nightly runs the full e2e suite. +# run concurrently when e2e-full runs the suite. [[profile.ci.overrides]] filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)' test-group = 'e2e-reliability' @@ -230,8 +231,8 @@ test-group = 'ecstore-serial-flaky' # the nightly profile derives its set as "the replication module MINUS this # allowlist", so any new replication test lands in nightly by default (never # silently unrun) until it is explicitly blessed as fast here. Keep the two -# regexes byte-identical. Count invariant: 20 here + 49 nightly = 69 total -# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md). +# regexes byte-identical. The committed profile selection digests make changes +# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md. # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane # (#4724) because they set a loopback (127.0.0.1) replication target that the # SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed — @@ -327,9 +328,8 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" } # the STS dual-node test actually exercises its path (it skips gracefully with # a visible log line when awscurl is absent), and routes scheduled failures # through .github/actions/schedule-failure-issue (ci-8). Explicit division of -# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not -# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci -# domain's consolidated scheduled e2e workflow once it exists. +# labor with e2e-full: these tests run only in the consolidated nightly +# workflow, not in the merge/main lane. [profile.e2e-repl-nightly] default-filter = """ package(e2e_test) @@ -343,26 +343,60 @@ fail-fast = false # workflow as the failure-triage artifact. path = "junit.xml" +# --------------------------------------------------------------------------- +# e2e-nightly profile — destructive multi-process cluster fault domains +# --------------------------------------------------------------------------- +# These seven modules are deliberately outside e2e-full's merge budget. Each +# starts a real multi-process or multi-disk topology and exercises node/disk +# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The +# consolidated nightly workflow runs them serially to avoid resource +# starvation; failures are never retried. +[profile.e2e-nightly] +default-filter = """ + package(e2e_test) + & test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/) +""" +fail-fast = false + +[profile.e2e-nightly.junit] +path = "junit.xml" + +[[profile.e2e-nightly.overrides]] +filter = 'package(e2e_test)' +test-group = 'e2e-cluster-nightly' + +# --------------------------------------------------------------------------- +# e2e-protocols profile — serial protocol lane +# --------------------------------------------------------------------------- +# The suite owns fixed ports, so the nightly workflow runs this exact profile +# with one nextest worker. +[profile.e2e-protocols] +default-filter = 'package(e2e_test) & test(/^protocols::/)' +fail-fast = false + +[profile.e2e-protocols.junit] +path = "junit.xml" + # --------------------------------------------------------------------------- # e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5) # --------------------------------------------------------------------------- # The merge gate (ci.yml `e2e-full` job: push main + merge_group + -# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40), -# object_lock (33), multipart_auth (109), quota, checksum, encryption, +# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth, +# quota, checksum, encryption, # security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately # skips. Budget <= 45 min; authority for the suite count is `cargo nextest list # --profile e2e-full` (see docs/testing/e2e-suite-inventory.md). # # The filter is "the whole e2e_test crate MINUS the sets owned by other lanes": -# * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed -# ports; they join a scheduled lane once ci-6 randomises the ports (ci-7). +# * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile +# with one worker because the suite owns fixed ports. # * the 7 cluster suites that spin up a RustFSTestClusterEnvironment # (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster, # namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression, -# object_lambda) — too heavy for the merge budget; they run in ci-7's -# nightly 4-node lane. +# object_lambda) — too heavy for the merge budget; they run in the +# e2e-nightly serial cluster-fault lane. # * replication_extension_test — repl-1 already splits it into the PR -# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (49 slow) lanes and reserves +# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (55 slow) lanes and reserves # it for those, so e2e-full does not double-run it. # * #[ignore]d tests — nextest skips them by default (no --run-ignored); the # manual-localhost:9000 reliant/policy tests are ci-13's migration. diff --git a/.github/actions/schedule-failure-issue/action.yml b/.github/actions/schedule-failure-issue/action.yml index 60e938690..d505387e4 100644 --- a/.github/actions/schedule-failure-issue/action.yml +++ b/.github/actions/schedule-failure-issue/action.yml @@ -14,9 +14,10 @@ name: "Schedule Failure Issue" description: >- - Open (or update) a tracking issue when a scheduled workflow run fails. + Open (or update) a tracking issue when a scheduled workflow run fails or + does not complete normally. Dedupes by workflow name: if an open issue titled - "[scheduled-failure] " already exists, the failure is + "[scheduled-failure] " already exists, the result is appended as a comment; otherwise a new issue is created. This is the single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8). @@ -38,6 +39,30 @@ inputs: Set to an empty string to skip labeling. required: false default: "infrastructure" + source-run-id: + description: "Run ID to report. Defaults to the current workflow run." + required: false + default: ${{ github.run_id }} + source-run-attempt: + description: "Run attempt to report. Defaults to the current attempt." + required: false + default: ${{ github.run_attempt }} + source-event: + description: "Trigger event of the run being reported." + required: false + default: ${{ github.event_name }} + source-ref-name: + description: "Ref name of the run being reported." + required: false + default: ${{ github.ref_name }} + source-sha: + description: "Commit SHA of the run being reported." + required: false + default: ${{ github.sha }} + details-file: + description: "Optional Markdown file appended to the issue body." + required: false + default: "" runs: using: "composite" @@ -48,17 +73,22 @@ runs: GH_TOKEN: ${{ inputs.github-token }} WORKFLOW_NAME: ${{ inputs.workflow-name }} ISSUE_LABEL: ${{ inputs.label }} + SOURCE_RUN_ID: ${{ inputs.source-run-id }} + SOURCE_RUN_ATTEMPT: ${{ inputs.source-run-attempt }} + SOURCE_EVENT: ${{ inputs.source-event }} + SOURCE_REF_NAME: ${{ inputs.source-ref-name }} + SOURCE_SHA: ${{ inputs.source-sha }} + DETAILS_FILE: ${{ inputs.details-file }} run: | set -euo pipefail title="[scheduled-failure] ${WORKFLOW_NAME}" - run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}" - # Failed job names for this run attempt. The alert job runs while the - # run as a whole is still in progress, so inspect the jobs that have - # already completed with a non-success conclusion. + # Inspect the reported run attempt. It can be the current in-workflow + # failure or a completed run observed by the external watchdog. failed_jobs="$(gh api \ - "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs" \ + "repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/attempts/${SOURCE_RUN_ATTEMPT}/jobs" \ --paginate \ --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled") @@ -67,15 +97,26 @@ runs: failed_jobs="- (failed job not recorded yet — see the run page)" fi + details="" + if [ -n "${DETAILS_FILE}" ]; then + if [ -f "${DETAILS_FILE}" ]; then + details="$(cat "${DETAILS_FILE}")" + else + details="Details file was not available: \`${DETAILS_FILE}\`" + fi + fi + body="$(cat <- + always() && github.event_name == 'schedule' && + (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Open or update failure-tracking issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/cache-warm.yml b/.github/workflows/cache-warm.yml index a9902fc4d..660d96b09 100644 --- a/.github/workflows/cache-warm.yml +++ b/.github/workflows/cache-warm.yml @@ -94,6 +94,9 @@ concurrency: env: CARGO_TERM_COLOR: always + # Swatinem/rust-cache hashes every RUST* variable. Keep this aligned with + # ci.yml or the writer and readers use disjoint cache keys. + RUST_BACKTRACE: 1 jobs: # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary, @@ -101,7 +104,7 @@ jobs: warm-ci-dev: name: Warm ci-dev runs-on: sm-standard-4 - timeout-minutes: 90 + timeout-minutes: 120 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: @@ -191,7 +194,7 @@ jobs: warm-ci-feat-rio: name: Warm ci-feat-rio runs-on: sm-standard-4 - timeout-minutes: 90 + timeout-minutes: 120 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: @@ -219,7 +222,7 @@ jobs: warm-ci-feat-proto: name: Warm ci-feat-proto runs-on: sm-standard-4 - timeout-minutes: 90 + timeout-minutes: 120 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: diff --git a/.github/workflows/ci-docs-only.yml b/.github/workflows/ci-docs-only.yml index 89ae4d18f..de78e7f7f 100644 --- a/.github/workflows/ci-docs-only.yml +++ b/.github/workflows/ci-docs-only.yml @@ -125,6 +125,12 @@ jobs: - name: Check no embedded secret material run: ./scripts/check_embedded_secrets.sh + - name: Check test wiring + run: | + python3 ./scripts/check_test_wiring.py --self-test + python3 ./scripts/check_scheduled_validation_freshness.py --self-test + python3 ./scripts/check_test_wiring.py + - name: Check no planning docs committed run: ./scripts/check_no_planning_docs.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4812b118f..ae74ec40a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -59,7 +59,7 @@ on: merge_group: types: [ checks_requested ] schedule: - - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC + - cron: "11 0 * * 0" # Weekly on Sunday 00:11 UTC workflow_dispatch: permissions: @@ -160,6 +160,12 @@ jobs: - name: Check no embedded secret material run: ./scripts/check_embedded_secrets.sh + - name: Check test wiring + run: | + python3 ./scripts/check_test_wiring.py --self-test + python3 ./scripts/check_scheduled_validation_freshness.py --self-test + python3 ./scripts/check_test_wiring.py + - name: Check no planning docs committed run: ./scripts/check_no_planning_docs.sh @@ -397,7 +403,7 @@ jobs: if: github.event_name != 'pull_request' || github.event.action != 'closed' needs: [ quick-checks ] runs-on: sm-standard-4 - timeout-minutes: 45 + timeout-minutes: 90 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: @@ -437,7 +443,7 @@ jobs: if: github.event_name != 'pull_request' || github.event.action != 'closed' needs: [ quick-checks ] runs-on: sm-standard-4 - timeout-minutes: 60 + timeout-minutes: 90 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: @@ -467,7 +473,7 @@ jobs: if: github.event_name != 'pull_request' || github.event.action != 'closed' needs: [ quick-checks ] runs-on: sm-standard-4 - timeout-minutes: 60 + timeout-minutes: 90 strategy: # On a PR, one failing protocol leg is enough to know the PR is not ready, # so stop the sibling leg instead of paying another ~40 minutes for it. @@ -686,9 +692,9 @@ jobs: - name: Make binary executable run: chmod +x ./target/debug/rustfs - # Build the e2e test graph once. The archive is reused by the security - # count-floor check and the smoke run below, avoiding a second compile of - # the same e2e_test target on cold runners (backlog#1645). + # Build the e2e test graph once. The archive is reused by the smoke + # selection guard, security exact-count check, and run below, avoiding a + # second compile of the same e2e_test target on cold runners (backlog#1645). - name: Archive e2e smoke test binaries env: NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst @@ -696,6 +702,7 @@ jobs: run: | cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}" cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}" + python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}" ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}" # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The @@ -760,7 +767,7 @@ jobs: # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption, # security-boundary, ... — via the e2e-full nextest profile. Too heavy for # every PR, so it is gated to main pushes, the merge queue, and manual - # dispatch. protocols / the 6 cluster suites / replication / #[ignore] are + # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are # owned by other lanes (see .config/nextest.toml profile.e2e-full). if: >- github.event_name == 'workflow_dispatch' || @@ -820,6 +827,13 @@ jobs: - name: Make binary executable run: chmod +x ./target/debug/rustfs + - name: Verify e2e full membership + env: + NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-full-list.json + run: | + cargo nextest list --profile e2e-full -p e2e_test --message-format json > "${NEXTEST_LISTING}" + python3 ./scripts/check_test_wiring.py --check-profile e2e-full "${NEXTEST_LISTING}" + # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full # default-filter in .config/nextest.toml is the single wiring mechanism — # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded @@ -832,7 +846,9 @@ jobs: uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: name: e2e-full-junit-${{ github.run_number }} - path: target/nextest/e2e-full/junit.xml + path: | + target/nextest/e2e-full/junit.xml + ${{ runner.temp }}/rustfs-e2e-full-list.json retention-days: 7 e2e-tests-rio-v2: @@ -1019,3 +1035,37 @@ jobs: path: artifacts/s3tests-single/** if-no-files-found: ignore retention-days: 3 + + alert-on-failure: + name: Alert on scheduled failure + needs: + - typos + - quick-checks + - test-and-lint + - test-ilm-integration-serial + - test-and-lint-rio-v2 + - test-and-lint-protocols + - build-rustfs-debug-binary + - build-rustfs-debug-binary-rio-v2 + - uring-integration + - e2e-tests + - e2e-full + - e2e-tests-rio-v2 + - s3-implemented-tests + - s3-lifecycle-behavior-tests + if: >- + always() && github.event_name == 'schedule' && + (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Open or update failure-tracking issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index baa3c87a7..ece00c2eb 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -37,7 +37,7 @@ on: # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17), # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes. - - cron: "0 7 * * 0" + - cron: "43 7 * * 0" # Only alert-on-failure needs more than read access; it declares its own # job-level `issues: write`. diff --git a/.github/workflows/e2e-replication-nightly.yml b/.github/workflows/e2e-replication-nightly.yml index 2dde9f2b3..145ad317e 100644 --- a/.github/workflows/e2e-replication-nightly.yml +++ b/.github/workflows/e2e-replication-nightly.yml @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4). +# Consolidated nightly e2e lane for replication, cluster faults, and protocols. # # The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the # FAST replication tests. This scheduled lane runs the remaining heavier @@ -28,28 +28,29 @@ # add ad-hoc cargo-test steps here; change the filterset instead. The # authoritative membership and count come from # `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly -# count invariant is maintained next to the filtersets in .config/nextest.toml -# (deliberately not duplicated here). +# selection digest is committed under .config/. # -# Explicit division of labor: the nightly subset runs ONLY here, never double-run -# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's -# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane -# into it rather than growing a second scheduled entrypoint. +# Explicit division of labor: these subsets run only here and never double-run +# in the e2e-full merge gate. -name: e2e-replication-nightly +name: e2e-nightly on: workflow_dispatch: schedule: # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00), # stale (01:30) and performance-ab (06:00). - - cron: "0 4 * * *" + - cron: "29 4 * * *" # Only alert-on-failure needs more than read access; it declares its own # job-level `issues: write`. permissions: contents: read +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: false + jobs: repl-nightly: name: Replication e2e (nightly) @@ -97,9 +98,20 @@ jobs: # demand otherwise, but a single explicit build avoids several parallel # nextest test processes racing to build it at once. - name: Build rustfs binary - run: cargo build -p rustfs --bins + run: | + cargo build -p rustfs --bins + : > target/debug/rustfs.features + + - name: Verify replication e2e membership + env: + NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json + run: | + cargo nextest list --profile e2e-repl-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}" + python3 ./scripts/check_test_wiring.py --check-profile e2e-repl-nightly "${NEXTEST_LISTING}" - name: Run replication e2e nightly suite + env: + RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs run: cargo nextest run --profile e2e-repl-nightly -p e2e_test - name: Upload nextest junit report @@ -107,13 +119,115 @@ jobs: uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: name: e2e-replication-nightly-junit-${{ github.run_number }} - path: target/nextest/e2e-repl-nightly/junit.xml + path: | + target/nextest/e2e-repl-nightly/junit.xml + ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json + ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs/ retention-days: 7 if-no-files-found: ignore + cluster-nightly: + name: Cluster fault e2e (nightly) + runs-on: sm-standard-4 + timeout-minutes: 90 + env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Setup Rust environment + uses: ./.github/actions/setup + with: + rust-version: stable + cache-shared-key: ci-e2e-nightly + cache-save-if: 'false' + install-build-packaging-tools: 'false' + + - name: Build rustfs binary + run: | + cargo build -p rustfs --bins --features e2e-test-hooks + : > target/debug/rustfs.features + + - name: Verify cluster fault e2e membership + env: + NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json + run: | + cargo nextest list --profile e2e-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}" + python3 ./scripts/check_test_wiring.py --check-profile e2e-nightly "${NEXTEST_LISTING}" + + - name: Run cluster fault e2e nightly suite + env: + RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs + run: cargo nextest run --profile e2e-nightly -p e2e_test + + - name: Upload cluster fault diagnostics + if: always() + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: e2e-cluster-nightly-${{ github.run_number }} + path: | + target/nextest/e2e-nightly/junit.xml + ${{ runner.temp }}/rustfs-e2e-nightly-list.json + ${{ runner.temp }}/rustfs-e2e-nightly-logs/ + retention-days: 7 + if-no-files-found: warn + + protocols-nightly: + name: Protocol e2e (nightly) + runs-on: sm-standard-4 + timeout-minutes: 90 + env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + RUSTFS_BUILD_FEATURES: ftps,webdav,sftp + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Setup Rust environment + uses: ./.github/actions/setup + with: + rust-version: stable + cache-shared-key: ci-e2e-protocols + cache-save-if: 'false' + install-build-packaging-tools: 'false' + + - name: Verify protocol socket oracle + run: ss -tn state CLOSE-WAIT >/dev/null + + # The suite owns fixed protocol ports and serializes its internal cases. + - name: Verify protocol e2e membership + env: + NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-protocols-list.json + run: | + cargo nextest list --profile e2e-protocols -p e2e_test --message-format json > "${NEXTEST_LISTING}" + python3 ./scripts/check_test_wiring.py --check-profile e2e-protocols "${NEXTEST_LISTING}" + + - name: Run protocol e2e nightly suite + env: + RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs + run: >- + cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture + + - name: Upload protocol diagnostics + if: always() + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: e2e-protocol-nightly-${{ github.run_number }} + path: | + target/nextest/e2e-protocols/junit.xml + ${{ runner.temp }}/rustfs-e2e-protocols-list.json + ${{ runner.temp }}/rustfs-protocol-e2e-logs/ + retention-days: 7 + if-no-files-found: warn + alert-on-failure: name: Alert on scheduled failure - needs: [repl-nightly] + needs: [repl-nightly, cluster-nightly, protocols-nightly] # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8); # manual workflow_dispatch runs stay quiet so a debugging run never files a # spurious alert. diff --git a/.github/workflows/e2e-s3tests.yml b/.github/workflows/e2e-s3tests.yml index f8eeaec10..1be61d73e 100644 --- a/.github/workflows/e2e-s3tests.yml +++ b/.github/workflows/e2e-s3tests.yml @@ -18,10 +18,9 @@ # runs only the implemented_tests.txt whitelist. This workflow complements it: # # - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream -# suite and reports promotion candidates (tests that newly pass) and -# unclassified tests. The job fails only on regressions in the implemented -# whitelist or on infrastructure errors — expected failures from -# not-yet-implemented features do not turn the run red. +# suite and reports promotion candidates. Regressions, unclassified tests, +# incomplete execution, and infrastructure errors fail the job; classified +# failures for not-yet-implemented features remain informational. # - Manual runs (workflow_dispatch): same, with configurable mode/scope. # # All test execution is delegated to scripts/s3-tests/run.sh (single source of @@ -45,13 +44,6 @@ # The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker # via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap. -# DISABLED. This workflow is switched off in the repository's Actions settings -# (state: disabled_manually) and does not run on any trigger, including its cron -# and workflow_dispatch. That state lives in GitHub's UI and is invisible when -# reading this file, which has already misled at least one audit — hence this -# banner. Re-enabling is a UI action; anyone doing so should first check that the -# workflow still matches the current CI layout. See rustfs/backlog#1603. -# name: e2e-s3tests on: @@ -81,14 +73,31 @@ on: description: "Stop after N failures. '0' to run everything." required: false default: "0" + shard-count: + description: "Exact-node-ID shard count for a targeted manual run" + required: false + default: "1" + type: choice + options: + - "1" + - "2" + - "4" + shard-index: + description: "Zero-based shard index for a targeted manual run" + required: false + default: "0" markexpr: description: "Optional pytest -m expression" required: false default: "" + testexpr: + description: "Optional pytest -k expression" + required: false + default: "" schedule: # Weekly full sweep (Sunday 02:00 UTC): full suite, run against BOTH the # single-node and the 4-node distributed topologies (matrix below). - - cron: "0 2 * * 0" + - cron: "19 2 * * 0" env: # main user @@ -111,6 +120,9 @@ env: XDIST: ${{ github.event.inputs.xdist || '4' }} MAXFAIL: ${{ github.event.inputs.maxfail || '0' }} MARKEXPR: ${{ github.event.inputs.markexpr || '' }} + TESTEXPR: ${{ github.event.inputs.testexpr || '' }} + S3_SHARD_COUNT: ${{ github.event_name == 'schedule' && '4' || github.event.inputs.shard-count || '1' }} + TEST_TIMEOUT: "300" concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }} @@ -127,19 +139,22 @@ defaults: jobs: s3tests: + name: s3tests (${{ matrix.test-mode }}, shard ${{ matrix.shard-index }}) # GitHub-hosted: reliably provides Docker + docker compose + python3/pip. # See the header note (ci-1) for why the self-hosted sm-standard-4 label - # was abandoned. TODO(ci-8): scheduled-failure alerting (auto-open issue) - # is added by the ci-8 composite action; do not implement it here. + # was abandoned. Scheduled failures are handled by alert-on-failure below. runs-on: ubuntu-latest timeout-minutes: 180 strategy: fail-fast: false + max-parallel: 2 matrix: # Scheduled sweeps cover both topologies; manual runs use the input. test-mode: ${{ github.event_name == 'schedule' && fromJSON('["single", "multi"]') || fromJSON(format('["{0}"]', github.event.inputs.test-mode || 'single')) }} + shard-index: ${{ github.event_name == 'schedule' && fromJSON('[0, 1, 2, 3]') || fromJSON(format('[{0}]', github.event.inputs.shard-index || '0')) }} env: TEST_MODE: ${{ matrix.test-mode }} + S3_SHARD_INDEX: ${{ matrix.shard-index }} steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 with: @@ -181,6 +196,7 @@ jobs: - name: Start single RustFS if: env.TEST_MODE == 'single' run: | + SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)" docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net docker rm -f rustfs-single >/dev/null 2>&1 || true # The four disks share one physical device on the runner (a single @@ -193,6 +209,7 @@ jobs: -e RUSTFS_ADDRESS=0.0.0.0:9000 \ -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \ -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \ + -e RUSTFS_SSE_S3_MASTER_KEY="${SSE_KEY}" \ -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \ -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \ -v /tmp/rustfs-single:/data \ @@ -201,6 +218,7 @@ jobs: - name: Start 4-node distributed cluster if: env.TEST_MODE == 'multi' run: | + SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)" # A real distributed deployment: every node lists all endpoints in # RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use # node-local volume paths here — that would create four independent @@ -213,6 +231,7 @@ jobs: RUSTFS_ADDRESS: "0.0.0.0:9000" RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} + RUSTFS_SSE_S3_MASTER_KEY: "${SSE_KEY}" RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}" # Each node's four disks share one physical device inside its # container, so bypass the local physical-disk-independence guard @@ -255,14 +274,20 @@ jobs: EOF cat > haproxy.cfg <<'EOF' + global + log stdout format raw local0 info + defaults mode http + log global + log-format '%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %tsc %HM %HP' timeout connect 5s timeout client 30s timeout server 30s frontend fe_s3 bind *:9000 + option http-buffer-request default_backend be_s3 backend be_s3 @@ -294,34 +319,14 @@ jobs: - name: Run ceph s3-tests run: | - set +e DEPLOY_MODE=existing \ TEST_MODE="${TEST_MODE}" \ TEST_SCOPE="${TEST_SCOPE}" \ XDIST="${XDIST}" \ MAXFAIL="${MAXFAIL}" \ MARKEXPR="${MARKEXPR}" \ + TESTEXPR="${TESTEXPR}" \ ./scripts/s3-tests/run.sh - RC=$? - set -e - - if [ "${TEST_SCOPE}" = "implemented" ]; then - # Whitelist run: every failure is a regression. - exit "${RC}" - fi - - # Full sweep: failures outside the implemented whitelist are - # inventory (promotion candidates / unimplemented features), not a - # gate. Fail only on whitelist regressions or infrastructure errors. - JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml" - if [ ! -f "${JUNIT}" ]; then - echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2 - exit "${RC}" - fi - python3 scripts/s3-tests/report_compat.py \ - --junit "${JUNIT}" \ - --lists-dir scripts/s3-tests \ - --fail-on-regression - name: Publish compatibility report if: always() @@ -346,7 +351,7 @@ jobs: if: always() && env.ACT != 'true' uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: - name: s3tests-${{ env.TEST_MODE }} + name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }} path: artifacts/** alert-on-failure: diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index f2967804f..6aa780fc6 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -12,30 +12,25 @@ # See the License for the specific language governing permissions and # limitations under the License. -# DISABLED. This workflow is switched off in the repository's Actions settings -# (state: disabled_manually) and does not run on any trigger, including its cron -# and workflow_dispatch. That state lives in GitHub's UI and is invisible when -# reading this file, which has already misled at least one audit — hence this -# banner. Re-enabling is a UI action; anyone doing so should first check that the -# workflow still matches the current CI layout. See rustfs/backlog#1603. -# name: Fuzz on: pull_request: types: [ opened, synchronize, reopened, closed ] - # PR trigger is intentionally narrow: only changes to the fuzz harness - # itself gate a PR. Broad crate paths (ecstore/filemeta/utils/policy/…) - # are covered by the nightly `schedule` run below, which fuzzes against - # whatever landed on main. Widening these paths previously queued a - # ~45min fuzz-build on nearly every PR and is why this workflow was - # disabled; do not re-add crate paths here. + # Run when the harness or any directly fuzzed production crate changes. paths: - "fuzz/**" - "scripts/fuzz/**" + - "crates/ecstore/**" + - "crates/filemeta/**" + - "crates/policy/**" + - "crates/security-governance/**" + - "crates/utils/**" + - "Cargo.toml" + - "Cargo.lock" - ".github/workflows/fuzz.yml" schedule: - - cron: "0 2 * * *" + - cron: "17 2 * * *" workflow_dispatch: inputs: profile: @@ -81,7 +76,7 @@ jobs: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' runs-on: sm-standard-4 - timeout-minutes: 45 + timeout-minutes: 60 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" steps: @@ -121,12 +116,7 @@ jobs: uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: name: fuzz-prebuilt-binaries-${{ github.run_number }} - path: | - fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/archive_extract - fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation - fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata - fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment - fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/policy_ingress + path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/ if-no-files-found: error retention-days: 1 compression-level: 0 @@ -192,10 +182,7 @@ jobs: nightly-fuzz-corpus: name: "Nightly / ${{ matrix.target }}" needs: fuzz-build - # TODO(ci-8): when the schedule-failure-issue composite action lands, - # add a step here (or a dependent job) that opens/updates a GitHub issue - # on nightly failure. ci-8 is the single alerting mechanism for all - # scheduled workflows; do not self-roll alerting in this workflow. + # Scheduled failures are handled by alert-on-failure below. if: > github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && diff --git a/.github/workflows/minio-interop.yml b/.github/workflows/minio-interop.yml index 3ee33e9bf..5f105ac41 100644 --- a/.github/workflows/minio-interop.yml +++ b/.github/workflows/minio-interop.yml @@ -121,3 +121,21 @@ jobs: cargo nextest run --run-ignored ignored-only --no-tests=fail \ -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \ -E "$INTEROP_FILTER" + + alert-on-failure: + name: Alert on scheduled failure + needs: [minio-interop] + if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure') + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Open or update failure-tracking issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/mint.yml b/.github/workflows/mint.yml index dd9acc6f8..25ae02916 100644 --- a/.github/workflows/mint.yml +++ b/.github/workflows/mint.yml @@ -45,13 +45,6 @@ # docker-capable self-hosted `dind-sm-standard-2` label was the alternative but # has fewer cores and reintroduces fleet-state risk for no reliability gain. -# DISABLED. This workflow is switched off in the repository's Actions settings -# (state: disabled_manually) and does not run on any trigger, including its cron -# and workflow_dispatch. That state lives in GitHub's UI and is invisible when -# reading this file, which has already misled at least one audit — hence this -# banner. Re-enabling is a UI action; anyone doing so should first check that the -# workflow still matches the current CI layout. See rustfs/backlog#1603. -# name: mint on: @@ -70,13 +63,13 @@ on: - core - full mint-image: - description: "Mint image reference" + description: "Mint image reference (empty = pinned default)" required: false - default: "minio/mint:edge" + default: "" schedule: # Weekly, after the Sunday s3-tests full sweep (starts 02:00 UTC, up to # 3h) has finished, so the two never contend for the same runner pool. - - cron: "0 6 * * 0" + - cron: "41 6 * * 0" env: S3_ACCESS_KEY: rustfsadmin-ci diff --git a/.github/workflows/nightly-gnu.yml b/.github/workflows/nightly-gnu.yml index 1f7c2d488..946761e54 100644 --- a/.github/workflows/nightly-gnu.yml +++ b/.github/workflows/nightly-gnu.yml @@ -16,7 +16,7 @@ name: Nightly GNU Build on: schedule: - - cron: "0 0 * * *" + - cron: "7 0 * * *" timezone: "Asia/Shanghai" workflow_dispatch: @@ -194,3 +194,23 @@ jobs: - name: Run HA leader failover live checks (three-node Raft cluster in Docker) run: bash scripts/test/vault_ha_kms_live.sh + + alert-on-failure: + name: Alert on scheduled failure + needs: [build, kms-vault-lane, kms-vault-ha-failover] + if: >- + always() && github.event_name == 'schedule' && + (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Open or update failure-tracking issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/performance-ab.yml b/.github/workflows/performance-ab.yml index ac8e8d7c5..de4986388 100644 --- a/.github/workflows/performance-ab.yml +++ b/.github/workflows/performance-ab.yml @@ -22,22 +22,15 @@ # correctness cost (e.g. the #4221 fsync durability fix) is recorded, not # blocked (rustfs/backlog#935 correction 1). -# DISABLED. This workflow is switched off in the repository's Actions settings -# (state: disabled_manually) and does not run on any trigger, including its cron -# and workflow_dispatch. That state lives in GitHub's UI and is invisible when -# reading this file, which has already misled at least one audit — hence this -# banner. Re-enabling is a UI action; anyone doing so should first check that the -# workflow still matches the current CI layout. See rustfs/backlog#1603. -# name: Performance A/B on: schedule: - - cron: "0 6 * * *" # 06:00 UTC nightly, against main + - cron: "31 6 * * *" # 06:31 UTC nightly, against main workflow_dispatch: inputs: duration: - description: "warp duration per round (short by default to fit the double-build budget)" + description: "warp duration per round" required: false default: "12s" type: string @@ -46,12 +39,8 @@ on: required: false default: false type: boolean - push: - # Every main commit pre-builds and caches its release binary (perf-3) so the - # nightly A/B restores a ready baseline instead of paying the double build. - branches: [main] - permissions: + actions: read contents: read env: @@ -59,83 +48,19 @@ env: RUST_BACKTRACE: 1 jobs: - # perf-3: on every push to main, build the release binary once and cache it - # keyed by commit SHA (rustfs-baseline-). The warp-ab measurements - # restore this instead of paying the ~32min-per-side source - # build. That double build is what pushed the expanded 24-cell nightly past its - # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental - # builds off the shared cargo cache keep each push cheap, and building on the - # same sm-standard-2 runner the A/B measures on guarantees the cached binary is - # ABI-identical. Do NOT source this from build.yml's per-merge artifact: those - # are cancelled ~7/8 of the time and are not a reliable baseline. - build-baseline-cache: - name: Build + cache baseline binary - if: github.event_name == 'push' - runs-on: sm-standard-2 - # Latest-wins: consumers only ever restore the binary for the *current* - # origin/main tip, so when pushes land faster than the ~65min build, a - # superseded build's output is dead weight — cancel it instead of stacking - # hour-long jobs on the shared runner pool. A skipped intermediate SHA at - # most costs one same-commit self-heal in the A/B job. - concurrency: - group: perf-baseline-build-main - cancel-in-progress: true - # #4806 put thin LTO + codegen-units=1 on [profile.release], pushing a - # single release build past 60min on this runner — every cache build on - # 2026-07-15 died on the old 60min ceiling ("exceeded the maximum execution - # time of 1h0m0s") and the cache never populated. The measured binary must - # keep the production profile, so the budget absorbs the build instead. - timeout-minutes: 100 - steps: - - name: Checkout repository - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 - with: - persist-credentials: false - - - name: Setup Rust environment - uses: ./.github/actions/setup - with: - rust-version: stable - cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }} - cache-save-if: ${{ github.ref == 'refs/heads/main' }} - - - name: Build release rustfs - run: cargo build --release --bin rustfs - - - name: Stage binary for cache - run: | - set -euo pipefail - mkdir -p baseline-bin - cp target/release/rustfs baseline-bin/rustfs - - - name: Cache baseline binary by SHA - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6 - with: - path: baseline-bin/rustfs - key: rustfs-baseline-${{ github.sha }} - warp-ab: name: Warp A/B budget gate - # Always run on schedule / manual dispatch. Never on push — that event only - # feeds build-baseline-cache above. - if: >- - github.event_name == 'schedule' || - github.event_name == 'workflow_dispatch' runs-on: sm-standard-2 - # With perf-3's cached baseline binary the common (cache-hit) nightly is - # measurement-only and finishes well under 50min. This ceiling stays - # generous only to absorb the same-commit cache-miss self-heal (~65min - # single build with the post-#4806 LTO profile + measurement). A timeout - # surfaces via the alert-on-failure job (it fires on cancelled/timed-out, - # not just failure). perf-6 recalibrates the budget once the noise study - # lands. - timeout-minutes: 120 + # A normal nightly restores the last successful binary and builds only the + # candidate; daily access keeps that cache warm. A cache miss may build both + # and needs room for the A/B run plus artifact and cache publication. + timeout-minutes: 180 steps: - name: Checkout repository uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 with: persist-credentials: false - fetch-depth: 0 # baseline is built from origin/main + fetch-depth: 0 # baseline may be an earlier successful scheduled head - name: Setup Rust environment uses: ./.github/actions/setup @@ -163,24 +88,55 @@ jobs: fi echo "allow_regression=$allow" >> "$GITHUB_OUTPUT" - # perf-3: resolve the commits so the cache can be keyed by SHA. The - # baseline is origin/main; the candidate is the checked-out ref. On the - # nightly (checkout == main) they are the same commit, so one cached binary - # serves both phases and the run does zero source builds. + # A failed regression run must keep comparing against the last known-good + # scheduled head. Otherwise the next nightly would absorb the regression + # into its baseline and turn green without a fix. + - name: Find last successful scheduled baseline + id: scheduled_baseline + if: github.event_name == 'schedule' + uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + with: + result-encoding: string + script: | + const { data } = await github.rest.actions.listWorkflowRuns({ + owner: context.repo.owner, + repo: context.repo.repo, + workflow_id: "performance-ab.yml", + event: "schedule", + status: "success", + per_page: 1, + }); + return data.workflow_runs[0]?.head_sha ?? ""; + + # Manual runs compare a selected ref with current main. Scheduled runs + # compare current main with the last successful scheduled head. With no + # history, the first run measures the candidate against itself and seeds + # that head only if the complete rig succeeds. - name: Resolve baseline / candidate commits id: commits + env: + SCHEDULED_BASELINE_SHA: ${{ steps.scheduled_baseline.outputs.result }} run: | set -euo pipefail - baseline_sha="$(git rev-parse origin/main)" candidate_sha="$(git rev-parse HEAD)" + if [[ "${{ github.event_name }}" == "schedule" ]]; then + baseline_sha="${SCHEDULED_BASELINE_SHA:-$candidate_sha}" + if ! git merge-base --is-ancestor "$baseline_sha" "$candidate_sha"; then + echo "::error::scheduled baseline $baseline_sha is not an ancestor of candidate $candidate_sha" >&2 + exit 1 + fi + else + baseline_sha="$(git rev-parse origin/main)" + fi + git cat-file -e "${baseline_sha}^{commit}" echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT" echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT" echo "baseline commit: $baseline_sha" echo "candidate commit: $candidate_sha" - # Exact-key restore of the baseline binary built by build-baseline-cache - # when origin/main last landed. A miss (binary evicted or not built yet) - # leaves cache-hit unset and the rig falls back to a source build. + # Exact-key restore of the candidate binary saved by its successful + # scheduled run. A miss leaves cache-hit unset and falls back to a source + # build of that known-good head. - name: Restore cached baseline binary id: baseline_cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6 @@ -270,11 +226,11 @@ jobs: elif [[ "$selfheal_built" == "true" ]]; then base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)" else - base_src="isolated origin/main source build (saved as rustfs-baseline-$baseline_sha)" + base_src="isolated baseline source build (saved as rustfs-baseline-$baseline_sha)" fi if [[ "$candidate_sha" == "$baseline_sha" ]]; then - # Nightly on main: the candidate is the same commit as the baseline, - # so reuse the one binary for both phases and skip all builds. + # No commits landed since the last successful baseline, so reuse + # the one binary for both phases and measure only rig drift. args+=(--candidate-bin "$base_bin") cand_src="same binary as baseline (same commit)" elif [[ "$candidate_built" == "true" ]]; then @@ -362,6 +318,23 @@ jobs: fi } >> "$GITHUB_STEP_SUMMARY" + - name: Stage successful candidate baseline + if: >- + steps.ab.outputs.status == '0' && + steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha + run: | + set -euo pipefail + cp candidate-bin/rustfs baseline-bin/rustfs + + - name: Cache successful candidate baseline + if: >- + steps.ab.outputs.status == '0' && + steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6 + with: + path: baseline-bin/rustfs + key: rustfs-baseline-${{ steps.commits.outputs.candidate_sha }} + # Scheduled failure alerting is handled by the alert-on-failure job below # (perf-2 consuming ci-8's schedule-failure-issue composite action). diff --git a/.github/workflows/runner-hygiene.yml b/.github/workflows/runner-hygiene.yml index c231b5706..bdebf2c77 100644 --- a/.github/workflows/runner-hygiene.yml +++ b/.github/workflows/runner-hygiene.yml @@ -30,7 +30,7 @@ name: Runner Hygiene on: schedule: - - cron: "0 6 1 * *" # Monthly, 1st at 06:00 UTC (after the daily audit cron) + - cron: "37 6 1 * *" # Monthly, 1st at 06:37 UTC workflow_dispatch: permissions: diff --git a/.github/workflows/scheduled-validation-freshness.yml b/.github/workflows/scheduled-validation-freshness.yml new file mode 100644 index 000000000..eef340869 --- /dev/null +++ b/.github/workflows/scheduled-validation-freshness.yml @@ -0,0 +1,57 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Scheduled Validation Freshness + +on: + schedule: + - cron: "47 23 * * *" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: scheduled-validation-freshness + cancel-in-progress: false + +jobs: + check-freshness: + name: Check scheduled validation freshness + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + actions: read + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Check latest scheduled runs + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set +e + python3 scripts/check_scheduled_validation_freshness.py \ + --report "${RUNNER_TEMP}/scheduled-validation-freshness.md" + status=$? + cat "${RUNNER_TEMP}/scheduled-validation-freshness.md" >> "${GITHUB_STEP_SUMMARY}" + exit "${status}" + - name: Open or update freshness issue + if: failure() + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + details-file: ${{ runner.temp }}/scheduled-validation-freshness.md diff --git a/.github/workflows/scheduled-validation-watchdog.yml b/.github/workflows/scheduled-validation-watchdog.yml new file mode 100644 index 000000000..e778ec640 --- /dev/null +++ b/.github/workflows/scheduled-validation-watchdog.yml @@ -0,0 +1,63 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Scheduled Validation Watchdog + +on: + workflow_run: + workflows: + - "Security Audit" + - "Build and Release" + - "Continuous Integration" + - "coverage" + - "e2e-nightly" + - "e2e-s3tests" + - "Fuzz" + - "mint" + - "minio-interop" + - "Nightly GNU Build" + - "Performance A/B" + - "Runner Hygiene" + types: [completed] + +permissions: + contents: read + +jobs: + alert-on-incomplete-run: + name: Alert on incomplete scheduled run + if: >- + github.event.workflow_run.event == 'schedule' && + github.event.workflow_run.conclusion != 'success' && + github.event.workflow_run.conclusion != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + actions: read + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Open or update incomplete-run issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + workflow-name: ${{ github.event.workflow_run.name }} + source-run-id: ${{ github.event.workflow_run.id }} + source-run-attempt: ${{ github.event.workflow_run.run_attempt }} + source-event: ${{ github.event.workflow_run.event }} + source-ref-name: ${{ github.event.workflow_run.head_branch }} + source-sha: ${{ github.event.workflow_run.head_sha }} diff --git a/AGENTS.md b/AGENTS.md index 2004c597d..2d9535399 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,402 +1,255 @@ -# RustFS Agent Instructions (Global) +# RustFS Agent Instructions -This root file keeps repository-wide rules only. -Use the nearest subdirectory `AGENTS.md` for path-specific guidance. +This file contains repository-wide rules. Use the nearest subdirectory +`AGENTS.md` for path-specific invariants. -## Rule Precedence +## Precedence 1. System/developer instructions. -2. Current user/task instructions. -3. The nearest `AGENTS.md` in the current path. -4. This file (global defaults). +2. The current user request. +3. The nearest `AGENTS.md`. +4. This file. -If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI. +## Operating Model -## Execution Discipline - -- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write). -- State assumptions when they affect the implementation or verification path. -- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky. -- For multi-step work, keep the plan minimal and tied to verifiable outcomes. -- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available. -- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below). +- Inquiry, diagnosis, review, and planning tasks are read-only unless the user + explicitly requests changes. +- For implementation, read the relevant code, tests, and local guidance, then + make the smallest change that satisfies the request. +- State assumptions only when they affect behavior or verification. Ask only + when a wrong assumption would materially change the result. +- Do not load every skill or inspect unrelated modules preemptively. Select a + skill only when its description directly matches the request or changed + surface. +- Avoid repeated reads and equivalent verification commands once enough + evidence exists. ## Worktree and Disk Hygiene -- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task. -- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing. -- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees. -- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed. +- Start implementation from the latest `origin/main` and confirm the requested + change is not already present. +- An existing clean, isolated task worktree is sufficient. Create another + worktree only when the current checkout is shared, dirty with unrelated work, + or belongs to another task. +- Never commit from a shared checkout. Use an `overtrue/` feature branch unless + the user requests another name. +- Check free space before artifact-heavy builds, tests, coverage, or downloads. + Re-check before a broad gate when space is tight. +- Remove only task-owned temporary/build artifacts. Never delete another task's + worktree or uncommitted data. +- At handoff, mention disk or cleanup details only when they affected execution + or artifacts/worktrees remain intentionally. -## PR Lifecycle Monitoring +## Change Style -- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action. -- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays. -- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result. -- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected. -- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled. -- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup. +- Preserve existing control flow unless changing it is required for correctness. +- Prefer a direct local edit over new files, wrappers, managers, or speculative + abstractions. +- Add a helper only when it removes current duplication, names a real domain + boundary, or isolates a non-trivial invariant. +- Remove an in-scope path superseded by the change. If compatibility requires it, + adapt at the boundary to one canonical core and use the repository's + `RUSTFS_COMPAT_TODO` policy. +- Comments explain non-obvious invariants or reasons. Do not narrate code or + record change history. +- Mention unrelated problems when useful; do not fix them in a narrow task. -## Autonomy and Approval Boundaries +## Reuse and Boundary Rules -- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested. -- Action tasks (change, build, fix): make in-scope local changes without asking for approval. -- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope. - -## Communication and Language - -- Respond in the same language used by the requester. -- Keep source code, comments, commit messages, and PR title/body in English. -- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting. - -## Change Style for Existing Logic - -- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough. -- Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`). - Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead. - If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work. -- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements. -- Prefer editing existing code over rewriting files or reshaping unrelated logic. -- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics. -- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths. -- Do not refactor existing code only to make it easier to unit test. -- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths. -- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible. -- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation. -- Mention unrelated issues when useful, but do not fix them as part of a narrow task. - -## Reuse Before You Write - -Search for an existing implementation before writing a new one; extend what exists instead of duplicating it: - -- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference. -- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call. -- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants. -- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '' /src /tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written. - -## Necessary Code Only - -Net-new code — files, types, branches, comments — is cost to justify, not progress: - -- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries. -- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious. -- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup. -- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data. -- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success. -- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value. +- Before adding helpers, constants, fixtures, or wrappers, search the touched + crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant + direct dependencies. +- Reuse requires matching semantics: normalization, error types, deadlines, + durability, and compatibility must fit the call site. A narrowly named local + helper is better than forced reuse with different semantics. +- Validate untrusted input at its trust boundary, then trust the validated type. + Values crossing disk, RPC, persistence, or version boundaries remain + untrusted at every consumer. +- Re-check boundary values immediately before destructive actions such as + delete, overwrite, or quorum decisions. +- Every new branch needs a concrete triggering input/state. For decoded or peer + data, corruption and mixed-version input are valid triggers. +- Required values must return a typed error when absent or corrupt; do not use a + default that converts corruption into a plausible result. +- Attach error context once where it is actionable. Do not erase typed errors + below aggregation or quorum layers. ## Sources of Truth -- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`) -- Local quality commands: `Makefile` and `.config/make/` -- CI quality gates: `.github/workflows/ci.yml` -- PR template: `.github/pull_request_template.md` -- High-level architecture and crate map: `ARCHITECTURE.md` -- Migration guardrails, readiness contracts, support matrices: - `docs/architecture/README.md` (routes by audience) -- Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries - a frontmatter `description` stating when it applies. Scan the descriptions - before starting a task and follow any skill that matches, even if your tool - does not auto-load skills: - `grep -m1 '^description:' .agents/skills/*/SKILL.md` - Claude Code reads them through the `.claude/skills` symlink; add new skills - to `.agents/skills/` only, never as separate copies per tool +- Workspace membership: `Cargo.toml`. +- Local gates: `Makefile` and `.config/make/`. +- CI gates: `.github/workflows/ci.yml`. +- PR format: `.github/pull_request_template.md`. +- Architecture routing: `ARCHITECTURE.md` and `docs/architecture/README.md`. +- Agent skills: `.agents/skills/*/SKILL.md`. -Avoid duplicating long crate lists or command matrices in instruction files. -Reference the source files above instead. +Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots, +or agent scratch notes. Durable architecture belongs under `docs/architecture/`, +operations under `docs/operations/`, and testing references under +`docs/testing/`. `scripts/check_no_planning_docs.sh` enforces this boundary. -Do not commit planning-type documents — one-shot implementation/optimization -plans, task trackers, migration-progress ledgers, phase/PR templates, -issue-scoped benchmark-result snapshots or optimization conclusions, or -agent-generated working notes (e.g. anything a `superpowers`/scratch workflow -produces). Keep that work in the issue tracker or your local worktree, not in -the repository. Only durable reference — the architecture set under -`docs/architecture/`, repeatable operational runbooks under `docs/operations/`, -and the test-suite references under `docs/testing/` — belongs in version -control; `.gitignore` ignores everything else under `docs/` by default, so a new -plan file will not be tracked unless someone force-adds it — don't. -`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and -CI) fails the build if anything is committed under `docs/superpowers/`, even via -`git add -f`. +## Verification -## Verification Before PR +Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a +concrete path-specific check, but must not replace this tiering with a generic +full-workspace gate. -Convert changes into independently verifiable outcomes. This section controls -agent-run local validation; preparing a commit or PR does not by itself require -the broadest gate. Inspect only the final task-owned diff, classify it by -behavioral impact rather than line count or path alone, and run the smallest -set of checks that provides meaningful coverage. Do not let unrelated -worktree changes or a generic contributor checklist expand the scope. -For non-exempt changes, complete the applicable multi-role adversarial review -before running `make pre-pr` (or an equivalent full gate). Resolve or rebut -every finding first, then run the gate against the reviewed final diff. +### Documentation and Instructions -### Validation floor +For prose, comments, agent instructions, and skill metadata that cannot affect +runtime/build output: -- Every change that is not documentation-only must finish with - `cargo fmt --all --check` passing. An umbrella gate that runs this exact - check satisfies the requirement; do not run it twice. Use `cargo fmt --all` - only when formatting needs to be fixed. Run the configured formatter or - validator for other changed languages when one exists. -- Documentation-only or instruction-only means all task-owned changes are - prose or documentation assets and cannot affect runtime, builds, CI, - dependencies, generated code, or tests. Run `git diff --check` and any - relevant documentation guard, but skip Cargo formatting, compilation, - Clippy, tests, `make pre-commit`, and `make pre-pr`. -- Behavior changes require relevant existing or new tests. Prefer the most - focused test or affected package. A passing targeted test can also provide - sufficient compilation coverage when it builds every changed target and - feature involved; do not add a redundant `cargo check` in that case. -- `cargo check` supplements compilation coverage; it never substitutes for a - behavioral test. If a relevant test cannot reasonably be added or run, use - the narrowest compilation check and report the reason and remaining risk. +- Run `git diff --check`. +- Run the relevant documentation guard or skill validator when applicable. +- Skip Cargo formatting, compilation, Clippy, tests, `make pre-commit`, and + `make pre-pr`. -### Validation tiers +### Non-Behavioral Source Changes -1. **Documentation/instruction-only:** Apply the exemption above. Run a guard - such as `make doc-paths-check` only when it is relevant to the edited text. -2. **Non-behavioral source change:** For comments, formatting, or another - demonstrably non-executable change, run the formatting floor. Compilation, - Clippy, and tests may be skipped only when the edit cannot affect - compilation or runtime behavior; run targeted doctests if executable - documentation examples changed. -3. **Localized or bounded behavior change:** Run the formatting floor and the - narrowest relevant tests. Add package-scoped `cargo check` or Clippy only - for changed targets, features, APIs, error handling, async behavior, or - control flow not already covered. When several crates are affected but the - dependency set is identifiable, validate those packages and known - dependents instead of the whole workspace. Use `make pre-commit` only when - a repository-wide fast gate adds useful confidence beyond those checks. -4. **Broad or high-risk change:** After the applicable adversarial review has - completed, run `make pre-pr` only when targeted coverage cannot bound the - impact, including: - - dependency, feature, build-script, procedural-macro, code-generation, - toolchain, or CI changes that alter compilation or the test matrix; - - cross-crate public APIs, shared foundational code, or broad refactors with - an unbounded dependent set; - - locking, storage durability or formats, erasure coding, replication, - RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other - security-sensitive behavior; - - a targeted check that reveals wider impact, an explicit user request, or - a release policy that requires the full gate. +- Run the formatter/validator for the changed language. +- Add compilation or doctests only when syntax or executable examples changed. -Documentation-only and non-behavioral classifications take precedence over -path-based triggers. A small diff can still be high-risk, while a CI comment, -manifest comment, or release-note edit does not require full validation. +### Localized Behavior Changes -`make pre-pr` includes `make pre-commit` coverage. Never run both for the same -unchanged diff, and do not repeat equivalent checks during PR preparation or -because a local hook already ran them. Rerun only checks whose scope is affected -by later edits. Full workspace checks do not replace a relevant integration or -E2E test for changed behavior; run that focused test when required and -available, or report why it was not run and the remaining risk. +- Run `cargo fmt --all --check` for Rust changes. +- Run the narrowest test that exercises the changed behavior. +- Add package-scoped `cargo check` or Clippy only for targets, features, public + APIs, error handling, or control flow not compiled by the focused test. +- Use `make pre-commit` only when its repository-wide fast checks add confidence + beyond the focused checks. -If `make` is unavailable, run the equivalent checks defined under -`.config/make/`. At handoff, list the checks actually run, checks intentionally -skipped, and the reason for the selected tier. +### Broad or High-Risk Changes -After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage. -Do not open a PR with code changes when the required checks fail. -Make a failing check pass by fixing the cause, never by weakening the gate: -do not loosen or skip a guard script, add entries to a baseline or allowance -list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or -delete or relax a failing assertion to get green. If a check itself is wrong, -change it deliberately and state the rationale in the PR. +After the required adversarial review, run `make pre-pr` when targeted coverage +cannot bound the impact, including dependency/toolchain/build-matrix changes, +unbounded cross-crate APIs, or locking, durability, erasure coding, replication, +RPC, IAM/KMS/auth, cryptography, on-disk/on-wire, and S3-visible behavior. -For flaky tests, do not paper over them with retries. Follow the flake policy -in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h, -quarantine with an issue link, fix or delete within 30 days); the local -`default` nextest profile never retries. +`make pre-pr` includes `make pre-commit`; never run both for the same unchanged +diff. Do not repeat a check already covered by a successful umbrella gate. +Rerun only checks affected by later edits. -## Adversarial Validation (Default On) +Never weaken a gate to get green: do not add baselines/allowances, suppress +lints, ignore tests, or relax assertions unless changing that policy is itself +the reviewed task. Follow `docs/testing/README.md` for flaky tests. -Every non-exempt output (see Risk tiers) — code change, bug fix, or -design/solution proposal — passes multi-role adversarial review before it -counts as done. -Author confidence is not evidence: each role's job is to refute the change, -not to bless it. +## Adversarial Validation -### Risk tiers +Adversarial validation applies to final implementation diffs, explicitly +requested adversarial/design reviews, and agent-instruction changes that alter +execution. Ordinary questions, diagnoses, status reports, non-adversarial code +reviews, and low-risk planning do not trigger it. -Pick the tier from the riskiest file touched; when in doubt, pick the higher. +Risk and review shape: -- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime, - builds, tests, or agent execution. Skip this section. -- **Mechanical:** pure renames, file moves, test-only or tooling changes, and - agent-instruction changes that alter execution — - correctness and simplicity adversaries only. -- **Standard (the default):** any change that affects behavior. -- **High risk:** touches locking, erasure coding, quorum/heal, replication, - multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`), - persistence/fsync, IAM/KMS/auth, on-disk or on-wire formats, or - S3 API-visible behavior. +- **Exempt:** documentation, comments, formatting, or typos with no runtime, + build, test, or agent-execution effect. +- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule + changes. Run correctness and simplicity lenses. +- **Standard:** localized behavior changes. Run one integrated final-diff pass + covering correctness, simplicity, and test coverage; add only domain lenses + matched by the diff. +- **High risk / substantial PR review:** high risk includes locking, + erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering, + persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and + S3-visible semantics. Cover all applicable lenses using exactly two + independent reviewers when delegation is explicitly authorized. Split the + lenses between them. Otherwise perform two fresh sequential passes. -### Roles +Available domain lenses are security, concurrency/durability, compatibility, +and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an +explicit adversarial request, a high-risk change, or a substantial PR review; +then read only its matching role references. A routine standard pass does not +load the playbook unless the reviewer needs a RustFS-specific probe. -Run each applicable role as an independent pass over the final diff (or -proposal text) — parallel reviewer agents where the tooling supports them, -otherwise sequential passes that each start fresh from the diff and the -nearest scoped `AGENTS.md`, discarding the writing session's assumptions. -Each role either produces findings or reports "attacked X, Y, Z — no break -found"; a bare pass is not a result. Repo-specific attack probes for every -role live in `.agents/skills/adversarial-validation/` — run them, they -encode this repo's shipped bugs. +A finding must name a concrete input/state/interleaving and wrong outcome, or a +specific missing regression check, with `file:line`. Resolve it by fixing the +diff or rebutting it with code-path/test/invariant evidence. After a non-trivial +fix, rerun only affected lenses. -- **Correctness adversary** — construct a concrete input/state/interleaving - that yields wrong output, data loss, or a crash. Probe error paths and edge - values (empty, nil UUID, zero-length, quorum−1, missing version). -- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence. -- **Security reviewer** — authn/authz bypass, injection, secret leakage, - untrusted deserialization (see Serde Safety), path traversal, timing leaks. -- **Concurrency/durability reviewer** — lock ordering, races, cancellation, - partial failure, retry/idempotency, crash and power-loss ordering. -- **Compatibility reviewer** — S3 API surface, MinIO interop, on-disk and - on-wire formats, mixed-version upgrade/downgrade paths. -- **Performance reviewer** — allocation and cloning on hot paths, lock hold - time across IO, sync or CPU-heavy work on async runtime threads, added - fsync/flush outside the durability gate, hot-path logging noise. A - measurable regression on a per-request or per-object path is a finding. -- **Test-coverage skeptic** — for each testable behavior claim, name the test - or executable check that detects a revert; then name a changed line that - could be wrong while all checks stay green. If a focused check is not - reasonable, require the reason and residual risk from the validation floor. - Test additions have no line-count or growth budget. +For high-risk PRs, record one concise verdict per covered lens in the PR body. -Standard tier: correctness adversary + simplicity adversary + test-coverage -skeptic, plus every role whose domain the diff touches (async or -shared-state code → concurrency; parsing of untrusted input → security; -public crate API shape → compatibility; per-request or per-object hot paths -→ performance). -High risk: all seven roles. +## Pull Request Lifecycle -### Protocol - -1. A finding states a concrete failure scenario (input/state → wrong - outcome) or names a missing test, with severity and file:line. "Looks - risky" is not a finding. -2. Resolve every finding: fix it, or rebut it with evidence — a test, a - traced code path, or a cited invariant. Restated intent and "unlikely" - are not rebuttals. -3. After non-trivial fixes, re-run the roles whose domain the fix touched. -4. For proposals with no diff, roles attack assumptions, failure modes, - migration/rollback, and testability instead — including the simplest - rejected alternative and the blast radius when the design fails. - -### Exit criteria - -- Every applicable role has run; every finding is fixed or rebutted with - evidence. -- Every testable behavior change has a focused regression check. Exceptions - follow the validation floor and state why a check is impractical and what - risk remains. -- After the applicable adversarial review has completed, the Verification - Before PR gates pass; adversarial review supplements those gates, never - replaces them. -- High risk only: record a one-line verdict per role in the PR description. +- Creating or updating a PR includes one immediate snapshot of checks, + mergeability, reviews, and unresolved threads. +- Unless the user explicitly requests monitoring, a release workflow requires + it, or an automation already owns it, hand off after the PR is open with the + current state and next event to watch. Do not delay ordinary handoff with + fixed quiet-period sleeps. +- For requested monitoring, use event-driven or bounded waits. Report only state + changes, actionable failures, or a meaningful prolonged delay. +- Investigate failures/comments before changing code. Fix task-attributable + issues, rerun affected verification, push, reply or resolve the thread, then + resume the requested monitor. +- Never merge without required reviewer approval or explicit authority. +- After an observed merge, verify the commit reached the base, then clean the + task worktree/branch when safe. Preserve unmerged work for closed PRs unless + deletion was explicitly authorized. ## Git and PR Baseline -- Use feature branches based on the latest `main`. -- Assume other agent sessions work this repository concurrently. Never commit - in a shared checkout; do all work on a dedicated feature branch, preferably - in a dedicated worktree. -- Immediately before branching, fetch `origin/main` and branch from it; - confirm the target issue is not already fixed there before writing code. -- Follow Conventional Commits, with subject length <= 72 characters. -- Keep PR title and description in English. -- Use `.github/pull_request_template.md` and keep all section headings. -- Use `N/A` for non-applicable template sections. -- Include verification commands in the PR description. -- When using `gh pr create`/`gh pr edit`, write the markdown body to a file - and pass `--body-file`; multiline inline `--body` is unsafe — backticks and - shell expansion can corrupt content or trigger unintended commands. - Pattern: `cat > /tmp/pr_body.md <<'EOF' ... EOF`, then - `--body-file /tmp/pr_body.md` (keep the file outside the checkout). -- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment. -- Do not hard-wrap prose in PR/issue/discussion bodies; write each paragraph as a - single line and let it reflow. GitHub renders single newlines inside a paragraph - as line breaks, so mid-sentence wrapping shows up as ugly breaks. Only break lines - for list items, code blocks, and deliberate separators. -- After fixing code review comments or CI findings, always mark corresponding review - comments/threads as resolved before returning to the user. -- In handling review comments, confirm the underlying issue before changing code. - If a suggested change is not appropriate for behavior or risk, reply with a - concise rationale instead of blindly applying it. +- Follow Conventional Commits; keep the subject at most 72 characters. +- Source comments, commits, PR titles, and PR bodies are in English. +- Keep every heading from `.github/pull_request_template.md`; use `N/A` where + needed and include commands actually run. +- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content. +- PR/issue/discussion content must not contain the literal sequence `\n` or + hard-wrapped prose paragraphs. +- Do not include local absolute paths or tool-specific labels/prefixes in GitHub + content. +- Resolve review threads after the underlying issue is fixed. If declining a + suggestion, reply with a short evidence-based reason. ## Security Baseline - Never commit secrets, credentials, or key material. - Use environment variables or vault tooling for sensitive configuration. -- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage. +- For localhost-sensitive tests, bypass proxies explicitly. +- Untrusted S3 XML/JSON, lifecycle, policy, replication, and RPC structures use + strict deserialization where compatibility permits. Security-critical + defaults require explicit validation. ## Logging -Applies to **every** `tracing` macro you add or edit, including a single line -added in passing while fixing something else — not only to log-focused changes. +For every added or edited `tracing` call: -- Fields first, message second: `event`, `component`, `subsystem`, - `result`/`state`, then key context. The message is a short label, not a - sentence with values interpolated into it. -- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*` - constants of the module you are editing; match the shape of the log sites - already in that file rather than introducing a second style next to them. -- Level policy: `error` for behavior/security-affecting failures, `warn` for - degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for - targeted diagnostics, `trace` for hot paths. Per-object and per-request - success paths are `trace`. -- Never log secrets, tokens, credential payloads, or merged config dumps. -- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files - it lists; passing it is a floor, not evidence the log matches the house style. +- Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` + constants and field shape. +- Put fields first and a short label last. +- Use `error` for behavior/security failure, `warn` for degradation/fallback, + `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for + repetitive request/object success paths. +- Never log secrets, credential payloads, or merged configs. -See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event -model, level policy, and guardrail-update checklist. +Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes. -## Tools +## Cross-Cutting Storage Invariants -### xl.meta decode tool Quick Use +- Write internal object metadata under both `x-rustfs-internal-` and + `x-minio-internal-` using + `crates/utils/src/http/metadata_compat.rs` helpers. +- Read binary UUID metadata with + `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())`; absent, + empty, and nil all mean no value. +- Remote-tier version `None` or `""` means an unversioned bucket; send no + `versionId` on tier GET/DELETE. +- `DataUsageCacheInfo` and `DataUsageEntry` keep their hand-written map + serialization and new fields remain `#[serde(default)]` for older readers. -``` -cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta" -``` +## Naming -## Serde Safety +Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case` +functions/variables, and `PascalCase` types. Do not rename unrelated existing +violations. -- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs). -- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level. -- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value. +## Scoped Guidance -## Cross-Cutting Domain Invariants - -- Write internal object metadata under **both** `x-rustfs-internal-` - and `x-minio-internal-` keys (MinIO interop). Use the helpers in - `crates/utils/src/http/metadata_compat.rs` (`get_bytes` prefers the RustFS - key); never write only one of the two. -- Read binary UUID metadata defensively: - `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` — - absent, empty, and nil all mean "no value", never `Uuid::nil()`. -- A remote-tier version of `None`/`""` means the tier bucket is unversioned: - send **no** `versionId` on tier GET/DELETE. -- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`, - `DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack - encodes derived structs as arrays, where an appended field makes the whole - cache a decode error for older readers — keep new fields `#[serde(default)]` - and keep the map encoding rather than reverting to `derive(Serialize)`. - -## Naming Conventions - -- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types. -- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`). -- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic). - -## Scoped Guidance in This Repository - -Many crates and modules carry their own `AGENTS.md` with path-specific rules -(security boundaries, lock ordering, domain invariants). Before editing a -path, check for the nearest one: +Before editing, locate the nearest instructions with: ```bash git ls-files '*AGENTS.md' ``` -The nearest file wins. Do not maintain a hand-written index of these files -here — it goes stale. +The nearest file wins for domain invariants. Keep generic workflow and +validation policy in this root file. diff --git a/CLAUDE.md b/CLAUDE.md index 5e75fe3cb..59893c48b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,8 @@ make build-docker BUILD_OS=ubuntu22.04 - Crate membership: `Cargo.toml` `[workspace].members` - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md) - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md) -- CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs) +- CI workflow steps: `.github/workflows/`; event, timeout, and required-status + matrix: [docs/testing/ci-gates.md](docs/testing/ci-gates.md) - Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake policy: [docs/testing/README.md](docs/testing/README.md) - Tier/ILM transition debugging (xl.meta inspection, versionId tracing): diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7882a486e..5842395f4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,6 +70,8 @@ make pre-pr > For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md). +> For the event, timeout, required-status, and local reproduction matrix, see [docs/testing/ci-gates.md](docs/testing/ci-gates.md). + ### 🔒 Automated Pre-commit Hooks #### What `make pre-commit` and `make pre-pr` actually run diff --git a/Cargo.lock b/Cargo.lock index 312fc1376..517983884 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1858,9 +1858,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.4.3" +version = "1.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "509591b7bcd67f4ef775afad7662703b4935daaa6ec0e5605cfb1090b32a2b6d" +checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273" dependencies = [ "find-msvc-tools", "jobserver", @@ -2522,12 +2522,6 @@ dependencies = [ "subtle", ] -[[package]] -name = "cty" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b365fabc795046672053e29c954733ec3b05e4be654ab130fe8f1f94d7051f35" - [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -3843,7 +3837,6 @@ dependencies = [ "s3s", "serde", "serde_json", - "serial_test", "sha2 0.11.0", "suppaftp", "time", @@ -5989,15 +5982,6 @@ version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" -[[package]] -name = "libmimalloc-sys" -version = "0.1.49" -source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11#6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" -dependencies = [ - "cc", - "cty", -] - [[package]] name = "libredox" version = "0.1.20" @@ -6398,14 +6382,6 @@ dependencies = [ "synstructure 0.13.2", ] -[[package]] -name = "mimalloc" -version = "0.1.52" -source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11#6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" -dependencies = [ - "libmimalloc-sys", -] - [[package]] name = "mime" version = "0.3.17" @@ -9163,13 +9139,11 @@ dependencies = [ "insta", "jiff", "libc", - "libmimalloc-sys", "libsystemd", "matchit 0.9.2", "md-5 0.11.0", "metrics", "metrics-util", - "mimalloc", "mime_guess", "opentelemetry", "opentelemetry_sdk", @@ -9205,6 +9179,8 @@ dependencies = [ "rustfs-lock", "rustfs-log-analyzer", "rustfs-madmin", + "rustfs-mimalloc", + "rustfs-mimalloc-sys", "rustfs-notify", "rustfs-object-capacity", "rustfs-object-data-cache", @@ -9257,6 +9233,7 @@ dependencies = [ "url", "urlencoding", "uuid", + "x509-parser", "zeroize", "zip", "zstd", @@ -9875,6 +9852,24 @@ dependencies = [ "tokio", ] +[[package]] +name = "rustfs-mimalloc" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a406f4aa07084301d485beec873af6dccc8e3f8762da244743df92038b1db1a6" +dependencies = [ + "rustfs-mimalloc-sys", +] + +[[package]] +name = "rustfs-mimalloc-sys" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3051b819175f58445d4c369a72f0ab88149f3885ba8bea2aff3be01f53fe7cd" +dependencies = [ + "cc", +] + [[package]] name = "rustfs-notify" version = "1.0.0-rc.3" @@ -9920,7 +9915,6 @@ dependencies = [ "rustfs-config", "rustfs-io-metrics", "rustfs-utils", - "serial_test", "temp-env", "tempfile", "tokio", @@ -10293,7 +10287,6 @@ dependencies = [ "s3s", "serde", "serde_json", - "serial_test", "sha2 0.11.0", "temp-env", "tempfile", @@ -12690,6 +12683,7 @@ dependencies = [ "js-sys", "rand 0.10.2", "serde_core", + "sha1_smol", "wasm-bindgen", ] diff --git a/Cargo.toml b/Cargo.toml index 652e931e2..717a0c6e6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -204,6 +204,7 @@ rsa = { version = "=0.10.0-rc.18" } rustls = { default-features = false, version = "0.23.43" } rustls-native-certs = "0.8" rustls-pki-types = "1.15.1" +x509-parser = "0.18.1" sha1 = "0.11.0" sha2 = "0.11.0" subtle = "2.6" @@ -349,8 +350,8 @@ russh-sftp = "2.4.0" dav-server = "0.11.0" # Performance Analysis and Memory Profiling -mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" } -libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] } +rustfs-mimalloc = { version = "0.5.0" } +rustfs-mimalloc-sys = { version = "0.5.0" } hotpath = { version = "0.23.3", default-features = false } # Snapshot testing for output format regression detection insta = { version = "1.48" } diff --git a/crates/AGENTS.md b/crates/AGENTS.md index 3417bab26..65b97334c 100644 --- a/crates/AGENTS.md +++ b/crates/AGENTS.md @@ -19,7 +19,9 @@ Applies to all paths under `crates/`. - Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths. - Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded. -- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics). +- Prefer direct atomic `fetch_*` operations for unconditional updates and + `compare_exchange` loops only for conditional updates such as peaks or + adaptive state. - When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff. - `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`. @@ -40,7 +42,9 @@ Applies to all paths under `crates/`. - Keep unit tests close to the module they test. - Keep integration tests under each crate's `tests/` directory. - Add regression tests for bug fixes and behavior changes. -- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test. +- Every test needs an observable failure criterion. Direct assertions, + delegated assertions, snapshots/properties, `#[should_panic]`, and meaningful + `Result` failures are all valid; a call that can silently succeed is not. - In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input. ## Async and Performance diff --git a/crates/audit/AGENTS.md b/crates/audit/AGENTS.md index 39870cd65..32a1f3c1c 100644 --- a/crates/audit/AGENTS.md +++ b/crates/audit/AGENTS.md @@ -50,4 +50,3 @@ crate. - `cargo test -p rustfs-audit` - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test` - Focused: `cargo test -p rustfs-audit pipeline` -- Full gate before commit: `make pre-commit` diff --git a/crates/common/src/metrics.rs b/crates/common/src/metrics.rs index 51eef967a..813e39ac3 100644 --- a/crates/common/src/metrics.rs +++ b/crates/common/src/metrics.rs @@ -901,6 +901,10 @@ pub struct Metrics { scanner_cycle_max_duration_millis: AtomicU64, scanner_cycle_max_objects: AtomicU64, scanner_cycle_max_directories: AtomicU64, + scanner_cycle_timeout_total: AtomicU64, + scanner_cycle_recovery_required_total: AtomicU64, + scanner_cycle_last_progress_age_seconds: AtomicU64, + scanner_leader_lease_without_progress: AtomicBool, scanner_bitrot_cycle_enabled: AtomicBool, scanner_bitrot_cycle_millis: AtomicU64, scanner_checkpoint: Mutex>, @@ -1370,6 +1374,14 @@ pub struct ScannerMetricsReport { #[serde(default)] pub cycle_max_directories: u64, #[serde(default)] + pub cycle_timeout_total: u64, + #[serde(default)] + pub cycle_recovery_required_total: u64, + #[serde(default)] + pub cycle_last_progress_age: u64, + #[serde(default)] + pub leader_lease_without_progress: bool, + #[serde(default)] pub bitrot_cycle_enabled: bool, #[serde(default)] pub bitrot_cycle_seconds: f64, @@ -1430,6 +1442,9 @@ const OTEL_SCANNER_BUCKETS_SCANNED: &str = "rustfs_scanner_buckets_scanned_total const OTEL_SCANNER_CYCLES: &str = "rustfs_scanner_cycles_total"; const OTEL_SCANNER_CYCLE_DURATION_SECONDS: &str = "rustfs_scanner_cycle_duration_seconds"; const OTEL_SCANNER_BUCKET_DRIVE_DURATION_SECONDS: &str = "rustfs_scanner_bucket_drive_duration_seconds"; +const OTEL_SCANNER_CYCLE_TIMEOUT_TOTAL: &str = "rustfs_scanner_cycle_timeout_total"; +const OTEL_SCANNER_CYCLE_LAST_PROGRESS_AGE: &str = "rustfs_scanner_cycle_last_progress_age"; +const OTEL_SCANNER_LEADER_LEASE_WITHOUT_PROGRESS: &str = "rustfs_scanner_leader_lease_without_progress"; fn scan_cycle_result_label(result: u8) -> &'static str { match result { @@ -1913,6 +1928,10 @@ impl Metrics { scanner_cycle_max_duration_millis: AtomicU64::new(0), scanner_cycle_max_objects: AtomicU64::new(0), scanner_cycle_max_directories: AtomicU64::new(0), + scanner_cycle_timeout_total: AtomicU64::new(0), + scanner_cycle_recovery_required_total: AtomicU64::new(0), + scanner_cycle_last_progress_age_seconds: AtomicU64::new(0), + scanner_leader_lease_without_progress: AtomicBool::new(false), scanner_bitrot_cycle_enabled: AtomicBool::new(false), scanner_bitrot_cycle_millis: AtomicU64::new(0), scanner_checkpoint: Mutex::new(None), @@ -2412,12 +2431,29 @@ impl Metrics { .store(cycle_max_objects.unwrap_or_default(), Ordering::Relaxed); self.scanner_cycle_max_directories .store(cycle_max_directories.unwrap_or_default(), Ordering::Relaxed); + self.scanner_leader_lease_without_progress.store(false, Ordering::Relaxed); + self.scanner_cycle_last_progress_age_seconds.store(0, Ordering::Relaxed); + metrics::gauge!(OTEL_SCANNER_LEADER_LEASE_WITHOUT_PROGRESS).set(0.0); + metrics::gauge!(OTEL_SCANNER_CYCLE_LAST_PROGRESS_AGE).set(0.0); self.scanner_bitrot_cycle_enabled .store(bitrot_cycle.is_some(), Ordering::Relaxed); self.scanner_bitrot_cycle_millis .store(bitrot_cycle.map(duration_millis_saturated).unwrap_or_default(), Ordering::Relaxed); } + pub fn record_scanner_cycle_timeout(&self, recovery_required: bool, progress_age: Duration) { + self.scanner_cycle_timeout_total.fetch_add(1, Ordering::Relaxed); + if recovery_required { + self.scanner_cycle_recovery_required_total.fetch_add(1, Ordering::Relaxed); + } + self.scanner_cycle_last_progress_age_seconds + .store(progress_age.as_secs(), Ordering::Relaxed); + self.scanner_leader_lease_without_progress.store(true, Ordering::Relaxed); + metrics::counter!(OTEL_SCANNER_CYCLE_TIMEOUT_TOTAL).increment(1); + metrics::gauge!(OTEL_SCANNER_CYCLE_LAST_PROGRESS_AGE).set(progress_age.as_secs_f64()); + metrics::gauge!(OTEL_SCANNER_LEADER_LEASE_WITHOUT_PROGRESS).set(1.0); + } + pub fn record_scanner_set_scan_state(&self, concurrency_limit: Option, queued: Option, active: Option) { if let Some(concurrency_limit) = concurrency_limit { self.scanner_set_scan_concurrency_limit @@ -3265,6 +3301,10 @@ impl Metrics { m.cycle_max_duration_seconds = self.scanner_cycle_max_duration_millis.load(Ordering::Relaxed) as f64 / 1000.0; m.cycle_max_objects = self.scanner_cycle_max_objects.load(Ordering::Relaxed); m.cycle_max_directories = self.scanner_cycle_max_directories.load(Ordering::Relaxed); + m.cycle_timeout_total = self.scanner_cycle_timeout_total.load(Ordering::Relaxed); + m.cycle_recovery_required_total = self.scanner_cycle_recovery_required_total.load(Ordering::Relaxed); + m.cycle_last_progress_age = self.scanner_cycle_last_progress_age_seconds.load(Ordering::Relaxed); + m.leader_lease_without_progress = self.scanner_leader_lease_without_progress.load(Ordering::Relaxed); m.bitrot_cycle_enabled = self.scanner_bitrot_cycle_enabled.load(Ordering::Relaxed); m.bitrot_cycle_seconds = self.scanner_bitrot_cycle_millis.load(Ordering::Relaxed) as f64 / 1000.0; m.scan_checkpoint = match self.scanner_checkpoint.lock() { @@ -4926,4 +4966,20 @@ mod tests { assert!(!report.bitrot_cycle_enabled); assert_eq!(report.bitrot_cycle_seconds, 0.0); } + + #[tokio::test] + async fn scanner_cycle_timeout_metrics_reset_for_a_new_cycle() { + let metrics = Metrics::new(); + metrics.record_scanner_cycle_timeout(true, Duration::from_secs(17)); + let timed_out = metrics.report().await; + assert_eq!(timed_out.cycle_timeout_total, 1); + assert_eq!(timed_out.cycle_last_progress_age, 17); + assert!(timed_out.leader_lease_without_progress); + + metrics.record_scanner_cycle_config(Duration::from_secs(60), None, Some(Duration::from_secs(1)), None, None); + let current = metrics.report().await; + assert_eq!(current.cycle_timeout_total, 1); + assert_eq!(current.cycle_last_progress_age, 0); + assert!(!current.leader_lease_without_progress); + } } diff --git a/crates/config/README.md b/crates/config/README.md index 02cf81d52..338b85dbf 100644 --- a/crates/config/README.md +++ b/crates/config/README.md @@ -84,6 +84,12 @@ Current guidance: - `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical) - `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical) +Scanner cycle budget controls: + +- When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance. +- An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`. +- A timed-out cycle cancels cooperative scanner work, then fences its leader epoch before releasing the lease. An uncooperative I/O operation is dropped after the bounded shutdown window; its cursor is not claimed to be durable and the scanner reports `recovery-required` when the worker cannot stop cooperatively, the cycle state was not confirmed durable, or epoch fencing cannot be persisted. + ## Mmap read environment aliases - `RUSTFS_OBJECT_MMAP_READ_ENABLE` (canonical) diff --git a/crates/config/src/constants/runtime.rs b/crates/config/src/constants/runtime.rs index 9d93c9f5e..783ad6993 100644 --- a/crates/config/src/constants/runtime.rs +++ b/crates/config/src/constants/runtime.rs @@ -57,6 +57,13 @@ pub const DEFAULT_MAX_IO_EVENTS_PER_TICK: usize = 1024; pub const DEFAULT_EVENT_INTERVAL: u32 = 61; pub const DEFAULT_RNG_SEED: Option = None; // None means random +/// Dedicated blocking thread pool for fsync/fdatasync operations. +/// When > 1, fsync operations are isolated from the main blocking pool to +/// prevent device-bound fsync from starving read operations (pread/stat/open). +/// Default 0 means auto (no isolation, use main runtime). +pub const ENV_FSYNC_BLOCKING_THREADS: &str = "RUSTFS_RUNTIME_FSYNC_BLOCKING_THREADS"; +pub const DEFAULT_FSYNC_BLOCKING_THREADS: usize = 0; + // Dial9 Tokio Telemetry Default values pub const DEFAULT_RUNTIME_DIAL9_ENABLED: bool = false; // Disabled by default pub const DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR: &str = "/var/log/rustfs/telemetry"; diff --git a/crates/config/src/constants/scanner.rs b/crates/config/src/constants/scanner.rs index 8086c3229..953ef789b 100644 --- a/crates/config/src/constants/scanner.rs +++ b/crates/config/src/constants/scanner.rs @@ -143,9 +143,12 @@ pub const ENV_SCANNER_MAX_WAIT_SECS: &str = "RUSTFS_SCANNER_MAX_WAIT_SECS"; /// Default scanner speed preset. pub const DEFAULT_SCANNER_SPEED: &str = "default"; -/// Default scanner cycle runtime budget. -/// `0` keeps the existing unbounded per-cycle behavior. -pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 0; +/// Default scanner cycle runtime budget when no override is configured. +/// +/// An explicit `0` remains the compatibility escape hatch for an unbounded +/// cycle. Keeping the unset default finite prevents a stalled scanner I/O +/// operation from holding the leader lease forever. +pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 30 * 60; /// Default scanner per-cycle object budget. /// `0` keeps the existing unbounded per-cycle behavior. diff --git a/crates/data-usage/src/data_usage.rs b/crates/data-usage/src/data_usage.rs index 9c4692a21..81a313125 100644 --- a/crates/data-usage/src/data_usage.rs +++ b/crates/data-usage/src/data_usage.rs @@ -585,9 +585,12 @@ impl VersionsHistogram { } } -/// Replication statistics for a single target -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct ReplicationStats { +/// Replication statistics for a single target. +/// +/// Renamed from `ReplicationStats`; serde field names are preserved +/// byte-identically to maintain wire compatibility with existing snapshots. +#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub struct ReplicationTargetUsage { pub pending_size: u64, pub replicated_size: u64, pub failed_size: u64, @@ -600,7 +603,7 @@ pub struct ReplicationStats { pub replicated_count: u64, } -impl ReplicationStats { +impl ReplicationTargetUsage { pub fn is_empty(&self) -> bool { let Self { pending_size, @@ -636,7 +639,7 @@ impl ReplicationStats { /// Replication statistics for all targets #[derive(Debug, Default, Clone, Serialize, Deserialize)] pub struct ReplicationAllStats { - pub targets: HashMap, + pub targets: HashMap, pub replica_size: u64, pub replica_count: u64, } @@ -649,7 +652,7 @@ impl ReplicationAllStats { targets, } = self; - *replica_size == 0 && *replica_count == 0 && targets.values().all(ReplicationStats::is_empty) + *replica_size == 0 && *replica_count == 0 && targets.values().all(ReplicationTargetUsage::is_empty) } #[deprecated(note = "use is_empty instead")] @@ -2466,7 +2469,7 @@ mod tests { #[test] fn replication_stats_empty_checks_every_field() { - type SetField = fn(&mut ReplicationStats); + type SetField = fn(&mut ReplicationTargetUsage); let cases: [(&str, SetField); 10] = [ ("pending_size", |stats| stats.pending_size = 1), @@ -2481,9 +2484,9 @@ mod tests { ("replicated_count", |stats| stats.replicated_count = 1), ]; - assert!(ReplicationStats::default().is_empty()); + assert!(ReplicationTargetUsage::default().is_empty()); for (field, set_nonzero) in cases { - let mut stats = ReplicationStats::default(); + let mut stats = ReplicationTargetUsage::default(); set_nonzero(&mut stats); assert!(!stats.is_empty(), "{field} must make replication stats non-empty"); } @@ -2514,17 +2517,17 @@ mod tests { } let empty_targets = ReplicationAllStats { - targets: HashMap::from([("arn:test:empty".to_string(), ReplicationStats::default())]), + targets: HashMap::from([("arn:test:empty".to_string(), ReplicationTargetUsage::default())]), ..Default::default() }; assert!(empty_targets.is_empty(), "all-empty targets must keep aggregate stats empty"); let stats = ReplicationAllStats { targets: HashMap::from([ - ("arn:test:empty".to_string(), ReplicationStats::default()), + ("arn:test:empty".to_string(), ReplicationTargetUsage::default()), ( "arn:test:non-empty".to_string(), - ReplicationStats { + ReplicationTargetUsage { pending_count: 1, ..Default::default() }, @@ -2565,7 +2568,7 @@ mod tests { replication_stats: Some(ReplicationAllStats { targets: HashMap::from([( "arn:test:pending".to_string(), - ReplicationStats { + ReplicationTargetUsage { pending_count: 1, ..Default::default() }, @@ -2714,7 +2717,7 @@ mod tests { targets: HashMap::from([ ( "arn:self-only".to_string(), - ReplicationStats { + ReplicationTargetUsage { pending_size: 7, pending_count: 1, ..Default::default() @@ -2722,7 +2725,7 @@ mod tests { ), ( "arn:shared".to_string(), - ReplicationStats { + ReplicationTargetUsage { failed_size: 3, failed_count: 1, missed_threshold_size: 2, @@ -2741,7 +2744,7 @@ mod tests { targets: HashMap::from([ ( "arn:shared".to_string(), - ReplicationStats { + ReplicationTargetUsage { failed_size: 5, failed_count: 2, after_threshold_size: 4, @@ -2751,7 +2754,7 @@ mod tests { ), ( "arn:other-only".to_string(), - ReplicationStats { + ReplicationTargetUsage { replicated_size: 11, replicated_count: 3, ..Default::default() @@ -2993,7 +2996,9 @@ mod tests { fn replication_target_deserialization_preserves_large_historical_maps() { let mut stats = ReplicationAllStats::default(); for index in 0..=1024 { - stats.targets.insert(format!("target-{index}"), ReplicationStats::default()); + stats + .targets + .insert(format!("target-{index}"), ReplicationTargetUsage::default()); } let encoded = rmp_serde::to_vec_named(&stats).expect("large replication target fixture should encode"); let decoded = rmp_serde::from_slice::(&encoded) @@ -3002,6 +3007,47 @@ mod tests { assert_eq!(decoded.targets.len(), stats.targets.len()); } + /// Round-trip test: encoding a [`ReplicationTargetUsage`] and decoding it back + /// must produce the exact same value. This guards against accidental serde + /// field-name drift during the `ReplicationStats` -> `ReplicationTargetUsage` + /// rename. Wire-level field names are the serialized Rust field identifiers, + /// which must remain byte-identical. + #[test] + fn replication_target_usage_rmp_round_trip() { + let original = ReplicationTargetUsage { + pending_size: 100, + replicated_size: 2_000, + failed_size: 50, + failed_count: 3, + pending_count: 7, + missed_threshold_size: 11, + after_threshold_size: 22, + missed_threshold_count: 1, + after_threshold_count: 2, + replicated_count: 99, + }; + + let buf = rmp_serde::to_vec_named(&original).expect("encode ReplicationTargetUsage to msgpack"); + let decoded: ReplicationTargetUsage = rmp_serde::from_slice(&buf).expect("decode ReplicationTargetUsage from msgpack"); + assert_eq!(original, decoded, "round-trip through rmp must preserve every field"); + + // Also verify that encoding as an unnamed sequence and then decoding + // with named fields produces the correct mapping (this catches reordering). + let named_buf = rmp_serde::to_vec_named(&original).expect("re-encode for field-name pinning"); + // Spot-check that known field names appear in the named encoding. + let named_str = String::from_utf8_lossy(&named_buf); + assert!(named_str.contains("pending_size"), "field 'pending_size' must survive the rename"); + assert!(named_str.contains("replicated_size"), "field 'replicated_size' must survive the rename"); + assert!( + named_str.contains("missed_threshold_size"), + "field 'missed_threshold_size' must survive the rename" + ); + assert!( + named_str.contains("after_threshold_count"), + "field 'after_threshold_count' must survive the rename" + ); + } + #[test] fn checked_merge_rejects_noncanonical_histograms_without_mutation() { let mut entry = DataUsageEntry { diff --git a/crates/e2e_test/AGENTS.md b/crates/e2e_test/AGENTS.md index 032f40e63..439ae8bcc 100644 --- a/crates/e2e_test/AGENTS.md +++ b/crates/e2e_test/AGENTS.md @@ -28,4 +28,3 @@ follow. ## Suggested Validation - `cargo test --package e2e_test` -- Full gate before commit: `make pre-commit` diff --git a/crates/e2e_test/Cargo.toml b/crates/e2e_test/Cargo.toml index b0bc6ed2a..302cd34d3 100644 --- a/crates/e2e_test/Cargo.toml +++ b/crates/e2e_test/Cargo.toml @@ -96,7 +96,6 @@ tokio-stream = { workspace = true } rustfs-madmin.workspace = true rustfs-filemeta.workspace = true bytes = { workspace = true, features = ["serde"] } -serial_test = { workspace = true } aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] } aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] } aws-config = { workspace = true } diff --git a/crates/e2e_test/README.md b/crates/e2e_test/README.md index a48ff6434..24567ac6f 100644 --- a/crates/e2e_test/README.md +++ b/crates/e2e_test/README.md @@ -48,16 +48,14 @@ cargo nextest run --profile e2e-smoke -p e2e_test cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \ -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' -# Protocols suite — fixed ports, MUST be single-threaded, gated by build features -RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \ - cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture ``` The protocols suite has its own contract (fixed bind ports 9022–9301, -`--test-threads=1`, feature-gated scheduling) documented in +single-worker execution, feature-gated scheduling) documented in [`src/protocols/README.md`](src/protocols/README.md). `RUSTFS_BUILD_FEATURES` selects which features the spawned binary is built with; leave it unset to run -every protocol entry. +every protocol entry. Use the exact profile command under +[Troubleshooting](#troubleshooting) for CI-equivalent execution. ### `#[ignore]` semantics @@ -159,27 +157,26 @@ construction (random port + isolated temp dir) and need no serialization. ## CI map `e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all` -pass ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) line 158, -`--exclude e2e_test`) — the whole crate is too slow to gate every PR. Subsets -join CI through the nextest profile system only (never as ad-hoc jobs): +pass (`--exclude e2e_test`) — the whole crate is too slow to gate every PR. +Subsets join CI through nextest profiles; the fixed-port protocol suite uses +the same profile for membership and execution with one nightly worker. | Suite | Runs where | Status | | --- | --- | --- | | Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) | +| Full single-node suite (`e2e-full` profile) | `e2e-full` job, merge queue + main | **Active** (backlog#1149 ci-5) | | `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) | | ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) | -| KMS suite | — | Not in CI yet (backlog#1149 ci-5) | -| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) | +| KMS suite | `e2e-full` job, merge queue + main | **Active** | +| Cluster faults (`e2e-nightly` profile) | consolidated nightly workflow | **Active** (backlog#1149 ci-7) | +| Protocols (FTPS/WebDAV/SFTP) | consolidated nightly workflow, serial | **Active** (backlog#1149 ci-7) | | Replication (fast subset) | `e2e-smoke` profile, `e2e-tests` job, every PR | **Active** (backlog#1147 repl-1) | -| Replication (slow + dual-node) | `e2e-repl-nightly` profile, scheduled workflow | **Active** (backlog#1147 repl-1) | -| `reliant/*` (pre-started server) | — | Manual only | +| Replication (slow + multi-node) | `e2e-repl-nightly` profile, consolidated nightly workflow | **Active** (backlog#1147 repl-1) | +| `reliant/*` | 19 tests in PR smoke; remaining default tests in `e2e-full` | **Active** except `#[ignore]` | -Links: [`ci.yml`](../../.github/workflows/ci.yml) `e2e-tests` (line 347), -`test-ilm-integration-serial` (line 196). The `e2e-smoke` `default-filter` in -[`.config/nextest.toml`](../../.config/nextest.toml) is the **single wiring -mechanism** — extend that filter (or add a sibling profile) to admit more -tests; do not add e2e jobs to `ci.yml`. repl-1 / ilm-3 are landing in parallel -and may add lanes; keep the table above easy to extend. +The profile filters in [`.config/nextest.toml`](../../.config/nextest.toml) are +the wiring source of truth. Committed test-ID digests under +`.config/e2e-*-selection.txt` make every membership change explicit. ## Troubleshooting @@ -188,9 +185,15 @@ and may add lanes; keep the table above easy to extend. ```bash # Smoke (e2e-tests job) — includes the 20 fast replication tests cargo nextest run --profile e2e-smoke -p e2e_test -# Replication nightly lane (16 slow + dual-node tests; install awscurl for the -# STS dual-node test, else it skips gracefully) +# Full single-node merge/main lane +cargo nextest run --profile e2e-full -p e2e_test +# Cluster fault nightly lane +cargo nextest run --profile e2e-nightly -p e2e_test +# Replication nightly lane; install awscurl so STS paths do not skip cargo nextest run --profile e2e-repl-nightly -p e2e_test +# Fixed-port protocol nightly lane +RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \ + cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture # ILM serial lane cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \ -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' @@ -273,4 +276,6 @@ current subset is. `docs/testing/e2e-suite-inventory.md` records the per-module test counts as listed by `cargo nextest list -p e2e_test`. Regenerate it when adding or moving e2e tests so acceptance numbers in the test-strategy issues -(backlog#1147–#1155) stay auditable. +(backlog#1147–#1155) stay auditable. When a profile membership change is +intentional, review its JSON listing before updating the matching +`.config/e2e-*-selection.txt` test-ID digest. diff --git a/crates/e2e_test/src/common.rs b/crates/e2e_test/src/common.rs index 42fe5f92a..7fad7ea76 100644 --- a/crates/e2e_test/src/common.rs +++ b/crates/e2e_test/src/common.rs @@ -30,6 +30,7 @@ use reqwest::StatusCode; use rustfs_signer::constants::UNSIGNED_PAYLOAD; use rustfs_signer::sign_v4; use s3s::Body; +use serde_json; use std::ffi::OsStr; use std::fs as stdfs; use std::io::ErrorKind; @@ -1583,6 +1584,156 @@ impl Drop for RustFSTestClusterEnvironment { } } +/// Send a SigV4-signed HTTP request and return the raw `reqwest::Response`. +/// +/// Unlike [`signed_s3_request`], this variant accepts `body: Option>` +/// (binary-safe) and reorders parameters so that `access_key`/`secret_key` +/// appear before the body — matching the convention used by the replication +/// extension and object-lambda e2e suites. +pub(crate) async fn signed_request( + method: http::Method, + url: &str, + access_key: &str, + secret_key: &str, + body: Option>, + content_type: Option<&str>, +) -> Result> { + let uri = url.parse::()?; + let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); + let mut request = http::Request::builder().method(method.clone()).uri(uri); + request = request.header(HOST, authority); + request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); + if let Some(content_type) = content_type { + request = request.header(CONTENT_TYPE, content_type); + } + + let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); + let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1"); + + let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; + let client = local_http_client(); + let mut request_builder = client.request(reqwest_method, url); + for (name, value) in signed.headers() { + request_builder = request_builder.header(name, value); + } + if let Some(body) = body { + request_builder = request_builder.body(body); + } + + Ok(request_builder.send().await?) +} + +/// Like [`signed_request`], but uses a caller-supplied `reqwest::Client` +/// instead of the shared [`local_http_client`]. +pub(crate) async fn signed_request_with_client( + client: &reqwest::Client, + method: http::Method, + url: &str, + access_key: &str, + secret_key: &str, + body: Option>, + content_type: Option<&str>, +) -> Result> { + let uri = url.parse::()?; + let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); + let mut request = http::Request::builder().method(method.clone()).uri(uri); + request = request.header(HOST, authority); + request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); + if let Some(content_type) = content_type { + request = request.header(CONTENT_TYPE, content_type); + } + + let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); + let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1"); + + let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; + let mut request_builder = client.request(reqwest_method, url); + for (name, value) in signed.headers() { + request_builder = request_builder.header(name, value); + } + if let Some(body) = body { + request_builder = request_builder.body(body); + } + + Ok(request_builder.send().await?) +} + +/// Like [`signed_request`], but includes a `session_token` in the +/// `x-amz-security-token` header and passes it to the SigV4 signer. +pub(crate) async fn signed_request_with_session_token( + method: http::Method, + url: &str, + access_key: &str, + secret_key: &str, + session_token: &str, + body: Option>, + content_type: Option<&str>, +) -> Result> { + let uri = url.parse::()?; + let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); + let mut request = http::Request::builder().method(method.clone()).uri(uri); + request = request.header(HOST, authority); + request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); + if !session_token.is_empty() { + request = request.header("x-amz-security-token", session_token); + } + if let Some(content_type) = content_type { + request = request.header(CONTENT_TYPE, content_type); + } + + let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); + let signed = sign_v4( + request.body(Body::empty())?, + content_len, + access_key, + secret_key, + session_token, + "us-east-1", + ); + + let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; + let client = local_http_client(); + let mut request_builder = client.request(reqwest_method, url); + for (name, value) in signed.headers() { + request_builder = request_builder.header(name, value); + } + if let Some(body) = body { + request_builder = request_builder.body(body); + } + + Ok(request_builder.send().await?) +} + +/// Create a new user via the admin API. +pub(crate) async fn admin_create_user( + env: &RustFSTestEnvironment, + username: &str, + secret_key: &str, +) -> Result<(), Box> { + let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username); + let body = serde_json::json!({ + "secretKey": secret_key, + "status": "enabled" + }); + let response = signed_request( + http::Method::PUT, + &url, + &env.access_key, + &env.secret_key, + Some(body.to_string().into_bytes()), + Some("application/json"), + ) + .await?; + + if response.status() != reqwest::StatusCode::OK { + let status = response.status(); + let body = response.text().await.unwrap_or_default(); + return Err(format!("create user failed: {status} {body}").into()); + } + + Ok(()) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/e2e_test/src/degraded_read_eof_regression_test.rs b/crates/e2e_test/src/degraded_read_eof_regression_test.rs index f5c2ada49..9b3c66649 100644 --- a/crates/e2e_test/src/degraded_read_eof_regression_test.rs +++ b/crates/e2e_test/src/degraded_read_eof_regression_test.rs @@ -55,7 +55,6 @@ mod tests { use aws_sdk_s3::Client; use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; - use serial_test::serial; use sha2::{Digest, Sha256}; use std::error::Error; use tokio::time::{Duration, timeout}; @@ -269,7 +268,6 @@ mod tests { /// stripes) and a multipart object (3 parts × 5 MiB) must GET back as a /// full, byte-identical body with the correct Content-Length. No early EOF. #[tokio::test] - #[serial] async fn degraded_read_large_objects_with_one_disk_offline_return_full_body() -> TestResult { init_logging(); info!("dist-13 (a): large-object degraded read with one of four disks offline"); @@ -335,7 +333,6 @@ mod tests { /// mid-stream — the exact window the fixes had to reconstruct through rather /// than truncate. #[tokio::test] - #[serial] async fn degraded_read_reconstructs_through_midstream_bitrot_within_quorum() -> TestResult { init_logging(); info!("dist-13 (b): mid-stream bitrot within quorum must reconstruct a full body"); @@ -393,7 +390,6 @@ mod tests { /// Content-Length. `get_checked` panics on that forbidden outcome, so this /// test fails loudly if the truncation bug ever returns. #[tokio::test] - #[serial] async fn beyond_quorum_degraded_read_never_silently_truncates() -> TestResult { init_logging(); info!("dist-13 (c): beyond-quorum degraded read must fail, never 200+truncated"); diff --git a/crates/e2e_test/src/get_stream_failure_observability_test.rs b/crates/e2e_test/src/get_stream_failure_observability_test.rs index 8777d1350..4f6bd597a 100644 --- a/crates/e2e_test/src/get_stream_failure_observability_test.rs +++ b/crates/e2e_test/src/get_stream_failure_observability_test.rs @@ -51,7 +51,6 @@ mod tests { use aws_sdk_s3::Client; use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; - use serial_test::serial; use std::error::Error; use tokio::time::{Duration, timeout}; use tracing::info; @@ -129,7 +128,6 @@ mod tests { /// the body — and assert the server log names the object, at the log level a /// default deployment actually runs with. #[tokio::test] - #[serial] async fn midstream_get_failure_is_logged_with_the_object_at_default_log_level() -> TestResult { init_logging(); info!("rustfs#4784: a mid-stream GET failure must name its object in the source log"); diff --git a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs index dac1eef5e..dd82512a5 100644 --- a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs +++ b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs @@ -380,10 +380,24 @@ mod tests { cluster.start_node(1).await?; let status_url = format!("{}/rustfs/admin/v3/background-heal/status", cluster.nodes[0].url); - let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?; - assert!( - !status_body.contains("MissingContentLength"), - "background heal status should not fail without an explicit Content-Length: {status_body}" + let mut recovered = serde_json::Value::Null; + for _ in 0..60 { + let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?; + assert!( + !status_body.contains("MissingContentLength"), + "background heal status should not fail without an explicit Content-Length: {status_body}" + ); + recovered = serde_json::from_str(&status_body) + .map_err(|err| format!("background heal status is not JSON ({err}): {status_body}"))?; + if recovered["clusterStatusComplete"] == serde_json::Value::Bool(true) { + break; + } + sleep(Duration::from_secs(1)).await; + } + assert_eq!( + recovered["clusterStatusComplete"], + serde_json::Value::Bool(true), + "cluster heal status should recover before root heal starts: {recovered}" ); let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#; diff --git a/crates/e2e_test/src/inline_fast_path_cluster_test.rs b/crates/e2e_test/src/inline_fast_path_cluster_test.rs index a4df7cd88..8de63303f 100644 --- a/crates/e2e_test/src/inline_fast_path_cluster_test.rs +++ b/crates/e2e_test/src/inline_fast_path_cluster_test.rs @@ -46,7 +46,6 @@ use prost::Message; use rustfs_signer::constants::UNSIGNED_PAYLOAD; use rustfs_signer::sign_v4; use s3s::Body; -use serial_test::serial; use std::collections::BTreeMap; use std::convert::Infallible; use std::error::Error; @@ -1695,7 +1694,6 @@ fn assert_storage_layout( } #[tokio::test] -#[serial] async fn four_node_inline_storage_and_get_boundaries() -> TestResult { init_logging(); @@ -1767,7 +1765,6 @@ async fn four_node_inline_storage_and_get_boundaries() -> TestResult { } #[tokio::test] -#[serial] async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult { init_logging(); @@ -1805,7 +1802,6 @@ async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult { } #[tokio::test] -#[serial] async fn four_node_inline_fallback_controls() -> TestResult { init_logging(); @@ -1870,7 +1866,6 @@ async fn four_node_inline_fallback_controls() -> TestResult { } #[tokio::test] -#[serial] async fn four_node_compressed_inline_fallback() -> TestResult { init_logging(); @@ -1905,7 +1900,6 @@ async fn four_node_compressed_inline_fallback() -> TestResult { /// Multipart disk compression is live again, so a compression-enabled cluster classifies multipart objects as compressed and the roundtrip (full GET plus partNumber GET) must still return the original bytes. /// Reverting the multipart compression fix must fail this test. #[tokio::test] -#[serial] async fn four_node_multipart_disk_compression_roundtrip() -> TestResult { init_logging(); @@ -1952,7 +1946,6 @@ async fn four_node_multipart_disk_compression_roundtrip() -> TestResult { /// read costs on the order of the covering part's block size against a ~5 MiB /// object. #[tokio::test] -#[serial] async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestResult { init_logging(); @@ -2019,7 +2012,6 @@ async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestRe } #[tokio::test] -#[serial] async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> TestResult { init_logging(); @@ -2123,7 +2115,6 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> Te } #[tokio::test] -#[serial] async fn four_node_add_tier_converges() -> TestResult { init_logging(); @@ -2142,7 +2133,6 @@ async fn four_node_add_tier_converges() -> TestResult { } #[tokio::test] -#[serial] async fn four_node_add_tier_converges_after_offline_node_restart_without_second_mutation() -> TestResult { init_logging(); @@ -2164,7 +2154,6 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_ } #[tokio::test] -#[serial] async fn four_node_manual_transition_job_status_survives_node_restart() -> TestResult { init_logging(); @@ -2239,7 +2228,6 @@ async fn four_node_manual_transition_job_status_survives_node_restart() -> TestR } #[tokio::test] -#[serial] async fn four_node_manual_transition_distributed_admission_conflict_reports_status_and_backpressure() -> TestResult { init_logging(); @@ -2381,7 +2369,6 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat } #[tokio::test] -#[serial] #[ignore = "manual #1508 evidence harness: starts a 4-node cluster, a remote tier, and an in-flight transition job"] async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> TestResult { init_logging(); @@ -2486,7 +2473,6 @@ async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> Tes } #[tokio::test] -#[serial] async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_transition() -> TestResult { init_logging(); @@ -2598,7 +2584,6 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_ } #[tokio::test] -#[serial] async fn four_node_transitioned_inline_fallback() -> TestResult { init_logging(); diff --git a/crates/e2e_test/src/kms/bucket_default_encryption_test.rs b/crates/e2e_test/src/kms/bucket_default_encryption_test.rs index fecba2b89..c0f0f0181 100644 --- a/crates/e2e_test/src/kms/bucket_default_encryption_test.rs +++ b/crates/e2e_test/src/kms/bucket_default_encryption_test.rs @@ -37,7 +37,7 @@ async fn test_bucket_default_sse_s3_put_object() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box> { + wait_for_kms_ready_with_timeout(base_url, access_key, secret_key, Duration::from_secs(5)).await +} + +async fn wait_for_kms_ready_with_timeout( + base_url: &str, + access_key: &str, + secret_key: &str, + total_deadline: Duration, +) -> Result<(), Box> { + let start = tokio::time::Instant::now(); + let deadline = start + total_deadline; + let mut backoff = Duration::from_millis(200); + let max_backoff = Duration::from_secs(1); + + loop { + match tokio::time::timeout_at(deadline, get_kms_status(base_url, access_key, secret_key)).await { + Ok(Ok(status)) => { + let backend_status = serde_json::from_str::(&status) + .ok() + .and_then(|value| value.get("backend_status")?.as_str().map(str::to_owned)); + if backend_status.as_deref() == Some("healthy") { + info!("KMS is ready (status: {})", status); + return Ok(()); + } + warn!( + backend_status = backend_status.as_deref().unwrap_or("missing"), + elapsed_ms = u64::try_from(start.elapsed().as_millis()).unwrap_or(u64::MAX), + "KMS not ready yet, retrying…" + ); + } + Ok(Err(e)) => { + let elapsed_ms = u64::try_from(start.elapsed().as_millis()).unwrap_or(u64::MAX); + warn!(error = %e, elapsed_ms, "KMS not ready yet, retrying…"); + } + Err(_) => return Err(format!("KMS failed to become ready within {} ms", total_deadline.as_millis()).into()), + } + + let now = tokio::time::Instant::now(); + if now >= deadline { + return Err(format!("KMS failed to become ready within {} ms", total_deadline.as_millis()).into()); + } + sleep((now + backoff).min(deadline) - now).await; + backoff = (backoff * 2).min(max_backoff); + } +} + +#[cfg(test)] +mod readiness_tests { + use super::{wait_for_kms_ready, wait_for_kms_ready_with_timeout}; + use std::sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }; + use std::time::Duration; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::TcpListener; + + #[tokio::test] + async fn kms_readiness_retries_http_success_until_backend_is_healthy() { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind readiness test server"); + let address = listener.local_addr().expect("read readiness test server address"); + let requests = Arc::new(AtomicUsize::new(0)); + let server_requests = Arc::clone(&requests); + let server = tokio::spawn(async move { + for backend_status in ["error", "healthy"] { + let (mut socket, _) = listener.accept().await.expect("accept readiness request"); + let mut request = Vec::new(); + let mut chunk = [0_u8; 1024]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket.read(&mut chunk).await.expect("read readiness request"); + if read == 0 { + break; + } + request.extend_from_slice(&chunk[..read]); + } + server_requests.fetch_add(1, Ordering::SeqCst); + + let body = format!(r#"{{"backend_status":"{backend_status}"}}"#); + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + socket.write_all(response.as_bytes()).await.expect("write readiness response"); + } + }); + + wait_for_kms_ready(&format!("http://{address}"), "access-key", "secret-key") + .await + .expect("KMS should become ready after the healthy response"); + + let observed_requests = requests.load(Ordering::SeqCst); + server.abort(); + assert_eq!(observed_requests, 2, "an HTTP 200 unhealthy status must be retried"); + } + + #[tokio::test] + async fn kms_readiness_deadline_covers_a_stalled_status_request() { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind readiness test server"); + let address = listener.local_addr().expect("read readiness test server address"); + let server = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.expect("accept readiness request"); + let mut request = [0_u8; 1024]; + let _ = socket.read(&mut request).await.expect("read readiness request"); + std::future::pending::<()>().await; + }); + + let result = tokio::time::timeout( + Duration::from_secs(1), + wait_for_kms_ready_with_timeout(&format!("http://{address}"), "access-key", "secret-key", Duration::from_millis(50)), + ) + .await + .expect("readiness helper must enforce its own deadline"); + + server.abort(); + assert!(result.is_err(), "a stalled status request must not outlive the readiness deadline"); + } +} + /// Create a default KMS key for testing and return the created key ID pub async fn create_default_key( base_url: &str, @@ -861,6 +991,13 @@ impl LocalKMSTestEnvironment { Ok(default_key_id.to_string()) } + /// Poll the KMS status endpoint until the backend reports ready. + /// + /// Prefer this over a fixed `sleep` after calling `start_rustfs_for_local_kms`. + pub async fn wait_for_kms_ready(&self) -> Result<(), Box> { + wait_for_kms_ready(&self.base_env.url, &self.base_env.access_key, &self.base_env.secret_key).await + } + /// Configure Local KMS backend with a predefined default key pub async fn configure_local_kms(&self) -> Result> { // Use a fixed, predictable default key ID diff --git a/crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs b/crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs index 1cf19a565..85a013ab3 100644 --- a/crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs +++ b/crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs @@ -61,7 +61,7 @@ async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() { ) .await .expect("failed to start RustFS with local KMS"); - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await.expect("KMS ready"); let client = kms_env.base_env.create_s3_client(); // Deliberately an UNVERSIONED bucket: that is the branch where the store layer can service @@ -160,7 +160,7 @@ async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() { ) .await .expect("failed to start RustFS with local KMS"); - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await.expect("KMS ready"); let client = kms_env.base_env.create_s3_client(); // Unversioned, and deliberately WITHOUT a bucket default-encryption rule, so the copy below @@ -256,7 +256,7 @@ async fn test_metadata_replace_self_copy_under_bucket_default_sse_stays_decrypta ) .await .expect("failed to start RustFS with local KMS"); - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await.expect("KMS ready"); let client = kms_env.base_env.create_s3_client(); let bucket = "copy-object-self-copy-bucket-default-sse-test"; diff --git a/crates/e2e_test/src/kms/copy_object_version_restore_sse_test.rs b/crates/e2e_test/src/kms/copy_object_version_restore_sse_test.rs index 3241a217d..c7a572e93 100644 --- a/crates/e2e_test/src/kms/copy_object_version_restore_sse_test.rs +++ b/crates/e2e_test/src/kms/copy_object_version_restore_sse_test.rs @@ -56,7 +56,7 @@ async fn test_self_copy_of_historical_sse_s3_version_is_readable() { ) .await .expect("failed to start RustFS with local KMS"); - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await.expect("KMS ready"); let client = kms_env.base_env.create_s3_client(); let bucket = "copy-object-version-restore-sse-test"; diff --git a/crates/e2e_test/src/kms/encryption_metadata_test.rs b/crates/e2e_test/src/kms/encryption_metadata_test.rs index a316668f6..59501430b 100644 --- a/crates/e2e_test/src/kms/encryption_metadata_test.rs +++ b/crates/e2e_test/src/kms/encryption_metadata_test.rs @@ -87,7 +87,7 @@ async fn test_head_reports_managed_metadata_for_sse_s3() -> Result<(), Box Result<(), let mut kms_env = LocalKMSTestEnvironment::new().await?; let default_key_id = kms_env.start_rustfs_for_local_kms().await?; - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await?; let s3_client = kms_env.base_env.create_s3_client(); kms_env.base_env.create_test_bucket(TEST_BUCKET).await?; @@ -250,7 +250,7 @@ async fn test_multipart_upload_writes_encrypted_data() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box> { init_logging(); if skip_if_kms_admin_tool_unavailable("test_vault_kms_end_to_end") { @@ -118,7 +115,6 @@ async fn test_vault_kms_end_to_end() -> Result<(), Box Result<(), Box> { init_logging(); if skip_if_kms_admin_tool_unavailable("test_vault_kms_key_isolation") { @@ -205,7 +201,6 @@ async fn test_vault_kms_key_isolation() -> Result<(), Box Result<(), Box> { init_logging(); if skip_if_kms_admin_tool_unavailable("test_vault_kms_large_file") { @@ -270,7 +265,6 @@ async fn test_vault_kms_large_file() -> Result<(), Box Result<(), Box> { init_logging(); if skip_if_kms_admin_tool_unavailable("test_vault_kms_multipart_upload") { @@ -301,7 +295,6 @@ async fn test_vault_kms_multipart_upload() -> Result<(), Box Result<(), Box> { init_logging(); if skip_if_kms_admin_tool_unavailable("test_vault_kms_key_operations") { diff --git a/crates/e2e_test/src/kms/mod.rs b/crates/e2e_test/src/kms/mod.rs index 5e6b9fe19..3b849fa1b 100644 --- a/crates/e2e_test/src/kms/mod.rs +++ b/crates/e2e_test/src/kms/mod.rs @@ -39,9 +39,6 @@ mod kms_edge_cases_test; #[cfg(test)] mod kms_fault_recovery_test; -#[cfg(test)] -mod test_runner; - #[cfg(test)] mod bucket_default_encryption_test; diff --git a/crates/e2e_test/src/kms/multipart_encryption_test.rs b/crates/e2e_test/src/kms/multipart_encryption_test.rs index 10d382aa9..66a03b7b2 100644 --- a/crates/e2e_test/src/kms/multipart_encryption_test.rs +++ b/crates/e2e_test/src/kms/multipart_encryption_test.rs @@ -33,7 +33,7 @@ async fn test_step1_basic_single_file_encryption() -> Result<(), Box Result<(), Bo let mut kms_env = LocalKMSTestEnvironment::new().await?; let _default_key_id = kms_env.start_rustfs_for_local_kms().await?; - tokio::time::sleep(tokio::time::Duration::from_secs(3)).await; + kms_env.wait_for_kms_ready().await?; let s3_client = kms_env.base_env.create_s3_client(); kms_env.base_env.create_test_bucket(TEST_BUCKET).await?; @@ -187,7 +187,7 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box Result<(), Box Result<(), Box &'static str { - match self { - TestCategory::CoreFunctionality => "core-functionality", - TestCategory::MultipartEncryption => "multipart-encryption", - TestCategory::EdgeCases => "edge-cases", - TestCategory::FaultRecovery => "fault-recovery", - TestCategory::Comprehensive => "comprehensive", - TestCategory::Performance => "performance", - } - } -} - -/// Test definition with metadata -#[derive(Debug, Clone)] -pub struct TestDefinition { - pub name: String, - pub description: String, - pub category: TestCategory, - pub estimated_duration: Duration, - pub is_critical: bool, -} - -impl TestDefinition { - pub fn new( - name: impl Into, - description: impl Into, - category: TestCategory, - estimated_duration: Duration, - is_critical: bool, - ) -> Self { - Self { - name: name.into(), - description: description.into(), - category, - estimated_duration, - is_critical, - } - } -} - -/// Test execution result -#[derive(Debug, Clone)] -pub struct TestResult { - pub test_name: String, - pub category: TestCategory, - pub success: bool, - pub duration: Duration, - pub error_message: Option, -} - -impl TestResult { - pub fn success(test_name: String, category: TestCategory, duration: Duration) -> Self { - Self { - test_name, - category, - success: true, - duration, - error_message: None, - } - } - - pub fn failure(test_name: String, category: TestCategory, duration: Duration, error: String) -> Self { - Self { - test_name, - category, - success: false, - duration, - error_message: Some(error), - } - } -} - -/// Comprehensive test suite configuration -#[derive(Debug, Clone)] -pub struct TestSuiteConfig { - pub categories: Vec, - pub include_critical_only: bool, - pub max_duration: Option, - pub parallel_execution: bool, -} - -impl Default for TestSuiteConfig { - fn default() -> Self { - Self { - categories: vec![ - TestCategory::CoreFunctionality, - TestCategory::MultipartEncryption, - TestCategory::EdgeCases, - TestCategory::FaultRecovery, - TestCategory::Comprehensive, - ], - include_critical_only: false, - max_duration: None, - parallel_execution: false, - } - } -} - -/// Unified KMS test suite runner -pub struct KMSTestSuite { - tests: Vec, - config: TestSuiteConfig, -} - -impl KMSTestSuite { - /// Create a new test suite with default configuration - pub fn new() -> Self { - let tests = vec![ - // Core Functionality Tests - TestDefinition::new( - "test_local_kms_end_to_end", - "End-to-end KMS test with all encryption types", - TestCategory::CoreFunctionality, - Duration::from_secs(60), - true, - ), - TestDefinition::new( - "test_local_kms_key_isolation", - "Test KMS key isolation and security", - TestCategory::CoreFunctionality, - Duration::from_secs(45), - true, - ), - // Multipart Encryption Tests - TestDefinition::new( - "test_local_kms_multipart_upload", - "Test large file multipart upload with encryption", - TestCategory::MultipartEncryption, - Duration::from_secs(120), - true, - ), - TestDefinition::new( - "test_step1_basic_single_file_encryption", - "Basic single file encryption test", - TestCategory::MultipartEncryption, - Duration::from_secs(30), - false, - ), - TestDefinition::new( - "test_step2_basic_multipart_upload_without_encryption", - "Basic multipart upload without encryption", - TestCategory::MultipartEncryption, - Duration::from_secs(45), - false, - ), - TestDefinition::new( - "test_step3_multipart_upload_with_sse_s3", - "Multipart upload with SSE-S3 encryption", - TestCategory::MultipartEncryption, - Duration::from_secs(60), - true, - ), - TestDefinition::new( - "test_step4_large_multipart_upload_with_encryption", - "Large file multipart upload with encryption", - TestCategory::MultipartEncryption, - Duration::from_secs(90), - false, - ), - TestDefinition::new( - "test_step5_all_encryption_types_multipart", - "All encryption types multipart test", - TestCategory::MultipartEncryption, - Duration::from_secs(120), - true, - ), - // Edge Cases Tests - TestDefinition::new( - "test_kms_zero_byte_file_encryption", - "Test encryption of zero-byte files", - TestCategory::EdgeCases, - Duration::from_secs(20), - false, - ), - TestDefinition::new( - "test_kms_single_byte_file_encryption", - "Test encryption of single-byte files", - TestCategory::EdgeCases, - Duration::from_secs(20), - false, - ), - TestDefinition::new( - "test_kms_multipart_boundary_conditions", - "Test multipart upload boundary conditions", - TestCategory::EdgeCases, - Duration::from_secs(45), - false, - ), - TestDefinition::new( - "test_kms_invalid_key_scenarios", - "Test invalid key scenarios", - TestCategory::EdgeCases, - Duration::from_secs(30), - false, - ), - TestDefinition::new( - "test_kms_concurrent_encryption", - "Test concurrent encryption operations", - TestCategory::EdgeCases, - Duration::from_secs(60), - false, - ), - TestDefinition::new( - "test_kms_key_validation_security", - "Test key validation security", - TestCategory::EdgeCases, - Duration::from_secs(30), - false, - ), - // Fault Recovery Tests - TestDefinition::new( - "test_kms_key_directory_unavailable", - "Test KMS when key directory is unavailable", - TestCategory::FaultRecovery, - Duration::from_secs(45), - false, - ), - TestDefinition::new( - "test_kms_corrupted_key_files", - "Test KMS with corrupted key files", - TestCategory::FaultRecovery, - Duration::from_secs(30), - false, - ), - TestDefinition::new( - "test_kms_multipart_upload_interruption", - "Test multipart upload interruption recovery", - TestCategory::FaultRecovery, - Duration::from_secs(60), - false, - ), - TestDefinition::new( - "test_kms_resource_constraints", - "Test KMS under resource constraints", - TestCategory::FaultRecovery, - Duration::from_secs(90), - false, - ), - // Comprehensive Tests - TestDefinition::new( - "test_comprehensive_kms_full_workflow", - "Full KMS workflow comprehensive test", - TestCategory::Comprehensive, - Duration::from_secs(300), - true, - ), - TestDefinition::new( - "test_comprehensive_stress_test", - "KMS stress test with large datasets", - TestCategory::Comprehensive, - Duration::from_secs(400), - false, - ), - TestDefinition::new( - "test_comprehensive_key_isolation", - "Comprehensive key isolation test", - TestCategory::Comprehensive, - Duration::from_secs(180), - false, - ), - TestDefinition::new( - "test_comprehensive_concurrent_operations", - "Comprehensive concurrent operations test", - TestCategory::Comprehensive, - Duration::from_secs(240), - false, - ), - TestDefinition::new( - "test_comprehensive_performance_benchmark", - "KMS performance benchmark test", - TestCategory::Comprehensive, - Duration::from_secs(360), - false, - ), - ]; - - Self { - tests, - config: TestSuiteConfig::default(), - } - } - - /// Configure the test suite - pub fn with_config(mut self, config: TestSuiteConfig) -> Self { - self.config = config; - self - } - - /// Filter tests based on category - pub fn filter_by_category(&self, category: &TestCategory) -> Vec<&TestDefinition> { - self.tests.iter().filter(|test| &test.category == category).collect() - } - - /// Filter tests based on criticality - pub fn filter_critical_tests(&self) -> Vec<&TestDefinition> { - self.tests.iter().filter(|test| test.is_critical).collect() - } - - /// Get test summary by category - pub fn get_category_summary(&self) -> std::collections::HashMap> { - let mut summary = std::collections::HashMap::new(); - for test in &self.tests { - summary.entry(test.category.clone()).or_insert_with(Vec::new).push(test); - } - summary - } - - /// Run the complete test suite - pub async fn run_test_suite(&self) -> Vec { - init_logging(); - info!("🚀 Starting unified KMS test suite"); - - let start_time = Instant::now(); - let mut results = Vec::new(); - - // Filter tests based on configuration - let tests_to_run: Vec<&TestDefinition> = self - .tests - .iter() - .filter(|test| self.config.categories.contains(&test.category)) - .filter(|test| !self.config.include_critical_only || test.is_critical) - .collect(); - - info!("📊 Test plan: {} test(s) scheduled", tests_to_run.len()); - for (i, test) in tests_to_run.iter().enumerate() { - info!(" {}. {} ({})", i + 1, test.name, test.category.as_str()); - } - - // Execute tests - for (i, test_def) in tests_to_run.iter().enumerate() { - info!("🧪 Running test {}/{}: {}", i + 1, tests_to_run.len(), test_def.name); - info!(" 📝 Description: {}", test_def.description); - info!(" 🏷️ Category: {}", test_def.category.as_str()); - info!(" ⏱️ Estimated duration: {:?}", test_def.estimated_duration); - - let test_start = Instant::now(); - let result = self.run_single_test(test_def).await; - let test_duration = test_start.elapsed(); - - match result { - Ok(_) => { - info!("✅ Test passed: {} ({:.2}s)", test_def.name, test_duration.as_secs_f64()); - results.push(TestResult::success(test_def.name.clone(), test_def.category.clone(), test_duration)); - } - Err(e) => { - error!("❌ Test failed: {} ({:.2}s): {}", test_def.name, test_duration.as_secs_f64(), e); - results.push(TestResult::failure( - test_def.name.clone(), - test_def.category.clone(), - test_duration, - e.to_string(), - )); - } - } - - // Add delay between tests to avoid resource conflicts - if i < tests_to_run.len() - 1 { - debug!("⏸️ Waiting two seconds before the next test..."); - sleep(Duration::from_secs(2)).await; - } - } - - let total_duration = start_time.elapsed(); - self.print_test_summary(&results, total_duration); - - results - } - - /// Run a single test by dispatching to the appropriate test function - async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box> { - // This is a placeholder for test dispatch logic - // In a real implementation, this would dispatch to actual test functions - warn!("⚠️ Test '{}' is not implemented in the unified runner; skipping", test_def.name); - Ok(()) - } - - /// Print comprehensive test summary - fn print_test_summary(&self, results: &[TestResult], total_duration: Duration) { - info!("📊 KMS test suite summary"); - info!("⏱️ Total duration: {:.2} seconds", total_duration.as_secs_f64()); - info!("📈 Total tests: {}", results.len()); - - let passed = results.iter().filter(|r| r.success).count(); - let failed = results.iter().filter(|r| !r.success).count(); - - info!("✅ Passed: {}", passed); - info!("❌ Failed: {}", failed); - info!("📊 Success rate: {:.1}%", (passed as f64 / results.len() as f64) * 100.0); - - // Summary by category - let mut category_summary: std::collections::HashMap = std::collections::HashMap::new(); - for result in results { - let (total, passed_count) = category_summary.entry(result.category.clone()).or_insert((0, 0)); - *total += 1; - if result.success { - *passed_count += 1; - } - } - - info!("📊 Category summary:"); - for (category, (total, passed_count)) in category_summary { - info!( - " 🏷️ {}: {}/{} ({:.1}%)", - category.as_str(), - passed_count, - total, - (passed_count as f64 / total as f64) * 100.0 - ); - } - - // List failed tests - if failed > 0 { - warn!("❌ Failing tests:"); - for result in results.iter().filter(|r| !r.success) { - warn!(" - {}: {}", result.test_name, result.error_message.as_deref().unwrap_or("Unknown error")); - } - } - } -} - -/// Quick test suite for critical tests only -#[tokio::test] -async fn test_kms_critical_suite() -> Result<(), Box> { - let config = TestSuiteConfig { - categories: vec![TestCategory::CoreFunctionality, TestCategory::MultipartEncryption], - include_critical_only: true, - max_duration: Some(Duration::from_secs(600)), // 10 minutes max - parallel_execution: false, - }; - - let suite = KMSTestSuite::new().with_config(config); - let results = suite.run_test_suite().await; - - let failed_count = results.iter().filter(|r| !r.success).count(); - if failed_count > 0 { - return Err(format!("Critical test suite failed: {failed_count} tests failed").into()); - } - - info!("✅ All critical tests passed"); - Ok(()) -} - -/// Full comprehensive test suite -#[tokio::test] -async fn test_kms_full_suite() -> Result<(), Box> { - let suite = KMSTestSuite::new(); - let results = suite.run_test_suite().await; - - let total_tests = results.len(); - let failed_count = results.iter().filter(|r| !r.success).count(); - let success_rate = ((total_tests - failed_count) as f64 / total_tests as f64) * 100.0; - - info!("📊 Full suite success rate: {:.1}%", success_rate); - - // Allow up to 10% failure rate for non-critical tests - if success_rate < 90.0 { - return Err(format!("Test suite success rate too low: {success_rate:.1}%").into()); - } - - info!("✅ Full test suite succeeded"); - Ok(()) -} diff --git a/crates/e2e_test/src/object_lambda_test.rs b/crates/e2e_test/src/object_lambda_test.rs index 66f259e6d..d2d5f0036 100644 --- a/crates/e2e_test/src/object_lambda_test.rs +++ b/crates/e2e_test/src/object_lambda_test.rs @@ -12,12 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client}; +use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client, signed_request}; use aws_sdk_s3::primitives::ByteStream; use http::header::{CONTENT_TYPE, HOST}; use reqwest::StatusCode; -use rustfs_signer::constants::UNSIGNED_PAYLOAD; -use rustfs_signer::{pre_sign_v4, sign_v4}; +use rustfs_config::{ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, ENV_NOTIFY_ENABLE}; +use rustfs_signer::pre_sign_v4; use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS; use s3s::Body; use std::collections::HashMap; @@ -227,39 +227,6 @@ async fn presigned_get_request( Ok(local_http_client().get(signed.uri().to_string()).send().await?) } -async fn signed_request( - method: http::Method, - url: &str, - access_key: &str, - secret_key: &str, - body: Option>, - content_type: Option<&str>, -) -> Result> { - let uri = url.parse::()?; - let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); - let mut request = http::Request::builder().method(method.clone()).uri(uri); - request = request.header(HOST, authority); - request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); - if let Some(content_type) = content_type { - request = request.header(CONTENT_TYPE, content_type); - } - - let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); - let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1"); - - let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; - let client = local_http_client(); - let mut request_builder = client.request(reqwest_method, url); - for (name, value) in signed.headers() { - request_builder = request_builder.header(name, value); - } - if let Some(body) = body { - request_builder = request_builder.body(body); - } - - Ok(request_builder.send().await?) -} - async fn configure_webhook_target( env: &RustFSTestEnvironment, target_name: &str, @@ -1010,7 +977,8 @@ async fn test_get_object_lambda_rejects_disabled_target() -> Result<(), Box Result<(), Box Resul init_logging(); let mut env = RustFSTestEnvironment::new().await?; - env.start_rustfs_server(vec![]).await?; + env.start_rustfs_server_with_env(vec![], &[(ENV_NOTIFY_ENABLE, "true")]) + .await?; let bucket = "object-lambda-e2e-invalid-endpoint"; @@ -1098,7 +1074,8 @@ async fn test_configure_object_lambda_notify_webhook_rejects_response_header_tim init_logging(); let mut env = RustFSTestEnvironment::new().await?; - env.start_rustfs_server(vec![]).await?; + env.start_rustfs_server_with_env(vec![], &[(ENV_NOTIFY_ENABLE, "true")]) + .await?; let response = send_configure_webhook_target_request( &env, @@ -1207,6 +1184,8 @@ async fn test_listen_notification_fans_in_remote_node_events() -> Result<(), Box init_logging(); let mut cluster = RustFSTestClusterEnvironment::new(2).await?; + cluster.set_env(ENV_NOTIFY_ENABLE, "true"); + cluster.set_env(ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, "1"); cluster.start().await?; let bucket = "listen-notification-cluster"; diff --git a/crates/e2e_test/src/policy/policy_variables_test.rs b/crates/e2e_test/src/policy/policy_variables_test.rs index 52c6492d0..e9b603822 100644 --- a/crates/e2e_test/src/policy/policy_variables_test.rs +++ b/crates/e2e_test/src/policy/policy_variables_test.rs @@ -17,7 +17,6 @@ use crate::common::{awscurl_delete, awscurl_put, init_logging}; use crate::policy::test_env::PolicyTestEnvironment; use aws_sdk_s3::primitives::ByteStream; -use serial_test::serial; use tracing::info; /// Helper function to create a regular user with given credentials @@ -122,7 +121,6 @@ async fn cleanup_user_and_policy(env: &PolicyTestEnvironment, username: &str, po /// Test AWS policy variables with single-value scenarios #[tokio::test(flavor = "multi_thread")] -#[serial] #[ignore = "Starts a rustfs server; enable when running full E2E"] pub async fn test_aws_policy_variables_single_value() -> Result<(), Box> { test_aws_policy_variables_single_value_impl().await @@ -275,7 +273,6 @@ pub async fn test_aws_policy_variables_single_value_impl_with_env( /// Test AWS policy variables with multi-value scenarios #[tokio::test(flavor = "multi_thread")] -#[serial] #[ignore = "Starts a rustfs server; enable when running full E2E"] pub async fn test_aws_policy_variables_multi_value() -> Result<(), Box> { test_aws_policy_variables_multi_value_impl().await @@ -401,7 +398,6 @@ pub async fn test_aws_policy_variables_multi_value_impl_with_env( /// Test AWS policy variables with variable concatenation #[tokio::test(flavor = "multi_thread")] -#[serial] #[ignore = "Starts a rustfs server; enable when running full E2E"] pub async fn test_aws_policy_variables_concatenation() -> Result<(), Box> { test_aws_policy_variables_concatenation_impl().await @@ -491,7 +487,6 @@ pub async fn test_aws_policy_variables_concatenation_impl_with_env( /// Test AWS policy variables with nested scenarios #[tokio::test(flavor = "multi_thread")] -#[serial] #[ignore = "Starts a rustfs server; enable when running full E2E"] pub async fn test_aws_policy_variables_nested() -> Result<(), Box> { test_aws_policy_variables_nested_impl().await @@ -509,7 +504,6 @@ pub async fn test_aws_policy_variables_nested_impl() -> Result<(), Box Result<(), Box> { test_aws_policy_variables_sts_impl().await @@ -705,7 +699,6 @@ pub async fn test_aws_policy_variables_sts_impl_with_env( /// Test AWS policy variables with deny scenarios #[tokio::test(flavor = "multi_thread")] -#[serial] #[ignore = "Starts a rustfs server; enable when running full E2E"] pub async fn test_aws_policy_variables_deny() -> Result<(), Box> { test_aws_policy_variables_deny_impl().await diff --git a/crates/e2e_test/src/policy/test_runner.rs b/crates/e2e_test/src/policy/test_runner.rs index 2194db2fb..5402f72eb 100644 --- a/crates/e2e_test/src/policy/test_runner.rs +++ b/crates/e2e_test/src/policy/test_runner.rs @@ -14,7 +14,6 @@ use crate::common::init_logging; use crate::policy::test_env::PolicyTestEnvironment; -use serial_test::serial; use std::time::Instant; use tokio::time::{Duration, sleep}; use tracing::{error, info}; @@ -213,7 +212,6 @@ impl PolicyTestSuite { /// Test suite #[tokio::test] -#[serial] #[ignore = "Connects to existing rustfs server"] async fn test_policy_critical_suite() -> Result<(), Box> { let config = TestSuiteConfig { diff --git a/crates/e2e_test/src/protocols/README.md b/crates/e2e_test/src/protocols/README.md index 0ecd3fad6..0cbb222b1 100644 --- a/crates/e2e_test/src/protocols/README.md +++ b/crates/e2e_test/src/protocols/README.md @@ -11,10 +11,17 @@ test process directly. ## Running Tests +Use the canonical CI-equivalent protocol command in the parent +[`e2e_test` README](../../README.md#troubleshooting). + +For targeted debugging of the core suite only: + ```bash RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture ``` +This targeted command does not cover the full `e2e-protocols` profile. + `RUSTFS_BUILD_FEATURES` controls which features the test rustfs binary is built with. When this variable is set, the protocol test runner schedules only entries whose protocol is present in the requested feature list. Leave @@ -133,4 +140,3 @@ property without consulting any external doc. Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with `RUSTFS_SFTP_IDLE_TIMEOUT=5`, sleeps 10 s past the timeout, then issues an SFTP request and asserts the server has closed the session. - diff --git a/crates/e2e_test/src/protocols/webdav_core.rs b/crates/e2e_test/src/protocols/webdav_core.rs index 3b15a1d43..90c266c69 100644 --- a/crates/e2e_test/src/protocols/webdav_core.rs +++ b/crates/e2e_test/src/protocols/webdav_core.rs @@ -41,7 +41,6 @@ use reqwest::Client; use rustfs_signer::constants::UNSIGNED_PAYLOAD; use rustfs_signer::sign_v4; use s3s::Body; -use serial_test::serial; use tokio::process::Command; use tracing::info; @@ -821,7 +820,6 @@ pub async fn test_webdav_core_operations() -> Result<()> { } #[tokio::test] -#[serial] async fn test_webdav_core_operations_direct() -> Result<()> { test_webdav_core_operations().await } diff --git a/crates/e2e_test/src/reliability_disk_fault_test.rs b/crates/e2e_test/src/reliability_disk_fault_test.rs index 3b2ae3e29..c77249716 100644 --- a/crates/e2e_test/src/reliability_disk_fault_test.rs +++ b/crates/e2e_test/src/reliability_disk_fault_test.rs @@ -27,7 +27,6 @@ mod tests { use aws_sdk_s3::Client; use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration}; - use serial_test::serial; use sha2::{Digest, Sha256}; use std::collections::HashSet; use std::error::Error; @@ -157,7 +156,6 @@ mod tests { /// content, degraded writes must succeed, and everything must still /// verify after the disk returns. #[tokio::test] - #[serial] async fn test_degraded_read_write_with_one_disk_offline() -> Result<(), Box> { init_logging(); info!("Reliability: degraded read/write with one of four disks offline"); @@ -210,7 +208,6 @@ mod tests { /// bytes to a reader: per-shard bitrot checksums reject the bad shard and /// the object is reconstructed from the remaining shards. #[tokio::test] - #[serial] async fn test_bitrot_corrupted_shard_read_returns_correct_data() -> Result<(), Box> { init_logging(); info!("Reliability: GET must read through a bitrot-corrupted shard"); @@ -253,7 +250,6 @@ mod tests { /// heal, and require the replaced disk to be rebuilt and all content to /// verify against the sha256 manifest. #[tokio::test] - #[serial] async fn test_fresh_disk_replacement_heals_after_sigkill_restart() -> Result<(), Box> { init_logging(); info!("Reliability: fresh-disk replacement heals after SIGKILL restart"); @@ -327,7 +323,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_versioned_shard_census_selects_each_version_data_dir() -> Result<(), Box> { init_logging(); info!("Reliability: physical shard census selects the requested object version"); diff --git a/crates/e2e_test/src/replacement_privileged_e2e_test.rs b/crates/e2e_test/src/replacement_privileged_e2e_test.rs index 67c1b0097..fc687d4e3 100644 --- a/crates/e2e_test/src/replacement_privileged_e2e_test.rs +++ b/crates/e2e_test/src/replacement_privileged_e2e_test.rs @@ -29,7 +29,6 @@ mod tests { use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration}; use http::Method; - use serial_test::serial; use sha2::{Digest, Sha256}; use std::collections::BTreeSet; use std::error::Error; @@ -1061,7 +1060,6 @@ mod tests { /// Linux mount namespaces are per-thread; keep mount setup and process /// spawning on one OS thread so child RustFS nodes inherit the test mounts. #[tokio::test(flavor = "current_thread")] - #[serial] #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_REPLACEMENT_E2E=1"] async fn test_privileged_3x4_auto_replacement_rebuilds_ec8_plus_4_without_admin_heal() -> Result<(), Box> { @@ -1075,7 +1073,6 @@ mod tests { /// Linux mount namespaces are per-thread; keep mount setup and process /// spawning on one OS thread so child RustFS nodes inherit the test mounts. #[tokio::test(flavor = "current_thread")] - #[serial] #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_REPLACEMENT_E2E=1"] async fn test_privileged_3x4_auto_replacement_rebuilds_ec6_plus_6_without_admin_heal() -> Result<(), Box> { diff --git a/crates/e2e_test/src/replication_extension_test.rs b/crates/e2e_test/src/replication_extension_test.rs index d0282c83a..3d51d139b 100644 --- a/crates/e2e_test/src/replication_extension_test.rs +++ b/crates/e2e_test/src/replication_extension_test.rs @@ -13,8 +13,9 @@ // limitations under the License. use crate::common::{ - RustFSTestEnvironment, awscurl_available, awscurl_post_sts_form_urlencoded, init_logging, local_http_client, - replication_fast_env, rustfs_binary_path, + RustFSTestEnvironment, admin_create_user, awscurl_available, awscurl_post_sts_form_urlencoded, init_logging, + local_http_client, replication_fast_env, rustfs_binary_path, signed_request, signed_request_with_client, + signed_request_with_session_token, }; use crate::fake_s3_target::{ FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation, @@ -35,7 +36,7 @@ use base64::{Engine, engine::general_purpose::STANDARD as BASE64_STANDARD}; use bytes::Bytes; use flate2::read::GzDecoder; use futures::{Stream, StreamExt}; -use http::header::{CONTENT_ENCODING, CONTENT_TYPE, HOST}; +use http::header::CONTENT_ENCODING; use http_body_util::{BodyExt, Full}; use hyper::body::Incoming; use hyper::server::conn::http1; @@ -56,9 +57,6 @@ use rustfs_madmin::{ AddServiceAccountReq, ListServiceAccountsResp, PeerInfo, PeerSite, ReplicateAddStatus, ReplicateEditStatus, ReplicateRemoveStatus, SRRemoveReq, SRResyncOpStatus, SRStatusInfo, SiteReplicationInfo, SyncStatus, }; -use rustfs_signer::constants::UNSIGNED_PAYLOAD; -use rustfs_signer::sign_v4; -use s3s::Body; use s3s::header::X_AMZ_REPLICATION_STATUS; use sha2::{Digest, Sha256}; use std::collections::BTreeMap; @@ -387,116 +385,6 @@ struct ReplicationResetStatusTarget { object: String, } -async fn signed_request( - method: http::Method, - url: &str, - access_key: &str, - secret_key: &str, - body: Option>, - content_type: Option<&str>, -) -> Result> { - let uri = url.parse::()?; - let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); - let mut request = http::Request::builder().method(method.clone()).uri(uri); - request = request.header(HOST, authority); - request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); - if let Some(content_type) = content_type { - request = request.header(CONTENT_TYPE, content_type); - } - - let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); - let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1"); - - let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; - let client = local_http_client(); - let mut request_builder = client.request(reqwest_method, url); - for (name, value) in signed.headers() { - request_builder = request_builder.header(name, value); - } - if let Some(body) = body { - request_builder = request_builder.body(body); - } - - Ok(request_builder.send().await?) -} - -async fn signed_request_with_client( - client: &reqwest::Client, - method: http::Method, - url: &str, - access_key: &str, - secret_key: &str, - body: Option>, - content_type: Option<&str>, -) -> Result> { - let uri = url.parse::()?; - let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); - let mut request = http::Request::builder().method(method.clone()).uri(uri); - request = request.header(HOST, authority); - request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); - if let Some(content_type) = content_type { - request = request.header(CONTENT_TYPE, content_type); - } - - let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); - let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1"); - - let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; - let mut request_builder = client.request(reqwest_method, url); - for (name, value) in signed.headers() { - request_builder = request_builder.header(name, value); - } - if let Some(body) = body { - request_builder = request_builder.body(body); - } - - Ok(request_builder.send().await?) -} - -async fn signed_request_with_session_token( - method: http::Method, - url: &str, - access_key: &str, - secret_key: &str, - session_token: &str, - body: Option>, - content_type: Option<&str>, -) -> Result> { - let uri = url.parse::()?; - let authority = uri.authority().ok_or("request URL missing authority")?.to_string(); - let mut request = http::Request::builder().method(method.clone()).uri(uri); - request = request.header(HOST, authority); - request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD); - if !session_token.is_empty() { - request = request.header("x-amz-security-token", session_token); - } - if let Some(content_type) = content_type { - request = request.header(CONTENT_TYPE, content_type); - } - - let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default(); - let signed = sign_v4( - request.body(Body::empty())?, - content_len, - access_key, - secret_key, - session_token, - "us-east-1", - ); - - let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?; - let client = local_http_client(); - let mut request_builder = client.request(reqwest_method, url); - for (name, value) in signed.headers() { - request_builder = request_builder.header(name, value); - } - if let Some(body) = body { - request_builder = request_builder.body(body); - } - - Ok(request_builder.send().await?) -} - fn extract_xml_tag(xml: &str, tag: &str) -> Option { let open = format!("<{tag}>"); let close = format!(""); @@ -1016,35 +904,6 @@ fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_k Client::from_conf(config) } -async fn admin_create_user( - env: &RustFSTestEnvironment, - username: &str, - secret_key: &str, -) -> Result<(), Box> { - let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username); - let body = serde_json::json!({ - "secretKey": secret_key, - "status": "enabled" - }); - let response = signed_request( - http::Method::PUT, - &url, - &env.access_key, - &env.secret_key, - Some(body.to_string().into_bytes()), - Some("application/json"), - ) - .await?; - - if response.status() != StatusCode::OK { - let status = response.status(); - let body = response.text().await.unwrap_or_default(); - return Err(format!("create user failed: {status} {body}").into()); - } - - Ok(()) -} - async fn admin_add_canned_policy( env: &RustFSTestEnvironment, policy_name: &str, diff --git a/crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs b/crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs index c5fc45e17..82d0924bc 100644 --- a/crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs +++ b/crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs @@ -15,7 +15,6 @@ use crate::common::{RustFSTestClusterEnvironment, init_logging}; use aws_sdk_s3::error::SdkError; use aws_sdk_s3::primitives::ByteStream; -use aws_sdk_s3::types::CompletedMultipartUpload; use tokio::time::{Duration, sleep}; use tracing::info; use uuid::Uuid; @@ -43,32 +42,18 @@ async fn list_parts_reports_missing_upload( } } -async fn complete_reports_missing_upload( +async fn multipart_listing_reports_missing_upload( client: &aws_sdk_s3::Client, bucket: &str, key: &str, upload_id: &str, ) -> Result> { - let result = client - .complete_multipart_upload() - .bucket(bucket) - .key(key) - .upload_id(upload_id) - .multipart_upload(CompletedMultipartUpload::builder().build()) - .send() - .await; - match result { - Ok(_) => Ok(false), - Err(SdkError::ServiceError(err)) => { - let code = err.err().meta().code().unwrap_or(""); - if code == "NoSuchUpload" { - Ok(true) - } else { - Err(format!("unexpected complete_multipart_upload service error: code={code}, err={err:?}").into()) - } - } - Err(err) => Err(format!("unexpected complete_multipart_upload error: {err:?}").into()), - } + let result = client.list_multipart_uploads().bucket(bucket).prefix(key).send().await?; + + Ok(!result + .uploads() + .iter() + .any(|upload| upload.key() == Some(key) && upload.upload_id() == Some(upload_id))) } async fn wait_for_cleanup_on_all_nodes( @@ -81,8 +66,8 @@ async fn wait_for_cleanup_on_all_nodes( let mut all_cleaned = true; for (idx, client) in clients.iter().enumerate() { let list_parts_missing = list_parts_reports_missing_upload(client, bucket, key, upload_id).await?; - let complete_missing = complete_reports_missing_upload(client, bucket, key, upload_id).await?; - if !(list_parts_missing && complete_missing) { + let listing_missing = multipart_listing_reports_missing_upload(client, bucket, key, upload_id).await?; + if !(list_parts_missing && listing_missing) { info!("stale multipart still visible on node {} at attempt {}", idx, attempt + 1); all_cleaned = false; break; @@ -146,6 +131,10 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster() 1, "multipart upload should be visible before background cleanup" ); + assert!( + !multipart_listing_reports_missing_upload(&clients[2], CLEANUP_BUCKET, &key, &upload_id).await?, + "multipart upload listing should contain the upload before background cleanup" + ); wait_for_cleanup_on_all_nodes(&clients, CLEANUP_BUCKET, &key, &upload_id).await?; diff --git a/crates/ecstore/AGENTS.md b/crates/ecstore/AGENTS.md index 10101f55e..a9c128522 100644 --- a/crates/ecstore/AGENTS.md +++ b/crates/ecstore/AGENTS.md @@ -49,4 +49,3 @@ Applies to `crates/ecstore/`. ## Suggested Validation - `cargo test -p rustfs-ecstore` -- Full gate before commit: `make pre-commit` diff --git a/crates/ecstore/src/api/mod.rs b/crates/ecstore/src/api/mod.rs index 17fffac3d..456d38ac7 100644 --- a/crates/ecstore/src/api/mod.rs +++ b/crates/ecstore/src/api/mod.rs @@ -317,8 +317,6 @@ pub mod config { } pub mod data_usage { - #[cfg(feature = "test-util")] - pub use crate::data_usage::seed_bucket_usage_memory_for_test; pub use crate::data_usage::{ DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage, init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache, @@ -330,6 +328,8 @@ pub mod data_usage { remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend, store_data_usage_in_backend, }; + #[cfg(feature = "test-util")] + pub use crate::data_usage::{get_bucket_usage_memory, seed_bucket_usage_memory_for_test}; } pub mod disk { diff --git a/crates/ecstore/src/bucket/bucket_target_sys.rs b/crates/ecstore/src/bucket/bucket_target_sys.rs index f5ac0c013..7e5a5df5f 100644 --- a/crates/ecstore/src/bucket/bucket_target_sys.rs +++ b/crates/ecstore/src/bucket/bucket_target_sys.rs @@ -866,7 +866,7 @@ impl BucketTargetSys { return Some(cli); } - // TODO: spawn a task to reload the target + // TODO(backlog): spawn an async task to proactively reload the replication target if self.is_reloading_target(bucket, arn).await { return None; } @@ -3425,6 +3425,44 @@ mod tests { assert!(mutexes.contains_key("second")); } + #[tokio::test] + async fn update_all_targets_publishes_disable_proxy_on_target_client() { + // The read-proxy selector (replication_proxy::get_proxy_targets) skips + // targets whose TargetClient carries disable_proxy — the persisted + // per-target opt-out must survive client publication. + let sys = BucketTargetSys::default(); + let target = |arn: &str, disable_proxy: bool| BucketTarget { + arn: arn.to_string(), + endpoint: "192.168.1.10:9000".to_string(), + target_bucket: "target-bucket".to_string(), + region: "us-east-1".to_string(), + disable_proxy, + credentials: Some(Credentials { + access_key: "access".to_string(), + secret_key: "secret".to_string(), + session_token: None, + expiration: None, + }), + ..Default::default() + }; + let targets = BucketTargets { + targets: vec![target("arn:proxied", false), target("arn:opted-out", true)], + }; + + sys.update_all_targets("bucket", Some(&targets)).await; + + let proxied = sys + .get_remote_target_client("bucket", "arn:proxied") + .await + .expect("client should be published"); + assert!(!proxied.disable_proxy); + let opted_out = sys + .get_remote_target_client("bucket", "arn:opted-out") + .await + .expect("client should be published"); + assert!(opted_out.disable_proxy, "disable_proxy must reach the published TargetClient"); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn target_updates_serialize_client_build_through_publication_per_bucket() { let sys = Arc::new(BucketTargetSys::default()); diff --git a/crates/ecstore/src/bucket/replication/replication_pool.rs b/crates/ecstore/src/bucket/replication/replication_pool.rs index efe517b99..61dbf2f1e 100644 --- a/crates/ecstore/src/bucket/replication/replication_pool.rs +++ b/crates/ecstore/src/bucket/replication/replication_pool.rs @@ -2855,7 +2855,7 @@ fn replicate_object_info_from_object_info( .map(|v| OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)); let mut rstate = oi.replication_state(); rstate.replicate_decision_str = dsc.to_string(); - let asz = oi.get_actual_size().unwrap_or_default(); + let asz = oi.get_actual_size_or_physical(); let ssec = replication_object_is_ssec_encrypted(&oi.user_defined); let checksum = if ssec { oi.checksum.clone() } else { None }; diff --git a/crates/ecstore/src/bucket/replication/replication_resyncer.rs b/crates/ecstore/src/bucket/replication/replication_resyncer.rs index d921ea411..e670a7d6f 100644 --- a/crates/ecstore/src/bucket/replication/replication_resyncer.rs +++ b/crates/ecstore/src/bucket/replication/replication_resyncer.rs @@ -1412,7 +1412,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC }; let mut replication_state = oi.replication_state(); replication_state.replicate_decision_str = dsc.to_string(); - let actual_size = oi.get_actual_size().unwrap_or_default(); + let actual_size = oi.get_actual_size_or_physical(); Ok(ReplicateObjectInfo { name: oi.name.clone(), diff --git a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs index 455a37ed0..4fe89967d 100644 --- a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs +++ b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs @@ -389,7 +389,7 @@ fn replication_source_object(object_info: &ObjectInfo) -> ReplicationSourceObjec .map(|mod_time| OffsetDateTime::from_unix_timestamp(mod_time.unix_timestamp()).unwrap_or(mod_time)), version_id: object_info.version_id.map(|version_id| version_id.to_string()), etag: object_info.etag.as_deref(), - actual_size: object_info.get_actual_size().unwrap_or_default(), + actual_size: object_info.get_actual_size_or_physical(), delete_marker: object_info.delete_marker, content_type: object_info.content_type.as_deref(), content_encoding: object_info.content_encoding.as_deref(), @@ -542,6 +542,20 @@ mod tests { assert!(replication_target_head_is_newer_null_version(&source, &target)); } + #[test] + fn replication_source_uses_physical_size_for_unknown_compressed_object() { + let mut metadata = HashMap::new(); + rustfs_utils::http::insert_str(&mut metadata, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string()); + let source = ObjectInfo { + size: 128, + actual_size: -1, + user_defined: Arc::new(metadata), + ..Default::default() + }; + + assert_eq!(replication_source_object(&source).actual_size, 128); + } + #[test] fn replication_target_head_content_matches_compare_etag_only() { let source = ObjectInfo { diff --git a/crates/ecstore/src/cluster/rpc/peer_s3_client.rs b/crates/ecstore/src/cluster/rpc/peer_s3_client.rs index d44651dce..72db2062c 100644 --- a/crates/ecstore/src/cluster/rpc/peer_s3_client.rs +++ b/crates/ecstore/src/cluster/rpc/peer_s3_client.rs @@ -454,7 +454,7 @@ impl S3PeerSys { } } topology_complete &= bucket_map.values().all(|count| *count >= quorum); - // TODO: MRF + // TODO(backlog): integrate MRF backlog stats into scanner bucket listing } let mut buckets: Vec = result_map.into_values().collect(); diff --git a/crates/ecstore/src/cluster/rpc/remote_disk.rs b/crates/ecstore/src/cluster/rpc/remote_disk.rs index ed8772f53..a47533a81 100644 --- a/crates/ecstore/src/cluster/rpc/remote_disk.rs +++ b/crates/ecstore/src/cluster/rpc/remote_disk.rs @@ -42,18 +42,19 @@ use futures::lock::Mutex; use metrics::counter; use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo}; use rustfs_io_metrics::internode_metrics::{ - INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE, - INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, + INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE, + INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, + INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE, INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, }; use rustfs_protos::ChannelClass; use rustfs_protos::evict_failed_connection; use rustfs_protos::proto_gen::node_service::RenamePartRequest; use rustfs_protos::proto_gen::node_service::{ BatchReadVersionRequest, BatchReadVersionResponse, CheckPartsRequest, DeletePathsRequest, DeleteRequest, - DeleteVersionRequest, DeleteVersionsRequest, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, - MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, - ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, - RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, + DeleteVersionRequest, DeleteVersionsRequest, DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, + ListVolumesRequest, MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, + ReadMetadataRequest, ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, + RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient, }; @@ -98,6 +99,7 @@ const NS_SCANNER_CAPABILITY_PROBE_TIMEOUT: Duration = Duration::from_secs(5); const REMOTE_DISK_READ_RETRY_BASE_BACKOFF: Duration = Duration::from_millis(50); const ENV_RUSTFS_METADATA_BATCH_READ: &str = "RUSTFS_METADATA_BATCH_READ"; const LEGACY_ENV_RUSTFS_BATCH_METADATA_RPC: &str = "RUSTFS_BATCH_METADATA_RPC"; +const ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE: &str = "RUSTFS_GET_METADATA_READ_VERSION_COALESCE"; const BATCH_METADATA_RPC_OFF: &str = "off"; const BATCH_METADATA_RPC_AUTO: &str = "auto"; const BATCH_METADATA_RPC_ON: &str = "on"; @@ -112,6 +114,28 @@ const EVENT_REMOTE_DISK_RPC: &str = "remote_disk_rpc"; const SNAPSHOT_LEASE_PROTOCOL_VERSION: u32 = 1; pub const REMOTE_SNAPSHOT_LEASE_TTL: Duration = Duration::from_secs(60); +fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len: usize) -> Vec> { + if !response.item_errors.is_empty() { + if response.item_errors.len() != expected_len { + return vec![Some(Error::other("malformed delete_versions item errors")); expected_len]; + } + return response + .item_errors + .into_iter() + .map(|error| (error.code != 0).then(|| error.into())) + .collect(); + } + + if response.errors.len() != expected_len { + return vec![Some(Error::other("malformed delete_versions errors")); expected_len]; + } + response + .errors + .into_iter() + .map(|error| (!error.is_empty()).then(|| Error::other(error))) + .collect() +} + fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result { if !response.success { return Err(response.error.unwrap_or_default().into()); @@ -180,7 +204,8 @@ fn parse_batch_metadata_rpc_mode(raw: &str) -> BatchMetadataRpcMode { } fn batch_metadata_rpc_mode_from_env() -> BatchMetadataRpcMode { - rustfs_utils::get_env_opt_str(ENV_RUSTFS_METADATA_BATCH_READ) + rustfs_utils::get_env_opt_str(ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE) + .or_else(|| rustfs_utils::get_env_opt_str(ENV_RUSTFS_METADATA_BATCH_READ)) .or_else(|| rustfs_utils::get_env_opt_str(LEGACY_ENV_RUSTFS_BATCH_METADATA_RPC)) .as_deref() .map(parse_batch_metadata_rpc_mode) @@ -1804,6 +1829,12 @@ fn record_read_version_stage(stage: &'static str, started_at: Option) { } } +fn record_batch_read_version_stage(stage: &'static str, started_at: Option) { + if let Some(started_at) = started_at { + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_stage(stage, started_at.elapsed()); + } +} + /// Aggregate encoded size (bytes) of a `ReadMultiple` response, preferring the msgpack payloads /// and falling back to the JSON compatibility strings. Used to size the RPC for the payload /// histogram / large-payload alerting (grpc-optimization P0 instrumentation). @@ -1914,6 +1945,27 @@ fn decode_batch_read_version_response_items( Ok(batch_read_version_resps) } +fn batch_read_version_request_payload_len(req: &BatchReadVersionReq, req_json: &str, req_bin: &[u8]) -> usize { + req.items + .iter() + .fold(req_json.len().saturating_add(req_bin.len()), |total, item| { + total + .saturating_add(item.org_volume.len()) + .saturating_add(item.volume.len()) + .saturating_add(item.path.len()) + .saturating_add(item.version_id.len()) + }) +} + +fn batch_read_version_response_payload_len(response: &BatchReadVersionResponse) -> usize { + response + .batch_read_version_resps + .iter() + .map(String::len) + .sum::() + .saturating_add(response.batch_read_version_resps_bin.iter().map(Bytes::len).sum::()) +} + fn validate_decoded_file_info(file_info: &FileInfo) -> Result<()> { file_info.validate_for_metadata_read().map_err(Into::into) } @@ -2406,8 +2458,6 @@ impl DiskAPI for RemoteDisk { return errors; } - // TODO: use Error not string - let result = self .execute_with_timeout( || async { @@ -2439,17 +2489,7 @@ impl DiskAPI for RemoteDisk { } return errors; } - response - .errors - .iter() - .map(|error| { - if error.is_empty() { - None - } else { - Some(Error::other(error.to_string())) - } - }) - .collect() + decode_delete_versions_errors(response, versions.len()) } #[tracing::instrument(level = "trace", skip_all)] @@ -2827,14 +2867,19 @@ impl DiskAPI for RemoteDisk { state = "started", "Remote disk RPC started" ); + let batch_read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); + let encode_started = read_version_stage_timer(batch_read_version_attribution_enabled); let batch_read_version_req = compat_json(&req)?; let batch_read_version_req_bin = encode_msgpack(&req)?; - + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started); + let request_payload_bytes = batch_read_version_attribution_enabled + .then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin)); let batch_result = self .execute_with_timeout_for_op( "batch_read_version", move || async move { let disk = self.disk_ref().await; + let disk_len = disk.len(); let mut client = self .get_bulk_client() .await @@ -2845,9 +2890,20 @@ impl DiskAPI for RemoteDisk { batch_read_version_req_bin: batch_read_version_req_bin.into(), }); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(); + if let Some(request_payload_bytes) = request_payload_bytes { + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes( + request_payload_bytes.saturating_add(disk_len), + ); + } + let rpc_started = read_version_stage_timer(batch_read_version_attribution_enabled); let response = match client.batch_read_version(request).await { - Ok(response) => response.into_inner(), + Ok(response) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started); + response.into_inner() + } Err(status) if status.code() == Code::Unimplemented => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started); if mode.should_fallback_on_unimplemented() { record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_FALLBACK_UNIMPLEMENTED); warn!( @@ -2864,6 +2920,7 @@ impl DiskAPI for RemoteDisk { } record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_UNSUPPORTED_NO_FALLBACK); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error(); warn!( event = EVENT_REMOTE_DISK_RPC, component = LOG_COMPONENT_ECSTORE, @@ -2876,14 +2933,33 @@ impl DiskAPI for RemoteDisk { ); return Err(Error::from(status)); } - Err(status) => return Err(Error::from(status)), + Err(status) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error(); + return Err(Error::from(status)); + } }; if !response.success { + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error(); return Err(response.error.unwrap_or_default().into()); } - decode_batch_read_version_response_items(response, &self.endpoint).map(Some) + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_recv_bytes( + batch_read_version_response_payload_len(&response), + ); + let decode_started = read_version_stage_timer(batch_read_version_attribution_enabled); + match decode_batch_read_version_response_items(response, &self.endpoint) { + Ok(batch_read_version_resps) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE, decode_started); + Ok(Some(batch_read_version_resps)) + } + Err(err) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE, decode_started); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error(); + Err(err) + } + } }, get_max_timeout_duration(), ) @@ -3760,6 +3836,63 @@ mod tests { static INIT: Once = Once::new(); + #[test] + fn delete_versions_response_preserves_typed_item_errors() { + let errors = decode_delete_versions_errors( + DeleteVersionsResponse { + success: true, + errors: vec!["file not found".to_string(), String::new()], + error: None, + item_errors: vec![ + rustfs_protos::proto_gen::node_service::Error { + code: DiskError::FileNotFound.to_u32(), + error_info: "file not found".to_string(), + }, + rustfs_protos::proto_gen::node_service::Error::default(), + ], + }, + 2, + ); + + assert!(matches!(errors.as_slice(), [Some(DiskError::FileNotFound), None])); + } + + #[test] + fn delete_versions_response_accepts_legacy_string_errors() { + let errors = decode_delete_versions_errors( + DeleteVersionsResponse { + success: true, + errors: vec!["legacy error".to_string(), String::new()], + error: None, + item_errors: Vec::new(), + }, + 2, + ); + + assert_eq!(errors.len(), 2); + assert_eq!(errors[0].as_ref().map(ToString::to_string).as_deref(), Some("io error legacy error")); + assert!(errors[1].is_none()); + } + + #[test] + fn delete_versions_response_rejects_misaligned_item_errors() { + let errors = decode_delete_versions_errors( + DeleteVersionsResponse { + success: true, + errors: vec!["file not found".to_string()], + error: None, + item_errors: vec![rustfs_protos::proto_gen::node_service::Error { + code: DiskError::FileNotFound.to_u32(), + error_info: "file not found".to_string(), + }], + }, + 2, + ); + + assert_eq!(errors.len(), 2); + assert!(errors.iter().all(Option::is_some)); + } + #[test] fn disk_mutation_digest_marks_rolling_compatibility() { let mut request = Request::new(()); @@ -4554,6 +4687,7 @@ mod tests { } else { "file version not found".to_string() }, + error_code: if success { 0 } else { DiskError::FileVersionNotFound.to_u32() }, } } @@ -4673,6 +4807,7 @@ mod tests { fn batch_metadata_rpc_mode_uses_documented_env_before_legacy_alias() { temp_env::with_vars( [ + (ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE, None::<&str>), (ENV_RUSTFS_METADATA_BATCH_READ, Some("auto")), (LEGACY_ENV_RUSTFS_BATCH_METADATA_RPC, Some("on")), ], @@ -4682,10 +4817,25 @@ mod tests { ); } + #[test] + fn batch_metadata_rpc_mode_uses_get_coalescer_env_before_batch_env() { + temp_env::with_vars( + [ + (ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE, Some("on")), + (ENV_RUSTFS_METADATA_BATCH_READ, Some("off")), + (LEGACY_ENV_RUSTFS_BATCH_METADATA_RPC, Some("off")), + ], + || { + assert_eq!(batch_metadata_rpc_mode_from_env(), BatchMetadataRpcMode::On); + }, + ); + } + #[test] fn batch_metadata_rpc_mode_falls_back_to_legacy_env_alias() { temp_env::with_vars( [ + (ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE, None::<&str>), (ENV_RUSTFS_METADATA_BATCH_READ, None::<&str>), (LEGACY_ENV_RUSTFS_BATCH_METADATA_RPC, Some("on")), ], diff --git a/crates/ecstore/src/cluster/rpc/runtime_sources.rs b/crates/ecstore/src/cluster/rpc/runtime_sources.rs index 0c8393a2e..ab9a34fe9 100644 --- a/crates/ecstore/src/cluster/rpc/runtime_sources.rs +++ b/crates/ecstore/src/cluster/rpc/runtime_sources.rs @@ -14,9 +14,10 @@ use rustfs_io_metrics::internode_metrics::{ INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE, - INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_READ_VERSION, - INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, - INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics, + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, + INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, + INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, + global_internode_metrics, }; use std::time::Duration; @@ -93,6 +94,59 @@ pub(crate) fn record_remote_disk_grpc_read_version_request() { ); } +pub(crate) fn record_remote_disk_grpc_batch_read_version_request() { + if !rustfs_io_metrics::get_stage_metrics_enabled() { + return; + } + global_internode_metrics().record_outgoing_request_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + ); +} + +pub(crate) fn record_remote_disk_grpc_batch_read_version_stage(stage: &'static str, duration: Duration) { + if !rustfs_io_metrics::get_stage_metrics_enabled() { + return; + } + global_internode_metrics().record_stage_duration_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + stage, + duration, + ); +} + +pub(crate) fn record_remote_disk_grpc_batch_read_version_error() { + if !rustfs_io_metrics::get_stage_metrics_enabled() { + return; + } + global_internode_metrics() + .record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC); +} + +pub(crate) fn record_remote_disk_grpc_batch_read_version_sent_bytes(bytes: usize) { + if !rustfs_io_metrics::get_stage_metrics_enabled() { + return; + } + global_internode_metrics().record_sent_bytes_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + bytes, + ); +} + +pub(crate) fn record_remote_disk_grpc_batch_read_version_recv_bytes(bytes: usize) { + if !rustfs_io_metrics::get_stage_metrics_enabled() { + return; + } + global_internode_metrics().record_recv_bytes_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + bytes, + ); + record_grpc_payload_size(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, bytes); +} + pub(crate) fn record_remote_disk_grpc_read_version_error() { if !rustfs_io_metrics::get_stage_metrics_enabled() { return; diff --git a/crates/ecstore/src/config/storageclass.rs b/crates/ecstore/src/config/storageclass.rs index 043bf508e..361b15702 100644 --- a/crates/ecstore/src/config/storageclass.rs +++ b/crates/ecstore/src/config/storageclass.rs @@ -248,10 +248,13 @@ impl Config { let shard_size = shard_size as usize; // Keep the historical two-data-shard object budget while preventing // wider EC layouts from multiplying the maximum inline object size. + // Use div_ceil to match the shard_file_size calculation (which also uses + // div_ceil), avoiding a 1-byte rounding discrepancy that prevents inline + // for objects right at the threshold. let inline_block = if self.initialized && self.inline_block_explicit { self.inline_block } else { - (DEFAULT_INLINE_OBJECT_BUDGET / data_shards).min(DEFAULT_INLINE_BLOCK) + DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK) }; if versioned { diff --git a/crates/ecstore/src/core/pools.rs b/crates/ecstore/src/core/pools.rs index fb983dc0c..4ca460c20 100644 --- a/crates/ecstore/src/core/pools.rs +++ b/crates/ecstore/src/core/pools.rs @@ -36,7 +36,8 @@ use crate::disk::error::DiskError; use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET}; use crate::error::{Error, Result}; use crate::error::{ - StorageError, is_err_bucket_exists, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, + StorageError, is_err_bucket_exists, is_err_bucket_not_found, is_err_object_not_found, is_err_operation_canceled, + is_err_version_not_found, }; use crate::layout::endpoints::EndpointServerPools; use crate::object_api::{GetObjectReader, ObjectOptions}; @@ -57,7 +58,6 @@ use http::HeaderMap; #[cfg(test)] use rmp_serde::Deserializer; use rmp_serde::Serializer; -use rustfs_common::defer; use rustfs_common::heal_channel::HealOpts; use rustfs_filemeta::{FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams}; use rustfs_utils::path::{encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path}; @@ -65,16 +65,17 @@ use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, Replicatio use serde::{Deserialize, Serialize}; use std::collections::{HashMap, HashSet}; use std::fmt::Display; +use std::future::Future; #[cfg(test)] use std::io::Cursor; use std::io::Write; use std::path::PathBuf; use std::sync::{ Arc, - atomic::{AtomicUsize, Ordering}, + atomic::{AtomicBool, AtomicUsize, Ordering}, }; use time::{Duration, OffsetDateTime}; -use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore, mpsc}; use tokio_util::sync::CancellationToken; use tracing::{debug, error, info, warn}; @@ -89,11 +90,17 @@ const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup"; const DECOMMISSION_STAGE_ENTRY_FINISHED: &str = "entry_finished"; const DECOMMISSION_PROGRESS_SAVE_INTERVAL: Duration = Duration::seconds(30); const DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD: usize = 1000; +const DECOMMISSION_PROGRESS_SAVE_RETRY_BACKOFF: Duration = Duration::seconds(1); const DECOMMISSION_BUCKET_CONCURRENCY_ENV: &str = "RUSTFS_DECOMMISSION_BUCKET_CONCURRENCY"; const DECOMMISSION_BUCKET_CONCURRENCY_DEFAULT_CAP: usize = 4; +const DECOMMISSION_ENTRY_CONCURRENCY_ENV: &str = "RUSTFS_DECOMMISSION_ENTRY_CONCURRENCY"; +const DECOMMISSION_ENTRY_CONCURRENCY_DEFAULT_CAP: usize = 8; +const DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP: usize = 64; +const DECOMMISSION_ENTRY_WORKERS_PER_SET: usize = 2; const DECOMMISSION_TARGET_CAPACITY_OVERHEAD_PERCENT: usize = 30; const DECOMMISSION_LISTING_MAX_ATTEMPTS: usize = 3; const DECOMMISSION_LISTING_RETRY_DELAY: std::time::Duration = std::time::Duration::from_secs(5); +const DECOMMISSION_TERMINAL_RETRY_DELAY: std::time::Duration = std::time::Duration::from_secs(1); /// Background decommission walks must tolerate slow object migrations; the /// stall timeout is the drive-health bound, not the total listing duration. const DECOMMISSION_BACKGROUND_WALKDIR_STALL_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60); @@ -102,6 +109,74 @@ pub const POOL_META_NAME: &str = "pool.bin"; pub const POOL_META_FORMAT: u16 = 1; pub const POOL_META_VERSION: u16 = 1; +#[derive(Clone, Debug)] +pub struct DecommissionCanceler { + operation: Arc, +} + +#[derive(Debug)] +struct DecommissionOperation { + token: CancellationToken, + active: AtomicBool, +} + +impl DecommissionCanceler { + fn new(token: CancellationToken) -> Self { + Self { + operation: Arc::new(DecommissionOperation { + token, + active: AtomicBool::new(true), + }), + } + } + + fn token(&self) -> &CancellationToken { + &self.operation.token + } + + fn is_active(&self) -> bool { + self.operation.active.load(Ordering::Acquire) + } + + #[cfg(test)] + fn is_cancelled(&self) -> bool { + self.token().is_cancelled() + } + + fn cancel(&self) { + self.token().cancel(); + } + + fn release(&self) { + self.cancel(); + self.operation.active.store(false, Ordering::Release); + } + + fn owns_same_operation(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.operation, &other.operation) + } +} + +struct DecommissionCancelerGuard { + canceler: DecommissionCanceler, +} + +impl DecommissionCancelerGuard { + fn new(canceler: DecommissionCanceler) -> Self { + Self { canceler } + } + + fn canceler(&self) -> &DecommissionCanceler { + &self.canceler + } +} + +impl Drop for DecommissionCancelerGuard { + fn drop(&mut self) { + self.canceler.release(); + } +} + fn dedup_indices(indices: &[usize]) -> Vec { let mut seen = HashSet::with_capacity(indices.len()); let mut output = Vec::with_capacity(indices.len()); @@ -117,18 +192,18 @@ fn dedup_indices(indices: &[usize]) -> Vec { fn bind_decommission_cancelers( indices: &[usize], parent: &CancellationToken, - cancelers: &mut [Option], -) -> Vec<(usize, CancellationToken)> { + cancelers: &mut [Option], +) -> Vec<(usize, DecommissionCanceler)> { let mut bound = Vec::with_capacity(indices.len()); for idx in indices { if let Some(slot) = cancelers.get_mut(*idx) { if let Some(existing) = slot.take() { - existing.cancel(); + existing.release(); } - let token = parent.child_token(); - *slot = Some(token.clone()); - bound.push((*idx, token)); + let canceler = DecommissionCanceler::new(parent.child_token()); + *slot = Some(canceler.clone()); + bound.push((*idx, canceler)); } } @@ -138,47 +213,104 @@ fn bind_decommission_cancelers( fn bind_missing_decommission_cancelers( indices: &[usize], parent: &CancellationToken, - cancelers: &mut [Option], -) -> Vec<(usize, CancellationToken)> { + cancelers: &mut [Option], +) -> Vec<(usize, DecommissionCanceler)> { let mut bound = Vec::with_capacity(indices.len()); for idx in indices { let Some(slot) = cancelers.get_mut(*idx) else { continue; }; - if slot.is_some() { + if slot.as_ref().is_some_and(DecommissionCanceler::is_active) { break; } - let token = parent.child_token(); - *slot = Some(token.clone()); - bound.push((*idx, token)); + if let Some(stale) = slot.take() { + stale.release(); + } + let canceler = DecommissionCanceler::new(parent.child_token()); + *slot = Some(canceler.clone()); + bound.push((*idx, canceler)); } bound } -fn take_decommission_canceler(cancelers: &mut [Option], idx: usize) -> Option { +fn take_decommission_canceler(cancelers: &mut [Option], idx: usize) -> Option { cancelers.get_mut(idx).and_then(Option::take) } -fn has_active_decommission_canceler(cancelers: &[Option]) -> bool { - cancelers.iter().any(Option::is_some) +fn take_decommission_canceler_for_operation( + cancelers: &mut [Option], + idx: usize, + owner: &DecommissionCanceler, +) -> Option { + let slot = cancelers.get_mut(idx)?; + if slot.as_ref().is_some_and(|canceler| canceler.owns_same_operation(owner)) { + slot.take() + } else { + None + } } -fn cancel_decommission_canceler(canceler: Option) -> bool { +fn decommission_canceler_is_owned_by( + cancelers: &[Option], + idx: usize, + owner: &DecommissionCanceler, +) -> bool { + cancelers + .get(idx) + .and_then(Option::as_ref) + .is_some_and(|canceler| canceler.owns_same_operation(owner)) +} + +fn update_decommission_for_operation( + cancelers: &[Option], + pool_meta: &mut PoolMeta, + idx: usize, + owner: Option<&DecommissionCanceler>, + update: impl FnOnce(&mut PoolMeta) -> T, +) -> Option { + if let Some(owner) = owner + && !decommission_canceler_is_owned_by(cancelers, idx, owner) + { + owner.release(); + return None; + } + + Some(update(pool_meta)) +} + +fn has_active_decommission_canceler(cancelers: &[Option]) -> bool { + cancelers.iter().flatten().any(DecommissionCanceler::is_active) +} + +fn cancel_decommission_canceler(canceler: Option) -> bool { if let Some(canceler) = canceler { - canceler.cancel(); + canceler.release(); true } else { false } } -fn take_and_cancel_decommission_canceler(cancelers: &mut [Option], idx: usize) -> bool { +fn take_and_cancel_decommission_canceler(cancelers: &mut [Option], idx: usize) -> bool { let canceler = take_decommission_canceler(cancelers, idx); cancel_decommission_canceler(canceler) } +fn take_and_cancel_decommission_canceler_for_operation( + cancelers: &mut [Option], + idx: usize, + owner: &DecommissionCanceler, +) -> bool { + let canceler = take_decommission_canceler_for_operation(cancelers, idx, owner); + if canceler.is_none() { + owner.release(); + return false; + } + cancel_decommission_canceler(canceler) +} + fn ensure_decommission_routines_scheduled(bound_count: usize, expected_count: usize) -> Result<()> { if bound_count == 0 || bound_count != expected_count { return Err(Error::other(format!( @@ -189,6 +321,36 @@ fn ensure_decommission_routines_scheduled(bound_count: usize, expected_count: us Ok(()) } +fn guard_decommission_cancelers(index_cancelers: Vec<(usize, DecommissionCanceler)>) -> Vec<(usize, DecommissionCancelerGuard)> { + index_cancelers + .into_iter() + .map(|(idx, canceler)| (idx, DecommissionCancelerGuard::new(canceler))) + .collect() +} + +async fn await_decommission_worker(idx: usize, worker: tokio::task::JoinHandle>) -> Result<()> { + worker + .await + .map_err(|err| Error::other(format!("decommission worker {idx} task join error: {err}")))? +} + +fn reserve_decommission_start_cancelers( + pool_meta: &PoolMeta, + indices: &[usize], + local_indices: &[usize], + parent: &CancellationToken, + cancelers: &mut [Option], +) -> Result> { + ensure_decommission_start_pool_states(pool_meta, indices)?; + if local_indices.is_empty() { + return Ok(Vec::new()); + } + let bound = bind_decommission_cancelers(local_indices, parent, cancelers); + let guards = guard_decommission_cancelers(bound); + ensure_decommission_routines_scheduled(guards.len(), local_indices.len())?; + Ok(guards) +} + fn default_decommission_bucket_concurrency(cpu_count: usize) -> usize { cpu_count.clamp(1, DECOMMISSION_BUCKET_CONCURRENCY_DEFAULT_CAP) } @@ -198,6 +360,19 @@ fn decommission_bucket_concurrency_limit() -> usize { rustfs_utils::get_env_usize(DECOMMISSION_BUCKET_CONCURRENCY_ENV, default_limit).max(1) } +fn default_decommission_entry_concurrency(cpu_count: usize) -> usize { + cpu_count.clamp(1, DECOMMISSION_ENTRY_CONCURRENCY_DEFAULT_CAP) +} + +fn clamp_decommission_entry_concurrency(limit: usize) -> usize { + limit.clamp(1, DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP) +} + +fn decommission_entry_concurrency_limit() -> usize { + let default_limit = default_decommission_entry_concurrency(num_cpus::get()); + clamp_decommission_entry_concurrency(rustfs_utils::get_env_usize(DECOMMISSION_ENTRY_CONCURRENCY_ENV, default_limit)) +} + fn is_decommission_meta_bucket(bucket: &DecomBucketInfo) -> bool { bucket.name == RUSTFS_META_BUCKET } @@ -307,11 +482,15 @@ fn first_resumable_decommission_queue_indices(meta: &PoolMeta) -> Vec { indices } -fn missing_decommission_worker_prefix(indices: &[usize], cancelers: &[Option]) -> Vec { +fn missing_decommission_worker_prefix(indices: &[usize], cancelers: &[Option]) -> Vec { let mut missing = Vec::with_capacity(indices.len()); for idx in indices { - if cancelers.get(*idx).and_then(Option::as_ref).is_some() { + if cancelers + .get(*idx) + .and_then(Option::as_ref) + .is_some_and(DecommissionCanceler::is_active) + { break; } missing.push(*idx); @@ -358,29 +537,27 @@ fn build_decommission_start_state( fn spawn_decommission_index_cancelers( store: Arc, rx: CancellationToken, - index_cancelers: Vec<(usize, CancellationToken)>, -) { + index_cancelers: Vec<(usize, DecommissionCancelerGuard)>, + entry_budget: Arc, +) -> tokio::task::JoinHandle<()> { tokio::spawn(async move { let mut stop_queue = false; - for (idx, canceler) in index_cancelers { + for (idx, canceler_guard) in index_cancelers { + let canceler = canceler_guard.canceler().clone(); if stop_queue || rx.is_cancelled() { canceler.cancel(); - if let Err(err) = store.decommission_cancel(idx).await { - warn!( - event = EVENT_DECOMMISSION_STATE, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - state = "queued_cancel_failed", - error = %err, - "Failed to cancel queued decommission" - ); - } + store.retry_decommission_cancel_for_operation(idx, &canceler).await; continue; } - if let Err(err) = store.do_decommission_in_routine(canceler, idx).await { + let worker = tokio::spawn({ + let store = store.clone(); + let canceler = canceler.clone(); + let entry_budget = entry_budget.clone(); + async move { store.do_decommission_in_routine(canceler, idx, entry_budget).await } + }); + if let Err(err) = await_decommission_worker(idx, worker).await { error!( event = EVENT_DECOMMISSION_STATE, component = LOG_COMPONENT_ECSTORE, @@ -390,6 +567,7 @@ fn spawn_decommission_index_cancelers( error = %err, "Decommission routine failed" ); + store.retry_decommission_failed_for_operation(idx, &canceler).await; stop_queue = true; continue; } @@ -399,7 +577,7 @@ fn spawn_decommission_index_cancelers( !should_continue_decommission_queue(&pool_meta, idx) }; } - }); + }) } fn decommission_meta_bucket_options() -> MakeBucketOptions { @@ -611,6 +789,47 @@ fn count_decommission_item(meta: &mut PoolMeta, idx: usize, size: usize, failed: Ok(()) } +fn ensure_decommission_generation(meta: &PoolMeta, idx: usize, generation: OffsetDateTime) -> Result<()> { + let Some(pool) = meta.pools.get(idx) else { + return Err(invalid_decommission_pool_index_error(meta.pools.len(), idx)); + }; + let Some(info) = pool.decommission.as_ref() else { + return Err(decommission_metadata_not_initialized_error("check decommission generation")); + }; + + if info.start_time == Some(generation) && !info.queued && is_decommission_active(info.complete, info.failed, info.canceled) { + Ok(()) + } else { + Err(Error::OperationCanceled) + } +} + +async fn run_decommission_side_effect( + rx: &CancellationToken, + operation_gate: &Arc>, + operation: F, +) -> Result +where + F: FnOnce() -> Fut, + Fut: std::future::Future>, +{ + let _operation_guard = tokio::select! { + biased; + _ = rx.cancelled() => return Err(Error::OperationCanceled), + guard = operation_gate.read() => guard, + }; + + if rx.is_cancelled() { + return Err(Error::OperationCanceled); + } + + let result = operation().await; + if rx.is_cancelled() { + return Err(Error::OperationCanceled); + } + result +} + fn track_decommission_current_object_stage( meta: &mut PoolMeta, idx: usize, @@ -638,22 +857,6 @@ fn track_decommission_current_object(meta: &mut PoolMeta, idx: usize, bucket: &s track_decommission_current_object_stage(meta, idx, bucket, object, "") } -fn touch_decommission_progress(meta: &mut PoolMeta, idx: usize) -> Result<()> { - let pool_count = meta.pools.len(); - ensure_valid_decommission_pool_index(pool_count, idx)?; - - let Some(pool) = meta.pools.get_mut(idx) else { - return Err(invalid_decommission_pool_index_error(pool_count, idx)); - }; - let Some(info) = pool.decommission.as_mut() else { - return Err(decommission_metadata_not_initialized_error("touch decommission progress")); - }; - - pool.last_update = OffsetDateTime::now_utc(); - info.mark_progress_saved(); - Ok(()) -} - fn resolve_decommission_update_after_result(result: Result) -> Result { result.map_err(|err| Error::other(format!("decommission metadata update failed: {err}"))) } @@ -714,16 +917,6 @@ fn observe_decommission_terminal_reload_result(result: Result<()>, stage: &str) .map(|err| Error::other(format!("decommission terminal pool meta reload failed during {stage}: {err}"))) } -fn resolve_decommission_spawn_failure_result(spawn_err: Error, rollback_err: Option) -> Error { - if let Some(rollback_err) = rollback_err { - Error::other(format!( - "decommission spawn routines failed: {spawn_err}; rollback failed: {rollback_err}" - )) - } else { - spawn_err - } -} - fn decommission_item_size(size: T) -> usize where usize: TryFrom, @@ -773,7 +966,60 @@ async fn load_decommission_entry_exact_versions( } fn resolve_decommission_check_after_list_result(list_result: Result<()>, entry_error: Option) -> Result<()> { - if let Some(err) = entry_error { Err(err) } else { list_result } + match list_result { + Ok(()) => entry_error.map_or(Ok(()), Err), + Err(list_err) => resolve_decommission_listing_error(Some(list_err), entry_error).map_or(Ok(()), Err), + } +} + +fn resolve_decommission_listing_error(listing_error: Option, entry_error: Option) -> Option { + match (listing_error, entry_error) { + (Some(listing_error), Some(entry_error)) if is_err_operation_canceled(&listing_error) => Some(entry_error), + (Some(listing_error), Some(entry_error)) if is_err_operation_canceled(&entry_error) => Some(listing_error), + (Some(listing_error), _) => Some(listing_error), + (None, entry_error) => entry_error, + } +} + +fn decommission_unresolved_listing_error( + bucket: &str, + prefix: &str, + candidate: Option<&str>, + candidate_count: usize, + disk_error_count: usize, + pool_index: usize, + set_index: usize, +) -> Error { + let location = candidate.unwrap_or(prefix); + Error::other(format!( + "decommission listing could not resolve metadata for {bucket}/{location} on pool {pool_index} set {set_index} ({candidate_count} candidate(s), {disk_error_count} disk error(s))" + )) +} + +fn resolve_decommission_partial_listing_entry( + entries: MetaCacheEntries, + resolver: MetadataResolutionParams, + bucket: &str, + prefix: &str, + disk_error_count: usize, + pool_index: usize, + set_index: usize, +) -> Result { + let candidate_count = entries.as_ref().iter().flatten().count(); + if let Some(entry) = entries.resolve(resolver) { + return Ok(entry); + } + + let candidate = entries.as_ref().iter().flatten().map(|entry| entry.name.as_str()).next(); + Err(decommission_unresolved_listing_error( + bucket, + prefix, + candidate, + candidate_count, + disk_error_count, + pool_index, + set_index, + )) } fn resolve_decommission_pool_meta_reload_result(result: Result<()>, stage: &str) -> Result<()> { @@ -908,6 +1154,7 @@ async fn wait_decommission_listing_retry(rx: &CancellationToken, delay: std::tim } } +#[cfg(test)] async fn run_decommission_listing_with_retry( rx: CancellationToken, bucket: String, @@ -915,11 +1162,31 @@ async fn run_decommission_listing_with_retry( pool_idx: usize, set_idx: usize, max_attempts: usize, - mut list: List, + list: List, ) -> Result<()> where List: FnMut(ListCallback) -> ListFuture, ListFuture: std::future::Future>, +{ + run_decommission_listing_with_retry_and_drain(rx, bucket, cb, pool_idx, set_idx, max_attempts, list, || async { false }).await +} + +#[allow(clippy::too_many_arguments)] +async fn run_decommission_listing_with_retry_and_drain( + rx: CancellationToken, + bucket: String, + cb: ListCallback, + pool_idx: usize, + set_idx: usize, + max_attempts: usize, + mut list: List, + mut drain: Drain, +) -> Result<()> +where + List: FnMut(ListCallback) -> ListFuture, + ListFuture: std::future::Future>, + Drain: FnMut() -> DrainFuture, + DrainFuture: std::future::Future, { let max_attempts = max_attempts.max(1); @@ -951,7 +1218,12 @@ where "Decommission listing started" ); - match list(cb.clone()).await { + let list_result = list(cb.clone()).await; + if drain().await { + return Ok(()); + } + + match list_result { Ok(()) => { debug!( event = EVENT_DECOMMISSION_BUCKET, @@ -1183,6 +1455,7 @@ where Ok(()) } +#[cfg(test)] async fn wait_decommission_worker_drain(workers: &Semaphore, limit: usize) -> Result<()> { let permits = u32::try_from(limit) .map_err(|_| Error::other(format!("decommission worker limit {limit} exceeds semaphore drain capacity")))?; @@ -1483,6 +1756,7 @@ impl TryFrom for PoolDecommissionInfo { terminal_reload_attempt_at: value.terminal_reload_attempt_at, terminal_reload_failures: value.terminal_reload_failures, progress_save_item_baseline: value.items_decommissioned.saturating_add(value.items_decommission_failed), + progress_save_retry_after: None, }) } } @@ -1514,6 +1788,7 @@ impl TryFrom for PoolDecommissionInfo { terminal_reload_attempt_at: None, terminal_reload_failures: Vec::new(), progress_save_item_baseline: value.items_decommissioned.saturating_add(value.items_decommission_failed), + progress_save_retry_after: None, }) } } @@ -1627,6 +1902,82 @@ impl PoolMeta { } } + fn decommission_progress_checkpoint( + &self, + idx: usize, + duration: Duration, + now: OffsetDateTime, + ) -> Result> { + let pool_count = self.pools.len(); + ensure_valid_decommission_pool_index(pool_count, idx)?; + + let Some(pool) = self.pools.get(idx) else { + return Err(invalid_decommission_pool_index_error(pool_count, idx)); + }; + let Some(info) = pool.decommission.as_ref() else { + return Err(decommission_metadata_not_initialized_error("update decommission metadata timestamp")); + }; + + if info.progress_save_retry_after.is_some_and(|retry_after| now < retry_after) { + return Ok(None); + } + + let time_threshold_reached = now.unix_timestamp() - pool.last_update.unix_timestamp() >= duration.whole_seconds(); + let item_threshold_reached = info.items_since_last_progress_save() >= DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD; + if !time_threshold_reached && !item_threshold_reached { + return Ok(None); + } + + Ok(Some(DecommissionProgressCheckpoint { + start_time: info.start_time, + queued: info.queued, + counted_items: info.counted_items(), + checkpoint_at: now, + })) + } + + fn commit_decommission_progress_checkpoint(&mut self, idx: usize, checkpoint: DecommissionProgressCheckpoint) -> bool { + let Some(pool) = self.pools.get_mut(idx) else { + return false; + }; + let Some(info) = pool.decommission.as_mut() else { + return false; + }; + + if info.start_time != checkpoint.start_time + || info.queued != checkpoint.queued + || !is_decommission_active(info.complete, info.failed, info.canceled) + { + return false; + } + + info.progress_save_item_baseline = info.progress_save_item_baseline.max(checkpoint.counted_items); + info.progress_save_retry_after = None; + pool.last_update = pool.last_update.max(checkpoint.checkpoint_at); + true + } + + fn defer_decommission_progress_checkpoint( + &mut self, + idx: usize, + checkpoint: DecommissionProgressCheckpoint, + retry_after: OffsetDateTime, + ) { + let Some(pool) = self.pools.get_mut(idx) else { + return; + }; + let Some(info) = pool.decommission.as_mut() else { + return; + }; + + if info.start_time == checkpoint.start_time + && info.queued == checkpoint.queued + && is_decommission_active(info.complete, info.failed, info.canceled) + { + info.progress_save_retry_after = Some(retry_after); + } + } + fn load_from_config_data(&mut self, data: Vec) -> Result<()> { if data.is_empty() { return Ok(()); @@ -1675,7 +2026,7 @@ impl PoolMeta { self.load_no_lock(pool).await } - async fn load_no_lock(&mut self, pool: Arc) -> Result<()> + pub(crate) async fn load_no_lock(&mut self, pool: Arc) -> Result<()> where S: EcstoreObjectIO, { @@ -1778,7 +2129,7 @@ impl PoolMeta { pub fn decommission_failed(&mut self, idx: usize) -> bool { if let Some(stats) = self.pools.get_mut(idx) { if let Some(d) = &stats.decommission { - if !d.failed { + if is_decommission_active(d.complete, d.failed, d.canceled) { stats.last_update = OffsetDateTime::now_utc(); let mut pd = d.clone(); @@ -1826,7 +2177,7 @@ impl PoolMeta { pub fn decommission_complete(&mut self, idx: usize) -> bool { if let Some(stats) = self.pools.get_mut(idx) { if let Some(d) = &stats.decommission { - if !d.complete { + if is_decommission_active(d.complete, d.failed, d.canceled) { stats.last_update = OffsetDateTime::now_utc(); let mut pd = d.clone(); @@ -1987,30 +2338,9 @@ impl PoolMeta { } pub fn update_after(&mut self, idx: usize, duration: Duration) -> Result { - let pool_count = self.pools.len(); - ensure_valid_decommission_pool_index(pool_count, idx)?; - - let (last_update, item_threshold_reached) = match self.pools.get(idx) { - Some(pool) if let Some(info) = pool.decommission.as_ref() => ( - pool.last_update, - info.items_since_last_progress_save() >= DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, - ), - Some(_) => { - return Err(decommission_metadata_not_initialized_error("update decommission metadata timestamp")); - } - None => return Err(invalid_decommission_pool_index_error(pool_count, idx)), - }; - let now = OffsetDateTime::now_utc(); - - if now.unix_timestamp() - last_update.unix_timestamp() >= duration.whole_seconds() || item_threshold_reached { - let Some(pool) = self.pools.get_mut(idx) else { - return Err(invalid_decommission_pool_index_error(pool_count, idx)); - }; - pool.last_update = now; - return Ok(true); - } - - Ok(false) + Ok(self + .decommission_progress_checkpoint(idx, duration, OffsetDateTime::now_utc())? + .is_some()) } pub fn validate(&self, pools: Vec>) -> Result { @@ -2151,6 +2481,16 @@ pub struct PoolDecommissionInfo { pub terminal_reload_failures: Vec, #[serde(skip)] pub progress_save_item_baseline: usize, + #[serde(skip)] + pub progress_save_retry_after: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct DecommissionProgressCheckpoint { + start_time: Option, + queued: bool, + counted_items: usize, + checkpoint_at: OffsetDateTime, } impl PoolDecommissionInfo { @@ -2185,6 +2525,7 @@ impl PoolDecommissionInfo { fn mark_progress_saved(&mut self) { self.progress_save_item_baseline = self.counted_items(); + self.progress_save_retry_after = None; } pub fn bucket_push(&mut self, bucket: &DecomBucketInfo) { @@ -2489,6 +2830,41 @@ impl ECStore { snapshot.save(self.pools.clone()).await } + async fn save_decommission_progress_checkpoint(&self, idx: usize, generation: OffsetDateTime) -> Result { + // Lock order: save gate, then the short pool metadata read/write sections. Peer + // reloads are intentionally performed by the caller after both locks are released. + let _save_guard = self.pool_meta_save_gate.lock().await; + let (snapshot, checkpoint) = { + let pool_meta = self.pool_meta.read().await; + ensure_decommission_generation(&pool_meta, idx, generation)?; + let Some(checkpoint) = pool_meta.decommission_progress_checkpoint( + idx, + DECOMMISSION_PROGRESS_SAVE_INTERVAL, + OffsetDateTime::now_utc(), + )? + else { + return Ok(false); + }; + + let mut snapshot = pool_meta.clone(); + let Some(pool) = snapshot.pools.get_mut(idx) else { + return Err(invalid_decommission_pool_index_error(snapshot.pools.len(), idx)); + }; + pool.last_update = checkpoint.checkpoint_at; + (snapshot, checkpoint) + }; + + if let Err(err) = snapshot.save(self.pools.clone()).await { + let retry_after = OffsetDateTime::now_utc() + DECOMMISSION_PROGRESS_SAVE_RETRY_BACKOFF; + let mut pool_meta = self.pool_meta.write().await; + pool_meta.defer_decommission_progress_checkpoint(idx, checkpoint, retry_after); + return Err(err); + } + + let mut pool_meta = self.pool_meta.write().await; + Ok(pool_meta.commit_decommission_progress_checkpoint(idx, checkpoint)) + } + async fn save_current_pool_meta_for_decommission_start( &self, indices: &[usize], @@ -2590,7 +2966,10 @@ impl ECStore { let active_workers = { let cancelers = self.decommission_cancelers.read().await; - cancelers.iter().map(Option::is_some).collect::>() + cancelers + .iter() + .map(|canceler| canceler.as_ref().is_some_and(DecommissionCanceler::is_active)) + .collect::>() }; let mut pool_meta = self.pool_meta.write().await; @@ -2626,9 +3005,98 @@ impl ECStore { #[tracing::instrument(skip(self))] pub async fn decommission_cancel(&self, idx: usize) -> Result<()> { - ensure_decommission_terminal_operation_supported(self.single_pool(), "cancel decommission")?; + self.decommission_cancel_with_owner(idx, None).await + } - let (should_save_pool_meta, should_reload_pool_meta, already_canceled, previous_pool_meta) = { + async fn decommission_cancel_for_operation(&self, idx: usize, owner: &DecommissionCanceler) -> Result<()> { + self.decommission_cancel_with_owner(idx, Some(owner)).await + } + + async fn release_decommission_canceler_slot(&self, idx: usize, owner: &DecommissionCanceler) { + let mut cancelers = self.decommission_cancelers.write().await; + take_and_cancel_decommission_canceler_for_operation(cancelers.as_mut_slice(), idx, owner); + } + + async fn decommission_terminal_retryable_for_operation(&self, idx: usize, owner: &DecommissionCanceler) -> bool { + let _start_guard = self.start_gate.lock().await; + let mut cancelers = self.decommission_cancelers.write().await; + if !decommission_canceler_is_owned_by(cancelers.as_slice(), idx, owner) { + owner.release(); + return false; + } + + let retryable = { + let pool_meta = self.pool_meta.read().await; + pool_meta + .pools + .get(idx) + .and_then(|pool| pool.decommission.as_ref()) + .is_some_and(|info| info.has_decommission_state() && !info.complete && !info.failed && !info.canceled) + }; + if !retryable { + take_and_cancel_decommission_canceler_for_operation(cancelers.as_mut_slice(), idx, owner); + } + retryable + } + + async fn retry_decommission_cancel_for_operation(&self, idx: usize, owner: &DecommissionCanceler) { + let mut attempt = 0usize; + loop { + let Err(err) = self.decommission_cancel_for_operation(idx, owner).await else { + return; + }; + if !self.decommission_terminal_retryable_for_operation(idx, owner).await { + return; + } + attempt += 1; + warn!( + event = EVENT_DECOMMISSION_STATE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + state = "terminal_save_retry", + terminal = "canceled", + attempt, + error = %err, + "Decommission terminal save will be retried" + ); + tokio::time::sleep(DECOMMISSION_TERMINAL_RETRY_DELAY).await; + } + } + + async fn retry_decommission_failed_for_operation(&self, idx: usize, owner: &DecommissionCanceler) { + let mut attempt = 0usize; + loop { + let Err(err) = self.decommission_failed_for_operation(idx, owner).await else { + return; + }; + if !self.decommission_terminal_retryable_for_operation(idx, owner).await { + return; + } + attempt += 1; + warn!( + event = EVENT_DECOMMISSION_STATE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + state = "terminal_save_retry", + terminal = "failed", + attempt, + error = %err, + "Decommission terminal save will be retried" + ); + tokio::time::sleep(DECOMMISSION_TERMINAL_RETRY_DELAY).await; + } + } + + async fn decommission_cancel_with_owner(&self, idx: usize, owner: Option<&DecommissionCanceler>) -> Result<()> { + ensure_decommission_terminal_operation_supported(self.single_pool(), "cancel decommission")?; + let _start_guard = self.start_gate.lock().await; + + // Lock order: decommission_cancelers before pool_meta. Holding both makes + // owner validation and the terminal transition one atomic operation. + let (should_save_pool_meta, should_reload_pool_meta, already_canceled, previous_pool_meta, terminal_canceler) = { + let cancelers = self.decommission_cancelers.read().await; let mut lock = self.pool_meta.write().await; let mut already_canceled = false; let (pool_present, decommission_present, terminal) = if let Some(pool) = lock.pools.get(idx) { @@ -2648,19 +3116,28 @@ impl ECStore { ensure_decommission_cancel_allowed(pool_present, decommission_present, terminal)?; let previous_pool_meta = lock.clone(); - let changed = lock.decommission_cancel(idx); + let Some(changed) = update_decommission_for_operation(cancelers.as_slice(), &mut lock, idx, owner, |pool_meta| { + pool_meta.decommission_cancel(idx) + }) else { + return Ok(()); + }; + let terminal_canceler = if let Some(owner) = owner { + Some(owner.clone()) + } else { + cancelers.get(idx).and_then(Option::as_ref).cloned() + }; + if let Some(canceler) = terminal_canceler.as_ref() { + canceler.cancel(); + } ( changed, should_retry_decommission_cancel_reload(changed, already_canceled), already_canceled, changed.then_some(previous_pool_meta), + terminal_canceler, ) }; - - let canceled_worker = { - let mut cancelers = self.decommission_cancelers.write().await; - take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), idx) - }; + let canceled_worker = terminal_canceler.as_ref().is_some_and(DecommissionCanceler::is_active); if !canceled_worker && !already_canceled { warn!( event = EVENT_DECOMMISSION_STATE, @@ -2673,6 +3150,8 @@ impl ECStore { ); } + self.wait_for_decommission_side_effects().await; + if should_save_pool_meta && let Err(err) = self.save_current_pool_meta().await { if let Some(previous_pool_meta) = previous_pool_meta { let mut pool_meta = self.pool_meta.write().await; @@ -2681,6 +3160,10 @@ impl ECStore { return Err(err); } + if let Some(canceler) = terminal_canceler.as_ref() { + self.release_decommission_canceler_slot(idx, canceler).await; + } + if should_reload_pool_meta && let Some(notification_sys) = runtime_sources::notification_sys() { let stage = format!("decommission_cancel for pool {idx}"); resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str())?; @@ -2692,6 +3175,23 @@ impl ECStore { #[tracing::instrument(skip(self))] pub async fn clear_decommission(&self, idx: usize) -> Result<()> { ensure_decommission_terminal_operation_supported(self.single_pool(), "clear decommission")?; + let _start_guard = self.start_gate.lock().await; + + { + let pool_meta = self.pool_meta.read().await; + let pool_count = pool_meta.pools.len(); + ensure_valid_decommission_pool_index(pool_count, idx)?; + let Some(pool) = pool_meta.pools.get(idx) else { + return Err(invalid_decommission_pool_index_error(pool_count, idx)); + }; + let (decommission_present, complete, failed, canceled) = pool + .decommission + .as_ref() + .map(|info| (info.has_decommission_state(), info.complete, info.failed, info.canceled)) + .unwrap_or((false, false, false, false)); + ensure_decommission_clear_allowed(true, decommission_present, complete, failed, canceled)?; + } + self.cancel_decommission_routines_and_wait(&[idx]).await; let (should_reload_pool_meta, previous_pool_meta) = { let mut pool_meta = self.pool_meta.write().await; @@ -2700,11 +3200,6 @@ impl ECStore { (changed, changed.then_some(previous_pool_meta)) }; - { - let mut cancelers = self.decommission_cancelers.write().await; - take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), idx); - } - if should_reload_pool_meta && let Err(err) = self.save_current_pool_meta().await { if let Some(previous_pool_meta) = previous_pool_meta { let mut pool_meta = self.pool_meta.write().await; @@ -2721,21 +3216,51 @@ impl ECStore { Ok(()) } - async fn promote_queued_decommission(&self, idx: usize) -> Result<()> { - let promoted = { + async fn promote_queued_decommission(&self, idx: usize, owner: &DecommissionCanceler) -> Result { + // Serialize promotion and generation capture with clear/restart transitions. + let (promoted, generation, save_error) = { + let _start_guard = self.start_gate.lock().await; let mut pool_meta = self.pool_meta.write().await; - pool_meta.promote_queued_decommission(idx) + if pool_meta.pools.get(idx).is_none() { + return Err(Error::other("failed to start decommission: target pool was not found")); + } + let promoted = pool_meta.promote_queued_decommission(idx); + drop(pool_meta); + + let save_error = if promoted { + self.save_current_pool_meta().await.err() + } else { + None + }; + + let generation = self.active_decommission_generation(idx).await?; + (promoted, generation, save_error) }; - if promoted { - self.save_current_pool_meta().await?; - if let Some(notification_sys) = runtime_sources::notification_sys() { - let stage = format!("promote_queued_decommission for pool {idx}"); - resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str())?; + if let Some(err) = save_error { + resolve_decommission_terminal_mark_after_error_result( + self.decommission_failed_for_operation(idx, owner).await, + idx, + &err, + )?; + return Err(err); + } + + if promoted && let Some(notification_sys) = runtime_sources::notification_sys() { + let stage = format!("promote_queued_decommission for pool {idx}"); + if let Err(err) = + resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str()) + { + resolve_decommission_terminal_mark_after_error_result( + self.decommission_failed_for_operation(idx, owner).await, + idx, + &err, + )?; + return Err(err); } } - Ok(()) + Ok(generation) } async fn record_decommission_terminal_reload_failure(&self, idx: usize, stage: &str, err: Error) -> Result<()> { @@ -2779,26 +3304,73 @@ impl ECStore { is_decommission_cancel_requested(rx.is_cancelled(), pool_meta.pools.get(idx)) } + async fn cancel_decommission_routines_and_wait(&self, indices: &[usize]) { + { + let mut cancelers = self.decommission_cancelers.write().await; + for idx in indices { + take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), *idx); + } + } + self.wait_for_decommission_side_effects().await; + } + + async fn wait_for_decommission_side_effects(&self) { + let operation_gate = self.ctx.decommission_operation_gate(); + let _operation_guard = operation_gate.write().await; + } + + async fn reserve_decommission_routines( + &self, + rx: &CancellationToken, + indices: &[usize], + ) -> Result> { + let indices = dedup_indices(indices); + if indices.is_empty() { + return Ok(Vec::new()); + } + + let _start_guard = self.start_gate.lock().await; + let indices = { + let pool_meta = self.pool_meta.read().await; + first_resumable_decommission_queue_indices(&pool_meta) + .into_iter() + .filter(|idx| indices.contains(idx)) + .collect::>() + }; + if indices.is_empty() { + return Ok(Vec::new()); + } + + let index_cancelers = { + let mut cancelers = self.decommission_cancelers.write().await; + let missing = missing_decommission_worker_prefix(indices.as_slice(), cancelers.as_slice()); + if missing.is_empty() { + return Ok(Vec::new()); + } + let bound = bind_missing_decommission_cancelers(missing.as_slice(), rx, cancelers.as_mut_slice()); + let guards = guard_decommission_cancelers(bound); + ensure_decommission_routines_scheduled(guards.len(), missing.len())?; + guards + }; + Ok(index_cancelers) + } + pub(crate) async fn spawn_decommission_routines( &self, store: Arc, rx: CancellationToken, indices: Vec, ) -> Result<()> { - let indices = dedup_indices(&indices); - if indices.is_empty() { - return Ok(()); + let index_cancelers = self.reserve_decommission_routines(&rx, indices.as_slice()).await?; + if !index_cancelers.is_empty() { + std::mem::drop(spawn_decommission_index_cancelers( + store, + rx, + index_cancelers, + Arc::new(Semaphore::new(decommission_entry_concurrency_limit())), + )); } - let index_cancelers = { - let mut cancelers = self.decommission_cancelers.write().await; - bind_decommission_cancelers(indices.as_slice(), &rx, cancelers.as_mut_slice()) - }; - - ensure_decommission_routines_scheduled(index_cancelers.len(), indices.len())?; - - spawn_decommission_index_cancelers(store, rx, index_cancelers); - Ok(()) } @@ -2813,17 +3385,17 @@ impl ECStore { } let rx = CancellationToken::new(); - let index_cancelers = { - let mut cancelers = self.decommission_cancelers.write().await; - let missing = missing_decommission_worker_prefix(indices.as_slice(), cancelers.as_slice()); - bind_missing_decommission_cancelers(missing.as_slice(), &rx, cancelers.as_mut_slice()) - }; - + let index_cancelers = self.reserve_decommission_routines(&rx, indices.as_slice()).await?; if index_cancelers.is_empty() { return Ok(()); } - spawn_decommission_index_cancelers(self.clone(), rx, index_cancelers); + std::mem::drop(spawn_decommission_index_cancelers( + self.clone(), + rx, + index_cancelers, + Arc::new(Semaphore::new(decommission_entry_concurrency_limit())), + )); Ok(()) } @@ -2845,74 +3417,364 @@ impl ECStore { let store = require_decommission_store(runtime_sources::object_store_handle(), "start decommission")?; let local_indices = local_decommission_queue_prefix(&self.endpoints(), &indices)?; - - self.start_decommission(indices.clone()).await?; - if let Err(err) = self.spawn_decommission_routines(store, rx, local_indices).await { - let mut rollback_err: Option = None; - for idx in indices { - if let Err(cancel_err) = self.decommission_cancel(idx).await { - error!( - event = EVENT_DECOMMISSION_STATE, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - state = "rollback_failed", - error = ?cancel_err, - "Decommission rollback failed after spawn error" - ); - if rollback_err.is_none() { - rollback_err = Some(Error::other(format!("decommission rollback failed for idx {idx}: {cancel_err}"))); - } - } - } - return Err(resolve_decommission_spawn_failure_result(err, rollback_err)); - } + let index_cancelers = self + .start_decommission_with_routines(indices, &rx, local_indices.as_slice()) + .await?; + std::mem::drop(spawn_decommission_index_cancelers( + store, + rx, + index_cancelers, + Arc::new(Semaphore::new(decommission_entry_concurrency_limit())), + )); Ok(()) } - async fn save_decommission_entry_progress_stage( + async fn active_decommission_generation(&self, idx: usize) -> Result { + let pool_meta = self.pool_meta.read().await; + let Some(pool) = pool_meta.pools.get(idx) else { + return Err(invalid_decommission_pool_index_error(pool_meta.pools.len(), idx)); + }; + let Some(info) = pool.decommission.as_ref() else { + return Err(decommission_metadata_not_initialized_error("load decommission generation")); + }; + let Some(generation) = info.start_time else { + return Err(Error::OperationCanceled); + }; + ensure_decommission_generation(&pool_meta, idx, generation)?; + Ok(generation) + } + + async fn ensure_decommission_generation_current(&self, idx: usize, generation: OffsetDateTime) -> Result<()> { + let pool_meta = self.pool_meta.read().await; + ensure_decommission_generation(&pool_meta, idx, generation) + } + + #[allow(clippy::too_many_arguments)] + async fn decommission_entry_worker( + self: Arc, + rx: CancellationToken, + idx: usize, + set_idx: usize, + generation: OffsetDateTime, + bucket: String, + set: Arc, + lifecycle_config: Option, + object_lock_config: Option, + replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>, + expected_bucket_incarnation_id: Option, + entry_budget: Arc, + queue: Arc>>, + entry_error: Arc>>, + ) { + loop { + let queued = tokio::select! { + biased; + _ = rx.cancelled() => return, + item = async { + let mut queue = queue.lock().await; + queue.recv().await + } => item, + }; + let Some(QueuedDecommissionEntry { entry, queue_permit }) = queued else { + return; + }; + let object_name = entry.name.clone(); + + if entry_error.lock().await.is_some() { + drop(queue_permit); + continue; + } + + if let Err(err) = self.ensure_decommission_generation_current(idx, generation).await { + if matches!(err, Error::OperationCanceled) { + rx.cancel(); + } else { + record_decommission_entry_error(&entry_error, &rx, err).await; + } + return; + } + + if let Err(err) = backpressure::wait_for_data_movement_admission(DataMovementOperation::Decommission, idx, &rx).await + { + if matches!(err, Error::OperationCanceled) { + return; + } + error!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + set_index = set_idx, + bucket = %bucket, + object = %object_name, + state = "entry_admission_failed", + error = %err, + "Decommission entry admission failed" + ); + record_decommission_entry_error(&entry_error, &rx, err).await; + return; + } + + let entry_budget_permit = match tokio::select! { + biased; + _ = rx.cancelled() => return, + permit = entry_budget.clone().acquire_owned() => permit, + } { + Ok(permit) => permit, + Err(err) => { + let err = Error::other(format!("decommission entry budget permit acquire failed: {err}")); + error!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + set_index = set_idx, + bucket = %bucket, + object = %object_name, + state = "entry_budget_acquire_failed", + error = %err, + "Decommission entry budget permit acquire failed" + ); + record_decommission_entry_error(&entry_error, &rx, err).await; + return; + } + }; + + let result = self + .decommission_entry( + rx.clone(), + idx, + generation, + entry, + bucket.clone(), + set.clone(), + lifecycle_config.clone(), + object_lock_config.clone(), + replication_config.clone(), + expected_bucket_incarnation_id, + ) + .await; + drop(entry_budget_permit); + drop(queue_permit); + + if let Err(err) = result { + error!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + set_index = set_idx, + bucket = %bucket, + object = %object_name, + state = "entry_failed", + error = %err, + "Decommission entry failed" + ); + record_decommission_entry_error(&entry_error, &rx, err).await; + return; + } + } + } + + #[allow(clippy::too_many_arguments)] + async fn decommission_set( + self: Arc, + rx: CancellationToken, + idx: usize, + set_idx: usize, + generation: OffsetDateTime, + set: Arc, + bi: DecomBucketInfo, + lifecycle_config: Option, + object_lock_config: Option, + replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>, + expected_bucket_incarnation_id: Option, + entry_budget: Arc, + entry_error: Arc>>, + ) -> Result<()> { + let worker_count = DECOMMISSION_ENTRY_WORKERS_PER_SET; + let queue_capacity = decommission_entry_queue_capacity(worker_count); + let outstanding_capacity = queue_capacity.saturating_add(worker_count); + let outstanding = Arc::new(Semaphore::new(outstanding_capacity)); + let (tx, rx_queue) = mpsc::channel(queue_capacity); + let queue = Arc::new(tokio::sync::Mutex::new(rx_queue)); + + let mut entry_workers = tokio::task::JoinSet::new(); + for _ in 0..worker_count { + let this = self.clone(); + let rx = rx.clone(); + let bucket = bi.name.clone(); + let set = set.clone(); + let lifecycle_config = lifecycle_config.clone(); + let object_lock_config = object_lock_config.clone(); + let replication_config = replication_config.clone(); + let queue = queue.clone(); + let entry_budget = entry_budget.clone(); + let entry_error = entry_error.clone(); + entry_workers.spawn(async move { + this.decommission_entry_worker( + rx, + idx, + set_idx, + generation, + bucket, + set, + lifecycle_config, + object_lock_config, + replication_config, + expected_bucket_incarnation_id, + entry_budget, + queue, + entry_error, + ) + .await; + }); + } + + let callback: ListCallback = Arc::new({ + let tx = tx.clone(); + let outstanding = outstanding.clone(); + let callback_rx = rx.clone(); + let entry_error = entry_error.clone(); + let bucket = bi.name.clone(); + move |entry: MetaCacheEntry| { + let tx = tx.clone(); + let outstanding = outstanding.clone(); + let callback_rx = callback_rx.clone(); + let entry_error = entry_error.clone(); + let bucket = bucket.clone(); + Box::pin(async move { + if callback_rx.is_cancelled() || entry_error.lock().await.is_some() { + return; + } + + if matches!( + enqueue_decommission_entry(&callback_rx, &outstanding, &tx, entry).await, + DecommissionEntryEnqueueResult::Closed + ) { + let err = Error::other("decommission entry queue closed"); + error!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + set_index = set_idx, + bucket = %bucket, + state = "entry_queue_closed", + error = %err, + "Decommission entry queue closed" + ); + record_decommission_entry_error(&entry_error, &callback_rx, err).await; + } + }) + } + }); + + let list_set = set.clone(); + let list_rx = rx.clone(); + let list_rx_for_list = list_rx.clone(); + let list_rx_for_drain = list_rx.clone(); + let list_bi = bi.clone(); + let list_outstanding = outstanding.clone(); + let list_entry_error = entry_error.clone(); + let mut listing = tokio::spawn(async move { + run_decommission_listing_with_retry_and_drain( + list_rx.clone(), + list_bi.name.clone(), + callback, + idx, + set_idx, + DECOMMISSION_LISTING_MAX_ATTEMPTS, + move |callback| { + let set = list_set.clone(); + let rx = list_rx_for_list.clone(); + let bucket = list_bi.clone(); + let entry_error = list_entry_error.clone(); + async move { + set.list_objects_to_decommission(rx, bucket, callback, entry_error, idx, set_idx) + .await + } + }, + move || { + let rx = list_rx_for_drain.clone(); + let outstanding = list_outstanding.clone(); + async move { drain_decommission_entry_queue(&rx, &outstanding, outstanding_capacity).await } + }, + ) + .await + }); + + let mut listing_result = None; + let mut workers_left = worker_count; + let mut sender = Some(tx); + while listing_result.is_none() || workers_left > 0 { + tokio::select! { + biased; + result = &mut listing, if listing_result.is_none() => { + let result = resolve_decommission_listing_worker_result(set_idx, result); + if result.is_err() { + rx.cancel(); + } + listing_result = Some(result); + drop(sender.take()); + } + worker_result = entry_workers.join_next(), if workers_left > 0 => { + workers_left -= 1; + if let Some(Err(err)) = worker_result { + let err = Error::other(format!("decommission entry worker {set_idx} task join error: {err}")); + error!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + set_index = set_idx, + bucket = %bi.name, + state = "entry_worker_join_failed", + error = %err, + "Decommission entry worker task failed" + ); + record_decommission_entry_error(&entry_error, &rx, err).await; + } + } + } + } + + let listing_result = listing_result.unwrap_or_else(|| Err(Error::other("decommission listing task did not complete"))); + if let Some(err) = entry_error.lock().await.clone() { + return Err(err); + } + listing_result + } + + async fn track_decommission_entry_progress_stage( &self, idx: usize, + generation: OffsetDateTime, bucket: &str, object: &str, stage: &'static str, ) -> Result<()> { { let mut pool_meta = self.pool_meta.write().await; + ensure_decommission_generation(&pool_meta, idx, generation)?; track_decommission_current_object_stage(&mut pool_meta, idx, bucket, object, stage) .map_err(|err| with_decommission_entry_context(stage, bucket, object, err))?; - touch_decommission_progress(&mut pool_meta, idx) - .map_err(|err| with_decommission_entry_context(stage, bucket, object, err))?; - } - - if let Some(err) = resolve_decommission_progress_save_result(self.save_current_pool_meta().await) { - warn!( - event = EVENT_DECOMMISSION_ENTRY, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - bucket = %bucket, - object = %object, - stage, - error = ?err, - "Decommission progress stage save failed" - ); } Ok(()) } #[allow(unused_assignments, clippy::too_many_arguments)] - #[tracing::instrument(skip(self, set, _worker_permit, lifecycle_config, object_lock_config, replication_config))] + #[tracing::instrument(skip(self, set, lifecycle_config, object_lock_config, replication_config))] async fn decommission_entry( self: &Arc, rx: CancellationToken, idx: usize, + generation: OffsetDateTime, entry: MetaCacheEntry, bucket: String, set: Arc, - _worker_permit: OwnedSemaphorePermit, lifecycle_config: Option, object_lock_config: Option, replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>, @@ -2945,6 +3807,8 @@ impl ECStore { rx.cancel(); } decommission_cancel_signal_result(rx.is_cancelled())?; + self.ensure_decommission_generation_current(idx, generation).await?; + let operation_gate = self.ctx.decommission_operation_gate(); let bucket_incarnation_fence = match expected_bucket_incarnation_id { Some(expected) => Some(self.acquire_bucket_incarnation_fence(&bucket, expected).await?), @@ -2966,15 +3830,18 @@ impl ECStore { } decommission_cancel_signal_result(rx.is_cancelled())?; - if should_skip_lifecycle_for_data_movement( - self.clone(), - &bucket, - version, - lifecycle_config.as_ref(), - object_lock_config.as_ref(), - true, - &LcEventSrc::Decom, - ) + if run_decommission_side_effect(&rx, &operation_gate, || async { + should_skip_lifecycle_for_data_movement( + self.clone(), + &bucket, + version, + lifecycle_config.as_ref(), + object_lock_config.as_ref(), + true, + &LcEventSrc::Decom, + ) + .await + }) .await .map_err(|err| with_decommission_entry_context("lifecycle_expiry", bucket.as_str(), version.name.as_str(), err))? { @@ -3007,13 +3874,15 @@ impl ECStore { let mut failure = false; let mut error = None; if version.deleted { - if let Err(err) = self - .delete_object( + if let Err(err) = run_decommission_side_effect(&rx, &operation_gate, || async { + self.delete_object( bucket.as_str(), &version.name, decommission_delete_marker_opts(version, version_id.clone(), idx, expected_bucket_incarnation_id), ) .await + }) + .await { if is_decommission_copy_cleanup_safe_error(&err) { warn!( @@ -3065,6 +3934,7 @@ impl ECStore { { let mut pool_meta = self.pool_meta.write().await; + ensure_decommission_generation(&pool_meta, idx, generation)?; if let Err(err) = count_decommission_item(&mut pool_meta, idx, 0, failure) { return Err(with_decommission_entry_context( "count_decommission_item", @@ -3096,14 +3966,16 @@ impl ECStore { for _i in 0..3 { if version.is_remote() { - if let Err(err) = self - .decommission_tiered_object( + if let Err(err) = run_decommission_side_effect(&rx, &operation_gate, || async { + self.decommission_tiered_object( bucket.as_str(), &version.name, version, &decommission_remote_tiered_opts(version, version_id.clone(), idx, expected_bucket_incarnation_id), ) .await + }) + .await { if is_decommission_copy_cleanup_safe_error(&err) { ignore = true; @@ -3165,18 +4037,21 @@ impl ECStore { let bucket_name = bucket.clone(); let object_name = rd.object_info.name.clone(); - self.save_decommission_entry_progress_stage( + self.track_decommission_entry_progress_stage( idx, + generation, bucket_name.as_str(), object_name.as_str(), DECOMMISSION_STAGE_MIGRATE_OBJECT, ) .await?; - if let Err(err) = self - .clone() - .decommission_object(idx, bucket, rd, expected_bucket_incarnation_id) - .await + if let Err(err) = run_decommission_side_effect(&rx, &operation_gate, || async { + self.clone() + .decommission_object(idx, bucket, rd, expected_bucket_incarnation_id) + .await + }) + .await { if is_decommission_copy_cleanup_safe_error(&err) { ignore = true; @@ -3234,6 +4109,7 @@ impl ECStore { { let mut pool_meta = self.pool_meta.write().await; + ensure_decommission_generation(&pool_meta, idx, generation)?; if let Err(err) = count_decommission_item(&mut pool_meta, idx, decommission_item_size(version.size), failure) { return Err(with_decommission_entry_context( "count_decommission_item", @@ -3258,45 +4134,55 @@ impl ECStore { return Err(Error::other("decommission bucket incarnation fence was lost before source cleanup")); } decommission_cancel_signal_result(rx.is_cancelled())?; + self.ensure_decommission_generation_current(idx, generation).await?; - self.save_decommission_entry_progress_stage( + self.track_decommission_entry_progress_stage( idx, + generation, bucket.as_str(), entry.name.as_str(), DECOMMISSION_STAGE_CLEANUP_PREFLIGHT, ) .await?; - self.save_decommission_entry_progress_stage( + self.track_decommission_entry_progress_stage( idx, + generation, bucket.as_str(), entry.name.as_str(), DECOMMISSION_STAGE_SOURCE_CLEANUP, ) .await?; - let cleanup_result = data_movement::cleanup_source_entry_if_unchanged( - set.clone(), - bucket.as_str(), - entry.name.as_str(), - &fivs, - &cleanup_preflight_allowed_missing, - data_movement::SourceCleanupBucketFence { - expected_incarnation_id: expected_bucket_incarnation_id, - lifecycle_guard: bucket_incarnation_fence - .as_ref() - .and_then(|guard| guard.namespace_lock_guard()), - }, - "decommission", - ) - .await - .map_err(|err| match err { - data_movement::SourceCleanupError::SourceChanged => Error::other(format!( - "decommission: source cleanup preflight failed for {}/{}: source versions changed after migration started", - bucket, entry.name - )), - data_movement::SourceCleanupError::Storage(err) => err, - }); + let source_cleanup_mutation_fence = self + .acquire_decommission_source_cleanup_fence(bucket.as_str(), entry.name.as_str(), set.as_ref()) + .await?; + let cleanup_result = run_decommission_side_effect(&rx, &operation_gate, || async { + data_movement::cleanup_source_entry_if_unchanged( + set.clone(), + bucket.as_str(), + entry.name.as_str(), + &fivs, + &cleanup_preflight_allowed_missing, + data_movement::SourceCleanupBucketFence { + expected_incarnation_id: expected_bucket_incarnation_id, + lifecycle_guard: bucket_incarnation_fence + .as_ref() + .and_then(|guard| guard.namespace_lock_guard()), + object_mutation_fence: Some(&source_cleanup_mutation_fence), + }, + "decommission", + ) + .await + .map_err(|err| match err { + data_movement::SourceCleanupError::SourceChanged => Error::other(format!( + "decommission: source cleanup preflight failed for {}/{}: source versions changed after migration started", + bucket, entry.name + )), + data_movement::SourceCleanupError::Storage(err) => err, + }) + }) + .await; resolve_decommission_entry_cleanup_delete_result(cleanup_result, bucket.as_str(), entry.name.as_str())? } else if decommissioned != fivs.versions.len() || expired > 0 { warn!( @@ -3316,6 +4202,7 @@ impl ECStore { let should_save_progress = { let mut pool_meta = self.pool_meta.write().await; + ensure_decommission_generation(&pool_meta, idx, generation)?; if let Err(err) = track_decommission_current_object(&mut pool_meta, idx, bucket.as_str(), entry.name.as_str()) { return Err(with_decommission_entry_context( @@ -3334,34 +4221,43 @@ impl ECStore { } }; - self.save_decommission_entry_progress_stage(idx, bucket.as_str(), entry.name.as_str(), DECOMMISSION_STAGE_ENTRY_FINISHED) - .await?; + self.track_decommission_entry_progress_stage( + idx, + generation, + bucket.as_str(), + entry.name.as_str(), + DECOMMISSION_STAGE_ENTRY_FINISHED, + ) + .await?; if should_save_progress { - let save_result = self.save_current_pool_meta().await; - if let Some(err) = resolve_decommission_progress_save_result(save_result) { - warn!( - event = EVENT_DECOMMISSION_ENTRY, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - bucket = %bucket, - object = %entry.name, - state = "progress_save_failed", - error = %err, - "Decommission progress save failed; continuing and will retry at the next checkpoint" - ); - } else { - let mut pool_meta = self.pool_meta.write().await; - pool_meta.mark_decommission_progress_saved(); - if let Some(notification_sys) = runtime_sources::notification_sys() - && let Err(err) = resolve_decommission_entry_reload_result( - notification_sys.reload_pool_meta().await, - bucket.as_str(), - entry.name.as_str(), - ) - { - warn!("{err}"); + match self.save_decommission_progress_checkpoint(idx, generation).await { + Ok(true) => { + if let Some(notification_sys) = runtime_sources::notification_sys() + && let Err(err) = resolve_decommission_entry_reload_result( + notification_sys.reload_pool_meta().await, + bucket.as_str(), + entry.name.as_str(), + ) + { + warn!("{err}"); + } + } + Ok(false) => {} + Err(err) => { + if let Some(err) = resolve_decommission_progress_save_result(Err(err)) { + warn!( + event = EVENT_DECOMMISSION_ENTRY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + bucket = %bucket, + object = %entry.name, + state = "progress_save_failed", + error = %err, + "Decommission progress save failed; continuing and will retry at the next checkpoint" + ); + } } } } @@ -3379,6 +4275,29 @@ impl ECStore { Ok(()) } + #[cfg(test)] + pub(crate) async fn decommission_entry_for_test( + self: &Arc, + idx: usize, + entry: MetaCacheEntry, + bucket: String, + set: Arc, + ) -> Result<()> { + self.decommission_entry( + CancellationToken::new(), + idx, + OffsetDateTime::now_utc(), + entry, + bucket, + set, + None, + None, + None, + None, + ) + .await + } + #[tracing::instrument(skip(self, rx))] async fn decommission_pool( self: &Arc, @@ -3386,13 +4305,10 @@ impl ECStore { idx: usize, pool: Arc, bi: DecomBucketInfo, + entry_budget: Arc, ) -> Result<()> { - let worker_limit = pool.disk_set.len() * 2; - if worker_limit == 0 { - return Err(Error::other("decommission worker limit must be greater than zero")); - } - let workers = Arc::new(Semaphore::new(worker_limit)); let entry_error = Arc::new(tokio::sync::Mutex::new(None::)); + let generation = self.active_decommission_generation(idx).await?; let mut listing_workers = Vec::with_capacity(pool.disk_set.len()); let mut lifecycle_config = None; @@ -3421,12 +4337,6 @@ impl ECStore { } for (set_idx, set) in pool.disk_set.iter().enumerate() { - let listing_permit = workers - .clone() - .acquire_owned() - .await - .map_err(|err| Error::other(format!("decommission listing worker permit acquire failed: {err}")))?; - debug!( event = EVENT_DECOMMISSION_BUCKET, component = LOG_COMPONENT_ECSTORE, @@ -3438,125 +4348,34 @@ impl ECStore { "Decommission listing worker started" ); - let decommission_entry: ListCallback = Arc::new({ - let this = Arc::clone(self); - let bucket = bi.name.clone(); - let workers = workers.clone(); - let set = set.clone(); - let lifecycle_config = lifecycle_config.clone(); - let object_lock_config = object_lock_config.clone(); - let replication_config = replication_config.clone(); - let entry_error = entry_error.clone(); - let callback_rx = rx.clone(); - move |entry: MetaCacheEntry| { - let this = this.clone(); - let bucket = bucket.clone(); - let workers = workers.clone(); - let set = set.clone(); - let lifecycle_config = lifecycle_config.clone(); - let object_lock_config = object_lock_config.clone(); - let replication_config = replication_config.clone(); - let expected_bucket_incarnation_id = expected_bucket_incarnation_id; - let entry_error = entry_error.clone(); - let callback_rx = callback_rx.clone(); - - Box::pin(async move { - if callback_rx.is_cancelled() { - return; - } - if entry_error.lock().await.is_some() { - return; - } - - if let Err(err) = - backpressure::wait_for_data_movement_admission(DataMovementOperation::Decommission, idx, &callback_rx) - .await - { - if matches!(err, Error::OperationCanceled) { - return; - } - error!("decommission_pool: data movement admission failed: {err}"); - let mut first_err = entry_error.lock().await; - if first_err.is_none() { - *first_err = Some(err); - callback_rx.cancel(); - } - return; - } - - if entry_error.lock().await.is_some() { - return; - } - - let worker_permit = match tokio::select! { - _ = callback_rx.cancelled() => return, - permit = workers.clone().acquire_owned() => permit, - } { - Ok(permit) => permit, - Err(err) => { - let err = Error::other(format!("decommission entry worker permit acquire failed: {err}")); - error!("decommission_pool: decommission_entry failed: {err}"); - let mut first_err = entry_error.lock().await; - if first_err.is_none() { - *first_err = Some(err); - callback_rx.cancel(); - } - return; - } - }; - if entry_error.lock().await.is_some() { - return; - } - let entry_rx = callback_rx.clone(); - if let Err(err) = this - .decommission_entry( - entry_rx, - idx, - entry, - bucket, - set, - worker_permit, - lifecycle_config, - object_lock_config, - replication_config, - expected_bucket_incarnation_id, - ) - .await - { - error!("decommission_pool: decommission_entry failed: {err}"); - let mut first_err = entry_error.lock().await; - if first_err.is_none() { - *first_err = Some(err); - callback_rx.cancel(); - } - } - }) - } - }); - let set = set.clone(); + let store = Arc::clone(self); let rx_clone = rx.clone(); - let bi = bi.clone(); - let set_id = set_idx; + let bi_clone = bi.clone(); + let lifecycle_config = lifecycle_config.clone(); + let object_lock_config = object_lock_config.clone(); + let replication_config = replication_config.clone(); + let entry_budget = entry_budget.clone(); + let entry_error = entry_error.clone(); let worker = tokio::spawn(async move { - let _listing_permit = listing_permit; - run_decommission_listing_with_retry( - rx_clone.clone(), - bi.name.clone(), - decommission_entry.clone(), - idx, - set_id, - DECOMMISSION_LISTING_MAX_ATTEMPTS, - |callback| { - let set = set.clone(); - let rx = rx_clone.clone(); - let bucket = bi.clone(); - async move { set.list_objects_to_decommission(rx, bucket, callback).await } - }, - ) - .await + store + .decommission_set( + rx_clone, + idx, + set_idx, + generation, + set, + bi_clone, + lifecycle_config, + object_lock_config, + replication_config, + expected_bucket_incarnation_id, + entry_budget, + entry_error, + ) + .await }); - listing_workers.push((set_id, worker)); + listing_workers.push((set_idx, worker)); } debug!( @@ -3579,8 +4398,6 @@ impl ECStore { } } - wait_decommission_worker_drain(&workers, worker_limit).await?; - if let Some(err) = listing_worker_error { return Err(err); } @@ -3616,26 +4433,36 @@ impl ECStore { Ok(()) } - #[tracing::instrument(skip(self, rx))] - pub async fn do_decommission_in_routine(self: &Arc, rx: CancellationToken, idx: usize) -> Result<()> { - defer!(|| async { - let mut cancelers = self.decommission_cancelers.write().await; - if take_decommission_canceler(cancelers.as_mut_slice(), idx).is_none() { - warn!( - event = EVENT_DECOMMISSION_STATE, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - state = "canceler_already_cleared", - "Decommission canceler already cleared" - ); - } - }); + #[tracing::instrument(skip(self, canceler))] + pub async fn do_decommission_in_routine( + self: &Arc, + canceler: DecommissionCanceler, + idx: usize, + entry_budget: Arc, + ) -> Result<()> { + let rx = canceler.token().clone(); + self.run_decommission_in_routine(rx, idx, &canceler, entry_budget).await + } - if let Err(err) = self.promote_queued_decommission(idx).await { - resolve_decommission_terminal_mark_after_error_result(self.decommission_failed(idx).await, idx, &err)?; - return Err(err); - } + async fn run_decommission_in_routine( + self: &Arc, + rx: CancellationToken, + idx: usize, + canceler: &DecommissionCanceler, + entry_budget: Arc, + ) -> Result<()> { + let generation = match self.promote_queued_decommission(idx, canceler).await { + Ok(generation) => generation, + Err(Error::OperationCanceled) => return Ok(()), + Err(err) => { + resolve_decommission_terminal_mark_after_error_result( + self.decommission_failed_for_operation(idx, canceler).await, + idx, + &err, + )?; + return Err(err); + } + }; if rx.is_cancelled() { let already_canceled = { let pool_meta = self.pool_meta.read().await; @@ -3652,13 +4479,17 @@ impl ECStore { ); return Ok(()); } - if let Err(err) = self.decommission_cancel(idx).await { - resolve_decommission_terminal_mark_after_error_result(self.decommission_failed(idx).await, idx, &err)?; + if let Err(err) = self.decommission_cancel_for_operation(idx, canceler).await { + resolve_decommission_terminal_mark_after_error_result( + self.decommission_failed_for_operation(idx, canceler).await, + idx, + &err, + )?; return Err(err); } return Ok(()); } - let result = self.decommission_in_background(rx.clone(), idx).await; + let result = self.decommission_in_background(rx.clone(), idx, entry_budget).await; let (final_state, canceled, cmd_line) = { let pool_meta = self.pool_meta.read().await; @@ -3712,7 +4543,11 @@ impl ECStore { return Ok(()); } - resolve_decommission_terminal_mark_after_error_result(self.decommission_failed(idx).await, idx, &err)?; + resolve_decommission_terminal_mark_after_error_result( + self.decommission_failed_for_operation(idx, canceler).await, + idx, + &err, + )?; warn!( event = EVENT_DECOMMISSION_STATE, component = LOG_COMPONENT_ECSTORE, @@ -3759,12 +4594,22 @@ impl ECStore { "Decommission completion verification started" ); if let Err(err) = self.check_after_decommission(idx).await { - resolve_decommission_terminal_mark_result(self.decommission_failed(idx).await, "failed", &cmd_line)?; + resolve_decommission_terminal_mark_result( + self.decommission_failed_for_operation(idx, canceler).await, + "failed", + &cmd_line, + )?; return Err(Error::other(format!( "failed to finalize decommission for pool {cmd_line}: post-check failed: {err}" ))); } + if self.decommission_cancel_requested(idx, &rx).await { + rx.cancel(); + } + decommission_cancel_signal_result(rx.is_cancelled())?; + self.ensure_decommission_generation_current(idx, generation).await?; + info!( event = EVENT_DECOMMISSION_STATE, component = LOG_COMPONENT_ECSTORE, @@ -3774,7 +4619,11 @@ impl ECStore { state = "marking_completed", "Decommission marking completed state" ); - resolve_decommission_terminal_mark_result(self.complete_decommission(idx).await, "completed", &cmd_line)?; + resolve_decommission_terminal_mark_result( + self.complete_decommission_for_operation(idx, canceler).await, + "completed", + &cmd_line, + )?; } DecommissionFinalState::Failed => { warn!( @@ -3786,7 +4635,11 @@ impl ECStore { state = "marking_failed", "Decommission marking failed state" ); - resolve_decommission_terminal_mark_result(self.decommission_failed(idx).await, "failed", &cmd_line)?; + resolve_decommission_terminal_mark_result( + self.decommission_failed_for_operation(idx, canceler).await, + "failed", + &cmd_line, + )?; } } @@ -3804,63 +4657,97 @@ impl ECStore { #[tracing::instrument(skip(self))] pub async fn decommission_failed(&self, idx: usize) -> Result<()> { - ensure_decommission_terminal_operation_supported(self.single_pool(), "mark decommission failed")?; + self.decommission_failed_with_owner(idx, None).await + } - let (should_reload_pool_meta, previous_pool_meta) = { + async fn decommission_failed_for_operation(&self, idx: usize, owner: &DecommissionCanceler) -> Result<()> { + self.decommission_failed_with_owner(idx, Some(owner)).await + } + + async fn decommission_failed_with_owner(&self, idx: usize, owner: Option<&DecommissionCanceler>) -> Result<()> { + self.decommission_failed_with_owner_and_save(idx, owner, self.save_current_pool_meta()) + .await + } + + async fn decommission_failed_with_owner_and_save( + &self, + idx: usize, + owner: Option<&DecommissionCanceler>, + save_pool_meta: SaveFuture, + ) -> Result<()> + where + SaveFuture: Future>, + { + ensure_decommission_terminal_operation_supported(self.single_pool(), "mark decommission failed")?; + let _start_guard = self.start_gate.lock().await; + + // Lock order: decommission_cancelers before pool_meta. Holding both makes + // owner validation and the terminal transition one atomic operation. + let (should_reload_pool_meta, previous_pool_meta, terminal_canceler) = { + let cancelers = self.decommission_cancelers.read().await; let mut pool_meta = self.pool_meta.write().await; let previous_pool_meta = pool_meta.clone(); - let changed = pool_meta.decommission_failed(idx); - (changed, changed.then_some(previous_pool_meta)) + let Some(changed) = + update_decommission_for_operation(cancelers.as_slice(), &mut pool_meta, idx, owner, |pool_meta| { + pool_meta.decommission_failed(idx) + }) + else { + return Ok(()); + }; + let terminal_canceler = if let Some(owner) = owner { + Some(owner.clone()) + } else { + cancelers.get(idx).and_then(Option::as_ref).cloned() + }; + (changed, changed.then_some(previous_pool_meta), terminal_canceler) }; - { - let mut cancelers = self.decommission_cancelers.write().await; - take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), idx); - } - - if should_reload_pool_meta { - if let Err(err) = self.save_current_pool_meta().await { - if let Some(previous_pool_meta) = previous_pool_meta { - let mut pool_meta = self.pool_meta.write().await; - rollback_decommission_pool_meta(&mut pool_meta, previous_pool_meta); - } - return Err(err); + if should_reload_pool_meta && let Err(err) = save_pool_meta.await { + if let Some(previous_pool_meta) = previous_pool_meta { + let mut pool_meta = self.pool_meta.write().await; + rollback_decommission_pool_meta(&mut pool_meta, previous_pool_meta); } + return Err(err); + } + if should_reload_pool_meta { { let mut pool_meta = self.pool_meta.write().await; pool_meta.mark_decommission_progress_saved(); } - if let Some(notification_sys) = runtime_sources::notification_sys() { - let stage = format!("decommission_failed for pool {idx}"); - if let Some(err) = observe_decommission_terminal_reload_result( - resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str()), - stage.as_str(), - ) { - if let Err(record_err) = self - .record_decommission_terminal_reload_failure(idx, stage.as_str(), err.clone()) - .await - { - warn!( - event = EVENT_DECOMMISSION_STATE, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - state = "terminal_reload_record_failed", - error = %record_err, - original_error = %err, - "Decommission terminal reload failure record failed" - ); - } + } + if let Some(canceler) = terminal_canceler.as_ref() { + self.release_decommission_canceler_slot(idx, canceler).await; + } + if should_reload_pool_meta && let Some(notification_sys) = runtime_sources::notification_sys() { + let stage = format!("decommission_failed for pool {idx}"); + if let Some(err) = observe_decommission_terminal_reload_result( + resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str()), + stage.as_str(), + ) { + if let Err(record_err) = self + .record_decommission_terminal_reload_failure(idx, stage.as_str(), err.clone()) + .await + { warn!( event = EVENT_DECOMMISSION_STATE, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_POOLS, pool_index = idx, - state = "terminal_reload_failed", - error = %err, - "Decommission terminal state saved but pool meta reload failed" + state = "terminal_reload_record_failed", + error = %record_err, + original_error = %err, + "Decommission terminal reload failure record failed" ); } + warn!( + event = EVENT_DECOMMISSION_STATE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + state = "terminal_reload_failed", + error = %err, + "Decommission terminal state saved but pool meta reload failed" + ); } } @@ -3869,63 +4756,84 @@ impl ECStore { #[tracing::instrument(skip(self))] pub async fn complete_decommission(&self, idx: usize) -> Result<()> { - ensure_decommission_terminal_operation_supported(self.single_pool(), "complete decommission")?; + self.complete_decommission_with_owner(idx, None).await + } - let (should_reload_pool_meta, previous_pool_meta) = { + async fn complete_decommission_for_operation(&self, idx: usize, owner: &DecommissionCanceler) -> Result<()> { + self.complete_decommission_with_owner(idx, Some(owner)).await + } + + async fn complete_decommission_with_owner(&self, idx: usize, owner: Option<&DecommissionCanceler>) -> Result<()> { + ensure_decommission_terminal_operation_supported(self.single_pool(), "complete decommission")?; + let _start_guard = self.start_gate.lock().await; + + // Lock order: decommission_cancelers before pool_meta. Holding both makes + // owner validation and the terminal transition one atomic operation. + let (should_reload_pool_meta, previous_pool_meta, terminal_canceler) = { + let cancelers = self.decommission_cancelers.read().await; let mut pool_meta = self.pool_meta.write().await; let previous_pool_meta = pool_meta.clone(); - let changed = pool_meta.decommission_complete(idx); - (changed, changed.then_some(previous_pool_meta)) + let Some(changed) = + update_decommission_for_operation(cancelers.as_slice(), &mut pool_meta, idx, owner, |pool_meta| { + pool_meta.decommission_complete(idx) + }) + else { + return Ok(()); + }; + let terminal_canceler = if let Some(owner) = owner { + Some(owner.clone()) + } else { + cancelers.get(idx).and_then(Option::as_ref).cloned() + }; + (changed, changed.then_some(previous_pool_meta), terminal_canceler) }; - { - let mut cancelers = self.decommission_cancelers.write().await; - take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), idx); - } - - if should_reload_pool_meta { - if let Err(err) = self.save_current_pool_meta().await { - if let Some(previous_pool_meta) = previous_pool_meta { - let mut pool_meta = self.pool_meta.write().await; - rollback_decommission_pool_meta(&mut pool_meta, previous_pool_meta); - } - return Err(err); + if should_reload_pool_meta && let Err(err) = self.save_current_pool_meta().await { + if let Some(previous_pool_meta) = previous_pool_meta { + let mut pool_meta = self.pool_meta.write().await; + rollback_decommission_pool_meta(&mut pool_meta, previous_pool_meta); } + return Err(err); + } + if should_reload_pool_meta { { let mut pool_meta = self.pool_meta.write().await; pool_meta.mark_decommission_progress_saved(); } - if let Some(notification_sys) = runtime_sources::notification_sys() { - let stage = format!("complete_decommission for pool {idx}"); - if let Some(err) = observe_decommission_terminal_reload_result( - resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str()), - stage.as_str(), - ) { - if let Err(record_err) = self - .record_decommission_terminal_reload_failure(idx, stage.as_str(), err.clone()) - .await - { - warn!( - event = EVENT_DECOMMISSION_STATE, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_POOLS, - pool_index = idx, - state = "terminal_reload_record_failed", - error = %record_err, - original_error = %err, - "Decommission terminal reload failure record failed" - ); - } + } + if let Some(canceler) = terminal_canceler.as_ref() { + self.release_decommission_canceler_slot(idx, canceler).await; + } + if should_reload_pool_meta && let Some(notification_sys) = runtime_sources::notification_sys() { + let stage = format!("complete_decommission for pool {idx}"); + if let Some(err) = observe_decommission_terminal_reload_result( + resolve_decommission_pool_meta_reload_result(notification_sys.reload_pool_meta().await, stage.as_str()), + stage.as_str(), + ) { + if let Err(record_err) = self + .record_decommission_terminal_reload_failure(idx, stage.as_str(), err.clone()) + .await + { warn!( event = EVENT_DECOMMISSION_STATE, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_POOLS, pool_index = idx, - state = "terminal_reload_failed", - error = %err, - "Decommission terminal state saved but pool meta reload failed" + state = "terminal_reload_record_failed", + error = %record_err, + original_error = %err, + "Decommission terminal reload failure record failed" ); } + warn!( + event = EVENT_DECOMMISSION_STATE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + pool_index = idx, + state = "terminal_reload_failed", + error = %err, + "Decommission terminal state saved but pool meta reload failed" + ); } } @@ -3938,6 +4846,7 @@ impl ECStore { idx: usize, pool: Arc, bucket: DecomBucketInfo, + entry_budget: Arc, ) -> Result<()> { let is_decommissioned = { let pool_meta = self.pool_meta.read().await; @@ -3963,7 +4872,10 @@ impl ECStore { warn!("decommission: currently on bucket {}", &bucket.name); - if let Err(err) = self.decommission_pool(rx.clone(), idx, pool, bucket.clone()).await { + if let Err(err) = self + .decommission_pool(rx.clone(), idx, pool, bucket.clone(), entry_budget) + .await + { error!("decommission: decommission_pool err {:?}", &err); return Err(err); } else { @@ -3997,40 +4909,53 @@ impl ECStore { pool: Arc, buckets: Vec, limit: usize, + entry_budget: Arc, ) -> Result<()> { let store = Arc::clone(self); run_decommission_buckets_bounded(rx, buckets, limit, move |bucket, rx| { let store = Arc::clone(&store); let pool = pool.clone(); - Box::pin(async move { store.decommission_pending_bucket(rx, idx, pool, bucket).await }) + let entry_budget = entry_budget.clone(); + Box::pin(async move { store.decommission_pending_bucket(rx, idx, pool, bucket, entry_budget).await }) }) .await } #[tracing::instrument(skip(self, rx))] - async fn decommission_in_background(self: &Arc, rx: CancellationToken, idx: usize) -> Result<()> { + async fn decommission_in_background( + self: &Arc, + rx: CancellationToken, + idx: usize, + entry_budget: Arc, + ) -> Result<()> { let pool = get_by_index(self.pools.as_slice(), idx, "load decommission background pool")?.clone(); let pending = { let pool_meta = self.pool_meta.read().await; pool_meta.pending_buckets(idx) }; - let bucket_concurrency = decommission_bucket_concurrency_limit(); if bucket_concurrency <= 1 { for bucket in pending { - self.decommission_pending_bucket(rx.clone(), idx, pool.clone(), bucket) + self.decommission_pending_bucket(rx.clone(), idx, pool.clone(), bucket, entry_budget.clone()) .await?; } return Ok(()); } let (regular_buckets, meta_buckets) = split_decommission_buckets(pending); - self.decommission_buckets_concurrently(rx.clone(), idx, pool.clone(), regular_buckets, bucket_concurrency) - .await?; + self.decommission_buckets_concurrently( + rx.clone(), + idx, + pool.clone(), + regular_buckets, + bucket_concurrency, + entry_budget.clone(), + ) + .await?; for bucket in meta_buckets { - self.decommission_pending_bucket(rx.clone(), idx, pool.clone(), bucket) + self.decommission_pending_bucket(rx.clone(), idx, pool.clone(), bucket, entry_budget.clone()) .await?; } @@ -4039,6 +4964,23 @@ impl ECStore { #[tracing::instrument(skip(self))] pub async fn start_decommission(&self, indices: Vec) -> Result<()> { + self.start_decommission_inner(indices, None).await.map(|_| ()) + } + + async fn start_decommission_with_routines( + &self, + indices: Vec, + rx: &CancellationToken, + local_indices: &[usize], + ) -> Result> { + self.start_decommission_inner(indices, Some((rx, local_indices))).await + } + + async fn start_decommission_inner( + &self, + indices: Vec, + reservation: Option<(&CancellationToken, &[usize])>, + ) -> Result> { let indices = dedup_indices(&indices); validate_start_decommission_request(&indices, self.single_pool())?; @@ -4081,11 +5023,21 @@ impl ECStore { self.ensure_decommission_rebalance_idle_after_refresh().await?; let all_space_infos = self.get_decommission_all_pool_space_infos().await?; - { + self.cancel_decommission_routines_and_wait(&indices).await; + + let index_cancelers = if let Some((rx, local_indices)) = reservation { + // Lock order matches terminal transitions: decommission_cancelers + // before pool_meta while start_gate excludes another start. + let mut cancelers = self.decommission_cancelers.write().await; + let pool_meta = self.pool_meta.read().await; + ensure_decommission_start_target_capacity(&pool_meta, &indices, &all_space_infos)?; + reserve_decommission_start_cancelers(&pool_meta, &indices, local_indices, rx, cancelers.as_mut_slice())? + } else { let pool_meta = self.pool_meta.read().await; ensure_decommission_start_pool_states(&pool_meta, &indices)?; ensure_decommission_start_target_capacity(&pool_meta, &indices, &all_space_infos)?; - } + Vec::new() + }; let mut space_infos = Vec::with_capacity(indices.len()); for (idx, pi) in all_space_infos.iter().copied() { @@ -4161,7 +5113,7 @@ impl ECStore { return Err(Error::other(format!("{err}; decommission start rollback succeeded"))); } - Ok(()) + Ok(index_cancelers) } async fn get_buckets_to_decommission(&self) -> Result> { @@ -4191,7 +5143,7 @@ impl ECStore { let buckets = self.get_buckets_to_decommission().await?; let pool = self.pools[idx].clone(); - for set in &pool.disk_set { + for (set_index, set) in pool.disk_set.iter().enumerate() { for bucket_info in &buckets { let mut lifecycle_config = None; let mut object_lock_config = None; @@ -4286,7 +5238,7 @@ impl ECStore { }); let list_result = set - .list_objects_to_decommission(callback_rx, bucket_info.clone(), callback) + .list_objects_to_decommission(callback_rx, bucket_info.clone(), callback, entry_error.clone(), idx, set_index) .await; let entry_error = entry_error.lock().await.clone(); resolve_decommission_check_after_list_result(list_result, entry_error)?; @@ -4314,15 +5266,20 @@ impl ECStore { ) -> Result<()> { warn!("decommission_object: start {} {}", &bucket, &rd.object_info.name); let object_name = rd.object_info.name.clone(); - let result = data_movement::migrate_object( + let mut migration = tokio::task::JoinSet::new(); + migration.spawn(data_movement::migrate_decommission_object( self, pool_idx, bucket.clone(), rd, expected_bucket_incarnation_id, "decommission_object", - ) - .await; + )); + let result = migration + .join_next() + .await + .ok_or_else(|| Error::other("decommission migration task was not started"))? + .map_err(|err| Error::other(format!("decommission migration task join error: {err}")))?; if result.is_ok() { warn!("decommission_object: migrated {} {}", &bucket, &object_name); } @@ -4637,6 +5594,14 @@ mod tests { let mut pool_meta = build_pool_meta(); assert!(pool_meta.decommission_cancel(0)); assert_eq!(pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time), None); + + let mut pool_meta = build_pool_meta(); + assert!(pool_meta.decommission_cancel(0)); + assert!(!pool_meta.decommission_complete(0)); + + let mut pool_meta = build_pool_meta(); + assert!(pool_meta.decommission_failed(0)); + assert!(!pool_meta.decommission_complete(0)); } #[test] @@ -5020,13 +5985,89 @@ mod tests { pub type ListCallback = Arc BoxFuture<'static, ()> + Send + Sync + 'static>; +const DECOMMISSION_ENTRY_QUEUE_HARD_CAP: usize = 256; + +struct QueuedDecommissionEntry { + entry: MetaCacheEntry, + queue_permit: OwnedSemaphorePermit, +} + +enum DecommissionEntryEnqueueResult { + Enqueued, + Canceled, + Closed, +} + +fn decommission_entry_queue_capacity(worker_limit: usize) -> usize { + worker_limit.saturating_mul(2).clamp(1, DECOMMISSION_ENTRY_QUEUE_HARD_CAP) +} + +async fn enqueue_decommission_entry( + rx: &CancellationToken, + outstanding: &Arc, + tx: &mpsc::Sender, + entry: MetaCacheEntry, +) -> DecommissionEntryEnqueueResult { + let queue_permit = match tokio::select! { + biased; + _ = rx.cancelled() => return DecommissionEntryEnqueueResult::Canceled, + permit = outstanding.clone().acquire_owned() => permit, + } { + Ok(permit) => permit, + Err(_) => return DecommissionEntryEnqueueResult::Closed, + }; + + let queued = QueuedDecommissionEntry { entry, queue_permit }; + tokio::select! { + biased; + _ = rx.cancelled() => DecommissionEntryEnqueueResult::Canceled, + result = tx.send(queued) => { + if result.is_ok() { + DecommissionEntryEnqueueResult::Enqueued + } else { + DecommissionEntryEnqueueResult::Closed + } + } + } +} + +async fn drain_decommission_entry_queue(rx: &CancellationToken, outstanding: &Arc, capacity: usize) -> bool { + let Ok(permits) = u32::try_from(capacity) else { + return true; + }; + + tokio::select! { + _ = rx.cancelled() => true, + result = outstanding.acquire_many(permits) => result.is_err(), + } +} + +async fn record_decommission_entry_error( + entry_error: &Arc>>, + rx: &CancellationToken, + err: Error, +) { + if rx.is_cancelled() { + return; + } + + let mut first_err = entry_error.lock().await; + if first_err.is_none() && !rx.is_cancelled() { + *first_err = Some(err); + rx.cancel(); + } +} + impl SetDisks { - #[tracing::instrument(skip(self, rx, cb_func))] + #[tracing::instrument(skip(self, rx, cb_func, entry_error))] async fn list_objects_to_decommission( self: &Arc, rx: CancellationToken, bucket_info: DecomBucketInfo, cb_func: ListCallback, + entry_error: Arc>>, + pool_index: usize, + set_index: usize, ) -> Result<()> { let (disks, _) = self.get_online_disks_with_healing(false).await; ensure_decommission_listing_disks_available(!disks.is_empty(), &bucket_info.name)?; @@ -5041,6 +6082,12 @@ impl SetDisks { }; let cb1 = cb_func.clone(); + let unresolved_error = entry_error.clone(); + let unresolved_rx = rx.clone(); + let unresolved_bucket = bucket_info.name.clone(); + let unresolved_prefix = bucket_info.prefix.clone(); + let unresolved_pool_index = pool_index; + let unresolved_set_index = set_index; list_path_raw( rx, @@ -5053,20 +6100,51 @@ impl SetDisks { skip_walkdir_total_timeout: true, walkdir_stall_timeout: Some(DECOMMISSION_BACKGROUND_WALKDIR_STALL_TIMEOUT), agreed: Some(Box::new(move |entry: MetaCacheEntry| Box::pin(cb1(entry)))), - partial: Some(Box::new(move |entries: MetaCacheEntries, _: &[Option]| { + partial: Some(Box::new(move |entries: MetaCacheEntries, errs: &[Option]| { let resolver = resolver.clone(); let cb_func = cb_func.clone(); - match entries.resolve(resolver) { - Some(entry) => { + let bucket = unresolved_bucket.clone(); + let prefix = unresolved_prefix.clone(); + let unresolved_error = unresolved_error.clone(); + let unresolved_rx = unresolved_rx.clone(); + let pool_index = unresolved_pool_index; + let set_index = unresolved_set_index; + let disk_error_count = errs.iter().flatten().count(); + if unresolved_rx.is_cancelled() { + return Box::pin(async {}); + } + + match resolve_decommission_partial_listing_entry( + entries, + resolver, + &bucket, + &prefix, + disk_error_count, + pool_index, + set_index, + ) { + Ok(entry) => { warn!("decommission_pool: list_objects_to_decommission get {}", &entry.name); Box::pin(async move { cb_func(entry).await; }) } - None => { - warn!("decommission_pool: list_objects_to_decommission get none"); - Box::pin(async {}) - } + Err(err) => Box::pin(async move { + if unresolved_rx.is_cancelled() { + return; + } + warn!( + event = EVENT_DECOMMISSION_BUCKET, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_POOLS, + bucket = %bucket, + prefix = %prefix, + state = "unresolved_entry", + error = %err, + "Decommission listing failed closed on unresolved metadata" + ); + record_decommission_entry_error(&unresolved_error, &unresolved_rx, err).await; + }), } })), ..Default::default() @@ -5074,6 +6152,10 @@ impl SetDisks { ) .await?; + if let Some(err) = entry_error.lock().await.clone() { + return Err(err); + } + Ok(()) } } @@ -5263,46 +6345,56 @@ pub(crate) fn fallback_free_capacity_dedup(disks: &[rustfs_madmin::Disk]) -> usi #[cfg(test)] mod pools_tests { + use super::DECOMMISSION_PROGRESS_SAVE_RETRY_BACKOFF; + use super::record_decommission_entry_error; + use super::resolve_decommission_listing_error; use super::{ - DECOMMISSION_PROGRESS_SAVE_INTERVAL, DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, DecomBucketInfo, - DecommissionStartPoolState, DecommissionTerminalState, ListCallback, PoolDecommissionInfo, PoolMeta, PoolSpaceInfo, - PoolStatus, apply_decommission_status_space_info, bind_decommission_cancelers, bind_missing_decommission_cancelers, - cancel_decommission_canceler, classify_decommission_terminal_state, count_decommission_item, - decommission_cancel_signal_result, decommission_item_size, decommission_meta_bucket_options, - decommission_start_pool_state, dedup_indices, default_decommission_bucket_concurrency, - ensure_decommission_cancel_allowed, ensure_decommission_clear_allowed, ensure_decommission_listing_disks_available, - ensure_decommission_not_rebalancing, ensure_decommission_start_allowed, ensure_decommission_start_keeps_active_pool, - ensure_decommission_start_local_leader, ensure_decommission_start_pool_states, - ensure_decommission_start_rebalance_meta_allowed, ensure_decommission_start_target_capacity, - ensure_decommission_terminal_operation_supported, ensure_local_decommission_pool_leaders, - ensure_valid_decommission_pool_index, first_resumable_decommission_queue_indices, get_by_index, - has_active_decommission_canceler, is_decommission_active, is_decommission_cancel_requested, - load_decommission_entry_versions, local_decommission_queue_prefix, mark_decommission_bucket_done, - merge_pool_status_refresh, missing_decommission_worker_prefix, observe_decommission_terminal_reload_result, - pool_meta_has_active_decommission, require_decommission_store, resolve_decommission_bucket_done_save_result, - resolve_decommission_bucket_state, resolve_decommission_check_after_list_result, - resolve_decommission_entry_cleanup_delete_result, resolve_decommission_entry_exact_versions, - resolve_decommission_entry_reload_result, resolve_decommission_listing_worker_result, - resolve_decommission_optional_bucket_config_result, resolve_decommission_pool_meta_reload_result, + DECOMMISSION_ENTRY_CONCURRENCY_DEFAULT_CAP, DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP, DECOMMISSION_ENTRY_QUEUE_HARD_CAP, + DECOMMISSION_PROGRESS_SAVE_INTERVAL, DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, DecomBucketInfo, DecommissionCanceler, + DecommissionEntryEnqueueResult, DecommissionStartPoolState, DecommissionTerminalState, ListCallback, + PoolDecommissionInfo, PoolMeta, PoolSpaceInfo, PoolStatus, QueuedDecommissionEntry, apply_decommission_status_space_info, + await_decommission_worker, bind_decommission_cancelers, bind_missing_decommission_cancelers, + cancel_decommission_canceler, clamp_decommission_entry_concurrency, classify_decommission_terminal_state, + count_decommission_item, decommission_cancel_signal_result, decommission_entry_queue_capacity, decommission_item_size, + decommission_meta_bucket_options, decommission_start_pool_state, dedup_indices, default_decommission_bucket_concurrency, + default_decommission_entry_concurrency, drain_decommission_entry_queue, enqueue_decommission_entry, + ensure_decommission_cancel_allowed, ensure_decommission_clear_allowed, ensure_decommission_generation, + ensure_decommission_listing_disks_available, ensure_decommission_not_rebalancing, ensure_decommission_start_allowed, + ensure_decommission_start_keeps_active_pool, ensure_decommission_start_local_leader, + ensure_decommission_start_pool_states, ensure_decommission_start_rebalance_meta_allowed, + ensure_decommission_start_target_capacity, ensure_decommission_terminal_operation_supported, + ensure_local_decommission_pool_leaders, ensure_valid_decommission_pool_index, first_resumable_decommission_queue_indices, + get_by_index, guard_decommission_cancelers, has_active_decommission_canceler, is_decommission_active, + is_decommission_cancel_requested, load_decommission_entry_versions, local_decommission_queue_prefix, + mark_decommission_bucket_done, merge_pool_status_refresh, missing_decommission_worker_prefix, + observe_decommission_terminal_reload_result, pool_meta_has_active_decommission, require_decommission_store, + reserve_decommission_start_cancelers, resolve_decommission_bucket_done_save_result, resolve_decommission_bucket_state, + resolve_decommission_check_after_list_result, resolve_decommission_entry_cleanup_delete_result, + resolve_decommission_entry_exact_versions, resolve_decommission_entry_reload_result, + resolve_decommission_listing_worker_result, resolve_decommission_optional_bucket_config_result, + resolve_decommission_partial_listing_entry, resolve_decommission_pool_meta_reload_result, resolve_decommission_preflight_heal_result, resolve_decommission_progress_save_result, - resolve_decommission_spawn_failure_result, resolve_decommission_terminal_mark_after_error_result, - resolve_decommission_terminal_mark_result, resolve_decommission_update_after_result, - resolve_start_decommission_pool_meta_reload_result, rollback_start_decommission_pool_meta, - run_decommission_buckets_bounded, run_decommission_listing_with_retry, should_cleanup_decommission_source_entry, + resolve_decommission_terminal_mark_after_error_result, resolve_decommission_terminal_mark_result, + resolve_decommission_update_after_result, resolve_start_decommission_pool_meta_reload_result, + rollback_start_decommission_pool_meta, run_decommission_buckets_bounded, run_decommission_listing_with_retry, + run_decommission_listing_with_retry_and_drain, run_decommission_side_effect, should_cleanup_decommission_source_entry, should_continue_decommission_queue, should_count_decommission_version_complete, should_preserve_decommission_canceled_state, should_reject_decommission_cancel_as_terminal, should_retry_decommission_cancel_reload, should_retry_decommission_listing, should_skip_canceled_decommission_routine, - split_decommission_buckets, take_and_cancel_decommission_canceler, take_decommission_canceler, - touch_decommission_progress, track_decommission_current_object, track_decommission_current_object_stage, - validate_start_decommission_request, wait_decommission_listing_retry, wait_decommission_worker_drain, - with_decommission_entry_context, + spawn_decommission_index_cancelers, split_decommission_buckets, take_and_cancel_decommission_canceler, + take_decommission_canceler, track_decommission_current_object, track_decommission_current_object_stage, + update_decommission_for_operation, validate_start_decommission_request, wait_decommission_listing_retry, + wait_decommission_worker_drain, with_decommission_entry_context, }; use crate::data_movement; use crate::disk::endpoint::Endpoint; use crate::error::{Error, StorageError}; use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints}; + use crate::runtime::instance::InstanceContext; use crate::services::rebalance::{RebalStatus, RebalanceInfo, RebalanceMeta, RebalanceStats}; + use crate::store::ECStore; use rustfs_filemeta::{FileInfo, FileInfoVersions, MetaCacheEntry, ObjectPartInfo}; + use rustfs_filemeta::{MetaCacheEntries, MetadataResolutionParams}; use rustfs_rio::Index; use std::sync::{ Arc, @@ -5317,6 +6409,24 @@ mod pools_tests { Arc::new(|_| Box::pin(async {})) } + fn decommission_worker_test_store(pool_meta: PoolMeta, cancelers: Vec>) -> Arc { + let ctx = Arc::new(InstanceContext::new()); + let endpoint_pools = EndpointServerPools::default(); + Arc::new(ECStore { + id: uuid::Uuid::new_v4(), + disk_map: std::collections::HashMap::new(), + pools: Vec::new(), + peer_sys: crate::cluster::rpc::S3PeerSys::new_with_instance_ctx(&endpoint_pools, ctx.clone()), + pool_meta: tokio::sync::RwLock::new(pool_meta), + rebalance_meta: tokio::sync::RwLock::new(None), + decommission_cancelers: tokio::sync::RwLock::new(cancelers), + start_gate: tokio::sync::Mutex::new(()), + pool_meta_save_gate: tokio::sync::Mutex::new(()), + ctx, + bucket_fence_registry: Arc::default(), + }) + } + fn decommission_test_pool_endpoint(idx: usize, is_local: bool) -> PoolEndpoints { let port = 9000usize + idx; let mut endpoint = @@ -5530,6 +6640,25 @@ mod pools_tests { assert_eq!(default_decommission_bucket_concurrency(8), 4); } + #[test] + fn test_default_decommission_entry_concurrency_is_conservative() { + assert_eq!(default_decommission_entry_concurrency(0), 1); + assert_eq!(default_decommission_entry_concurrency(1), 1); + assert_eq!(default_decommission_entry_concurrency(4), 4); + assert_eq!(default_decommission_entry_concurrency(16), DECOMMISSION_ENTRY_CONCURRENCY_DEFAULT_CAP); + } + + #[test] + fn test_decommission_entry_concurrency_clamps_operator_configuration() { + assert_eq!(clamp_decommission_entry_concurrency(0), 1); + assert_eq!(clamp_decommission_entry_concurrency(1), 1); + assert_eq!( + clamp_decommission_entry_concurrency(DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP), + DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP + ); + assert_eq!(clamp_decommission_entry_concurrency(usize::MAX), DECOMMISSION_ENTRY_CONCURRENCY_HARD_CAP); + } + #[test] fn test_split_decommission_buckets_keeps_meta_buckets_last() { let (regular, meta) = split_decommission_buckets(vec![ @@ -5701,6 +6830,190 @@ mod pools_tests { assert!(result.is_ok()); } + #[test] + fn test_decommission_entry_queue_capacity_is_bounded() { + assert_eq!(decommission_entry_queue_capacity(0), 1); + assert_eq!(decommission_entry_queue_capacity(1), 2); + assert_eq!( + decommission_entry_queue_capacity(DECOMMISSION_ENTRY_QUEUE_HARD_CAP), + DECOMMISSION_ENTRY_QUEUE_HARD_CAP + ); + assert_eq!(decommission_entry_queue_capacity(usize::MAX), DECOMMISSION_ENTRY_QUEUE_HARD_CAP); + } + + #[tokio::test] + async fn test_drain_decommission_entry_queue_waits_for_all_outstanding_entries() { + let outstanding = Arc::new(Semaphore::new(1)); + let held = outstanding + .clone() + .acquire_owned() + .await + .expect("test outstanding permit should acquire"); + let rx = CancellationToken::new(); + let drain = tokio::spawn({ + let outstanding = outstanding.clone(); + let rx = rx.clone(); + async move { drain_decommission_entry_queue(&rx, &outstanding, 1).await } + }); + + tokio::task::yield_now().await; + assert!(!drain.is_finished(), "queue drain must wait for active entry work"); + drop(held); + + let drained = tokio::time::timeout(StdDuration::from_secs(1), drain) + .await + .expect("queue drain should finish after entry completion") + .expect("queue drain task should not panic"); + assert!(!drained); + } + + #[tokio::test] + async fn test_enqueue_decommission_entry_observes_cancellation_when_queue_is_full() { + let outstanding = Arc::new(Semaphore::new(2)); + let (tx, mut queue) = tokio::sync::mpsc::channel(1); + let held = outstanding + .clone() + .acquire_owned() + .await + .expect("first queue permit should acquire"); + tx.send(QueuedDecommissionEntry { + entry: MetaCacheEntry::default(), + queue_permit: held, + }) + .await + .expect("first entry should fill the queue"); + + let rx = CancellationToken::new(); + let enqueue = tokio::spawn({ + let rx = rx.clone(); + let outstanding = outstanding.clone(); + let tx = tx.clone(); + async move { enqueue_decommission_entry(&rx, &outstanding, &tx, MetaCacheEntry::default()).await } + }); + + tokio::task::yield_now().await; + rx.cancel(); + let result = tokio::time::timeout(StdDuration::from_secs(1), enqueue) + .await + .expect("full queue enqueue should observe cancellation") + .expect("enqueue task should not panic"); + assert!(matches!(result, DecommissionEntryEnqueueResult::Canceled)); + drop(queue.recv().await); + } + + #[tokio::test] + async fn test_decommission_side_effect_gate_quiesces_before_transition() { + let operation_gate = Arc::new(tokio::sync::RwLock::new(())); + let rx = CancellationToken::new(); + let started = Arc::new(tokio::sync::Notify::new()); + let release = Arc::new(tokio::sync::Notify::new()); + let operation = tokio::spawn({ + let operation_gate = operation_gate.clone(); + let rx = rx.clone(); + let started = started.clone(); + let release = release.clone(); + async move { + run_decommission_side_effect(&rx, &operation_gate, || async { + started.notify_one(); + release.notified().await; + Ok::<_, Error>(()) + }) + .await + } + }); + + started.notified().await; + rx.cancel(); + let transition = tokio::spawn({ + let operation_gate = operation_gate.clone(); + async move { + let _guard = operation_gate.write().await; + } + }); + tokio::task::yield_now().await; + assert!(!transition.is_finished(), "transition must wait for the in-flight side effect"); + + release.notify_one(); + let operation_result = operation.await.expect("operation task should not panic"); + assert!(matches!(operation_result, Err(Error::OperationCanceled))); + transition.await.expect("transition task should not panic"); + + let called = Arc::new(AtomicBool::new(false)); + let result = run_decommission_side_effect(&rx, &operation_gate, { + let called = called.clone(); + move || async move { + called.store(true, Ordering::SeqCst); + Ok::<_, Error>(()) + } + }) + .await; + assert!(matches!(result, Err(Error::OperationCanceled))); + assert!(!called.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn test_decommission_transition_waits_without_registered_canceler() { + let store = decommission_worker_test_store(PoolMeta::default(), vec![None]); + let operation_gate = store.ctx.decommission_operation_gate(); + let operation_guard = operation_gate.read().await; + let transition = tokio::spawn({ + let store = store.clone(); + async move { store.cancel_decommission_routines_and_wait(&[0]).await } + }); + + tokio::task::yield_now().await; + assert!( + !transition.is_finished(), + "a transition must wait for an in-flight side effect even after its canceler slot is gone" + ); + + drop(operation_guard); + tokio::time::timeout(StdDuration::from_secs(1), transition) + .await + .expect("transition should finish after the side effect") + .expect("transition task should not panic"); + } + + #[tokio::test(start_paused = true)] + async fn test_run_decommission_listing_with_retry_drains_before_each_retry() { + let attempts = Arc::new(AtomicUsize::new(0)); + let drains = Arc::new(AtomicUsize::new(0)); + let err = run_decommission_listing_with_retry_and_drain( + CancellationToken::new(), + "bucket-a".to_string(), + noop_decommission_list_callback(), + 1, + 2, + 2, + { + let attempts = attempts.clone(); + move |_| { + let attempts = attempts.clone(); + async move { + attempts.fetch_add(1, Ordering::SeqCst); + Err(Error::SlowDown) + } + } + }, + { + let drains = drains.clone(); + move || { + let drains = drains.clone(); + async move { + drains.fetch_add(1, Ordering::SeqCst); + false + } + } + }, + ) + .await + .expect_err("permanent listing failure must be returned"); + + assert!(err.to_string().contains("attempt 2/2")); + assert_eq!(attempts.load(Ordering::SeqCst), 2); + assert_eq!(drains.load(Ordering::SeqCst), 2); + } + #[test] fn test_get_by_index_returns_value_when_in_range() { let values = vec!["a", "b", "c"]; @@ -6187,20 +7500,6 @@ mod pools_tests { assert!(message.contains(Error::SlowDown.to_string().as_str())); } - #[test] - fn test_resolve_decommission_spawn_failure_result_keeps_primary_without_rollback_error() { - let err = resolve_decommission_spawn_failure_result(Error::SlowDown, None); - assert!(matches!(err, Error::SlowDown)); - } - - #[test] - fn test_resolve_decommission_spawn_failure_result_wraps_rollback_error() { - let err = resolve_decommission_spawn_failure_result(Error::SlowDown, Some(Error::OperationCanceled)); - let message = err.to_string(); - assert!(message.contains("decommission spawn routines failed")); - assert!(message.contains("rollback failed")); - } - #[test] fn test_decommission_item_size_converts_positive_values() { assert_eq!(decommission_item_size(42_i64), 42); @@ -6321,6 +7620,65 @@ mod pools_tests { assert!(matches!(err, Error::SlowDown)); } + #[test] + fn test_resolve_decommission_partial_listing_entry_rejects_unresolved_metadata() { + let err = resolve_decommission_partial_listing_entry( + MetaCacheEntries(vec![None]), + MetadataResolutionParams { + dir_quorum: 2, + obj_quorum: 2, + bucket: "bucket-a".to_string(), + ..Default::default() + }, + "bucket-a", + "prefix/", + 1, + 2, + 3, + ) + .expect_err("unresolved partial listing must fail closed"); + + let message = err.to_string(); + assert!(message.contains("decommission listing could not resolve metadata")); + assert!(message.contains("bucket-a/prefix/")); + assert!(message.contains("pool 2 set 3")); + assert!(message.contains("1 disk error(s)")); + } + + #[tokio::test] + async fn test_record_decommission_entry_error_cancels_listing_and_preserves_first_error() { + let entry_error = Arc::new(tokio::sync::Mutex::new(None)); + let rx = CancellationToken::new(); + + record_decommission_entry_error(&entry_error, &rx, Error::SlowDown).await; + record_decommission_entry_error(&entry_error, &rx, Error::OperationCanceled).await; + + assert!(rx.is_cancelled()); + assert!(matches!(*entry_error.lock().await, Some(Error::SlowDown))); + } + + #[tokio::test] + async fn test_record_decommission_entry_error_ignores_already_canceled_listing() { + let entry_error = Arc::new(tokio::sync::Mutex::new(None)); + let rx = CancellationToken::new(); + rx.cancel(); + + record_decommission_entry_error(&entry_error, &rx, Error::SlowDown).await; + + assert!(entry_error.lock().await.is_none()); + } + + #[test] + fn test_resolve_decommission_listing_error_preserves_real_listing_failure() { + let err = resolve_decommission_listing_error(Some(Error::SlowDown), Some(Error::OperationCanceled)) + .expect("listing failure should be returned"); + assert!(matches!(err, Error::SlowDown)); + + let err = resolve_decommission_listing_error(Some(Error::OperationCanceled), Some(Error::SlowDown)) + .expect("entry failure should be returned"); + assert!(matches!(err, Error::SlowDown)); + } + #[test] fn test_resolve_decommission_check_after_list_result_returns_list_result_without_entry_error() { let err = resolve_decommission_check_after_list_result(Err(Error::OperationCanceled), None) @@ -6538,7 +7896,7 @@ mod pools_tests { } #[test] - fn test_touch_decommission_progress_updates_last_update_and_save_baseline() { + fn test_track_decommission_stage_does_not_advance_checkpoint_state() { let mut meta = PoolMeta { pools: vec![PoolStatus { id: 0, @@ -6553,11 +7911,13 @@ mod pools_tests { ..Default::default() }; - touch_decommission_progress(&mut meta, 0).expect("valid decommission progress should be touched"); + track_decommission_current_object_stage(&mut meta, 0, "bucket", "object", "migrate_object") + .expect("valid decommission progress should be tracked"); - assert!(meta.pools[0].last_update > OffsetDateTime::UNIX_EPOCH); + assert_eq!(meta.pools[0].last_update, OffsetDateTime::UNIX_EPOCH); let info = meta.pools[0].decommission.as_ref().expect("decommission info should exist"); - assert_eq!(info.items_since_last_progress_save(), 0); + assert_eq!(info.items_since_last_progress_save(), 5); + assert_eq!(info.stage, "migrate_object"); } #[test] @@ -6632,6 +7992,134 @@ mod pools_tests { assert_eq!(info.items_since_last_progress_save(), 1); } + #[test] + fn test_pool_meta_update_after_does_not_advance_last_update_before_save() { + let last_update = OffsetDateTime::UNIX_EPOCH; + let mut meta = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update, + decommission: Some(PoolDecommissionInfo { + start_time: Some(last_update), + items_decommissioned: DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, + ..Default::default() + }), + }], + ..Default::default() + }; + + assert!( + meta.update_after(0, DECOMMISSION_PROGRESS_SAVE_INTERVAL) + .expect("item threshold should request a checkpoint") + ); + assert_eq!(meta.pools[0].last_update, last_update); + } + + #[test] + fn test_decommission_progress_checkpoint_commits_exact_snapshot_watermark() { + let start_time = OffsetDateTime::UNIX_EPOCH; + let checkpoint_at = start_time + Duration::seconds(30); + let mut meta = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: start_time, + decommission: Some(PoolDecommissionInfo { + start_time: Some(start_time), + items_decommissioned: DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, + ..Default::default() + }), + }], + ..Default::default() + }; + + let checkpoint = meta + .decommission_progress_checkpoint(0, DECOMMISSION_PROGRESS_SAVE_INTERVAL, checkpoint_at) + .expect("valid decommission state should produce a checkpoint") + .expect("item threshold should produce a checkpoint"); + meta.count_item(0, 1, false); + + assert!(meta.commit_decommission_progress_checkpoint(0, checkpoint)); + let info = meta.pools[0].decommission.as_ref().expect("decommission info should exist"); + assert_eq!(info.progress_save_item_baseline, checkpoint.counted_items); + assert_eq!(info.items_since_last_progress_save(), 1); + assert_eq!(meta.pools[0].last_update, checkpoint_at); + } + + #[test] + fn test_decommission_progress_checkpoint_backoff_does_not_advance_baseline() { + let start_time = OffsetDateTime::UNIX_EPOCH; + let checkpoint_at = start_time + Duration::seconds(30); + let retry_after = checkpoint_at + DECOMMISSION_PROGRESS_SAVE_RETRY_BACKOFF; + let mut meta = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: start_time, + decommission: Some(PoolDecommissionInfo { + start_time: Some(start_time), + items_decommissioned: DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD, + ..Default::default() + }), + }], + ..Default::default() + }; + + let checkpoint = meta + .decommission_progress_checkpoint(0, DECOMMISSION_PROGRESS_SAVE_INTERVAL, checkpoint_at) + .expect("valid decommission state should produce a checkpoint") + .expect("item threshold should produce a checkpoint"); + meta.defer_decommission_progress_checkpoint(0, checkpoint, retry_after); + + assert!( + meta.decommission_progress_checkpoint(0, DECOMMISSION_PROGRESS_SAVE_INTERVAL, checkpoint_at) + .expect("retry backoff check should succeed") + .is_none() + ); + assert_eq!(meta.pools[0].last_update, start_time); + assert_eq!( + meta.pools[0] + .decommission + .as_ref() + .expect("decommission info should exist") + .progress_save_item_baseline, + 0 + ); + } + + #[test] + fn test_decommission_progress_checkpoint_count_scales_with_threshold() { + let start_time = OffsetDateTime::UNIX_EPOCH; + let checkpoint_at = start_time; + let mut meta = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: start_time, + decommission: Some(PoolDecommissionInfo { + start_time: Some(start_time), + ..Default::default() + }), + }], + ..Default::default() + }; + let mut checkpoint_count = 0; + + for _ in 0..(DECOMMISSION_PROGRESS_SAVE_ITEM_THRESHOLD * 10) { + meta.count_item(0, 1, false); + if let Some(checkpoint) = meta + .decommission_progress_checkpoint(0, DECOMMISSION_PROGRESS_SAVE_INTERVAL, checkpoint_at) + .expect("valid decommission state should produce a checkpoint") + { + checkpoint_count += 1; + assert!(meta.commit_decommission_progress_checkpoint(0, checkpoint)); + } + } + + assert_eq!(checkpoint_count, 10); + } + #[test] fn test_ensure_decommission_not_rebalancing_rejects_running_rebalance() { let err = ensure_decommission_not_rebalancing(true).expect_err("rebalance running should be rejected"); @@ -6753,6 +8241,43 @@ mod pools_tests { assert!(!is_decommission_active(false, false, true)); } + #[test] + fn test_ensure_decommission_generation_rejects_stale_or_queued_workers() { + let generation = OffsetDateTime::UNIX_EPOCH; + let mut meta = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: generation, + decommission: Some(PoolDecommissionInfo { + start_time: Some(generation), + ..Default::default() + }), + }], + ..Default::default() + }; + + assert!(ensure_decommission_generation(&meta, 0, generation).is_ok()); + assert!(ensure_decommission_generation(&meta, 0, generation + Duration::seconds(1)).is_err()); + + meta.pools[0] + .decommission + .as_mut() + .expect("decommission metadata should exist") + .queued = true; + assert!(ensure_decommission_generation(&meta, 0, generation).is_err()); + + let replacement_generation = generation + Duration::seconds(2); + let info = meta.pools[0] + .decommission + .as_mut() + .expect("decommission metadata should exist"); + info.queued = false; + info.start_time = Some(replacement_generation); + assert!(ensure_decommission_generation(&meta, 0, generation).is_err()); + assert!(ensure_decommission_generation(&meta, 0, replacement_generation).is_ok()); + } + #[test] fn test_pool_meta_has_active_decommission_counts_running_and_queued_states() { let active_meta = PoolMeta { @@ -7742,7 +9267,7 @@ mod pools_tests { #[test] fn test_bind_decommission_cancelers_replaces_existing_slot() { let parent = CancellationToken::new(); - let existing = CancellationToken::new(); + let existing = DecommissionCanceler::new(CancellationToken::new()); let mut cancelers = vec![Some(existing.clone())]; let bound = bind_decommission_cancelers(&[0], &parent, cancelers.as_mut_slice()); @@ -7759,7 +9284,7 @@ mod pools_tests { #[test] fn test_bind_missing_decommission_cancelers_stops_at_existing_slot() { let parent = CancellationToken::new(); - let existing = CancellationToken::new(); + let existing = DecommissionCanceler::new(CancellationToken::new()); let mut cancelers = vec![None, Some(existing.clone()), None]; let bound = bind_missing_decommission_cancelers(&[0, 1, 2], &parent, cancelers.as_mut_slice()); @@ -7772,6 +9297,38 @@ mod pools_tests { assert!(!existing.is_cancelled()); } + #[test] + fn test_serialized_decommission_double_start_preserves_first_operation() { + let mut pool_meta = PoolMeta { + pools: vec![decommission_test_pool_status(0, None), decommission_test_pool_status(1, None)], + ..Default::default() + }; + let first_parent = CancellationToken::new(); + let second_parent = CancellationToken::new(); + let mut cancelers = vec![None, None]; + + let first = reserve_decommission_start_cancelers(&pool_meta, &[0], &[0], &first_parent, cancelers.as_mut_slice()) + .expect("first start should reserve its worker"); + pool_meta + .decommission( + 0, + PoolSpaceInfo { + total: 100, + free: 40, + used: 60, + }, + ) + .expect("first start should install active metadata"); + + let second = reserve_decommission_start_cancelers(&pool_meta, &[0], &[0], &second_parent, cancelers.as_mut_slice()); + + assert!(matches!(second, Err(Error::DecommissionAlreadyRunning))); + let current = cancelers[0].as_ref().expect("first operation should retain the slot"); + assert!(current.owns_same_operation(first[0].1.canceler())); + assert!(current.is_active()); + assert!(!first_parent.is_cancelled()); + } + #[test] fn test_local_decommission_queue_prefix_stops_at_remote_leader() { let endpoints = EndpointServerPools::from(vec![ @@ -7822,7 +9379,7 @@ mod pools_tests { #[test] fn test_missing_decommission_worker_prefix_stops_at_active_worker() { - let cancelers = vec![None, Some(CancellationToken::new()), None]; + let cancelers = vec![None, Some(DecommissionCanceler::new(CancellationToken::new())), None]; let missing = missing_decommission_worker_prefix(&[0, 1, 2], cancelers.as_slice()); @@ -7931,8 +9488,8 @@ mod pools_tests { #[test] fn test_take_decommission_canceler_takes_and_clears_slot() { - let token = CancellationToken::new(); - let mut cancelers = vec![Some(token)]; + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let mut cancelers = vec![Some(canceler)]; let taken = take_decommission_canceler(cancelers.as_mut_slice(), 0); assert!(taken.is_some()); @@ -7941,13 +9498,13 @@ mod pools_tests { #[test] fn test_take_decommission_canceler_returns_none_for_missing_slot() { - let mut cancelers: Vec> = Vec::new(); + let mut cancelers: Vec> = Vec::new(); assert!(take_decommission_canceler(cancelers.as_mut_slice(), 0).is_none()); } #[test] fn test_has_active_decommission_canceler_true_when_any_slot_present() { - let cancelers = vec![None, Some(CancellationToken::new())]; + let cancelers = vec![None, Some(DecommissionCanceler::new(CancellationToken::new()))]; assert!(has_active_decommission_canceler(cancelers.as_slice())); } @@ -7959,11 +9516,12 @@ mod pools_tests { #[test] fn test_cancel_decommission_canceler_cancels_when_present() { - let token = CancellationToken::new(); - let canceled = cancel_decommission_canceler(Some(token.clone())); + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let canceled = cancel_decommission_canceler(Some(canceler.clone())); assert!(canceled); - assert!(token.is_cancelled()); + assert!(canceler.is_cancelled()); + assert!(!canceler.is_active()); } #[test] @@ -7973,12 +9531,13 @@ mod pools_tests { #[test] fn test_take_and_cancel_decommission_canceler_clears_slot() { - let token = CancellationToken::new(); - let mut cancelers = vec![Some(token.clone())]; + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let mut cancelers = vec![Some(canceler.clone())]; assert!(take_and_cancel_decommission_canceler(cancelers.as_mut_slice(), 0)); assert!(cancelers[0].is_none()); - assert!(token.is_cancelled()); + assert!(canceler.is_cancelled()); + assert!(!canceler.is_active()); } #[test] @@ -7989,6 +9548,195 @@ mod pools_tests { assert!(cancelers[0].is_none()); } + #[test] + fn test_guarded_decommission_future_releases_without_first_poll() { + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let cancelers = vec![Some(canceler.clone())]; + let guards = guard_decommission_cancelers(vec![(0, canceler.clone())]); + let unpolled = async move { + let _guards = guards; + std::future::pending::<()>().await; + }; + + drop(unpolled); + + assert!(canceler.is_cancelled()); + assert!(!has_active_decommission_canceler(cancelers.as_slice())); + } + + #[test] + fn test_partial_decommission_spawn_reservation_releases_bound_slot() { + let parent = CancellationToken::new(); + let mut cancelers = vec![None]; + let bound = bind_decommission_cancelers(&[0, 1], &parent, cancelers.as_mut_slice()); + let guards = guard_decommission_cancelers(bound); + + let result = super::ensure_decommission_routines_scheduled(guards.len(), 2); + drop(guards); + + assert!(result.is_err()); + assert!(!has_active_decommission_canceler(cancelers.as_slice())); + } + + #[tokio::test] + async fn test_decommission_supervisor_observes_worker_abort() { + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let worker = tokio::spawn(async move { + started_tx.send(()).expect("worker start should be observed"); + std::future::pending::<()>().await; + #[allow(unreachable_code)] + Ok(()) + }); + + started_rx.await.expect("worker start should be observed"); + worker.abort(); + let err = await_decommission_worker(3, worker) + .await + .expect_err("supervisor should observe aborted worker"); + + assert!(err.to_string().contains("decommission worker 3 task join error")); + } + + #[tokio::test] + async fn test_decommission_supervisor_observes_worker_panic() { + let worker = tokio::spawn(async move { + panic!("injected decommission worker panic"); + #[allow(unreachable_code)] + Ok(()) + }); + + let err = await_decommission_worker(4, worker) + .await + .expect_err("supervisor should observe panicked worker"); + + assert!(err.to_string().contains("decommission worker 4 task join error")); + } + + #[tokio::test] + async fn test_decommission_worker_metadata_missing_releases_owned_slot() { + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let store = decommission_worker_test_store(PoolMeta::default(), vec![Some(canceler.clone())]); + canceler.cancel(); + + let err = store + .do_decommission_in_routine(canceler.clone(), 0, Arc::new(Semaphore::new(1))) + .await + .expect_err("missing worker metadata should fail the routine"); + + assert!(err.to_string().contains("target pool was not found")); + assert!(!canceler.is_active()); + assert!(store.decommission_cancelers.read().await[0].is_none()); + } + + #[tokio::test] + async fn test_decommission_supervisor_failure_cancels_queued_successor() { + let first = DecommissionCanceler::new(CancellationToken::new()); + let queued = DecommissionCanceler::new(CancellationToken::new()); + let store = decommission_worker_test_store(PoolMeta::default(), vec![Some(first.clone()), Some(queued.clone())]); + let guards = guard_decommission_cancelers(vec![(0, first.clone()), (1, queued.clone())]); + + spawn_decommission_index_cancelers(store.clone(), CancellationToken::new(), guards, Arc::new(Semaphore::new(1))) + .await + .expect("decommission supervisor should finish after queued cleanup"); + + assert!(!first.is_active()); + assert!(!queued.is_active()); + assert!(queued.is_cancelled()); + assert!(store.decommission_cancelers.read().await.iter().all(Option::is_none)); + } + + #[tokio::test] + async fn test_decommission_failed_save_failure_preserves_owner_until_retry_succeeds() { + let canceler = DecommissionCanceler::new(CancellationToken::new()); + let pool_meta = PoolMeta { + pools: vec![decommission_test_pool_status( + 0, + Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }), + )], + ..Default::default() + }; + let store = decommission_worker_test_store(pool_meta, vec![Some(canceler.clone())]); + + store + .decommission_failed_with_owner_and_save(0, Some(&canceler), async { Err(Error::SlowDown) }) + .await + .expect_err("injected terminal save failure should be returned"); + + { + let cancelers = store.decommission_cancelers.read().await; + let current = cancelers[0].as_ref().expect("failed save must retain the exact owner slot"); + assert!(current.owns_same_operation(&canceler)); + assert!(current.is_active()); + } + { + let pool_meta = store.pool_meta.read().await; + let info = pool_meta.pools[0] + .decommission + .as_ref() + .expect("rollback must retain active decommission metadata"); + assert!(info.has_decommission_state()); + assert!(!info.failed); + assert!(!info.complete); + assert!(!info.canceled); + } + assert!(store.decommission_terminal_retryable_for_operation(0, &canceler).await); + + store + .decommission_failed_with_owner_and_save(0, Some(&canceler), async { Ok(()) }) + .await + .expect("terminal retry should commit"); + + let pool_meta = store.pool_meta.read().await; + assert!( + pool_meta.pools[0] + .decommission + .as_ref() + .expect("terminal metadata should remain") + .failed + ); + drop(pool_meta); + assert!(store.decommission_cancelers.read().await[0].is_none()); + assert!(!canceler.is_active()); + assert!(canceler.is_cancelled()); + } + + #[test] + fn test_stale_decommission_operation_cannot_cancel_replacement() { + let stale = DecommissionCanceler::new(CancellationToken::new()); + let replacement = DecommissionCanceler::new(CancellationToken::new()); + let cancelers = vec![Some(replacement.clone())]; + let mut pool_meta = PoolMeta { + pools: vec![decommission_test_pool_status( + 0, + Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }), + )], + ..Default::default() + }; + + let changed = update_decommission_for_operation(cancelers.as_slice(), &mut pool_meta, 0, Some(&stale), |pool_meta| { + pool_meta.decommission_cancel(0) + }); + + assert!(changed.is_none()); + assert!( + !pool_meta.pools[0] + .decommission + .as_ref() + .expect("replacement metadata should remain") + .canceled + ); + assert!(replacement.is_active()); + assert!(!replacement.is_cancelled()); + assert!(!stale.is_active()); + assert!(stale.is_cancelled()); + } + #[test] fn test_ensure_decommission_routines_scheduled_accepts_positive_bound_count() { assert!(super::ensure_decommission_routines_scheduled(2, 2).is_ok()); diff --git a/crates/ecstore/src/core/sets.rs b/crates/ecstore/src/core/sets.rs index 1d1bcedeb..0c79c080f 100644 --- a/crates/ecstore/src/core/sets.rs +++ b/crates/ecstore/src/core/sets.rs @@ -21,7 +21,7 @@ use crate::storage_api_contracts::{ bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions}, list::{StorageListObjectVersionsInfo, StorageListObjectsV2Info, StorageObjectInfoOrErr, StorageWalkOptions}, multipart::{CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartUploadResult, PartInfo}, - object::{DeletedObject, ObjectIO as _, ObjectOperations as _, ObjectToDelete}, + object::{DeleteAccounting, DeletedObject, ObjectIO as _, ObjectOperations as _, ObjectToDelete}, range::HTTPRangeSpec, }; use crate::{ @@ -249,7 +249,7 @@ impl Sets { self.connect_disks().await; - // TODO: config interval + // TODO(backlog): make monitor_and_connect interval configurable instead of hardcoded 15s let mut interval = tokio::time::interval(Duration::from_secs(15)); loop { tokio::select! { @@ -414,6 +414,66 @@ fn apply_delete_objects_results( } } +fn apply_delete_accounting_results( + accounting: &mut [Option], + set_objects: &[DelObj], + set_accounting: &[Option], +) { + for (obj, value) in set_objects.iter().zip(set_accounting.iter()) { + accounting[obj.orig_idx] = value.clone(); + } +} + +impl Sets { + pub(crate) async fn delete_objects_with_accounting( + &self, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + ) -> (Vec, Vec>, Vec>) { + let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut del_errs = vec![None; objects.len()]; + let mut accounting = vec![None; objects.len()]; + let mut set_obj_map = HashMap::new(); + + for (i, obj) in objects.iter().enumerate() { + let idx = self.get_hashed_set_index(obj.object_name.as_str()); + set_obj_map.entry(idx).or_insert_with(Vec::new).push(DelObj { + orig_idx: i, + obj: obj.clone(), + }); + } + + let max_concurrent = set_obj_map.len().min(num_cpus::get()).max(1); + let semaphore = Arc::new(tokio::sync::Semaphore::new(max_concurrent)); + let mut futures = FuturesUnordered::new(); + let bucket = bucket.to_owned(); + + for (set_index, set_objects) in set_obj_map { + let disks = self.get_disks(set_index); + let objects = set_objects.iter().map(|entry| entry.obj.clone()).collect::>(); + let bucket = bucket.clone(); + let opts = opts.clone(); + let semaphore = semaphore.clone(); + futures.push(async move { + let _permit = semaphore + .acquire_owned() + .await + .expect("delete_objects semaphore should remain open"); + let (deleted, errors, accounting) = disks.delete_objects_with_accounting(&bucket, objects, opts).await; + (set_objects, deleted, errors, accounting) + }); + } + + while let Some((set_objects, deleted, errors, set_accounting)) = futures.next().await { + apply_delete_objects_results(&mut del_objects, &mut del_errs, &set_objects, &deleted, errors); + apply_delete_accounting_results(&mut accounting, &set_objects, &set_accounting); + } + + (del_objects, del_errs, accounting) + } +} + #[async_trait::async_trait] impl crate::storage_api_contracts::object::ObjectIO for Sets { type Error = Error; @@ -655,65 +715,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for Sets { objects: Vec, opts: ObjectOptions, ) -> (Vec, Vec>) { - // Default return value - let mut del_objects = vec![DeletedObject::default(); objects.len()]; - - let mut del_errs = Vec::with_capacity(objects.len()); - for _ in 0..objects.len() { - del_errs.push(None) - } - - let mut set_obj_map = HashMap::new(); - - // hash key - for (i, obj) in objects.iter().enumerate() { - let idx = self.get_hashed_set_index(obj.object_name.as_str()); - - if !set_obj_map.contains_key(&idx) { - set_obj_map.insert( - idx, - vec![DelObj { - // set_idx: idx, - orig_idx: i, - obj: obj.clone(), - }], - ); - } else if let Some(val) = set_obj_map.get_mut(&idx) { - val.push(DelObj { - // set_idx: idx, - orig_idx: i, - obj: obj.clone(), - }); - } - } - - let max_concurrent = set_obj_map.len().min(num_cpus::get()).max(1); - let semaphore = Arc::new(tokio::sync::Semaphore::new(max_concurrent)); - let mut futures = FuturesUnordered::new(); - let bucket = bucket.to_string(); - - for (k, v) in set_obj_map { - let disks = self.get_disks(k); - let objs: Vec = v.iter().map(|v| v.obj.clone()).collect(); - let bucket = bucket.clone(); - let opts = opts.clone(); - let semaphore = semaphore.clone(); - - futures.push(async move { - let _permit = semaphore - .acquire_owned() - .await - .expect("delete_objects semaphore should remain open"); - let (dobjects, errs) = disks.delete_objects(&bucket, objs, opts).await; - (v, dobjects, errs) - }); - } - - while let Some((v, dobjects, errs)) = futures.next().await { - apply_delete_objects_results(&mut del_objects, &mut del_errs, &v, &dobjects, errs); - } - - (del_objects, del_errs) + let (deleted, errors, _) = self.delete_objects_with_accounting(bucket, objects, opts).await; + (deleted, errors) } #[tracing::instrument(skip(self))] @@ -985,14 +988,11 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for Sets { } } -#[async_trait::async_trait] -impl crate::storage_api_contracts::heal::HealOperations for Sets { - type Error = Error; - type HealResultItem = HealResultItem; - type HealOptions = HealOpts; - - #[tracing::instrument(skip(self))] - async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option)> { +impl Sets { + pub(crate) async fn heal_format_with_fence(&self, dry_run: bool, fence_lost: F) -> Result<(HealResultItem, Option)> + where + F: Fn() -> bool + Send + Sync, + { let (disks, init_errs) = init_storage_disks_with_errors( &self.endpoints.endpoints, &DiskOption { @@ -1065,6 +1065,9 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets { // Save new formats `format.json` on unformatted disks. for (index, (fm, disk)) in tmp_new_formats.iter_mut().zip(disks.iter()).enumerate() { if fm.is_some() && disk.is_some() { + if fence_lost() { + return Ok((res, Some(StorageError::SlowDown))); + } if let Err(err) = save_format_file(disk, fm).await { if let Some(disk) = disk.as_ref() { let _ = disk.close().await; @@ -1098,6 +1101,18 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets { } Ok((res, None)) } +} + +#[async_trait::async_trait] +impl crate::storage_api_contracts::heal::HealOperations for Sets { + type Error = Error; + type HealResultItem = HealResultItem; + type HealOptions = HealOpts; + + #[tracing::instrument(skip(self))] + async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option)> { + self.heal_format_with_fence(dry_run, || false).await + } #[tracing::instrument(skip(self))] async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result { let mut result = HealResultItem { diff --git a/crates/ecstore/src/data_movement/mod.rs b/crates/ecstore/src/data_movement/mod.rs index 1f4daa703..4fdda942e 100644 --- a/crates/ecstore/src/data_movement/mod.rs +++ b/crates/ecstore/src/data_movement/mod.rs @@ -26,7 +26,7 @@ use crate::storage_api_contracts::{ namespace::NamespaceLocking as _, object::{HTTPPreconditions, ObjectOperations as _}, }; -use crate::store::ECStore; +use crate::store::{ECStore, ObjectLockDiagGuard, SourceCleanupMutationFence}; use bytes::Bytes; use rustfs_filemeta::{FileInfo, FileInfoVersions, ObjectPartInfo}; use rustfs_rio::{EtagResolvable, HashReader, HashReaderDetector, Index, TryGetIndex}; @@ -856,7 +856,6 @@ fn is_equivalent_data_movement_object(source: &ObjectInfo, target: &ObjectInfo) fn is_superseding_unversioned_data_movement_object(source: &ObjectInfo, target: &ObjectInfo) -> bool { is_unversioned_data_movement_object(source) && is_unversioned_data_movement_object(target) - && !target.delete_marker && source .mod_time .zip(target.mod_time) @@ -1028,6 +1027,7 @@ pub(crate) enum SourceCleanupError { pub(crate) struct SourceCleanupBucketFence<'a> { pub(crate) expected_incarnation_id: Option, pub(crate) lifecycle_guard: Option<&'a rustfs_lock::NamespaceLockGuard>, + pub(crate) object_mutation_fence: Option<&'a SourceCleanupMutationFence>, } fn ensure_source_cleanup_versions_match( @@ -1065,7 +1065,9 @@ pub(crate) async fn ensure_source_cleanup_versions_unchanged( struct SourceCleanupDeleteBarrierState { bucket: String, object: String, + fence_pending: tokio::sync::Notify, arrived: tokio::sync::Notify, + is_paused: AtomicBool, release: tokio::sync::Notify, } @@ -1079,7 +1081,7 @@ pub(crate) struct SourceCleanupDeleteBarrier { } #[cfg(test)] -static SOURCE_CLEANUP_DELETE_BARRIER: std::sync::OnceLock>>> = +static SOURCE_CLEANUP_DELETE_BARRIERS: std::sync::OnceLock>>> = std::sync::OnceLock::new(); #[cfg(test)] @@ -1092,15 +1094,22 @@ impl SourceCleanupDeleteBarrier { let state = Arc::new(SourceCleanupDeleteBarrierState { bucket: bucket.to_string(), object: object.to_string(), + fence_pending: tokio::sync::Notify::new(), arrived: tokio::sync::Notify::new(), + is_paused: AtomicBool::new(false), release: tokio::sync::Notify::new(), }); - let mut slot = SOURCE_CLEANUP_DELETE_BARRIER - .get_or_init(|| std::sync::Mutex::new(None)) + let mut barriers = SOURCE_CLEANUP_DELETE_BARRIERS + .get_or_init(|| std::sync::Mutex::new(Vec::new())) .lock() .expect("source cleanup delete barrier mutex should not poison"); - assert!(slot.is_none(), "source cleanup delete barrier must be unique"); - *slot = Some(Arc::clone(&state)); + assert!( + !barriers + .iter() + .any(|barrier| barrier.bucket == bucket && barrier.object == object), + "source cleanup delete barrier must be unique per object" + ); + barriers.push(Arc::clone(&state)); Self { state } } @@ -1110,35 +1119,58 @@ impl SourceCleanupDeleteBarrier { .expect("source cleanup should reach the pre-delete barrier"); } + pub(crate) async fn wait_until_fence_pending(&self) { + tokio::time::timeout(StdDuration::from_secs(30), self.state.fence_pending.notified()) + .await + .expect("source cleanup should attempt the fixed mutation fence"); + } + + pub(crate) fn is_paused(&self) -> bool { + self.state.is_paused.load(Ordering::Acquire) + } + pub(crate) fn release(&self) { self.state.release.notify_one(); } } +#[cfg(test)] +pub(crate) fn notify_source_cleanup_mutation_fence_pending(bucket: &str, object: &str) { + let barrier = SOURCE_CLEANUP_DELETE_BARRIERS + .get_or_init(|| std::sync::Mutex::new(Vec::new())) + .lock() + .expect("source cleanup delete barrier mutex should not poison") + .iter() + .find(|barrier| barrier.bucket == bucket && barrier.object == object) + .cloned(); + if let Some(barrier) = barrier { + barrier.fence_pending.notify_one(); + } +} + #[cfg(test)] impl Drop for SourceCleanupDeleteBarrier { fn drop(&mut self) { self.state.release.notify_one(); - let mut slot = SOURCE_CLEANUP_DELETE_BARRIER - .get_or_init(|| std::sync::Mutex::new(None)) + let mut barriers = SOURCE_CLEANUP_DELETE_BARRIERS + .get_or_init(|| std::sync::Mutex::new(Vec::new())) .lock() .expect("source cleanup delete barrier mutex should not poison"); - if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) { - *slot = None; - } + barriers.retain(|state| !Arc::ptr_eq(state, &self.state)); } } #[cfg(test)] async fn pause_source_cleanup_before_delete(bucket: &str, object: &str) { - let barrier = SOURCE_CLEANUP_DELETE_BARRIER - .get_or_init(|| std::sync::Mutex::new(None)) + let barrier = SOURCE_CLEANUP_DELETE_BARRIERS + .get_or_init(|| std::sync::Mutex::new(Vec::new())) .lock() .expect("source cleanup delete barrier mutex should not poison") - .as_ref() - .filter(|barrier| barrier.bucket == bucket && barrier.object == object) + .iter() + .find(|barrier| barrier.bucket == bucket && barrier.object == object) .cloned(); if let Some(barrier) = barrier { + barrier.is_paused.store(true, Ordering::Release); barrier.arrived.notify_one(); barrier.release.notified().await; } @@ -1154,11 +1186,20 @@ pub(crate) async fn cleanup_source_entry_if_unchanged( op_label: &str, ) -> std::result::Result { let cleanup_key = encode_dir_object(object); - let ns_lock = set.new_ns_lock(bucket, cleanup_key.as_str()).await?; - let _guard = ns_lock - .get_write_lock(get_lock_acquire_timeout()) - .await - .map_err(Error::from)?; + let source_guard = if bucket_fence + .object_mutation_fence + .is_some_and(SourceCleanupMutationFence::source_lock_covered) + { + None + } else { + let ns_lock = set.new_ns_lock(bucket, cleanup_key.as_str()).await?; + Some( + ns_lock + .get_write_lock(get_lock_acquire_timeout()) + .await + .map_err(Error::from)?, + ) + }; if bucket_fence .lifecycle_guard @@ -1168,6 +1209,14 @@ pub(crate) async fn cleanup_source_entry_if_unchanged( "{op_label}: bucket incarnation fence was lost before source cleanup" )))); } + if bucket_fence + .object_mutation_fence + .is_some_and(SourceCleanupMutationFence::is_lock_lost) + { + return Err(SourceCleanupError::Storage(Error::other(format!( + "{op_label}: object mutation fence was lost before source cleanup" + )))); + } ensure_source_cleanup_versions_unchanged(set.clone(), bucket, object, expected, allowed_missing, op_label).await?; @@ -1182,7 +1231,12 @@ pub(crate) async fn cleanup_source_entry_if_unchanged( expected_bucket_incarnation_id: bucket_fence.expected_incarnation_id, ..Default::default() }; - opts.add_namespace_lock_guard(&_guard); + if let Some(source_guard) = source_guard.as_ref() { + opts.add_namespace_lock_guard(source_guard); + } + if let Some(object_mutation_fence) = bucket_fence.object_mutation_fence { + object_mutation_fence.add_namespace_lock_fence(&mut opts); + } if let Some(bucket_lifecycle_guard) = bucket_fence.lifecycle_guard { opts.add_bucket_lifecycle_lock_guard(bucket_lifecycle_guard); } @@ -1330,6 +1384,37 @@ fn data_movement_part_upload_failure_stage(err: &Error) -> &'static str { } } +pub(crate) async fn migrate_decommission_object( + store: Arc, + pool_idx: usize, + bucket: String, + rd: GetObjectReader, + source_bucket_incarnation_id: Option, + op_label: &str, +) -> Result<()> { + let source = rd.object_info.clone(); + let _mutation_fence = store + .acquire_decommission_object_mutation_fence(&bucket, &source.name) + .await?; + let current = find_data_movement_target_info(store.as_ref(), pool_idx, &bucket, &source) + .await? + .ok_or(Error::FileNotFound)?; + if !is_equivalent_data_movement_object_identity(&source, ¤t, true, false) { + return Err(Error::FileNotFound); + } + + migrate_object_inner( + store, + pool_idx, + bucket, + rd, + source_bucket_incarnation_id, + op_label, + Some(&_mutation_fence), + ) + .await +} + pub(crate) async fn migrate_object( store: Arc, pool_idx: usize, @@ -1337,6 +1422,18 @@ pub(crate) async fn migrate_object( rd: GetObjectReader, source_bucket_incarnation_id: Option, op_label: &str, +) -> Result<()> { + migrate_object_inner(store, pool_idx, bucket, rd, source_bucket_incarnation_id, op_label, None).await +} + +async fn migrate_object_inner( + store: Arc, + pool_idx: usize, + bucket: String, + rd: GetObjectReader, + source_bucket_incarnation_id: Option, + op_label: &str, + mutation_fence: Option<&ObjectLockDiagGuard>, ) -> Result<()> { let object_info = rd.object_info.clone(); let has_part_checksums = object_info @@ -1350,7 +1447,7 @@ pub(crate) async fn migrate_object( let mut new_multipart_opts = data_movement_new_multipart_opts(&object_info, pool_idx); new_multipart_opts.expected_bucket_incarnation_id = source_bucket_incarnation_id; let (res, target_pool_idx, expected_bucket_incarnation_id) = match store - .handle_new_multipart_upload_with_pool_idx(&bucket, &object_info.name, &new_multipart_opts) + .handle_new_multipart_upload_with_pool_idx(&bucket, &object_info.name, &new_multipart_opts, mutation_fence) .await { Ok(res) => res, @@ -1448,7 +1545,7 @@ pub(crate) async fn migrate_object( if let Err(err) = store .clone() .complete_multipart_upload_for_data_movement( - target_pool_idx, + (target_pool_idx, mutation_fence), &bucket, &object_info.name, &res.upload_id, @@ -1609,7 +1706,7 @@ pub(crate) async fn migrate_object( let mut put_opts = data_movement_put_object_opts(&object_info, pool_idx); put_opts.expected_bucket_incarnation_id = source_bucket_incarnation_id; let (target_pool_idx, put_result) = store - .put_object_for_data_movement(&bucket, &object_info.name, &mut data, &put_opts) + .put_object_for_data_movement(&bucket, &object_info.name, &mut data, &put_opts, mutation_fence) .await .map_err(|err| data_movement_stage_error(op_label, "prepare_put_object", &bucket, &object_info.name, err))?; if let Err(err) = put_result { @@ -3541,25 +3638,47 @@ mod tests { } #[test] - fn test_precondition_conflict_rejects_newer_delete_marker() { - let source = ObjectInfo { - size: 128, - etag: Some("etag-source".to_string()), - mod_time: Some(OffsetDateTime::UNIX_EPOCH), - ..Default::default() - }; - let target = ObjectInfo { - delete_marker: true, - etag: None, - mod_time: OffsetDateTime::UNIX_EPOCH.checked_add(time::Duration::SECOND), - ..source.clone() - }; + fn test_precondition_conflict_accepts_only_newer_null_delete_marker() { + for version_id in [None, Some(Uuid::nil())] { + let source = ObjectInfo { + version_id, + size: 128, + etag: Some("etag-source".to_string()), + mod_time: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }; + let target = ObjectInfo { + delete_marker: true, + etag: None, + mod_time: OffsetDateTime::UNIX_EPOCH.checked_add(time::Duration::SECOND), + ..source.clone() + }; - let should_resume = - resolve_data_movement_overwrite_resume_result(&Error::PreconditionFailed, Ok(Some(target)), &source, 0, 1) - .expect("delete marker conflict should be evaluated"); + assert!( + resolve_data_movement_overwrite_resume_result( + &Error::PreconditionFailed, + Ok(Some(target.clone())), + &source, + 0, + 1, + ) + .expect("newer null delete marker should be evaluated") + ); - assert!(!should_resume); + let mut same_time = target.clone(); + same_time.mod_time = source.mod_time; + assert!( + !resolve_data_movement_overwrite_resume_result(&Error::PreconditionFailed, Ok(Some(same_time)), &source, 0, 1,) + .expect("same-generation null delete marker should be rejected") + ); + + let mut versioned = target; + versioned.version_id = Some(Uuid::new_v4()); + assert!( + !resolve_data_movement_overwrite_resume_result(&Error::PreconditionFailed, Ok(Some(versioned)), &source, 0, 1,) + .expect("a UUID delete marker must not erase a null source version") + ); + } } #[test] diff --git a/crates/ecstore/src/data_usage/mod.rs b/crates/ecstore/src/data_usage/mod.rs index 917edc649..3bf9bf500 100644 --- a/crates/ecstore/src/data_usage/mod.rs +++ b/crates/ecstore/src/data_usage/mod.rs @@ -1391,7 +1391,37 @@ impl BucketUsageAccumulator { } pub fn quota_object_size(object: &ObjectInfo) -> Result { - let logical_size = u64::try_from(object.get_actual_size().map_err(Error::other)?).map_err(|_| Error::PartMissingOrCorrupt)?; + // A compressed object may carry -1 while the transformed size is unknown + // (legacy streaming sentinel). In that case the persisted physical size + // is still a valid accounting floor; every other negative value is corrupt. + // An explicit negative `actual-size` metadata value is corrupt, however: + // the sentinel is only valid in the in-memory/object-part field written by + // the legacy streaming path, not as a persisted declared size. + let compressed = object.is_compressed(); + if object.actual_size < -1 || (object.actual_size == -1 && !compressed) { + return Err(Error::PartMissingOrCorrupt); + } + if object + .parts + .iter() + .any(|part| part.actual_size < -1 || (part.actual_size < 0 && !compressed)) + { + return Err(Error::PartMissingOrCorrupt); + } + let declared_actual_size = rustfs_utils::http::get_str(&object.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE) + .filter(|value| !value.is_empty()); + if declared_actual_size + .as_deref() + .and_then(|value| value.parse::().ok()) + .is_some_and(|size| size < 0) + { + return Err(Error::PartMissingOrCorrupt); + } + let logical_size = match object.get_actual_size().map_err(Error::other)? { + size if size == -1 && compressed && declared_actual_size.is_none() => None, + size if size >= 0 => Some(u64::try_from(size).map_err(|_| Error::PartMissingOrCorrupt)?), + _ => return Err(Error::PartMissingOrCorrupt), + }; let persisted_part_size = if object.parts.is_empty() { u64::try_from(object.size).map_err(|_| Error::PartMissingOrCorrupt)? } else { @@ -1399,12 +1429,8 @@ pub fn quota_object_size(object: &ObjectInfo) -> Result { // Compressed streaming objects persist -1 when the transformed // part size is unknown. The physical part size remains a valid // quota floor; reject only non-negative values that overflow. - let actual_size = if part.actual_size < 0 { - if object.is_compressed() { - 0 - } else { - return Err(Error::PartMissingOrCorrupt); - } + let actual_size = if part.actual_size == -1 { + 0 } else { u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)? }; @@ -1412,7 +1438,7 @@ pub fn quota_object_size(object: &ObjectInfo) -> Result { total.checked_add(part_size).ok_or(Error::PartMissingOrCorrupt) })? }; - Ok(logical_size.max(persisted_part_size)) + Ok(logical_size.unwrap_or(0).max(persisted_part_size)) } type UsageVersionPage = StorageListObjectVersionsInfo; @@ -3320,6 +3346,80 @@ mod tests { ); } + #[test] + fn quota_object_size_accepts_compressed_unknown_actual_size_sentinel() { + let mut metadata = HashMap::new(); + rustfs_utils::http::insert_str( + &mut metadata, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + let object = ObjectInfo { + size: 400, + actual_size: -1, + user_defined: Arc::new(metadata), + ..Default::default() + }; + + assert_eq!(quota_object_size(&object).expect("compressed sentinel is valid"), 400); + } + + #[test] + fn quota_object_size_rejects_compressed_part_sum_overflow() { + let mut metadata = HashMap::new(); + rustfs_utils::http::insert_str( + &mut metadata, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + let object = ObjectInfo { + size: 1, + user_defined: Arc::new(metadata), + parts: Arc::new(vec![ + rustfs_filemeta::ObjectPartInfo { + actual_size: i64::MAX, + ..Default::default() + }, + rustfs_filemeta::ObjectPartInfo { + actual_size: 1, + ..Default::default() + }, + ]), + ..Default::default() + }; + + assert!(matches!(quota_object_size(&object), Err(Error::Io(_)))); + } + + #[test] + fn quota_object_size_rejects_negative_values_other_than_the_compressed_sentinel() { + let mut metadata = HashMap::new(); + rustfs_utils::http::insert_str( + &mut metadata, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + let corrupt_object = ObjectInfo { + size: 400, + actual_size: -2, + user_defined: Arc::new(metadata.clone()), + ..Default::default() + }; + assert!(matches!(quota_object_size(&corrupt_object), Err(Error::PartMissingOrCorrupt))); + + let corrupt_part = ObjectInfo { + size: 400, + user_defined: Arc::new(metadata), + parts: Arc::new(vec![rustfs_filemeta::ObjectPartInfo { + size: 400, + actual_size: -2, + ..Default::default() + }]), + ..Default::default() + }; + assert!(matches!(quota_object_size(&corrupt_part), Err(Error::PartMissingOrCorrupt))); + } + #[tokio::test] #[serial] async fn live_bucket_usage_refreshes_are_coalesced_only_while_in_flight() { diff --git a/crates/ecstore/src/disk/local.rs b/crates/ecstore/src/disk/local.rs index 0a9b11ef1..9994f8612 100644 --- a/crates/ecstore/src/disk/local.rs +++ b/crates/ecstore/src/disk/local.rs @@ -858,6 +858,7 @@ const EVENT_DISK_LOCAL_DIRECT_IO_FALLBACK: &str = "disk_local_direct_io_fallback #[cfg(target_os = "linux")] const EVENT_DISK_LOCAL_URING_LATCH_OFF: &str = "disk_local_uring_latch_off"; const EVENT_DISK_LOCAL_DELETE_FAILED: &str = "disk_local_delete_failed"; +const EVENT_DISK_LOCAL_DELETE_ROLLBACK_FAILED: &str = "disk_local_delete_rollback_failed"; const EVENT_DISK_LOCAL_CHECK_PARTS: &str = "disk_local_check_parts"; const EVENT_DISK_LOCAL_ACCESS_FAILED: &str = "disk_local_access_failed"; const EVENT_DISK_LOCAL_VOLUME_SETUP_FAILED: &str = "disk_local_volume_setup_failed"; @@ -5215,8 +5216,8 @@ impl LocalDisk { let cache = Cache::new(update_fn, Duration::from_secs(1), Opts::default()); - // TODO: DIRECT support - // TODD: DiskInfo + // TODO(backlog): add O_DIRECT I/O support for performance-critical paths + // TODO(backlog): populate DiskInfo in constructor let mut disk = Self { root: root.clone(), publication_root, @@ -5751,7 +5752,7 @@ impl LocalDisk { // return Ok(()); - // TODO: async notifications for disk space checks and trash cleanup + // TODO(backlog): make disk space checks and trash cleanup event-driven instead of poll-based let trash_path = self.io_get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, Uuid::new_v4().to_string().as_str())?; // if let Some(parent) = trash_path.parent() { @@ -5997,7 +5998,7 @@ impl LocalDisk { #[hotpath::measure(impl_type = "LocalDisk")] async fn read_all_data(&self, volume: &str, volume_dir: impl AsRef, file_path: impl AsRef) -> Result> { - // TODO: timeout support + // TODO(backlog): add configurable timeout for read_all_data operations let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?; Ok(data) } @@ -6106,6 +6107,43 @@ impl LocalDisk { Ok((bytes, modtime)) } + async fn write_missing_delete_marker( + &self, + volume: &str, + path: &str, + fi: FileInfo, + object_dir: &Path, + xl_path: &Path, + rollback_dir: Option, + ) -> Result<()> { + if let Some(rollback_dir) = rollback_dir { + let rollback_path = object_dir.join(rollback_dir.to_string()); + fs::create_dir_all(&rollback_path).await.map_err(to_file_error)?; + fs::write(rollback_path.join(DELETE_MARKER_ROLLBACK_FILE), []) + .await + .map_err(to_file_error)?; + } + if let Err(err) = self.write_metadata("", volume, path, fi).await { + if let Some(rollback_dir) = rollback_dir + && let Err(restore_err) = restore_delete_rollback(object_dir, xl_path, rollback_dir, &self.publication_root).await + { + warn!( + event = EVENT_DISK_LOCAL_DELETE_ROLLBACK_FAILED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK_LOCAL, + result = "failed", + volume, + path, + rollback_dir = %rollback_dir, + error = ?restore_err, + "Disk local delete rollback failed" + ); + } + return Err(err); + } + Ok(()) + } + async fn delete_versions_internal(&self, volume: &str, path: &str, fis: &[FileInfo], opts: &DeleteOptions) -> Result<()> { let volume_dir = self.io_get_bucket_path(volume)?; let xlpath = self.io_get_object_path(volume, format!("{path}/{STORAGE_FORMAT_FILE}").as_str())?; @@ -6123,7 +6161,20 @@ impl LocalDisk { return restore_metadata_backup(object_dir, &xlpath, rollback_dir, &self.publication_root).await; } - let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir.as_path(), &xlpath).await?; + let (data, _) = match self.read_all_data_with_dmtime(volume, volume_dir.as_path(), &xlpath).await { + Ok(data) => data, + Err(DiskError::FileNotFound) => { + // `deleted` alone can be an explicit marker purge; only + // `mark_deleted` may create metadata that was not present. + let Some(delete_marker) = fis.iter().find(|fi| fi.deleted && fi.mark_deleted).cloned() else { + return Err(DiskError::FileNotFound); + }; + return self + .write_missing_delete_marker(volume, path, delete_marker, object_dir, &xlpath, opts.old_data_dir) + .await; + } + Err(err) => return Err(err), + }; if data.is_empty() { return Err(DiskError::FileNotFound); @@ -6674,7 +6725,7 @@ impl LocalDisk { return Ok(()); } - // TODO: add lock + // TODO(backlog): add directory listing lock to prevent concurrent enumeration let stall = opts.stall_timeout_duration(); @@ -8796,7 +8847,7 @@ impl DiskAPI for LocalDisk { Ok(entries) } - // FIXME: TODO: io.writer TODO cancel + // TODO(backlog): support io.writer cancellation and early termination in walk_dir #[tracing::instrument(level = "trace", skip_all)] async fn walk_dir(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> { self.wait_for_startup_cleanup().await; @@ -9880,7 +9931,7 @@ impl DiskAPI for LocalDisk { ); return Err(e); } - // TODO: health check + // TODO(backlog): add post-setup disk health verification } Ok(()) } @@ -10422,29 +10473,9 @@ impl DiskAPI for LocalDisk { } if fi.deleted && force_del_marker { - if let Some(rollback_dir) = rollback_dir { - let rollback_path = file_path.join(rollback_dir.to_string()); - fs::create_dir_all(&rollback_path).await.map_err(to_file_error)?; - fs::write(rollback_path.join(DELETE_MARKER_ROLLBACK_FILE), []) - .await - .map_err(to_file_error)?; - } - if let Err(err) = self.write_metadata("", volume, path, fi).await { - if let Some(rollback_dir) = rollback_dir - && let Err(restore_err) = - restore_delete_rollback(file_path.as_path(), &xl_path, rollback_dir, &self.publication_root).await - { - warn!( - volume, - path, - rollback_dir = %rollback_dir, - error = ?restore_err, - "failed to restore metadata after delete marker commit error" - ); - } - return Err(err); - } - return Ok(()); + return self + .write_missing_delete_marker(volume, path, fi, file_path.as_path(), &xl_path, rollback_dir) + .await; } return if fi.version_id.is_some() { diff --git a/crates/ecstore/src/disk/mod.rs b/crates/ecstore/src/disk/mod.rs index ecd0179e0..fb8c8de5a 100644 --- a/crates/ecstore/src/disk/mod.rs +++ b/crates/ecstore/src/disk/mod.rs @@ -44,6 +44,8 @@ pub const PART_TRANSACTION_ROLLBACK: &str = "rollback"; const LOG_COMPONENT_ECSTORE: &str = "ecstore"; const LOG_SUBSYSTEM_DISK: &str = "disk"; const EVENT_DISK_PART_ERR_UNCLASSIFIED: &str = "disk_part_err_unclassified"; +const ENV_BATCH_READ_VERSION_SERVER_PARALLELISM: &str = "RUSTFS_BATCH_READ_VERSION_SERVER_PARALLELISM"; +const BATCH_READ_VERSION_SERVER_PARALLELISM: usize = 4; pub fn part_transaction_path(part_path: &str) -> String { match part_path.rsplit_once('/') { @@ -62,6 +64,7 @@ use bytes::Bytes; use endpoint::Endpoint; use error::DiskError; use error::{Error, Result}; +use futures::stream::{self, StreamExt}; use local::LocalDisk; use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo}; use rustfs_madmin::info_commands::DiskMetrics; @@ -417,6 +420,14 @@ impl DiskAPI for Disk { } } + #[tracing::instrument(level = "trace", skip_all)] + async fn batch_read_version(&self, req: BatchReadVersionReq) -> Result> { + match self { + Disk::Local(local_disk) => local_disk.batch_read_version(req).await, + Disk::Remote(remote_disk) => remote_disk.batch_read_version(req).await, + } + } + #[tracing::instrument(level = "trace", skip_all)] async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result { match self { @@ -1028,36 +1039,47 @@ where D: DiskAPI + ?Sized, { validate_batch_read_version_item_count(req.items.len())?; + let parallelism = batch_read_version_server_parallelism(); - let mut responses = Vec::with_capacity(req.items.len()); - for (index, item) in req.items.iter().enumerate() { - let response = match disk - .read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts) - .await - { - Ok(file_info) => BatchReadVersionResp { - index, - path: item.path.clone(), - version_id: item.version_id.clone(), - success: true, - file_info, - error: String::new(), - }, - Err(err) => BatchReadVersionResp { - index, - path: item.path.clone(), - version_id: item.version_id.clone(), - success: false, - file_info: FileInfo::default(), - error: err.to_string(), - }, - }; - responses.push(response); - } + let mut responses = stream::iter(req.items.into_iter().enumerate()) + .map(|(index, item)| async move { + match disk + .read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts) + .await + { + Ok(file_info) => BatchReadVersionResp { + index, + path: item.path, + version_id: item.version_id, + success: true, + file_info, + error: String::new(), + error_code: 0, + }, + Err(err) => BatchReadVersionResp { + index, + path: item.path, + version_id: item.version_id, + success: false, + file_info: FileInfo::default(), + error: err.to_string(), + error_code: err.to_u32(), + }, + } + }) + .buffer_unordered(parallelism) + .collect::>() + .await; + responses.sort_unstable_by_key(|response| response.index); Ok(responses) } +fn batch_read_version_server_parallelism() -> usize { + rustfs_utils::get_env_usize(ENV_BATCH_READ_VERSION_SERVER_PARALLELISM, BATCH_READ_VERSION_SERVER_PARALLELISM) + .clamp(1, BATCH_READ_VERSION_MAX_ITEMS) +} + #[derive(Debug, Default, Serialize, Deserialize)] pub struct CheckPartsResp { pub results: Vec, @@ -1322,6 +1344,8 @@ pub struct BatchReadVersionResp { pub success: bool, pub file_info: FileInfo, pub error: String, + #[serde(default)] + pub error_code: u32, } pub fn validate_batch_read_version_item_count(item_count: usize) -> Result<()> { @@ -1417,6 +1441,26 @@ mod tests { assert!(!partial_valid_location.valid()); } + #[test] + fn batch_read_version_server_parallelism_defaults_to_conservative_four() { + temp_env::with_var(ENV_BATCH_READ_VERSION_SERVER_PARALLELISM, None::<&str>, || { + assert_eq!(batch_read_version_server_parallelism(), 4); + }); + } + + #[test] + fn batch_read_version_server_parallelism_honors_env_with_bounds() { + temp_env::with_var(ENV_BATCH_READ_VERSION_SERVER_PARALLELISM, Some("8"), || { + assert_eq!(batch_read_version_server_parallelism(), 8); + }); + temp_env::with_var(ENV_BATCH_READ_VERSION_SERVER_PARALLELISM, Some("0"), || { + assert_eq!(batch_read_version_server_parallelism(), 1); + }); + temp_env::with_var(ENV_BATCH_READ_VERSION_SERVER_PARALLELISM, Some("9999"), || { + assert_eq!(batch_read_version_server_parallelism(), BATCH_READ_VERSION_MAX_ITEMS); + }); + } + /// Test FileInfoVersions find_version_index #[test] fn test_file_info_versions_find_version_index() { diff --git a/crates/ecstore/src/disk/os.rs b/crates/ecstore/src/disk/os.rs index ba97f67c5..77ea01991 100644 --- a/crates/ecstore/src/disk/os.rs +++ b/crates/ecstore/src/disk/os.rs @@ -315,7 +315,7 @@ pub async fn fsync_dir(dir: impl AsRef) -> io::Result<()> { #[cfg(unix)] { let dir = dir.as_ref().to_path_buf(); - tokio::task::spawn_blocking(move || fsync_dir_std(dir)).await? + fsync_spawn_blocking(move || fsync_dir_std(dir)).await? } #[cfg(not(unix))] @@ -683,7 +683,7 @@ async fn fsync_open_dst_dir_group(group: &DstDirFsyncGroup) -> io::Result<()> { #[cfg(test)] let dir = group.dir.clone(); let dir_file = group.dir_file.clone(); - tokio::task::spawn_blocking(move || { + fsync_spawn_blocking(move || { #[cfg(test)] { if let Some(kind) = fsync_dir_recorder::take_grouped_failure(&dir) { @@ -1080,6 +1080,44 @@ const TEST_GLOBAL_FILE_SYNCS: usize = 64; static FILE_SYNC_PERMITS: LazyLock = LazyLock::new(|| Semaphore::new(global_file_sync_limit())); static DISK_FILE_SYNC_LIMITERS: LazyLock>>> = LazyLock::new(|| Mutex::new(HashMap::new())); + +/// Dedicated tokio runtime for fsync/fdatasync blocking operations. When +/// configured with >1 threads, isolates device-bound fsync from the main +/// blocking pool so reads (pread/stat/open) are not starved. `None` means +/// fall back to the main runtime (zero behavior change). +static FSYNC_RUNTIME: LazyLock> = LazyLock::new(|| { + let threads = + rustfs_utils::get_env_usize(rustfs_config::ENV_FSYNC_BLOCKING_THREADS, rustfs_config::DEFAULT_FSYNC_BLOCKING_THREADS); + if threads <= 1 { + return None; + } + let mut builder = tokio::runtime::Builder::new_multi_thread(); + builder + .worker_threads(num_cpus::get().min(8)) + .max_blocking_threads(threads) + .thread_name("rustfs-fsync") + .thread_stack_size(512 * 1024) + .enable_all(); + match builder.build() { + Ok(rt) => { + tracing::info!(threads, "fsync dedicated blocking pool enabled"); + Some(rt) + } + Err(err) => { + tracing::warn!(%err, "failed to build fsync runtime, falling back to main pool"); + None + } + } +}); + +/// Spawn a blocking task on the fsync-dedicated runtime if configured, +/// otherwise fall back to the main tokio blocking pool. +fn fsync_spawn_blocking(f: impl FnOnce() -> T + Send + 'static) -> tokio::task::JoinHandle { + match FSYNC_RUNTIME.as_ref() { + Some(rt) => rt.spawn_blocking(f), + None => tokio::task::spawn_blocking(f), + } +} static DISK_VOLUME_MUTATION_LOCKS: LazyLock>>>> = LazyLock::new(|| Mutex::new(HashMap::new())); type NamespaceMutationLock = AsyncMutex<()>; @@ -1217,7 +1255,7 @@ where F: FnOnce() -> io::Result + Send + 'static, { let (disk_permit, global_permit) = acquire_file_sync_permits(disk_permits).await?; - let result = tokio::task::spawn_blocking(move || { + let result = fsync_spawn_blocking(move || { let _disk_permit = disk_permit; work() }) @@ -2146,7 +2184,7 @@ async fn run_blocking_namespace_file_sync_operation_with_global>>, #[cfg(test)] - forced_lost: Arc, + forced_lost: Arc>>, } impl Debug for NamespaceLockFence { @@ -40,13 +40,17 @@ impl NamespaceLockFence { Self { signals: Arc::default(), #[cfg(test)] - forced_lost: Arc::new(std::sync::atomic::AtomicBool::new(false)), + forced_lost: Arc::new(vec![Arc::new(std::sync::atomic::AtomicBool::new(false))]), } } pub(crate) fn is_lock_lost(&self) -> bool { #[cfg(test)] - if self.forced_lost.load(std::sync::atomic::Ordering::Acquire) { + if self + .forced_lost + .iter() + .any(|lost| lost.load(std::sync::atomic::Ordering::Acquire)) + { return true; } self.signals.iter().any(|signal| signal.is_lost()) @@ -57,27 +61,26 @@ impl NamespaceLockFence { } fn extend(&mut self, other: &Self) { - if Arc::ptr_eq(&self.signals, &other.signals) { - return; + if !Arc::ptr_eq(&self.signals, &other.signals) { + Arc::make_mut(&mut self.signals).extend(other.signals.iter().cloned()); } - Arc::make_mut(&mut self.signals).extend(other.signals.iter().cloned()); #[cfg(test)] - if other.forced_lost.load(std::sync::atomic::Ordering::Acquire) { - self.forced_lost.store(true, std::sync::atomic::Ordering::Release); + if !Arc::ptr_eq(&self.forced_lost, &other.forced_lost) { + Arc::make_mut(&mut self.forced_lost).extend(other.forced_lost.iter().cloned()); } } #[cfg(test)] pub(crate) fn lost_for_test() -> Self { let fence = Self::new(); - fence.forced_lost.store(true, std::sync::atomic::Ordering::Release); + fence.forced_lost[0].store(true, std::sync::atomic::Ordering::Release); fence } #[cfg(test)] pub(crate) fn loss_handle_for_test() -> (Self, Arc) { let fence = Self::new(); - (fence.clone(), Arc::clone(&fence.forced_lost)) + (fence.clone(), Arc::clone(&fence.forced_lost[0])) } } @@ -411,6 +414,13 @@ impl ObjectOptions { self.namespace_lock_fence.get_or_insert_with(NamespaceLockFence::new); } + #[cfg(test)] + pub(crate) fn add_namespace_lock_fence_for_test(&mut self, fence: &NamespaceLockFence) { + self.namespace_lock_fence + .get_or_insert_with(NamespaceLockFence::new) + .extend(fence); + } + pub(crate) fn ensure_lifecycle_delete_all_journal(&mut self) { self.lifecycle_delete_all_journal .get_or_insert_with(|| Arc::new(parking_lot::Mutex::new(LifecycleDeleteAllJournalState::default()))); @@ -689,6 +699,9 @@ impl ObjectInfo { } pub fn get_actual_size(&self) -> std::io::Result { + if self.actual_size < -1 || (self.actual_size == -1 && !self.is_compressed()) { + return Err(std::io::Error::other("invalid negative actual size")); + } if self.actual_size > 0 { return Ok(self.actual_size); } @@ -700,10 +713,25 @@ impl ObjectInfo { let size = size_str.parse::().map_err(|e| std::io::Error::other(e.to_string()))?; return Ok(size); } - let mut actual_size = 0; - self.parts.iter().for_each(|part| { - actual_size += part.actual_size; - }); + if self.actual_size == -1 && self.parts.is_empty() { + return Ok(-1); + } + let mut actual_size = 0_i64; + let mut unknown = false; + for part in self.parts.iter() { + match part.actual_size { + -1 => unknown = true, + size if size >= 0 => { + actual_size = actual_size + .checked_add(size) + .ok_or_else(|| std::io::Error::other("compressed actual size overflow"))?; + } + _ => return Err(std::io::Error::other("invalid negative compressed part size")), + } + } + if unknown { + return Ok(-1); + } if actual_size == 0 && actual_size != self.size { return Err(std::io::Error::other(format!("invalid decompressed size {} {}", actual_size, self.size))); } @@ -718,6 +746,18 @@ impl ObjectInfo { Ok(self.size) } + /// Returns a non-negative size for client and replication boundaries. + /// + /// Compressed legacy metadata can retain the internal `-1` unknown-size + /// sentinel. Those boundaries cannot emit a negative length, so they use + /// the persisted physical size while quota accounting keeps the sentinel + /// distinction in [`crate::data_usage::quota_object_size`]. + pub fn get_actual_size_or_physical(&self) -> i64 { + self.get_actual_size() + .map(|size| if size >= 0 { size } else { self.size.max(0) }) + .unwrap_or_else(|_| self.size.max(0)) + } + pub fn from_file_info(fi: &FileInfo, bucket: &str, object: &str, versioned: bool) -> ObjectInfo { let mut version_id = fi.version_id; @@ -1091,7 +1131,7 @@ impl ObjectInfo { } }; - // TODO:VersionPurgeStatus + // TODO(backlog): handle VersionPurgeStatus in object listing let versioned = vcfg.clone().map(|v| v.0.versioned(&entry.name)).unwrap_or_default(); objects.push(ObjectInfo::from_file_info(&fi, bucket, &entry.name, versioned)); diff --git a/crates/ecstore/src/runtime/global.rs b/crates/ecstore/src/runtime/global.rs index fcc4411f0..56ba22a0a 100644 --- a/crates/ecstore/src/runtime/global.rs +++ b/crates/ecstore/src/runtime/global.rs @@ -25,7 +25,10 @@ use lazy_static::lazy_static; use rustfs_lock::client::LockClient; use std::{ collections::HashMap, - sync::{Arc, OnceLock}, + sync::{ + Arc, OnceLock, + atomic::{AtomicBool, Ordering}, + }, time::SystemTime, }; use tokio::sync::{OnceCell, RwLock}; @@ -37,6 +40,16 @@ pub const DISK_MIN_INODES: u64 = 1000; pub const DISK_FILL_FRACTION: f64 = 0.99; pub const DISK_RESERVE_FRACTION: f64 = 0.15; +static GET_METADATA_READ_VERSION_COALESCING_SERVICE_READY: AtomicBool = AtomicBool::new(false); + +pub(crate) fn mark_get_metadata_read_version_coalescing_service_ready() { + GET_METADATA_READ_VERSION_COALESCING_SERVICE_READY.store(true, Ordering::Release); +} + +pub(crate) fn get_metadata_read_version_coalescing_service_ready() -> bool { + GET_METADATA_READ_VERSION_COALESCING_SERVICE_READY.load(Ordering::Acquire) +} + // Global singletons for backward compatibility with MinIO port. // These should be migrated to AppContext over time. // See issue #730 for migration plan. diff --git a/crates/ecstore/src/runtime/instance.rs b/crates/ecstore/src/runtime/instance.rs index 71a1898cf..9453ed02b 100644 --- a/crates/ecstore/src/runtime/instance.rs +++ b/crates/ecstore/src/runtime/instance.rs @@ -160,6 +160,10 @@ pub struct InstanceContext { /// workers (scanner/heal/tier/lifecycle) without touching another instance. /// Replaces the process-global cancel-token static. background_cancel_token: OnceLock, + /// Serializes decommission data-movement operations with cancellation and + /// a subsequent restart. Readers are held across one object side effect; + /// the transition path takes the writer after cancelling the routine. + decommission_operation_gate: Arc>, /// Resolves object-encryption material at the application boundary. object_encryption_resolver: OnceLock>, tier_delete_journal_recovery_stores: std::sync::Mutex>, @@ -200,6 +204,7 @@ impl InstanceContext { local_disk_set_drives: Arc::new(RwLock::new(Vec::new())), bucket_metadata_sys: std::sync::Mutex::new(None), background_cancel_token: OnceLock::new(), + decommission_operation_gate: Arc::new(RwLock::new(())), object_encryption_resolver: OnceLock::new(), tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()), transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()), @@ -218,6 +223,10 @@ impl InstanceContext { self.lock_manager.clone() } + pub(crate) fn decommission_operation_gate(&self) -> Arc> { + Arc::clone(&self.decommission_operation_gate) + } + /// Install the application-owned object-encryption resolver once. pub fn set_object_encryption_resolver( &self, diff --git a/crates/ecstore/src/services/metrics_realtime.rs b/crates/ecstore/src/services/metrics_realtime.rs index e4fc919e6..6e2e5cd7d 100644 --- a/crates/ecstore/src/services/metrics_realtime.rs +++ b/crates/ecstore/src/services/metrics_realtime.rs @@ -256,6 +256,10 @@ fn to_madmin_scanner_metrics(metrics: rustfs_common::metrics::ScannerMetricsRepo cycle_max_duration_seconds: metrics.cycle_max_duration_seconds, cycle_max_objects: metrics.cycle_max_objects, cycle_max_directories: metrics.cycle_max_directories, + cycle_timeout_total: metrics.cycle_timeout_total, + cycle_recovery_required_total: metrics.cycle_recovery_required_total, + cycle_last_progress_age: metrics.cycle_last_progress_age, + leader_lease_without_progress: metrics.leader_lease_without_progress, bitrot_cycle_enabled: metrics.bitrot_cycle_enabled, bitrot_cycle_seconds: metrics.bitrot_cycle_seconds, scan_checkpoint: metrics.scan_checkpoint.map(|checkpoint| MadminScannerCheckpointReport { @@ -611,6 +615,10 @@ mod test { current_started: chrono_to_jiff_timestamp(current_started), last_cycle_partial_source: "usage".to_string(), last_cycle_partial_source_code: 1, + cycle_timeout_total: 3, + cycle_recovery_required_total: 2, + cycle_last_progress_age: 17, + leader_lease_without_progress: true, partial_cycles_by_source: vec![rustfs_common::metrics::ScannerSourceCycleSnapshot { source: "usage".to_string(), cycles: 2, @@ -622,6 +630,10 @@ mod test { assert_eq!(scanner.current_started, chrono_to_jiff_timestamp(current_started)); assert_eq!(scanner.last_cycle_partial_source, "usage"); assert_eq!(scanner.last_cycle_partial_source_code, 1); + assert_eq!(scanner.cycle_timeout_total, 3); + assert_eq!(scanner.cycle_recovery_required_total, 2); + assert_eq!(scanner.cycle_last_progress_age, 17); + assert!(scanner.leader_lease_without_progress); let usage = scanner .partial_cycles_by_source .iter() diff --git a/crates/ecstore/src/services/rebalance/entry.rs b/crates/ecstore/src/services/rebalance/entry.rs index 764a68500..e9e1e2343 100644 --- a/crates/ecstore/src/services/rebalance/entry.rs +++ b/crates/ecstore/src/services/rebalance/entry.rs @@ -334,6 +334,7 @@ impl ECStore { lifecycle_guard: bucket_incarnation_fence .as_ref() .and_then(|guard| guard.namespace_lock_guard()), + ..Default::default() }, "rebalance", ), diff --git a/crates/ecstore/src/set_disk/core/io_primitives.rs b/crates/ecstore/src/set_disk/core/io_primitives.rs index d17c1c751..1a77ba267 100644 --- a/crates/ecstore/src/set_disk/core/io_primitives.rs +++ b/crates/ecstore/src/set_disk/core/io_primitives.rs @@ -53,11 +53,12 @@ use crate::diagnostics::get::{ GetObjectFailureReason, classify_disk_error, get_stage_timer_if_enabled, record_get_object_pipeline_failure, record_get_object_pipeline_failure_for_path, record_get_stage_duration_if_enabled, }; -use crate::disk::disk_store::DiskStoreRenameDataExt; +use crate::disk::disk_store::{DiskStoreRenameDataExt, get_drive_metadata_timeout}; use crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX; use crate::disk::{ - DataDirDeleteStatus, OldCurrentSize, PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, - PartTransactionAction, STORAGE_FORMAT_FILE_BACKUP, part_transaction_path, + BATCH_READ_VERSION_MAX_ITEMS, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp, DataDirDeleteStatus, Disk, + OldCurrentSize, PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, PartTransactionAction, + STORAGE_FORMAT_FILE_BACKUP, part_transaction_path, }; use crate::erasure::coding::BitrotReader; use crate::io_support::bitrot::ShardReader; @@ -75,7 +76,7 @@ use std::{ future::Future, pin::Pin, sync::{ - OnceLock, + Arc, OnceLock, atomic::{AtomicUsize, Ordering}, }, task::{Context, Poll}, @@ -94,6 +95,242 @@ fn metadata_distribution_key(bucket: &str, object: &str) -> String { [bucket, object].join("/") } +fn read_version_coalescing_enabled() -> bool { + let enabled = || { + rustfs_utils::get_env_opt_str(ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE) + .is_some_and(|value| value.eq_ignore_ascii_case("auto") || value.eq_ignore_ascii_case("on")) + }; + + #[cfg(test)] + { + enabled() + } + + #[cfg(not(test))] + { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(enabled) + } +} + +fn read_version_coalescing_delay() -> Duration { + #[cfg(test)] + { + let micros = rustfs_utils::get_env_u64( + ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS, + DEFAULT_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS, + ); + Duration::from_micros(micros) + } + + #[cfg(not(test))] + { + static DELAY: OnceLock = OnceLock::new(); + *DELAY.get_or_init(|| { + Duration::from_micros(rustfs_utils::get_env_u64( + ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS, + DEFAULT_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS, + )) + }) + } +} + +struct CoalescedReadVersionRequest { + item: BatchReadVersionItem, + tx: oneshot::Sender>, +} + +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] +struct ReadVersionCoalescerKey { + disk: usize, + incl_free_versions: bool, + read_data: bool, + healing: bool, +} + +impl ReadVersionCoalescerKey { + fn new(disk: &DiskStore, opts: &ReadOptions) -> Self { + Self { + disk: Arc::as_ptr(disk) as usize, + incl_free_versions: opts.incl_free_versions, + read_data: opts.read_data, + healing: opts.healing, + } + } +} + +#[derive(Default)] +struct ReadVersionCoalescer { + lanes: HashMap>, +} + +fn read_version_coalescer() -> &'static Mutex { + static COALESCER: OnceLock> = OnceLock::new(); + COALESCER.get_or_init(|| Mutex::new(ReadVersionCoalescer::default())) +} + +fn record_read_version_coalescer_event(event: &'static str, item_count: usize) { + counter!( + METRIC_GET_METADATA_READ_VERSION_COALESCER_TOTAL, + "event" => event, + "item_count" => item_count.to_string() + ) + .increment(1); +} + +async fn read_version_via_coalescer( + disk: DiskStore, + org_bucket: &str, + bucket: &str, + object: &str, + version_id: &str, + opts: &ReadOptions, + allow_coalescing: bool, +) -> disk::error::Result { + if !allow_coalescing || !read_version_coalescing_enabled() { + return disk.read_version(org_bucket, bucket, object, version_id, opts).await; + } + if !matches!(disk.as_ref(), Disk::Remote(_)) { + record_read_version_coalescer_event("bypass_non_remote", 1); + return disk.read_version(org_bucket, bucket, object, version_id, opts).await; + } + + let (tx, rx) = oneshot::channel(); + let item = BatchReadVersionItem { + org_volume: org_bucket.to_string(), + volume: bucket.to_string(), + path: object.to_string(), + version_id: version_id.to_string(), + }; + let lane_key = ReadVersionCoalescerKey::new(&disk, opts); + let pending = { + let mut coalescer = read_version_coalescer().lock().await; + let lane = coalescer.lanes.entry(lane_key).or_default(); + let schedule_delayed_flush = lane.is_empty(); + lane.push(CoalescedReadVersionRequest { item, tx }); + if lane.len() >= BATCH_READ_VERSION_MAX_ITEMS { + coalescer.lanes.remove(&lane_key) + } else if schedule_delayed_flush { + let disk = disk.clone(); + let task_opts = *opts; + tokio::spawn(async move { + tokio::time::sleep(read_version_coalescing_delay()).await; + flush_read_version_coalescer_lane(lane_key, disk, task_opts).await; + }); + None + } else { + None + } + }; + + if let Some(pending) = pending { + flush_read_version_coalescer_pending(lane_key, disk, *opts, pending).await; + } + + rx.await + .unwrap_or_else(|_| Err(DiskError::other("coalesced read_version response channel closed"))) +} + +async fn flush_read_version_coalescer_lane(lane_key: ReadVersionCoalescerKey, disk: DiskStore, opts: ReadOptions) { + let pending = { + let mut coalescer = read_version_coalescer().lock().await; + coalescer.lanes.remove(&lane_key).unwrap_or_default() + }; + flush_read_version_coalescer_pending(lane_key, disk, opts, pending).await; +} + +async fn flush_read_version_coalescer_pending( + lane_key: ReadVersionCoalescerKey, + disk: DiskStore, + opts: ReadOptions, + pending: Vec, +) { + if pending.is_empty() { + return; + } + + #[cfg(test)] + { + let mut observed_paths = HashSet::new(); + for request in &pending { + if observed_paths.insert(request.item.path.as_str()) { + disk_call_counters::record(&request.item.path, disk_call_counters::KIND_BATCH_READ_VERSION, lane_key.disk); + } + } + } + + let mut senders = Vec::with_capacity(pending.len()); + let mut items = Vec::with_capacity(pending.len()); + for request in pending { + senders.push(request.tx); + items.push(request.item); + } + + let expected_items = items.clone(); + record_read_version_coalescer_event("attempted_batch", items.len()); + let result = + match tokio::time::timeout(get_drive_metadata_timeout(), disk.batch_read_version(BatchReadVersionReq { items, opts })) + .await + { + Ok(result) => result, + Err(_) => Err(DiskError::Timeout), + }; + match result { + Ok(responses) => { + let results = map_batch_read_version_responses(&expected_items, responses); + for (tx, result) in senders.into_iter().zip(results) { + let _ = tx.send(result); + } + } + Err(err) => { + let message = err.to_string(); + for tx in senders { + let _ = tx.send(Err(DiskError::other(message.clone()))); + } + } + } +} + +fn map_batch_read_version_responses( + expected_items: &[BatchReadVersionItem], + responses: Vec, +) -> Vec> { + let mut results = (0..expected_items.len()) + .map(|_| Err(DiskError::other("coalesced read_version response missing"))) + .collect::>(); + let mut seen = vec![false; expected_items.len()]; + for response in responses { + let Some(expected) = expected_items.get(response.index) else { + continue; + }; + let Some(slot) = results.get_mut(response.index) else { + continue; + }; + if seen[response.index] { + *slot = Err(DiskError::other("coalesced read_version response duplicate index")); + continue; + } + seen[response.index] = true; + if response.path != expected.path || response.version_id != expected.version_id { + *slot = Err(DiskError::other("coalesced read_version response identity mismatch")); + } else { + *slot = if response.success { + Ok(response.file_info) + } else { + Err(batch_read_version_response_error(response.error_code, response.error)) + }; + } + } + results +} + +fn batch_read_version_response_error(error_code: u32, error: String) -> DiskError { + match DiskError::from_u32(error_code) { + Some(DiskError::Io(_)) | None => DiskError::other(error), + Some(error) => error, + } +} + pub(in crate::set_disk) fn bounded_metadata_fanout_order( bucket: &str, object: &str, @@ -133,11 +370,15 @@ pub(in crate::set_disk) fn bounded_metadata_fanout_order( order } use tokio::io::{AsyncRead, ReadBuf}; -use tokio::sync::RwLock; +use tokio::sync::{Mutex, RwLock, oneshot}; use tokio::task::JoinSet; pub(in crate::set_disk) const EVENT_SET_DISK_READ: &str = "set_disk_read"; pub(in crate::set_disk) const ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP: &str = "RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP"; +const ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE: &str = "RUSTFS_GET_METADATA_READ_VERSION_COALESCE"; +const ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS: &str = "RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS"; +const DEFAULT_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS: u64 = 200; +const METRIC_GET_METADATA_READ_VERSION_COALESCER_TOTAL: &str = "rustfs_get_metadata_read_version_coalescer_total"; pub(in crate::set_disk) const ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE: &str = "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE"; /// Default reader-setup strategy for the GET read path (rustfs/backlog#1215, /// #1159, #923). @@ -2356,6 +2597,7 @@ impl SetDisks { false, true, 0, + false, ) .await?; Ok((ress, errors)) @@ -2386,6 +2628,36 @@ impl SetDisks { true, caller_allows_early_stop, default_parity_count, + false, + ) + .await + } + + #[allow(clippy::too_many_arguments)] + pub(in crate::set_disk) async fn read_all_fileinfo_observed_for_get_object( + disks: &[Option], + org_bucket: &str, + bucket: &str, + object: &str, + version_id: &str, + read_data: bool, + incl_free_versions: bool, + caller_allows_early_stop: bool, + default_parity_count: usize, + ) -> disk::error::Result<(Vec, Vec>, MetadataFanoutDiagnostics)> { + Self::read_all_fileinfo_inner( + disks, + org_bucket, + bucket, + object, + version_id, + read_data, + false, + incl_free_versions, + true, + caller_allows_early_stop, + default_parity_count, + true, ) .await } @@ -2408,6 +2680,7 @@ impl SetDisks { // subset would fail write quorum (backlog#872 regression). caller_allows_early_stop: bool, default_parity_count: usize, + allow_coalescing: bool, ) -> disk::error::Result<(Vec, Vec>, MetadataFanoutDiagnostics)> { let early_stop_enabled = caller_allows_early_stop && observe && (is_get_metadata_early_stop_enabled() || is_version_early_stop_enabled()); @@ -2424,6 +2697,7 @@ impl SetDisks { healing, incl_free_versions, default_parity_count, + allow_coalescing, ) .await; } @@ -2446,6 +2720,7 @@ impl SetDisks { healing, incl_free_versions, observe, + allow_coalescing, ) .await } @@ -2461,6 +2736,7 @@ impl SetDisks { healing: bool, incl_free_versions: bool, observe: bool, + allow_coalescing: bool, ) -> disk::error::Result<(Vec, Vec>, MetadataFanoutDiagnostics)> { let fanout_start = observe.then(Instant::now); let mut ress = Vec::with_capacity(disks.len()); @@ -2492,7 +2768,7 @@ impl SetDisks { if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) { tokio::time::sleep(delay).await; } - disk.read_version(&org_bucket, &bucket, &object, &version_id, &task_opts) + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) .await } else { Err(DiskError::DiskNotFound) @@ -2559,6 +2835,7 @@ impl SetDisks { healing: bool, incl_free_versions: bool, default_parity_count: usize, + allow_coalescing: bool, ) -> disk::error::Result<(Vec, Vec>, MetadataFanoutDiagnostics)> { let fanout_start = Instant::now(); let mut ress = vec![FileInfo::default(); disks.len()]; @@ -2607,7 +2884,7 @@ impl SetDisks { if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) { tokio::time::sleep(delay).await; } - disk.read_version(&org_bucket, &bucket, &object, &version_id, &task_opts) + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) .await } else { Err(DiskError::DiskNotFound) @@ -5737,6 +6014,7 @@ pub(crate) mod disk_call_counters { /// Kind label for the per-disk `read_version` metadata RPC. pub const KIND_READ_VERSION: &str = "read_version"; + pub const KIND_BATCH_READ_VERSION: &str = "batch_read_version"; /// Registry key: (object, kind, disk_index). type CountKey = (String, String, usize); @@ -6460,6 +6738,286 @@ mod tests { drop(dirs); } + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn metadata_read_version_coalescer_bypasses_local_disks() { + const DISKS: usize = 4; + let bucket = "coalesced-read-version-local-bypass-bucket"; + let object_a = "coalesced-local-object-a"; + let object_b = "coalesced-local-object-b"; + let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + install_metadata_fanout_fileinfo(&disks, bucket, object_a, None).await; + install_metadata_fanout_fileinfo(&disks, bucket, object_b, None).await; + + temp_env::async_with_vars( + [ + (ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE, Some("auto")), + (ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS, Some("5000")), + ], + async { + let calls = disk_call_counters::observe(object_a); + let disks_a = disks.clone(); + let disks_b = disks.clone(); + let read_a = tokio::spawn(async move { + SetDisks::read_all_fileinfo_observed_for_get_object( + &disks_a, "", bucket, object_a, "", false, false, false, 2, + ) + .await + .map(|(file_infos, errors, _)| (file_infos, errors)) + }); + tokio::task::yield_now().await; + let read_b = tokio::spawn(async move { + SetDisks::read_all_fileinfo_observed_for_get_object( + &disks_b, "", bucket, object_b, "", false, false, false, 2, + ) + .await + .map(|(file_infos, errors, _)| (file_infos, errors)) + }); + + let (metadata_a, errs_a) = read_a + .await + .expect("first read task should not panic") + .expect("first coalesced read should resolve"); + let (metadata_b, errs_b) = read_b + .await + .expect("second read task should not panic") + .expect("second coalesced read should resolve"); + + assert_eq!(metadata_a.iter().filter(|fi| fi.name == object_a).count(), DISKS); + assert_eq!(metadata_b.iter().filter(|fi| fi.name == object_b).count(), DISKS); + assert!(errs_a.iter().all(Option::is_none)); + assert!(errs_b.iter().all(Option::is_none)); + assert_eq!( + calls.total(disk_call_counters::KIND_READ_VERSION), + DISKS as u64, + "local disks still execute the ordinary per-disk read_version path" + ); + assert_eq!( + calls.total(disk_call_counters::KIND_BATCH_READ_VERSION), + 0, + "GET coalescing targets internode RPC count only and must not batch local disk reads" + ); + }, + ) + .await; + + drop(dirs); + } + + #[tokio::test] + async fn metadata_read_version_coalescer_requires_get_object_intent() { + const DISKS: usize = 4; + let bucket = "coalesced-read-version-default-bypass-bucket"; + let object = "default-bypass-object"; + let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + install_metadata_fanout_fileinfo(&disks, bucket, object, None).await; + + temp_env::async_with_vars([(ENV_RUSTFS_GET_METADATA_READ_VERSION_COALESCE, Some("auto"))], async { + let calls = disk_call_counters::observe(object); + let (metadata, errs) = SetDisks::read_all_fileinfo(&disks, "", bucket, object, "", false, false, false) + .await + .expect("default metadata read should resolve"); + + assert_eq!(metadata.iter().filter(|fi| fi.name == object).count(), DISKS); + assert!(errs.iter().all(Option::is_none)); + assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), DISKS as u64); + assert_eq!( + calls.total(disk_call_counters::KIND_BATCH_READ_VERSION), + 0, + "non-GET metadata paths must bypass coalescer even when the env gate is enabled" + ); + }) + .await; + + drop(dirs); + } + + #[test] + fn batch_read_version_response_mapping_preserves_index_and_errors() { + let expected_items = vec![ + BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-a".to_string(), + version_id: "v-a".to_string(), + }, + BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-b".to_string(), + version_id: "v-b".to_string(), + }, + BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-c".to_string(), + version_id: "v-c".to_string(), + }, + ]; + let ok_file_info = FileInfo { + name: "object-a".to_string(), + ..Default::default() + }; + let responses = vec![ + BatchReadVersionResp { + index: 2, + path: "object-c".to_string(), + version_id: "v-c".to_string(), + success: false, + file_info: FileInfo::default(), + error: "disk read failed".to_string(), + error_code: 0, + }, + BatchReadVersionResp { + index: 0, + path: "object-a".to_string(), + version_id: "v-a".to_string(), + success: true, + file_info: ok_file_info, + error: String::new(), + error_code: 0, + }, + ]; + + let mut results = map_batch_read_version_responses(&expected_items, responses).into_iter(); + let first = results + .next() + .expect("slot 0 should exist") + .expect("slot 0 should map the success response by index"); + assert_eq!(first.name, "object-a"); + + let missing = results + .next() + .expect("slot 1 should exist") + .expect_err("slot 1 should stay missing"); + assert!( + missing.to_string().contains("response missing"), + "unexpected missing response error: {missing}" + ); + + let failed = results + .next() + .expect("slot 2 should exist") + .expect_err("slot 2 should map the response error"); + assert!(failed.to_string().contains("disk read failed"), "unexpected per-item error: {failed}"); + assert!(results.next().is_none()); + } + + #[test] + fn batch_read_version_response_mapping_preserves_typed_not_found_errors() { + let expected_items = vec![ + BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-a".to_string(), + version_id: "v-a".to_string(), + }, + BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-b".to_string(), + version_id: "v-b".to_string(), + }, + ]; + let results = map_batch_read_version_responses( + &expected_items, + vec![ + BatchReadVersionResp { + index: 0, + path: "object-a".to_string(), + version_id: "v-a".to_string(), + success: false, + file_info: FileInfo::default(), + error: DiskError::FileNotFound.to_string(), + error_code: DiskError::FileNotFound.to_u32(), + }, + BatchReadVersionResp { + index: 1, + path: "object-b".to_string(), + version_id: "v-b".to_string(), + success: false, + file_info: FileInfo::default(), + error: DiskError::FileVersionNotFound.to_string(), + error_code: DiskError::FileVersionNotFound.to_u32(), + }, + ], + ); + + assert!(matches!(results.first().expect("slot 0 should exist"), Err(DiskError::FileNotFound))); + assert!(matches!( + results.get(1).expect("slot 1 should exist"), + Err(DiskError::FileVersionNotFound) + )); + } + + #[test] + fn batch_read_version_response_mapping_rejects_identity_mismatch_and_duplicate_index() { + let expected_items = vec![BatchReadVersionItem { + org_volume: String::new(), + volume: "bucket".to_string(), + path: "object-a".to_string(), + version_id: "v-a".to_string(), + }]; + let mismatched = map_batch_read_version_responses( + &expected_items, + vec![BatchReadVersionResp { + index: 0, + path: "object-b".to_string(), + version_id: "v-a".to_string(), + success: true, + file_info: FileInfo { + name: "object-b".to_string(), + ..Default::default() + }, + error: String::new(), + error_code: 0, + }], + ) + .pop() + .expect("slot 0 should exist") + .expect_err("identity mismatch should fail closed"); + assert!( + mismatched.to_string().contains("identity mismatch"), + "unexpected mismatch error: {mismatched}" + ); + + let duplicate = map_batch_read_version_responses( + &expected_items, + vec![ + BatchReadVersionResp { + index: 0, + path: "object-a".to_string(), + version_id: "v-a".to_string(), + success: true, + file_info: FileInfo { + name: "object-a".to_string(), + ..Default::default() + }, + error: String::new(), + error_code: 0, + }, + BatchReadVersionResp { + index: 0, + path: "object-a".to_string(), + version_id: "v-a".to_string(), + success: true, + file_info: FileInfo { + name: "object-a".to_string(), + ..Default::default() + }, + error: String::new(), + error_code: 0, + }, + ], + ) + .pop() + .expect("slot 0 should exist") + .expect_err("duplicate response index should fail closed"); + assert!( + duplicate.to_string().contains("duplicate index"), + "unexpected duplicate error: {duplicate}" + ); + } + /// Isolation guard: unobserved objects record nothing (so parallel tests do /// not inflate one another), and a scope clears its own counts on drop. #[tokio::test] diff --git a/crates/ecstore/src/set_disk/mod.rs b/crates/ecstore/src/set_disk/mod.rs index 5625d7583..0422ba94e 100644 --- a/crates/ecstore/src/set_disk/mod.rs +++ b/crates/ecstore/src/set_disk/mod.rs @@ -97,7 +97,7 @@ use crate::storage_api_contracts::{ CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations as _, MultipartUploadResult, PartInfo, }, namespace::NamespaceLocking as _, - object::{DeletedObject, HTTPPreconditions, ObjectIO as _, ObjectOperations as _, ObjectToDelete}, + object::{DeleteAccounting, DeletedObject, HTTPPreconditions, ObjectIO as _, ObjectOperations as _, ObjectToDelete}, range::HTTPRangeSpec, }; use crate::store::utils::is_reserved_or_invalid_bucket; @@ -735,8 +735,12 @@ pub(crate) use core::io_primitives::disk_call_counters; mod ctx; mod metadata; mod ops; +#[cfg(test)] +pub(crate) use ops::multipart::NewMultipartUploadCommitObservation; #[cfg(any(test, feature = "test-util"))] pub use ops::multipart::{MultipartCommitBarrier, MultipartCommitPause}; +#[cfg(test)] +pub(crate) use ops::object::DeleteObjectCommitBarrier; #[cfg(feature = "test-util")] pub(crate) use ops::object::TransitionCleanupStoreBarrier as SetDiskTransitionCleanupStoreBarrier; pub(crate) use ops::object::body_cache_plaintext_len; @@ -922,14 +926,10 @@ mod prepared_get_object_metadata_tests { .expect("test should find an object whose initial fanout covers both data shards") } - #[allow( - dead_code, - reason = "test fixture no assertion in this module uses today; the live namesake lives in io_primitives tests (backlog#1823)" - )] - fn bounded_spare_disk_index(bucket: &str, object: &str) -> usize { + fn bounded_initial_parity_disk_index(bucket: &str, object: &str) -> usize { *bounded_metadata_fanout_order(bucket, object, 4, 2) - .get(3) - .expect("4-disk test geometry should leave one bounded spare disk") + .get(2) + .expect("4-disk test geometry should schedule one parity disk initially") } #[tokio::test] @@ -1087,7 +1087,7 @@ mod prepared_get_object_metadata_tests { ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", None::<&str>), ], async { - let slow_parity_disk = bounded_spare_disk_index(bucket, &object); + let slow_parity_disk = bounded_initial_parity_disk_index(bucket, &object); let barrier = rename_fanout_barrier::arm(&object, slow_parity_disk, rename_fanout_barrier::PHASE_READ_VERSION); let calls = disk_call_counters::observe(&object); @@ -3029,6 +3029,16 @@ pub struct SetDisks { storage_class_config_override: Arc>>>, } +// DistributedLock sends the raw ObjectKey to its clients; LockRegistry clones +// each endpoint's canonical Arc, so an exact Arc set identifies the lock domain. +pub(crate) fn same_distributed_lock_domain(left: &[Arc], right: &[Arc]) -> bool { + left.iter() + .all(|left_client| right.iter().any(|right_client| Arc::ptr_eq(left_client, right_client))) + && right + .iter() + .all(|right_client| left.iter().any(|left_client| Arc::ptr_eq(left_client, right_client))) +} + const ERASURE_CACHE_MAX_ENTRIES: usize = 32; #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] @@ -3604,6 +3614,15 @@ impl SetDisks { &self.ctx } + /// Whether both sets' namespace-lock implementations cover the same object key. + pub(crate) async fn shares_namespace_lock_domain(&self, other: &Self) -> bool { + match (self.ctx.is_dist_erasure().await, other.ctx.is_dist_erasure().await) { + (false, false) => Arc::ptr_eq(&self.local_lock_manager, &other.local_lock_manager), + (true, true) => same_distributed_lock_domain(&self.lockers, &other.lockers), + _ => false, + } + } + /// The lock manager this set actually uses (test-only; Phase 5 Slice 3). #[cfg(test)] pub(crate) fn local_lock_manager_for_test(&self) -> &Arc { @@ -4588,11 +4607,11 @@ fn should_preserve_delete_replication_state(opts: &ObjectOptions) -> bool { } fn should_force_delete_marker_for_missing_version(opts: &ObjectOptions) -> bool { - opts.delete_marker || (opts.versioned && opts.version_id.is_none() && !opts.data_movement) + opts.delete_marker || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.data_movement) } fn resolve_delete_version_state(opts: &ObjectOptions, goi: &ObjectInfo, version_found: bool) -> (bool, bool) { - let mut mark_delete = goi.version_id.is_some() || (opts.versioned && opts.version_id.is_none()); + let mut mark_delete = goi.version_id.is_some() || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none()); let mut delete_marker = opts.versioned; if opts.version_id.is_some() { diff --git a/crates/ecstore/src/set_disk/ops/multipart.rs b/crates/ecstore/src/set_disk/ops/multipart.rs index 0f418fa7f..5aa59b158 100644 --- a/crates/ecstore/src/set_disk/ops/multipart.rs +++ b/crates/ecstore/src/set_disk/ops/multipart.rs @@ -32,6 +32,8 @@ use crate::crash_inject::{self, CrashPoint}; use crate::multipart_listing::paginate_multipart_listing; use futures::{StreamExt, stream}; use std::future::Future; +#[cfg(test)] +use std::sync::atomic::AtomicBool; #[cfg(any(test, feature = "test-util"))] use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Duration; @@ -65,6 +67,7 @@ impl StaleMultipartCleanupGuard { #[cfg(any(test, feature = "test-util"))] #[derive(Clone, Copy, PartialEq, Eq)] pub enum MultipartCommitPause { + NewUploadBeforeLockLost, PutPartBeforeLockAcquire, PutPartBeforeLockLost, PutPartAfterRename, @@ -156,6 +159,72 @@ impl Drop for MultipartCommitBarrier { } } +#[cfg(test)] +struct NewMultipartUploadCommitObservationState { + bucket: String, + object: String, + committed: AtomicBool, +} + +#[cfg(test)] +pub(crate) struct NewMultipartUploadCommitObservation { + state: Arc, +} + +#[cfg(test)] +static NEW_MULTIPART_UPLOAD_COMMIT_OBSERVATION: std::sync::OnceLock< + std::sync::Mutex>>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +impl NewMultipartUploadCommitObservation { + pub(crate) fn install(bucket: &str, object: &str) -> Self { + let state = Arc::new(NewMultipartUploadCommitObservationState { + bucket: bucket.to_string(), + object: object.to_string(), + committed: AtomicBool::new(false), + }); + let mut slot = NEW_MULTIPART_UPLOAD_COMMIT_OBSERVATION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("new multipart upload commit observation mutex should not poison"); + assert!(slot.is_none(), "new multipart upload commit observation must be unique"); + *slot = Some(Arc::clone(&state)); + Self { state } + } + + pub(crate) fn committed(&self) -> bool { + self.state.committed.load(Ordering::Acquire) + } +} + +#[cfg(test)] +impl Drop for NewMultipartUploadCommitObservation { + fn drop(&mut self) { + let mut slot = NEW_MULTIPART_UPLOAD_COMMIT_OBSERVATION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("new multipart upload commit observation mutex should not poison"); + if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) { + *slot = None; + } + } +} + +#[cfg(test)] +fn observe_new_multipart_upload_commit(bucket: &str, object: &str) { + let state = NEW_MULTIPART_UPLOAD_COMMIT_OBSERVATION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("new multipart upload commit observation mutex should not poison") + .as_ref() + .filter(|state| state.bucket == bucket && state.object == object) + .cloned(); + if let Some(state) = state { + state.committed.store(true, Ordering::Release); + } +} + #[cfg(any(test, feature = "test-util"))] async fn pause_multipart_commit(bucket: &str, object: &str, pause: MultipartCommitPause) { let barrier = { @@ -1575,7 +1644,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { let parts_metadata = vec![fi.clone(); disks.len()]; if !user_defined.contains_key("content-type") { - // TODO: get content-type + // TODO(backlog): detect content-type from part data when header is missing } if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS) @@ -1615,6 +1684,30 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { let upload_path = Self::get_multipart_upload_dir(bucket, object, upload_uuid.as_str(), opts.data_movement); + #[cfg(any(test, feature = "test-util"))] + pause_multipart_commit(bucket, object, MultipartCommitPause::NewUploadBeforeLockLost).await; + if _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "new_multipart_upload_commit", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } + if opts + .namespace_lock_fence + .as_ref() + .is_some_and(NamespaceLockFence::is_lock_lost) + { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "new_multipart_upload_outer_lock", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; Self::write_unique_file_info( &shuffle_disks, @@ -1626,6 +1719,8 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { ) .await .map_err(|e| to_object_err(e.into(), vec![bucket, object]))?; + #[cfg(test)] + observe_new_multipart_upload_commit(bucket, object); // evalDisks @@ -1971,7 +2066,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { return Err(Error::InvalidPart(p.part_num, ext_part.etag.clone(), p.etag.clone().unwrap_or_default())); } - // TODO: crypto + // TODO(backlog): integrate encryption verification during complete multipart if (i < uploaded_parts.len() - 1) && !(opts.data_movement && ext_part.actual_size < 0) diff --git a/crates/ecstore/src/set_disk/ops/object.rs b/crates/ecstore/src/set_disk/ops/object.rs index 9ba7f8da7..bb3c9fbf9 100644 --- a/crates/ecstore/src/set_disk/ops/object.rs +++ b/crates/ecstore/src/set_disk/ops/object.rs @@ -45,6 +45,7 @@ use crate::bucket::replication::{ DeleteReplicationConfigSnapshot, ReplicationLifecycleBridge, ReplicationStatusType, VersionPurgeStatusType, replication_state_to_filemeta, replication_status_from_filemeta, version_purge_status_to_filemeta, }; +use crate::data_usage::quota_object_size; use crate::diagnostics::get::GetObjectFailureReason; use crate::disk::{DataDirDeleteStatus, OldCurrentSize}; use crate::error::is_err_invalid_upload_id; @@ -1293,7 +1294,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { (prepared.snapshot, prepared.object_info) } else { match self - .get_object_fileinfo( + .get_object_fileinfo_for_get_object_reader( bucket, object, opts, @@ -2122,11 +2123,11 @@ impl SetDisks { let erasure = Arc::new(erasure_from_file_info(&fi, false)?); let put_object_size = known_put_object_storage_size(data.size()); - let is_inline_buffer = - storage_class_config.should_inline(erasure.shard_file_size(put_object_size), erasure.data_shards, opts.versioned); + let shard_file_size_raw = erasure.shard_file_size(put_object_size); + let is_inline_buffer = storage_class_config.should_inline(shard_file_size_raw, erasure.data_shards, opts.versioned); let collect_stage_timing = rustfs_io_metrics::put_stage_metrics_enabled() || issue3031_diag_enabled(); - let shard_file_size = erasure.shard_file_size(put_object_size); + let shard_file_size = shard_file_size_raw; let shard_size = erasure.shard_size(); let write_path = classify_put_write_path(is_inline_buffer, put_object_size, fi.erasure.block_size); let direct_inline_commit = matches!(write_path, SmallWritePath::Inline); @@ -2483,6 +2484,7 @@ impl SetDisks { }) .await?, ); + notify_put_object_commit_namespace_acquired(bucket, object); } #[cfg(not(any(test, feature = "test-util")))] { @@ -4630,6 +4632,7 @@ struct PutObjectCommitBarrierState { arrived: tokio::sync::Notify, release: tokio::sync::Notify, namespace_pending: tokio::sync::Notify, + namespace_acquired: std::sync::atomic::AtomicBool, } #[cfg(any(test, feature = "test-util"))] @@ -4651,6 +4654,7 @@ impl PutObjectCommitBarrier { arrived: tokio::sync::Notify::new(), release: tokio::sync::Notify::new(), namespace_pending: tokio::sync::Notify::new(), + namespace_acquired: std::sync::atomic::AtomicBool::new(false), }); let mut slot = PUT_OBJECT_COMMIT_BARRIER .get_or_init(|| std::sync::Mutex::new(Vec::new())) @@ -4685,6 +4689,10 @@ impl PutObjectCommitBarrier { .await .expect("put object should wait for the namespace lock after leaving the commit barrier"); } + + pub fn namespace_acquired(&self) -> bool { + self.state.namespace_acquired.load(std::sync::atomic::Ordering::Acquire) + } } #[cfg(any(test, feature = "test-util"))] @@ -4741,6 +4749,22 @@ fn notify_put_object_commit_namespace_pending(bucket: &str, object: &str) { } } +#[cfg(any(test, feature = "test-util"))] +fn notify_put_object_commit_namespace_acquired(bucket: &str, object: &str) { + let barrier = PUT_OBJECT_COMMIT_BARRIER + .get_or_init(|| std::sync::Mutex::new(Vec::new())) + .lock() + .expect("put object commit barrier mutex should not poison") + .iter() + .find(|barrier| { + barrier.bucket == bucket && barrier.object == object && barrier.pause == PutObjectCommitPause::BeforeNamespace + }) + .cloned(); + if let Some(barrier) = barrier { + barrier.namespace_acquired.store(true, std::sync::atomic::Ordering::Release); + } +} + #[cfg(test)] struct DeleteObjectCommitBarrierState { bucket: String, @@ -4750,7 +4774,7 @@ struct DeleteObjectCommitBarrierState { } #[cfg(test)] -struct DeleteObjectCommitBarrier { +pub(crate) struct DeleteObjectCommitBarrier { state: Arc, } @@ -4760,7 +4784,7 @@ static DELETE_OBJECT_COMMIT_BARRIER: std::sync::OnceLock Self { + pub(crate) fn install(bucket: &str, object: &str) -> Self { let state = Arc::new(DeleteObjectCommitBarrierState { bucket: bucket.to_string(), object: object.to_string(), @@ -4776,13 +4800,13 @@ impl DeleteObjectCommitBarrier { Self { state } } - async fn wait_until_paused(&self) { + pub(crate) async fn wait_until_paused(&self) { tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified()) .await .expect("delete object should reach the deterministic commit barrier"); } - fn release(&self) { + pub(crate) fn release(&self) { self.state.release.notify_one(); } } @@ -5655,7 +5679,18 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { objects: Vec, opts: ObjectOptions, ) -> (Vec, Vec>) { + let (deleted, errors, _) = self.delete_objects_with_accounting(bucket, objects, opts).await; + (deleted, errors) + } + + async fn delete_objects_with_accounting( + &self, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + ) -> (Vec, Vec>, Vec>) { let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut accounting = vec![None; objects.len()]; let delete_config_snapshot = opts .delete_replication_config_snapshot .clone() @@ -5745,7 +5780,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { *item = Some(Error::other(message.clone())); } } - return (del_objects, del_errs); + return (del_objects, del_errs, accounting); } }, } @@ -5792,6 +5827,22 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let source_missing = gerr .as_ref() .is_some_and(|err| is_err_object_not_found(err) || is_err_version_not_found(err)); + // Resolve accounting from the generation selected under this + // object's write lock. A request-layer pre-stat is only an + // optimization and cannot identify a concurrent overwrite. + let (accounting_size, accounting_version_id, removed_current_object) = if source_missing + || dobj.synthetic_version_id + || set_disk_delete_creates_delete_marker(&check_opts) + || goi.delete_marker + { + (None, None, false) + } else { + ( + quota_object_size(&goi).ok(), + goi.version_id.filter(|version_id| !version_id.is_nil()), + (dobj.version_id.is_none() || is_explicit_null_version(dobj.version_id)) && !dobj.synthetic_version_id, + ) + }; // Normalize both sides before comparing. `goi.version_id` is the // client-facing identity, where `from_file_info` synthesizes // `Some(Uuid::nil())` for a null version on a versioned or @@ -5877,6 +5928,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { if dobj.version_id.is_none() && (version_suspended || versioned) { vr.mod_time = Some(OffsetDateTime::now_utc()); vr.deleted = true; + vr.mark_deleted = true; if versioned { vr.version_id = Some(Uuid::new_v4()); } @@ -5920,7 +5972,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { }, replication_state: vr.replication_state_internal.clone(), ..Default::default() - } + }; + accounting[i] = Some(DeleteAccounting { + size: accounting_size, + version_id: accounting_version_id, + removed_current_object, + }); } // Only add to vers_map if we hold the lock @@ -5966,7 +6023,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { }); } } - return (del_objects, del_errs); + return (del_objects, del_errs, accounting); } let mut persisted_journal_entries = Vec::with_capacity(journal_entries.len()); @@ -6161,7 +6218,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { join_all(rollback_futures).await; - // TODO: add_partial + // TODO(backlog): support partial object deletion for multi-part objects if let Some(api) = opts.tier_delete_journal_api.as_ref() { for (idx, je) in persisted_journal_entries { @@ -6204,7 +6261,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } } - (del_objects, del_errs) + // An accounting identity is actionable only when the delete result is + // successful. Never let a failed commit (including a partial quorum + // failure) reach the request-layer fast delta path. + for (index, err) in del_errs.iter().enumerate() { + if err.is_some() { + accounting[index] = None; + } + } + + (del_objects, del_errs, accounting) } #[tracing::instrument(skip(self))] @@ -6371,7 +6437,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } } - // TODO: Lifecycle + // TODO(backlog): integrate lifecycle evaluation before object deletion let mut version_found = true; // delete_object_version below derives its own majority quorum from the @@ -6465,7 +6531,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { mark_deleted: mark_delete, mod_time: Some(mod_time), replication_state_internal: opts.delete_replication.as_ref().map(replication_state_to_filemeta), - ..Default::default() // TODO: Transition + ..Default::default() // TODO(backlog): populate transition state on delete markers }; fi.set_tier_free_version_id(&find_vid.to_string()); @@ -6533,6 +6599,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let mut obj_info = ObjectInfo::from_file_info(&dfi, bucket, object, opts.versioned || opts.version_suspended); obj_info.size = goi.size; + // Keep the committed source metadata on the internal delete result so + // the request layer can derive canonical accounting for this exact + // generation. Delete responses do not expose these fields. + obj_info.actual_size = goi.actual_size; + obj_info.user_defined = Arc::clone(&goi.user_defined); + obj_info.parts = Arc::clone(&goi.parts); obj_info.user_tags = Arc::clone(&goi.user_tags); self.invalidate_get_object_metadata_cache(bucket, object).await; Ok(obj_info) @@ -7824,6 +7896,113 @@ mod replication_quota_safety_tests { assert_eq!(stored.get_actual_size().expect("stored logical size should parse"), 1); } + #[tokio::test] + async fn delete_returns_canonical_compressed_accounting_size() { + let (_temp_dirs, disks, set_disks) = hermetic_set_disks(4).await; + let bucket = "compressed-delete-accounting"; + for disk in &disks { + disk.make_volume(bucket).await.expect("bucket volume should be created"); + } + + let mut user_defined = HashMap::new(); + insert_str( + &mut user_defined, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + insert_str(&mut user_defined, SUFFIX_ACTUAL_SIZE, "1000".to_string()); + let mut reader = PutObjReader::new( + HashReader::from_stream(Cursor::new(vec![0x5a; 400]), 400, 1000, None, None, false) + .expect("compressed fixture reader should be valid"), + ); + set_disks + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + user_defined, + ..Default::default() + }, + ) + .await + .expect("compressed object should be written"); + + let (deleted, errors, accounting) = set_disks + .delete_objects_with_accounting( + bucket, + vec![ObjectToDelete { + object_name: "object".to_string(), + ..Default::default() + }], + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + + assert!(errors[0].is_none(), "compressed delete should succeed: {:?}", errors[0]); + assert!(deleted[0].found, "the committed object must be reported as found"); + assert_eq!(accounting[0].as_ref().and_then(|value| value.size), Some(1000)); + assert!(accounting[0].as_ref().is_some_and(|value| value.version_id.is_none())); + assert!(accounting[0].as_ref().is_some_and(|value| value.removed_current_object)); + } + + #[tokio::test] + async fn suspended_delete_marker_does_not_return_body_accounting() { + let (_temp_dirs, disks, set_disks) = hermetic_set_disks(4).await; + let bucket = "suspended-delete-accounting"; + for disk in &disks { + disk.make_volume(bucket).await.expect("bucket volume should be created"); + } + + let mut user_defined = HashMap::new(); + insert_str( + &mut user_defined, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + insert_str(&mut user_defined, SUFFIX_ACTUAL_SIZE, "1000".to_string()); + let mut reader = PutObjReader::new( + HashReader::from_stream(Cursor::new(vec![0x5a; 400]), 400, 1000, None, None, false) + .expect("compressed fixture reader should be valid"), + ); + let suspended_opts = ObjectOptions { + version_suspended: true, + delete_replication_config_snapshot: Some(Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test( + s3s::dto::VersioningConfiguration { + status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::SUSPENDED)), + ..Default::default() + }, + None, + ))), + user_defined, + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(ObjectLockConfigState::ConfirmedAbsent))), + ..Default::default() + }; + set_disks + .put_object(bucket, "object", &mut reader, &suspended_opts) + .await + .expect("compressed object should be written"); + + let (deleted, errors, accounting) = set_disks + .delete_objects_with_accounting( + bucket, + vec![ObjectToDelete { + object_name: "object".to_string(), + ..Default::default() + }], + suspended_opts, + ) + .await; + assert!(errors[0].is_none(), "suspended delete should create a marker: {:?}", errors[0]); + assert!(deleted[0].delete_marker); + assert!(accounting[0].is_none(), "a delete marker must not carry body accounting"); + } + #[tokio::test] async fn direct_put_cannot_persist_a_tiny_logical_size() { let (_temp_dirs, disks, set_disks) = hermetic_set_disks(4).await; @@ -11636,6 +11815,7 @@ mod transition_upload_integrity_tests { crate::data_movement::SourceCleanupBucketFence { expected_incarnation_id: None, lifecycle_guard: Some(&bucket_guard), + ..Default::default() }, "test_data_movement", ) diff --git a/crates/ecstore/src/set_disk/read.rs b/crates/ecstore/src/set_disk/read.rs index 2b556b143..15e7e4e2a 100644 --- a/crates/ecstore/src/set_disk/read.rs +++ b/crates/ecstore/src/set_disk/read.rs @@ -259,10 +259,33 @@ impl SetDisks { read_data: bool, caller_allows_early_stop: bool, ) -> Result { - self.get_object_fileinfo_gated(bucket, object, opts, read_data, caller_allows_early_stop) + self.get_object_fileinfo_gated_inner(bucket, object, opts, read_data, caller_allows_early_stop, false) .await } + #[tracing::instrument(level = "debug", skip(self))] + #[hotpath::measure(impl_type = "SetDisks")] + pub(super) async fn get_object_fileinfo_for_get_object_reader( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + read_data: bool, + caller_allows_early_stop: bool, + ) -> Result { + let allow_read_version_coalescing = !crate::bucket::utils::is_meta_bucketname(bucket) + && crate::runtime::global::get_metadata_read_version_coalescing_service_ready(); + self.get_object_fileinfo_gated_inner( + bucket, + object, + opts, + read_data, + caller_allows_early_stop, + allow_read_version_coalescing, + ) + .await + } + /// Like `get_object_fileinfo`, but `allow_early_stop=false` forces the full /// quorum fanout. Read-before-write callers (object tagging) must use this: /// the returned online-disk set is the write target, and the early-stop @@ -275,6 +298,20 @@ impl SetDisks { opts: &ObjectOptions, read_data: bool, allow_early_stop: bool, + ) -> Result { + self.get_object_fileinfo_gated_inner(bucket, object, opts, read_data, allow_early_stop, false) + .await + } + + #[allow(clippy::too_many_arguments)] + async fn get_object_fileinfo_gated_inner( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + read_data: bool, + allow_early_stop: bool, + allow_read_version_coalescing: bool, ) -> Result { let vid = opts.version_id.clone().unwrap_or_default(); let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); @@ -337,19 +374,34 @@ impl SetDisks { // read_all_fileinfo_observed (see read_all_fileinfo_early_stop in // core/io_primitives.rs); unsafe requests and callers that opt out // (allow_early_stop=false) fall back to full-wait. - let (mut parts_metadata, errs, metadata_fanout_diagnostics) = Self::read_all_fileinfo_observed( - &disks, - "", - bucket, - object, - vid.as_str(), - read_data, - false, - opts.incl_free_versions, - allow_early_stop, - self.default_parity_count, - ) - .await?; + let (mut parts_metadata, errs, metadata_fanout_diagnostics) = if allow_read_version_coalescing { + Self::read_all_fileinfo_observed_for_get_object( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await? + } else { + Self::read_all_fileinfo_observed( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + false, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await? + }; let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) { GET_OBJECT_PATH_INTERNAL_META } else { diff --git a/crates/ecstore/src/storage_api_contracts/mod.rs b/crates/ecstore/src/storage_api_contracts/mod.rs index 11e7b800f..78a3f9c2d 100644 --- a/crates/ecstore/src/storage_api_contracts/mod.rs +++ b/crates/ecstore/src/storage_api_contracts/mod.rs @@ -62,8 +62,8 @@ pub(crate) mod object { use super::{Debug, Error, FileInfo, GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader}; use crate::storage_api_contracts::range::HTTPRangeSpec; pub(crate) use rustfs_storage_api::{ - DeletedObject, HTTPPreconditions, ObjectIO, ObjectLockDeleteOptions, ObjectLockRetentionOptions, ObjectOperations, - ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState, ObjectToDelete, + DeleteAccounting, DeletedObject, HTTPPreconditions, ObjectIO, ObjectLockDeleteOptions, ObjectLockRetentionOptions, + ObjectOperations, ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState, ObjectToDelete, }; pub(crate) trait EcstoreObjectIO: diff --git a/crates/ecstore/src/store/bucket.rs b/crates/ecstore/src/store/bucket.rs index 1fac01899..77cdbc954 100644 --- a/crates/ecstore/src/store/bucket.rs +++ b/crates/ecstore/src/store/bucket.rs @@ -601,7 +601,7 @@ impl ECStore { #[instrument(skip(self))] pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result> { - // TODO: opts.cached + // TODO(backlog): support cached bucket listing via opts.cached let mut buckets = self.peer_sys.list_bucket(opts).await?; diff --git a/crates/ecstore/src/store/heal.rs b/crates/ecstore/src/store/heal.rs index d10abe740..a7aed758d 100644 --- a/crates/ecstore/src/store/heal.rs +++ b/crates/ecstore/src/store/heal.rs @@ -13,7 +13,12 @@ // limitations under the License. use super::*; +use crate::core::pools::POOL_META_NAME; +use crate::services::rebalance::{REBAL_META_NAME, RebalStatus}; +use crate::set_disk::get_lock_acquire_timeout; use crate::storage_api_contracts::heal::HealOperations as _; +use crate::storage_api_contracts::namespace::NamespaceLocking as _; +use rustfs_lock::NamespaceLockGuard; use tracing::trace; const LOG_COMPONENT_ECSTORE: &str = "ecstore"; @@ -30,7 +35,119 @@ fn invalid_heal_pool_index(pool_idx: usize, pool_count: usize) -> Error { ) } +#[derive(Debug, Clone, Copy)] +enum HealFormatPoolSkip { + Completed, + Retryable, +} + +fn classify_heal_format_pool( + pool_idx: usize, + pool_cmd_line: &str, + pool_meta: &PoolMeta, + rebalance_meta: Option<&RebalanceMeta>, +) -> Option { + let Some(pool) = pool_meta.pools.get(pool_idx) else { + return Some(HealFormatPoolSkip::Retryable); + }; + + if pool.id != pool_idx || pool_cmd_line.is_empty() || pool.cmd_line.is_empty() || pool.cmd_line != pool_cmd_line { + return Some(HealFormatPoolSkip::Retryable); + } + + if let Some(decommission) = pool.decommission.as_ref() { + if decommission.complete { + return Some(HealFormatPoolSkip::Completed); + } + if decommission.failed || decommission.canceled || decommission.queued || pool_meta.is_suspended(pool_idx) { + return Some(HealFormatPoolSkip::Retryable); + } + } + + if let Some(meta) = rebalance_meta { + let Some(pool_stats) = meta.pool_stats.get(pool_idx) else { + return Some(HealFormatPoolSkip::Retryable); + }; + if pool_stats.info.stopping || (pool_stats.participating && pool_stats.info.status == RebalStatus::Started) { + return Some(HealFormatPoolSkip::Retryable); + } + } + + None +} + +fn heal_format_pool_skip_error(skip: HealFormatPoolSkip) -> Error { + match skip { + HealFormatPoolSkip::Completed => StorageError::NoHealRequired, + HealFormatPoolSkip::Retryable => StorageError::SlowDown, + } +} + +fn heal_format_fence_lost_error() -> Error { + StorageError::SlowDown +} + impl ECStore { + async fn acquire_heal_format_fence( + &self, + ) -> Result<(NamespaceLockGuard, NamespaceLockGuard, PoolMeta, Option)> { + let metadata_pool = self + .pools + .first() + .cloned() + .ok_or_else(|| Error::other("heal format requires at least one storage pool"))?; + + // Metadata fence order is part of the decommission/rebalance protocol: + // pool.bin must always be acquired before rebalance.bin. + let pool_lock = metadata_pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?; + let pool_guard = pool_lock.get_write_lock(get_lock_acquire_timeout()).await?; + let rebalance_lock = metadata_pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?; + let rebalance_guard = rebalance_lock.get_write_lock(get_lock_acquire_timeout()).await?; + + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + return Err(heal_format_fence_lost_error()); + } + + let mut pool_meta = PoolMeta::default(); + pool_meta.load_no_lock(metadata_pool.clone()).await?; + if pool_meta.pools.len() != self.pools.len() + || pool_meta.pools.iter().enumerate().any(|(pool_idx, pool)| { + pool.id != pool_idx || pool.cmd_line.is_empty() || pool.cmd_line != self.pools[pool_idx].endpoints.cmd_line + }) + { + return Err(heal_format_fence_lost_error()); + } + + let mut rebalance_meta = RebalanceMeta::new(); + let rebalance_meta = match rebalance_meta + .load_with_opts( + metadata_pool, + ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(()) => Some(rebalance_meta), + Err(Error::ConfigNotFound) => None, + Err(err) => return Err(err), + }; + + if rebalance_meta + .as_ref() + .is_some_and(|meta| meta.pool_stats.len() != self.pools.len()) + { + return Err(heal_format_fence_lost_error()); + } + + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + return Err(heal_format_fence_lost_error()); + } + + Ok((pool_guard, rebalance_guard, pool_meta, rebalance_meta)) + } + fn get_pools_for_heal_object(&self, opts: &HealOpts) -> Result>> { match opts.pool { Some(pool_idx) => Ok(vec![ @@ -52,9 +169,26 @@ impl ECStore { }; let mut count_no_heal = 0; + let mut count_completed = 0; let mut first_error = None; - for pool in self.pools.iter() { - let (mut result, err) = pool.heal_format(dry_run).await?; + for (pool_idx, pool) in self.pools.iter().enumerate() { + let (pool_guard, rebalance_guard, pool_meta, rebalance_meta) = self.acquire_heal_format_fence().await?; + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + first_error.get_or_insert(heal_format_fence_lost_error()); + break; + } + if let Some(skip) = classify_heal_format_pool(pool_idx, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.as_ref()) + { + if matches!(skip, HealFormatPoolSkip::Completed) { + count_completed += 1; + } else { + first_error.get_or_insert(heal_format_pool_skip_error(skip)); + } + continue; + } + + let fence_lost = || pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost(); + let (mut result, err) = pool.heal_format_with_fence(dry_run, fence_lost).await?; if let Some(err) = err { match err { StorageError::NoHealRequired => { @@ -69,11 +203,18 @@ impl ECStore { r.set_count += result.set_count; r.before.drives.append(&mut result.before.drives); r.after.drives.append(&mut result.after.drives); + + // A lease can be lost after the final write; fail closed before + // reporting the pool as successfully healed. + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + first_error.get_or_insert(heal_format_fence_lost_error()); + break; + } } if let Some(err) = first_error { return Ok((r, Some(err))); } - if count_no_heal == self.pools.len() { + if count_no_heal + count_completed == self.pools.len() { info!( event = EVENT_HEAL_FORMAT_COMPLETED, component = LOG_COMPONENT_ECSTORE, @@ -297,10 +438,17 @@ impl ECStore { #[cfg(test)] mod tests { use super::*; + use crate::bucket::metadata_sys; use crate::core::pools::{PoolDecommissionInfo, PoolStatus}; - use crate::disk::{DiskOption, format::FormatV3, new_disk}; - use crate::layout::endpoints::{Endpoints, PoolEndpoints}; + use crate::disk::{DeleteOptions, DiskOption, format::FormatV3, new_disk}; + use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints}; + use crate::runtime::instance::InstanceContext; + use crate::services::rebalance::{RebalanceInfo, RebalanceStats}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations}; use crate::store::init_format::{load_format_erasure, save_format_file}; + use crate::store::init_local_disks_with_instance_ctx; + use tokio_util::sync::CancellationToken; async fn minimal_heal_pool(pool_idx: usize) -> Arc { let format = FormatV3::new(1, 1); @@ -347,6 +495,209 @@ mod tests { } } + fn pool_meta_with_decommission(info: PoolDecommissionInfo) -> PoolMeta { + PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: OffsetDateTime::UNIX_EPOCH, + decommission: Some(info), + }], + ..Default::default() + } + } + + #[test] + fn heal_format_pool_state_barriers_are_classified() { + let active = pool_meta_with_decommission(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }); + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &active, None), + Some(HealFormatPoolSkip::Retryable) + )); + + for info in [ + PoolDecommissionInfo { + failed: true, + ..Default::default() + }, + PoolDecommissionInfo { + canceled: true, + ..Default::default() + }, + ] { + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &pool_meta_with_decommission(info), None), + Some(HealFormatPoolSkip::Retryable) + )); + } + + let completed = pool_meta_with_decommission(PoolDecommissionInfo { + complete: true, + ..Default::default() + }); + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &completed, None), + Some(HealFormatPoolSkip::Completed) + )); + } + + #[test] + fn heal_format_pool_rebalance_barriers_and_identity_are_fail_closed() { + let identity_meta = pool_meta_with_decommission(PoolDecommissionInfo::default()); + let rebalance = RebalanceMeta { + pool_stats: vec![RebalanceStats { + participating: true, + info: RebalanceInfo { + status: RebalStatus::Started, + ..Default::default() + }, + ..Default::default() + }], + ..Default::default() + }; + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &identity_meta, Some(&rebalance)), + Some(HealFormatPoolSkip::Retryable) + )); + + let stopping = RebalanceMeta { + pool_stats: vec![RebalanceStats { + info: RebalanceInfo { + stopping: true, + ..Default::default() + }, + ..Default::default() + }], + ..Default::default() + }; + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &identity_meta, Some(&stopping)), + Some(HealFormatPoolSkip::Retryable) + )); + + let identity = pool_meta_with_decommission(PoolDecommissionInfo::default()); + assert!(matches!( + classify_heal_format_pool(0, "pool-new", &identity, None), + Some(HealFormatPoolSkip::Retryable) + )); + + let identity_without_decommission = PoolMeta { + pools: vec![PoolStatus { + id: 0, + cmd_line: "pool-0".to_string(), + last_update: OffsetDateTime::UNIX_EPOCH, + decommission: None, + }], + ..Default::default() + }; + assert!(matches!( + classify_heal_format_pool(0, "pool-new", &identity_without_decommission, None), + Some(HealFormatPoolSkip::Retryable) + )); + + assert!(matches!( + classify_heal_format_pool(0, "", &identity_meta, None), + Some(HealFormatPoolSkip::Retryable) + )); + + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &PoolMeta::default(), None), + Some(HealFormatPoolSkip::Retryable) + )); + + let stopped = RebalanceMeta { + stopped_at: Some(OffsetDateTime::UNIX_EPOCH), + pool_stats: vec![RebalanceStats { + participating: true, + info: RebalanceInfo { + status: RebalStatus::Stopped, + ..Default::default() + }, + ..Default::default() + }], + ..Default::default() + }; + assert!(classify_heal_format_pool(0, "pool-0", &identity_meta, Some(&stopped)).is_none()); + + let stopping_after_stop = RebalanceMeta { + stopped_at: Some(OffsetDateTime::UNIX_EPOCH), + pool_stats: vec![RebalanceStats { + participating: true, + info: RebalanceInfo { + status: RebalStatus::Started, + stopping: true, + ..Default::default() + }, + ..Default::default() + }], + ..Default::default() + }; + assert!(matches!( + classify_heal_format_pool(0, "pool-0", &identity_meta, Some(&stopping_after_stop)), + Some(HealFormatPoolSkip::Retryable) + )); + } + + #[test] + fn skipped_heal_format_pool_is_never_reported_as_success() { + assert!(matches!( + heal_format_pool_skip_error(HealFormatPoolSkip::Retryable), + StorageError::SlowDown + )); + assert!(matches!( + heal_format_pool_skip_error(HealFormatPoolSkip::Completed), + StorageError::NoHealRequired + )); + } + + async fn multi_pool_heal_store() -> (tempfile::TempDir, Arc, CancellationToken) { + let temp_dir = tempfile::tempdir().expect("multi-pool heal test directory should be created"); + let mut pool_endpoints = Vec::new(); + for pool_index in 0..2 { + let mut endpoints = Vec::new(); + for disk_index in 0..4 { + let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}")); + tokio::fs::create_dir_all(&disk_path) + .await + .expect("multi-pool heal test disk should be created"); + let mut endpoint = Endpoint::try_from(disk_path.to_str().expect("disk path should be utf8")) + .expect("test endpoint should parse"); + endpoint.set_pool_index(pool_index); + endpoint.set_set_index(0); + endpoint.set_disk_index(disk_index); + endpoints.push(endpoint); + } + pool_endpoints.push(PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 4, + endpoints: Endpoints::from(endpoints), + cmd_line: format!("heal-owner-pool-{pool_index}"), + platform: "test".to_string(), + }); + } + + let endpoint_pools = EndpointServerPools::from(pool_endpoints); + let instance_ctx = Arc::new(InstanceContext::new()); + init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone()) + .await + .expect("multi-pool local disks should initialize"); + let shutdown = CancellationToken::new(); + let store = ECStore::new_with_instance_ctx( + "127.0.0.1:0".parse().expect("test address should parse"), + endpoint_pools, + shutdown.clone(), + instance_ctx, + ) + .await + .expect("multi-pool test store should initialize"); + metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + (temp_dir, store, shutdown) + } + #[tokio::test] async fn heal_object_pool_scope_selects_only_requested_pool() { let store = minimal_heal_store().await; @@ -506,6 +857,229 @@ mod tests { } } + #[tokio::test] + #[serial_test::serial] + async fn unscoped_heal_object_suspended_owner_semantics() { + let (_temp_dir, store, shutdown) = multi_pool_heal_store().await; + let bucket = format!("heal-owner-{}", Uuid::new_v4().simple()); + let active_object = "active-owner"; + let suspended_only_object = "suspended-only"; + let duplicate_object = "duplicate-owner"; + let marker_object = "marker-owner"; + let quorum_object = "quorum-owner"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created in all pools"); + + let mut active_reader = PutObjReader::from_vec(b"active owner".to_vec()); + store.pools[0] + .put_object(&bucket, active_object, &mut active_reader, &ObjectOptions::default()) + .await + .expect("active owner object should be written"); + let active_disks = store.pools[0].disk_set[0].disks.read().await.clone(); + let missing_active_disk = active_disks[0].clone().expect("active disk should be online"); + missing_active_disk + .delete( + &bucket, + active_object, + DeleteOptions { + recursive: true, + immediate: true, + ..Default::default() + }, + ) + .await + .expect("active owner shard should be removed for repair"); + assert!( + missing_active_disk.read_xl(&bucket, active_object, false).await.is_err(), + "the active owner fixture must start with one missing metadata copy" + ); + + let mut suspended_reader = PutObjReader::from_vec(b"suspended owner".to_vec()); + store.pools[1] + .put_object(&bucket, suspended_only_object, &mut suspended_reader, &ObjectOptions::default()) + .await + .expect("suspended owner object should be written"); + for (pool_index, mod_time) in [1_i64, 2_i64].into_iter().enumerate() { + let mut duplicate_reader = PutObjReader::from_vec(format!("duplicate-pool-{pool_index}").into_bytes()); + store.pools[pool_index] + .put_object( + &bucket, + duplicate_object, + &mut duplicate_reader, + &ObjectOptions { + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(mod_time)), + ..Default::default() + }, + ) + .await + .expect("duplicate owner object should be written"); + } + let duplicate_missing_disk = store.pools[0].disk_set[0].disks.read().await[0] + .clone() + .expect("duplicate active owner disk should be online"); + duplicate_missing_disk + .delete( + &bucket, + duplicate_object, + DeleteOptions { + recursive: true, + immediate: true, + ..Default::default() + }, + ) + .await + .expect("duplicate active owner shard should be removed for repair"); + let history_version = Uuid::new_v4(); + let mut history_reader = PutObjReader::from_vec(b"marker history".to_vec()); + store.pools[0] + .put_object( + &bucket, + marker_object, + &mut history_reader, + &ObjectOptions { + versioned: true, + version_id: Some(history_version.to_string()), + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)), + ..Default::default() + }, + ) + .await + .expect("versioned marker history should be written"); + store.pools[0] + .delete_object( + &bucket, + marker_object, + ObjectOptions { + versioned: true, + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(2)), + ..Default::default() + }, + ) + .await + .expect("delete marker should be written"); + let mut quorum_reader = PutObjReader::from_vec(b"quorum boundary".to_vec()); + store.pools[0] + .put_object(&bucket, quorum_object, &mut quorum_reader, &ObjectOptions::default()) + .await + .expect("quorum boundary object should be written"); + { + let mut pool_meta = store.pool_meta.write().await; + let mut next = PoolMeta::new(&store.pools, &pool_meta); + next.pools[1].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }); + *pool_meta = next; + } + + let (_, duplicate_owner) = store + .get_latest_object_info_with_idx(&bucket, duplicate_object, &ObjectOptions::default()) + .await + .expect("duplicate owner should resolve"); + assert_eq!(duplicate_owner, 1, "latest duplicate must win when all pools are eligible"); + let (_, active_duplicate_owner) = store + .get_latest_object_info_with_idx( + &bucket, + duplicate_object, + &ObjectOptions { + skip_decommissioned: true, + ..Default::default() + }, + ) + .await + .expect("active duplicate owner should resolve"); + assert_eq!( + active_duplicate_owner, 0, + "suspended duplicate must be excluded from active owner selection" + ); + let (duplicate_result, duplicate_err) = store + .handle_heal_object(&bucket, duplicate_object, "", &HealOpts::default()) + .await + .expect("duplicate owner heal should complete through the production path"); + assert_eq!(duplicate_result.object, duplicate_object); + assert!(duplicate_err.is_none(), "active duplicate should be repaired: {duplicate_err:?}"); + assert!( + duplicate_missing_disk.read_xl(&bucket, duplicate_object, false).await.is_ok(), + "production heal must repair the active duplicate owner rather than the suspended owner" + ); + let (marker_info, marker_owner) = store + .get_latest_object_info_with_idx( + &bucket, + marker_object, + &ObjectOptions { + skip_decommissioned: true, + versioned: true, + ..Default::default() + }, + ) + .await + .expect("latest delete marker should resolve"); + assert_eq!(marker_owner, 0); + assert!(marker_info.delete_marker, "latest version must preserve delete-marker semantics"); + + let (active_result, active_err) = store + .handle_heal_object(&bucket, active_object, "", &HealOpts::default()) + .await + .expect("unscoped active-owner heal should complete"); + assert_eq!(active_result.object, active_object); + assert!(active_err.is_none(), "active owner must be selected even with a suspended pool"); + assert!( + missing_active_disk.read_xl(&bucket, active_object, false).await.is_ok(), + "active owner heal must write the missing disk metadata: result={active_result:?}, err={active_err:?}" + ); + assert!( + store.pools[1] + .get_object_info(&bucket, active_object, &ObjectOptions::default()) + .await + .is_err(), + "the suspended pool must not be written for an active-owner object" + ); + + let (suspended_result, suspended_err) = store + .handle_heal_object(&bucket, suspended_only_object, "", &HealOpts::default()) + .await + .expect("unscoped suspended-only heal should return a terminal result"); + assert!(suspended_result.object.is_empty()); + assert!(matches!(suspended_err, Some(Error::FileNotFound))); + assert!( + store.pools[1] + .get_object_info(&bucket, suspended_only_object, &ObjectOptions::default()) + .await + .is_ok(), + "suspended-only data must remain untouched when unscoped heal reports absent" + ); + + let (_, explicit_err) = store + .handle_heal_object( + &bucket, + suspended_only_object, + "", + &HealOpts { + pool: Some(1), + ..Default::default() + }, + ) + .await + .expect("explicit suspended-owner heal should return a mapped error"); + assert!(matches!(explicit_err, Some(Error::SlowDown))); + + let original_quorum_disks = store.pools[0].disk_set[0].disks.read().await.clone(); + let surviving_quorum_disk = original_quorum_disks[3].clone(); + *store.pools[0].disk_set[0].disks.write().await = vec![None, None, None, surviving_quorum_disk]; + let (_, quorum_err) = store + .handle_heal_object(&bucket, quorum_object, "", &HealOpts::default()) + .await + .expect("quorum boundary heal should return a mapped result"); + *store.pools[0].disk_set[0].disks.write().await = original_quorum_disks; + assert!( + matches!(quorum_err, Some(Error::ErasureReadQuorum)), + "quorum-boundary heal must preserve quorum error, got {quorum_err:?}" + ); + shutdown.cancel(); + } + #[tokio::test] async fn handle_heal_format_continues_after_a_pool_error() { let canonical_format = FormatV3::new(1, 3); @@ -615,6 +1189,18 @@ mod tests { bucket_fence_registry: std::sync::Arc::default(), }; + let err = store + .handle_heal_format(false) + .await + .expect_err("missing pool metadata must fail closed before format writes"); + assert!(matches!(err, StorageError::SlowDown)); + + let pool_meta = PoolMeta::new(&store.pools, &PoolMeta::default()); + pool_meta + .save(store.pools.clone()) + .await + .expect("pool metadata should be persisted before format heal"); + let (result, err) = store .handle_heal_format(false) .await @@ -628,5 +1214,22 @@ mod tests { .await .expect("the later pool should be healed despite the first pool error"); assert_eq!(healed.erasure.this, recoverable_format.erasure.sets[0][2]); + + let mut completed_meta = PoolMeta::new(&store.pools, &PoolMeta::default()); + for status in &mut completed_meta.pools { + status.decommission = Some(PoolDecommissionInfo { + complete: true, + ..Default::default() + }); + } + completed_meta + .save(store.pools.clone()) + .await + .expect("completed pool metadata should be persisted"); + let (_, err) = store + .handle_heal_format(false) + .await + .expect("completed pools should be reported as a no-op"); + assert!(matches!(err, Some(StorageError::NoHealRequired))); } } diff --git a/crates/ecstore/src/store/init.rs b/crates/ecstore/src/store/init.rs index e0452c5bd..451e42242 100644 --- a/crates/ecstore/src/store/init.rs +++ b/crates/ecstore/src/store/init.rs @@ -604,15 +604,15 @@ mod tests { storage_api_contracts::{ bucket::{BucketOperations as _, MakeBucketOptions}, multipart::MultipartOperations as _, - object::{ObjectIO, ObjectOperations as _}, + object::{ObjectIO, ObjectOperations as _, ObjectToDelete}, range::HTTPRangeSpec, }, }; use http::HeaderMap; use rustfs_config::server_config::KVS; - use rustfs_filemeta::ObjectPartInfo; #[cfg(feature = "test-util")] use rustfs_filemeta::{FileInfo, FileMeta}; + use rustfs_filemeta::{FileInfoVersions, MetaCacheEntry, ObjectPartInfo}; #[cfg(feature = "test-util")] use rustfs_protos::{TIER_MUTATION_RPC_PROTOCOL_VERSION, TierMutationRpcPhase}; use rustfs_rio::{Checksum, ChecksumType}; @@ -1226,6 +1226,212 @@ mod tests { shutdown.cancel(); } + async fn migrate_versioned_decommission_test_object( + store: &Arc, + bucket: &str, + object: &str, + payload: &[u8], + op_label: &'static str, + ) -> (uuid::Uuid, FileInfoVersions) { + let mut source = PutObjReader::from_vec(payload.to_vec()); + let source_info = store.pools[0] + .put_object( + bucket, + object, + &mut source, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("write versioned source to the pool being decommissioned"); + let source_version = source_info.version_id.expect("versioned source must have a version ID"); + let expected_source_versions = store.pools[0] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + .expect("source versions should be readable before migration") + .expect("source versions should exist before migration"); + { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.pools[0].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + } + + let barrier = crate::set_disk::PutObjectCommitBarrier::install( + bucket, + object, + crate::set_disk::PutObjectCommitPause::AfterNamespace, + ); + let migration_store = Arc::clone(store); + let migration_bucket = bucket.to_string(); + let migration_object = object.to_string(); + let migration = tokio::spawn(async move { + let source_reader = migration_store.pools[0] + .get_object_reader( + &migration_bucket, + &migration_object, + None, + HeaderMap::new(), + &ObjectOptions { + versioned: true, + version_id: Some(source_version.to_string()), + no_lock: true, + data_movement: true, + raw_data_movement_read: true, + ..Default::default() + }, + ) + .await?; + crate::data_movement::migrate_decommission_object(migration_store, 0, migration_bucket, source_reader, None, op_label) + .await + }); + barrier.wait_until_paused().await; + barrier.release(); + migration + .await + .expect("versioned decommission migration task should join") + .expect("versioned decommission migration should commit"); + + (source_version, expected_source_versions) + } + + async fn mark_test_pool_decommissioning(store: &Arc, pool_idx: usize) { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.pools[pool_idx].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + } + + async fn write_decommission_test_multipart_source( + store: &Arc, + pool_idx: usize, + bucket: &str, + object: &str, + ) { + let pool = &store.pools[pool_idx]; + let upload = pool + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create decommission multipart source upload"); + let first_part = vec![b'm'; 5 * 1024 * 1024]; + let second_part = b"decommission multipart tail".to_vec(); + let mut completed_parts = Vec::with_capacity(2); + for (part_number, body) in [(1, first_part), (2, second_part)] { + let mut reader = PutObjReader::from_vec(body); + let part = pool + .put_object_part(bucket, object, &upload.upload_id, part_number, &mut reader, &ObjectOptions::default()) + .await + .expect("write decommission multipart source part"); + completed_parts.push(crate::storage_api_contracts::multipart::CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }); + } + pool.clone() + .complete_multipart_upload(bucket, object, &upload.upload_id, completed_parts, &ObjectOptions::default()) + .await + .expect("complete decommission multipart source object"); + } + + async fn assert_pool_object_present(pool: &Arc, bucket: &str, object: &str) { + pool.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .expect("expected object generation must remain present"); + } + + async fn assert_pool_object_absent(pool: &Arc, bucket: &str, object: &str) { + let err = pool + .get_object_info(bucket, object, &ObjectOptions::default()) + .await + .expect_err("fenced decommission target must remain absent"); + assert!( + matches!(err, StorageError::ObjectNotFound(_, _) | StorageError::VersionNotFound(_, _, _)), + "unexpected fenced target result: {err:?}" + ); + } + + async fn write_suspended_decommission_source(store: &Arc, bucket: &str, object: &str) { + let mut reader = PutObjReader::from_vec(b"suspended source generation".to_vec()); + let source = store.pools[0] + .put_object( + bucket, + object, + &mut reader, + &ObjectOptions { + version_suspended: true, + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND), + ..Default::default() + }, + ) + .await + .expect("write suspended null source version"); + assert!( + source.version_id.is_none_or(|version_id| version_id.is_nil()), + "suspended source must use the null version identity" + ); + } + + async fn assert_suspended_null_source_present(store: &Arc, bucket: &str, object: &str) { + let versions = store.pools[0] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + .expect("suspended source versions should be readable") + .expect("suspended source must exist before worker convergence"); + assert!( + versions + .versions + .iter() + .any(|version| !version.deleted && version.version_id.is_none_or(|version_id| version_id.is_nil())), + "the source pool must retain its null data version while DELETE owns the fixed fence" + ); + } + + async fn assert_suspended_decommission_converged(store: &Arc, bucket: &str, object: &str) { + let source_versions = store.pools[0] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + .expect("source versions should remain readable after suspended convergence"); + assert!( + source_versions.is_none_or(|versions| versions.versions.is_empty()), + "worker convergence must remove only the decommissioned source null version" + ); + + let target_versions = store.pools[1] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + .expect("active target versions should be readable") + .expect("active target must retain the suspended DELETE marker"); + assert!( + matches!(target_versions.versions.as_slice(), [marker] if marker.deleted && marker.version_id.is_none_or(|version_id| version_id.is_nil())), + "active target must contain only its null delete marker: {target_versions:?}" + ); + + let err = store + .get_object_info( + bucket, + object, + &ObjectOptions { + version_suspended: true, + ..Default::default() + }, + ) + .await + .expect_err("the active null delete marker must hide the migrated source generation"); + assert!( + matches!(err, StorageError::ObjectNotFound(_, _)), + "unexpected suspended latest-object result: {err:?}" + ); + } + #[tokio::test] #[serial_test::serial(storage_class_env)] async fn tag_updates_skip_active_rebalance_source_pool() { @@ -2752,6 +2958,1150 @@ mod tests { .expect_err("suspended delete must remove the requested UUID version"); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn decommission_entry_carries_migration_and_cleanup_mutation_fences() { + let temp_dir = tempfile::tempdir().expect("create decommission delete-fence store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "decommission-delete-fence", + &[(2, 4), (1, 4)], + CancellationToken::new(), + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("decom-delete-fence-{}", uuid::Uuid::new_v4()); + let object = (0..128) + .map(|index| format!("object-{index}.bin")) + .find(|candidate| store.pools[0].get_disks_by_key(candidate).set_index == 1) + .expect("the deterministic object search should select source set 1"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create decommission delete-fence bucket"); + let mut source = PutObjReader::from_vec(b"source generation".to_vec()); + store.pools[0] + .put_object(&bucket, &object, &mut source, &ObjectOptions::default()) + .await + .expect("write source object to the pool being decommissioned"); + { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.pools[0].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + } + assert!(store.is_suspended(0).await, "pool 0 must be a suspended decommission source"); + + let barrier = crate::set_disk::PutObjectCommitBarrier::install( + &bucket, + &object, + crate::set_disk::PutObjectCommitPause::AfterNamespace, + ); + let cleanup_barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(&bucket, &object); + let source_set = store.pools[0].get_disks_by_key(&object); + assert_eq!(source_set.set_index, 1, "the source entry must exercise the non-fixed set cleanup lock"); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker_object = object.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: worker_object, + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + barrier.wait_until_paused().await; + + let delete_barrier = crate::store::object::DeleteAfterObjectLockSnapshotBarrier::install(&bucket); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete_object = object.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_object(&delete_bucket, &delete_object, ObjectOptions::default()) + .await + }); + delete_barrier.wait_until_paused().await; + delete_barrier.release_and_wait_until_namespace_pending().await; + assert!( + !delete_barrier.namespace_acquired() && !delete.is_finished(), + "DELETE must remain before namespace acquisition behind the decommission worker's target-commit mutation fence" + ); + + barrier.release(); + cleanup_barrier.wait_until_paused().await; + drop(barrier); + + let fixed_set = Arc::clone(&store.pools[0].disk_set[0]); + let fixed_mutation_barrier = crate::set_disk::PutObjectCommitBarrier::install( + &bucket, + &object, + crate::set_disk::PutObjectCommitPause::BeforeNamespace, + ); + let mutation_bucket = bucket.clone(); + let mutation_object = object.clone(); + let fixed_mutation = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(b"fixed-domain replacement".to_vec()); + fixed_set + .put_object(&mutation_bucket, &mutation_object, &mut reader, &ObjectOptions::default()) + .await + }); + fixed_mutation_barrier.wait_until_paused().await; + fixed_mutation_barrier.release_and_wait_until_namespace_pending().await; + assert!( + !fixed_mutation_barrier.namespace_acquired() && !fixed_mutation.is_finished(), + "the source cleanup must retain the fixed mutation fence before the set-0 mutation acquires its namespace" + ); + fixed_mutation.abort(); + assert!( + fixed_mutation + .await + .expect_err("the fixed-domain mutation should be canceled") + .is_cancelled(), + "the competing fixed-domain mutation must remain cancelable while blocked" + ); + drop(fixed_mutation_barrier); + + cleanup_barrier.release(); + worker + .await + .expect("decommission entry worker should join") + .expect("decommission entry should migrate and clean its source"); + delete + .await + .expect("DELETE task should join") + .expect("DELETE should remove the source and migrated target generations"); + + for pool in &store.pools { + let err = pool + .get_object_info(&bucket, &object, &ObjectOptions::default()) + .await + .expect_err("DELETE must remove the source and migrated target copies"); + assert!( + matches!(err, StorageError::ObjectNotFound(_, _)), + "unexpected post-delete pool result: {err:?}" + ); + } + let err = store + .get_object_info(&bucket, &object, &ObjectOptions::default()) + .await + .expect_err("the deleted generation must not become visible again"); + assert!( + matches!(err, StorageError::ObjectNotFound(_, _)), + "unexpected post-delete store result: {err:?}" + ); + + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn decommission_outer_fence_loss_blocks_target_put_commit() { + let temp_dir = tempfile::tempdir().expect("create decommission PUT fence-loss store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "decommission-put-fence-loss", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("decom-put-fence-loss-{}", uuid::Uuid::new_v4()); + let object = "ordinary.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create decommission PUT fence-loss bucket"); + let mut source = PutObjReader::from_vec(b"source generation".to_vec()); + store.pools[0] + .put_object(&bucket, object, &mut source, &ObjectOptions::default()) + .await + .expect("write decommission PUT source"); + mark_test_pool_decommissioning(&store, 0).await; + + let loss_hook = crate::store::object::DecommissionMutationFenceLossHook::install( + &bucket, + object, + crate::store::object::DecommissionMutationFenceTestPhase::Migration, + ); + let barrier = crate::set_disk::PutObjectCommitBarrier::install( + &bucket, + object, + crate::set_disk::PutObjectCommitPause::BeforeQuotaRename, + ); + let source_set = store.pools[0].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + + barrier.wait_until_paused().await; + loss_hook.mark_lost(); + barrier.release(); + drop(barrier); + worker + .await + .expect("decommission PUT fence-loss worker should join") + .expect("a fenced migration failure should remain retryable at entry scope"); + + assert_pool_object_absent(&store.pools[1], &bucket, object).await; + assert_pool_object_present(&store.pools[0], &bucket, object).await; + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn decommission_outer_fence_loss_blocks_multipart_commits() { + let temp_dir = tempfile::tempdir().expect("create decommission multipart fence-loss store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "decommission-multipart-fence-loss", &[4, 4])) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("decom-mpu-fence-loss-{}", uuid::Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create decommission multipart fence-loss bucket"); + for object in ["new-upload.bin", "complete.bin"] { + write_decommission_test_multipart_source(&store, 0, &bucket, object).await; + } + mark_test_pool_decommissioning(&store, 0).await; + + for (object, pause) in [ + ("new-upload.bin", crate::set_disk::MultipartCommitPause::NewUploadBeforeLockLost), + ("complete.bin", crate::set_disk::MultipartCommitPause::BeforeLockLost), + ] { + let loss_hook = crate::store::object::DecommissionMutationFenceLossHook::install( + &bucket, + object, + crate::store::object::DecommissionMutationFenceTestPhase::Migration, + ); + let commit_observation = (pause == crate::set_disk::MultipartCommitPause::NewUploadBeforeLockLost) + .then(|| crate::set_disk::NewMultipartUploadCommitObservation::install(&bucket, object)); + let barrier = crate::set_disk::MultipartCommitBarrier::install(&bucket, object, pause); + let source_set = store.pools[0].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + + barrier.wait_until_paused().await; + loss_hook.mark_lost(); + barrier.release(); + drop(barrier); + worker + .await + .expect("decommission multipart fence-loss worker should join") + .expect("a fenced multipart migration failure should remain retryable at entry scope"); + + if let Some(commit_observation) = commit_observation { + assert!( + !commit_observation.committed(), + "new multipart upload metadata must not commit after the outer fence is lost" + ); + } + assert_pool_object_absent(&store.pools[1], &bucket, object).await; + assert_pool_object_present(&store.pools[0], &bucket, object).await; + let uploads = store.pools[1] + .list_multipart_uploads(&bucket, object, None, None, None, 100) + .await + .expect("list target multipart uploads after fenced migration"); + assert!(uploads.uploads.is_empty(), "fenced multipart migration must not retain target staging"); + } + + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn decommission_outer_fence_loss_blocks_source_cleanup_delete_commit() { + let temp_dir = tempfile::tempdir().expect("create decommission cleanup fence-loss store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "decommission-cleanup-fence-loss", &[4, 4])) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("decom-cleanup-fence-loss-{}", uuid::Uuid::new_v4()); + let object = "cleanup.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create decommission cleanup fence-loss bucket"); + let mut source = PutObjReader::from_vec(b"source generation".to_vec()); + store.pools[0] + .put_object(&bucket, object, &mut source, &ObjectOptions::default()) + .await + .expect("write decommission cleanup source"); + mark_test_pool_decommissioning(&store, 0).await; + + let loss_hook = crate::store::object::DecommissionMutationFenceLossHook::install( + &bucket, + object, + crate::store::object::DecommissionMutationFenceTestPhase::SourceCleanup, + ); + let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(&bucket, object); + let source_set = store.pools[0].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + + barrier.wait_until_paused().await; + loss_hook.mark_lost(); + barrier.release(); + drop(barrier); + let err = worker + .await + .expect("decommission cleanup fence-loss worker should join") + .expect_err("source cleanup must fail after its outer fence is lost"); + assert!( + err.to_string().contains("delete_object_commit"), + "cleanup failure must come from the delete commit fence: {err:?}" + ); + + assert_pool_object_present(&store.pools[0], &bucket, object).await; + assert_pool_object_present(&store.pools[1], &bucket, object).await; + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn reverse_decommission_reuses_fixed_target_fence_for_put_and_multipart() { + let temp_dir = tempfile::tempdir().expect("create reverse decommission store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "reverse-decommission-fixed-target", + &[(1, 4), (1, 4)], + CancellationToken::new(), + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("reverse-decom-fixed-target-{}", uuid::Uuid::new_v4()); + let object = "ordinary.bin"; + let object_body = b"reverse ordinary generation".to_vec(); + let multipart_object = "multipart.bin"; + let first_part = vec![b'm'; 5 * 1024 * 1024]; + let second_part = b"reverse multipart tail".to_vec(); + let mut multipart_body = first_part.clone(); + multipart_body.extend_from_slice(&second_part); + + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create reverse decommission bucket"); + let mut source = PutObjReader::from_vec(object_body.clone()); + store.pools[1] + .put_object(&bucket, object, &mut source, &ObjectOptions::default()) + .await + .expect("write ordinary source object to pool 1"); + + let upload = store.pools[1] + .new_multipart_upload(&bucket, multipart_object, &ObjectOptions::default()) + .await + .expect("create source multipart upload in pool 1"); + let mut completed_parts = Vec::with_capacity(2); + for (part_number, bytes) in [(1, first_part.as_slice()), (2, second_part.as_slice())] { + let mut reader = PutObjReader::from_vec(bytes.to_vec()); + let part = store.pools[1] + .put_object_part( + &bucket, + multipart_object, + &upload.upload_id, + part_number, + &mut reader, + &ObjectOptions::default(), + ) + .await + .expect("write source multipart part"); + completed_parts.push(crate::storage_api_contracts::multipart::CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }); + } + store.pools[1] + .clone() + .complete_multipart_upload(&bucket, multipart_object, &upload.upload_id, completed_parts, &ObjectOptions::default()) + .await + .expect("complete source multipart object in pool 1"); + + { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.pools[1].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + } + assert!(store.is_suspended(1).await, "pool 1 must be the reverse decommission source"); + + let commit_barrier = crate::set_disk::PutObjectCommitBarrier::install( + &bucket, + object, + crate::set_disk::PutObjectCommitPause::AfterNamespace, + ); + let source_set = store.pools[1].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 1, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + commit_barrier.wait_until_paused().await; + + let delete_barrier = crate::store::object::DeleteAfterObjectLockSnapshotBarrier::install(&bucket); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_object(&delete_bucket, object, ObjectOptions::default()) + .await + }); + delete_barrier.wait_until_paused().await; + delete_barrier.release_and_wait_until_namespace_pending().await; + assert!( + !delete_barrier.namespace_acquired() && !delete.is_finished(), + "the reverse target commit must keep DELETE behind the fixed read fence" + ); + delete.abort(); + assert!( + delete + .await + .expect_err("the blocked DELETE should be canceled") + .is_cancelled(), + "canceling the blocked DELETE must not mutate either pool" + ); + drop(delete_barrier); + + commit_barrier.release(); + drop(commit_barrier); + tokio::time::timeout(Duration::from_secs(60), worker) + .await + .expect("reverse ordinary decommission must not self-deadlock on the fixed target set") + .expect("reverse ordinary decommission worker should join") + .expect("reverse ordinary decommission should complete"); + + let mut ordinary_reader = store.pools[0] + .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("read the ordinary object from the fixed target set"); + let mut ordinary_target_body = Vec::new(); + ordinary_reader + .stream + .read_to_end(&mut ordinary_target_body) + .await + .expect("drain the ordinary target body"); + assert_eq!(ordinary_target_body, object_body, "ordinary migration must preserve the full body"); + let ordinary_source_err = store.pools[1] + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect_err("ordinary source generation must be cleaned after migration"); + assert!(matches!(ordinary_source_err, StorageError::ObjectNotFound(_, _))); + + let multipart_source_set = store.pools[1].get_disks_by_key(multipart_object); + let multipart_store = Arc::clone(&store); + let multipart_bucket = bucket.clone(); + let multipart_worker = tokio::spawn(async move { + multipart_store + .decommission_entry_for_test( + 1, + MetaCacheEntry { + name: multipart_object.to_string(), + ..Default::default() + }, + multipart_bucket, + multipart_source_set, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(60), multipart_worker) + .await + .expect("reverse multipart decommission must not self-deadlock on new or complete") + .expect("reverse multipart decommission worker should join") + .expect("reverse multipart decommission should complete"); + + let target_info = store.pools[0] + .get_object_info(&bucket, multipart_object, &ObjectOptions::default()) + .await + .expect("read migrated multipart metadata from the fixed target set"); + assert!(target_info.is_multipart(), "migration must retain multipart identity"); + let mut multipart_reader = store.pools[0] + .get_object_reader(&bucket, multipart_object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("read migrated multipart object from the fixed target set"); + let mut multipart_target_body = Vec::new(); + multipart_reader + .stream + .read_to_end(&mut multipart_target_body) + .await + .expect("drain the multipart target body"); + assert_eq!(multipart_target_body, multipart_body, "multipart migration must preserve the full body"); + let multipart_source_err = store.pools[1] + .get_object_info(&bucket, multipart_object, &ObjectOptions::default()) + .await + .expect_err("multipart source generation must be cleaned after migration"); + assert!(matches!(multipart_source_err, StorageError::ObjectNotFound(_, _))); + + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn batch_delete_real_path_preserves_source_pool_errors_in_any_pool_order() { + let temp_dir = tempfile::tempdir().expect("create batch delete pool-error store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "batch-delete-pool-errors", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + for source_pool_idx in [0, 1] { + { + let mut pool_meta = store.pool_meta.write().await; + for pool in &mut pool_meta.pools { + pool.decommission = None; + } + } + + let bucket = format!("batch-del-pool-error-{source_pool_idx}-{}", uuid::Uuid::new_v4()); + let object_names = vec![ + format!("third-{source_pool_idx}.bin"), + format!("first-{source_pool_idx}.bin"), + format!("second-{source_pool_idx}.bin"), + ]; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create batch delete pool-error bucket"); + for pool in &store.pools { + for object_name in &object_names { + let mut reader = PutObjReader::from_vec(format!("pool {} {object_name}", pool.pool_idx).into_bytes()); + pool.put_object(&bucket, object_name, &mut reader, &ObjectOptions::default()) + .await + .expect("seed each object in both the source and active pools"); + } + } + { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.pools[source_pool_idx].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + } + assert!( + store.is_suspended(source_pool_idx).await, + "the injected error pool must be the decommission source" + ); + + let expected_errors = [ + StorageError::ErasureWriteQuorum, + StorageError::NamespaceLockQuorumUnavailable { + mode: "delete_objects_commit", + bucket: bucket.clone(), + object: object_names[1].clone(), + required: 3, + achieved: 2, + }, + StorageError::ErasureWriteQuorum, + ]; + let injection = crate::store::object::BatchDeletePoolErrorInjection::install( + &bucket, + source_pool_idx, + object_names.iter().cloned().zip(expected_errors.iter().cloned()).collect(), + ); + let requests = object_names + .iter() + .map(|object_name| ObjectToDelete { + object_name: object_name.clone(), + ..Default::default() + }) + .collect(); + + let (deleted, errors) = store.delete_objects(&bucket, requests, ObjectOptions::default()).await; + + assert_eq!( + injection.observed(), + object_names.len(), + "the source pool must first complete every real delete" + ); + assert_eq!( + errors, + expected_errors.iter().cloned().map(Some).collect::>(), + "a successful pool must not clear a source pool failure at any request index" + ); + assert_eq!( + deleted.iter().map(|object| object.object_name.as_str()).collect::>(), + object_names.iter().map(String::as_str).collect::>(), + "DeleteObjects must preserve request index mapping while aggregating pool failures" + ); + assert!( + deleted.iter().all(|object| object.found), + "the injected source results must retain real delete success data" + ); + + for pool in &store.pools { + for object_name in &object_names { + let error = pool + .get_object_info(&bucket, object_name, &ObjectOptions::default()) + .await + .expect_err("both the active and source pool delete calls must execute"); + assert!( + matches!(error, StorageError::ObjectNotFound(_, _)), + "unexpected residual object: {error:?}" + ); + } + } + drop(injection); + } + + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn decommission_source_cleanup_holds_hashed_set_lock_across_preflight() { + let temp_dir = tempfile::tempdir().expect("create multi-set decommission cleanup store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "multi-set-decommission-source-cleanup", + &[(2, 4)], + CancellationToken::new(), + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("decom-source-cleanup-lock-{}", uuid::Uuid::new_v4()); + let object = (0..128) + .map(|index| format!("object-{index}.bin")) + .find(|candidate| store.pools[0].get_disks_by_key(candidate).set_index == 1) + .expect("the deterministic object search should select source set 1"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create multi-set decommission cleanup bucket"); + let mut source = PutObjReader::from_vec(b"source generation".to_vec()); + store.pools[0] + .put_object(&bucket, &object, &mut source, &ObjectOptions::default()) + .await + .expect("write the source generation to set 1"); + let source_set = store.pools[0].get_disks_by_key(&object); + assert_eq!(source_set.set_index, 1, "the source must not share the fixed set-0 namespace"); + let expected_source_versions = source_set + .load_file_info_versions_exact(&bucket, &object) + .await + .expect("source versions should be readable") + .expect("the source generation should exist"); + + let cleanup_barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(&bucket, &object); + let cleanup_store = Arc::clone(&store); + let cleanup_bucket = bucket.clone(); + let cleanup_object = object.clone(); + let cleanup = tokio::spawn(async move { + let mutation_fence = cleanup_store + .acquire_decommission_source_cleanup_fence(&cleanup_bucket, &cleanup_object, source_set.as_ref()) + .await?; + crate::data_movement::cleanup_source_entry_if_unchanged( + source_set, + &cleanup_bucket, + &cleanup_object, + &expected_source_versions, + &[], + crate::data_movement::SourceCleanupBucketFence { + object_mutation_fence: Some(&mutation_fence), + ..Default::default() + }, + "test_multi_set_decommission_source_cleanup", + ) + .await + }); + cleanup_barrier.wait_until_paused().await; + + let put_barrier = crate::set_disk::PutObjectCommitBarrier::install( + &bucket, + &object, + crate::set_disk::PutObjectCommitPause::BeforeNamespace, + ); + let mutation_pool = Arc::clone(&store.pools[0]); + let mutation_bucket = bucket.clone(); + let mutation_object = object.clone(); + let replacement = b"replacement generation".to_vec(); + let expected_replacement = replacement.clone(); + let mutation = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(replacement); + mutation_pool + .put_object(&mutation_bucket, &mutation_object, &mut reader, &ObjectOptions::default()) + .await + }); + put_barrier.wait_until_paused().await; + put_barrier.release_and_wait_until_namespace_pending().await; + assert!(!mutation.is_finished(), "a source mutation must wait behind cleanup's set-1 write lock"); + + cleanup_barrier.release(); + cleanup + .await + .expect("source cleanup task should join") + .expect("source cleanup should remove only the preflight generation"); + mutation + .await + .expect("source mutation task should join") + .expect("source mutation should commit after cleanup releases the set lock"); + + let mut reader = store.pools[0] + .get_object_reader(&bucket, &object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("the replacement generation must remain readable"); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("read the replacement generation"); + assert_eq!(actual, expected_replacement, "cleanup must not delete the replacement generation"); + + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn versioned_delete_marker_survives_decommission_source_cleanup() { + let temp_dir = tempfile::tempdir().expect("create versioned decommission delete-fence store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "versioned-decommission-delete-fence", &[4, 4])) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("versioned-decom-delete-{}", uuid::Uuid::new_v4()); + let object = "object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create versioned decommission delete-fence bucket"); + let (source_version, expected_source_versions) = migrate_versioned_decommission_test_object( + &store, + &bucket, + object, + b"source generation", + "test_versioned_decommission_delete_fence", + ) + .await; + + let delete_barrier = crate::store::object::VersionedDeleteMarkerCommitBarrier::install(&bucket, object); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_object( + &delete_bucket, + object, + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + }); + delete_barrier.wait_until_paused().await; + let cleanup_set = store.pools[0].get_disks_by_key(object); + crate::data_movement::ensure_source_cleanup_versions_unchanged( + Arc::clone(&cleanup_set), + &bucket, + object, + &expected_source_versions, + &[], + "test_versioned_decommission_delete_fence", + ) + .await + .expect("the committed delete marker must not be published to the suspended source pool"); + + let cleanup_delete_barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(&bucket, object); + let cleanup_store = Arc::clone(&store); + let cleanup_bucket = bucket.clone(); + let cleanup = tokio::spawn(async move { + let mutation_fence = cleanup_store + .acquire_decommission_source_cleanup_fence(&cleanup_bucket, object, cleanup_set.as_ref()) + .await?; + crate::data_movement::cleanup_source_entry_if_unchanged( + cleanup_set, + &cleanup_bucket, + object, + &expected_source_versions, + &[], + crate::data_movement::SourceCleanupBucketFence { + object_mutation_fence: Some(&mutation_fence), + ..Default::default() + }, + "test_versioned_decommission_delete_fence", + ) + .await + }); + cleanup_delete_barrier.wait_until_fence_pending().await; + assert!( + !cleanup_delete_barrier.is_paused(), + "source cleanup must wait for the versioned DELETE mutation fence" + ); + + delete_barrier.release(); + let marker = delete + .await + .expect("versioned DELETE task should join") + .expect("versioned DELETE should publish a delete marker after migration"); + assert!(marker.delete_marker, "versioned DELETE must publish a delete marker"); + assert!( + marker.version_id.is_some_and(|version_id| !version_id.is_nil()), + "the delete marker must have a non-nil version ID" + ); + + cleanup_delete_barrier.wait_until_paused().await; + cleanup_delete_barrier.release(); + cleanup + .await + .expect("source cleanup task should join") + .expect("source cleanup should preserve the active-pool delete marker"); + + let err = store + .get_object_info( + &bucket, + object, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect_err("the post-migration delete marker must hide the migrated version"); + assert!( + matches!(err, StorageError::ObjectNotFound(_, _)), + "unexpected latest-version result: {err:?}" + ); + store + .get_object_info( + &bucket, + object, + &ObjectOptions { + versioned: true, + version_id: Some(source_version.to_string()), + ..Default::default() + }, + ) + .await + .expect("the migrated source version must remain addressable below the delete marker"); + store.pools[0] + .get_object_info( + &bucket, + object, + &ObjectOptions { + versioned: true, + version_id: Some(source_version.to_string()), + ..Default::default() + }, + ) + .await + .expect_err("source cleanup must remove the decommissioned source versions"); + + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn versioned_batch_delete_marker_skips_decommission_source() { + let temp_dir = tempfile::tempdir().expect("create versioned batch decommission store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store( + temp_dir.path(), + "versioned-batch-decommission-delete-fence", + &[4, 4, 4], + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("vbatch-decom-delete-{}", uuid::Uuid::new_v4()); + let object = "batch-object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create versioned batch decommission bucket"); + let (_source_version, expected_source_versions) = migrate_versioned_decommission_test_object( + &store, + &bucket, + object, + b"batch source generation", + "test_versioned_batch_decommission_delete_fence", + ) + .await; + + let delete_config_snapshot = + Arc::new(crate::bucket::replication::DeleteReplicationConfigSnapshot::from_configs_for_test( + s3s::dto::VersioningConfiguration { + status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)), + ..Default::default() + }, + None, + )); + let delete_barrier = crate::store::object::VersionedDeleteMarkerCommitBarrier::install(&bucket, object); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_objects( + &delete_bucket, + vec![ObjectToDelete { + object_name: object.to_string(), + ..Default::default() + }], + ObjectOptions { + delete_replication_config_snapshot: Some(delete_config_snapshot), + ..Default::default() + }, + ) + .await + }); + delete_barrier.wait_until_paused().await; + + let source_set = store.pools[0].get_disks_by_key(object); + crate::data_movement::ensure_source_cleanup_versions_unchanged( + source_set, + &bucket, + object, + &expected_source_versions, + &[], + "test_versioned_batch_decommission_delete_fence", + ) + .await + .expect("batch DELETE must not publish a marker to the suspended source"); + + delete_barrier.release(); + let (deleted, errors) = delete.await.expect("versioned batch DELETE task should join"); + assert!(errors.iter().all(Option::is_none), "versioned batch DELETE should succeed: {errors:?}"); + assert_eq!(deleted.len(), 1); + assert!(deleted[0].delete_marker, "versioned batch DELETE must return a marker"); + assert!( + deleted[0] + .delete_marker_version_id + .is_some_and(|version_id| !version_id.is_nil()), + "versioned batch DELETE marker must have a non-nil version ID" + ); + + let mut active_marker_count = 0; + for pool in store.pools.iter().skip(1) { + let Some(versions) = pool + .get_disks_by_key(object) + .load_file_info_versions_exact(&bucket, object) + .await + .expect("active-pool versions should be readable") + else { + continue; + }; + active_marker_count += versions.versions.iter().filter(|version| version.deleted).count(); + } + assert_eq!(active_marker_count, 1, "batch DELETE must publish exactly one active-pool marker"); + + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn suspended_delete_marker_then_decommission_worker_converges_null_source() { + let temp_dir = tempfile::tempdir().expect("create suspended decommission DELETE store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store( + temp_dir.path(), + "suspended-decommission-delete-convergence", + &[4, 4], + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("suspended-decom-delete-{}", uuid::Uuid::new_v4()); + let object = "single.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create suspended decommission DELETE bucket"); + write_suspended_decommission_source(&store, &bucket, object).await; + mark_test_pool_decommissioning(&store, 0).await; + + let delete_barrier = crate::store::object::VersionedDeleteMarkerCommitBarrier::install(&bucket, object); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_object( + &delete_bucket, + object, + ObjectOptions { + version_suspended: true, + ..Default::default() + }, + ) + .await + }); + delete_barrier.wait_until_paused().await; + assert_suspended_null_source_present(&store, &bucket, object).await; + + let source_set = store.pools[0].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + + delete_barrier.release(); + let marker = delete + .await + .expect("suspended DELETE task should join") + .expect("suspended DELETE should commit its active-pool marker"); + drop(delete_barrier); + assert!(marker.delete_marker, "suspended DELETE must create a marker"); + assert!( + marker.version_id.is_none_or(|version_id| version_id.is_nil()), + "suspended DELETE marker must keep the null version identity" + ); + worker + .await + .expect("suspended decommission worker should join") + .expect("worker must treat the newer active null marker as a completed migration"); + + assert_suspended_decommission_converged(&store, &bucket, object).await; + shutdown.cancel(); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(storage_class_env)] + async fn suspended_batch_delete_marker_then_decommission_worker_converges_null_source() { + let temp_dir = tempfile::tempdir().expect("create suspended batch decommission DELETE store dir"); + let (_ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store( + temp_dir.path(), + "suspended-batch-decommission-delete-convergence", + &[4, 4], + )) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let bucket = format!("susp-batch-decom-delete-{}", uuid::Uuid::new_v4()); + let object = "batch.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create suspended batch decommission DELETE bucket"); + write_suspended_decommission_source(&store, &bucket, object).await; + mark_test_pool_decommissioning(&store, 0).await; + + let delete_config_snapshot = + Arc::new(crate::bucket::replication::DeleteReplicationConfigSnapshot::from_configs_for_test( + s3s::dto::VersioningConfiguration { + status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::SUSPENDED)), + ..Default::default() + }, + None, + )); + let delete_barrier = crate::store::object::VersionedDeleteMarkerCommitBarrier::install(&bucket, object); + let delete_store = Arc::clone(&store); + let delete_bucket = bucket.clone(); + let delete = tokio::spawn(async move { + delete_store + .delete_objects( + &delete_bucket, + vec![ObjectToDelete { + object_name: object.to_string(), + ..Default::default() + }], + ObjectOptions { + delete_replication_config_snapshot: Some(delete_config_snapshot), + ..Default::default() + }, + ) + .await + }); + delete_barrier.wait_until_paused().await; + assert_suspended_null_source_present(&store, &bucket, object).await; + + let source_set = store.pools[0].get_disks_by_key(object); + let worker_store = Arc::clone(&store); + let worker_bucket = bucket.clone(); + let worker = tokio::spawn(async move { + worker_store + .decommission_entry_for_test( + 0, + MetaCacheEntry { + name: object.to_string(), + ..Default::default() + }, + worker_bucket, + source_set, + ) + .await + }); + + delete_barrier.release(); + let (deleted, errors) = delete.await.expect("suspended batch DELETE task should join"); + drop(delete_barrier); + assert!(errors.iter().all(Option::is_none), "suspended batch DELETE should succeed: {errors:?}"); + assert!( + matches!(deleted.as_slice(), [marker] if marker.delete_marker && marker.delete_marker_version_id.is_none_or(|version_id| version_id.is_nil())), + "suspended batch DELETE must create one null marker: {deleted:?}" + ); + worker + .await + .expect("suspended batch decommission worker should join") + .expect("worker must treat the newer batch null marker as a completed migration"); + + assert_suspended_decommission_converged(&store, &bucket, object).await; + shutdown.cancel(); + } + #[cfg(feature = "test-util")] #[tokio::test] #[serial_test::serial(storage_class_env)] diff --git a/crates/ecstore/src/store/list_objects.rs b/crates/ecstore/src/store/list_objects.rs index 558bb7c94..0d8d238f9 100644 --- a/crates/ecstore/src/store/list_objects.rs +++ b/crates/ecstore/src/store/list_objects.rs @@ -4673,7 +4673,7 @@ async fn gather_results( entry.name = entry.name.replace("\\", "/"); } - // TODO: rx.recv() + // TODO(backlog): integrate rx.recv() for incremental listing results if let Some(marker) = &opts.marker && ((!opts.include_marker && &entry.name <= marker) || (opts.include_marker && &entry.name < marker)) @@ -4703,7 +4703,7 @@ async fn gather_results( continue; } - // TODO: Lifecycle + // TODO(backlog): integrate lifecycle evaluation during object listing entries.push(Some(entry)); candidate_entries += 1; diff --git a/crates/ecstore/src/store/mod.rs b/crates/ecstore/src/store/mod.rs index 4b8a9cb71..a40edd929 100644 --- a/crates/ecstore/src/store/mod.rs +++ b/crates/ecstore/src/store/mod.rs @@ -33,7 +33,7 @@ use crate::bucket::utils::check_put_object_part_args; use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname}; use crate::cluster::rpc::{RemoteClient, S3PeerSys}; use crate::config::storageclass; -use crate::core::pools::PoolMeta; +use crate::core::pools::{DecommissionCanceler, PoolMeta}; use crate::disk::endpoint::{Endpoint, EndpointType}; use crate::disk::{DiskAPI, DiskInfo, DiskInfoOptions}; use crate::error::{Error, Result}; @@ -151,7 +151,7 @@ pub(crate) mod init_format; pub(crate) mod list_objects; mod multipart; mod object; -pub(crate) use object::ObjectLockDiagGuard; +pub(crate) use object::{ObjectLockDiagGuard, SourceCleanupMutationFence}; pub use object::{ PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError, SnapshotConsistencyError, @@ -176,7 +176,7 @@ pub struct ECStore { // pub local_disks: Vec, pub pool_meta: RwLock, pub rebalance_meta: RwLock>, - pub decommission_cancelers: RwLock>>, + pub decommission_cancelers: RwLock>>, /// Serializes rebalance/decommission start transitions. /// /// Lock order: acquire `start_gate` before `pool_meta`, `rebalance_meta`, diff --git a/crates/ecstore/src/store/multipart.rs b/crates/ecstore/src/store/multipart.rs index 3d460d4d5..8d72287d4 100644 --- a/crates/ecstore/src/store/multipart.rs +++ b/crates/ecstore/src/store/multipart.rs @@ -332,7 +332,7 @@ impl ECStore { let expected_incarnation_id = opts.expected_bucket_incarnation_id; if request.prefix.is_empty() { - // TODO: return from cache + // TODO(backlog): return cached multipart listing when prefix is empty } if self.single_pool() { @@ -400,7 +400,7 @@ impl ECStore { object: &str, opts: &ObjectOptions, ) -> Result { - self.handle_new_multipart_upload_with_pool_idx(bucket, object, opts) + self.handle_new_multipart_upload_with_pool_idx(bucket, object, opts, None) .await .map(|(res, _, _)| res) } @@ -410,20 +410,22 @@ impl ECStore { bucket: &str, object: &str, opts: &ObjectOptions, + mutation_fence: Option<&ObjectLockDiagGuard>, ) -> Result<(MultipartUploadResult, usize, Option)> { check_new_multipart_args(bucket, object)?; - let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; - let opts = &opts; + let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; if self.single_pool() { + self.apply_decommission_target_mutation_fence(0, object, &mut opts, mutation_fence) + .await; return self.pools[0] - .new_multipart_upload(bucket, object, opts) + .new_multipart_upload(bucket, object, &opts) .await .map(|res| (res, 0, opts.expected_bucket_incarnation_id)); } if opts.data_movement && opts.version_id.is_some() { - let idx = self.select_data_movement_pool_idx(bucket, object, -1, opts, false).await?; + let idx = self.select_data_movement_pool_idx(bucket, object, -1, &opts, false).await?; if idx == opts.src_pool_idx { return Err(StorageError::DataMovementOverwriteErr( bucket.to_owned(), @@ -431,7 +433,9 @@ impl ECStore { opts.version_id.clone().unwrap_or_default(), )); } - let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?; + self.apply_decommission_target_mutation_fence(idx, object, &mut opts, mutation_fence) + .await; + let res = self.pools[idx].new_multipart_upload(bucket, object, &opts).await?; return Ok((res, idx, opts.expected_bucket_incarnation_id)); } @@ -454,7 +458,9 @@ impl ECStore { .await?; if !res.uploads.is_empty() { - let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?; + self.apply_decommission_target_mutation_fence(idx, object, &mut opts, mutation_fence) + .await; + let res = self.pools[idx].new_multipart_upload(bucket, object, &opts).await?; return Ok((res, idx, opts.expected_bucket_incarnation_id)); } } @@ -467,7 +473,9 @@ impl ECStore { )); } - let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?; + self.apply_decommission_target_mutation_fence(idx, object, &mut opts, mutation_fence) + .await; + let res = self.pools[idx].new_multipart_upload(bucket, object, &opts).await?; Ok((res, idx, opts.expected_bucket_incarnation_id)) } @@ -610,7 +618,7 @@ impl ECStore { let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; let opts = &opts; - // TODO: defer DeleteUploadID + // TODO(backlog): defer DeleteUploadID to background for faster abort response if self.single_pool() { return self.pools[0].abort_multipart_upload(bucket, object, upload_id, opts).await; @@ -704,13 +712,14 @@ impl ECStore { pub(crate) async fn complete_multipart_upload_for_data_movement( self: Arc, - target_pool_idx: usize, + target: (usize, Option<&ObjectLockDiagGuard>), bucket: &str, object: &str, upload_id: &str, uploaded_parts: Vec, opts: &ObjectOptions, ) -> Result { + let (target_pool_idx, mutation_fence) = target; check_complete_multipart_args(bucket, object, upload_id)?; if !opts.data_movement { return Err(Error::other("targeted multipart completion requires data_movement options")); @@ -739,6 +748,8 @@ impl ECStore { snapshot.add_lock_fences(&mut opts); opts.object_lock_config_snapshot = Some(snapshot); } + self.apply_decommission_target_mutation_fence(target_pool_idx, object, &mut opts, mutation_fence) + .await; #[cfg(test)] pause_data_movement_multipart_before_selected_completion(bucket).await; let pool = self diff --git a/crates/ecstore/src/store/object.rs b/crates/ecstore/src/store/object.rs index 312a25662..7a8653415 100644 --- a/crates/ecstore/src/store/object.rs +++ b/crates/ecstore/src/store/object.rs @@ -32,16 +32,17 @@ use crate::bucket::metadata_sys::{ use crate::bucket::object_lock::objectlock_sys::{ check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state, }; -use crate::bucket::replication::ReplicationObjectBridge; +use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge}; +use crate::bucket::versioning::VersioningApi; use crate::disk::OldCurrentSize; use crate::object_api::{NamespaceLockFence, ObjectLockConfigSnapshot}; use crate::set_disk::{ - get_lock_acquire_timeout, get_object_lock_diag_slow_acquire_threshold, get_object_lock_diag_slow_hold_threshold, - is_lock_optimization_enabled, is_object_lock_diag_enabled, + SetDisks, get_lock_acquire_timeout, get_object_lock_diag_slow_acquire_threshold, get_object_lock_diag_slow_hold_threshold, + is_lock_optimization_enabled, is_object_lock_diag_enabled, same_distributed_lock_domain, }; use crate::storage_api_contracts::{ namespace::NamespaceLocking as _, - object::{ObjectIO as _, ObjectOperations as _}, + object::{DeleteAccounting, ObjectIO as _, ObjectOperations as _}, }; use parking_lot::Mutex as ParkingMutex; use rustfs_io_metrics::{ @@ -352,6 +353,8 @@ impl fmt::Display for ObjectLockDiagMode { pub(crate) struct ObjectLockDiagGuard { guard: rustfs_lock::NamespaceLockGuard, + #[cfg(test)] + test_namespace_lock_fence: Option, enabled: bool, op: &'static str, bucket: Option, @@ -373,6 +376,8 @@ impl ObjectLockDiagGuard { ) -> Self { Self { guard, + #[cfg(test)] + test_namespace_lock_fence: None, enabled, op, bucket, @@ -393,6 +398,115 @@ impl ObjectLockDiagGuard { pub(crate) fn is_lock_lost(&self) -> bool { self.guard.is_lock_lost() } + + pub(crate) fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) { + opts.ensure_namespace_lock_fence(); + if let Some(signal) = self.lock_lost_signal() { + opts.add_namespace_lock_lost_signal(signal); + } + #[cfg(test)] + if let Some(fence) = self.test_namespace_lock_fence.as_ref() { + opts.add_namespace_lock_fence_for_test(fence); + } + } +} + +#[cfg(test)] +#[derive(Clone, Copy, PartialEq, Eq)] +pub(crate) enum DecommissionMutationFenceTestPhase { + Migration, + SourceCleanup, +} + +#[cfg(test)] +struct DecommissionMutationFenceLossState { + bucket: String, + object: String, + phase: DecommissionMutationFenceTestPhase, + fence: NamespaceLockFence, + loss_handle: Arc, +} + +#[cfg(test)] +pub(crate) struct DecommissionMutationFenceLossHook { + state: Arc, +} + +#[cfg(test)] +static DECOMMISSION_MUTATION_FENCE_LOSS_HOOK: std::sync::OnceLock< + std::sync::Mutex>>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +impl DecommissionMutationFenceLossHook { + pub(crate) fn install(bucket: &str, object: &str, phase: DecommissionMutationFenceTestPhase) -> Self { + let (fence, loss_handle) = NamespaceLockFence::loss_handle_for_test(); + let state = Arc::new(DecommissionMutationFenceLossState { + bucket: bucket.to_string(), + object: object.to_string(), + phase, + fence, + loss_handle, + }); + let mut slot = DECOMMISSION_MUTATION_FENCE_LOSS_HOOK + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("decommission mutation fence loss hooks should not poison"); + assert!(slot.is_none(), "decommission mutation fence loss hook must be unique"); + *slot = Some(Arc::clone(&state)); + Self { state } + } + + pub(crate) fn mark_lost(&self) { + self.state.loss_handle.store(true, Ordering::Release); + } +} + +#[cfg(test)] +impl Drop for DecommissionMutationFenceLossHook { + fn drop(&mut self) { + let mut slot = DECOMMISSION_MUTATION_FENCE_LOSS_HOOK + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("decommission mutation fence loss hooks should not poison"); + if slot.as_ref().is_some_and(|hook| Arc::ptr_eq(hook, &self.state)) { + *slot = None; + } + } +} + +#[cfg(test)] +fn decommission_mutation_fence_for_test( + bucket: &str, + object: &str, + phase: DecommissionMutationFenceTestPhase, +) -> Option { + DECOMMISSION_MUTATION_FENCE_LOSS_HOOK + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("decommission mutation fence loss hooks should not poison") + .as_ref() + .filter(|hook| hook.bucket == bucket && hook.object == object && hook.phase == phase) + .map(|hook| hook.fence.clone()) +} + +pub(crate) struct SourceCleanupMutationFence { + guard: ObjectLockDiagGuard, + source_lock_covered: bool, +} + +impl SourceCleanupMutationFence { + pub(crate) fn source_lock_covered(&self) -> bool { + self.source_lock_covered + } + + pub(crate) fn is_lock_lost(&self) -> bool { + self.guard.is_lock_lost() + } + + pub(crate) fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) { + self.guard.add_namespace_lock_fence(opts); + } } /// Opaque write-lock guard for the RestoreObject accept path; see @@ -410,10 +524,7 @@ impl RestoreAcceptGuard { } pub fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) { - opts.ensure_namespace_lock_fence(); - if let Some(signal) = self.0.lock_lost_signal() { - opts.add_namespace_lock_lost_signal(signal); - } + self.0.add_namespace_lock_fence(opts); } } @@ -690,16 +801,6 @@ impl SelectObjectSnapshotLockLossWake { } } -// LockRegistry clones its canonical client Arc for each endpoint host, so an -// exact Arc set identifies one distributed namespace-lock quorum domain. -fn same_distributed_lock_domain(left: &[Arc], right: &[Arc]) -> bool { - left.iter() - .all(|left_client| right.iter().any(|right_client| Arc::ptr_eq(left_client, right_client))) - && right - .iter() - .all(|right_client| left.iter().any(|left_client| Arc::ptr_eq(left_client, right_client))) -} - impl AsyncRead for SelectObjectSnapshotReader { fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { if self.lock_loss_wake.poll_lost(cx) || self.lease.is_lost() { @@ -805,7 +906,7 @@ fn resolve_latest_object_access( } fn should_create_delete_marker_for_missing_object(opts: &ObjectOptions) -> bool { - opts.versioned && opts.version_id.is_none() && !opts.delete_marker && !opts.data_movement + (opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.delete_marker && !opts.data_movement } #[cfg(test)] @@ -813,6 +914,8 @@ struct DeleteAfterObjectLockSnapshotBarrierState { bucket: String, arrived: tokio::sync::Notify, release: tokio::sync::Notify, + namespace_pending: tokio::sync::Notify, + namespace_acquired: AtomicBool, } #[cfg(test)] @@ -832,6 +935,8 @@ impl DeleteAfterObjectLockSnapshotBarrier { bucket: bucket.to_string(), arrived: tokio::sync::Notify::new(), release: tokio::sync::Notify::new(), + namespace_pending: tokio::sync::Notify::new(), + namespace_acquired: AtomicBool::new(false), }); let mut slot = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER .get_or_init(|| std::sync::Mutex::new(None)) @@ -849,6 +954,18 @@ impl DeleteAfterObjectLockSnapshotBarrier { pub(crate) fn release(&self) { self.state.release.notify_one(); } + + pub(crate) async fn release_and_wait_until_namespace_pending(&self) { + let namespace_pending = self.state.namespace_pending.notified(); + self.release(); + tokio::time::timeout(Duration::from_secs(5), namespace_pending) + .await + .expect("delete should proceed to its namespace lock after leaving the snapshot barrier"); + } + + pub(crate) fn namespace_acquired(&self) -> bool { + self.state.namespace_acquired.load(Ordering::Acquire) + } } #[cfg(test)] @@ -873,6 +990,97 @@ async fn pause_delete_after_object_lock_snapshot(bucket: &str) { .as_ref() .filter(|state| state.bucket == bucket) .cloned(); + if let Some(state) = state { + state.arrived.notify_one(); + state.release.notified().await; + state.namespace_pending.notify_one(); + } +} + +#[cfg(test)] +fn notify_delete_namespace_acquired(bucket: &str) { + let state = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("delete snapshot barrier mutex should not poison") + .as_ref() + .filter(|state| state.bucket == bucket) + .cloned(); + if let Some(state) = state { + state.namespace_acquired.store(true, Ordering::Release); + } +} + +#[cfg(test)] +struct VersionedDeleteMarkerCommitBarrierState { + bucket: String, + object: String, + arrived: tokio::sync::Notify, + release: tokio::sync::Notify, +} + +#[cfg(test)] +pub(crate) struct VersionedDeleteMarkerCommitBarrier { + state: Arc, +} + +#[cfg(test)] +static VERSIONED_DELETE_MARKER_COMMIT_BARRIER: std::sync::OnceLock< + std::sync::Mutex>>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +impl VersionedDeleteMarkerCommitBarrier { + pub(crate) fn install(bucket: &str, object: &str) -> Self { + let state = Arc::new(VersionedDeleteMarkerCommitBarrierState { + bucket: bucket.to_string(), + object: object.to_string(), + arrived: tokio::sync::Notify::new(), + release: tokio::sync::Notify::new(), + }); + let mut slot = VERSIONED_DELETE_MARKER_COMMIT_BARRIER + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("versioned delete-marker commit barrier mutex should not poison"); + assert!(slot.is_none(), "versioned delete-marker commit barrier must be unique"); + *slot = Some(Arc::clone(&state)); + Self { state } + } + + pub(crate) async fn wait_until_paused(&self) { + tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified()) + .await + .expect("versioned DELETE should reach the post-marker-commit barrier"); + } + + pub(crate) fn release(&self) { + self.state.release.notify_one(); + } +} + +#[cfg(test)] +impl Drop for VersionedDeleteMarkerCommitBarrier { + fn drop(&mut self) { + self.state.release.notify_one(); + let mut slot = VERSIONED_DELETE_MARKER_COMMIT_BARRIER + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("versioned delete-marker commit barrier mutex should not poison"); + if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) { + *slot = None; + } + } +} + +#[cfg(test)] +async fn pause_versioned_delete_marker_after_commit(bucket: &str, object: &str) { + let state = VERSIONED_DELETE_MARKER_COMMIT_BARRIER + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("versioned delete-marker commit barrier mutex should not poison") + .as_ref() + .filter(|state| state.bucket == bucket && state.object == object) + .cloned(); if let Some(state) = state { state.arrived.notify_one(); state.release.notified().await; @@ -913,6 +1121,160 @@ fn writer_pool_lookup_opts(opts: &ObjectOptions, no_lock: bool) -> ObjectOptions lookup_opts } +fn delete_pool_lookup_opts(opts: &ObjectOptions, no_lock: bool) -> ObjectOptions { + let mut lookup_opts = writer_pool_lookup_opts(opts, no_lock); + lookup_opts.skip_decommissioned = opts.data_movement; + lookup_opts +} + +fn should_delete_from_all_pools(opts: &ObjectOptions, pool_count: usize) -> bool { + pool_count > 0 && (!opts.versioned && !opts.version_suspended || opts.version_id.is_some()) +} + +fn batch_delete_creates_latest_marker(object: &ObjectToDelete, delete_config_snapshot: &DeleteReplicationConfigSnapshot) -> bool { + if object.version_id.is_some() { + return false; + } + + let object_name = decode_dir_object(&object.object_name); + let (versioned, version_suspended) = delete_config_snapshot.versioning_config().delete_state(&object_name); + versioned || version_suspended +} + +fn batch_delete_targets_pool(creates_latest_marker: bool, marker_target_pool_idx: Option, pool_idx: usize) -> bool { + !creates_latest_marker || marker_target_pool_idx == Some(pool_idx) +} + +#[cfg(test)] +struct BatchDeletePoolErrorInjectionState { + bucket: String, + pool_idx: usize, + errors: std::collections::HashMap, + observed: std::sync::atomic::AtomicUsize, +} + +#[cfg(test)] +pub(crate) struct BatchDeletePoolErrorInjection { + state: Arc, +} + +#[cfg(test)] +static BATCH_DELETE_POOL_ERROR_INJECTION: std::sync::OnceLock>>> = + std::sync::OnceLock::new(); + +#[cfg(test)] +impl BatchDeletePoolErrorInjection { + pub(crate) fn install(bucket: &str, pool_idx: usize, errors: Vec<(String, Error)>) -> Self { + let state = Arc::new(BatchDeletePoolErrorInjectionState { + bucket: bucket.to_string(), + pool_idx, + errors: errors.into_iter().collect(), + observed: std::sync::atomic::AtomicUsize::new(0), + }); + let mut slot = BATCH_DELETE_POOL_ERROR_INJECTION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("batch delete pool error injection mutex should not poison"); + assert!(slot.is_none(), "batch delete pool error injection must be unique"); + *slot = Some(Arc::clone(&state)); + Self { state } + } + + pub(crate) fn observed(&self) -> usize { + self.state.observed.load(Ordering::Acquire) + } +} + +#[cfg(test)] +impl Drop for BatchDeletePoolErrorInjection { + fn drop(&mut self) { + let mut slot = BATCH_DELETE_POOL_ERROR_INJECTION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("batch delete pool error injection mutex should not poison"); + if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) { + *slot = None; + } + } +} + +#[cfg(test)] +fn inject_batch_delete_pool_errors( + bucket: &str, + pool_idx: usize, + object_names: &[String], + result: &mut (Vec, Vec>), +) { + let state = BATCH_DELETE_POOL_ERROR_INJECTION + .get_or_init(|| std::sync::Mutex::new(None)) + .lock() + .expect("batch delete pool error injection mutex should not poison") + .as_ref() + .filter(|state| state.bucket == bucket && state.pool_idx == pool_idx) + .cloned(); + let Some(state) = state else { + return; + }; + + for (idx, object_name) in object_names.iter().enumerate() { + let Some(error) = state.errors.get(object_name) else { + continue; + }; + if result.1[idx].is_none() && result.0[idx].found { + result.1[idx] = Some(error.clone()); + state.observed.fetch_add(1, Ordering::AcqRel); + } + } +} + +fn resolve_batch_delete_pool_results<'a>( + initial_error: Option, + pool_results: impl IntoIterator)>, +) -> (Option, Option, bool) { + let mut failure = initial_error.map(|err| (None, err)); + let mut deleted = None; + let mut fallback: Option<(DeletedObject, Option)> = None; + let mut attempted = false; + + for (pool_delete, pool_error) in pool_results { + attempted = true; + match pool_error { + Some(err) if is_err_object_not_found(err) || is_err_version_not_found(err) => { + if fallback.as_ref().is_none_or(|(_, error)| error.is_none()) { + fallback = Some(((*pool_delete).clone(), Some(err.clone()))); + } + } + Some(err) => { + if failure.is_none() { + failure = Some((Some((*pool_delete).clone()), err.clone())); + } + } + None if pool_delete.found => { + if deleted.is_none() { + deleted = Some((*pool_delete).clone()); + } + } + None => { + if fallback.is_none() { + fallback = Some(((*pool_delete).clone(), None)); + } + } + } + } + + if let Some((failed_delete, err)) = failure { + return (failed_delete, Some(err), attempted); + } + if let Some(deleted) = deleted { + return (Some(deleted), None, attempted); + } + if let Some((deleted, err)) = fallback { + return (Some(deleted), err, attempted); + } + + (None, None, attempted) +} + fn transition_restore_pool_opts(opts: &ObjectOptions) -> ObjectOptions { let mut lookup_opts = opts.clone(); lookup_opts.skip_decommissioned = true; @@ -1216,6 +1578,14 @@ fn return_batch_delete_lock_error(objects: &[ObjectToDelete], err: Error) -> (Ve (del_objects, del_errs) } +fn return_batch_delete_lock_error_with_accounting( + objects: &[ObjectToDelete], + err: Error, +) -> (Vec, Vec>, Vec>) { + let (deleted, errors) = return_batch_delete_lock_error(objects, err); + (deleted, errors, vec![None; objects.len()]) +} + fn sorted_unique_delete_object_names(objects: &[ObjectToDelete]) -> Vec<&str> { let mut object_names: Vec<&str> = objects.iter().map(|object| object.object_name.as_str()).collect(); object_names.sort_unstable(); @@ -1533,6 +1903,89 @@ impl ECStore { ))) } + pub(crate) async fn acquire_decommission_object_mutation_fence( + &self, + bucket: &str, + object: &str, + ) -> Result { + if self.ctx.lock_manager().is_disabled() { + return Err(Error::other("decommission object migration requires namespace locking")); + } + + #[cfg(test)] + let test_namespace_lock_fence = + decommission_mutation_fence_for_test(bucket, object, DecommissionMutationFenceTestPhase::Migration); + let object = encode_dir_object(object); + let mut opts = ObjectOptions::default(); + let guard = self + .acquire_object_read_lock_if_needed("decommission_object", bucket, &object, &mut opts) + .await? + .ok_or_else(|| Error::other("decommission object migration failed to acquire its namespace fence"))?; + #[cfg(test)] + let guard = { + let mut guard = guard; + guard.test_namespace_lock_fence = test_namespace_lock_fence; + guard + }; + Ok(guard) + } + + pub(super) async fn apply_decommission_target_mutation_fence( + &self, + target_pool_idx: usize, + object: &str, + opts: &mut ObjectOptions, + mutation_fence: Option<&ObjectLockDiagGuard>, + ) { + let Some(mutation_fence) = mutation_fence else { + return; + }; + + mutation_fence.add_namespace_lock_fence(opts); + let fixed_set = self.pools.first().and_then(|pool| pool.disk_set.first()); + let target_set = self.pools.get(target_pool_idx).map(|pool| pool.get_disks_by_key(object)); + opts.no_lock = match (fixed_set, target_set) { + (Some(fixed), Some(target)) => fixed.shares_namespace_lock_domain(&target).await, + _ => false, + }; + } + + pub(crate) async fn acquire_decommission_source_cleanup_fence( + &self, + bucket: &str, + object: &str, + source_set: &SetDisks, + ) -> Result { + if self.ctx.lock_manager().is_disabled() { + return Err(Error::other("decommission source cleanup requires namespace locking")); + } + + #[cfg(test)] + crate::data_movement::notify_source_cleanup_mutation_fence_pending(bucket, object); + #[cfg(test)] + let test_namespace_lock_fence = + decommission_mutation_fence_for_test(bucket, object, DecommissionMutationFenceTestPhase::SourceCleanup); + let object = encode_dir_object(object); + let fixed_set = Arc::clone(&self.pools[0].disk_set[0]); + let source_lock_covered = fixed_set.shares_namespace_lock_domain(source_set).await; + // Lock order: fixed store mutation domain first; source cleanup takes its + // hashed source-domain lock second only when this guard does not cover it. + let guard = self + .acquire_object_write_lock("decommission_source_cleanup", bucket, &object) + .await?; + #[cfg(test)] + let guard = { + let mut guard = guard; + guard.test_namespace_lock_fence = test_namespace_lock_fence; + guard + }; + + Ok(SourceCleanupMutationFence { + guard, + source_lock_covered, + }) + } + pub(crate) async fn acquire_all_object_read_locks( &self, op: &'static str, @@ -1986,14 +2439,17 @@ impl ECStore { object: &str, data: &mut PutObjReader, opts: &ObjectOptions, + mutation_fence: Option<&ObjectLockDiagGuard>, ) -> Result<(usize, Result)> { if !opts.data_movement { return Err(Error::other("data movement PUT requires data_movement options")); } - let (object, opts) = self.prepare_put_object(bucket, object, opts).await?; + let (object, mut opts) = self.prepare_put_object(bucket, object, opts).await?; let idx = self .select_put_object_pool_idx(bucket, object.as_str(), data.size(), &opts) .await?; + self.apply_decommission_target_mutation_fence(idx, object.as_str(), &mut opts, mutation_fence) + .await; let result = self.pools[idx] .put_object_with_old_current_size(bucket, &object, data, &opts) .await @@ -2312,6 +2768,22 @@ impl ECStore { result } + pub async fn delete_objects_with_tier_delete_journal_and_accounting( + self: &Arc, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + ) -> (Vec, Vec>, Vec>) { + let result = self + .handle_delete_objects_with_journal_and_accounting(bucket, objects, opts, Some(Arc::clone(self))) + .await; + let success_count = result.1.iter().filter(|err| err.is_none()).count(); + if success_count > 0 { + list_objects::observe_list_objects_mutations(self, bucket, success_count).await; + } + result + } + #[instrument(skip(self))] pub(super) async fn handle_delete_object(&self, bucket: &str, object: &str, opts: ObjectOptions) -> Result { self.handle_delete_object_with_journal(bucket, object, opts, None).await @@ -2446,6 +2918,10 @@ impl ECStore { } else { None }; + #[cfg(test)] + if _object_lock_guard.is_some() { + notify_delete_namespace_acquired(bucket); + } if let Some(trigger) = opts.lifecycle_delete_all.as_ref() { let configs = delete_all_configs.as_ref().ok_or(StorageError::PreconditionFailed)?; let expected_bucket_incarnation_id = opts.expected_bucket_incarnation_id.ok_or(StorageError::PreconditionFailed)?; @@ -2479,7 +2955,7 @@ impl ECStore { return Ok(ObjectInfo::default()); } - let gopts = writer_pool_lookup_opts(&opts, true); + let gopts = delete_pool_lookup_opts(&opts, true); if opts.data_movement { let existing_pool_info = self.get_pool_info_existing_with_opts(bucket, object, &gopts).await; @@ -2584,6 +3060,8 @@ impl ECStore { Err(err) if is_err_object_not_found(&err) && should_create_delete_marker_for_missing_object(&opts) => { let target_pool_idx = self.get_pool_idx_no_lock(bucket, object, 0).await?; let mut obj = self.pools[target_pool_idx].delete_object(bucket, object, opts).await?; + #[cfg(test)] + pause_versioned_delete_marker_after_commit(bucket, object).await; obj.name = decode_dir_object(object); return Ok(obj); } @@ -2622,7 +3100,7 @@ impl ECStore { None }; - if !errs.is_empty() && !opts.versioned && !opts.version_suspended { + if should_delete_from_all_pools(&opts, errs.len()) { let mut obj = match self.delete_object_from_all_pools(bucket, object, &opts, errs).await { Ok(obj) => obj, Err(err) => { @@ -2646,6 +3124,8 @@ impl ECStore { match pool.delete_object(bucket, object, opts.clone()).await { Ok(res) => { + #[cfg(test)] + pause_versioned_delete_marker_after_commit(bucket, object).await; if let (Some(api), Some(je)) = (tier_journal_api.as_ref(), journal_entry.as_ref()) { commit_prepared_tier_delete_journal_entry(api, je).await; } @@ -2689,6 +3169,19 @@ impl ECStore { opts: ObjectOptions, tier_journal_api: Option>, ) -> (Vec, Vec>) { + let (deleted, errors, _) = self + .handle_delete_objects_with_journal_and_accounting(bucket, objects, opts, tier_journal_api) + .await; + (deleted, errors) + } + + pub(super) async fn handle_delete_objects_with_journal_and_accounting( + &self, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + tier_journal_api: Option>, + ) -> (Vec, Vec>, Vec>) { // encode object name let objects: Vec = objects .iter() @@ -2701,6 +3194,7 @@ impl ECStore { // Default return value let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let accounting = vec![None; objects.len()]; let mut del_errs = Vec::with_capacity(objects.len()); for _ in 0..objects.len() { @@ -2714,7 +3208,7 @@ impl ECStore { } else { match self.acquire_bucket_lifecycle_read_lock(bucket).await { Ok(guard) => Some(guard), - Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), } }; if let Some(guard) = _bucket_lifecycle_guard.as_ref() { @@ -2726,21 +3220,21 @@ impl ECStore { Err(err) => { let message = err.to_string(); let errors = (0..objects.len()).map(|_| Some(Error::other(message.clone()))).collect(); - return (del_objects, errors); + return (del_objects, errors, accounting); } } } if !is_meta_bucketname(bucket) && let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await { - return return_batch_delete_lock_error(objects.as_slice(), err); + return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err); } let _object_lock_metadata_guard = if is_meta_bucketname(bucket) { None } else { Some(match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await { Ok(guard) => guard, - Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), }) }; if let Some(guard) = _object_lock_metadata_guard.as_ref() { @@ -2750,7 +3244,7 @@ impl ECStore { let (state, incarnation_id, config_revision) = match get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await { Ok(snapshot) => snapshot, - Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), }; opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket( self.id, @@ -2766,7 +3260,10 @@ impl ECStore { if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id) && expected != current { - return return_batch_delete_lock_error(objects.as_slice(), StorageError::BucketNotFound(bucket.to_string())); + return return_batch_delete_lock_error_with_accounting( + objects.as_slice(), + StorageError::BucketNotFound(bucket.to_string()), + ); } #[cfg(test)] if current_bucket_incarnation_id.is_some() { @@ -2774,32 +3271,106 @@ impl ECStore { } let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await { Ok(guards) => guards, - Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), }; + #[cfg(test)] + if !_object_lock_guards.is_empty() { + notify_delete_namespace_acquired(bucket); + } + + let delete_config_snapshot = opts + .delete_replication_config_snapshot + .as_deref() + .expect("batch delete replication config snapshot should be loaded"); + let latest_marker_objects = objects + .iter() + .map(|object| batch_delete_creates_latest_marker(object, delete_config_snapshot)) + .collect::>(); + let marker_target_results = join_all(objects.iter().zip(&latest_marker_objects).map( + |(object, creates_marker)| async move { + if *creates_marker { + Some(self.get_pool_idx_no_lock(bucket, &object.object_name, 0).await) + } else { + None + } + }, + )) + .await; + let mut marker_target_pool_indices = Vec::with_capacity(objects.len()); + for (idx, target_result) in marker_target_results.into_iter().enumerate() { + match target_result { + Some(Ok(pool_idx)) => marker_target_pool_indices.push(Some(pool_idx)), + Some(Err(err)) => { + del_errs[idx] = Some(err); + marker_target_pool_indices.push(None); + } + None => marker_target_pool_indices.push(None), + } + } let mut futures = Vec::with_capacity(self.pools.len()); - for pool in self.pools.iter() { if self.is_pool_rebalancing(pool.pool_idx).await { continue; } - futures.push(pool.delete_objects(bucket, objects.clone(), opts.clone())); + + let (object_indices, pool_objects): (Vec<_>, Vec<_>) = objects + .iter() + .enumerate() + .filter(|(idx, _)| { + batch_delete_targets_pool(latest_marker_objects[*idx], marker_target_pool_indices[*idx], pool.pool_idx) + }) + .map(|(idx, object)| (idx, object.clone())) + .unzip(); + if pool_objects.is_empty() { + continue; + } + + let pool_opts = opts.clone(); + futures.push(async move { + #[cfg(test)] + let pool_object_names = pool_objects + .iter() + .map(|object| object.object_name.clone()) + .collect::>(); + let result = pool.delete_objects(bucket, pool_objects, pool_opts).await; + #[cfg(test)] + let result = { + let mut result = result; + inject_batch_delete_pool_errors(bucket, pool.pool_idx, &pool_object_names, &mut result); + result + }; + (object_indices, result) + }); } let results = join_all(futures).await; for idx in 0..del_objects.len() { - for (dels, errs) in results.iter() { - if errs[idx].is_none() && dels[idx].found { - del_errs[idx] = None; - del_objects[idx] = dels[idx].clone(); - break; - } + let pool_results = results.iter().filter_map(|(object_indices, (dels, errs))| { + let pool_object_idx = object_indices.binary_search(&idx).ok()?; + Some((&dels[pool_object_idx], &errs[pool_object_idx])) + }); + let (deleted, error, attempted) = resolve_batch_delete_pool_results(del_errs[idx].take(), pool_results); + if let Some(deleted) = deleted { + del_objects[idx] = deleted; + } + del_errs[idx] = error; - if del_errs[idx].is_none() { - del_errs[idx] = errs[idx].clone(); - del_objects[idx] = dels[idx].clone(); - } + if !attempted && del_errs[idx].is_none() && latest_marker_objects[idx] { + del_objects[idx] = DeletedObject { + object_name: objects[idx].object_name.clone(), + version_id: objects[idx].version_id, + ..Default::default() + }; + del_errs[idx] = Some(StorageError::ObjectNotFound(bucket.to_owned(), objects[idx].object_name.clone())); + } + } + + #[cfg(test)] + for (idx, object) in objects.iter().enumerate() { + if del_errs[idx].is_none() && del_objects[idx].delete_marker { + pause_versioned_delete_marker_after_commit(bucket, &object.object_name).await; } } @@ -2807,7 +3378,7 @@ impl ECStore { v.object_name = decode_dir_object(&v.object_name); }); - (del_objects, del_errs) + (del_objects, del_errs, accounting) // let mut futures = Vec::with_capacity(objects.len()); @@ -3374,6 +3945,80 @@ mod tests { assert!(!same_distributed_lock_domain(&[first, second], &[other])); } + #[tokio::test] + async fn decommission_fence_covers_dist_sets_with_same_clients_despite_different_namespaces() { + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let (_dirs, original_sets) = make_local_two_set_sets_with_ctx(Arc::clone(&ctx)).await; + let mut second_set = (*original_sets.disk_set[1]).clone(); + second_set.lockers = original_sets.disk_set[0].lockers.clone(); + let mut sets = (*original_sets).clone(); + sets.disk_set[1] = Arc::new(second_set); + let sets = Arc::new(sets); + ctx.update_erasure_type(SetupType::DistErasure).await; + + assert!( + sets.disk_set[0] + .lockers + .iter() + .zip(&sets.disk_set[1].lockers) + .all(|(fixed, hashed)| Arc::ptr_eq(fixed, hashed)), + "the regression requires identical distributed lock clients" + ); + assert_ne!(sets.disk_set[0].set_index, sets.disk_set[1].set_index); + + let pool_config = sets.endpoints.clone(); + let store = new_prepared_reader_test_store_from_pools(vec![Arc::clone(&sets)], vec![pool_config], ctx); + let object = (0..1_000) + .map(|index| format!("decommission-dist-domain-{index}.bin")) + .find(|candidate| Arc::ptr_eq(&sets.get_disks_by_key(candidate), &sets.disk_set[1])) + .expect("a key should hash to the second set namespace"); + let mutation_fence = store + .acquire_decommission_object_mutation_fence("bucket", &object) + .await + .expect("the fixed distributed mutation fence should be acquired"); + let target_lock = sets.disk_set[1] + .new_ns_lock("bucket", &object) + .await + .expect("the hashed-set namespace lock should be created"); + let target_err = target_lock + .get_write_lock(Duration::from_millis(50)) + .await + .expect_err("the fixed read fence must conflict through the shared clients"); + assert!(matches!(target_err, rustfs_lock::LockError::Timeout { .. })); + + let mut put_opts = ObjectOptions::default(); + store + .apply_decommission_target_mutation_fence(0, &object, &mut put_opts, Some(&mutation_fence)) + .await; + assert!(put_opts.no_lock, "migration target PUT must reuse the covering fixed fence"); + + let mut multipart_opts = ObjectOptions::default(); + store + .apply_decommission_target_mutation_fence(0, &object, &mut multipart_opts, Some(&mutation_fence)) + .await; + assert!(multipart_opts.no_lock, "migration target multipart must reuse the covering fixed fence"); + drop(mutation_fence); + + let cleanup_object = (0..1_000) + .map(|index| format!("decommission-dist-cleanup-{index}.bin")) + .find(|candidate| Arc::ptr_eq(&sets.get_disks_by_key(candidate), &sets.disk_set[1])) + .expect("a cleanup key should hash to the second set namespace"); + let source_fence = store + .acquire_decommission_source_cleanup_fence("bucket", &cleanup_object, sets.disk_set[1].as_ref()) + .await + .expect("the fixed distributed cleanup fence should be acquired"); + assert!(source_fence.source_lock_covered(), "source cleanup must reuse the covering fixed fence"); + let source_lock = sets.disk_set[1] + .new_ns_lock("bucket", &cleanup_object) + .await + .expect("the source-set namespace lock should be created"); + let source_err = source_lock + .get_read_lock(Duration::from_millis(50)) + .await + .expect_err("the fixed write fence must conflict through the shared clients"); + assert!(matches!(source_err, rustfs_lock::LockError::Timeout { .. })); + } + #[test] fn select_snapshot_version_matching_normalizes_null_and_uuid_forms() { let nil = Uuid::nil(); @@ -4433,6 +5078,159 @@ mod tests { assert_eq!(lookup_opts.version_id.as_deref(), Some("vid-1")); } + #[test] + fn ordinary_delete_lookup_includes_decommission_source_and_skips_rebalance_source() { + let lookup_opts = delete_pool_lookup_opts(&ObjectOptions::default(), true); + + assert!(lookup_opts.no_lock); + assert!(!lookup_opts.skip_decommissioned); + assert!(lookup_opts.skip_rebalancing); + + let explicit_version = delete_pool_lookup_opts( + &ObjectOptions { + versioned: true, + version_id: Some(uuid::Uuid::new_v4().to_string()), + ..Default::default() + }, + true, + ); + assert!(!explicit_version.skip_decommissioned); + } + + #[test] + fn delete_fans_out_for_unversioned_and_explicit_version_mutations() { + assert!(should_delete_from_all_pools(&ObjectOptions::default(), 1)); + assert!(should_delete_from_all_pools( + &ObjectOptions { + versioned: true, + version_id: Some(uuid::Uuid::new_v4().to_string()), + ..Default::default() + }, + 2, + )); + assert!(!should_delete_from_all_pools( + &ObjectOptions { + versioned: true, + ..Default::default() + }, + 1, + )); + assert!(!should_delete_from_all_pools(&ObjectOptions::default(), 0)); + } + + #[test] + fn batch_delete_identifies_only_latest_versioned_markers() { + let versioned = DeleteReplicationConfigSnapshot::from_configs_for_test( + s3s::dto::VersioningConfiguration { + status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)), + ..Default::default() + }, + None, + ); + let latest = ObjectToDelete { + object_name: "latest".to_string(), + ..Default::default() + }; + assert!(batch_delete_creates_latest_marker(&latest, &versioned)); + assert!(!batch_delete_targets_pool(true, Some(1), 0)); + assert!(batch_delete_targets_pool(true, Some(1), 1)); + assert!(!batch_delete_targets_pool(true, Some(1), 2)); + + let explicit = ObjectToDelete { + object_name: "explicit".to_string(), + version_id: Some(uuid::Uuid::new_v4()), + ..Default::default() + }; + assert!(!batch_delete_creates_latest_marker(&explicit, &versioned)); + assert!(batch_delete_targets_pool(false, Some(1), 0)); + + let unversioned = DeleteReplicationConfigSnapshot::default(); + assert!(!batch_delete_creates_latest_marker(&latest, &unversioned)); + assert!(batch_delete_targets_pool(false, None, 0)); + } + + #[test] + fn batch_delete_pool_failures_override_success_in_any_pool_order() { + let success = DeletedObject { + object_name: "object".to_string(), + found: true, + ..Default::default() + }; + let source_errors = [ + StorageError::ErasureWriteQuorum, + StorageError::NamespaceLockQuorumUnavailable { + mode: "delete_objects_commit", + bucket: "bucket".to_string(), + object: "object".to_string(), + required: 1, + achieved: 0, + }, + ]; + + for source_error in source_errors { + for source_first in [true, false] { + let failed = (DeletedObject::default(), Some(source_error.clone())); + let succeeded = (success.clone(), None); + let pool_results = if source_first { + vec![failed, succeeded] + } else { + vec![succeeded, failed] + }; + + let (_, error, attempted) = + resolve_batch_delete_pool_results(None, pool_results.iter().map(|(deleted, error)| (deleted, error))); + + assert!(attempted); + assert_eq!(error, Some(source_error.clone())); + } + } + } + + #[test] + fn batch_delete_ignores_missing_pool_only_after_another_pool_succeeds() { + let success = DeletedObject { + object_name: "object".to_string(), + found: true, + ..Default::default() + }; + let missing_errors = [ + StorageError::ObjectNotFound("bucket".to_string(), "object".to_string()), + StorageError::VersionNotFound("bucket".to_string(), "object".to_string(), "version".to_string()), + ]; + + for missing_error in missing_errors { + let missing = (DeletedObject::default(), Some(missing_error.clone())); + for missing_first in [true, false] { + let succeeded = (success.clone(), None); + let pool_results = if missing_first { + vec![missing.clone(), succeeded] + } else { + vec![succeeded, missing.clone()] + }; + let (deleted, error, attempted) = + resolve_batch_delete_pool_results(None, pool_results.iter().map(|(deleted, error)| (deleted, error))); + + assert!(attempted); + let deleted = deleted.expect("successful pool result should be retained"); + assert!(deleted.found); + assert_eq!(deleted.object_name, success.object_name.as_str()); + assert!(error.is_none()); + } + + let missing_only = [missing]; + let (_, error, attempted) = + resolve_batch_delete_pool_results(None, missing_only.iter().map(|(deleted, error)| (deleted, error))); + assert!(attempted); + assert_eq!(error, Some(missing_error)); + } + + let silent_missing = [(DeletedObject::default(), None)]; + let (_, error, attempted) = + resolve_batch_delete_pool_results(None, silent_missing.iter().map(|(deleted, error)| (deleted, error))); + assert!(attempted); + assert!(error.is_none()); + } + #[test] fn data_movement_pool_lookup_opts_keeps_no_lock_for_tiered_moves() { let lookup_opts = data_movement_pool_lookup_opts( diff --git a/crates/ecstore/src/store/rebalance.rs b/crates/ecstore/src/store/rebalance.rs index 79e3abf91..dc2fa2ee5 100644 --- a/crates/ecstore/src/store/rebalance.rs +++ b/crates/ecstore/src/store/rebalance.rs @@ -385,7 +385,7 @@ impl ECStore { } pub(super) async fn is_suspended(&self, idx: usize) -> bool { - // TODO: LOCK + // TODO(backlog): acquire pool metadata lock for consistent suspension check let pool_meta = self.pool_meta.read().await; @@ -859,6 +859,7 @@ fn lifecycle_delete_all_test_failure(phase: crate::object_api::LifecycleDeleteAl #[cfg(test)] mod tests { use super::*; + use crate::bucket::replication::{ReplicationStatusType, VersionPurgeStatusType}; use crate::config::storageclass::{CLASS_RRS, CLASS_STANDARD, lookup_config_for_pools_without_env}; use crate::disk::error::DiskError; use crate::layout::endpoint::Endpoint; @@ -1423,6 +1424,14 @@ mod tests { } } + fn object_info_with_identity(unix_ts: i64, delete_marker: bool, version_id: Uuid, etag: Option) -> ObjectInfo { + ObjectInfo { + version_id: Some(version_id), + etag, + ..object_info_with_mod_time(unix_ts, delete_marker) + } + } + #[test] fn resolve_latest_object_info_candidates_returns_latest_delete_marker() { let candidates = vec![ @@ -1446,7 +1455,7 @@ mod tests { } #[test] - fn resolve_latest_object_info_candidates_prefers_higher_pool_idx_on_equal_mod_time() { + fn resolve_latest_object_info_candidates_prefers_higher_pool_idx_on_equal_mod_time_for_equivalent_candidates() { let candidates = vec![ LatestObjectInfoCandidate { info: Some(object_info_with_mod_time(10, false)), @@ -1466,6 +1475,382 @@ mod tests { assert_eq!(idx, 1); } + #[test] + fn resolve_latest_object_info_candidates_keeps_index_fallback_for_fully_equivalent_identities() { + let candidates = vec![ + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string()))), + idx: 2, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string()))), + idx: 7, + err: None, + }, + ]; + + let (info, idx) = resolve_latest_object_info_candidates(candidates, "bucket", "object", &ObjectOptions::default()) + .expect("equivalent replicas must resolve deterministically"); + + assert_eq!(idx, 7); + assert_eq!(info.version_id, Some(Uuid::from_u128(1))); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_equal_time_version_id_conflict() { + let candidates = vec![ + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string()))), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(2), Some("etag-a".to_string()))), + idx: 1, + err: None, + }, + ]; + + let err = resolve_latest_object_info_candidates(candidates, "bucket", "object", &ObjectOptions::default()) + .expect_err("divergent version ids must not silently resolve to the higher pool index"); + + assert_eq!(err, Error::ErasureReadQuorum); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_equal_time_etag_conflict() { + let candidates = vec![ + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-old".to_string()))), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-new".to_string()))), + idx: 1, + err: None, + }, + ]; + + let err = resolve_latest_object_info_candidates(candidates, "bucket", "object", &ObjectOptions::default()) + .expect_err("divergent etags must not silently resolve to the higher pool index"); + + assert_eq!(err, Error::ErasureReadQuorum); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_equal_time_delete_marker_conflict() { + let candidates = vec![ + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), None)), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, true, Uuid::from_u128(1), Some("etag-a".to_string()))), + idx: 1, + err: None, + }, + ]; + + let err = resolve_latest_object_info_candidates(candidates, "bucket", "object", &ObjectOptions::default()) + .expect_err("a delete marker tied with a live version must not be masked by the pool index"); + + assert_eq!(err, Error::ErasureReadQuorum); + } + + fn assert_equal_time_identity_conflict(left: ObjectInfo, right: ObjectInfo) { + let err = resolve_latest_object_info_candidates( + vec![ + LatestObjectInfoCandidate { + info: Some(left), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(right), + idx: 1, + err: None, + }, + ], + "bucket", + "object", + &ObjectOptions::default(), + ) + .expect_err("equal-time identity divergence must fail closed"); + + assert_eq!(err, Error::ErasureReadQuorum); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_equal_time_payload_identity_conflicts() { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + + let mut data_dir = base.clone(); + data_dir.data_dir = Some(Uuid::from_u128(2)); + assert_equal_time_identity_conflict(base.clone(), data_dir); + + let mut size = base.clone(); + size.size = 1; + assert_equal_time_identity_conflict(base.clone(), size); + + let mut actual_size = base.clone(); + actual_size.actual_size = 1; + assert_equal_time_identity_conflict(base.clone(), actual_size); + + let mut checksum = base.clone(); + checksum.checksum = Some(bytes::Bytes::from_static(b"checksum")); + assert_equal_time_identity_conflict(base.clone(), checksum); + + let mut parts = base.clone(); + parts.parts = std::sync::Arc::new(vec![rustfs_filemeta::ObjectPartInfo { + etag: "part-etag".to_string(), + number: 1, + size: 1, + ..Default::default() + }]); + assert_equal_time_identity_conflict(base.clone(), parts); + + let mut transition = base; + transition.transitioned_object.tier = "tier-a".to_string(); + assert_equal_time_identity_conflict( + object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())), + transition, + ); + } + + #[test] + fn resolve_latest_object_info_candidates_accepts_internal_metadata_aliases() { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + let mut rustfs_alias = base.clone(); + rustfs_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + "x-rustfs-internal-compression".to_string(), + "zstd".to_string(), + )])); + let mut minio_alias = base.clone(); + minio_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + "X-MINIO-INTERNAL-COMPRESSION".to_string(), + "zstd".to_string(), + )])); + + let (_, idx) = resolve_latest_object_info_candidates( + vec![ + LatestObjectInfoCandidate { + info: Some(rustfs_alias), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(minio_alias), + idx: 1, + err: None, + }, + ], + "bucket", + "object", + &ObjectOptions::default(), + ) + .expect("same-value internal aliases should resolve"); + assert_eq!(idx, 1); + + let mut dual_alias = base.clone(); + dual_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([ + ("x-rustfs-internal-compression".to_string(), "zstd".to_string()), + ("x-minio-internal-compression".to_string(), "zstd".to_string()), + ])); + let mut single_alias = base; + single_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + "x-rustfs-internal-compression".to_string(), + "zstd".to_string(), + )])); + + let (_, idx) = resolve_latest_object_info_candidates( + vec![ + LatestObjectInfoCandidate { + info: Some(dual_alias), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(single_alias), + idx: 1, + err: None, + }, + ], + "bucket", + "object", + &ObjectOptions::default(), + ) + .expect("dual-key and single-key internal metadata should resolve"); + assert_eq!(idx, 1); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_different_internal_metadata_alias_values() { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + let mut rustfs_alias = base.clone(); + rustfs_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + "x-rustfs-internal-compression".to_string(), + "zstd".to_string(), + )])); + let mut minio_alias = base; + minio_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + "x-minio-internal-compression".to_string(), + "snappy".to_string(), + )])); + + assert_equal_time_identity_conflict(rustfs_alias, minio_alias); + } + + #[test] + fn resolve_latest_object_info_candidates_preserves_dynamic_internal_metadata_identity_case() { + for suffix_prefix in ["replication-reset-", "replication-delete-marker-version-"] { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + let mut rustfs_alias = base.clone(); + rustfs_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + format!( + "X-RUSTFS-INTERNAL-{}{suffix}", + suffix_prefix.to_uppercase(), + suffix = "arn:aws:s3:::Bucket" + ), + "value".to_string(), + )])); + let mut minio_alias = base.clone(); + minio_alias.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + format!("x-minio-internal-{suffix_prefix}arn:aws:s3:::Bucket"), + "value".to_string(), + )])); + + let (_, idx) = resolve_latest_object_info_candidates( + vec![ + LatestObjectInfoCandidate { + info: Some(rustfs_alias.clone()), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(minio_alias), + idx: 1, + err: None, + }, + ], + "bucket", + "object", + &ObjectOptions::default(), + ) + .expect("dynamic internal aliases with the same target should resolve"); + assert_eq!(idx, 1); + + let mut different_target_case = base; + different_target_case.user_defined = std::sync::Arc::new(std::collections::HashMap::from([( + format!("x-minio-internal-{suffix_prefix}arn:aws:s3:::bucket"), + "value".to_string(), + )])); + + assert_equal_time_identity_conflict(rustfs_alias, different_target_case); + } + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_conflicting_internal_metadata_aliases_in_one_candidate() { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + let mut first = base.clone(); + first.user_defined = std::sync::Arc::new(std::collections::HashMap::from([ + ("x-rustfs-internal-compression".to_string(), "zstd".to_string()), + ("x-minio-internal-compression".to_string(), "snappy".to_string()), + ])); + let mut second = base; + second.user_defined = first.user_defined.clone(); + + assert_equal_time_identity_conflict(first, second); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_replication_identity_conflict() { + let base = object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())); + + let mut replication = base.clone(); + replication.replication_status_internal = Some("PENDING".to_string()); + replication.replication_status = ReplicationStatusType::Pending; + assert_equal_time_identity_conflict(base.clone(), replication); + + let mut purge = base.clone(); + purge.version_purge_status_internal = Some("PENDING".to_string()); + purge.version_purge_status = VersionPurgeStatusType::Pending; + assert_equal_time_identity_conflict(base.clone(), purge); + + let mut decision = base; + decision.replication_decision = "replicate".to_string(); + assert_equal_time_identity_conflict( + object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string())), + decision, + ); + } + + #[test] + fn resolve_latest_object_info_candidates_rejects_none_vs_unix_epoch_mod_time() { + let mut without_mod_time = object_info_with_identity(0, false, Uuid::from_u128(1), Some("etag-a".to_string())); + without_mod_time.mod_time = None; + let with_unix_epoch = object_info_with_identity(0, false, Uuid::from_u128(1), Some("etag-a".to_string())); + + assert_equal_time_identity_conflict(without_mod_time, with_unix_epoch); + } + + #[test] + fn resolve_latest_object_info_candidates_ignores_older_identity_conflicts() { + let latest = object_info_with_identity(20, false, Uuid::from_u128(1), Some("etag-latest".to_string())); + let mut older = object_info_with_identity(10, true, Uuid::from_u128(2), Some("etag-old".to_string())); + older.data_dir = Some(Uuid::from_u128(2)); + + let (info, idx) = resolve_latest_object_info_candidates( + vec![ + LatestObjectInfoCandidate { + info: Some(latest), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: Some(older), + idx: 9, + err: None, + }, + ], + "bucket", + "object", + &ObjectOptions::default(), + ) + .expect("older identity divergence must not affect the latest candidate"); + + assert_eq!(idx, 0); + assert_eq!( + info.mod_time, + Some(OffsetDateTime::from_unix_timestamp(20).expect("operation should succeed")) + ); + } + + #[test] + fn resolve_latest_object_info_candidates_ignores_not_found_pools_when_resolving() { + let candidates = vec![ + LatestObjectInfoCandidate { + info: Some(object_info_with_identity(10, false, Uuid::from_u128(1), Some("etag-a".to_string()))), + idx: 0, + err: None, + }, + LatestObjectInfoCandidate { + info: None, + idx: 1, + err: Some(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), + }, + ]; + + let (info, idx) = resolve_latest_object_info_candidates(candidates, "bucket", "object", &ObjectOptions::default()) + .expect("not-found pools must not block resolution of found candidates"); + + assert_eq!(idx, 0); + assert_eq!(info.version_id, Some(Uuid::from_u128(1))); + } + #[test] fn resolve_latest_object_info_candidates_returns_non_not_found_error() { let err = resolve_latest_object_info_candidates( diff --git a/crates/ecstore/src/store/rebalance/support.rs b/crates/ecstore/src/store/rebalance/support.rs index 6035e58d8..434f9c29d 100644 --- a/crates/ecstore/src/store/rebalance/support.rs +++ b/crates/ecstore/src/store/rebalance/support.rs @@ -12,10 +12,14 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::cmp::Ordering; +use std::collections::HashMap; use crate::error::{Error, Result, StorageError, is_err_object_not_found, is_err_version_not_found}; use crate::object_api::{ObjectInfo, ObjectOptions}; +use rustfs_utils::http::metadata_compat::{ + SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX, SUFFIX_REPLICATION_RESET_ARN_PREFIX, + strip_internal_prefix_preserving_case, +}; use rustfs_utils::path::decode_dir_object; use time::OffsetDateTime; @@ -73,7 +77,7 @@ pub(super) fn resolve_rebalance_delete_from_all_pools_result( object: &str, ) -> Result { result.map_err(|err| { - if err == Error::PreconditionFailed { + if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { err } else { Error::other(format!("failed to delete rebalance source object {bucket}/{object}: {err}")) @@ -86,7 +90,7 @@ fn is_ignorable_rebalance_delete_error(err: &Error) -> bool { } fn rebalance_delete_pool_error(pool_idx: usize, bucket: &str, object: &str, err: Error) -> Error { - if err == Error::PreconditionFailed { + if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { err } else { Error::other(format!("pool {pool_idx} delete failed for {bucket}/{object}: {err}")) @@ -137,37 +141,158 @@ pub(super) fn rebalance_disk_set_lookup_error(pool_idx: usize, set_idx: usize, p )) } +fn latest_candidate_mod_time(candidate: &LatestObjectInfoCandidate) -> Option { + candidate + .info + .as_ref() + .map(|info| info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)) +} + +fn same_transition_identity(left: &ObjectInfo, right: &ObjectInfo) -> bool { + left.transition_version_state == right.transition_version_state + && left.transitioned_object.name == right.transitioned_object.name + && left.transitioned_object.version_id == right.transitioned_object.version_id + && left.transitioned_object.tier == right.transitioned_object.tier + && left.transitioned_object.free_version == right.transitioned_object.free_version + && left.transitioned_object.status == right.transitioned_object.status +} + +#[derive(PartialEq, Eq)] +struct LatestUserDefinedIdentity { + internal: HashMap, + other: HashMap, +} + +fn normalize_internal_identity_suffix(key: &str) -> Option { + let suffix = strip_internal_prefix_preserving_case(key)?; + + for dynamic_prefix in [ + SUFFIX_REPLICATION_RESET_ARN_PREFIX, + SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX, + ] { + let prefix_len = dynamic_prefix.len(); + if let (Some(prefix), Some(remainder)) = (suffix.get(..prefix_len), suffix.get(prefix_len..)) + && prefix.eq_ignore_ascii_case(dynamic_prefix) + { + return Some(format!("{dynamic_prefix}{remainder}")); + } + } + + Some(suffix.to_lowercase()) +} + +fn normalize_user_defined_identity(user_defined: &HashMap) -> Option { + let mut identity = LatestUserDefinedIdentity { + internal: HashMap::with_capacity(user_defined.len()), + other: HashMap::with_capacity(user_defined.len()), + }; + + for (key, value) in user_defined { + if let Some(suffix) = normalize_internal_identity_suffix(key) { + if identity + .internal + .insert(suffix, value.clone()) + .is_some_and(|previous| previous != *value) + { + return None; + } + } else { + identity.other.insert(key.clone(), value.clone()); + } + } + + Some(identity) +} + +fn same_user_defined_identity(left: &ObjectInfo, right: &ObjectInfo) -> bool { + match ( + normalize_user_defined_identity(&left.user_defined), + normalize_user_defined_identity(&right.user_defined), + ) { + (Some(left), Some(right)) => left == right, + _ => false, + } +} + +/// Pool-specific erasure geometry is intentionally excluded: `get_object_info` +/// returns each pool's own `data_blocks`/`parity_blocks`, so those values can +/// differ for the same object version while the selected winner still carries +/// the chosen pool's layout. `put_object_reader` is also intentionally +/// excluded because it is a transient request handle that `ObjectInfo::clone` +/// drops. Every other ObjectInfo field is part of the production-visible +/// identity and must agree before the pool index can provide a deterministic +/// tie-break. +fn same_latest_object_info_identity(left: &ObjectInfo, right: &ObjectInfo) -> bool { + left.bucket == right.bucket + && left.name == right.name + && left.storage_class == right.storage_class + && left.mod_time == right.mod_time + && left.size == right.size + && left.actual_size == right.actual_size + && left.is_dir == right.is_dir + && same_user_defined_identity(left, right) + && left.user_tags == right.user_tags + && left.version_id == right.version_id + && left.data_dir == right.data_dir + && left.delete_marker == right.delete_marker + && same_transition_identity(left, right) + && left.restore_ongoing == right.restore_ongoing + && left.restore_expires == right.restore_expires + && left.parts == right.parts + && left.is_latest == right.is_latest + && left.content_type == right.content_type + && left.content_encoding == right.content_encoding + && left.expires == right.expires + && left.num_versions == right.num_versions + && left.successor_mod_time == right.successor_mod_time + && left.etag == right.etag + && left.inlined == right.inlined + && left.metadata_only == right.metadata_only + && left.version_only == right.version_only + && left.replication_status_internal == right.replication_status_internal + && left.replication_status == right.replication_status + && left.version_purge_status_internal == right.version_purge_status_internal + && left.version_purge_status == right.version_purge_status + && left.replication_decision == right.replication_decision + && left.checksum == right.checksum +} + pub(super) fn resolve_latest_object_info_candidates( - mut candidates: Vec, + candidates: Vec, bucket: &str, object: &str, opts: &ObjectOptions, ) -> Result<(ObjectInfo, usize)> { - candidates.sort_by(|a, b| { - let a_mod = if let Some(info) = &a.info { - info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH) - } else { - OffsetDateTime::UNIX_EPOCH + let latest_mod_time = candidates.iter().filter_map(latest_candidate_mod_time).max(); + + if let Some(latest_mod_time) = latest_mod_time { + let mut latest_candidates = candidates + .into_iter() + .filter(|candidate| latest_candidate_mod_time(candidate) == Some(latest_mod_time)) + .collect::>(); + + latest_candidates.sort_by_key(|candidate| std::cmp::Reverse(candidate.idx)); + + let Some(winner) = latest_candidates.first() else { + return Err(Error::ErasureReadQuorum); + }; + let Some(winner_info) = winner.info.as_ref() else { + return Err(Error::ErasureReadQuorum); }; - let b_mod = if let Some(info) = &b.info { - info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH) - } else { - OffsetDateTime::UNIX_EPOCH - }; - - if a_mod == b_mod { - return if a.idx < b.idx { Ordering::Greater } else { Ordering::Less }; + if latest_candidates.iter().skip(1).any(|candidate| { + candidate + .info + .as_ref() + .is_none_or(|info| !same_latest_object_info_identity(winner_info, info)) + }) { + return Err(Error::ErasureReadQuorum); } - b_mod.cmp(&a_mod) - }); + return Ok((winner_info.clone(), winner.idx)); + } for candidate in candidates { - if let Some(info) = candidate.info { - return Ok((info, candidate.idx)); - } - if let Some(err) = candidate.err && !is_err_object_not_found(&err) && !is_err_version_not_found(&err) @@ -191,6 +316,18 @@ mod tests { assert_eq!(err, Error::PreconditionFailed); } + #[test] + fn rebalance_delete_result_preserves_prefix_access_denied() { + let err = resolve_rebalance_delete_from_all_pools_result( + Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), + "bucket", + "object", + ) + .expect_err("prefix access denial should remain structured"); + + assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + } + #[test] fn rebalance_delete_pool_result_preserves_precondition_failed() { let err = resolve_rebalance_delete_from_all_pools_results( @@ -205,4 +342,19 @@ mod tests { assert_eq!(err, Error::PreconditionFailed); } + + #[test] + fn rebalance_delete_pool_result_preserves_prefix_access_denied() { + let err = resolve_rebalance_delete_from_all_pools_results( + vec![RebalanceDeletePoolResult { + pool_idx: 0, + result: Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), + }], + "bucket", + "object", + ) + .expect_err("prefix access denial should remain structured"); + + assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + } } diff --git a/crates/heal/src/heal/channel.rs b/crates/heal/src/heal/channel.rs index edfcf2613..e00435100 100644 --- a/crates/heal/src/heal/channel.rs +++ b/crates/heal/src/heal/channel.rs @@ -1640,6 +1640,60 @@ mod tests { assert_eq!(payload["items"].as_array().expect("items should be an array").len(), 0); } + #[tokio::test] + async fn test_process_query_request_reports_displaced_terminal_detail() { + let heal_manager = Arc::new(HealManager::new( + Arc::new(MockStorage), + Some(HealConfig { + queue_size: 1, + ..HealConfig::default() + }), + )); + let mut displaced = HealRequest::new( + HealType::Bucket { + bucket: "displaced-channel".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + displaced.id = "displaced-channel-task".to_string(); + let displaced_id = displaced.id.clone(); + heal_manager + .submit_heal_request(displaced) + .await + .expect("initial channel task should queue"); + heal_manager + .submit_heal_request(HealRequest::new( + HealType::Bucket { + bucket: "successor-channel".to_string(), + }, + HealOptions::default(), + HealPriority::High, + )) + .await + .expect("successor channel task should displace the initial task"); + + let processor = HealChannelProcessor::new(heal_manager); + let (tx, rx) = oneshot::channel(); + processor + .process_query_request("displaced-channel".to_string(), displaced_id, None, tx) + .await + .expect("displaced query should process"); + let response = rx + .await + .expect("query response should be returned") + .expect("displaced query should remain successful"); + let payload: serde_json::Value = serde_json::from_slice(response.data.as_deref().expect("status payload should exist")) + .expect("status payload should be json"); + assert_eq!(payload["summary"], "stopped"); + assert!( + response + .error + .as_deref() + .is_some_and(|detail| detail.contains("reason=displaced")) + ); + } + #[tokio::test] async fn test_process_query_request_reports_running_for_queued_task() { let heal_manager = create_test_heal_manager(); diff --git a/crates/heal/src/heal/manager.rs b/crates/heal/src/heal/manager.rs index bad6d4b5b..fc8255e0b 100644 --- a/crates/heal/src/heal/manager.rs +++ b/crates/heal/src/heal/manager.rs @@ -40,6 +40,7 @@ use tracing::{debug, error, info, warn}; use super::{DiskError, Endpoint, HealDiskExt as _, local_disk_map_read}; const KEEP_HEAL_TASK_STATUS_DURATION: Duration = Duration::from_secs(10 * 60); +const DISPLACED_HEAL_REASON: &str = "reason=displaced; retry_hint=submit_again"; const LOG_COMPONENT_HEAL: &str = "heal"; const LOG_SUBSYSTEM_DISK_SCANNER: &str = "disk_scanner"; const LOG_SUBSYSTEM_MANAGER: &str = "manager"; @@ -120,26 +121,30 @@ struct MrfRepairNoticeTarget { version_id: Option<[u8; 16]>, } -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone)] struct HealAdmissionDecision { result: HealAdmissionResult, - displaced_task_id: Option, + displaced_request: Option, } impl HealAdmissionDecision { const fn new(result: HealAdmissionResult) -> Self { Self { result, - displaced_task_id: None, + displaced_request: None, } } - fn accepted_with_displacement(displaced_task_id: String) -> Self { + fn accepted_with_displacement(displaced_request: HealRequest) -> Self { Self { result: HealAdmissionResult::Accepted, - displaced_task_id: Some(displaced_task_id), + displaced_request: Some(displaced_request), } } + + fn displaced_task_id(&self) -> Option<&str> { + self.displaced_request.as_ref().map(|request| request.id.as_str()) + } } fn lock_mrf_repair_notice_targets( @@ -151,6 +156,55 @@ fn lock_mrf_repair_notice_targets( } } +fn lock_displaced_terminals( + registry: &StdMutex>>, +) -> StdMutexGuard<'_, HashMap>> { + match registry.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + } +} + +fn record_displaced_terminal( + registry: &StdMutex>>, + request: &HealRequest, +) -> Arc { + let terminal = Arc::new(CompletedHealStatus { + heal_type: request.heal_type.clone(), + status: HealTaskStatus::Failed { + error: format!("heal task displaced by a higher-priority request ({DISPLACED_HEAL_REASON})"), + }, + result_items_truncated: false, + completed_at: SystemTime::now(), + seqed_items: Vec::new(), + next_seq: 0, + min_seq: 0, + }); + let mut terminals = lock_displaced_terminals(registry); + prune_completed_heal_statuses(&mut terminals); + terminals.insert(request.id.clone(), Arc::clone(&terminal)); + terminal +} + +async fn remove_displaced_task_aliases( + aliases: &Arc>>, + terminals: &StdMutex>>, + task_id: &str, + terminal: &Arc, +) { + let mut aliases = aliases.lock().await; + let alias_ids = aliases + .iter() + .filter_map(|(alias_id, alias)| (alias.task_id == task_id).then_some(alias_id.clone())) + .collect::>(); + let mut displaced_terminals = lock_displaced_terminals(terminals); + prune_completed_heal_statuses(&mut displaced_terminals); + for alias_id in alias_ids { + displaced_terminals.insert(alias_id, Arc::clone(terminal)); + } + aliases.retain(|alias_id, alias| alias_id != task_id && alias.task_id != task_id); +} + async fn remove_task_aliases_for_task(registry: &Arc>>, task_id: &str) { registry .lock() @@ -618,6 +672,14 @@ pub struct HealManager { /// are shared so the lookup helper can hand a completed entry to a /// caller without cloning the retained result window. completed_heals: Arc>>>, + /// Terminals for requests removed by priority displacement. An Accepted + /// task ID remains queryable for the same process lifetime and the normal + /// ten-minute status TTL; clients should treat `reason=displaced` as a + /// terminal result and submit a fresh request. This sidecar is synchronous + /// so admission can publish the terminal while the queue transition is + /// still under its lock, without awaiting another tokio lock. Queue state + /// is process-local, so this guarantee does not extend across restart. + displaced_terminals: Arc>>>, /// Client tokens merged into an existing task id. task_aliases: Arc>>, /// Heal tasks waiting for a retry backoff to expire. @@ -659,6 +721,7 @@ struct HealQueueContext<'a> { heal_queue: &'a Arc>, active_heals: &'a Arc>>>, completed_heals: &'a Arc>>>, + displaced_terminals: &'a Arc>>>, task_aliases: &'a Arc>>, retrying_heals: &'a Arc>>, mrf_repair_notice_targets: &'a Arc>>>, @@ -874,7 +937,7 @@ impl HealManager { result = "accepted_by_displacement", "Heal queue request accepted by displacement" }); - return HealAdmissionDecision::accepted_with_displacement(displaced.id); + return HealAdmissionDecision::accepted_with_displacement(displaced); } demote_to_debug_when!(per_object_request, warn, target: "rustfs::heal::manager", { @@ -1105,6 +1168,7 @@ impl HealManager { active_heals: Arc::new(Mutex::new(HashMap::new())), heal_queue: Arc::new(Mutex::new(PriorityHealQueue::new())), completed_heals: Arc::new(Mutex::new(HashMap::new())), + displaced_terminals: Arc::new(StdMutex::new(HashMap::new())), task_aliases: Arc::new(Mutex::new(HashMap::new())), retrying_heals: Arc::new(Mutex::new(HashMap::new())), mrf_repair_notice_targets: Arc::new(StdMutex::new(HashMap::new())), @@ -1209,6 +1273,10 @@ impl HealManager { active_heals.clear(); publish_active_heal_count(&active_heals); self.completed_heals.lock().await.clear(); + // Do not let the synchronous guard live across the following async lock. + { + lock_displaced_terminals(&self.displaced_terminals).clear(); + } self.task_aliases.lock().await.clear(); self.retrying_heals.lock().await.clear(); lock_mrf_repair_notice_targets(&self.mrf_repair_notice_targets).clear(); @@ -1459,7 +1527,11 @@ impl HealManager { task_id = queued_id.to_owned(); } let should_notify = matches!(admission, HealAdmissionResult::Accepted) && config.event_driven_scheduler_enable; - let displaced_task_id = admission_decision.displaced_task_id; + let displaced_task_id = admission_decision.displaced_task_id().map(ToOwned::to_owned); + let displaced_terminal = admission_decision + .displaced_request + .as_ref() + .map(|request| record_displaced_terminal(&self.displaced_terminals, request)); if matches!(admission, HealAdmissionResult::Accepted | HealAdmissionResult::Merged) && let Some(target) = mrf_notice_target { @@ -1473,8 +1545,12 @@ impl HealManager { drop(queue); drop(active_heals); - if let Some(displaced_task_id) = displaced_task_id { - self.remove_aliases_for_task(&displaced_task_id).await; + if let (Some(displaced_task_id), Some(displaced_terminal)) = (displaced_task_id, displaced_terminal) { + // The queue has already removed the displaced request, so the + // synchronous terminal sidecar was published before aliases and + // MRF ownership are cleaned up. + remove_displaced_task_aliases(&self.task_aliases, &self.displaced_terminals, &displaced_task_id, &displaced_terminal) + .await; } if should_notify { @@ -1549,6 +1625,15 @@ impl HealManager { } } + if terminal_completed.is_none() { + let mut displaced_terminals = lock_displaced_terminals(&self.displaced_terminals); + prune_completed_heal_statuses(&mut displaced_terminals); + terminal_completed = displaced_terminals + .get(canonical_task_id) + .filter(|terminal| matches_path(&terminal.heal_type)) + .cloned(); + } + match terminal_completed { Some(completed) => TaskStateLookup::Completed(completed), None => TaskStateLookup::NotFound, @@ -1669,9 +1754,19 @@ impl HealManager { let mut completed_heals = self.completed_heals.lock().await; prune_completed_heal_statuses(&mut completed_heals); - completed_heals + if completed_heals .values() .any(|completed| heal_type_matches_path(&completed.heal_type, heal_path)) + { + return true; + } + drop(completed_heals); + + let mut displaced_terminals = lock_displaced_terminals(&self.displaced_terminals); + prune_completed_heal_statuses(&mut displaced_terminals); + displaced_terminals + .values() + .any(|terminal| heal_type_matches_path(&terminal.heal_type, heal_path)) } /// Get task progress diff --git a/crates/heal/src/heal/manager/auto_scan.rs b/crates/heal/src/heal/manager/auto_scan.rs index 10ac136a4..f7eb4482d 100644 --- a/crates/heal/src/heal/manager/auto_scan.rs +++ b/crates/heal/src/heal/manager/auto_scan.rs @@ -21,6 +21,7 @@ impl HealManager { let heal_queue = self.heal_queue.clone(); let active_heals = self.active_heals.clone(); let task_aliases = self.task_aliases.clone(); + let displaced_terminals = self.displaced_terminals.clone(); let mrf_repair_notice_targets = self.mrf_repair_notice_targets.clone(); let storage = self.storage.clone(); let replacement_recovery_anchors = self.replacement_recovery_anchors.clone(); @@ -481,6 +482,10 @@ impl HealManager { let admission = admission_decision.result; let should_notify = matches!(admission, HealAdmissionResult::Accepted) && config.event_driven_scheduler_enable; + let displaced_terminal = admission_decision + .displaced_request + .as_ref() + .map(|request| record_displaced_terminal(&displaced_terminals, request)); if matches!(admission, HealAdmissionResult::Accepted) && let Some(anchor) = recovery_anchor { @@ -491,8 +496,16 @@ impl HealManager { } drop(queue); drop(config); - if let Some(displaced_task_id) = admission_decision.displaced_task_id { - remove_task_aliases_for_task(&task_aliases, &displaced_task_id).await; + if let (Some(displaced_task_id), Some(displaced_terminal)) = + (admission_decision.displaced_task_id().map(ToOwned::to_owned), displaced_terminal) + { + remove_displaced_task_aliases( + &task_aliases, + &displaced_terminals, + &displaced_task_id, + &displaced_terminal, + ) + .await; lock_mrf_repair_notice_targets(&mrf_repair_notice_targets).remove(&displaced_task_id); } if matches!(admission, HealAdmissionResult::Accepted) { diff --git a/crates/heal/src/heal/manager/scheduler.rs b/crates/heal/src/heal/manager/scheduler.rs index fbee9f212..c76a037ae 100644 --- a/crates/heal/src/heal/manager/scheduler.rs +++ b/crates/heal/src/heal/manager/scheduler.rs @@ -21,6 +21,7 @@ impl HealManager { let heal_queue = self.heal_queue.clone(); let active_heals = self.active_heals.clone(); let completed_heals = self.completed_heals.clone(); + let displaced_terminals = self.displaced_terminals.clone(); let task_aliases = self.task_aliases.clone(); let retrying_heals = self.retrying_heals.clone(); let mrf_repair_notice_targets = self.mrf_repair_notice_targets.clone(); @@ -53,6 +54,7 @@ impl HealManager { heal_queue: &heal_queue, active_heals: &active_heals, completed_heals: &completed_heals, + displaced_terminals: &displaced_terminals, task_aliases: &task_aliases, retrying_heals: &retrying_heals, mrf_repair_notice_targets: &mrf_repair_notice_targets, @@ -71,6 +73,7 @@ impl HealManager { heal_queue: &heal_queue, active_heals: &active_heals, completed_heals: &completed_heals, + displaced_terminals: &displaced_terminals, task_aliases: &task_aliases, retrying_heals: &retrying_heals, mrf_repair_notice_targets: &mrf_repair_notice_targets, @@ -98,6 +101,7 @@ impl HealManager { heal_queue, active_heals, completed_heals, + displaced_terminals, task_aliases, retrying_heals, mrf_repair_notice_targets, @@ -183,6 +187,7 @@ impl HealManager { let active_heals_clone = active_heals.clone(); let heal_queue_clone = heal_queue.clone(); let completed_heals_clone = completed_heals.clone(); + let displaced_terminals_clone = displaced_terminals.clone(); let task_aliases_clone = task_aliases.clone(); let retrying_heals_clone = retrying_heals.clone(); let mrf_repair_notice_targets_clone = mrf_repair_notice_targets.clone(); @@ -363,6 +368,7 @@ impl HealManager { let retry_heal_queue = heal_queue_clone.clone(); let retrying_heals_for_spawn = retrying_heals_clone.clone(); let retry_task_aliases = task_aliases_clone.clone(); + let retry_displaced_terminals = displaced_terminals_clone.clone(); let retry_mrf_repair_notice_targets = mrf_repair_notice_targets_clone.clone(); let retry_completed_heals = completed_heals_clone.clone(); let retry_notify = notify_clone.clone(); @@ -430,6 +436,14 @@ impl HealManager { let admission = admission_decision.result; let should_notify = matches!(admission, HealAdmissionResult::Accepted) && retry_config.event_driven_scheduler_enable; + // Publish the terminal synchronously while the + // queue transition is protected. The subsequent + // queue -> retrying handoff retains the lock order + // used by operations_snapshot. + let displaced_terminal = admission_decision + .displaced_request + .as_ref() + .map(|request| record_displaced_terminal(&retry_displaced_terminals, request)); match admission { HealAdmissionResult::Accepted => { // Transfer ownership while holding queue -> retrying, @@ -437,10 +451,18 @@ impl HealManager { #[cfg(test)] pause_retry_ownership_transition(&retry_request_id, true).await; retrying_heals_for_spawn.lock().await.remove(&retry_request_id); - let displaced_task_id = admission_decision.displaced_task_id; + let displaced_task_id = admission_decision.displaced_task_id().map(ToOwned::to_owned); drop(queue); - if let Some(displaced_task_id) = displaced_task_id { - remove_task_aliases_for_task(&retry_task_aliases, &displaced_task_id).await; + if let (Some(displaced_task_id), Some(displaced_terminal)) = + (displaced_task_id, displaced_terminal) + { + remove_displaced_task_aliases( + &retry_task_aliases, + &retry_displaced_terminals, + &displaced_task_id, + &displaced_terminal, + ) + .await; remove_mrf_repair_notice_targets( &retry_mrf_repair_notice_targets, &displaced_task_id, diff --git a/crates/heal/src/heal/manager/tests.rs b/crates/heal/src/heal/manager/tests.rs index ad50c4bf7..585b7a4e7 100644 --- a/crates/heal/src/heal/manager/tests.rs +++ b/crates/heal/src/heal/manager/tests.rs @@ -84,6 +84,7 @@ async fn process_manager_queue_once(manager: &HealManager) { heal_queue: &manager.heal_queue, active_heals: &manager.active_heals, completed_heals: &manager.completed_heals, + displaced_terminals: &manager.displaced_terminals, task_aliases: &manager.task_aliases, retrying_heals: &manager.retrying_heals, mrf_repair_notice_targets: &manager.mrf_repair_notice_targets, @@ -2778,7 +2779,10 @@ async fn test_high_priority_request_displaces_lower_priority_when_queue_full() { HealAdmissionResult::Accepted ); assert_eq!(manager.get_queue_length().await, 1); - assert!(matches!(manager.get_task_status(&low_id).await, Err(Error::TaskNotFound { .. }))); + assert!(matches!( + manager.get_task_status(&low_id).await, + Ok(HealTaskStatus::Failed { error }) if error.contains("reason=displaced") + )); assert_eq!( manager .get_task_status(&high_id) @@ -2788,6 +2792,263 @@ async fn test_high_priority_request_displaces_lower_priority_when_queue_full() { ); } +#[tokio::test] +async fn displaced_task_remains_queryable() { + let manager = HealManager::new( + Arc::new(MockStorage), + Some(HealConfig { + queue_size: 1, + ..HealConfig::default() + }), + ); + let mut displaced = HealRequest::new( + HealType::Bucket { + bucket: "displaced-bucket".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + displaced.id = "displaced-task".to_string(); + let displaced_id = displaced.id.clone(); + manager + .submit_heal_request(displaced) + .await + .expect("displaced request should queue"); + + let successor = HealRequest::new( + HealType::Bucket { + bucket: "successor-bucket".to_string(), + }, + HealOptions::default(), + HealPriority::High, + ); + manager + .submit_heal_request(successor) + .await + .expect("successor should displace low work"); + + let report = manager + .get_task_report(&displaced_id) + .await + .expect("displaced report should remain queryable"); + assert!(matches!(report.status, HealTaskStatus::Failed { ref error } if error.contains("reason=displaced"))); +} + +#[tokio::test] +async fn displaced_archive_failure_keeps_queryable_terminal() { + let manager = HealManager::new(Arc::new(MockStorage), None); + let mut request = HealRequest::new( + HealType::Bucket { + bucket: "archive-failure".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + request.id = "archive-failure-task".to_string(); + let request_id = request.id.clone(); + // The synchronous sidecar is the authoritative fallback when the normal + // completed-task archive has no entry (the failure window that must not + // turn an Accepted ID into NotFound). + record_displaced_terminal(&manager.displaced_terminals, &request); + assert!(manager.completed_heals.lock().await.is_empty()); + assert!(matches!( + manager.get_task_status(&request_id).await, + Ok(HealTaskStatus::Failed { error }) if error.contains("reason=displaced") + )); +} + +#[tokio::test] +async fn scheduler_retry_displacement_keeps_evicted_task_queryable() { + let manager = Arc::new(HealManager::new( + Arc::new(MockStorage), + Some(HealConfig { + queue_size: 1, + event_driven_scheduler_enable: false, + ..HealConfig::default() + }), + )); + let mut retry_request = HealRequest::object("retry-transition".to_string(), "object".to_string(), None); + retry_request.priority = HealPriority::High; + let retry_id = retry_request.id.clone(); + manager + .submit_heal_request(retry_request) + .await + .expect("retry request should queue"); + + // Process exactly one queue cycle so the retry task is spawned without a + // background scheduler consuming the filler request before the retry wakes. + process_manager_queue_once(&manager).await; + tokio::time::timeout(Duration::from_secs(1), async { + loop { + if manager.retrying_heals.lock().await.contains_key(&retry_id) { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("retry request should enter backoff"); + + let filler = HealRequest::new( + HealType::Bucket { + bucket: "retry-displaced-filler".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + let filler_id = filler.id.clone(); + manager + .submit_heal_request(filler) + .await + .expect("filler request should occupy the queue"); + + tokio::time::timeout(Duration::from_secs(5), async { + loop { + if matches!( + manager.get_task_status(&filler_id).await, + Ok(HealTaskStatus::Failed { ref error }) if error.contains("reason=displaced") + ) { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("retry admission should displace the filler request"); + assert_eq!(manager.get_queue_length().await, 1); + assert_eq!( + manager.get_task_status(&retry_id).await.expect("retry should be queued"), + HealTaskStatus::Pending + ); +} + +#[tokio::test] +async fn concurrent_displacers_produce_one_terminal_generation() { + let manager = Arc::new(HealManager::new( + Arc::new(MockStorage), + Some(HealConfig { + queue_size: 1, + ..HealConfig::default() + }), + )); + let mut displaced = HealRequest::new( + HealType::Bucket { + bucket: "concurrent-displaced".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + displaced.id = "concurrent-displaced-task".to_string(); + let displaced_id = displaced.id.clone(); + manager + .submit_heal_request(displaced) + .await + .expect("initial request should queue"); + + let first = HealRequest::new( + HealType::Bucket { + bucket: "concurrent-successor-a".to_string(), + }, + HealOptions::default(), + HealPriority::High, + ); + let second = HealRequest::new( + HealType::Bucket { + bucket: "concurrent-successor-b".to_string(), + }, + HealOptions::default(), + HealPriority::High, + ); + let (first_result, second_result) = tokio::join!(manager.submit_heal_request(first), manager.submit_heal_request(second)); + let accepted = [&first_result, &second_result] + .into_iter() + .filter(|result| matches!(result, Ok(HealAdmissionResult::Accepted))) + .count(); + assert_eq!(accepted, 1, "exactly one concurrent displacer should win the full queue"); + assert!( + first_result.is_ok() && second_result.is_ok(), + "the losing request should receive a typed Full result" + ); + let terminals = lock_displaced_terminals(&manager.displaced_terminals); + assert_eq!(terminals.len(), 1); + assert!(terminals.contains_key(&displaced_id)); +} + +#[tokio::test] +async fn successor_chain_is_bounded_and_authorized() { + let manager = HealManager::new( + Arc::new(MockStorage), + Some(HealConfig { + queue_size: 1, + ..HealConfig::default() + }), + ); + let mut original = HealRequest::new( + HealType::Bucket { + bucket: "authorized-original".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + original.id = "authorized-original-task".to_string(); + let original_id = original.id.clone(); + manager.submit_heal_request(original).await.expect("original should queue"); + let mut duplicate = HealRequest::new( + HealType::Bucket { + bucket: "authorized-original".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + duplicate.id = "authorized-duplicate-task".to_string(); + let duplicate_id = duplicate.id.clone(); + manager + .submit_heal_request(duplicate) + .await + .expect("same-target duplicate should merge"); + let successor = HealRequest::new( + HealType::Bucket { + bucket: "authorized-successor".to_string(), + }, + HealOptions::default(), + HealPriority::High, + ); + let successor_id = successor.id.clone(); + manager.submit_heal_request(successor).await.expect("successor should queue"); + assert!(manager.task_aliases.lock().await.is_empty()); + assert!(matches!(manager.get_task_status(&original_id).await, Ok(HealTaskStatus::Failed { .. }))); + assert!(matches!(manager.get_task_status(&duplicate_id).await, Ok(HealTaskStatus::Failed { .. }))); + assert_eq!( + manager + .get_task_status(&successor_id) + .await + .expect("successor should remain queued"), + HealTaskStatus::Pending + ); +} + +#[tokio::test] +async fn displaced_terminal_expires_after_bounded_ttl() { + let manager = HealManager::new(Arc::new(MockStorage), None); + let mut request = HealRequest::new( + HealType::Bucket { + bucket: "expires".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + request.id = "expires-task".to_string(); + let request_id = request.id.clone(); + record_displaced_terminal(&manager.displaced_terminals, &request); + { + let mut terminals = lock_displaced_terminals(&manager.displaced_terminals); + let entry = + Arc::get_mut(terminals.get_mut(&request_id).expect("terminal should be retained")).expect("test owns terminal entry"); + entry.completed_at = SystemTime::now() - KEEP_HEAL_TASK_STATUS_DURATION - Duration::from_secs(1); + } + assert!(matches!(manager.get_task_status(&request_id).await, Err(Error::TaskNotFound { .. }))); +} + #[tokio::test] async fn test_displacing_registered_mrf_task_drops_notice_ownership() { let storage: Arc = Arc::new(MockStorage); diff --git a/crates/heal/src/heal/task/heal_erasure_set.rs b/crates/heal/src/heal/task/heal_erasure_set.rs index e0cfe5a90..7b25bcba2 100644 --- a/crates/heal/src/heal/task/heal_erasure_set.rs +++ b/crates/heal/src/heal/task/heal_erasure_set.rs @@ -231,6 +231,10 @@ impl HealTask { "Heal erasure set format repair skipped because no format heal was required" ); } else { + let error = e; + if error.is_recoverable_heal() { + return Err(error); + } error!( target: "rustfs::heal::task", event = EVENT_HEAL_ERASURE_SET_RESULT, @@ -239,7 +243,7 @@ impl HealTask { task_id = %self.id, set_disk_id, result = "format_failed", - error = %e, + error = %error, "Heal erasure set failed" ); { @@ -247,7 +251,7 @@ impl HealTask { progress.update_progress(4, 4, 0, 0); } return Err(Error::TaskExecutionFailed { - message: format!("Failed to heal disk format for {set_disk_id}: {e}"), + message: format!("Failed to heal disk format for {set_disk_id}: {error}"), }); } } else { @@ -284,6 +288,9 @@ impl HealTask { Err(Error::TaskCancelled) => return Err(Error::TaskCancelled), Err(Error::TaskTimeout) => return Err(Error::TaskTimeout), Err(e) => { + if e.is_recoverable_heal() { + return Err(e); + } error!( target: "rustfs::heal::task", event = EVENT_HEAL_ERASURE_SET_RESULT, diff --git a/crates/heal/src/heal/task/tests.rs b/crates/heal/src/heal/task/tests.rs index 464ff3606..f2b442205 100644 --- a/crates/heal/src/heal/task/tests.rs +++ b/crates/heal/src/heal/task/tests.rs @@ -547,6 +547,7 @@ struct MockStorage { heal_object_outcome: Mutex>, heal_object_outcomes: Mutex>>, format_no_heal_required: Mutex, + format_error: Mutex>, global_format_calls: Mutex, replacement_format_calls: Mutex)>>, replacement_targets_ready: Mutex, @@ -867,6 +868,9 @@ impl HealStorageAPI for MockStorage { async fn heal_format(&self, _dry_run: bool) -> Result<(HealResultItem, Option)> { *self.global_format_calls.lock().unwrap() += 1; + if let Some(error) = self.format_error.lock().unwrap().take() { + return Err(error); + } let no_heal_required = *self.format_no_heal_required.lock().unwrap(); if no_heal_required { Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::NoHealRequired)))) @@ -2052,6 +2056,30 @@ async fn test_erasure_set_heal_continues_after_format_no_heal_required() { ); } +#[tokio::test] +async fn erasure_set_format_slowdown_is_propagated() { + let storage = Arc::new(MockStorage { + format_error: Mutex::new(Some(Error::Storage(EcstoreError::SlowDown))), + ..Default::default() + }); + let request = HealRequest::new( + HealType::ErasureSet { + buckets: Vec::new(), + set_disk_id: "pool_0_set_0".to_string(), + }, + HealOptions::default(), + HealPriority::Normal, + ); + let task = HealTask::from_request(request, storage); + + let error = task + .execute() + .await + .expect_err("format SlowDown must remain recoverable for the task manager"); + + assert!(matches!(error, Error::Storage(EcstoreError::SlowDown))); +} + #[tokio::test] async fn erasure_set_bucket_prepass_failure_stops_before_object_heal() { let temp = TempDir::new().expect("temporary directory should be created"); diff --git a/crates/iam/AGENTS.md b/crates/iam/AGENTS.md index c9c8332fa..c8bce3778 100644 --- a/crates/iam/AGENTS.md +++ b/crates/iam/AGENTS.md @@ -24,4 +24,3 @@ Applies to `crates/iam/`. ## Suggested Validation - `cargo test -p rustfs-iam` -- Full gate before commit: `make pre-commit` diff --git a/crates/io-metrics/src/internode_metrics.rs b/crates/io-metrics/src/internode_metrics.rs index 65f93c2a1..6b1ea382c 100644 --- a/crates/io-metrics/src/internode_metrics.rs +++ b/crates/io-metrics/src/internode_metrics.rs @@ -54,6 +54,13 @@ pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE: &str = "read_versio pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE: &str = "read_version_response_msgpack_encode"; pub const INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP: &str = "read_version_rpc_roundtrip"; pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE: &str = "read_version_response_decode"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE: &str = "batch_read_version_request_encode"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_DECODE: &str = "batch_read_version_request_decode"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_DISK_READ: &str = "batch_read_version_disk_read"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_JSON_ENCODE: &str = "batch_read_version_response_json_encode"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_MSGPACK_ENCODE: &str = "batch_read_version_response_msgpack_encode"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP: &str = "batch_read_version_rpc_roundtrip"; +pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE: &str = "batch_read_version_response_decode"; const OPERATION_LABEL: &str = "operation"; const BACKEND_LABEL: &str = "backend"; diff --git a/crates/lifecycle/Cargo.toml b/crates/lifecycle/Cargo.toml index ae8232f20..483b0e7d7 100644 --- a/crates/lifecycle/Cargo.toml +++ b/crates/lifecycle/Cargo.toml @@ -69,7 +69,7 @@ uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnos [dev-dependencies] metrics-util = { workspace = true, features = ["debugging"] } proptest = "1" -serial_test.workspace = true +serial_test = { workspace = true } temp-env.workspace = true tokio = { workspace = true, features = ["macros", "fs", "rt-multi-thread"] } diff --git a/crates/lifecycle/src/core.rs b/crates/lifecycle/src/core.rs index 4d1f5298d..6180d0022 100644 --- a/crates/lifecycle/src/core.rs +++ b/crates/lifecycle/src/core.rs @@ -1564,7 +1564,6 @@ mod tests { } #[tokio::test] - #[serial] async fn abort_incomplete_multipart_upload_due_accepts_zero_days() { let initiated = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -1625,7 +1624,6 @@ mod tests { } #[tokio::test] - #[serial] async fn predict_expiration_selects_closest_expiry_for_put_object() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -1872,7 +1870,6 @@ mod tests { } #[tokio::test] - #[serial] async fn empty_transition_vectors_are_not_active_or_due() { let lc = BucketLifecycleConfiguration { expiry_updated_at: None, @@ -1938,7 +1935,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_keeps_latest_object_before_days_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -1972,7 +1968,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_transitions_latest_object_after_days_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2010,7 +2005,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_transitions_latest_object_after_date_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let transition_date = base_time - Duration::days(1); @@ -2050,7 +2044,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_selects_earliest_due_among_multiple_past_due_events() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); // Two enabled rules both yield a past-due DeleteAction and a third yields a @@ -2164,7 +2157,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_expires_noncurrent_version_after_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2202,7 +2194,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_skips_noncurrent_expiration_without_successor() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp"); let lc = BucketLifecycleConfiguration { @@ -2238,7 +2229,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_missing_successor_does_not_skip_noncurrent_transition() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp"); let lc = BucketLifecycleConfiguration { @@ -2281,7 +2271,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_noncurrent_expiration_one_day_respects_due_boundary() { let successor_time = datetime!(2025-06-15 12:00:00 UTC); let due = expected_expiry_time(successor_time, 1); @@ -2323,7 +2312,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_expires_noncurrent_version_immediately_when_zero_days() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2361,7 +2349,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_transitions_noncurrent_version_after_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2437,7 +2424,6 @@ mod tests { } #[tokio::test] - #[serial] async fn evaluator_honors_newer_noncurrent_versions_retention_count() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = Arc::new(BucketLifecycleConfiguration { @@ -2726,7 +2712,6 @@ mod tests { } #[tokio::test] - #[serial] async fn expired_object_delete_marker_ignores_marker_with_noncurrent_versions_present() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2803,7 +2788,6 @@ mod tests { } #[tokio::test] - #[serial] async fn expired_object_delete_marker_deletes_only_delete_marker_immediately() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2881,7 +2865,6 @@ mod tests { } #[tokio::test] - #[serial] async fn expiration_days_deletes_only_expired_delete_marker_when_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -2932,7 +2915,6 @@ mod tests { } #[tokio::test] - #[serial] async fn expiration_days_uses_earliest_due_rule_for_expired_delete_marker() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let make_rule = |id: &str, days| LifecycleRule { @@ -3263,7 +3245,6 @@ mod tests { } #[tokio::test] - #[serial] async fn del_marker_expiration_deletes_marker_and_older_versions_when_due() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid"); let lc = BucketLifecycleConfiguration { @@ -3303,7 +3284,6 @@ mod tests { // --- TASK-003 tests: Round up to next UTC processing boundary --- #[test] - #[serial] fn expected_expiry_time_rounds_up_to_next_midnight_utc() { with_default_ilm_process_time(|| { // Object created at 2025-01-15T10:30:45Z, expire in 30 days @@ -3319,7 +3299,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_immediate_expiry_returns_epoch() { with_default_ilm_process_time(|| { let mod_time = datetime!(2025-06-01 12:00:00 UTC); @@ -3329,7 +3308,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_preserves_exact_midnight_boundary() { with_default_ilm_process_time(|| { let mod_time = datetime!(2025-03-01 00:00:00 UTC); @@ -3339,7 +3317,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_rounds_end_of_day_to_following_midnight() { with_default_ilm_process_time(|| { let mod_time = datetime!(2025-06-15 23:59:59 UTC); @@ -3349,7 +3326,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_uses_canonical_process_time_boundary() { let mod_time = datetime!(2025-01-15 10:30:45 UTC); @@ -3362,7 +3338,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_uses_deprecated_process_time_alias() { let mod_time = datetime!(2025-01-15 10:30:45 UTC); @@ -3375,7 +3350,6 @@ mod tests { } #[test] - #[serial] fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() { let mod_time = datetime!(2025-01-15 10:30:45 UTC); @@ -3398,7 +3372,6 @@ mod tests { // (a) Default path (env unset) is byte-identical: one day == 86400s. #[test] - #[serial] fn ilm_day_secs_defaults_to_86400_when_unset() { temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || { assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS); @@ -3427,7 +3400,6 @@ mod tests { // (b) End-to-end env read scales the day length. #[test] - #[serial] fn ilm_day_secs_scales_when_env_set() { temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || { assert_eq!(ilm_day_secs(), 2); @@ -3436,7 +3408,6 @@ mod tests { // (c) Invalid env value falls back to 86400. #[test] - #[serial] fn ilm_day_secs_falls_back_on_invalid_env() { temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("bogus"), || { assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS); @@ -3449,7 +3420,6 @@ mod tests { // Deadline math scales: with a 1s day and PROCESS_TIME unset, a Days=1 rule is // due 1s after mod_time (rounded up to the next 1s boundary => same instant). #[test] - #[serial] fn expected_expiry_time_scales_with_debug_day_secs() { let mod_time = datetime!(2025-01-15 10:30:45 UTC); temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("1"), || { @@ -3465,7 +3435,6 @@ mod tests { // days == 0 still yields the immediate-expiry sentinel regardless of the switch. #[test] - #[serial] fn expected_expiry_time_zero_days_ignores_debug_day_secs() { let mod_time = datetime!(2025-06-01 12:00:00 UTC); temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || { @@ -3476,7 +3445,6 @@ mod tests { // (③) Interaction with an explicit RUSTFS_ILM_PROCESS_TIME: the deadline offset // uses the accelerated day length, but the rounding boundary honors PROCESS_TIME. #[test] - #[serial] fn expected_expiry_time_debug_day_secs_respects_explicit_process_time() { let mod_time = datetime!(2025-01-15 10:30:00 UTC); // day == 10s, but round up to the next 60s (PROCESS_TIME) boundary. @@ -3493,7 +3461,6 @@ mod tests { // (③) With the switch unset, an explicit PROCESS_TIME behaves exactly as before. #[test] - #[serial] fn expected_expiry_time_unset_debug_day_secs_matches_legacy_process_time() { let mod_time = datetime!(2025-01-15 10:30:45 UTC); temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || { @@ -3521,7 +3488,6 @@ mod tests { // The abort-incomplete-multipart deadline path also scales through the switch. #[test] - #[serial] fn abort_incomplete_multipart_due_scales_with_debug_day_secs() { use s3s::dto::AbortIncompleteMultipartUpload; let initiated = datetime!(2025-01-15 10:30:45 UTC); @@ -3566,7 +3532,6 @@ mod tests { // (⑤ evaluator seam) A Days=1 rule fires under RUSTFS_ILM_DEBUG_DAY_SECS=1 once // `now` advances a few seconds past a mod_time only ~seconds in the past. #[test] - #[serial] fn eval_inner_expires_days_one_rule_under_debug_day_secs() { let lc = BucketLifecycleConfiguration { expiry_updated_at: None, @@ -3615,7 +3580,6 @@ mod tests { // Absolute Date-based rules must NOT scale with the switch (regression guard). #[test] - #[serial] fn eval_inner_date_rule_ignores_debug_day_secs() { let expiry_date = datetime!(2025-06-01 00:00:00 UTC); let lc = BucketLifecycleConfiguration { @@ -3873,7 +3837,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_triggers_delete_all_versions_when_expired_object_all_versions_set() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -3912,7 +3875,6 @@ mod tests { } #[tokio::test] - #[serial] async fn expired_object_all_versions_does_not_apply_to_current_delete_marker() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid"); let lc = BucketLifecycleConfiguration { @@ -3942,7 +3904,6 @@ mod tests { } #[tokio::test] - #[serial] async fn eval_inner_uses_delete_action_when_all_versions_not_set() { let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap(); let lc = BucketLifecycleConfiguration { @@ -4061,7 +4022,6 @@ mod tests { use super::*; use proptest::prelude::*; use s3s::dto::{NoncurrentVersionExpiration, Tag}; - use serial_test::serial; const DAY_SECS: i64 = 86400; @@ -4292,7 +4252,6 @@ mod tests { /// combination, and must be deterministic: the same input /// evaluated twice yields an identical event. #[test] - #[serial] fn eval_inner_never_panics_and_is_deterministic( rules in prop::collection::vec(arb_rule(), 0..4), obj in arb_object_opts(), @@ -4432,7 +4391,6 @@ mod tests { /// candidate set — earliest due wins, ties prefer delete-class — /// and must be `NoneAction` exactly when that set is empty. #[test] - #[serial] fn eval_inner_winner_matches_selection_oracle( rules in prop::collection::vec(arb_selection_rule(), 0..5), mod_off in 0i64..(2 * DAY_SECS), @@ -4486,7 +4444,6 @@ mod tests { /// non-decreasing in `days` (days == 0 maps to UNIX_EPOCH, below /// any post-1970 deadline). #[test] - #[serial] fn expected_expiry_time_is_monotonic_in_days( mod_off in 0i64..(3650 * DAY_SECS), d1 in 0i32..2000, @@ -4508,7 +4465,6 @@ mod tests { /// to the next whole-day boundary: the result is day-aligned, not /// before `mod_time + days`, and less than one boundary beyond it. #[test] - #[serial] fn expected_expiry_time_lands_on_default_day_boundary( mod_off in 0i64..(3650 * DAY_SECS), days in 1i32..2000, @@ -4526,7 +4482,6 @@ mod tests { /// to that boundary instead: aligned to it, never early, and less /// than one boundary late. #[test] - #[serial] fn expected_expiry_time_lands_on_explicit_process_boundary( mod_off in 0i64..(365 * DAY_SECS), days in 1i32..400, diff --git a/crates/madmin/src/metrics.rs b/crates/madmin/src/metrics.rs index 49e1e5ff2..4e55563a5 100644 --- a/crates/madmin/src/metrics.rs +++ b/crates/madmin/src/metrics.rs @@ -689,6 +689,14 @@ pub struct ScannerMetrics { pub cycle_max_objects: u64, #[serde(rename = "cycle_max_directories", default)] pub cycle_max_directories: u64, + #[serde(rename = "cycle_timeout_total", default)] + pub cycle_timeout_total: u64, + #[serde(rename = "cycle_recovery_required_total", default)] + pub cycle_recovery_required_total: u64, + #[serde(rename = "cycle_last_progress_age", default)] + pub cycle_last_progress_age: u64, + #[serde(rename = "leader_lease_without_progress", default)] + pub leader_lease_without_progress: bool, #[serde(rename = "bitrot_cycle_enabled", default)] pub bitrot_cycle_enabled: bool, #[serde(rename = "bitrot_cycle_seconds", default)] @@ -764,6 +772,8 @@ impl ScannerMetrics { self.cycle_max_duration_seconds = other.cycle_max_duration_seconds; self.cycle_max_objects = other.cycle_max_objects; self.cycle_max_directories = other.cycle_max_directories; + self.cycle_last_progress_age = other.cycle_last_progress_age; + self.leader_lease_without_progress = other.leader_lease_without_progress; self.bitrot_cycle_enabled = other.bitrot_cycle_enabled; self.bitrot_cycle_seconds = other.bitrot_cycle_seconds; } @@ -857,6 +867,12 @@ impl ScannerMetrics { .saturating_add(other.last_cycle_replication_checks); self.last_cycle_usage_saves = self.last_cycle_usage_saves.saturating_add(other.last_cycle_usage_saves); self.failed_cycles = self.failed_cycles.saturating_add(other.failed_cycles); + self.cycle_timeout_total = self.cycle_timeout_total.saturating_add(other.cycle_timeout_total); + self.cycle_recovery_required_total = self + .cycle_recovery_required_total + .saturating_add(other.cycle_recovery_required_total); + self.cycle_last_progress_age = self.cycle_last_progress_age.max(other.cycle_last_progress_age); + self.leader_lease_without_progress |= other.leader_lease_without_progress; self.superseded_cycles = self.superseded_cycles.saturating_add(other.superseded_cycles); self.partial_cycles_unknown = self.partial_cycles_unknown.saturating_add(other.partial_cycles_unknown); self.partial_cycles_runtime = self.partial_cycles_runtime.saturating_add(other.partial_cycles_runtime); diff --git a/crates/madmin/src/trace.rs b/crates/madmin/src/trace.rs index 1c63d81e5..9863973c6 100644 --- a/crates/madmin/src/trace.rs +++ b/crates/madmin/src/trace.rs @@ -12,18 +12,16 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::{collections::HashMap, time::Duration}; - -use jiff::Timestamp; use serde::{Deserialize, Serialize}; -use crate::heal_commands::HealResultItem; - +/// Bitflag helper for service trace categories. +/// +/// Each variant occupies a single bit so that a `TraceType` value can represent +/// an arbitrary combination of categories via bitwise OR. #[derive(Debug, Clone, Copy, Serialize, Deserialize, Default)] pub struct TraceType(u64); impl TraceType { - // Define some constants pub const OS: TraceType = TraceType(1 << 0); pub const STORAGE: TraceType = TraceType(1 << 1); pub const S3: TraceType = TraceType(1 << 2); @@ -40,15 +38,13 @@ impl TraceType { pub const FTP: TraceType = TraceType(1 << 13); pub const ILM: TraceType = TraceType(1 << 14); - // MetricsAll must be last. + /// All trace categories combined. Must be updated when adding new variants. pub const ALL: TraceType = TraceType((1 << 15) - 1); pub fn new(t: u64) -> Self { Self(t) } -} -impl TraceType { pub fn contains(&self, x: &TraceType) -> bool { (self.0 & x.0) == x.0 } @@ -76,140 +72,38 @@ impl TraceType { } } -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceInfo { - #[serde(rename = "type")] - trace_type: u64, - #[serde(rename = "nodename")] - node_name: String, - #[serde(rename = "funcname")] - func_name: String, - #[serde(rename = "time")] - time: Timestamp, - #[serde(rename = "path")] - path: String, - #[serde(rename = "dur")] - duration: Duration, - #[serde(rename = "bytes", skip_serializing_if = "Option::is_none")] - bytes: Option, - #[serde(rename = "msg", skip_serializing_if = "Option::is_none")] - message: Option, - #[serde(rename = "error", skip_serializing_if = "Option::is_none")] - error: Option, - #[serde(rename = "custom", skip_serializing_if = "Option::is_none")] - custom: Option>, - #[serde(rename = "http", skip_serializing_if = "Option::is_none")] - http: Option, - #[serde(rename = "healResult", skip_serializing_if = "Option::is_none")] - heal_result: Option, -} - -impl TraceInfo { - pub fn mask(&self) -> u64 { - TraceType::new(self.trace_type).mask() - } -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceInfoLegacy { - trace_info: TraceInfo, - #[serde(rename = "request")] - req_info: Option, - #[serde(rename = "response")] - resp_info: Option, - #[serde(rename = "stats")] - call_stats: Option, - #[serde(rename = "storageStats")] - storage_stats: Option, - #[serde(rename = "osStats")] - os_stats: Option, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct StorageStats { - path: String, - duration: Duration, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct OSStats { - path: String, - duration: Duration, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceHTTPStats { - req_info: TraceRequestInfo, - resp_info: TraceResponseInfo, - call_stats: TraceCallStats, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceCallStats { - input_bytes: i32, - output_bytes: i32, - latency: Duration, - time_to_first_byte: Duration, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceRequestInfo { - time: Timestamp, - proto: String, - method: String, - #[serde(skip_serializing_if = "Option::is_none")] - path: Option, - #[serde(skip_serializing_if = "Option::is_none")] - raw_query: Option, - #[serde(skip_serializing_if = "Option::is_none")] - headers: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - body: Option>, - client: String, -} - -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct TraceResponseInfo { - time: Timestamp, - #[serde(skip_serializing_if = "Option::is_none")] - headers: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - body: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - status_code: Option, -} - #[cfg(test)] mod tests { use super::*; #[test] - fn trace_timestamps_serialize_as_rfc3339_utc() { - let timestamp = Timestamp::constant(1_700_000_000, 123_456_000); - let trace = TraceInfo { - time: timestamp, - http: Some(TraceHTTPStats { - req_info: TraceRequestInfo { - time: timestamp, - ..Default::default() - }, - resp_info: TraceResponseInfo { - time: timestamp, - ..Default::default() - }, - ..Default::default() - }), - ..Default::default() - }; + fn trace_type_contains_and_overlaps() { + let mut combined = TraceType::default(); + combined.merge(&TraceType::S3); + combined.merge(&TraceType::HEALING); - let value = serde_json::to_value(trace).expect("trace should serialize"); - assert_eq!(value["time"], "2023-11-14T22:13:20.123456Z"); - assert_eq!(value["http"]["req_info"]["time"], "2023-11-14T22:13:20.123456Z"); - assert_eq!(value["http"]["resp_info"]["time"], "2023-11-14T22:13:20.123456Z"); - let trace: TraceInfo = serde_json::from_value(value).expect("trace should deserialize"); - assert_eq!(trace.time, timestamp); - let http = trace.http.expect("http trace should deserialize"); - assert_eq!(http.req_info.time, timestamp); - assert_eq!(http.resp_info.time, timestamp); + assert!(combined.contains(&TraceType::S3)); + assert!(combined.contains(&TraceType::HEALING)); + assert!(!combined.contains(&TraceType::SCANNER)); + assert!(combined.overlaps(&TraceType::S3)); + assert!(combined.overlaps(&TraceType::HEALING)); + assert!(!combined.overlaps(&TraceType::SCANNER)); + } + + #[test] + fn trace_type_set_if() { + let mut tt = TraceType::default(); + tt.set_if(true, &TraceType::OS); + tt.set_if(false, &TraceType::S3); + assert!(tt.contains(&TraceType::OS)); + assert!(!tt.contains(&TraceType::S3)); + } + + #[test] + fn trace_type_single_type() { + assert!(TraceType::S3.single_type()); + let mut combined = TraceType::S3; + combined.merge(&TraceType::HEALING); + assert!(!combined.single_type()); } } diff --git a/crates/notify/AGENTS.md b/crates/notify/AGENTS.md index 04d996a90..9a142ac13 100644 --- a/crates/notify/AGENTS.md +++ b/crates/notify/AGENTS.md @@ -55,4 +55,3 @@ shared plugin/runtime primitives from `rustfs-targets`. - Focused: `cargo test -p rustfs-notify runtime_facade` - Focused: `cargo test -p rustfs-notify runtime_view` - Focused: `cargo test -p rustfs-notify config_manager` -- Full gate before commit: `make pre-commit` diff --git a/crates/object-capacity/Cargo.toml b/crates/object-capacity/Cargo.toml index a1eea7740..b37d58f3b 100644 --- a/crates/object-capacity/Cargo.toml +++ b/crates/object-capacity/Cargo.toml @@ -73,7 +73,6 @@ walkdir = { workspace = true } [dev-dependencies] criterion = { workspace = true, features = ["html_reports"] } -serial_test = { workspace = true } temp-env = { workspace = true, features = ["async_closure"] } tempfile = { workspace = true } tokio = { workspace = true, features = ["test-util", "macros", "fs", "rt-multi-thread"] } diff --git a/crates/object-capacity/src/capacity_manager.rs b/crates/object-capacity/src/capacity_manager.rs index 2d70f985c..99c3a63bc 100644 --- a/crates/object-capacity/src/capacity_manager.rs +++ b/crates/object-capacity/src/capacity_manager.rs @@ -1484,7 +1484,6 @@ mod tests { ENV_CAPACITY_SAMPLE_RATE, ENV_CAPACITY_STAT_TIMEOUT, ENV_CAPACITY_WRITE_FREQUENCY_THRESHOLD, ENV_CAPACITY_WRITE_TRIGGER_DELAY, }; - use serial_test::serial; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -1669,7 +1668,6 @@ mod tests { } #[test] - #[serial] fn test_config_getter_defaults() { for (env_var, getter, default, _, _) in config_getter_cases() { temp_env::with_var(env_var, None::<&str>, || { @@ -1679,7 +1677,6 @@ mod tests { } #[test] - #[serial] fn test_config_getter_env_overrides() { for (env_var, getter, _, override_value, expected) in config_getter_cases() { temp_env::with_var(env_var, Some(override_value), || { @@ -1689,7 +1686,6 @@ mod tests { } #[test] - #[serial] fn test_zero_env_values_clamp_to_defaults() { // A zero threshold makes small disks report 0 bytes; a zero timeout // (with dynamic timeout off) makes every scan fail. Both must fall @@ -1709,7 +1705,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_update_capacity_preserves_retrieval_metadata() { let manager = HybridCapacityManager::from_env(); @@ -1725,7 +1720,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_record_write_operation() { let manager = HybridCapacityManager::from_env(); @@ -1736,7 +1730,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_write_frequency_window() { let manager = HybridCapacityManager::from_env(); @@ -1824,7 +1817,6 @@ mod tests { } #[test] - #[serial] fn test_recent_write_count_ignores_future_buckets() { let record = WriteRecord::new(); record.write_buckets[0].store(120, 3); @@ -1838,7 +1830,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_needs_fast_update() { let manager = HybridCapacityManager::from_env(); @@ -1855,7 +1846,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_cache_age_tracking() { let manager = HybridCapacityManager::from_env(); @@ -1875,7 +1865,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_data_source_tracking() { let manager = HybridCapacityManager::from_env(); @@ -1891,7 +1880,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_needs_fast_update_waits_for_write_trigger_delay() { let manager = create_isolated_manager(HybridStrategyConfig { scheduled_update_interval: Duration::from_secs(60), @@ -1922,7 +1910,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_needs_fast_update_respects_enable_write_trigger() { let manager = create_isolated_manager(HybridStrategyConfig { scheduled_update_interval: Duration::from_secs(60), @@ -1949,7 +1936,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_concurrent_access() { let manager = Arc::new(HybridCapacityManager::from_env()); let mut handles = Vec::new(); @@ -1976,7 +1962,6 @@ mod tests { // exact under heavy same-second contention or the frequency window (and the // write-trigger decision) would undercount. #[tokio::test(flavor = "multi_thread", worker_threads = 8)] - #[serial] async fn test_record_write_operation_lock_free_is_exact_under_contention() { let manager = Arc::new(HybridCapacityManager::from_env()); let mut handles = Vec::new(); @@ -2001,7 +1986,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_performance_overhead() { let manager = Arc::new(HybridCapacityManager::from_env()); let start = Instant::now(); @@ -2018,7 +2002,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_refresh_or_join_singleflight() { let manager = Arc::new(HybridCapacityManager::from_env()); let calls = Arc::new(AtomicUsize::new(0)); @@ -2058,7 +2041,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_refresh_or_join_recovers_after_leader_cancellation() { let manager = Arc::new(HybridCapacityManager::from_env()); @@ -2087,7 +2069,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_refresh_or_join_cancelled_leader_unblocks_joiner() { let manager = Arc::new(HybridCapacityManager::from_env()); @@ -2115,7 +2096,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_spawn_refresh_if_needed_deduplicates_background_refresh() { let manager = Arc::new(HybridCapacityManager::from_env()); let calls = Arc::new(AtomicUsize::new(0)); @@ -2153,7 +2133,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_record_write_operation_with_scope_token_marks_dirty_disks() { let manager = create_isolated_manager(HybridStrategyConfig::default()); let token = uuid::Uuid::new_v4(); @@ -2177,7 +2156,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_get_dirty_disks_drains_global_dirty_scope_registry() { let manager = create_isolated_manager(HybridStrategyConfig::default()); record_global_dirty_scope(CapacityScope { @@ -2197,7 +2175,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_update_capacity_recomputes_total_from_disk_cache_for_subset_refresh() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2308,7 +2285,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_update_capacity_degraded_full_refresh_merges_cache_and_does_not_oscillate() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2354,7 +2330,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_update_capacity_degraded_with_empty_per_disk_serves_merged_cache() { let manager = create_isolated_manager(HybridStrategyConfig::default()); manager.update_capacity(full_two_disk_update(), DataSource::RealTime).await; @@ -2384,7 +2359,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_update_capacity_degraded_without_complete_cache_keeps_partial_sum() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2425,7 +2399,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_commit_keeps_dirty_marks_recorded_after_scan_start() { let manager = create_isolated_manager(HybridStrategyConfig::default()); let disk = scope_disk("node-a", "/tmp/disk-a"); @@ -2456,7 +2429,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_commit_clears_dirty_marks_recorded_before_scan_start() { let manager = create_isolated_manager(HybridStrategyConfig::default()); let disk = scope_disk("node-a", "/tmp/disk-a"); @@ -2477,7 +2449,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_retain_dirty_disks_within_drops_ghost_entries() { let manager = create_isolated_manager(HybridStrategyConfig::default()); let local = scope_disk("node-a", "/tmp/disk-a"); @@ -2496,7 +2467,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_spawn_refresh_recovers_from_construction_panic() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2563,7 +2533,6 @@ mod tests { } #[tokio::test(start_paused = true)] - #[serial] async fn test_refresh_or_join_joiner_times_out_when_leader_wedges() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2591,7 +2560,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_refresh_or_join_returns_cluster_total_for_dirty_subset() { let manager = create_isolated_manager(HybridStrategyConfig::default()); @@ -2673,7 +2641,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_config_from_env() { let config = HybridStrategyConfig::from_env(); @@ -2687,7 +2654,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_config_from_env_with_override() { temp_env::with_var(ENV_CAPACITY_SCHEDULED_INTERVAL, Some("600"), || { let config = HybridStrategyConfig::from_env(); diff --git a/crates/object-capacity/src/scan.rs b/crates/object-capacity/src/scan.rs index bf974cda6..d93b99736 100644 --- a/crates/object-capacity/src/scan.rs +++ b/crates/object-capacity/src/scan.rs @@ -1069,7 +1069,6 @@ mod tests { #[cfg(unix)] use rustfs_config::ENV_CAPACITY_FOLLOW_SYMLINKS; use rustfs_config::{ENV_CAPACITY_MAX_FILES_THRESHOLD, ENV_CAPACITY_SAMPLE_RATE}; - use serial_test::serial; /// Reference implementation using unbounded `u128` arithmetic, clamped to /// `u64::MAX`, used as the source of truth for the sampling extrapolation. @@ -1274,7 +1273,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_get_dir_size_async_nonexistent_directory() { let result = get_dir_size_async(Path::new("/nonexistent/path")).await; assert!(result.is_err()); @@ -1648,7 +1646,6 @@ mod tests { } #[tokio::test] - #[serial] async fn test_metadata_incomplete_aggregate_does_not_replace_disk_cache() { use std::fs::File; use std::io::Write; @@ -1783,7 +1780,6 @@ mod tests { #[cfg(unix)] #[tokio::test] - #[serial] async fn test_get_dir_size_async_ignores_symlink_targets_when_follow_disabled() { use std::fs::File; use std::io::Write; @@ -1809,7 +1805,6 @@ mod tests { #[cfg(unix)] #[tokio::test] - #[serial] async fn test_get_dir_size_async_counts_symlink_targets_when_follow_enabled() { use std::fs::File; use std::io::Write; diff --git a/crates/obs/src/metrics/collectors/bucket_replication.rs b/crates/obs/src/metrics/collectors/bucket_replication.rs index a393c8e38..bbe7de6fe 100644 --- a/crates/obs/src/metrics/collectors/bucket_replication.rs +++ b/crates/obs/src/metrics/collectors/bucket_replication.rs @@ -75,7 +75,7 @@ pub struct BucketReplicationBandwidthStats { } #[derive(Debug, Clone, Default)] -pub struct BucketReplicationStats { +pub struct BucketReplicationMetricsSnapshot { pub bucket: String, pub total_failed_bytes: u64, pub total_failed_count: u64, @@ -107,7 +107,7 @@ pub struct BucketReplicationStats { #[derive(Debug, Clone, Default)] pub(crate) struct BucketReplicationRuntimeStats { - pub(crate) stats: BucketReplicationStats, + pub(crate) stats: BucketReplicationMetricsSnapshot, pub(crate) target_flows: Vec, } @@ -182,7 +182,7 @@ fn push_proxy_request_result_metrics( } } -pub fn collect_bucket_replication_metrics(stats: &[BucketReplicationStats]) -> Vec { +pub fn collect_bucket_replication_metrics(stats: &[BucketReplicationMetricsSnapshot]) -> Vec { if stats.is_empty() { return Vec::new(); } @@ -572,7 +572,7 @@ mod tests { #[test] fn test_collect_bucket_replication_metrics() { let stats = vec![BucketReplicationRuntimeStats { - stats: BucketReplicationStats { + stats: BucketReplicationMetricsSnapshot { bucket: "b1".to_string(), total_failed_bytes: 64, total_failed_count: 2, @@ -876,7 +876,7 @@ mod tests { #[test] fn test_collect_bucket_replication_metrics_empty() { - let stats: Vec = Vec::new(); + let stats: Vec = Vec::new(); let metrics = collect_bucket_replication_metrics(&stats); assert!(metrics.is_empty()); } diff --git a/crates/obs/src/metrics/collectors/mod.rs b/crates/obs/src/metrics/collectors/mod.rs index 67df19035..c4f849a4d 100644 --- a/crates/obs/src/metrics/collectors/mod.rs +++ b/crates/obs/src/metrics/collectors/mod.rs @@ -48,7 +48,7 @@ pub(crate) use bucket_replication::{ BucketReplicationTargetFlowStats, collect_bucket_replication_backlog_metrics, collect_bucket_replication_runtime_metrics, }; pub use bucket_replication::{ - BucketReplicationBandwidthStats, BucketReplicationStats, BucketReplicationTargetStats, + BucketReplicationBandwidthStats, BucketReplicationMetricsSnapshot, BucketReplicationTargetStats, collect_bucket_replication_bandwidth_metrics, collect_bucket_replication_metrics, }; pub use cluster::{ClusterStats, collect_cluster_metrics}; @@ -68,8 +68,8 @@ pub(crate) use notification::collect_notification_runtime_metrics; pub use notification::{NotificationStats, collect_notification_metrics}; pub(crate) use notification_target::{NotificationTargetRuntimeStats, collect_notification_target_runtime_metrics}; pub use notification_target::{NotificationTargetStats, collect_notification_target_metrics}; +pub use replication::{ReplicationMetricsSnapshot, collect_replication_metrics}; pub(crate) use replication::{ReplicationRuntimeStats, collect_replication_runtime_metrics}; -pub use replication::{ReplicationStats, collect_replication_metrics}; pub(crate) use request::{ApiRequestMetricSupport, ApiRequestStats, collect_request_metrics}; pub use resource::{ResourceStats, collect_resource_metrics}; pub(crate) use scanner::{ScannerRuntimeStats, collect_scanner_runtime_metrics}; diff --git a/crates/obs/src/metrics/collectors/replication.rs b/crates/obs/src/metrics/collectors/replication.rs index 4dc347058..d483c71ef 100644 --- a/crates/obs/src/metrics/collectors/replication.rs +++ b/crates/obs/src/metrics/collectors/replication.rs @@ -22,7 +22,7 @@ use crate::metrics::schema::replication::*; /// Replication statistics. #[derive(Debug, Clone, Default)] -pub struct ReplicationStats { +pub struct ReplicationMetricsSnapshot { /// Average number of active replication workers pub average_active_workers: f64, /// Average queued bytes since server start @@ -54,13 +54,13 @@ pub struct ReplicationStats { #[derive(Debug, Clone, Default)] pub(crate) struct ReplicationRuntimeStats { pub(crate) server: String, - pub(crate) stats: ReplicationStats, + pub(crate) stats: ReplicationMetricsSnapshot, } /// Collects replication metrics from the given stats. /// /// Returns a vector of Prometheus metrics for replication statistics. -pub fn collect_replication_metrics(stats: &ReplicationStats) -> Vec { +pub fn collect_replication_metrics(stats: &ReplicationMetricsSnapshot) -> Vec { vec![ PrometheusMetric::from_descriptor(&REPLICATION_AVERAGE_ACTIVE_WORKERS_MD, stats.average_active_workers), PrometheusMetric::from_descriptor(&REPLICATION_AVERAGE_QUEUED_BYTES_MD, stats.average_queued_bytes as f64), @@ -120,7 +120,7 @@ mod tests { #[test] fn test_collect_replication_metrics() { - let stats = ReplicationStats { + let stats = ReplicationMetricsSnapshot { average_active_workers: 8.5, average_queued_bytes: 1024 * 1024 * 40, average_queued_count: 240, @@ -182,7 +182,7 @@ mod tests { #[test] fn test_collect_replication_metrics_default() { - let stats = ReplicationStats::default(); + let stats = ReplicationMetricsSnapshot::default(); let metrics = collect_replication_metrics(&stats); assert_eq!(metrics.len(), 13); @@ -194,7 +194,7 @@ mod tests { #[test] fn replication_stats_struct_literal_keeps_legacy_fields() { - let stats = ReplicationStats { + let stats = ReplicationMetricsSnapshot { average_active_workers: 1.0, average_queued_bytes: 2, average_queued_count: 3, diff --git a/crates/obs/src/metrics/scheduler.rs b/crates/obs/src/metrics/scheduler.rs index 088799d5e..29734268f 100644 --- a/crates/obs/src/metrics/scheduler.rs +++ b/crates/obs/src/metrics/scheduler.rs @@ -2811,14 +2811,14 @@ mod tests { #[test] fn replication_proxy_bucket_keys_detect_removed_buckets() { let previous = repl_proxy_bucket_live_keys(&[BucketReplicationRuntimeStats { - stats: crate::metrics::BucketReplicationStats { + stats: crate::metrics::BucketReplicationMetricsSnapshot { bucket: "photos".to_string(), ..Default::default() }, ..Default::default() }]); let current = repl_proxy_bucket_live_keys(&[BucketReplicationRuntimeStats { - stats: crate::metrics::BucketReplicationStats { + stats: crate::metrics::BucketReplicationMetricsSnapshot { bucket: "logs".to_string(), ..Default::default() }, diff --git a/crates/obs/src/metrics/stats_collector.rs b/crates/obs/src/metrics/stats_collector.rs index 20e461952..0d3863762 100644 --- a/crates/obs/src/metrics/stats_collector.rs +++ b/crates/obs/src/metrics/stats_collector.rs @@ -21,12 +21,12 @@ use crate::metrics::collectors::scanner::{ScannerActiveBucketDriveStats, ScannerBucketDriveResultStats, ScannerSourceWorkStats}; use crate::metrics::collectors::{ ApiRequestMetricSupport, ApiRequestStats, BucketReplicationBacklogStats, BucketReplicationBandwidthStats, - BucketReplicationRuntimeStats, BucketReplicationStats, BucketReplicationTargetBacklogStats, BucketReplicationTargetFlowStats, - BucketReplicationTargetStats, BucketStats, BucketUsageStats, ClusterConfigStats, ClusterHealthStats, ClusterStats, - ClusterUsageStats, CompressionClusterStats, CpuStats, DiskStats, DriveCountStats, DriveDetailedStats, - DriveRuntimeDetailedStats, ErasureSetStats, HostNetworkStats, IamStats, IlmActionTaskStats, IlmBackpressureStats, - IlmQueueTaskStats, IlmRuntimeStats, IlmStats, IlmTaskEventStats, MemoryStats, NetworkStats, ProcessStats, ProcessStatusType, - ReplicationStats, ResourceStats, ScannerRuntimeStats, ScannerStats, + BucketReplicationMetricsSnapshot, BucketReplicationRuntimeStats, BucketReplicationTargetBacklogStats, + BucketReplicationTargetFlowStats, BucketReplicationTargetStats, BucketStats, BucketUsageStats, ClusterConfigStats, + ClusterHealthStats, ClusterStats, ClusterUsageStats, CompressionClusterStats, CpuStats, DiskStats, DriveCountStats, + DriveDetailedStats, DriveRuntimeDetailedStats, ErasureSetStats, HostNetworkStats, IamStats, IlmActionTaskStats, + IlmBackpressureStats, IlmQueueTaskStats, IlmRuntimeStats, IlmStats, IlmTaskEventStats, MemoryStats, NetworkStats, + ProcessStats, ProcessStatusType, ReplicationMetricsSnapshot, ResourceStats, ScannerRuntimeStats, ScannerStats, }; use crate::metrics::runtime_sources::{ObsIlmRuntimeSnapshot, bucket_monitor_handle, iam_metrics_snapshot, ilm_runtime_snapshot}; use crate::metrics::{ @@ -266,7 +266,7 @@ fn bucket_replication_detail_from_snapshot(stats: ObsBucketReplicationStatsSnaps BucketReplicationRuntimeStats { target_flows, - stats: BucketReplicationStats { + stats: BucketReplicationMetricsSnapshot { bucket, total_failed_bytes: stats.total_failed_bytes, total_failed_count: stats.total_failed_count, @@ -298,7 +298,7 @@ fn bucket_replication_detail_from_snapshot(stats: ObsBucketReplicationStatsSnaps } } -async fn obs_site_replication_stats() -> ReplicationStats { +async fn obs_site_replication_stats() -> ReplicationMetricsSnapshot { let current_data_transfer_rate = obs_bucket_replication_bandwidth_stats() .into_iter() .flatten() @@ -306,7 +306,7 @@ async fn obs_site_replication_stats() -> ReplicationStats { .sum::(); let stats = obs_replication_site_stats_snapshot(current_data_transfer_rate).await; - ReplicationStats { + ReplicationMetricsSnapshot { average_active_workers: stats.average_active_workers, average_queued_bytes: stats.average_queued_bytes, average_queued_count: stats.average_queued_count, @@ -648,7 +648,7 @@ pub fn collect_bucket_replication_bandwidth_stats() -> Vec Vec { +pub async fn collect_bucket_replication_detail_stats() -> Vec { obs_bucket_replication_stats_snapshot() .await .into_iter() @@ -662,7 +662,7 @@ pub(crate) async fn collect_bucket_replication_stats_bundle() } /// Collect site-level replication stats from the global replication runtime. -pub async fn collect_replication_stats() -> ReplicationStats { +pub async fn collect_replication_stats() -> ReplicationMetricsSnapshot { obs_site_replication_stats().await } diff --git a/crates/policy/AGENTS.md b/crates/policy/AGENTS.md index 75abf24ea..ac3f86ebe 100644 --- a/crates/policy/AGENTS.md +++ b/crates/policy/AGENTS.md @@ -23,4 +23,3 @@ Applies to `crates/policy/`. ## Suggested Validation - `cargo test -p rustfs-policy` -- Full gate before commit: `make pre-commit` diff --git a/crates/policy/src/policy/policy.rs b/crates/policy/src/policy/policy.rs index d412fa789..946f930a8 100644 --- a/crates/policy/src/policy/policy.rs +++ b/crates/policy/src/policy/policy.rs @@ -195,7 +195,8 @@ pub struct BucketPolicyArgs<'a> { #[derive(Serialize, Deserialize, Clone, Default, Debug)] #[serde(deny_unknown_fields)] pub struct BucketPolicy { - #[serde(default, rename = "Id", skip_serializing_if = "ID::is_empty")] + // RUSTFS_COMPAT_TODO(rustfs-6339): accept bucket policies persisted with the legacy "ID" key. Remove after migration tooling rewrites every retained legacy bucket policy. + #[serde(default, rename = "Id", alias = "ID", skip_serializing_if = "ID::is_empty")] pub id: ID, #[serde(rename = "Version")] pub version: String, @@ -2786,7 +2787,7 @@ mod test { let parsed: serde_json::Value = serde_json::from_str(&json).expect("Should parse"); // Verify empty fields are omitted - assert!(!parsed.as_object().unwrap().contains_key("ID"), "Empty ID should be omitted"); + assert!(parsed.get("Id").is_none(), "Empty ID should be omitted"); let statement = &parsed["Statement"][0]; assert!(!statement.as_object().unwrap().contains_key("Sid"), "Empty Sid should be omitted"); @@ -2809,6 +2810,43 @@ mod test { assert_eq!(statement["Principal"]["AWS"], "*"); } + #[test] + fn test_bucket_policy_deserializes_legacy_id() { + let legacy_policy = br#"{"ID":"","Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":["*"]},"Action":["s3:GetObject"],"NotAction":[],"Resource":["arn:aws:s3:::bucket/*"],"NotResource":[],"Condition":{}}]}"#; + + let policy: BucketPolicy = + serde_json::from_slice(legacy_policy).expect("bucket policy with legacy ID should deserialize"); + assert!(policy.id.is_empty()); + policy.is_valid().expect("legacy bucket policy should remain valid"); + + let policy: BucketPolicy = serde_json::from_str(r#"{"ID":"legacy-policy","Version":"2012-10-17","Statement":[]}"#) + .expect("non-empty legacy ID should deserialize"); + assert_eq!(policy.id.0, "legacy-policy"); + + let serialized = serde_json::to_value(&policy).expect("bucket policy should serialize"); + assert_eq!(serialized["Id"], "legacy-policy"); + assert!(serialized.get("ID").is_none(), "legacy ID spelling should not be serialized"); + } + + #[test] + fn test_bucket_policy_legacy_id_alias_remains_strict() { + let unknown_field = r#"{"Version":"2012-10-17","Statement":[],"Unexpected":true}"#; + let error = + serde_json::from_str::(unknown_field).expect_err("unrelated unknown fields should remain rejected"); + assert!( + error.to_string().contains("unknown field `Unexpected`"), + "unexpected deserialization error: {error}" + ); + + let duplicate_id = r#"{"Id":"current-policy","ID":"legacy-policy","Version":"2012-10-17","Statement":[]}"#; + let error = serde_json::from_str::(duplicate_id) + .expect_err("canonical and legacy ID fields should not be accepted together"); + assert!( + error.to_string().contains("duplicate field `Id`"), + "unexpected deserialization error: {error}" + ); + } + #[test] fn test_existing_object_tag_condition_helpers() { let identity_policy = Policy::parse_config( diff --git a/crates/protos/src/generated/proto_gen/node_service.rs b/crates/protos/src/generated/proto_gen/node_service.rs index 6fa01bf1e..3885be4a3 100644 --- a/crates/protos/src/generated/proto_gen/node_service.rs +++ b/crates/protos/src/generated/proto_gen/node_service.rs @@ -722,6 +722,10 @@ pub struct DeleteVersionsResponse { pub errors: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, #[prost(message, optional, tag = "3")] pub error: ::core::option::Option, + /// Senders dual-write the legacy strings and typed entries. Receivers prefer typed entries + /// when present and fall back to strings for peers that predate this field. Code zero means success. + #[prost(message, repeated, tag = "4")] + pub item_errors: ::prost::alloc::vec::Vec, } #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct ReadMultipleRequest { diff --git a/crates/protos/src/lib.rs b/crates/protos/src/lib.rs index a9e59b8a0..ccfb8197d 100644 --- a/crates/protos/src/lib.rs +++ b/crates/protos/src/lib.rs @@ -2106,6 +2106,9 @@ pub enum ChannelClass { Bulk, } +// Keep multiplexed unary RPCs below h2's per-connection small-frame budget. +const INTERNODE_RPC_CONCURRENCY_LIMIT: usize = 64; + /// Whether control/bulk channel isolation is enabled (env-gated, default off for safe rollout). fn channel_isolation_enabled() -> bool { rustfs_utils::get_env_bool( @@ -2188,6 +2191,7 @@ async fn build_channel(dial_addr: &str, cache_key: &str) -> Result( } } +/// Read only the object revision without materializing its body. +pub(crate) async fn read_config_revision(store: Arc, path: &str) -> StorageResult { + match store + .get_object_reader( + RUSTFS_META_BUCKET, + path, + None, + HeaderMap::new(), + &ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(reader) => reader + .object_info + .etag + .filter(|etag| !etag.is_empty()) + .map(DataUsageCacheRevision::Etag) + .ok_or_else(|| StorageError::other(format!("scanner config object {path} has no ETag"))), + Err(Error::FileNotFound | Error::VolumeNotFound | Error::ObjectNotFound(_, _) | Error::BucketNotFound(_)) => { + Ok(DataUsageCacheRevision::Missing) + } + Err(err) => Err(err), + } +} + #[derive(Clone, Debug)] pub(crate) struct DataUsageCacheRevisions { main: DataUsageCacheRevision, @@ -146,6 +174,11 @@ pub static LEGACY_DATA_USAGE_OBJ_NAME_PATH: LazyLock = pub static DATA_USAGE_BLOOM_NAME_PATH: LazyLock = LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{DATA_USAGE_BLOOM_NAME}")); +/// Durable companion object for a cycle-state object which cannot be decoded. +/// The primary object is deliberately never replaced or deleted by recovery. +pub static DATA_USAGE_BLOOM_RECOVERY_PATH: LazyLock = + LazyLock::new(|| format!("{}.recovery-required.json", DATA_USAGE_BLOOM_NAME_PATH.as_str())); + pub static BACKGROUND_HEAL_INFO_PATH: LazyLock = LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}.background-heal.json")); diff --git a/crates/scanner/src/data_usage_define/persistence.rs b/crates/scanner/src/data_usage_define/persistence.rs index 2ac453cf4..b0a28f504 100644 --- a/crates/scanner/src/data_usage_define/persistence.rs +++ b/crates/scanner/src/data_usage_define/persistence.rs @@ -74,7 +74,7 @@ impl DataUsageCache { let loaded = Self::load_cache(store.clone(), name).await?; let backup = match loaded.backup_revision { Some(revision) => Some(revision), - None => match Self::revision_for_path(store, &backup_path).await { + None => match read_config_revision(store, &backup_path).await { Ok(revision) => Some(revision), Err(err) => { counter!(METRIC_CACHE_BACKUP_REVISION_FAILURE_TOTAL).increment(1); @@ -336,33 +336,6 @@ impl DataUsageCache { } } - async fn revision_for_path(store: Arc, path: &str) -> StorageResult { - match store - .get_object_reader( - RUSTFS_META_BUCKET, - path, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { - Ok(reader) => reader - .object_info - .etag - .filter(|etag| !etag.is_empty()) - .map(DataUsageCacheRevision::Etag) - .ok_or_else(|| StorageError::other(format!("scanner cache object {path} has no ETag"))), - Err(Error::FileNotFound | Error::VolumeNotFound | Error::ObjectNotFound(_, _) | Error::BucketNotFound(_)) => { - Ok(DataUsageCacheRevision::Missing) - } - Err(err) => Err(err), - } - } - pub(super) fn cache_save_timeout() -> Duration { crate::runtime_config::scanner_cache_save_timeout() } diff --git a/crates/scanner/src/data_usage_define/tests.rs b/crates/scanner/src/data_usage_define/tests.rs index ed3fcd544..bdccb11c1 100644 --- a/crates/scanner/src/data_usage_define/tests.rs +++ b/crates/scanner/src/data_usage_define/tests.rs @@ -16,7 +16,7 @@ use super::persistence::DataUsageCacheLoadAttempt; use super::*; use crate::storage_api::scanner_io::{HTTPRangeSpec, ObjectIO}; use crate::{ScannerGetObjectReader, ScannerPutObjReader}; -use rustfs_data_usage::{ReplicationAllStats, ReplicationStats}; +use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage}; use serde_json::Value; use std::io::Cursor; use std::pin::Pin; @@ -1636,7 +1636,7 @@ fn size_recursive_prunes_empty_and_preserves_threshold_replication_stats() { replication_stats: Some(ReplicationAllStats { targets: HashMap::from([( "arn:test:threshold".to_string(), - ReplicationStats { + ReplicationTargetUsage { after_threshold_count: 1, ..Default::default() }, diff --git a/crates/scanner/src/lib.rs b/crates/scanner/src/lib.rs index 0e347a8c1..ab01964a5 100644 --- a/crates/scanner/src/lib.rs +++ b/crates/scanner/src/lib.rs @@ -75,7 +75,10 @@ pub use remote_scanner::{ }; pub use runtime_config::{apply_scanner_runtime_config, scanner_runtime_config_status, validate_scanner_runtime_config}; pub use rustfs_common::last_minute; -pub use scanner::{ScannerCycleScheduleStatus, init_data_scanner, scanner_cycle_schedule_status, scanner_topology_digest}; +pub use scanner::{ + ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus, ScannerCycleScheduleStatus, init_data_scanner, + reset_scanner_cycle_recovery, scanner_cycle_recovery_status, scanner_cycle_schedule_status, scanner_topology_digest, +}; pub use scanner_io::{ ScannerDirtyUsageAckError, ScannerDirtyUsageState, acknowledge_dirty_usage_generation, clear_dirty_usage_bucket, record_dirty_usage_bucket, record_scanner_maintenance_change, scanner_activity_epoch, scanner_dirty_usage_state, @@ -599,10 +602,8 @@ impl ScannerConfigObjectDelete for ECStore { #[cfg(test)] mod tests { use super::*; - use serial_test::serial; #[tokio::test] - #[serial] async fn runtime_tier_names_serves_cached_arc_within_ttl() { reset_tier_name_cache_for_test(); // The tier config manager is unconfigured in unit tests, so the @@ -616,7 +617,6 @@ mod tests { } #[test] - #[serial] fn foreground_read_guard_tracks_stream_lifetime() { reset_foreground_read_activity_for_test(); assert_eq!(current_foreground_read_activity(), 0); @@ -630,7 +630,6 @@ mod tests { } #[test] - #[serial] fn foreground_read_activity_keeps_larger_signal() { reset_foreground_read_activity_for_test(); let _guard = ForegroundReadGuard::new(); @@ -643,7 +642,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_guard_tracks_runtime_lifetime() { reset_scanner_runtime_instances_for_test(); assert!(!scanner_runtime_initialized()); diff --git a/crates/scanner/src/runtime_config.rs b/crates/scanner/src/runtime_config.rs index da951339d..2fef6f9f4 100644 --- a/crates/scanner/src/runtime_config.rs +++ b/crates/scanner/src/runtime_config.rs @@ -125,7 +125,10 @@ impl Default for ScannerRuntimeConfig { cycle_interval_source: ScannerRuntimeConfigSource::Default, bitrot_cycle: Some(Duration::from_secs(DEFAULT_HEAL_BITROT_CYCLE_SECS)), bitrot_cycle_source: ScannerRuntimeConfigSource::Default, - cycle_budget: ScannerCycleBudgetConfig::default(), + cycle_budget: ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS)), + ..Default::default() + }, cycle_max_duration_source: ScannerRuntimeConfigSource::Default, cycle_max_objects_source: ScannerRuntimeConfigSource::Default, cycle_max_directories_source: ScannerRuntimeConfigSource::Default, @@ -374,7 +377,10 @@ fn validate_persisted_scanner_runtime_config(config: &ServerConfig) -> Result<() } validate_optional_config_u64(scanner_kvs, SCANNER_START_DELAY, "")?; validate_optional_config_u64(scanner_kvs, SCANNER_CYCLE, "")?; - validate_optional_config_u64(scanner_kvs, SCANNER_CYCLE_MAX_DURATION, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS)?; + if let Some(value) = config_value(scanner_kvs, SCANNER_CYCLE_MAX_DURATION, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS) { + let secs = parse_config_u64(SCANNER_CYCLE_MAX_DURATION, value)?; + cycle_duration_from_secs(SCANNER_CYCLE_MAX_DURATION, secs)?; + } validate_optional_config_u64(scanner_kvs, SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS)?; validate_optional_config_u64(scanner_kvs, SCANNER_CYCLE_MAX_DIRECTORIES, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES)?; if let Some(value) = config_value(heal_kvs, HEAL_BITROT_CYCLE, DEFAULT_HEAL_BITROT_CYCLE_SECS) { @@ -436,19 +442,46 @@ fn lookup_max_wait( Ok((speed.max_sleep(), speed_source)) } -fn lookup_optional_seconds( - kvs: Option<&KVS>, - key: &'static str, - env_key: &'static str, - default: u64, -) -> Result<(Option, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { - if let Some(secs) = rustfs_utils::get_env_opt_u64(env_key) { - return Ok((Some(Duration::from_secs(secs)), ScannerRuntimeConfigSource::Env)); +fn lookup_cycle_duration(kvs: Option<&KVS>) -> Result<(Option, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { + match rustfs_utils::get_env_parse_outcome::(ENV_SCANNER_CYCLE_MAX_DURATION_SECS) { + rustfs_utils::EnvParseOutcome::Parsed(secs) => { + return cycle_duration_from_secs(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, secs) + .map(|duration| (duration, ScannerRuntimeConfigSource::Env)); + } + rustfs_utils::EnvParseOutcome::Invalid => { + // Do not include the raw environment value in the typed error: + // deployments occasionally put sensitive material in inherited + // environment snapshots. The key still identifies the control. + return Err(invalid_value( + ENV_SCANNER_CYCLE_MAX_DURATION_SECS, + "", + "expected unsigned integer seconds", + )); + } + rustfs_utils::EnvParseOutcome::Absent => {} } - if let Some(value) = config_value(kvs, key, default) { - return parse_config_u64(key, value).map(|secs| (Some(Duration::from_secs(secs)), ScannerRuntimeConfigSource::Config)); + + if let Some(value) = config_value(kvs, SCANNER_CYCLE_MAX_DURATION, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS) { + let secs = parse_config_u64(SCANNER_CYCLE_MAX_DURATION, value)?; + return cycle_duration_from_secs(SCANNER_CYCLE_MAX_DURATION, secs) + .map(|duration| (duration, ScannerRuntimeConfigSource::Config)); } - Ok((None, ScannerRuntimeConfigSource::Default)) + + Ok(( + Some(Duration::from_secs(DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS)), + ScannerRuntimeConfigSource::Default, + )) +} + +fn cycle_duration_from_secs(key: &'static str, secs: u64) -> Result, ScannerRuntimeConfigError> { + if secs == 0 { + return Ok(None); + } + let duration = Duration::from_secs(secs); + if std::time::Instant::now().checked_add(duration).is_none() { + return Err(invalid_value(key, "", "duration exceeds the timer range")); + } + Ok(Some(duration)) } fn lookup_start_delay(kvs: Option<&KVS>) -> Result<(Option, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { @@ -553,12 +586,7 @@ pub(crate) fn lookup_scanner_runtime_config( (speed.cycle_interval(), speed_source) }; - let (cycle_max_duration, cycle_max_duration_source) = lookup_optional_seconds( - scanner_kvs, - SCANNER_CYCLE_MAX_DURATION, - ENV_SCANNER_CYCLE_MAX_DURATION_SECS, - DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, - )?; + let (cycle_max_duration, cycle_max_duration_source) = lookup_cycle_duration(scanner_kvs)?; let (cycle_max_objects, cycle_max_objects_source) = lookup_count_budget( scanner_kvs, SCANNER_CYCLE_MAX_OBJECTS, @@ -863,12 +891,11 @@ mod tests { use rustfs_config::server_config::{Config as ServerConfig, KVS}; use rustfs_config::{ DEFAULT_DELIMITER, DEFAULT_HEAL_BITROT_CYCLE_SECS, ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, - ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, - HEAL_BITROT_CYCLE, HEAL_SUB_SYS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, - SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, - SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed, + ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, + ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, SCANNER_BITROT_CYCLE, + SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, + SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed, }; - use serial_test::serial; use std::collections::HashMap; use std::time::Duration; use temp_env::{with_var, with_var_unset}; @@ -916,7 +943,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_uses_persisted_values_when_env_is_unset() { let config = server_config_with_scanner(&[ (SCANNER_SPEED, "slow"), @@ -944,7 +970,50 @@ mod tests { } #[test] - #[serial] + fn scanner_unset_budget_uses_safe_default_but_explicit_zero_is_unbounded() { + let config = server_config_with_scanner(&[]); + with_var_unset(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.cycle_budget.max_duration, Some(Duration::from_secs(1800))); + assert_eq!(resolved.cycle_max_duration_source, ScannerRuntimeConfigSource::Default); + }); + + let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_DURATION, "0")]); + with_var_unset(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.cycle_budget.max_duration, None); + assert_eq!(resolved.cycle_max_duration_source, ScannerRuntimeConfigSource::Config); + }); + } + + #[test] + fn cycle_budget_invalid_or_overflow_config_is_rejected() { + with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("invalid"), || { + let error = lookup_scanner_runtime_config(None).expect_err("invalid duration env must be rejected"); + assert!(error.to_string().contains(ENV_SCANNER_CYCLE_MAX_DURATION_SECS)); + assert!(error.to_string().contains("")); + assert!(!error.to_string().contains(": invalid (")); + }); + with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("18446744073709551616"), || { + assert!(lookup_scanner_runtime_config(None).is_err()); + }); + with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("18446744073709551615"), || { + assert!(lookup_scanner_runtime_config(None).is_err()); + }); + let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_DURATION, "not-a-duration")]); + assert!(lookup_scanner_runtime_config(Some(&config)).is_err()); + } + + #[test] + fn scanner_runtime_config_validation_rejects_overflow_persisted_duration() { + let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_DURATION, "18446744073709551615")]); + + let error = validate_scanner_runtime_config(&config) + .expect_err("persisted duration that exceeds the timer range must be rejected"); + assert!(error.to_string().contains(SCANNER_CYCLE_MAX_DURATION)); + } + + #[test] fn scanner_runtime_config_normalizes_persisted_default_speed() { let config = server_config_with_scanner(&[(SCANNER_SPEED, "default")]); @@ -960,7 +1029,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_prefers_env_over_persisted_config() { let config = server_config_with_scanner(&[(SCANNER_SPEED, "slowest"), (SCANNER_CYCLE, "600")]); @@ -977,7 +1045,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_prefers_heal_bitrot_cycle_over_scanner_compat_config() { let config = server_config_with_scanner_and_heal(&[(SCANNER_BITROT_CYCLE, "3600")], &[(HEAL_BITROT_CYCLE, "off")]); @@ -990,7 +1057,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_marks_scanner_bitrot_cycle_as_compat_source() { let config = server_config_with_scanner(&[(SCANNER_BITROT_CYCLE, "3600")]); @@ -1007,7 +1073,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_normalizes_persisted_default_bitrot_cycles() { let default_cycle = DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string(); for config in [ @@ -1032,7 +1097,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_validation_rejects_invalid_persisted_speed_with_env_override() { let config = server_config_with_scanner(&[(SCANNER_SPEED, "warp")]); @@ -1066,7 +1130,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_uses_derived_delay_for_excessive_env_override() { let config = server_config_with_scanner(&[(SCANNER_SPEED, "slow")]); @@ -1087,7 +1150,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_status_reports_value_sources() { let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_OBJECTS, "100"), (SCANNER_CACHE_SAVE_TIMEOUT, "5")]); @@ -1108,7 +1170,6 @@ mod tests { } #[test] - #[serial] fn applied_runtime_config_is_the_authoritative_scheduler_state() { let config = server_config_with_scanner(&[(SCANNER_CYCLE, "321")]); @@ -1125,7 +1186,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_status_reports_persisted_pacing_overrides() { let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]); @@ -1147,7 +1207,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_status_prefers_env_pacing_overrides() { let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]); @@ -1169,7 +1228,6 @@ mod tests { } #[test] - #[serial] fn scanner_runtime_config_status_preserves_subsecond_max_wait() { let config = server_config_with_scanner(&[(SCANNER_SPEED, "fast")]); diff --git a/crates/scanner/src/scanner.rs b/crates/scanner/src/scanner.rs index 4db558a54..059de1fb4 100644 --- a/crates/scanner/src/scanner.rs +++ b/crates/scanner/src/scanner.rs @@ -20,7 +20,7 @@ use std::sync::{Arc, LazyLock, RwLock}; use crate::data_usage_define::{ BACKGROUND_HEAL_INFO_PATH, DATA_USAGE_BLOOM_NAME_PATH, DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, - DataUsageCache, DataUsageCacheRevision, LEGACY_DATA_USAGE_OBJ_NAME_PATH, read_config_with_revision, + DataUsageCache, DataUsageCacheRevision, LEGACY_DATA_USAGE_OBJ_NAME_PATH, read_config_revision, read_config_with_revision, }; use crate::runtime_config::{ ScannerRuntimeConfig, ScannerRuntimeConfigSource, refresh_scanner_runtime_config_from_global, scanner_bitrot_cycle, @@ -52,11 +52,10 @@ use rustfs_config::{ }; use rustfs_config::{ENV_SCANNER_CYCLE, ENV_SCANNER_SPEED, ENV_SCANNER_START_DELAY_SECS}; use rustfs_data_usage::observed_data_usage_is_newer; +use rustfs_lock::NamespaceLockGuard; use serde::{Deserialize, Serialize}; use sha2::{Digest as _, Sha256}; -#[cfg(test)] -use tokio::sync::Notify; -use tokio::sync::mpsc; +use tokio::sync::{Notify, mpsc}; use tokio::time::{Duration, Instant}; use tokio_util::sync::CancellationToken; use tokio_util::task::AbortOnDropHandle; @@ -104,6 +103,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2; /// unavailable peer cannot drive a tight retry loop. const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5); const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60); +/// A transient backend outage remains self-healing after the short retry +/// budget is exhausted, but the probe is intentionally sparse until storage +/// recovers or an operator reset wakes the scanner. +const SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL: Duration = Duration::from_secs(5 * 60); +/// Permanent recovery states still get a sparse status probe so a reset that +/// races the wait registration cannot leave the scanner asleep forever. +const SCANNER_CYCLE_RECOVERY_BLOCKED_PROBE_INTERVAL: Duration = Duration::from_secs(5 * 60); const SCANNER_LEADER_LOCK_POLL_INTERVAL: Duration = Duration::from_secs(1); #[cfg(not(test))] const SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(30); @@ -125,6 +131,12 @@ type ScannerCycleStatePersistTestHook = (u64, Arc); static SCANNER_CYCLE_STATE_PERSIST_TEST_HOOK: LazyLock>> = LazyLock::new(|| StdMutex::new(None)); +static SCANNER_CYCLE_RECOVERY_WAKE: LazyLock = LazyLock::new(Notify::new); + +pub(super) fn notify_scanner_cycle_recovery_wake() { + SCANNER_CYCLE_RECOVERY_WAKE.notify_one(); +} + #[cfg(test)] struct ScannerCycleStatePersistTestHookGuard; @@ -576,19 +588,21 @@ pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc) { tokio::time::sleep(sleep_time).await; } + let mut transient_backoff = ScannerRetryBackoff::default(); + let mut recovery_retry_count = 0_u32; loop { if ctx_clone.is_cancelled() { break; } - if let Err(e) = run_data_scanner_with_maintenance_state( + let run_result = run_data_scanner_with_maintenance_state( ctx_clone.clone(), storeapi_clone.clone(), startup_features, startup_maintenance_generation, ) - .await - { + .await; + if let Err(e) = &run_result { error!( target: "rustfs::scanner", event = EVENT_SCANNER_CYCLE_STATE, @@ -599,11 +613,52 @@ pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc) { "Scanner runtime iteration failed" ); } + let recovery_status = scanner_cycle_recovery_status(); + if recovery_status.retryable { + recovery_retry_count = recovery_retry_count.saturating_add(1); + let _ = record_scanner_cycle_recovery_retry(recovery_retry_count); + } else { + recovery_retry_count = 0; + } + + let recovery_status = scanner_cycle_recovery_status(); + if recovery_status.state == "paused" { + transient_backoff.record_retryable_cycle(false); + tokio::select! { + _ = ctx_clone.cancelled() => break, + _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {}, + _ = tokio::time::sleep(SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL) => {}, + } + recovery_retry_count = 0; + continue; + } + if !recovery_status.retryable + && matches!(recovery_status.state.as_str(), "blocked" | "recovery-required" | "cleanup-pending") + { + transient_backoff.record_retryable_cycle(false); + tokio::select! { + _ = ctx_clone.cancelled() => break, + _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {}, + _ = tokio::time::sleep(SCANNER_CYCLE_RECOVERY_BLOCKED_PROBE_INTERVAL) => {}, + } + continue; + } + + let retry_delay = if recovery_status.retryable || run_result.is_err() { + transient_backoff.record_retryable_cycle(true); + transient_backoff + .retry_interval(scanner_cycle_interval()) + .unwrap_or(SCANNER_RETRY_BASE_INTERVAL) + } else { + transient_backoff.record_retryable_cycle(false); + randomized_cycle_delay() + }; // Backoff before retrying after lock contention or scanner-level failures. // Keep this cancellation-aware so shutdown is not delayed by backoff sleep. tokio::select! { _ = ctx_clone.cancelled() => break, - _ = tokio::time::sleep(randomized_cycle_delay()) => {} + _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {}, + _ = tokio::time::sleep(retry_delay) => {} } } }); @@ -983,20 +1038,116 @@ fn data_usage_persist_timeout() -> Duration { DataUsageCache::persistence_timeout() } +#[cfg(not(test))] +const SCANNER_CYCLE_EPOCH_FENCE_TIMEOUT: Duration = Duration::from_secs(30); +#[cfg(test)] +const SCANNER_CYCLE_EPOCH_FENCE_TIMEOUT: Duration = Duration::from_millis(50); + +async fn fence_scanner_epoch_after_cycle_timeout( + ctx: &CancellationToken, + storeapi: Arc, + cycle_info: &mut CurrentCycle, + cycle_revision: &mut DataUsageCacheRevision, + leader_epoch: &mut u64, + lock_lost: LockLost, +) -> bool +where + Store: ScannerObjectIO, + LockLost: Future, +{ + let fence_ctx = ctx.child_token(); + let claim = claim_scanner_leadership(&fence_ctx, storeapi, cycle_info, cycle_revision, leader_epoch); + tokio::pin!(claim); + tokio::pin!(lock_lost); + tokio::select! { + biased; + _ = &mut lock_lost => { + fence_ctx.cancel(); + false + } + result = tokio::time::timeout(SCANNER_CYCLE_EPOCH_FENCE_TIMEOUT, &mut claim) => { + result.unwrap_or(false) && !fence_ctx.is_cancelled() + } + } +} + +struct ScannerCycleDeadlineState<'a> { + cycle_info: &'a mut CurrentCycle, + cycle_revision: &'a mut DataUsageCacheRevision, + leader_epoch: &'a mut u64, + cycle_budget: &'a ScannerCycleBudget, +} + +fn cycle_timeout_requires_recovery(worker_stopped: bool, cycle_state_persisted: bool, generation_fenced: bool) -> bool { + !worker_stopped || !cycle_state_persisted || !generation_fenced +} + +async fn handle_scanner_cycle_deadline( + ctx: &CancellationToken, + storeapi: Arc, + state: ScannerCycleDeadlineState<'_>, + worker_stopped: bool, + guard: &mut NamespaceLockGuard, +) where + Store: ScannerObjectIO, +{ + let fenced = fence_scanner_epoch_after_cycle_timeout( + ctx, + storeapi, + state.cycle_info, + state.cycle_revision, + state.leader_epoch, + guard.lock_lost_notified(), + ) + .await; + let cycle_state_persisted = state.cycle_budget.cycle_state_persisted(); + let recovery_required = cycle_timeout_requires_recovery(worker_stopped, cycle_state_persisted, fenced); + warn!( + target: "rustfs::scanner", + event = EVENT_SCANNER_CYCLE_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + state = "cycle_timeout", + worker_stopped, + cycle_state_persisted, + generation_fenced = fenced, + recovery_required, + "Scanner cycle deadline expired; durable cursor/generation fencing completed when possible" + ); + global_metrics().record_scanner_cycle_timeout(recovery_required, state.cycle_budget.progress_age()); + // Stop renewing before releasing the lease. A new leader can then claim the + // higher persisted generation instead of inheriting the expired worker. + guard.release(); + global_metrics().set_cycle(None).await; +} + async fn mark_scan_cycle_idle(cycle_info: &mut CurrentCycle, cycle_metrics_guard: &mut ScannerCycleMetricsGuard) { cycle_info.current = 0; global_metrics().clear_current_scan_mode(); cycle_metrics_guard.finish(cycle_info.clone()).await; } -#[instrument(skip_all)] -#[hotpath::measure] +#[cfg(test)] async fn run_data_scanner_cycle( ctx: &CancellationToken, storeapi: &Arc, cycle_info: &mut CurrentCycle, cycle_revision: &mut DataUsageCacheRevision, leader_epoch: u64, +) -> ScannerCycleOutcome { + let cycle_budget = ScannerCycleBudget::new(ctx, scanner_cycle_budget_config()); + run_data_scanner_cycle_with_budget(ctx, storeapi, cycle_info, cycle_revision, leader_epoch, cycle_budget).await +} + +#[instrument(skip_all)] +#[hotpath::measure] +async fn run_data_scanner_cycle_with_budget( + ctx: &CancellationToken, + storeapi: &Arc, + cycle_info: &mut CurrentCycle, + cycle_revision: &mut DataUsageCacheRevision, + leader_epoch: u64, + cycle_budget: Arc, ) -> ScannerCycleOutcome { let _activity_guard = ScannerActivityGuard::new(); if let Err(err) = refresh_scanner_runtime_config_from_global() { @@ -1012,7 +1163,11 @@ async fn run_data_scanner_cycle( } let configured_cycle_interval = scanner_cycle_interval(); let configured_bitrot_cycle = scanner_bitrot_cycle(); - let cycle_budget_config = scanner_cycle_budget_config(); + let cycle_budget_config = ScannerCycleBudgetConfig { + max_duration: cycle_budget.max_duration(), + max_objects: cycle_budget.max_objects(), + max_directories: cycle_budget.max_directories(), + }; let usage_persist_timeout = data_usage_persist_timeout(); global_metrics().record_scanner_cycle_config( configured_cycle_interval, @@ -1083,7 +1238,6 @@ async fn run_data_scanner_cycle( let (sender, receiver) = mpsc::channel::(1); let done_cycle = Metrics::time(Metric::ScanCycle); - let cycle_budget = ScannerCycleBudget::new(ctx, cycle_budget_config); let scan_result = storeapi .clone() .nsscanner_with_status( @@ -1223,7 +1377,7 @@ async fn run_data_scanner_cycle( "Scanner cycle is recovering to a newer durable cache generation" ); emit_scan_cycle_partial_with_source(cycle_start.elapsed(), ScanCyclePartialReason::Unknown, None); - return if persist_required_scanner_cycle_floor( + let persisted = persist_required_scanner_cycle_floor( ctx, storeapi.clone(), cycle_info, @@ -1232,8 +1386,9 @@ async fn run_data_scanner_cycle( required_cycle, &mut cycle_metrics_guard, ) - .await - { + .await; + return if persisted { + cycle_budget.mark_cycle_state_persisted(); ScannerCycleOutcome::Partial } else { ScannerCycleOutcome::Failed @@ -1291,7 +1446,7 @@ async fn run_data_scanner_cycle( scan_cycle_partial_reason(budget_reason), scan_cycle_partial_source(budget_reason), ); - return if finalize_partial_scan_cycle( + let persisted = finalize_partial_scan_cycle( ctx, storeapi.clone(), cycle_info, @@ -1299,8 +1454,9 @@ async fn run_data_scanner_cycle( leader_epoch, &mut cycle_metrics_guard, ) - .await - { + .await; + return if persisted { + cycle_budget.mark_cycle_state_persisted(); ScannerCycleOutcome::Partial } else { ScannerCycleOutcome::Failed @@ -1375,7 +1531,7 @@ async fn run_data_scanner_cycle( ); } emit_scan_cycle_partial_with_source(cycle_start.elapsed(), ScanCyclePartialReason::Unknown, None); - return if finalize_partial_scan_cycle( + let persisted = finalize_partial_scan_cycle( ctx, storeapi.clone(), cycle_info, @@ -1383,8 +1539,9 @@ async fn run_data_scanner_cycle( leader_epoch, &mut cycle_metrics_guard, ) - .await - { + .await; + return if persisted { + cycle_budget.mark_cycle_state_persisted(); ScannerCycleOutcome::Partial } else { ScannerCycleOutcome::Failed @@ -1425,6 +1582,7 @@ async fn run_data_scanner_cycle( ) .await { + cycle_budget.mark_cycle_state_persisted(); emit_scan_cycle_superseded(cycle_start.elapsed()); return ScannerCycleOutcome::Superseded; } @@ -1457,6 +1615,7 @@ async fn run_data_scanner_cycle( emit_scan_cycle_complete(false, cycle_start.elapsed()); return ScannerCycleOutcome::Failed; } + cycle_budget.mark_cycle_state_persisted(); done_cycle(); emit_scan_cycle_complete(true, cycle_start.elapsed()); @@ -1521,7 +1680,7 @@ async fn run_data_scanner_with_maintenance_state( ) -> Result<(), ScannerError> { reset_scanner_cycle_schedule(); // Acquire leader lock (write lock) to ensure only one scanner runs - let guard = match storeapi.new_ns_lock(RUSTFS_META_BUCKET, "leader.lock").await { + let mut guard = match storeapi.new_ns_lock(RUSTFS_META_BUCKET, "leader.lock").await { Ok(ns_lock) => match ns_lock.get_write_lock_quiet(get_lock_acquire_timeout()).await { Ok(guard) => { record_scanner_leader_lock_state("acquired"); @@ -1606,40 +1765,22 @@ async fn run_data_scanner_with_maintenance_state( observe_scanner_activity(&storeapi, distributed, &mut scanner_activity_seen).await; } - let (buf, mut cycle_revision) = match read_config_with_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await { - Ok((buf, revision)) => (buf.unwrap_or_default(), revision), - Err(err) => { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "revision_load_failed", - error = %err, - "Scanner cycle state revision load failed" - ); - global_metrics().set_cycle(None).await; - return Ok(()); - } - }; - let (mut cycle_info, mut leader_epoch) = match decode_scanner_cycle_state_for_startup(&buf) { - Ok(state) => state, - Err(err) => { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "cycle_decode_failed", - error = %err, - "Scanner stopped because persisted cycle state is invalid" - ); - global_metrics().set_cycle(None).await; - return Ok(()); - } - }; + let (mut cycle_info, mut leader_epoch, mut cycle_revision) = + match load_scanner_cycle_state_for_startup(storeapi.clone()).await { + ScannerCycleStateStartup::Ready { + cycle, + leader_epoch, + revision, + } => (cycle, leader_epoch, revision), + ScannerCycleStateStartup::Blocked => { + global_metrics().set_cycle(None).await; + return Ok(()); + } + ScannerCycleStateStartup::Transient(err) => { + global_metrics().set_cycle(None).await; + return Err(err); + } + }; let usage_floor = match persisted_usage_floor(storeapi.clone()).await { Ok(floor) => floor, Err(err) => { @@ -1704,13 +1845,49 @@ async fn run_data_scanner_with_maintenance_state( return Ok(()); } let cycle_ctx = ctx.child_token(); - let initial_outcome = await_scanner_cycle_with_lock_fence( + let cycle_budget = ScannerCycleBudget::new_with_runtime_progress_tracking(&cycle_ctx, scanner_cycle_budget_config()); + let initial_outcome = match await_scanner_cycle_with_budget_fence( &cycle_ctx, - run_data_scanner_cycle(&cycle_ctx, &storeapi, &mut cycle_info, &mut cycle_revision, leader_epoch), + &cycle_budget, + run_data_scanner_cycle_with_budget( + &cycle_ctx, + &storeapi, + &mut cycle_info, + &mut cycle_revision, + leader_epoch, + cycle_budget.clone(), + ), guard.lock_lost_notified(), ) .await - .unwrap_or(ScannerCycleOutcome::Failed); + { + ScannerCycleWaitOutcome::Completed(outcome) => outcome, + ScannerCycleWaitOutcome::LockLost => { + record_scanner_leader_lock_lost("Scanner leader lock lost during the initial cycle").await; + global_metrics().set_cycle(None).await; + return Ok(()); + } + ScannerCycleWaitOutcome::Cancelled => { + global_metrics().set_cycle(None).await; + return Ok(()); + } + ScannerCycleWaitOutcome::Deadline { worker_stopped } => { + handle_scanner_cycle_deadline( + &ctx, + storeapi.clone(), + ScannerCycleDeadlineState { + cycle_info: &mut cycle_info, + cycle_revision: &mut cycle_revision, + leader_epoch: &mut leader_epoch, + cycle_budget: &cycle_budget, + }, + worker_stopped, + &mut guard, + ) + .await; + return Ok(()); + } + }; superseded_backoff.record_retryable_cycle(initial_outcome == ScannerCycleOutcome::Superseded); deferred_backoff.record_retryable_cycle(matches!(initial_outcome, ScannerCycleOutcome::Deferred(_))); dirty_usage_generation_seen = dirty_generation_before_cycle; @@ -1916,13 +2093,49 @@ async fn run_data_scanner_with_maintenance_state( } let dirty_generation_before_cycle = dirty_usage_generation(); let cycle_ctx = ctx.child_token(); - let outcome = await_scanner_cycle_with_lock_fence( + let cycle_budget = ScannerCycleBudget::new_with_runtime_progress_tracking(&cycle_ctx, scanner_cycle_budget_config()); + let outcome = match await_scanner_cycle_with_budget_fence( &cycle_ctx, - run_data_scanner_cycle(&cycle_ctx, &storeapi, &mut cycle_info, &mut cycle_revision, leader_epoch), + &cycle_budget, + run_data_scanner_cycle_with_budget( + &cycle_ctx, + &storeapi, + &mut cycle_info, + &mut cycle_revision, + leader_epoch, + cycle_budget.clone(), + ), guard.lock_lost_notified(), ) .await - .unwrap_or(ScannerCycleOutcome::Failed); + { + ScannerCycleWaitOutcome::Completed(outcome) => outcome, + ScannerCycleWaitOutcome::LockLost => { + record_scanner_leader_lock_lost("Scanner leader lock lost during a scanner cycle").await; + global_metrics().set_cycle(None).await; + return Ok(()); + } + ScannerCycleWaitOutcome::Cancelled => { + global_metrics().set_cycle(None).await; + return Ok(()); + } + ScannerCycleWaitOutcome::Deadline { worker_stopped } => { + handle_scanner_cycle_deadline( + &ctx, + storeapi.clone(), + ScannerCycleDeadlineState { + cycle_info: &mut cycle_info, + cycle_revision: &mut cycle_revision, + leader_epoch: &mut leader_epoch, + cycle_budget: &cycle_budget, + }, + worker_stopped, + &mut guard, + ) + .await; + return Ok(()); + } + }; superseded_backoff.record_retryable_cycle(outcome == ScannerCycleOutcome::Superseded); deferred_backoff.record_retryable_cycle(matches!(outcome, ScannerCycleOutcome::Deferred(_))); dirty_usage_generation_seen = dirty_generation_before_cycle; @@ -2219,7 +2432,12 @@ pub(crate) use activity::{ pub(crate) use activity::{ScannerCycleOutcome, scanner_cycle_outcome_with_pending_maintenance}; #[cfg(test)] pub(crate) use cycle_state::encode_scanner_cycle_fence_for_test; -pub(crate) use cycle_state::{current_scanner_leader_epoch, decode_persisted_scanner_cycle_fence}; +pub use cycle_state::{ + ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus, reset_scanner_cycle_recovery, scanner_cycle_recovery_status, +}; +pub(crate) use cycle_state::{ + current_scanner_leader_epoch, decode_persisted_scanner_cycle_fence, load_scanner_cycle_state_for_startup, +}; pub use heal_info::{BackgroundHealInfo, read_background_heal_info, save_background_heal_info}; pub use usage_store::store_data_usage_in_backend; diff --git a/crates/scanner/src/scanner/cycle_state.rs b/crates/scanner/src/scanner/cycle_state.rs index 6f3af4b81..6459ae6e5 100644 --- a/crates/scanner/src/scanner/cycle_state.rs +++ b/crates/scanner/src/scanner/cycle_state.rs @@ -13,6 +13,1067 @@ // limitations under the License. /// Scanner cycle-state codec, persisted usage floors, and cycle-state persistence. use super::*; +use crate::ScannerGetObjectReader; +use crate::data_usage_define::DATA_USAGE_BLOOM_RECOVERY_PATH; +use crate::storage_api::owner::ObjectIO as _; +use tokio::io::AsyncReadExt as _; + +const SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION: u16 = 1; +const MAX_SCANNER_CYCLE_STATE_BYTES: u64 = 1024 * 1024; +pub(super) const MAX_SCANNER_CYCLE_RECOVERY_RETRIES: u32 = 5; +const METRIC_SCANNER_CYCLE_RECOVERY_REQUIRED: &str = "rustfs_scanner_cycle_recovery_required"; +const METRIC_SCANNER_CYCLE_RECOVERY_RETRY_COUNT: &str = "rustfs_scanner_cycle_recovery_retry_count"; + +#[derive(Clone, Debug, Default, Serialize)] +pub struct ScannerCycleRecoveryStatus { + /// The immutable primary object whose revision is being guarded. + pub path: String, + /// The companion marker/quarantine object containing the recovery evidence. + pub quarantine_path: Option, + pub state: String, + pub classification: Option, + pub primary_revision: Option, + pub generation: Option, + pub leader_epoch: Option, + pub first_detected_at_unix_secs: Option, + pub last_attempt_at_unix_secs: Option, + pub retry_count: u64, + pub max_retries: u32, + /// Whether the scanner may retry this state automatically. + pub retryable: bool, + pub reason: Option, +} + +static SCANNER_CYCLE_RECOVERY_STATUS: LazyLock> = LazyLock::new(|| { + RwLock::new(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "healthy".to_string(), + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + ..Default::default() + }) +}); + +pub fn scanner_cycle_recovery_status() -> ScannerCycleRecoveryStatus { + SCANNER_CYCLE_RECOVERY_STATUS + .read() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .clone() +} + +fn set_scanner_cycle_recovery_status(status: ScannerCycleRecoveryStatus) { + let recovery_required = if matches!(status.state.as_str(), "blocked" | "paused" | "recovery-required" | "cleanup-pending") { + 1.0 + } else { + 0.0 + }; + metrics::gauge!(METRIC_SCANNER_CYCLE_RECOVERY_REQUIRED).set(recovery_required); + metrics::gauge!(METRIC_SCANNER_CYCLE_RECOVERY_RETRY_COUNT).set(status.retry_count as f64); + *SCANNER_CYCLE_RECOVERY_STATUS + .write() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = status; +} + +pub(super) fn record_scanner_cycle_recovery_retry(attempt: u32) -> bool { + let mut status = scanner_cycle_recovery_status(); + status.retry_count = u64::from(attempt); + status.last_attempt_at_unix_secs = Some(unix_now_secs()); + if attempt >= MAX_SCANNER_CYCLE_RECOVERY_RETRIES { + status.state = "paused".to_string(); + status.retryable = false; + status.reason = Some("scanner cycle recovery retry budget reached; sparse backend probes continue".to_string()); + set_scanner_cycle_recovery_status(status); + false + } else { + status.retryable = true; + set_scanner_cycle_recovery_status(status); + true + } +} + +fn unix_now_secs() -> u64 { + u64::try_from(Utc::now().timestamp()).unwrap_or(0) +} + +fn recovery_status(state: &str, reason: Option<&str>, retryable: bool) -> ScannerCycleRecoveryStatus { + ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: state.to_string(), + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable, + last_attempt_at_unix_secs: Some(unix_now_secs()), + reason: reason.map(str::to_string), + ..Default::default() + } +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ScannerCycleRecoveryMarker { + pub schema_version: u16, + pub primary_revision: String, + pub generation: u64, + pub leader_epoch: u64, + pub classification: String, + pub first_detected_at_unix_secs: u64, + pub last_attempt_at_unix_secs: u64, + pub retry_count: u64, + pub reason: String, + pub path: String, + pub quarantine_path: String, + /// `blocked` means the marker guards the primary revision; `cleanup-pending` + /// means an operator reset is in progress and must remain fenced across a + /// restart, even if the primary object is subsequently rewritten. + #[serde(default = "default_recovery_marker_state")] + pub state: String, +} + +fn default_recovery_marker_state() -> String { + "blocked".to_string() +} + +#[derive(Debug, Deserialize)] +struct ScannerCycleRecoveryMarkerCompat { + schema_version: Option, + primary_revision: Option, + classification: Option, + first_detected_at_unix_secs: Option, + last_attempt_at_unix_secs: Option, + retry_count: Option, + reason: Option, + path: Option, + quarantine_path: Option, + state: Option, +} + +#[derive(Debug)] +pub(crate) enum ScannerCycleStateStartup { + Ready { + cycle: CurrentCycle, + leader_epoch: u64, + revision: DataUsageCacheRevision, + }, + Blocked, + Transient(ScannerError), +} + +#[derive(Debug, thiserror::Error)] +enum CycleRecoveryMarkerReadError { + #[error("cycle recovery marker backend read failed: {0}")] + Backend(#[source] EcstoreError), + #[error("invalid cycle recovery marker: {0}")] + Invalid(&'static str), + #[error("cycle recovery marker revision changed while publishing")] + Conflict, +} + +#[derive(Debug, thiserror::Error)] +enum CycleStateBodyReadError { + #[error("scanner cycle state exceeds the bounded object size")] + TooLarge, + #[error("scanner cycle state body read failed: {0}")] + Backend(#[source] EcstoreError), +} + +fn recovery_status_from_marker(marker: &ScannerCycleRecoveryMarker, state: &str) -> ScannerCycleRecoveryStatus { + ScannerCycleRecoveryStatus { + path: marker.path.clone(), + quarantine_path: Some(marker.quarantine_path.clone()), + state: state.to_string(), + classification: Some(marker.classification.clone()), + primary_revision: Some(marker.primary_revision.clone()), + generation: Some(marker.generation), + leader_epoch: Some(marker.leader_epoch), + first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs), + last_attempt_at_unix_secs: Some(marker.last_attempt_at_unix_secs), + retry_count: marker.retry_count, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: false, + reason: Some(marker.reason.clone()), + } +} + +fn marker_matches_revision(marker: &ScannerCycleRecoveryMarker, revision: &DataUsageCacheRevision) -> bool { + matches!(revision, DataUsageCacheRevision::Etag(etag) if marker.primary_revision == *etag) +} + +fn validate_recovery_marker(marker: &ScannerCycleRecoveryMarker) -> Result<(), &'static str> { + if marker.schema_version != SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION { + return Err("cycle recovery marker schema is unsupported"); + } + if marker.primary_revision.is_empty() { + return Err("cycle recovery marker has no primary revision"); + } + if marker.path != *DATA_USAGE_BLOOM_NAME_PATH { + return Err("cycle recovery marker path does not match the scanner scope"); + } + if marker.quarantine_path != *DATA_USAGE_BLOOM_RECOVERY_PATH { + return Err("cycle recovery marker quarantine path does not match the scanner scope"); + } + if !matches!(marker.classification.as_str(), "corrupt" | "future_schema") { + return Err("cycle recovery marker classification is invalid"); + } + if !matches!(marker.state.as_str(), "blocked" | "cleanup-pending") { + return Err("cycle recovery marker state is invalid"); + } + Ok(()) +} + +/// Decode only the stable scope and revision fields needed by an authenticated +/// full-rescan reset. Startup keeps the strict decoder above so a newer marker +/// cannot be interpreted as a trusted cursor; reset deliberately rebuilds from +/// the persisted usage floor instead. +pub(super) fn decode_recovery_marker_for_reset( + data: &[u8], + marker_revision: &DataUsageCacheRevision, +) -> Result { + if !matches!(marker_revision, DataUsageCacheRevision::Etag(_)) { + return Err(ScannerError::Other("cycle recovery marker has no object revision".to_string())); + } + let compat = serde_json::from_slice::(data).ok(); + let _schema_version = compat.as_ref().and_then(|marker| marker.schema_version); + let primary_revision = compat + .as_ref() + .and_then(|marker| marker.primary_revision.clone()) + .filter(|revision| !revision.is_empty()) + .unwrap_or_default(); + let path = compat + .as_ref() + .and_then(|marker| marker.path.clone()) + .unwrap_or_else(|| DATA_USAGE_BLOOM_NAME_PATH.clone()); + let quarantine_path = compat + .as_ref() + .and_then(|marker| marker.quarantine_path.clone()) + .unwrap_or_else(|| DATA_USAGE_BLOOM_RECOVERY_PATH.clone()); + if path != *DATA_USAGE_BLOOM_NAME_PATH || quarantine_path != *DATA_USAGE_BLOOM_RECOVERY_PATH { + return Err(ScannerError::Other( + "cycle recovery marker path does not match the scanner scope".to_string(), + )); + } + let classification = match compat.as_ref().and_then(|marker| marker.classification.as_deref()) { + Some("corrupt") => "corrupt", + Some("future_schema") | None => "future_schema", + Some(_) => "future_schema", + }; + let state = match compat.as_ref().and_then(|marker| marker.state.as_deref()) { + Some("cleanup-pending") => "cleanup-pending", + _ => "blocked", + }; + let now = unix_now_secs(); + Ok(ScannerCycleRecoveryMarker { + schema_version: SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION, + primary_revision, + // Cursor and epoch values from an unknown marker are audit-only data; + // the reset path intentionally rebuilds both from the verified usage + // floor instead of carrying them across a version boundary. + generation: 0, + leader_epoch: 0, + classification: classification.to_string(), + first_detected_at_unix_secs: compat + .as_ref() + .and_then(|marker| marker.first_detected_at_unix_secs) + .unwrap_or(now), + last_attempt_at_unix_secs: compat + .as_ref() + .and_then(|marker| marker.last_attempt_at_unix_secs) + .unwrap_or(now), + retry_count: compat.as_ref().and_then(|marker| marker.retry_count).unwrap_or(0), + reason: compat + .as_ref() + .and_then(|marker| marker.reason.clone()) + .unwrap_or_else(|| "operator requested full scanner rescan".to_string()), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: state.to_string(), + }) +} + +async fn read_cycle_state_body(reader: &mut ScannerGetObjectReader) -> Result, CycleStateBodyReadError> { + let max_len = usize::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(usize::MAX); + let mut data = Vec::new(); + reader + .take(MAX_SCANNER_CYCLE_STATE_BYTES.saturating_add(1)) + .read_to_end(&mut data) + .await + .map_err(|err| CycleStateBodyReadError::Backend(EcstoreError::other(err)))?; + if data.len() > max_len { + return Err(CycleStateBodyReadError::TooLarge); + } + Ok(data) +} + +fn cycle_state_classification(buf: &[u8]) -> (&'static str, &'static str) { + if buf.len() >= 16 && &buf[8..12] == b"RSCY" && &buf[8..16] != SCANNER_CYCLE_STATE_MAGIC { + ("future_schema", "scanner cycle state schema is newer than this reader") + } else { + ("corrupt", "scanner cycle state failed validation") + } +} + +fn cycle_state_generation_and_epoch(buf: &[u8]) -> (u64, u64) { + let generation = buf + .get(..8) + .and_then(|bytes| bytes.try_into().ok()) + .map(u64::from_le_bytes) + .unwrap_or(0); + let leader_epoch = if buf.len() >= SCANNER_CYCLE_STATE_HEADER_LEN && &buf[8..16] == SCANNER_CYCLE_STATE_MAGIC { + u64::from_le_bytes(buf[16..24].try_into().unwrap_or([0; 8])) + } else { + 0 + }; + (generation, leader_epoch) +} + +async fn persist_cycle_recovery_marker( + storeapi: Arc, + primary_revision: &DataUsageCacheRevision, + generation: u64, + leader_epoch: u64, + classification: &'static str, + reason: &'static str, +) -> Result { + let now = unix_now_secs(); + let (existing, existing_revision) = match read_cycle_recovery_marker_bytes(storeapi.clone()).await { + Ok(result) => result, + Err(err) => return Err(err), + }; + let existing_marker = existing + .as_deref() + .and_then(|bytes| serde_json::from_slice::(bytes).ok()); + let primary_revision = match primary_revision { + DataUsageCacheRevision::Etag(etag) => etag.clone(), + DataUsageCacheRevision::Missing => { + return Err(CycleRecoveryMarkerReadError::Invalid("cycle state recovery requires a primary revision")); + } + }; + let marker = ScannerCycleRecoveryMarker { + schema_version: SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION, + primary_revision: primary_revision.clone(), + generation, + leader_epoch, + classification: classification.to_string(), + first_detected_at_unix_secs: existing_marker + .as_ref() + .filter(|marker| marker.primary_revision == primary_revision) + .map(|marker| marker.first_detected_at_unix_secs) + .unwrap_or(now), + last_attempt_at_unix_secs: now, + retry_count: existing_marker + .as_ref() + .filter(|marker| marker.primary_revision == primary_revision) + .map(|marker| marker.retry_count.saturating_add(1)) + .unwrap_or(0), + reason: reason.to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "blocked".to_string(), + }; + let bytes = serde_json::to_vec(&marker).map_err(|_| CycleRecoveryMarkerReadError::Invalid("marker serialization failed"))?; + let save_result = save_config_with_preconditions( + storeapi.clone(), + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + bytes, + existing_revision.preconditions(), + ) + .await; + match save_result { + Ok(_) => Ok(marker), + Err(EcstoreError::PreconditionFailed) => Err(CycleRecoveryMarkerReadError::Conflict), + Err(err) => Err(CycleRecoveryMarkerReadError::Backend(err)), + } +} + +async fn read_cycle_recovery_marker_bytes( + storeapi: Arc, +) -> Result<(Option>, DataUsageCacheRevision), CycleRecoveryMarkerReadError> { + let mut reader = match storeapi + .get_object_reader( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + None, + http::HeaderMap::new(), + &ScannerObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(reader) => reader, + Err( + EcstoreError::FileNotFound + | EcstoreError::VolumeNotFound + | EcstoreError::ObjectNotFound(_, _) + | EcstoreError::BucketNotFound(_) + | EcstoreError::ConfigNotFound, + ) => { + return Ok((None, DataUsageCacheRevision::Missing)); + } + Err(err) => return Err(CycleRecoveryMarkerReadError::Backend(err)), + }; + let revision = reader + .object_info + .etag + .as_ref() + .filter(|etag| !etag.is_empty()) + .cloned() + .map(DataUsageCacheRevision::Etag) + .ok_or(CycleRecoveryMarkerReadError::Invalid("marker has no revision"))?; + if reader.object_info.is_dir || reader.object_info.size < 0 || reader.object_info.size > 64 * 1024 { + return Err(CycleRecoveryMarkerReadError::Invalid("marker exceeds the bounded object size")); + } + let mut data = Vec::new(); + (&mut reader) + .take(64 * 1024 + 1) + .read_to_end(&mut data) + .await + .map_err(|err| CycleRecoveryMarkerReadError::Backend(EcstoreError::other(err)))?; + if data.len() > 64 * 1024 { + return Err(CycleRecoveryMarkerReadError::Invalid("marker exceeds the bounded object size")); + } + if data.is_empty() { + return Err(CycleRecoveryMarkerReadError::Invalid("marker is empty")); + } + Ok((Some(data), revision)) +} + +async fn read_cycle_recovery_marker_revision( + storeapi: Arc, +) -> Result { + let reader = match storeapi + .get_object_reader( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + None, + http::HeaderMap::new(), + &ScannerObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(reader) => reader, + Err( + EcstoreError::FileNotFound + | EcstoreError::VolumeNotFound + | EcstoreError::ObjectNotFound(_, _) + | EcstoreError::BucketNotFound(_) + | EcstoreError::ConfigNotFound, + ) => return Ok(DataUsageCacheRevision::Missing), + Err(err) => return Err(CycleRecoveryMarkerReadError::Backend(err)), + }; + if reader.object_info.is_dir || reader.object_info.size < 0 { + return Err(CycleRecoveryMarkerReadError::Invalid("marker is not a regular object")); + } + reader + .object_info + .etag + .as_ref() + .filter(|etag| !etag.is_empty()) + .cloned() + .map(DataUsageCacheRevision::Etag) + .ok_or(CycleRecoveryMarkerReadError::Invalid("marker has no revision")) +} + +async fn quarantine_invalid_cycle_state( + storeapi: Arc, + revision: &DataUsageCacheRevision, + buf: &[u8], +) -> ScannerCycleStateStartup { + let (classification, reason) = cycle_state_classification(buf); + let (generation, leader_epoch) = cycle_state_generation_and_epoch(buf); + quarantine_invalid_cycle_state_with_reason(storeapi, revision, generation, leader_epoch, classification, reason).await +} + +async fn quarantine_invalid_cycle_state_with_reason( + storeapi: Arc, + revision: &DataUsageCacheRevision, + generation: u64, + leader_epoch: u64, + classification: &'static str, + reason: &'static str, +) -> ScannerCycleStateStartup { + let now = unix_now_secs(); + let base_status = ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "recovery-required".to_string(), + classification: Some(classification.to_string()), + primary_revision: match revision { + DataUsageCacheRevision::Etag(etag) => Some(etag.clone()), + DataUsageCacheRevision::Missing => None, + }, + generation: Some(generation), + leader_epoch: Some(leader_epoch), + first_detected_at_unix_secs: Some(now), + last_attempt_at_unix_secs: Some(now), + retry_count: 0, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: true, + reason: Some(reason.to_string()), + }; + set_scanner_cycle_recovery_status(base_status); + match persist_cycle_recovery_marker(storeapi, revision, generation, leader_epoch, classification, reason).await { + Ok(marker) => set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, "blocked")), + Err(CycleRecoveryMarkerReadError::Backend(_)) => { + // Keep the poison object untouched and retry marker creation with the + // bounded startup backoff; recovery-required never becomes healthy. + return ScannerCycleStateStartup::Transient(ScannerError::Other( + "failed to persist scanner cycle recovery marker".to_string(), + )); + } + Err(CycleRecoveryMarkerReadError::Conflict) => { + set_scanner_cycle_recovery_status(recovery_status( + "transient", + Some("cycle recovery marker revision changed while publishing"), + true, + )); + return ScannerCycleStateStartup::Transient(ScannerError::Other( + "cycle recovery marker revision changed while publishing".to_string(), + )); + } + Err(CycleRecoveryMarkerReadError::Invalid(reason)) => { + set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false)); + return ScannerCycleStateStartup::Blocked; + } + } + ScannerCycleStateStartup::Blocked +} + +async fn mark_cycle_recovery_cleanup_pending( + storeapi: Arc, + mut marker: ScannerCycleRecoveryMarker, + marker_revision: &DataUsageCacheRevision, +) -> Result<(ScannerCycleRecoveryMarker, DataUsageCacheRevision), ScannerError> { + marker.state = "cleanup-pending".to_string(); + marker.last_attempt_at_unix_secs = unix_now_secs(); + let bytes = serde_json::to_vec(&marker) + .map_err(|err| ScannerError::Other(format!("failed to encode cycle recovery marker: {err}")))?; + let info = save_config_with_preconditions( + storeapi.clone(), + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + bytes, + marker_revision.preconditions(), + ) + .await + .map_err(|err| ScannerError::Other(format!("failed to mark cycle recovery cleanup pending: {err}")))?; + let revision = info + .etag + .filter(|etag| !etag.is_empty()) + .map(DataUsageCacheRevision::Etag) + .ok_or_else(|| ScannerError::Other("cycle recovery marker save returned no revision".to_string()))?; + Ok((marker, revision)) +} + +pub(crate) async fn load_scanner_cycle_state_for_startup(storeapi: Arc) -> ScannerCycleStateStartup { + let marker = match read_cycle_recovery_marker_bytes(storeapi.clone()).await { + Ok((None, _)) => None, + Ok((Some(data), marker_revision)) => match serde_json::from_slice::(&data) { + Ok(marker) => match validate_recovery_marker(&marker) { + Ok(()) => Some((marker, marker_revision)), + Err(reason) => { + set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false)); + return ScannerCycleStateStartup::Blocked; + } + }, + Err(_) => { + set_scanner_cycle_recovery_status(recovery_status( + "recovery-required", + Some("cycle recovery marker is invalid"), + false, + )); + return ScannerCycleStateStartup::Blocked; + } + }, + Err(CycleRecoveryMarkerReadError::Backend(err)) => { + let status = recovery_status("transient", Some("cycle recovery marker I/O is temporarily unavailable"), true); + set_scanner_cycle_recovery_status(status); + return ScannerCycleStateStartup::Transient(ScannerError::Other(format!( + "failed to read scanner cycle recovery marker: {err}" + ))); + } + Err(CycleRecoveryMarkerReadError::Invalid(reason)) => { + set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false)); + return ScannerCycleStateStartup::Blocked; + } + Err(CycleRecoveryMarkerReadError::Conflict) => { + set_scanner_cycle_recovery_status(recovery_status( + "transient", + Some("cycle recovery marker revision changed while being inspected"), + true, + )); + return ScannerCycleStateStartup::Transient(ScannerError::Other( + "cycle recovery marker revision changed while being inspected".to_string(), + )); + } + }; + + let mut reader = match storeapi + .get_object_reader( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + None, + http::HeaderMap::new(), + &ScannerObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(reader) => reader, + Err( + EcstoreError::FileNotFound + | EcstoreError::VolumeNotFound + | EcstoreError::ObjectNotFound(_, _) + | EcstoreError::BucketNotFound(_) + | EcstoreError::ConfigNotFound, + ) => { + if let Some((marker, _)) = marker { + let state = if marker.state == "cleanup-pending" { + "cleanup-pending" + } else { + "recovery-required" + }; + set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, state)); + return ScannerCycleStateStartup::Blocked; + } + set_scanner_cycle_recovery_status(recovery_status("healthy", None, false)); + return ScannerCycleStateStartup::Ready { + cycle: CurrentCycle::default(), + leader_epoch: 0, + revision: DataUsageCacheRevision::Missing, + }; + } + Err(err) => { + set_scanner_cycle_recovery_status(recovery_status("transient", Some("cycle state could not be inspected"), true)); + return ScannerCycleStateStartup::Transient(ScannerError::Other(format!( + "failed to inspect scanner cycle state: {err}" + ))); + } + }; + let revision = reader + .object_info + .etag + .as_ref() + .filter(|etag| !etag.is_empty()) + .cloned() + .map(DataUsageCacheRevision::Etag); + let Some(revision) = revision else { + set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some("cycle state has no revision"), false)); + return ScannerCycleStateStartup::Blocked; + }; + let max_size = i64::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(i64::MAX); + if reader.object_info.is_dir || reader.object_info.size < 0 || reader.object_info.size > max_size { + return quarantine_invalid_cycle_state_with_reason( + storeapi, + &revision, + 0, + 0, + "corrupt", + "scanner cycle state object is oversized or not a regular object", + ) + .await; + } + if let Some((marker, _)) = marker + .as_ref() + .filter(|(marker, _)| marker.state == "cleanup-pending" || marker_matches_revision(marker, &revision)) + { + let state = if marker.state == "cleanup-pending" { + "cleanup-pending" + } else { + "blocked" + }; + set_scanner_cycle_recovery_status(recovery_status_from_marker(marker, state)); + return ScannerCycleStateStartup::Blocked; + } + let data = match read_cycle_state_body(&mut reader).await { + Ok(data) => data, + Err(CycleStateBodyReadError::TooLarge) => { + return quarantine_invalid_cycle_state_with_reason( + storeapi, + &revision, + 0, + 0, + "corrupt", + "scanner cycle state exceeds the bounded object size", + ) + .await; + } + Err(CycleStateBodyReadError::Backend(err)) => { + set_scanner_cycle_recovery_status(recovery_status("transient", Some("cycle state read failed"), true)); + return ScannerCycleStateStartup::Transient(ScannerError::Other(format!( + "failed to read scanner cycle state: {err}" + ))); + } + }; + if data.is_empty() { + return quarantine_invalid_cycle_state_with_reason( + storeapi, + &revision, + 0, + 0, + "corrupt", + "scanner cycle state object is empty", + ) + .await; + } + match decode_scanner_cycle_state_for_startup(&data) { + Ok((cycle, leader_epoch)) => { + set_scanner_cycle_recovery_status(recovery_status("healthy", None, false)); + ScannerCycleStateStartup::Ready { + cycle, + leader_epoch, + revision, + } + } + Err(_) => quarantine_invalid_cycle_state(storeapi, &revision, &data).await, + } +} + +/// Reset a blocked cycle state after an operator has explicitly requested a full +/// usage rebuild. The primary object is changed first with its observed ETag; +/// the recovery marker is removed only when its own ETag still matches. +pub async fn reset_scanner_cycle_recovery(ctx: CancellationToken, storeapi: Arc) -> Result<(), ScannerError> { + let lock = storeapi + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .map_err(|err| ScannerError::Other(format!("failed to acquire scanner leader lock: {err}")))?; + let guard = lock + .get_write_lock_quiet(Duration::from_secs(5)) + .await + .map_err(|err| ScannerError::Other(format!("scanner leader lock is busy: {err}")))?; + + if guard.is_lock_lost() { + return Err(ScannerError::Other("scanner leader lock was lost before recovery reset".to_string())); + } + + let (marker_data, marker_revision, marker_body_invalid) = match read_cycle_recovery_marker_bytes(storeapi.clone()).await { + Ok((marker_data, marker_revision)) => (marker_data, marker_revision, false), + Err(CycleRecoveryMarkerReadError::Invalid(_)) => { + let marker_revision = read_cycle_recovery_marker_revision(storeapi.clone()) + .await + .map_err(|err| ScannerError::Other(format!("failed to read cycle recovery marker: {err}")))?; + (Some(Vec::new()), marker_revision, true) + } + Err(err) => return Err(ScannerError::Other(format!("failed to read cycle recovery marker: {err}"))), + }; + let marker_data = marker_data.ok_or_else(|| ScannerError::Other("scanner cycle recovery marker is absent".to_string()))?; + let (marker, force_full_rescan) = match serde_json::from_slice::(&marker_data) { + Ok(marker) if validate_recovery_marker(&marker).is_ok() => (marker, false), + _ => (decode_recovery_marker_for_reset(&marker_data, &marker_revision)?, true), + }; + let force_full_rescan = force_full_rescan || marker_body_invalid; + + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost while reading recovery state".to_string(), + )); + } + + let (mut primary_reader, primary_revision) = match storeapi + .get_object_reader( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + None, + http::HeaderMap::new(), + &ScannerObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await + { + Ok(reader) => { + let revision = reader + .object_info + .etag + .as_ref() + .filter(|etag| !etag.is_empty()) + .cloned() + .ok_or_else(|| ScannerError::Other("scanner cycle state has no revision".to_string()))?; + (Some(reader), DataUsageCacheRevision::Etag(revision)) + } + Err( + EcstoreError::FileNotFound + | EcstoreError::VolumeNotFound + | EcstoreError::ObjectNotFound(_, _) + | EcstoreError::BucketNotFound(_) + | EcstoreError::ConfigNotFound, + ) => (None, DataUsageCacheRevision::Missing), + Err(err) => return Err(ScannerError::Other(format!("failed to inspect scanner cycle state: {err}"))), + }; + let marker_cleanup_pending = marker.state == "cleanup-pending"; + let marker_matches_primary = marker_matches_revision(&marker, &primary_revision); + if (marker_cleanup_pending || !marker_matches_primary) + && let Some(mut reader) = primary_reader.take() + { + // A newer, independently fenced primary is authoritative. A + // full-rescan reset must not overwrite that progress; it only + // removes the stale recovery marker after validating and re-fencing + // the state. + let max_size = i64::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(i64::MAX); + if reader.object_info.is_dir || reader.object_info.size < 0 { + return Err(ScannerError::Other("scanner cycle state changed since recovery was recorded".to_string())); + } + let primary_is_oversized = reader.object_info.size > max_size; + let primary_state = if primary_is_oversized { + None + } else { + match read_cycle_state_body(&mut reader).await { + Ok(data) if data.is_empty() => None, + Ok(data) => decode_scanner_cycle_state_for_startup(&data).ok(), + Err(CycleStateBodyReadError::TooLarge) if force_full_rescan || marker_cleanup_pending => None, + Err(err) => { + return Err(ScannerError::Other(format!( + "scanner cycle state changed since recovery was recorded: {err}" + ))); + } + } + }; + if let Some((primary_cycle, primary_epoch)) = primary_state { + let (cleanup_marker, cleanup_marker_revision) = + mark_cycle_recovery_cleanup_pending(storeapi.clone(), marker.clone(), &marker_revision).await?; + set_scanner_cycle_recovery_status(recovery_status_from_marker(&cleanup_marker, "cleanup-pending")); + let usage_floor = persisted_usage_floor(storeapi.clone()).await?; + let fence_epoch = primary_epoch + .max(usage_floor.leader_epoch) + .checked_add(1) + .filter(|epoch| *epoch < u64::MAX) + .ok_or_else(|| ScannerError::Other("scanner leader epoch is exhausted".to_string()))?; + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost before preserving newer cycle state".to_string(), + )); + } + let preserved_data = encode_scanner_cycle_state(&primary_cycle, fence_epoch) + .map_err(|err| ScannerError::Other(format!("failed to encode preserved scanner cycle state: {err}")))?; + if u64::try_from(preserved_data.len()).unwrap_or(u64::MAX) > MAX_SCANNER_CYCLE_STATE_BYTES { + return Err(ScannerError::Other( + "preserved scanner cycle state exceeds the bounded object size".to_string(), + )); + } + let preserved_info = save_config_with_preconditions( + storeapi.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + preserved_data, + primary_revision.preconditions(), + ) + .await + .map_err(|err| ScannerError::Other(format!("failed to fence preserved scanner cycle state: {err}")))?; + let preserved_revision = preserved_info + .etag + .filter(|etag| !etag.is_empty()) + .map(DataUsageCacheRevision::Etag) + .ok_or_else(|| ScannerError::Other("preserved scanner cycle state has no revision".to_string()))?; + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost after fencing newer cycle state".to_string(), + )); + } + fence_scanner_usage_epoch(&ctx, storeapi.clone(), fence_epoch) + .await + .map_err(|err| ScannerError::Other(format!("failed to fence preserved scanner usage epoch: {err}")))?; + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost after fencing newer cycle state".to_string(), + )); + } + let current_revision = read_config_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .map_err(|err| ScannerError::Other(format!("failed to verify preserved scanner cycle state: {err}")))?; + if current_revision != preserved_revision { + return Err(ScannerError::Other( + "scanner cycle state changed before recovery marker cleanup".to_string(), + )); + } + storeapi + .delete_config_object( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + ScannerObjectOptions { + // This is one exact metadata object. Prefix-delete mode + // bypasses HTTP preconditions in the ECStore path. + delete_prefix: false, + http_preconditions: Some(cleanup_marker_revision.preconditions()), + ..Default::default() + }, + ) + .await + .map_err(|err| ScannerError::Other(format!("failed to clear stale cycle recovery marker: {err}")))?; + set_scanner_cycle_recovery_status(recovery_status("healthy", None, false)); + super::notify_scanner_cycle_recovery_wake(); + return Ok(()); + } else if !force_full_rescan && !marker_cleanup_pending { + // An invalid compatibility marker cannot fence a corrupt primary + // by revision, so rebuild it from the verified usage floor below. + // A strict marker keeps the existing fail-closed behavior for an + // unexpected stale-primary mutation. + return Err(ScannerError::Other("scanner cycle state changed since recovery was recorded".to_string())); + } + } + + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost before rebuilding cycle state".to_string(), + )); + } + + let floor = persisted_usage_floor(storeapi.clone()).await?; + // A full rescan must not trust a cursor recovered from a corrupt, future, + // or mixed-version marker. The durable usage floor is the only verified + // starting point; marker generation/epoch fields remain audit evidence. + let next = floor.next_cycle; + if next == u64::MAX { + return Err(ScannerError::Other("scanner cycle counter is exhausted".to_string())); + } + let leader_epoch = floor + .leader_epoch + .checked_add(1) + .filter(|epoch| *epoch < u64::MAX) + .ok_or_else(|| ScannerError::Other("scanner leader epoch is exhausted".to_string()))?; + let cycle = CurrentCycle { + next, + ..Default::default() + }; + let data = encode_scanner_cycle_state(&cycle, leader_epoch) + .map_err(|err| ScannerError::Other(format!("failed to encode rebuilt scanner cycle state: {err}")))?; + // Persist the cleanup-pending phase before rewriting the primary. If the + // process dies after the rewrite, startup still sees a durable fence and + // cannot mistake the partially completed reset for a healthy state. + let (marker, marker_revision) = if marker.state == "cleanup-pending" { + (marker, marker_revision) + } else { + mark_cycle_recovery_cleanup_pending(storeapi.clone(), marker, &marker_revision).await? + }; + let rebuilt_info = save_config_with_preconditions( + storeapi.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + data, + primary_revision.preconditions(), + ) + .await + .map_err(|err| ScannerError::Other(format!("failed to persist rebuilt scanner cycle state: {err}")))?; + let rebuilt_revision = rebuilt_info + .etag + .filter(|etag| !etag.is_empty()) + .ok_or_else(|| ScannerError::Other("rebuilt scanner cycle state has no revision".to_string()))?; + if guard.is_lock_lost() { + return Err(ScannerError::Other( + "scanner leader lock was lost after rebuilding cycle state".to_string(), + )); + } + if let Err(err) = fence_scanner_usage_epoch(&ctx, storeapi.clone(), leader_epoch).await { + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "cleanup-pending".to_string(), + classification: Some(marker.classification.clone()), + primary_revision: Some(rebuilt_revision.clone()), + generation: Some(next), + leader_epoch: Some(leader_epoch), + first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs), + last_attempt_at_unix_secs: Some(unix_now_secs()), + retry_count: marker.retry_count, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: false, + reason: Some("cycle state rebuilt but usage epoch fencing failed".to_string()), + }); + return Err(err); + } + + let current_revision = match read_config_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .map_err(|err| ScannerError::Other(format!("failed to verify rebuilt scanner cycle state: {err}")))? + { + DataUsageCacheRevision::Etag(etag) => etag, + DataUsageCacheRevision::Missing => { + return Err(ScannerError::Other("rebuilt scanner cycle state lost its revision".to_string())); + } + }; + if current_revision != rebuilt_revision { + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "cleanup-pending".to_string(), + classification: Some(marker.classification.clone()), + primary_revision: Some(current_revision), + generation: Some(next), + leader_epoch: Some(leader_epoch), + first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs), + last_attempt_at_unix_secs: Some(unix_now_secs()), + retry_count: marker.retry_count, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: false, + reason: Some("rebuilt scanner cycle state changed before marker cleanup".to_string()), + }); + return Err(ScannerError::Other( + "rebuilt scanner cycle state changed before recovery marker cleanup".to_string(), + )); + } + + if guard.is_lock_lost() { + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "cleanup-pending".to_string(), + classification: Some(marker.classification.clone()), + primary_revision: Some(rebuilt_revision.clone()), + generation: Some(next), + leader_epoch: Some(leader_epoch), + retry_count: marker.retry_count, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: false, + reason: Some("cycle state rebuilt but recovery marker was not cleared".to_string()), + ..Default::default() + }); + return Err(ScannerError::Other( + "scanner leader lock was lost before clearing recovery marker".to_string(), + )); + } + + if let Err(err) = storeapi + .delete_config_object( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + ScannerObjectOptions { + // This is one exact metadata object. Prefix-delete mode + // bypasses HTTP preconditions in the ECStore path. + delete_prefix: false, + http_preconditions: Some(marker_revision.preconditions()), + ..Default::default() + }, + ) + .await + { + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "cleanup-pending".to_string(), + classification: Some(marker.classification.clone()), + primary_revision: Some(rebuilt_revision.clone()), + generation: Some(next), + leader_epoch: Some(leader_epoch), + retry_count: marker.retry_count, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: false, + reason: Some("cycle state rebuilt but recovery marker cleanup failed".to_string()), + ..Default::default() + }); + return Err(ScannerError::Other(format!("failed to clear cycle recovery marker: {err}"))); + } + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "healthy".to_string(), + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + ..Default::default() + }); + super::notify_scanner_cycle_recovery_wake(); + Ok(()) +} #[derive(Debug, thiserror::Error)] pub(super) enum ScannerCycleStateError { @@ -86,7 +1147,11 @@ pub(super) fn decode_scanner_cycle_state(buf: &[u8]) -> Result<(CurrentCycle, u6 (0, &buf[8..]) }; - let cycle_info = rmp_serde::from_slice::(payload)?; + let mut deserializer = rmp_serde::Deserializer::new(std::io::Cursor::new(payload)); + let cycle_info = CurrentCycle::deserialize(&mut deserializer)?; + if deserializer.position() != u64::try_from(payload.len()).unwrap_or(u64::MAX) { + return Err(ScannerCycleStateError::InvalidData("scanner cycle state has trailing bytes")); + } if cycle_info.next != persisted_next { return Err(ScannerCycleStateError::InvalidData("scanner cycle counter disagrees with encoded state")); } @@ -146,7 +1211,7 @@ pub(super) fn advance_scanner_cycle(cycle_info: &mut CurrentCycle) -> Result<(), pub(super) async fn persisted_usage_floor(storeapi: Arc) -> Result { let mut floor = PersistedUsageFloor::default(); - let update_floor = |floor: &mut PersistedUsageFloor, usage: DataUsageInfo, path: &str| -> Result<(), ScannerError> { + let update_floor = |floor: &mut PersistedUsageFloor, usage: &DataUsageInfo, path: &str| -> Result<(), ScannerError> { floor.leader_epoch = floor.leader_epoch.max(usage.scanner_epoch.unwrap_or_default()); if let Some(completed_cycle) = usage.scanner_cycle { let next_cycle = completed_cycle @@ -159,25 +1224,45 @@ pub(super) async fn persisted_usage_floor(storeapi: Arc) - }; for primary_path in [DATA_USAGE_OBJ_NAME_PATH.as_str(), LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str()] { let backup_path = format!("{primary_path}.bkp"); - let mut pair_found = false; - for path in [primary_path, backup_path.as_str()] { - let data = match read_config(storeapi.clone(), path).await { - Ok(data) => { - pair_found = true; - data + let primary_epoch = match read_config(storeapi.clone(), primary_path).await { + Ok(data) => { + let usage = serde_json::from_slice::(&data).map_err(|err| { + ScannerError::Other(format!("failed to decode scanner usage floor from {primary_path}: {err}")) + })?; + let epoch = usage.scanner_epoch.unwrap_or_default(); + update_floor(&mut floor, &usage, primary_path)?; + Some(epoch) + } + Err(EcstoreError::ConfigNotFound) => None, + Err(err) => { + return Err(ScannerError::Other(format!( + "failed to read scanner usage epoch floor from {primary_path}: {err}" + ))); + } + }; + let mut any_found = primary_epoch.is_some(); + match read_config(storeapi.clone(), &backup_path).await { + Ok(data) => { + any_found = true; + let usage = serde_json::from_slice::(&data).map_err(|err| { + ScannerError::Other(format!("failed to decode scanner usage floor from {backup_path}: {err}")) + })?; + let backup_epoch = usage.scanner_epoch.unwrap_or_default(); + // A backup write from an older leader may complete after the + // primary epoch has been fenced. It must not advance the startup + // floor unless its epoch is at least as new as the primary. + if primary_epoch.is_none_or(|epoch| backup_epoch >= epoch) { + update_floor(&mut floor, &usage, &backup_path)?; } - Err(EcstoreError::ConfigNotFound) => continue, - Err(err) => { - return Err(ScannerError::Other(format!( - "failed to read scanner usage epoch floor from {path}: {err}" - ))); - } - }; - let usage = serde_json::from_slice::(&data) - .map_err(|err| ScannerError::Other(format!("failed to decode scanner usage floor from {path}: {err}")))?; - update_floor(&mut floor, usage, path)?; + } + Err(EcstoreError::ConfigNotFound) => {} + Err(err) => { + return Err(ScannerError::Other(format!( + "failed to read scanner usage epoch floor from {backup_path}: {err}" + ))); + } } - if pair_found { + if any_found { break; } } @@ -496,3 +1581,63 @@ where output = &mut cycle => Some(output), } } + +#[derive(Debug, PartialEq, Eq)] +pub(super) enum ScannerCycleWaitOutcome { + Completed(T), + LockLost, + Cancelled, + Deadline { worker_stopped: bool }, +} + +pub(super) async fn await_scanner_cycle_with_budget_fence( + cycle_ctx: &CancellationToken, + budget: &ScannerCycleBudget, + cycle: Cycle, + lock_lost: LockLost, +) -> ScannerCycleWaitOutcome +where + Cycle: Future, + LockLost: Future, +{ + tokio::pin!(cycle); + tokio::pin!(lock_lost); + let deadline = async { + if let Some(deadline) = budget.deadline() { + tokio::time::sleep_until(deadline).await; + } else { + std::future::pending::<()>().await; + } + }; + tokio::pin!(deadline); + tokio::select! { + biased; + _ = &mut lock_lost => { + cycle_ctx.cancel(); + let _ = tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, &mut cycle).await; + ScannerCycleWaitOutcome::LockLost + } + _ = &mut deadline => { + budget.cancel_for_runtime(); + // Let the budget cancellation reach the scanner first so it can + // persist a partial cursor. Only an uncooperative worker gets the + // parent cancellation, and it is dropped after the bounded window; + // the caller fences its epoch next. + let worker_stopped = if tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, &mut cycle) + .await + .is_ok() + { + true + } else { + cycle_ctx.cancel(); + false + }; + ScannerCycleWaitOutcome::Deadline { worker_stopped } + } + _ = cycle_ctx.cancelled() => { + let _ = tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, &mut cycle).await; + ScannerCycleWaitOutcome::Cancelled + } + output = &mut cycle => ScannerCycleWaitOutcome::Completed(output), + } +} diff --git a/crates/scanner/src/scanner/leadership.rs b/crates/scanner/src/scanner/leadership.rs index 0ac948549..ab22f56d9 100644 --- a/crates/scanner/src/scanner/leadership.rs +++ b/crates/scanner/src/scanner/leadership.rs @@ -196,7 +196,7 @@ pub(super) async fn claim_scanner_leadership( if ctx.is_cancelled() { return false; } - let Some(claimed_epoch) = persisted_epoch.checked_add(1) else { + let Some(claimed_epoch) = persisted_epoch.checked_add(1).filter(|epoch| *epoch < u64::MAX) else { error!( target: "rustfs::scanner", event = EVENT_SCANNER_PERSIST_STATE, diff --git a/crates/scanner/src/scanner/tests.rs b/crates/scanner/src/scanner/tests.rs index fd321bfc3..d73d63cde 100644 --- a/crates/scanner/src/scanner/tests.rs +++ b/crates/scanner/src/scanner/tests.rs @@ -15,17 +15,18 @@ use super::*; use crate::EcstoreResult; use crate::{ - Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints, ScannerGetObjectReader as GetObjectReader, - ScannerObjectInfo as ObjectInfo, ScannerObjectOptions as ObjectOptions, ScannerPutObjReader as PutObjReader, - init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, + DATA_USAGE_BLOOM_RECOVERY_PATH, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints, + ScannerGetObjectReader as GetObjectReader, ScannerObjectInfo as ObjectInfo, ScannerObjectOptions as ObjectOptions, + ScannerPutObjReader as PutObjReader, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, + init_local_disks_with_instance_ctx, }; -use serial_test::serial; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::io::Cursor; use std::task::Poll; use temp_env::{with_var, with_var_unset}; use tokio::io::AsyncReadExt; use tokio::sync::Mutex; +use tokio::time::{Duration, advance}; const TEST_DEFAULT_SCANNER_CYCLE_SECS: u64 = 24 * 60 * 60; @@ -118,6 +119,187 @@ async fn scanner_cycle_lock_fence_bounds_uncooperative_shutdown() { assert!(cycle_ctx.is_cancelled()); } +#[tokio::test(start_paused = true)] +async fn cycle_budget_fences_late_writer_after_timeout() { + let cycle_ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &cycle_ctx, + ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(5)), + ..Default::default() + }, + ); + let outcome = { + let cycle = std::future::pending::<()>(); + let lock_lost = std::future::pending::<()>(); + let waiter = await_scanner_cycle_with_budget_fence(&cycle_ctx, &budget, cycle, lock_lost); + tokio::pin!(waiter); + tokio::task::yield_now().await; + advance(Duration::from_secs(5)).await; + tokio::task::yield_now().await; + advance(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT).await; + waiter.await + }; + assert_eq!(outcome, ScannerCycleWaitOutcome::Deadline { worker_stopped: false }); + assert!(cycle_ctx.is_cancelled()); + assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Runtime)); + + // A newer leadership epoch is the durable fence that rejects a late + // writer after the timed-out future has been dropped. + let store = Arc::new(MemoryConfigStore::default()); + let mut revision = DataUsageCacheRevision::Missing; + let mut cycle = CurrentCycle { + current: 0, + next: 12, + ..Default::default() + }; + let persist_ctx = CancellationToken::new(); + assert!(persist_scanner_cycle_state(&persist_ctx, store.clone(), &mut cycle, &mut revision, 1).await); + let newer = encode_scanner_cycle_state(&cycle, 2).expect("new epoch fence should encode"); + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.interleaving_puts.lock().await.insert(key, (2, newer)); + let mut late_cycle = CurrentCycle { next: 13, ..cycle }; + assert!(!persist_scanner_cycle_state(&persist_ctx, store, &mut late_cycle, &mut revision, 1).await); +} + +#[tokio::test(start_paused = true)] +async fn cycle_budget_parent_cancellation_is_not_reported_as_timeout() { + let cycle_ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &cycle_ctx, + ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(5)), + ..Default::default() + }, + ); + let waiter = await_scanner_cycle_with_budget_fence(&cycle_ctx, &budget, std::future::pending::<()>(), std::future::pending()); + tokio::pin!(waiter); + tokio::task::yield_now().await; + cycle_ctx.cancel(); + tokio::task::yield_now().await; + advance(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT).await; + assert_eq!(waiter.await, ScannerCycleWaitOutcome::Cancelled); +} + +#[tokio::test(start_paused = true)] +async fn cycle_budget_deadline_wins_same_tick_as_parent_cancellation() { + let cycle_ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &cycle_ctx, + ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(5)), + ..Default::default() + }, + ); + let waiter = await_scanner_cycle_with_budget_fence(&cycle_ctx, &budget, std::future::pending::<()>(), std::future::pending()); + tokio::pin!(waiter); + tokio::task::yield_now().await; + advance(Duration::from_secs(5)).await; + cycle_ctx.cancel(); + tokio::task::yield_now().await; + advance(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT).await; + + assert_eq!(waiter.await, ScannerCycleWaitOutcome::Deadline { worker_stopped: false }); + assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Runtime)); +} + +#[tokio::test] +async fn cycle_budget_persist_cursor_failure_is_recovery_required() { + let store = Arc::new(MemoryConfigStore::default()); + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.fail_put_number.lock().await.insert(key, 1); + + let ctx = CancellationToken::new(); + let mut revision = DataUsageCacheRevision::Missing; + let mut cycle = CurrentCycle { + current: 12, + next: 12, + ..Default::default() + }; + let mut leader_epoch = 1; + let fenced = fence_scanner_epoch_after_cycle_timeout( + &ctx, + store, + &mut cycle, + &mut revision, + &mut leader_epoch, + std::future::pending(), + ) + .await; + assert!(!fenced, "a failed cursor/generation write must require recovery"); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + assert!(cycle_timeout_requires_recovery(true, budget.cycle_state_persisted(), fenced)); + + let metrics = Metrics::new(); + metrics.record_scanner_cycle_timeout(!fenced, Duration::from_secs(17)); + let report = metrics.report().await; + assert_eq!(report.cycle_timeout_total, 1); + assert_eq!(report.cycle_recovery_required_total, 1); + assert_eq!(report.cycle_last_progress_age, 17); + assert!(report.leader_lease_without_progress); +} + +#[tokio::test] +async fn cycle_budget_deadline_handler_fences_and_releases_guard() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .expect("scanner leader lock should be created"); + let mut guard = lock + .get_write_lock(Duration::from_secs(1)) + .await + .expect("scanner leader lock should be acquired"); + + let ctx = CancellationToken::new(); + let mut cycle_info = CurrentCycle { + current: 12, + next: 12, + ..Default::default() + }; + let mut cycle_revision = DataUsageCacheRevision::Missing; + let mut leader_epoch = 1; + let budget = ScannerCycleBudget::new( + &ctx, + ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(60)), + ..Default::default() + }, + ); + budget.mark_cycle_state_persisted(); + + handle_scanner_cycle_deadline( + &ctx, + store.clone(), + ScannerCycleDeadlineState { + cycle_info: &mut cycle_info, + cycle_revision: &mut cycle_revision, + leader_epoch: &mut leader_epoch, + cycle_budget: &budget, + }, + true, + &mut guard, + ) + .await; + + assert!(guard.is_released()); + let persisted = read_config(store, &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("deadline handler should persist a fenced cursor"); + let (_, persisted_epoch) = decode_scanner_cycle_state(&persisted).expect("fenced cursor should decode"); + assert_eq!(persisted_epoch, 2); + global_metrics().set_cycle(None).await; +} + +#[tokio::test] +async fn scanner_cycle_recovery_wake_survives_wait_registration_race() { + notify_scanner_cycle_recovery_wake(); + + tokio::time::timeout(Duration::from_secs(1), SCANNER_CYCLE_RECOVERY_WAKE.notified()) + .await + .expect("recovery wake should retain a permit until the waiter registers"); +} + struct ScannerDefaultSpeedGuard; impl ScannerDefaultSpeedGuard { @@ -152,6 +334,7 @@ impl Drop for ScannerDefaultCycleGuard { struct MemoryConfigStore { objects: Mutex>>, revisions: Mutex>, + non_regular_objects: Mutex>, fail_put_number: Mutex>, object_not_found_put_number: Mutex>, error_after_commit_put_number: Mutex>, @@ -192,12 +375,16 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { .get(&key) .cloned() .ok_or(EcstoreError::FileNotFound)?; - let revision = *self.revisions.lock().await.entry(key).or_insert(1); + let data_len = i64::try_from(data.len()).expect("memory test object length should fit in i64"); + let revision = *self.revisions.lock().await.entry(key.clone()).or_insert(1); + let is_dir = self.non_regular_objects.lock().await.contains(&key); Ok(GetObjectReader { stream: Box::new(Cursor::new(data)), object_info: ObjectInfo { etag: Some(format!("memory-{revision}")), + size: data_len, + is_dir, ..Default::default() }, buffered_body: None, @@ -362,7 +549,6 @@ fn test_initial_scanner_delay_uses_configured_start_delay() { } #[test] -#[serial] fn test_initial_scanner_delay_uses_cycle_without_explicit_start_delay() { with_var(ENV_SCANNER_CYCLE, Some("120"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -409,21 +595,12 @@ fn test_initial_scanner_delay_keeps_delay_for_replication_without_buckets() { } #[test] -#[serial] fn test_scanner_cycle_max_duration_uses_env() { with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("42"), || { assert_eq!(scanner_cycle_max_duration(), Some(Duration::from_secs(42))); }); } -#[test] -#[serial] -fn test_scanner_cycle_max_duration_default_is_disabled() { - with_var_unset(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, || { - assert_eq!(scanner_cycle_max_duration(), None); - }); -} - #[tokio::test] async fn test_scanner_cycle_budget_cancels_after_duration() { let parent = CancellationToken::new(); @@ -461,7 +638,6 @@ async fn test_scanner_cycle_budget_drop_cancels_child_without_elapsed() { } #[test] -#[serial] fn test_scanner_cycle_budget_config_uses_work_budget_env() { with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("100"), || { with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("25"), || { @@ -473,7 +649,6 @@ fn test_scanner_cycle_budget_config_uses_work_budget_env() { } #[test] -#[serial] fn test_scanner_cycle_budget_config_disables_zero_work_budgets() { with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("0"), || { with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("0"), || { @@ -516,7 +691,6 @@ fn test_scan_cycle_partial_source_maps_budget_reason() { } #[tokio::test] -#[serial] async fn test_mark_scan_cycle_idle_clears_published_cycle_state() { let mut cycle_info = CurrentCycle { current: 12, @@ -545,7 +719,6 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() { } #[tokio::test] -#[serial] async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() { let cycle_started = Utc::now() - chrono::Duration::seconds(5); let mut cycle_info = CurrentCycle { @@ -572,7 +745,6 @@ async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() { } #[tokio::test] -#[serial] async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finalization() { let mut cycle_info = CurrentCycle { current: 12, @@ -597,7 +769,6 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz } #[tokio::test] -#[serial] async fn scanner_cycle_metrics_guard_drop_clears_activity() { let guard = ScannerCycleMetricsGuard::new(CurrentCycle { current: 12, @@ -615,7 +786,6 @@ async fn scanner_cycle_metrics_guard_drop_clears_activity() { } #[tokio::test] -#[serial] async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() { let (_temp_dir, store) = setup_scanner_cycle_store().await; let ctx = CancellationToken::new(); @@ -666,7 +836,6 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() { } #[tokio::test] -#[serial] async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -702,7 +871,6 @@ async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() { } #[tokio::test] -#[serial] async fn scanner_cycle_recovers_to_newer_durable_cache_floor() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -742,7 +910,6 @@ async fn scanner_cycle_recovers_to_newer_durable_cache_floor() { } #[tokio::test] -#[serial] async fn scanner_cycle_rejects_invalid_cache_floor() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -811,6 +978,10 @@ fn scanner_cycle_state_decodes_legacy_and_fenced_formats() { let (fenced_cycle, fenced_epoch) = decode_scanner_cycle_state(&fenced).expect("fenced cycle state should decode"); assert_eq!(fenced_cycle.next, 13); assert_eq!(fenced_epoch, 7); + + let mut trailing = fenced; + trailing.push(0); + assert!(decode_scanner_cycle_state(&trailing).is_err()); } #[test] @@ -837,6 +1008,840 @@ fn scanner_startup_fails_closed_on_nonempty_corrupt_cycle_state() { assert!(encode_scanner_cycle_state(&exhausted, 7).is_err()); } +#[tokio::test] +async fn corrupt_cycle_state_is_quarantined_once() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.objects.lock().await.insert(state_key.clone(), vec![1]); + store.revisions.lock().await.insert(state_key.clone(), 7); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Blocked + )); + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + let marker_data = store + .objects + .lock() + .await + .get(&marker_key) + .cloned() + .expect("corrupt state must leave a durable recovery marker"); + let marker: ScannerCycleRecoveryMarker = serde_json::from_slice(&marker_data).expect("marker should be valid JSON"); + assert_eq!(marker.primary_revision, "memory-7"); + assert_eq!(marker.path, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + assert_eq!(marker.quarantine_path, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + assert_eq!(marker.classification, "corrupt"); + + // A second startup sees the matching marker before consuming the poison body. + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Blocked + )); + + // Replacing the primary object advances its revision; the stale marker must + // not quarantine the newer, valid state. + let cycle = CurrentCycle { + next: 9, + ..Default::default() + }; + let encoded = encode_scanner_cycle_state(&cycle, 3).expect("valid state should encode"); + store.objects.lock().await.insert(state_key.clone(), encoded); + store.revisions.lock().await.insert(state_key, 8); + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Ready { + cycle: CurrentCycle { next: 9, .. }, + leader_epoch: 3, + .. + } + )); +} + +#[tokio::test] +async fn empty_cycle_state_object_is_quarantined_as_corrupt() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.objects.lock().await.insert(state_key.clone(), Vec::new()); + store.revisions.lock().await.insert(state_key, 6); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Blocked + )); + assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("corrupt")); + assert!( + scanner_cycle_recovery_status() + .reason + .as_deref() + .is_some_and(|reason| reason.contains("empty")) + ); +} + +#[tokio::test] +async fn future_cycle_state_schema_is_recovery_required() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + let mut future = 17_u64.to_le_bytes().to_vec(); + future.extend_from_slice(b"RSCYC999"); + future.extend_from_slice(&4_u64.to_le_bytes()); + future.extend_from_slice(&[0x90]); + store.objects.lock().await.insert(state_key.clone(), future); + store.revisions.lock().await.insert(state_key, 13); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Blocked + )); + assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("future_schema")); +} + +#[tokio::test] +async fn concurrent_leaders_cannot_quarantine_newer_cycle_state() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.objects.lock().await.insert(state_key.clone(), vec![1]); + store.revisions.lock().await.insert(state_key, 4); + + let (first, second) = tokio::join!( + load_scanner_cycle_state_for_startup(store.clone()), + load_scanner_cycle_state_for_startup(store.clone()), + ); + assert!(matches!(first, ScannerCycleStateStartup::Blocked)); + assert!(matches!(second, ScannerCycleStateStartup::Blocked)); + + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + let marker_data = store + .objects + .lock() + .await + .get(&marker_key) + .cloned() + .expect("one contender must publish the recovery marker"); + let marker: ScannerCycleRecoveryMarker = serde_json::from_slice(&marker_data).expect("marker should decode"); + assert_eq!(marker.primary_revision, "memory-4"); +} + +#[tokio::test] +async fn cleanup_pending_marker_blocks_a_rewritten_primary_after_restart() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + let encoded = encode_scanner_cycle_state( + &CurrentCycle { + next: 12, + ..Default::default() + }, + 8, + ) + .expect("valid state should encode"); + store.objects.lock().await.insert(state_key.clone(), encoded); + store.revisions.lock().await.insert(state_key, 22); + let marker = ScannerCycleRecoveryMarker { + schema_version: 1, + primary_revision: "memory-21".to_string(), + generation: 11, + leader_epoch: 7, + classification: "corrupt".to_string(), + first_detected_at_unix_secs: 1, + last_attempt_at_unix_secs: 2, + retry_count: 1, + reason: "reset in progress".to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "cleanup-pending".to_string(), + }; + store + .objects + .lock() + .await + .insert(marker_key.clone(), serde_json::to_vec(&marker).expect("marker should encode")); + store.revisions.lock().await.insert(marker_key, 3); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Blocked + )); + assert_eq!(scanner_cycle_recovery_status().state, "cleanup-pending"); +} + +#[test] +fn full_rescan_reset_accepts_unknown_marker_fields_without_trusting_cursor() { + let marker = br#"{ + "schema_version": 99, + "primary_revision": "memory-7", + "generation": 9000, + "leader_epoch": 9000, + "classification": "new-future-classification", + "first_detected_at_unix_secs": 1, + "last_attempt_at_unix_secs": 2, + "retry_count": 9, + "reason": "future marker", + "path": "buckets/.bloomcycle.bin", + "quarantine_path": "buckets/.bloomcycle.bin.recovery-required.json", + "future_field": {"cursor": "untrusted"} + }"#; + let decoded = + super::cycle_state::decode_recovery_marker_for_reset(marker, &DataUsageCacheRevision::Etag("memory-3".to_string())) + .expect("full-rescan compatibility decoder should accept additive fields"); + assert_eq!(decoded.primary_revision, "memory-7"); + assert_eq!(decoded.classification, "future_schema"); + assert_eq!(decoded.generation, 0); + assert_eq!(decoded.leader_epoch, 0); + assert_eq!(decoded.state, "blocked"); + + let malformed = + super::cycle_state::decode_recovery_marker_for_reset(b"{not-json", &DataUsageCacheRevision::Etag("memory-4".to_string())) + .expect("a full-rescan reset must recover even when the marker is malformed"); + assert!(malformed.primary_revision.is_empty()); + assert_eq!(malformed.classification, "future_schema"); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_after_malformed_marker_without_trusting_cursor() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01]) + .await + .expect("corrupt cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), br#"{not-json"#.to_vec()) + .await + .expect("malformed marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("full-rescan reset should recover malformed marker"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(cycle.next, 0, "reset must use the verified usage floor, not marker cursor"); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_ignores_epoch_from_malformed_future_primary() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let mut future_primary = vec![0; 24]; + future_primary[8..16].copy_from_slice(b"RSCY9999"); + future_primary[16..24].copy_from_slice(&u64::MAX.to_le_bytes()); + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), future_primary) + .await + .expect("future cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), br#"{not-json"#.to_vec()) + .await + .expect("malformed marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("full-rescan reset should recover malformed future state"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (_, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(leader_epoch, 1, "invalid persisted bytes must not raise the recovery epoch"); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn ecstore_exact_recovery_marker_delete_honors_etag() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"marker-v1".to_vec()) + .await + .expect("initial recovery marker should be persisted"); + let (_, stale_revision) = read_config_with_revision(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("initial marker revision should load"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"marker-v2".to_vec()) + .await + .expect("replacement recovery marker should be persisted"); + + let delete_result = store + .delete_config_object( + RUSTFS_META_BUCKET, + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + ObjectOptions { + http_preconditions: Some(stale_revision.preconditions()), + ..Default::default() + }, + ) + .await; + assert!(matches!(delete_result, Err(EcstoreError::PreconditionFailed))); + assert_eq!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("replacement marker should remain durable"), + b"marker-v2" + ); +} + +#[tokio::test] +async fn full_rescan_reset_rejects_corrupt_primary_under_stale_blocked_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let corrupt_primary = vec![0xff, 0x00, 0x01]; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), corrupt_primary.clone()) + .await + .expect("corrupt cycle state should be persisted"); + let (_, primary_revision) = read_config_with_revision(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("primary revision should load"); + let marker = ScannerCycleRecoveryMarker { + schema_version: 1, + primary_revision: "memory-stale".to_string(), + generation: 1, + leader_epoch: 1, + classification: "corrupt".to_string(), + first_detected_at_unix_secs: 1, + last_attempt_at_unix_secs: 2, + retry_count: 1, + reason: "blocked primary changed".to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "blocked".to_string(), + }; + let marker_data = serde_json::to_vec(&marker).expect("blocked marker should encode"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), marker_data.clone()) + .await + .expect("blocked marker should be persisted"); + + assert!( + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .is_err(), + "a strict marker must fail closed when its primary revision changed" + ); + assert_eq!( + read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("primary should remain readable"), + corrupt_primary + ); + assert_eq!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("blocked marker should remain durable"), + marker_data + ); + assert!(!matches!(primary_revision, DataUsageCacheRevision::Missing)); +} + +#[tokio::test] +async fn full_rescan_reset_preserves_valid_primary_when_marker_is_malformed() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let primary = CurrentCycle { + next: 42, + ..Default::default() + }; + let old_primary_data = encode_scanner_cycle_state(&primary, 7).expect("valid cycle state should encode"); + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), old_primary_data.clone()) + .await + .expect("valid cycle state should be persisted"); + let (_, old_primary_revision) = read_config_with_revision(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("primary state revision should load"); + let old_usage = DataUsageInfo { + scanner_epoch: Some(7), + scanner_cycle: Some(41), + ..Default::default() + }; + let old_usage_data = serde_json::to_vec(&old_usage).expect("usage snapshot should encode"); + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), old_usage_data.clone()) + .await + .expect("usage snapshot should be persisted"); + let (_, old_usage_revision) = read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("usage snapshot revision should load"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("reset should clear a stale malformed marker"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("valid primary should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("primary cycle state should decode"); + assert_eq!(cycle.next, 42, "reset must not regress an independently fenced primary"); + assert_eq!(leader_epoch, 8, "reset must advance the preserved primary epoch"); + let stale_primary_save = save_config_with_preconditions( + store.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + old_primary_data, + old_primary_revision.preconditions(), + ) + .await; + assert!(matches!(stale_primary_save, Err(EcstoreError::PreconditionFailed))); + let usage = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("usage epoch fence should remain durable"); + assert_eq!( + serde_json::from_slice::(&usage) + .expect("fenced usage should decode") + .scanner_epoch, + Some(8) + ); + let stale_save = save_config_with_preconditions( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + old_usage_data, + old_usage_revision.preconditions(), + ) + .await; + assert!(matches!(stale_save, Err(EcstoreError::PreconditionFailed))); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_resumes_cleanup_pending_preserved_primary() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let completed_at = Utc::now(); + let primary = CurrentCycle { + current: 3, + next: 42, + cycle_completed: vec![completed_at], + started: completed_at, + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + encode_scanner_cycle_state(&primary, 7).expect("valid cycle state should encode"), + ) + .await + .expect("valid cycle state should be persisted"); + let usage = DataUsageInfo { + scanner_epoch: Some(7), + scanner_cycle: Some(41), + ..Default::default() + }; + save_config( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&usage).expect("usage snapshot should encode"), + ) + .await + .expect("usage snapshot should be persisted"); + let marker = ScannerCycleRecoveryMarker { + schema_version: 1, + primary_revision: "memory-old".to_string(), + generation: 41, + leader_epoch: 7, + classification: "corrupt".to_string(), + first_detected_at_unix_secs: 1, + last_attempt_at_unix_secs: 2, + retry_count: 1, + reason: "reset in progress".to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "cleanup-pending".to_string(), + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + serde_json::to_vec(&marker).expect("marker should encode"), + ) + .await + .expect("cleanup marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("reset should resume a cleanup-pending preserved primary"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("preserved cycle state should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("cycle state should decode"); + assert_eq!(cycle.current, 3, "cleanup retry must preserve the in-progress cursor"); + assert_eq!(cycle.next, 42); + assert_eq!(cycle.cycle_completed, vec![completed_at]); + assert_eq!(cycle.started, completed_at); + assert_eq!(leader_epoch, 8); + let usage = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("usage epoch fence should remain durable"); + assert_eq!( + serde_json::from_slice::(&usage) + .expect("usage should decode") + .scanner_epoch, + Some(8) + ); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_oversized_regular_primary_with_malformed_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0; 1024 * 1024 + 1]) + .await + .expect("oversized cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("explicit full-rescan reset should replace an oversized regular primary"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(cycle.next, 0); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_oversized_primary_after_cleanup_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0; 1024 * 1024 + 1]) + .await + .expect("oversized cycle state should be persisted"); + let (_, primary_revision) = read_config_with_revision(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("primary revision should load"); + let marker = ScannerCycleRecoveryMarker { + schema_version: 1, + primary_revision: match primary_revision { + DataUsageCacheRevision::Etag(etag) => etag, + DataUsageCacheRevision::Missing => panic!("primary revision should be present"), + }, + generation: 1, + leader_epoch: 1, + classification: "corrupt".to_string(), + first_detected_at_unix_secs: 1, + last_attempt_at_unix_secs: 2, + retry_count: 1, + reason: "reset in progress".to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "cleanup-pending".to_string(), + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + serde_json::to_vec(&marker).expect("cleanup marker should encode"), + ) + .await + .expect("cleanup marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("cleanup retry should rebuild an oversized primary"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(cycle.next, 0); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_with_oversized_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01]) + .await + .expect("corrupt cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), vec![b'x'; 64 * 1024 + 1]) + .await + .expect("oversized recovery marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("full-rescan reset should recover an oversized marker"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (_, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_with_empty_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01]) + .await + .expect("corrupt cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), Vec::new()) + .await + .expect("empty recovery marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("full-rescan reset should recover an empty marker"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (_, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_keeps_cleanup_marker_when_preserved_epoch_is_exhausted() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let primary = CurrentCycle { + next: 42, + ..Default::default() + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + encode_scanner_cycle_state(&primary, u64::MAX).expect("valid cycle state should encode"), + ) + .await + .expect("valid cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + assert!( + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .is_err() + ); + + let marker = read_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("cleanup marker should remain durable"); + assert_eq!( + serde_json::from_slice::(&marker) + .expect("cleanup marker should decode") + .state, + "cleanup-pending" + ); + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Blocked + )); +} + +#[tokio::test] +async fn full_rescan_reset_rejects_preserved_epoch_that_would_be_terminal() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let primary = CurrentCycle { + next: 42, + ..Default::default() + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + encode_scanner_cycle_state(&primary, u64::MAX - 1).expect("valid cycle state should encode"), + ) + .await + .expect("valid cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + assert!( + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .is_err(), + "reset must not persist the terminal leader epoch" + ); + + let marker = read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("cleanup marker should remain durable"); + assert_eq!( + serde_json::from_slice::(&marker) + .expect("cleanup marker should decode") + .state, + "cleanup-pending" + ); +} + +#[tokio::test] +async fn full_rescan_reset_rejects_usage_floor_that_would_be_terminal() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01]) + .await + .expect("corrupt cycle state should be persisted"); + save_config( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&DataUsageInfo { + scanner_epoch: Some(u64::MAX - 1), + ..Default::default() + }) + .expect("usage floor should encode"), + ) + .await + .expect("usage floor should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + assert!( + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .is_err(), + "reset must not persist the terminal leader epoch" + ); + assert_eq!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("recovery marker should remain durable"), + b"{not-json" + ); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_empty_primary_with_malformed_marker() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), Vec::new()) + .await + .expect("empty cycle state should be persisted"); + save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), b"{not-json".to_vec()) + .await + .expect("malformed marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("explicit full-rescan reset should replace an empty primary"); + + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("rebuilt cycle state should remain durable"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(cycle.next, 0); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn full_rescan_reset_rebuilds_when_primary_cycle_state_is_missing() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let marker = ScannerCycleRecoveryMarker { + schema_version: 1, + primary_revision: "memory-missing".to_string(), + generation: u64::MAX, + leader_epoch: u64::MAX, + classification: "corrupt".to_string(), + first_detected_at_unix_secs: 1, + last_attempt_at_unix_secs: 2, + retry_count: 0, + reason: "missing primary".to_string(), + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(), + state: "blocked".to_string(), + }; + save_config( + store.clone(), + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), + serde_json::to_vec(&marker).expect("marker should encode"), + ) + .await + .expect("marker should be persisted"); + + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("full-rescan reset should recreate missing primary"); + let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .expect("missing primary should be rebuilt"); + let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode"); + assert_eq!(cycle.next, 0); + assert_eq!(leader_epoch, 1); + assert!(matches!( + read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); +} + +#[tokio::test] +async fn corrupt_cycle_state_rename_or_marker_failure_stays_recovery_required() { + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + store.objects.lock().await.insert(state_key.clone(), vec![1]); + store.revisions.lock().await.insert(state_key, 9); + store.fail_put_number.lock().await.insert(marker_key, 1); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Transient(_) + )); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.state, "recovery-required"); + assert!(status.retryable); + assert!( + store + .objects + .lock() + .await + .contains_key(&memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str())) + ); +} + +#[tokio::test] +async fn oversized_or_symlinked_cycle_state_is_rejected() { + let store = Arc::new(MemoryConfigStore::default()); + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.objects.lock().await.insert(key.clone(), vec![0; 1024 * 1024 + 1]); + store.revisions.lock().await.insert(key.clone(), 11); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Blocked + )); + assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("corrupt")); + assert!( + scanner_cycle_recovery_status() + .reason + .as_deref() + .is_some_and(|reason| reason.contains("oversized")) + ); + + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + store.objects.lock().await.remove(&marker_key); + store.objects.lock().await.insert(key.clone(), vec![1]); + store.revisions.lock().await.insert(key.clone(), 12); + store.non_regular_objects.lock().await.insert(key); + // The object contract exposes a non-regular object as `is_dir`; local + // backends reject symlink/reparse entries before they become an object. + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Blocked + )); +} + #[tokio::test] async fn scanner_startup_uses_primary_and_backup_usage_floor() { let store = Arc::new(MemoryConfigStore::default()); @@ -869,6 +1874,31 @@ async fn scanner_startup_uses_primary_and_backup_usage_floor() { assert_eq!(epoch, 11); } +#[tokio::test] +async fn scanner_usage_floor_ignores_older_backup_after_primary_epoch_fence() { + let store = Arc::new(MemoryConfigStore::default()); + let backup_path = format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()); + for (path, epoch, cycle) in [(DATA_USAGE_OBJ_NAME_PATH.as_str(), 8, 100), (backup_path.as_str(), 7, 10_000)] { + store.objects.lock().await.insert( + memory_config_key(RUSTFS_META_BUCKET, path), + serde_json::to_vec(&DataUsageInfo { + scanner_epoch: Some(epoch), + scanner_cycle: Some(cycle), + ..Default::default() + }) + .expect("usage snapshot should encode"), + ); + } + + assert_eq!( + persisted_usage_floor(store).await.expect("usage floor should load"), + PersistedUsageFloor { + next_cycle: 101, + leader_epoch: 8, + } + ); +} + #[test] fn scanner_startup_treats_incomplete_usage_snapshot_as_cold() { let mut legacy = complete_usage_with_bucket_count(Some(std::time::SystemTime::now()), 1); @@ -1001,6 +2031,15 @@ async fn scanner_usage_floor_fails_closed_on_corrupt_or_exhausted_usage_state() assert!(persisted_usage_floor(store.clone()).await.is_err()); + store.objects.lock().await.insert( + memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str()), + br#"{}"#.to_vec(), + ); + assert!( + persisted_usage_floor(store.clone()).await.is_err(), + "a structurally incomplete usage snapshot must not be treated as an empty floor" + ); + store.objects.lock().await.insert( memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str()), serde_json::to_vec(&DataUsageInfo { @@ -1013,7 +2052,6 @@ async fn scanner_usage_floor_fails_closed_on_corrupt_or_exhausted_usage_state() } #[tokio::test] -#[serial] async fn scanner_usage_backup_uses_durable_cycle_cadence_across_tasks() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -1087,7 +2125,6 @@ fn scanner_cycle_advance_fails_before_reserved_exhausted_value() { } #[tokio::test] -#[serial] async fn test_finalize_partial_scan_cycle_reports_persist_failure() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -1111,7 +2148,6 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() { } #[tokio::test] -#[serial] async fn test_persist_scanner_cycle_state_reconciles_newer_winner() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); @@ -1261,6 +2297,22 @@ async fn test_leadership_claim_preserves_usage_epoch_floor_across_old_epoch_conf assert_eq!(store.put_counts.lock().await.get(&key), Some(&3)); } +#[tokio::test] +async fn test_leadership_claim_rejects_terminal_epoch() { + let store = Arc::new(MemoryConfigStore::default()); + let ctx = CancellationToken::new(); + let mut revision = DataUsageCacheRevision::Missing; + let mut cycle = CurrentCycle { + next: 12, + ..Default::default() + }; + let mut persisted_epoch = u64::MAX - 1; + + assert!(!claim_scanner_leadership(&ctx, store.clone(), &mut cycle, &mut revision, &mut persisted_epoch).await); + assert_eq!(persisted_epoch, u64::MAX - 1); + assert!(read_config(store, &DATA_USAGE_BLOOM_NAME_PATH).await.is_err()); +} + #[tokio::test] async fn test_leadership_claim_confirms_commit_after_returned_error() { let store = Arc::new(MemoryConfigStore::default()); @@ -1356,7 +2408,7 @@ async fn test_leadership_claim_usage_fence_rejects_old_inflight_writer() { } #[tokio::test] -async fn test_successful_old_epoch_commit_is_fenced_after_cancellation() { +async fn cycle_budget_lease_takeover_rejects_old_generation() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); let mut revision = DataUsageCacheRevision::Missing; @@ -1401,12 +2453,17 @@ async fn test_successful_old_epoch_commit_is_fenced_after_cancellation() { .await ); - let state = read_config(store, &DATA_USAGE_BLOOM_NAME_PATH) + let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) .await .expect("replacement leadership claim should persist"); let (claimed_cycle, claimed_epoch) = decode_scanner_cycle_state(&state).expect("replacement cycle state should decode"); assert_eq!(claimed_cycle.next, 14); assert_eq!(claimed_epoch, 2); + + let mut stale_cycle = CurrentCycle { next: 15, ..cycle }; + let mut stale_revision = DataUsageCacheRevision::Etag("memory-2".to_string()); + let stale_ctx = CancellationToken::new(); + assert!(!persist_scanner_cycle_state(&stale_ctx, store, &mut stale_cycle, &mut stale_revision, 1,).await); } #[tokio::test] @@ -1534,7 +2591,6 @@ async fn test_usage_save_route_barrier_prevents_missing_snapshot_creation() { } #[tokio::test] -#[serial] async fn test_usage_route_barrier_precedes_durable_reconciliation() { let store = Arc::new(MemoryConfigStore::default()); let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str()); @@ -1562,7 +2618,6 @@ async fn test_usage_route_barrier_precedes_durable_reconciliation() { } #[tokio::test] -#[serial] async fn test_deferred_usage_save_keeps_last_real_save_metric() { let metrics = global_metrics(); metrics.record_scanner_usage_save_result(ScannerUsageSaveResult::Success); @@ -2626,7 +3681,6 @@ fn scanner_cycle_cache_floor_stays_pending_during_deferred_usage_publication() { } #[test] -#[serial] fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() { crate::scanner_io::clear_dirty_usage_bucket("photos"); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -2653,7 +3707,6 @@ fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() { } #[test] -#[serial] fn finalizing_a_deferred_usage_save_keeps_dirty_work_pending() { crate::scanner_io::clear_dirty_usage_bucket("photos"); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -2696,7 +3749,6 @@ async fn scanner_cycle_keeps_remote_pending_acknowledgement() { } #[test] -#[serial] fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() { crate::scanner_io::clear_dirty_usage_bucket("photos"); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -2711,7 +3763,6 @@ fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() { } #[test] -#[serial] fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() { crate::scanner_io::clear_dirty_usage_bucket("photos"); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -2727,7 +3778,6 @@ fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() { } #[test] -#[serial] fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() { crate::scanner_io::clear_dirty_usage_bucket("photos"); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -2743,7 +3793,6 @@ fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() { } #[test] -#[serial] fn data_usage_persist_wait_covers_cache_retries_and_backup() { with_var(rustfs_config::ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -2796,7 +3845,6 @@ async fn maintenance_feature_inspection_preserves_base_cycle_after_timeout() { } #[tokio::test(start_paused = true)] -#[serial] async fn stable_maintenance_detection_preserves_base_cycle_after_timeout() { let ctx = CancellationToken::new(); @@ -2862,7 +3910,6 @@ async fn maintenance_feature_inspection_stops_on_cancellation() { } #[test] -#[serial] fn test_cycle_interval_prefers_explicit_cycle_override() { with_var(ENV_SCANNER_SPEED, Some("slowest"), || { with_var(ENV_SCANNER_CYCLE, Some("42"), || { @@ -2872,7 +3919,6 @@ fn test_cycle_interval_prefers_explicit_cycle_override() { } #[test] -#[serial] fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() { let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS); @@ -2882,7 +3928,6 @@ fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() { } #[test] -#[serial] fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() { let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest); @@ -2892,7 +3937,6 @@ fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() { } #[test] -#[serial] fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() { let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest); @@ -2910,7 +3954,6 @@ fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() { } #[test] -#[serial] fn test_cycle_interval_uses_default_cycle_override_when_unconfigured() { let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS); @@ -3006,6 +4049,24 @@ fn superseded_retry_backoff_grows_from_the_default_cycle() { } } +#[tokio::test(start_paused = true)] +async fn corrupt_cycle_state_backoff_uses_virtual_clock() { + let mut backoff = ScannerRetryBackoff::default(); + backoff.record_retryable_cycle(true); + let first_delay = backoff + .retry_interval(Duration::from_secs(60)) + .expect("the first recovery retry should be scheduled"); + assert_eq!(first_delay, Duration::from_secs(5)); + + let deadline = Instant::now() + first_delay; + assert!(Instant::now() < deadline); + tokio::time::advance(first_delay).await; + assert!(Instant::now() >= deadline); + + backoff.record_retryable_cycle(true); + assert_eq!(backoff.retry_interval(Duration::from_secs(60)), Some(Duration::from_secs(10))); +} + #[test] fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() { let runtime_config = ScannerRuntimeConfig { @@ -3074,7 +4135,6 @@ fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() { } #[test] -#[serial] fn scanner_cycle_schedule_status_reports_effective_backoff() { record_scanner_cycle_schedule(Duration::from_millis(86_400_001), true, 2_048, true, 7); @@ -3354,7 +4414,6 @@ fn dirty_usage_wakes_are_disabled_for_explicit_cycle_policy() { } #[test] -#[serial] fn clean_idle_cap_preserves_default_bitrot_coverage_window() { let config = ScannerRuntimeConfig { bitrot_cycle: Some(Duration::from_secs(30 * 24 * 60 * 60)), @@ -3384,7 +4443,6 @@ fn clean_idle_cap_allows_policy_max_when_bitrot_is_disabled() { } #[test] -#[serial] fn clean_idle_cap_never_shortens_the_base_cycle() { let config = ScannerRuntimeConfig { bitrot_cycle: Some(Duration::from_secs(60)), @@ -3398,7 +4456,6 @@ fn clean_idle_cap_never_shortens_the_base_cycle() { } #[test] -#[serial] fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() { let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS); @@ -3416,7 +4473,6 @@ fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() { } #[test] -#[serial] fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() { let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS); @@ -3430,7 +4486,6 @@ fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() { } #[test] -#[serial] fn test_cycle_interval_supports_minio_speed_alias() { with_var_unset(ENV_SCANNER_SPEED, || { with_var_unset(ENV_SCANNER_CYCLE, || { @@ -3444,7 +4499,6 @@ fn test_cycle_interval_supports_minio_speed_alias() { } #[test] -#[serial] fn test_cycle_interval_supports_minio_cycle_alias() { with_var_unset(ENV_SCANNER_CYCLE, || { with_var_unset(ENV_SCANNER_START_DELAY_SECS, || { @@ -3464,7 +4518,6 @@ fn test_randomized_cycle_delay_handles_small_start_delay() { } #[tokio::test] -#[serial] async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); @@ -3490,7 +4543,6 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() { } #[tokio::test] -#[serial] async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let dirty_generation = crate::scanner_io::dirty_usage_generation(); @@ -3512,7 +4564,6 @@ async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() { } #[tokio::test(start_paused = true)] -#[serial] async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -3534,7 +4585,6 @@ async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer() } #[tokio::test(start_paused = true)] -#[serial] async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3553,7 +4603,6 @@ async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() { } #[tokio::test] -#[serial] async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); crate::scanner_io::record_dirty_usage_bucket("photos"); @@ -3579,7 +4628,6 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() { } #[tokio::test] -#[serial] async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let observed_generation = crate::runtime_config::scanner_runtime_config_generation(); @@ -3607,7 +4655,6 @@ async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() { } #[tokio::test] -#[serial] async fn test_wait_for_next_scanner_cycle_reschedules_for_maintenance_change() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let observed_generation = crate::scanner_io::scanner_maintenance_generation(); @@ -3851,7 +4898,6 @@ fn scanner_activity_after_a_cycle_restores_the_base_interval() { } #[tokio::test(start_paused = true)] -#[serial] async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3879,7 +4925,6 @@ async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity( } #[tokio::test(start_paused = true)] -#[serial] async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3907,7 +4952,6 @@ async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() { } #[tokio::test(start_paused = true)] -#[serial] async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3934,7 +4978,6 @@ async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance } #[tokio::test(start_paused = true)] -#[serial] async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3961,7 +5004,6 @@ async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable() } #[tokio::test(start_paused = true)] -#[serial] async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_clean() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -3989,7 +5031,6 @@ async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_ } #[tokio::test(start_paused = true)] -#[serial] async fn scanner_activity_probe_wait_is_cancellation_aware() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -4020,7 +5061,6 @@ async fn scanner_activity_probe_wait_is_cancellation_aware() { } #[tokio::test(start_paused = true)] -#[serial] async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let ctx = CancellationToken::new(); @@ -4052,7 +5092,6 @@ async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() { } #[test] -#[serial] fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() { with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || { let mode = get_cycle_scan_mode(10, 0, Some(Utc::now()), bitrot_scan_cycle()); @@ -4061,7 +5100,6 @@ fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() { } #[test] -#[serial] fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() { with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || { let recent = Utc::now() - chrono::Duration::minutes(30); @@ -4073,7 +5111,6 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() { } #[test] -#[serial] fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() { with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("off"), || { assert_eq!(get_cycle_scan_mode(1, 0, None, bitrot_scan_cycle()), HealScanMode::Normal); @@ -4081,7 +5118,6 @@ fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() { } #[test] -#[serial] fn test_background_heal_info_for_scan_start_marks_deep_active() { let now = Utc::now(); let info = @@ -4094,7 +5130,6 @@ fn test_background_heal_info_for_scan_start_marks_deep_active() { } #[test] -#[serial] fn test_background_heal_info_for_scan_start_keeps_deep_window_start() { with_var_unset(ENV_SCANNER_BITROT_CYCLE_SECS, || { let started_at = Utc::now(); diff --git a/crates/scanner/src/scanner_budget.rs b/crates/scanner/src/scanner_budget.rs index 43ce732d5..65743439f 100644 --- a/crates/scanner/src/scanner_budget.rs +++ b/crates/scanner/src/scanner_budget.rs @@ -14,17 +14,16 @@ use std::sync::{ Arc, - atomic::{AtomicU8, AtomicU64, Ordering}, + atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering}, }; -use std::time::Instant; - -use tokio::time::Duration; +use tokio::time::{Duration, Instant}; use tokio_util::sync::CancellationToken; const BUDGET_REASON_NONE: u8 = 0; const BUDGET_REASON_RUNTIME: u8 = 1; const BUDGET_REASON_OBJECTS: u8 = 2; const BUDGET_REASON_DIRECTORIES: u8 = 3; +const PROGRESS_CLOCK_SAMPLE_INTERVAL: u64 = 128; #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub(crate) struct ScannerCycleBudgetConfig { @@ -63,29 +62,51 @@ pub struct ScannerCycleBudget { token: CancellationToken, reason: Arc, started_at: Instant, + deadline: Option, max_duration: Option, max_objects: Option, max_directories: Option, track_progress: bool, + track_unbounded_counts: bool, objects_scanned: AtomicU64, directories_started: AtomicU64, entries_visited: AtomicU64, + last_progress_millis: AtomicU64, + cycle_state_persisted: AtomicBool, } impl ScannerCycleBudget { + #[cfg(test)] pub(crate) fn new(parent: &CancellationToken, config: ScannerCycleBudgetConfig) -> Arc { - Self::new_inner(parent, config, false) + Self::new_inner(parent, config, false, false) } pub(crate) fn new_with_progress_tracking(parent: &CancellationToken, config: ScannerCycleBudgetConfig) -> Arc { - Self::new_inner(parent, config, true) + Self::new_inner(parent, config, true, true) } - fn new_inner(parent: &CancellationToken, config: ScannerCycleBudgetConfig, track_progress: bool) -> Arc { + pub(crate) fn new_with_runtime_progress_tracking(parent: &CancellationToken, config: ScannerCycleBudgetConfig) -> Arc { + let track_progress = config.max_duration.is_some(); + Self::new_inner(parent, config, track_progress, false) + } + + fn new_inner( + parent: &CancellationToken, + config: ScannerCycleBudgetConfig, + track_progress: bool, + track_unbounded_counts: bool, + ) -> Arc { let token = parent.child_token(); let reason = Arc::new(AtomicU8::new(BUDGET_REASON_NONE)); + let started_at = Instant::now(); + let deadline = config.max_duration.map(|duration| match started_at.checked_add(duration) { + Some(deadline) => deadline, + // Runtime config rejects this range, but keep programmatic callers + // fail-closed instead of panicking or silently disabling the wall clock. + None => started_at, + }); - if let Some(duration) = config.max_duration { + if let Some(deadline) = deadline { let parent = parent.clone(); let token_wait = token.clone(); let token_cancel = token.clone(); @@ -94,7 +115,7 @@ impl ScannerCycleBudget { tokio::select! { _ = parent.cancelled() => {} _ = token_wait.cancelled() => {} - _ = tokio::time::sleep(duration) => { + _ = tokio::time::sleep_until(deadline) => { Self::cancel_for_reason(&reason, &token_cancel, ScannerCycleBudgetReason::Runtime); } } @@ -104,14 +125,18 @@ impl ScannerCycleBudget { Arc::new(Self { token, reason, - started_at: Instant::now(), + started_at, + deadline, max_duration: config.max_duration, max_objects: config.max_objects, max_directories: config.max_directories, track_progress, + track_unbounded_counts, objects_scanned: AtomicU64::new(0), directories_started: AtomicU64::new(0), entries_visited: AtomicU64::new(0), + last_progress_millis: AtomicU64::new(0), + cycle_state_persisted: AtomicBool::new(false), }) } @@ -131,6 +156,14 @@ impl ScannerCycleBudget { self.max_duration } + pub(crate) fn deadline(&self) -> Option { + self.deadline + } + + pub(crate) fn cancel_for_runtime(&self) { + self.cancel_for(ScannerCycleBudgetReason::Runtime); + } + pub(crate) fn max_objects(&self) -> Option { self.max_objects } @@ -173,15 +206,43 @@ impl ScannerCycleBudget { self.entries_visited.load(Ordering::Relaxed) } + pub(crate) fn mark_cycle_state_persisted(&self) { + self.cycle_state_persisted.store(true, Ordering::Release); + } + + pub(crate) fn cycle_state_persisted(&self) -> bool { + self.cycle_state_persisted.load(Ordering::Acquire) + } + + pub(crate) fn progress_age(&self) -> Duration { + let elapsed_millis = u64::try_from(self.started_at.elapsed().as_millis()).unwrap_or(u64::MAX); + let last_progress = self.last_progress_millis.load(Ordering::Relaxed); + Duration::from_millis(elapsed_millis.saturating_sub(last_progress)) + } + + fn record_progress_sample(&self, event: u64) { + // Clock reads are sampled at batch/count boundaries; the scanner's + // per-object path does not add a second progress atomic. + if event == 0 || (event != 1 && !event.is_multiple_of(PROGRESS_CLOCK_SAMPLE_INTERVAL)) { + return; + } + let elapsed_millis = u64::try_from(self.started_at.elapsed().as_millis()).unwrap_or(u64::MAX); + self.last_progress_millis.store(elapsed_millis, Ordering::Relaxed); + } + pub(crate) fn record_entries_visited(&self, entries_visited: u64) { if self.track_progress { - saturating_fetch_add(&self.entries_visited, entries_visited); + let entries = saturating_fetch_add(&self.entries_visited, entries_visited); + self.record_progress_sample(entries); } } pub(crate) fn record_remote_progress(&self, objects_scanned: u64, directories_started: u64) { if self.track_progress || self.max_objects.is_some() { let objects = saturating_fetch_add(&self.objects_scanned, objects_scanned); + if self.track_progress { + self.record_progress_sample(objects); + } if self.max_objects.is_some_and(|max_objects| objects >= max_objects) { self.cancel_for(ScannerCycleBudgetReason::Objects); } @@ -189,9 +250,12 @@ impl ScannerCycleBudget { if self.track_progress || self.max_directories.is_some() { let directories = saturating_fetch_add(&self.directories_started, directories_started); + if self.track_progress { + self.record_progress_sample(directories); + } if self .max_directories - .is_some_and(|max_directories| directories > max_directories) + .is_some_and(|max_directories| directory_budget_exhausted(directories, max_directories)) { self.cancel_for(ScannerCycleBudgetReason::Directories); } @@ -207,14 +271,17 @@ impl ScannerCycleBudget { } pub(crate) fn try_start_directory(&self) -> bool { - if !self.track_progress && self.max_directories.is_none() { + if self.max_directories.is_none() && !self.track_unbounded_counts { return true; } let directories = saturating_fetch_add(&self.directories_started, 1); + if self.track_progress { + self.record_progress_sample(directories); + } if self .max_directories - .is_some_and(|max_directories| directories > max_directories) + .is_some_and(|max_directories| directory_budget_exhausted(directories, max_directories)) { self.cancel_for(ScannerCycleBudgetReason::Directories); return false; @@ -224,11 +291,14 @@ impl ScannerCycleBudget { } pub(crate) fn record_object_scanned(&self) { - if !self.track_progress && self.max_objects.is_none() { + if self.max_objects.is_none() && !self.track_unbounded_counts { return; } let objects = saturating_fetch_add(&self.objects_scanned, 1); + if self.track_progress { + self.record_progress_sample(objects); + } if self.max_objects.is_some_and(|max_objects| objects >= max_objects) { self.cancel_for(ScannerCycleBudgetReason::Objects); } @@ -259,6 +329,13 @@ fn saturating_fetch_add(value: &AtomicU64, delta: u64) -> u64 { } } +fn directory_budget_exhausted(directories: u64, max_directories: u64) -> bool { + // Saturation hides a remote max+1 update when the configured limit is the + // largest representable counter. Treat that boundary as exhausted rather + // than allowing work to continue indefinitely. + directories > max_directories || (directories == u64::MAX && max_directories == u64::MAX) +} + impl Drop for ScannerCycleBudget { fn drop(&mut self) { self.token.cancel(); @@ -401,6 +478,35 @@ mod tests { assert_eq!(directory_budget.reason(), Some(ScannerCycleBudgetReason::Directories)); } + #[test] + fn directory_budget_fails_closed_when_progress_saturates() { + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &parent, + ScannerCycleBudgetConfig { + max_directories: Some(u64::MAX), + ..Default::default() + }, + ); + + budget.record_remote_progress(0, u64::MAX); + + assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Directories)); + assert!(budget.token().is_cancelled()); + + let local_budget = ScannerCycleBudget::new( + &parent, + ScannerCycleBudgetConfig { + max_directories: Some(u64::MAX), + ..Default::default() + }, + ); + local_budget.record_remote_progress(0, u64::MAX - 1); + assert!(!local_budget.budget_elapsed()); + assert!(!local_budget.try_start_directory()); + assert_eq!(local_budget.reason(), Some(ScannerCycleBudgetReason::Directories)); + } + #[test] fn explicit_progress_tracking_counts_unbounded_remote_work_without_cancelling() { let parent = CancellationToken::new(); @@ -461,4 +567,29 @@ mod tests { assert!(object_limited.requires_serial_progress_accounting()); assert!(directory_limited.requires_serial_progress_accounting()); } + + #[tokio::test(start_paused = true)] + async fn progress_age_uses_virtual_time_and_sampled_progress() { + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_runtime_progress_tracking( + &parent, + ScannerCycleBudgetConfig { + max_duration: Some(Duration::from_secs(60)), + ..Default::default() + }, + ); + + tokio::time::advance(Duration::from_secs(5)).await; + assert_eq!(budget.progress_age(), Duration::from_secs(5)); + budget.record_entries_visited(1); + assert_eq!(budget.progress_age(), Duration::ZERO); + + tokio::time::advance(Duration::from_secs(2)).await; + for _ in 0..126 { + budget.record_entries_visited(1); + } + assert_eq!(budget.progress_age(), Duration::from_secs(2)); + budget.record_entries_visited(1); + assert_eq!(budget.progress_age(), Duration::ZERO); + } } diff --git a/crates/scanner/src/scanner_folder/tests.rs b/crates/scanner/src/scanner_folder/tests.rs index dc4460b15..503b0e75f 100644 --- a/crates/scanner/src/scanner_folder/tests.rs +++ b/crates/scanner/src/scanner_folder/tests.rs @@ -18,7 +18,6 @@ use super::*; use crate::storage_api::VersionPurgeStatusType; use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass}; use rustfs_filemeta::{FileInfo, FileMeta}; -use serial_test::serial; use std::io::Write; #[cfg(unix)] use std::os::unix::fs::{PermissionsExt, symlink}; @@ -356,7 +355,6 @@ impl Drop for TestGuard { } #[tokio::test] -#[serial] async fn test_should_skip_failed_respects_ttl() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir); @@ -378,7 +376,6 @@ async fn test_should_skip_failed_respects_ttl() { } #[tokio::test] -#[serial] async fn test_record_failed_ttl_zero_noop() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(0, 100, &mut scanner, temp_dir); @@ -467,7 +464,6 @@ fn test_should_account_replication_stats_only_for_live_object_versions() { } #[tokio::test] -#[serial] async fn test_heal_replication_only_queues_pending_null_deletes() { async fn replication_skipped_count() -> u64 { global_metrics() @@ -716,7 +712,6 @@ async fn test_scanner_heal_admission_accounting_maps_deep_scan_to_bitrot() { } #[test] -#[serial] fn test_excessive_version_alert_thresholds_use_env() { with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSIONS, Some("3"), || { with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, Some("100"), || { @@ -731,7 +726,6 @@ fn test_excessive_version_alert_thresholds_use_env() { } #[test] -#[serial] fn test_excessive_folders_threshold_uses_env() { with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, Some("3"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -741,7 +735,6 @@ fn test_excessive_folders_threshold_uses_env() { } #[test] -#[serial] fn test_excessive_folders_threshold_default_supports_pbs_layout() { with_var_unset(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -751,7 +744,6 @@ fn test_excessive_folders_threshold_default_supports_pbs_layout() { } #[test] -#[serial] fn test_scanner_yield_every_n_objects_uses_env() { with_var(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, Some("32"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -761,7 +753,6 @@ fn test_scanner_yield_every_n_objects_uses_env() { } #[test] -#[serial] fn test_scanner_yield_every_n_objects_uses_default() { with_var_unset(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -888,7 +879,6 @@ fn test_order_folders_for_resume_reports_stale_hint() { } #[tokio::test] -#[serial] async fn test_record_failed_prunes_to_max_entries() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(1000, 2, &mut scanner, temp_dir); @@ -920,7 +910,6 @@ async fn test_record_failed_prunes_to_max_entries() { } #[tokio::test] -#[serial] async fn test_prune_failed_objects_cache_drops_expired() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(5, 10, &mut scanner, temp_dir); @@ -944,7 +933,6 @@ async fn test_prune_failed_objects_cache_drops_expired() { } #[tokio::test] -#[serial] async fn test_prune_failed_objects_max_zero_keeps_fresh() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir); @@ -1701,7 +1689,6 @@ async fn test_heal_actions_returns_actual_size_without_inline_heal() { } #[tokio::test] -#[serial] #[cfg(unix)] async fn test_scan_folder_skips_unreadable_child_directory() { let (mut scanner, temp_dir) = build_test_scanner().await; @@ -1734,7 +1721,6 @@ async fn test_scan_folder_skips_unreadable_child_directory() { } #[tokio::test] -#[serial] async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -1813,7 +1799,6 @@ async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() { } #[tokio::test] -#[serial] async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -1859,7 +1844,6 @@ async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() { } #[tokio::test(flavor = "current_thread")] -#[serial] async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() { let logs = CapturedLogs::default(); let subscriber = tracing_subscriber::fmt() @@ -2021,7 +2005,6 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() { } #[tokio::test] -#[serial] async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -2099,7 +2082,6 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() { } #[tokio::test] -#[serial] async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -2161,7 +2143,6 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() { } #[tokio::test] -#[serial] async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -2203,7 +2184,6 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() { } #[tokio::test] -#[serial] async fn test_scan_folder_compacted_parent_sends_partial_update() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -2245,7 +2225,6 @@ async fn test_scan_folder_compacted_parent_sends_partial_update() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2290,7 +2269,6 @@ async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone()); @@ -2346,7 +2324,6 @@ async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2391,7 +2368,6 @@ async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2465,7 +2441,6 @@ async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() { } #[tokio::test] -#[serial] async fn scan_data_folder_missing_bucket_returns_partial() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2517,7 +2492,6 @@ async fn scan_data_folder_missing_bucket_returns_partial() { } #[tokio::test] -#[serial] async fn scan_data_folder_missing_scan_root_returns_partial() { let (scanner, temp_dir) = build_test_scanner().await; tokio::fs::remove_dir_all(&temp_dir) @@ -2563,7 +2537,6 @@ async fn scan_data_folder_missing_scan_root_returns_partial() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2632,7 +2605,6 @@ async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folder } #[tokio::test] -#[serial] async fn test_scan_data_folder_partial_object_budget_accumulates_progress() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2715,7 +2687,6 @@ async fn test_scan_data_folder_partial_object_budget_accumulates_progress() { } #[tokio::test] -#[serial] async fn test_partial_compacted_entry_does_not_carry_children() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2761,7 +2732,6 @@ async fn test_partial_compacted_entry_does_not_carry_children() { } #[tokio::test] -#[serial] async fn test_partial_entry_does_not_carry_missing_old_child() { let (mut scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2794,7 +2764,6 @@ async fn test_partial_entry_does_not_carry_missing_old_child() { } #[tokio::test] -#[serial] async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2861,7 +2830,6 @@ async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_success_clears_resume_hint() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2904,7 +2872,6 @@ async fn test_scan_data_folder_success_clears_resume_hint() { } #[tokio::test] -#[serial] async fn test_scan_data_folder_keeps_unresolved_objects_partial() { let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { @@ -2951,7 +2918,6 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() { } #[tokio::test] -#[serial] #[cfg(unix)] async fn test_scan_folder_ignores_symlinked_child_directory() { let (mut scanner, temp_dir) = build_test_scanner().await; diff --git a/crates/scanner/src/scanner_io.rs b/crates/scanner/src/scanner_io.rs index 677719a69..c104fb263 100644 --- a/crates/scanner/src/scanner_io.rs +++ b/crates/scanner/src/scanner_io.rs @@ -48,6 +48,7 @@ use time::OffsetDateTime; use tokio::sync::{Mutex, Notify, Semaphore, mpsc}; use tokio::time::Duration; use tokio_util::sync::CancellationToken; +use tokio_util::task::AbortOnDropHandle; use tracing::{debug, error, warn}; use crate::ScannerObjectInfo as ObjectInfo; diff --git a/crates/scanner/src/scanner_io/io_cache.rs b/crates/scanner/src/scanner_io/io_cache.rs index 3151b4c0c..aa0339e4b 100644 --- a/crates/scanner/src/scanner_io/io_cache.rs +++ b/crates/scanner/src/scanner_io/io_cache.rs @@ -314,7 +314,7 @@ impl ScannerIOCache for SetDisks { let ctx_clone = ctx.clone(); let completed_bucket_count = Arc::new(AtomicUsize::new(0)); let completed_bucket_count_clone = completed_bucket_count.clone(); - let collect_bucket_results_fut = tokio::spawn(async move { + let collect_bucket_results_fut = AbortOnDropHandle::new(tokio::spawn(async move { let mut cancelled = false; loop { @@ -333,7 +333,7 @@ impl ScannerIOCache for SetDisks { } } } - }); + })); let mut futs = Vec::new(); @@ -365,7 +365,7 @@ impl ScannerIOCache for SetDisks { NamespaceScannerWorkerMode::RemoteV4(server_epoch) => Some(server_epoch), NamespaceScannerWorkerMode::Coordinator => None, }; - futs.push(tokio::spawn(async move { + futs.push(AbortOnDropHandle::new(tokio::spawn(async move { let remote_session_id = uuid::Uuid::new_v4(); let mut remote_session_sequence = 0_u64; loop { @@ -1038,7 +1038,7 @@ impl ScannerIOCache for SetDisks { ); } } - })); + }))); } drop(bucket_tx); drop(bucket_result_tx); diff --git a/crates/scanner/src/scanner_io/io_cycle.rs b/crates/scanner/src/scanner_io/io_cycle.rs index c2a8eb253..64763655f 100644 --- a/crates/scanner/src/scanner_io/io_cycle.rs +++ b/crates/scanner/src/scanner_io/io_cycle.rs @@ -242,7 +242,7 @@ impl ScannerIOCycle for ECStore { results[results_index_clone] = result; } }); - wait_futs.push(receiver_fut); + wait_futs.push(AbortOnDropHandle::new(receiver_fut)); let scan_plan = ScannerBucketScanPlan { buckets: set_buckets, @@ -318,7 +318,7 @@ impl ScannerIOCycle for ECStore { record_set_scan_failure(&mut first_err, e); } }); - wait_futs.push(scanner_fut); + wait_futs.push(AbortOnDropHandle::new(scanner_fut)); } } diff --git a/crates/scanner/src/scanner_io/publish_gate_tests.rs b/crates/scanner/src/scanner_io/publish_gate_tests.rs index 56961c89e..c6acdea1a 100644 --- a/crates/scanner/src/scanner_io/publish_gate_tests.rs +++ b/crates/scanner/src/scanner_io/publish_gate_tests.rs @@ -13,7 +13,7 @@ // limitations under the License. use super::*; -use rustfs_data_usage::{ReplicationAllStats, ReplicationStats}; +use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage}; const TEST_PLAN_DIGEST: DataUsageScanPlanDigest = DataUsageScanPlanDigest([7; 32]); @@ -271,7 +271,7 @@ fn completed_data_usage_info_flattens_nested_bucket_entries() { replication_stats: Some(ReplicationAllStats { targets: HashMap::from([( "arn:target".to_string(), - ReplicationStats { + ReplicationTargetUsage { replicated_size: 2048, replicated_count: 2, ..Default::default() diff --git a/crates/scanner/src/scanner_io/tests.rs b/crates/scanner/src/scanner_io/tests.rs index e6ef3bafb..9d2c1586d 100644 --- a/crates/scanner/src/scanner_io/tests.rs +++ b/crates/scanner/src/scanner_io/tests.rs @@ -27,7 +27,6 @@ use crate::{ init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path, }; use rustfs_filemeta::FileInfo; -use serial_test::serial; use temp_env::with_var; use time::OffsetDateTime; use uuid::Uuid; @@ -103,7 +102,6 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc) { } #[tokio::test] -#[serial] async fn scanner_cache_locks_block_same_source_workers() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let set = &store.pools[0].disk_set[0]; @@ -130,7 +128,6 @@ async fn scanner_cache_locks_block_same_source_workers() { } #[tokio::test] -#[serial] async fn scanner_cache_locks_allow_cross_source_workers() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let first_set = &store.pools[0].disk_set[0]; @@ -149,7 +146,6 @@ async fn scanner_cache_locks_allow_cross_source_workers() { } #[tokio::test] -#[serial] async fn scanner_cycle_is_deferred_while_rebalance_is_active() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let mut pool_stats = vec![EcstoreRebalanceStats::default(); store.pools.len()]; @@ -185,7 +181,6 @@ async fn scanner_cycle_is_deferred_while_rebalance_is_active() { } #[tokio::test] -#[serial] async fn scanner_cycle_is_deferred_while_terminal_decommission_is_blocked() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; for decommission in [ @@ -230,7 +225,6 @@ async fn data_usage_publish_fails_when_receiver_is_closed() { } #[tokio::test] -#[serial] async fn multi_pool_scanner_cycle_publishes_combined_usage() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let bucket = format!("scanner-union-{}", Uuid::new_v4().simple()); @@ -278,7 +272,6 @@ async fn multi_pool_scanner_cycle_publishes_combined_usage() { } #[tokio::test] -#[serial] async fn multi_pool_scanner_cycle_zero_fills_bucket_absent_from_first_pool() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let bucket = format!("scanner-second-pool-{}", Uuid::new_v4().simple()); @@ -366,7 +359,6 @@ fn object_lock_config_enabled_accepts_enabled_only() { } #[test] -#[serial] fn dirty_usage_snapshot_clear_preserves_newer_generation() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -381,7 +373,6 @@ fn dirty_usage_snapshot_clear_preserves_newer_generation() { } #[test] -#[serial] fn dirty_usage_generation_acknowledgement_preserves_newer_mutations() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -407,7 +398,6 @@ fn dirty_usage_generation_acknowledgement_preserves_newer_mutations() { } #[test] -#[serial] fn dirty_usage_generation_acknowledgement_rejects_stale_process_and_future_generation() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -437,7 +427,6 @@ fn dirty_usage_generation_acknowledgement_rejects_stale_process_and_future_gener } #[test] -#[serial] fn dirty_usage_snapshot_detects_uncovered_generation() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -462,7 +451,6 @@ fn generation_saturates_instead_of_wrapping() { } #[test] -#[serial] fn dirty_usage_snapshot_clears_a_stably_absent_bucket_after_durable_save() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -484,7 +472,6 @@ fn dirty_usage_snapshot_clears_a_stably_absent_bucket_after_durable_save() { } #[test] -#[serial] fn dirty_usage_snapshot_preserves_an_absent_bucket_recorded_after_listing_started() { clear_dirty_usage_buckets_for_tests(); let generation_before_bucket_list = dirty_usage_generation(); @@ -499,7 +486,6 @@ fn dirty_usage_snapshot_preserves_an_absent_bucket_recorded_after_listing_starte } #[test] -#[serial] fn deleting_a_clean_bucket_invalidates_an_inflight_usage_snapshot() { clear_dirty_usage_buckets_for_tests(); let snapshot = snapshot_dirty_usage_buckets(&[bucket_info("photos")], dirty_usage_generation()); @@ -513,7 +499,6 @@ fn deleting_a_clean_bucket_invalidates_an_inflight_usage_snapshot() { } #[test] -#[serial] fn deleting_a_bucket_during_listing_invalidates_the_resulting_usage_snapshot() { clear_dirty_usage_buckets_for_tests(); let generation_before_bucket_list = dirty_usage_generation(); @@ -527,7 +512,6 @@ fn deleting_a_bucket_during_listing_invalidates_the_resulting_usage_snapshot() { } #[test] -#[serial] fn scanner_maintenance_change_advances_generation_and_marks_usage_dirty() { clear_dirty_usage_buckets_for_tests(); let generation = scanner_maintenance_generation(); @@ -540,7 +524,6 @@ fn scanner_maintenance_change_advances_generation_and_marks_usage_dirty() { } #[test] -#[serial] fn dirty_usage_clear_excludes_failed_buckets() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -572,7 +555,6 @@ fn dirty_usage_clear_plan_excludes_cache_save_failures() { } #[test] -#[serial] fn dirty_usage_is_acknowledged_only_after_durable_usage_confirmation() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -590,7 +572,6 @@ fn dirty_usage_is_acknowledged_only_after_durable_usage_confirmation() { } #[test] -#[serial] fn clear_dirty_usage_bucket_removes_deleted_bucket_marker() { clear_dirty_usage_buckets_for_tests(); record_dirty_usage_bucket("photos"); @@ -917,35 +898,30 @@ async fn bucket_cache_pending_heal_reaches_cycle_maintenance_state() { } #[test] -#[serial] fn scanner_concurrency_limit_preserves_available_when_unconfigured() { crate::reset_foreground_read_activity_for_test(); assert_eq!(scanner_concurrency_limit(0, 4), 4); } #[test] -#[serial] fn scanner_concurrency_limit_caps_to_configured_value() { crate::reset_foreground_read_activity_for_test(); assert_eq!(scanner_concurrency_limit(2, 4), 2); } #[test] -#[serial] fn scanner_concurrency_limit_never_exceeds_available_work() { crate::reset_foreground_read_activity_for_test(); assert_eq!(scanner_concurrency_limit(8, 4), 4); } #[test] -#[serial] fn scanner_concurrency_limit_handles_no_available_work() { crate::reset_foreground_read_activity_for_test(); assert_eq!(scanner_concurrency_limit(2, 0), 0); } #[test] -#[serial] fn scanner_concurrency_limit_yields_to_foreground_reads() { crate::reset_foreground_read_activity_for_test(); crate::set_foreground_read_activity(8); @@ -955,7 +931,6 @@ fn scanner_concurrency_limit_yields_to_foreground_reads() { } #[test] -#[serial] fn scanner_concurrency_limit_yields_to_streaming_reads() { crate::reset_foreground_read_activity_for_test(); let _guard = crate::ForegroundReadGuard::new(); @@ -979,7 +954,6 @@ fn increment_atomic_usize_saturates_at_max() { } #[test] -#[serial] fn scanner_max_concurrent_set_scans_uses_env_cap() { with_var(ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, Some("2"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); @@ -989,7 +963,6 @@ fn scanner_max_concurrent_set_scans_uses_env_cap() { } #[test] -#[serial] fn scanner_max_concurrent_disk_scans_uses_env_cap() { with_var(ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, Some("1"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); diff --git a/crates/scanner/src/sleeper.rs b/crates/scanner/src/sleeper.rs index 200b97e08..de46d53ef 100644 --- a/crates/scanner/src/sleeper.rs +++ b/crates/scanner/src/sleeper.rs @@ -258,7 +258,6 @@ impl SleepTimer { #[cfg(test)] mod tests { use super::*; - use serial_test::serial; use temp_env::{with_var, with_var_unset}; struct ScannerDefaultSpeedGuard; @@ -326,7 +325,6 @@ mod tests { } #[test] - #[serial] fn test_refresh_from_env_applies_speed_and_idle_mode_for_next_cycle() { let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed); SCANNER_IDLE_MODE.store(true, Ordering::Relaxed); @@ -346,7 +344,6 @@ mod tests { } #[test] - #[serial] fn test_refresh_from_env_uses_default_speed_override_when_speed_unset() { let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest); let s = DynamicSleeper::new(ScannerSpeed::Default); @@ -362,7 +359,6 @@ mod tests { } #[tokio::test(start_paused = true)] - #[serial] async fn test_fastest_never_sleeps() { let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed); SCANNER_IDLE_MODE.store(true, Ordering::Relaxed); @@ -376,7 +372,6 @@ mod tests { } #[tokio::test(start_paused = true)] - #[serial] async fn test_idle_mode_off_skips_sleep() { let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed); SCANNER_IDLE_MODE.store(false, Ordering::Relaxed); diff --git a/crates/scanner/tests/lifecycle_integration_test.rs b/crates/scanner/tests/lifecycle_integration_test.rs index 12d928513..22e42732e 100644 --- a/crates/scanner/tests/lifecycle_integration_test.rs +++ b/crates/scanner/tests/lifecycle_integration_test.rs @@ -14,7 +14,6 @@ #![recursion_limit = "256"] -use futures::FutureExt; use rustfs_config::ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT; use rustfs_scanner::scanner_folder::ScannerItem; use rustfs_scanner::scanner_io::ScannerIODisk; @@ -23,10 +22,8 @@ use rustfs_scanner::{ scanner::init_data_scanner, }; use s3s::dto::RestoreRequest; -use serial_test::serial; use std::{ collections::HashMap, - env, path::{Path, PathBuf}, sync::{Arc, Once, OnceLock}, time::Duration, @@ -535,31 +532,15 @@ async fn wait_for_transition(ecstore: &Arc, bucket: &str, object: &str, } } -// SAFETY: this helper is used only by `#[serial]` tests and runs under the single-threaded Tokio -// runtime (`worker_threads = 1`), so no concurrent test can mutate process environment during the -// `env::set_var` / `env::remove_var` window. -#[allow(unsafe_code)] +// Run `test_fn` with `ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT` +// set to `"1"` for its duration. `temp_env` serializes environment mutations +// globally, preventing data races when multiple tests run in parallel. async fn with_forced_immediate_enqueue_timeout(test_fn: F) where F: FnOnce() -> Fut, Fut: std::future::Future, { - let original = env::var_os(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT); - unsafe { - env::set_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, "1"); - } - let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await; - match original { - Some(value) => unsafe { - env::set_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, value); - }, - None => unsafe { - env::remove_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT); - }, - } - if let Err(err) = result { - std::panic::resume_unwind(err); - } + temp_env::async_with_vars([(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, Some("1"))], test_fn()).await; } mod serial_tests { @@ -592,7 +573,6 @@ mod serial_tests { /// body (GET won) or a clean object/version-not-found (expiry won). A /// tier-fetch failure -- the #3491 symptom -- is never tolerated. #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-2)"] async fn test_expire_transitioned_object_never_races_concurrent_get() { let (_disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -738,7 +718,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"] async fn rejected_transition_candidate_is_recovered_from_persisted_delete_journal() { let (_disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -825,7 +804,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"] async fn cancelled_before_cleanup_store_resolution_persists_journal() { let (_disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -919,7 +897,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"] async fn rejected_transition_cleanup_durability_matrix() { #[derive(Clone, Copy)] @@ -1059,7 +1036,6 @@ mod serial_tests { } #[test] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] fn test_transition_and_restore_flows() { std::thread::Builder::new() @@ -1385,7 +1361,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_scanner_enqueues_free_version_cleanup_for_stale_transitioned_object() { let (disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -1446,7 +1421,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_scanner_cleanup_still_works_after_immediate_compensation_transition() { let (disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -1504,7 +1478,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_existing_object_backfill_is_idempotent_after_immediate_compensation_transition() { let (_disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -1547,7 +1520,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "FAILING on main: excluded from the serial ILM lane pending a fix, see rustfs/backlog#1148 (ilm-1 partial)"] async fn test_noncurrent_expiry_still_works_after_immediate_compensation_transition() { let (disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -1631,7 +1603,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "FAILING on main: excluded from the serial ILM lane pending a fix, see rustfs/backlog#1148 (ilm-1 partial)"] async fn test_noncurrent_transition_still_works_after_immediate_compensation_transition() { let (disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -1714,7 +1685,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_modeled_versioned_delete_creates_delete_marker_after_immediate_compensation_transition() { let (_disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -1762,7 +1732,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_modeled_delete_marker_cleanup_after_immediate_compensation_transition() { let (disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -1839,7 +1808,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_scanner_expires_zero_day_current_version() { let (disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -1866,7 +1834,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_put_object_immediately_enqueues_zero_day_current_expiry() { let (_disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -1904,7 +1871,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_scanner_expires_zero_day_noncurrent_version() { let (disk_paths, ecstore) = setup_isolated_test_env(false).await; @@ -1971,7 +1937,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_put_object_immediately_enqueues_zero_day_noncurrent_expiry() { let (_disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -2032,7 +1997,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] async fn test_background_scanner_expires_zero_day_current_version() { let (_disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -2056,7 +2020,6 @@ mod serial_tests { } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"] async fn test_background_scanner_expires_zero_day_current_version_for_exact_key_prefix() { let (_disk_paths, ecstore) = setup_isolated_test_env(true).await; @@ -2122,7 +2085,6 @@ mod serial_tests { /// tier object is untouched (zero `remove` calls) -> GET streams from the /// tier again -> a second restore succeeds. #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"] async fn test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore() { let (_disk_paths, ecstore) = setup_test_env().await; @@ -2254,7 +2216,6 @@ mod serial_tests { /// parts) must reassemble the exact part layout: part count and sizes, /// the multipart ETag, and byte-identical content across part boundaries. #[tokio::test(flavor = "multi_thread", worker_threads = 1)] - #[serial] #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"] async fn test_multipart_restore_preserves_parts_and_etag() { let (_disk_paths, ecstore) = setup_test_env().await; diff --git a/crates/storage-api/src/lib.rs b/crates/storage-api/src/lib.rs index e011327f9..1114349e1 100644 --- a/crates/storage-api/src/lib.rs +++ b/crates/storage-api/src/lib.rs @@ -76,6 +76,7 @@ pub use bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOption pub use capability::{CapabilitySnapshotError, CapabilityState, CapabilityStatus}; pub use error::{StorageErrorCode, StorageResult}; pub use multipart::{CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartUploadResult, PartInfo}; +pub use object::DeleteAccounting; pub use object::ObjectLockDeleteOptions; pub use object::{DeletedObject, ObjectToDelete}; pub use object::{ExpirationOptions, TransitionedObject}; diff --git a/crates/storage-api/src/object.rs b/crates/storage-api/src/object.rs index 9f0957bde..7959354dc 100644 --- a/crates/storage-api/src/object.rs +++ b/crates/storage-api/src/object.rs @@ -218,6 +218,17 @@ pub struct DeletedObject { pub force_delete_generation: Option, } +/// Accounting identity returned by the internal commit-time delete path. +/// +/// This is carried separately from [`DeletedObject`] so adding quota details +/// does not change the source shape of the public S3 delete result contract. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct DeleteAccounting { + pub size: Option, + pub version_id: Option, + pub removed_current_object: bool, +} + impl DeletedObject { pub fn version_purge_status(&self) -> VersionPurgeStatusType { self.replication_state @@ -341,6 +352,19 @@ pub trait ObjectOperations: Send + Sync + fmt::Debug { objects: Vec, opts: Self::ObjectOptions, ) -> (Vec, Vec>); + /// Delete objects and optionally return commit-time accounting identities. + /// The default preserves the ordinary delete contract for implementations + /// that do not expose storage-level accounting details. + async fn delete_objects_with_accounting( + &self, + bucket: &str, + objects: Vec, + opts: Self::ObjectOptions, + ) -> (Vec, Vec>, Vec>) { + let object_count = objects.len(); + let (deleted, errors) = self.delete_objects(bucket, objects, opts).await; + (deleted, errors, vec![None; object_count]) + } async fn put_object_metadata( &self, bucket: &str, diff --git a/crates/targets/AGENTS.md b/crates/targets/AGENTS.md index 03d79992e..5935a7b65 100644 --- a/crates/targets/AGENTS.md +++ b/crates/targets/AGENTS.md @@ -75,4 +75,3 @@ run commands. - `cargo test -p rustfs-targets plugin` - `cargo test -p rustfs-targets runtime` - `cargo test -p rustfs-targets control_plane` -- Full gate before commit: `make pre-commit` diff --git a/crates/test-utils/src/lib.rs b/crates/test-utils/src/lib.rs index 59b4f3e0c..0188c7064 100644 --- a/crates/test-utils/src/lib.rs +++ b/crates/test-utils/src/lib.rs @@ -245,6 +245,18 @@ impl TestECStoreEnvBuilder { .await .expect("build test ECStore"); + // The production bootstrap only persists pool.bin from the elected + // first cluster node. Test stores intentionally have no cluster + // election, but heal-format still requires that durable fence before + // it can write any disk format. Materialize the validated topology + // here so the shared fixture models a ready single-node store. + let mut pool_meta = ecstore.pool_meta.read().await.clone(); + pool_meta.dont_save = false; + pool_meta + .save(ecstore.pools.clone()) + .await + .expect("persist test pool metadata"); + if self.init_bucket_metadata { let buckets_list = ecstore .list_bucket(&BucketOptions { diff --git a/crates/utils/src/envs.rs b/crates/utils/src/envs.rs index 2d039dec5..5e572e5c0 100644 --- a/crates/utils/src/envs.rs +++ b/crates/utils/src/envs.rs @@ -268,7 +268,7 @@ where .parse::() .map_err(|_| { log_once(&format!("env_invalid_value:{used_key}"), || { - format!("Invalid {} value for {used_key}: {value}. Treating as unset.", type_name::()) + format!("Invalid {} value for {used_key}. Treating as unset.", type_name::()) }); }) .ok() @@ -570,7 +570,7 @@ where Ok(parsed) => EnvParseOutcome::Parsed(parsed), Err(_) => { log_once(&format!("env_invalid_value:{used_key}"), || { - format!("Invalid {} value for {used_key}: {value}. Treating as unset.", type_name::()) + format!("Invalid {} value for {used_key}. Treating as unset.", type_name::()) }); EnvParseOutcome::Invalid } diff --git a/crates/utils/src/net.rs b/crates/utils/src/net.rs index 3489ba41c..4fa656776 100644 --- a/crates/utils/src/net.rs +++ b/crates/utils/src/net.rs @@ -622,7 +622,8 @@ mod test { let _resolver_lock = DNS_RESOLVER_TEST_LOCK.lock().unwrap(); reset_dns_resolver_inner(); - let err = resolve_domain("rustfs-resolver-provenance.invalid").unwrap_err(); + // DNS labels are limited to 63 bytes, so the system resolver rejects this before lookup. + let err = resolve_domain("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.invalid").unwrap_err(); assert_ne!(err.kind(), std::io::ErrorKind::Other, "system resolver error was wrapped: {err}"); } diff --git a/deny.toml b/deny.toml index 296b229e4..c6fb8facc 100644 --- a/deny.toml +++ b/deny.toml @@ -43,9 +43,6 @@ allow-git = [ # RustFS fork carrying presigned expiry and constant-time authentication fixes. # owner: rustfs-maintainers review: 2026-10 "https://github.com/rustfs/s3s.git", - # MiMalloc fork pinned for hotpath allocation counting support. - # owner: houseme review: 2026-10 - "https://github.com/xonatius/mimalloc_rust.git", ] [bans] diff --git a/docs/architecture/compat-cleanup-register.md b/docs/architecture/compat-cleanup-register.md index 9904b41c9..d7ecec7f5 100644 --- a/docs/architecture/compat-cleanup-register.md +++ b/docs/architecture/compat-cleanup-register.md @@ -12,6 +12,7 @@ for later deletion. ## Open Items +- `rustfs-6339` legacy bucket policy ID casing: earlier RustFS releases persisted the top-level policy identifier as "ID", while current writes use the S3-compatible "Id" spelling. Readers accept both spellings so retained bucket metadata remains usable after upgrade. Remove the legacy alias after migration tooling has rewritten every retained bucket policy using "ID". - `table-publication-fence-v1` table publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations. - `table-catalog-strong-snapshot-v1` durable strong catalog snapshot compatibility: version 1 writes continue during mixed-version rollout until operators confirm that every serving node reads version 2, and version 1 table/view identifier collisions remain available only for cleanup. Remove version 1 writes and collision cleanup after the minimum supported RustFS release reads version 2 and every retained durable strong snapshot is collision-free and has been upgraded to version 2. - `table-catalog-migration-fence-v1` durable strong migration fence compatibility: version 1 "PREPARING" fences did not distinguish a known-absent global strong snapshot from an unknown baseline, so retries read them but fail closed if the global snapshot is missing. Version 2 preserves the same JSON shape and records the pre-migration global snapshot ETag in the existing target_snapshot_etag field while the fence is "PREPARING". Remove version 1 reads after every supported direct-upgrade source writes version 2 fences and operators have completed or cancelled every older in-progress backing migration. diff --git a/docs/operations/scanner-runtime-controls.md b/docs/operations/scanner-runtime-controls.md index 33206d699..de54b8f40 100644 --- a/docs/operations/scanner-runtime-controls.md +++ b/docs/operations/scanner-runtime-controls.md @@ -52,7 +52,7 @@ The `/v3/scanner/status` response reports each effective runtime value with a | `scanner.max_wait` | `RUSTFS_SCANNER_MAX_WAIT_SECS` | seconds | preset-derived | Caps one scanner sleep. | | `scanner.cycle` | `RUSTFS_SCANNER_CYCLE` | seconds | preset-derived | Sets the interval between scanner cycles. | | `scanner.start_delay` | `RUSTFS_SCANNER_START_DELAY_SECS` | seconds | unset | Sets startup delay and, for compatibility, the cycle interval when `scanner.cycle` is unset. | -| `scanner.cycle_max_duration` | `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` | seconds | `0` | Caps one cycle's runtime. `0` disables this budget. | +| `scanner.cycle_max_duration` | `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` | seconds | `1800` | Caps one cycle's runtime. An explicit `0` disables this budget. | | `scanner.cycle_max_objects` | `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` | objects | `0` | Caps objects processed by one cycle. `0` disables this budget. | | `scanner.cycle_max_directories` | `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` | directories | `0` | Caps directories entered by one cycle. `0` disables this budget. | | `heal.bitrot_cycle` | `RUSTFS_SCANNER_BITROT_CYCLE_SECS` | seconds | `2592000` | Controls periodic deep bitrot scans. `false`, `off`, `no`, or `disabled` disables periodic deep scans; `0`, `true`, `on`, or `yes` runs deep mode every scanner cycle. | @@ -70,6 +70,21 @@ sleep multiplier, maximum wait, and cycle interval. Use `scanner.delay`, `scanner.max_wait`, and `scanner.cycle` when the preset is close but one axis needs a precise override. +When the cycle duration control is unset, RustFS uses a finite 1800-second +(30-minute) default, matching the scanner benchmark guidance. An explicit `0` +preserves the compatibility behavior of an unbounded cycle; object and +directory budgets likewise remain unbounded when explicitly set to `0`. Invalid +or overflowing duration environment values are configuration errors rather than +silent fallback values. + +When a finite deadline expires, RustFS cancels cooperative scanner work and +waits only for the existing bounded shutdown window. A non-yielding I/O future +is dropped after that window. RustFS then attempts a higher leadership epoch so +late cycle, usage, cache, and remote writes from the old generation fail closed. +If the worker cannot stop cooperatively, the cycle state was not confirmed +durable, or that epoch fence cannot be durably persisted, the scanner reports +`recovery-required`; it does not claim an uncooperative cursor was saved. + An explicit `scanner.cycle` or `RUSTFS_SCANNER_CYCLE` is a minimum inter-cycle cadence: dirty-usage notifications do not bypass that configured interval. The default adaptive policy continues to use dirty-usage notifications to wake @@ -144,6 +159,10 @@ metrics.maintenance_control.primary_control metrics.source_work metrics.replication_repair metrics.scan_checkpoint +metrics.cycle_timeout_total +metrics.cycle_last_progress_age +metrics.leader_lease_without_progress +metrics.cycle_recovery_required_total ``` ## Reading Pacing Pressure diff --git a/docs/testing/ci-gates.md b/docs/testing/ci-gates.md new file mode 100644 index 000000000..d87ce1668 --- /dev/null +++ b/docs/testing/ci-gates.md @@ -0,0 +1,149 @@ +# CI gate matrix + +This file is the source of truth for which validation runs on each event, its +configured wall-clock budget, and whether it can block a merge. Test taxonomy, +naming, and nextest serialization rules remain in [README.md](README.md); e2e +membership and counts remain in +[e2e-suite-inventory.md](e2e-suite-inventory.md). + +The distinction between **required** and **report-only** is load-bearing: +a failing job blocks a merge only when its exact check name is present in the +live `main` ruleset. A workflow name, a `merge_group` trigger, or a red PR check +does not make a job required by itself. + +## Required merge checks + +The live `main` ruleset (`6436880`) currently requires exactly these contexts: + +| Required context | Producer | Validation | +|---|---|---| +| `CLA Check` | `.github/workflows/cla.yml` | Contributor agreement | +| `Quick Checks` | `.github/workflows/ci.yml` | Formatting and repository guard scripts | +| `Test and Lint` | `.github/workflows/ci.yml` | Clippy, workspace nextest excluding `e2e_test`, doctests, and migration proofs | + +For pull requests limited to the paths excluded by the main CI workflow, +`.github/workflows/ci-docs-only.yml` reports `Quick Checks` and +`Test and Lint` under the same names. It runs the real quick checks and the +planning-document guard; it does not claim that Rust compilation or runtime +tests ran. Despite the workflow name, these paths also include selected deploy, +workflow, and lock files. + +Verify the live rule rather than trusting this snapshot before changing merge +policy: + +```bash +gh api repos/rustfs/rustfs/rulesets/6436880 \ + --jq '.rules[] | select(.type == "required_status_checks") | .parameters' +``` + +The ruleset currently has `strict_required_status_checks_policy=false`. +`Continuous Integration` accepts `merge_group` events and runs `e2e-full` for +them, but `End-to-End Tests (full merge gate)` is not currently a required +context. Therefore the repository is prepared to test a merge-queue SHA, but +the workflow alone does not prove that every merge passed that lane. + +## Pull request and merge matrix + +Budgets below are job `timeout-minutes`, not typical runtimes. “Report-only” +means the result is visible and actionable but is not in the live required +context list. + +| Event | Validation | Budget | Merge status | Reproduction | +|---|---|---:|---|---| +| PR, non-doc change | `Quick Checks` | 10 min | Required | `make pre-commit` (broader local umbrella) | +| PR, non-doc change | `Test and Lint` | 90 min | Required | `cargo nextest run --profile ci --all --exclude e2e_test` | +| PR, non-doc change | `Typos` | 10 min | Report-only | `typos` | +| PR, non-doc change | `ILM Integration (serial)` | 90 min | Report-only | Use the exact command in `.github/workflows/ci.yml` | +| PR, non-doc change | rio-v2 / swift / sftp test-and-lint variants | 90 min each | Report-only | `cargo nextest run` with the workflow's feature set | +| PR, non-doc change | `Build RustFS Debug Binary` | 30 min | Report-only; prerequisite for black-box lanes | `cargo build -p rustfs --bins` | +| PR, non-doc change | `io_uring Integration (real)` | 30 min | Report-only | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` | +| PR, non-doc change | `End-to-End Tests` (`e2e-smoke` plus `s3s-e2e`) | 30 min | Report-only | `cargo nextest run --profile e2e-smoke -p e2e_test`; then `./scripts/e2e-run.sh ./target/debug/rustfs ` | +| PR, non-doc change | `S3 Implemented Tests` | 60 min | Report-only | Build `rustfs`, then run `scripts/s3-tests/run.sh` with `DEPLOY_MODE=binary`, `TEST_MODE=single`, and `MAXFAIL=0` | +| PR, non-doc change | `S3 Lifecycle Behavior Tests` | 30 min | Report-only | Use the accelerated scanner environment in `.github/workflows/ci.yml` with `scripts/s3-tests/run.sh` | +| PR touching dependency or workflow inputs | Cargo Deny / Workflow Pin Report / Dependency Review | 20 / 5 / 30 min | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` | +| PR touching architecture rules or architecture docs | `Architecture Migration Rules` | 10 min | Report-only | `scripts/check_architecture_migration_rules.sh` | +| PR touching Nix or workspace manifests | `Nix Build & Check` | 60 min | Report-only | `nix flake check` | +| PR limited to main-CI-excluded paths | companion `Quick Checks` and `Test and Lint` | 10 min each | Required | `git diff --check`; `make doc-paths-check` when documentation paths changed | +| `merge_group` | Standard CI plus `e2e-full` | 55 min for `e2e-full` | Standard required contexts only; `e2e-full` report-only | `cargo nextest run --profile e2e-full -p e2e_test` | +| Push to `main` | Standard CI plus `e2e-full` | 55 min for `e2e-full` | Post-merge detection | Same as `merge_group` | +| PR touching fuzz inputs or harness paths | Build plus five 60-second fuzz smoke targets | 60 min build; 30 min per target | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` | +| PR touching selected ecstore disk/format paths | `Rename Safety` on Windows | 60 min | Report-only | Run the four `cargo test -p rustfs-ecstore --lib ` commands in `windows-filesystem.yml` on Windows | + +The authoritative e2e filters live in `.config/nextest.toml`; extend a profile +instead of adding a second ad-hoc selector. Before a profile runs, +`scripts/check_test_wiring.py` compares its exact membership to the committed +digest so a silent test drop fails closed. + +## Scheduled and manual validation + +Scheduled lanes are independent fault domains. They do not block a pull +request, but their workflow-local gate can fail the run and scheduled failures +are routed to the shared failure-issue action. The scheduled-validation +watchdog and freshness workflow separately detect incomplete runs and missing +schedules. + +| Cadence (UTC unless noted) | Workflow / validation | Budget | Verdict and artifacts | Reproduction | +|---|---|---:|---|---| +| Daily 02:17 | Fuzz: five nightly corpus targets | 60 min build; 60 min per target | Gate; corpus/crash artifacts, scheduled failure alert | `MAX_TOTAL_TIME= ./scripts/fuzz/run.sh` | +| Daily 03:17 | MinIO interop (EC + SSE read parity) | 40 min | Gate; scheduled failure alert | Dispatch `minio-interop.yml` or follow its pinned Docker fixture steps | +| Daily 04:29 | Replication / cluster-fault / protocol e2e | 45 / 90 / 90 min | Three independent gates; JUnit, membership, and server logs | `cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--profile e2e-nightly`; `-j 1 --profile e2e-protocols` | +| Daily 06:31 | Warp performance A/B | 180 min | Regression budget gate; A/B summaries and server logs | `bash scripts/run_hotpath_warp_abba.sh --help` | +| Daily 00:07 Asia/Shanghai (16:07 UTC previous day) | Nightly GNU build and Vault lanes | 150 / 90 / 60 min | Build, live Vault, and HA failover gates | Use the commands and pinned Vault images in `nightly-gnu.yml` | +| Daily 03:23 | Security Audit | 20 / 5 min, plus 30 min on PR dependency review | Cargo Deny and workflow-pin gates; scheduled failure alert | `cargo deny check`; `scripts/security/check_workflow_pins.sh` | +| Daily 23:47 | Scheduled Validation Freshness | 10 min | Fails when a critical schedule was never created or is stale | Dispatch `scheduled-validation-freshness.yml` | +| Sunday 00:11 | Full `Continuous Integration` matrix | Per-job budgets above | Weekly variant coverage, including dormant rio-v2 binary/e2e lanes | Dispatch `ci.yml` | +| Sunday 01:13 | Seven-platform build matrix | 150 min per platform | Build/package integrity; scheduled failure alert | Dispatch `build.yml` with an exact platform set | +| Sunday 02:19 | Ceph s3-tests full sweep: single and real four-node, four shards each | 180 min per shard | Compatibility gate; report, JUnit, exact node IDs, and server logs | `scripts/s3-tests/run.sh` against an existing single or distributed target | +| Sunday 06:41 | Mint | 120 min | **Report-only by design**; per-suite PASS/FAIL/NA and raw `log.json` | Reproduce the pinned Docker sequence in `mint.yml` or dispatch it | +| Sunday 07:43 | Workspace line coverage | 120 min | Report-only trend; lcov and JSON retained 90 days | `make coverage` | +| Monthly, day 1 06:37 | Runner Hygiene | 15 min | Validates runner ephemerality; scheduled failure alert | Dispatch `runner-hygiene.yml` | + +Manual `workflow_dispatch` exists for the scheduled workflows above. Manual +runs are debugging evidence and intentionally do not open scheduled-failure +issues. A manual performance run may explicitly allow a known regression; that +override must not be treated as an ordinary passing baseline. + +## Release validation + +Release validation is post-merge and tag-driven; it does not substitute for a +pull-request gate. + +| Event | Validation | Budget | Result | +|---|---|---:|---| +| Push to `main` or weekly schedule | `Build and Release` platform matrix | 150 min per platform | Build artifacts for all selected targets; no release publication on a main push | +| Valid release or preview tag | `Build and Release` plus asset checks | 150 min per platform | Draft release, checksummed assets, and publish step | +| Successful non-preview release-tag build | Docker image build and image scan | 60 min build; 30 min scan | Multi-architecture images plus vulnerability report | +| Successful release-tag build | DEB/RPM packaging | 30 min per architecture | Packages and checksum files uploaded to the release | +| Successful non-preview release-tag build | Helm template test and package | 30 min build; 30 min publish | Versioned chart and repository index | + +Use an exact preview tag for end-to-end release rehearsal. Manual dispatches +are backfill/debug paths and do not prove the automatic `workflow_run` chain. + +## Evidence requirements + +A green check is useful only when it proves the intended behavior ran: + +- Record the exact commit SHA and run URL. +- Separate product failure from runner prerequisites, service readiness, and + cancellation. Repair the precondition, then rerun the exact workload. +- Preserve membership manifests, JUnit, raw compatibility logs, seeds, and + server logs where the workflow provides them. +- For a bug fix or a new fault checker, provide sensitivity evidence: the old + behavior or an intentional mutation must fail the new oracle, and the fixed + behavior must pass it. +- Never promote a report-only lane to required from one green run. Require at + least 14 days and 30 representative pull requests with at least 99% complete + execution, then update the ruleset and this table together. + +## Change checklist + +Update this file in the same pull request when any of these change: + +- workflow triggers, job names, timeouts, or nextest profile ownership; +- required status contexts or strict/merge-queue policy; +- scheduled cadence, alert routing, artifact contract, or local reproduction; +- report-only versus gating semantics. + +Do not copy per-module test counts here. Update +[e2e-suite-inventory.md](e2e-suite-inventory.md) and its enforced membership +digest instead. diff --git a/docs/testing/e2e-suite-inventory.md b/docs/testing/e2e-suite-inventory.md index 0885c32e9..df2fc0847 100644 --- a/docs/testing/e2e-suite-inventory.md +++ b/docs/testing/e2e-suite-inventory.md @@ -5,12 +5,10 @@ > ```bash > cargo nextest list -p e2e_test --message-format json | jq -r '.["rust-suites"][]?.testcases | to_entries[] | select(.value.ignored == false) | .key | split("::")[0]' | sort | uniq -c > ``` -> Modules marked ✅ are in the PR smoke profile `e2e-smoke` -> (`.config/nextest.toml`); admission criteria: `crates/e2e_test/README.md`. -> 🌙 marks tests in the scheduled `e2e-repl-nightly` profile (backlog#1147 -> repl-1): `replication_extension_test` splits 20 fast tests into the PR smoke -> lane and 28 slow / `_real_dual_node` / `_real_three_node` / `_real_single_node` tests into the -> nightly lane (`.github/workflows/e2e-replication-nightly.yml`). +> Modules marked ✅ are in the PR smoke profile `e2e-smoke`; 🌙 marks the +> cluster, protocol, and replication subsets in the consolidated nightly +> workflow. The `e2e-full` merge/main profile covers the remaining default +> single-node tests. Committed test-ID digests are enforced before each run. > Note: counts exclude `#[ignore]`d tests (nextest lists them separately). > Managed-SSE (SSE-S3/SSE-KMS) replication contracts assert successful > re-encryption on the target (backlog#1783); SSE-C replication still pins a @@ -19,19 +17,21 @@ | module | tests | PR smoke | |---|---|---| | admin_auth_test | 4 | ✅ | -| admin_iam_crud_test | 2 | ✅ | +| admin_iam_crud_test | 3 | ✅ | | admin_pools_test | 1 | ✅ | -| admin_timeout_regression_test | 1 | | +| admin_timeout_regression_test | 1 | 🌙 | | anonymous_access_test | 4 | ✅ | | api_rate_limit_test | 3 | | | archive_download_integrity_test | 13 | | | bucket_logging_test | 3 | | | bucket_policy_check_test | 1 | ✅ | +| bucket_stats_regression_test | 3 | | +| chaos | 2 | | | checksum_upload_test | 7 | | -| cluster_concurrency_test | 2 | | -| cluster_multidrive_pool_test | 2 | | -| common | 12 | | -| compression_test | 1 | | +| cluster_concurrency_test | 2 | 🌙 | +| cluster_multidrive_pool_test | 2 | 🌙 | +| common | 14 | | +| compression_test | 6 | ✅ | | connection_cap_test | 2 | | | console_smoke_test | 1 | ✅ | | content_encoding_test | 3 | ✅ | @@ -45,48 +45,58 @@ | delete_marker_migration_semantics_test | 2 | ✅ | | delete_object_no_content_length_test | 1 | | | delete_objects_versioning_test | 2 | ✅ | +| delete_regression_test | 5 | | +| distributed_startup_regression_test | 3 | | | existing_object_tag_policy_test | 4 | | -| fake_s3_target | 4 | ✅ | +| fake_s3_target | 6 | ✅ | | fault_proxy | 7 | | | get_codec_streaming_compat_test | 1 | | +| get_stream_failure_observability_test | 1 | | +| group_delete_test | 1 | | | head_object_consistency_test | 1 | ✅ | | head_object_range_test | 1 | ✅ | -| heal_erasure_disk_rebuild_test | 3 | | -| inline_fast_path_cluster_test | 14 | | +| heal_erasure_disk_rebuild_test | 4 | 🌙 | +| inline_fast_path_cluster_test | 16 | | | internode_rpc_signature_e2e_test | 5 | | -| kms | 41 | | +| kms | 46 | | | leading_slash_key_test | 2 | ✅ | +| lifecycle_regression_test | 4 | | +| list_buckets_auth_test | 1 | ✅ | | list_buckets_double_slash_test | 3 | ✅ | +| list_buckets_iam_filter_test | 1 | ✅ | | list_object_versions_metadata_extension_test | 1 | | | list_object_versions_regression_test | 2 | ✅ | | list_objects_duplicates_test | 3 | ✅ | | list_objects_v2_metadata_extension_test | 1 | | | list_objects_v2_pagination_test | 12 | ✅ | +| listing_regression_test | 4 | | | mc_mirror_small_bucket_test | 1 | | | multipart_auth_test | 75 | | | multipart_storage_class_test | 3 | ✅ | -| namespace_lock_quorum_test | 2 | | +| namespace_lock_quorum_test | 2 | 🌙 | | negative_sigv4_test | 6 | ✅ | +| notification_startup_regression_test | 2 | | | notification_webhook_test | 3 | ✅ | -| object_lambda_test | 16 | | -| object_lock | 33 | | +| object_lambda_test | 16 | 🌙 | +| object_lock | 34 | | | overwrite_cleanup_regression_test | 1 | | | presigned_negative_test | 7 | ✅ | -| protocols | 16 | | +| protocols | 16 | 🌙 | | quota_test | 14 | | -| reliability_disk_fault_test | 3 | | -| reliant | 24 | 18 ✅ | -| replication_extension_test | 50 | 20 ✅ +30 🌙 | +| reliability_disk_fault_test | 4 | | +| reliant | 25 | 19 ✅ | +| replication_extension_test | 75 | 20 ✅ +55 🌙 | | security_boundary_test | 4 | | -| ssec_copy_test | 2 | ✅ | | server_startup_failfast_test | 1 | | | snowball_auto_extract_test | 6 | | | special_chars_test | 14 | ✅ | -| stale_multipart_cleanup_cluster_test | 1 | | +| ssec_copy_test | 2 | ✅ | +| stale_multipart_cleanup_cluster_test | 1 | 🌙 | | storage_class_capability_test | 4 | ✅ | -| sts_query_compat_test | 3 | ✅ | +| sts_query_compat_test | 6 | ✅ | +| tier_transition_regression_test | 3 | | | tls_gen | 3 | | | tls_hot_reload_test | 1 | ✅ | | version_id_regression_test | 10 | ✅ | -**Total listed: 530 tests across 70 modules · PR smoke subset: 148 tests / 33 modules** (31 full modules + 18 `reliant` tests + 20 of `replication_extension_test`) **· nightly `e2e-repl-nightly`: 30 tests** · updated 2026-08-09. +**Total listed: 575 tests across 82 modules · PR smoke: 163 tests / 36 modules · merge/main full: 453 tests / 73 modules · nightly replication: 55 tests · nightly cluster faults: 28 tests / 7 modules · nightly protocols: 16 tests** · updated 2026-08-23. diff --git a/fuzz/fuzz_targets/archive_extract.rs b/fuzz/fuzz_targets/archive_extract.rs index 1597a9a83..b0d40bd99 100644 --- a/fuzz/fuzz_targets/archive_extract.rs +++ b/fuzz/fuzz_targets/archive_extract.rs @@ -48,11 +48,8 @@ fn materialize_case(path: String, prefix: Option, flags: &[String]) -> ( .fold((path, prefix), |(path, prefix), flag| apply_flag(path, prefix, flag)) } -fn has_dot_segments(path: &str) -> bool { - path.split(['/', '\\']).any(|segment| { - let trimmed = segment.trim(); - trimmed == "." || trimmed == ".." - }) +fn has_parent_segments(path: &str) -> bool { + path.split(['/', '\\']).any(|segment| segment == "..") } fuzz_target!(|data: &[u8]| { @@ -66,8 +63,8 @@ fuzz_target!(|data: &[u8]| { if let Ok(key) = normalize_extract_entry_key(&path, prefix.as_deref(), is_dir) { assert!( - !has_dot_segments(&key), - "accepted archive entry retained dot segments: path={:?} prefix={:?} key={:?}", + !has_parent_segments(&key), + "accepted archive entry retained parent segments: path={:?} prefix={:?} key={:?}", path, prefix, key diff --git a/fuzz/fuzz_targets/policy_ingress.rs b/fuzz/fuzz_targets/policy_ingress.rs index 426991d76..32a2de9f2 100644 --- a/fuzz/fuzz_targets/policy_ingress.rs +++ b/fuzz/fuzz_targets/policy_ingress.rs @@ -69,7 +69,11 @@ fuzz_target!(|data: &[u8]| { && let Some(object) = value.as_object() { let mut legacy_doc = Map::new(); - if let Some(policy) = object.get("Policy").or_else(|| object.get("policy")) { + if let Some(policy) = object + .get("Policy") + .or_else(|| object.get("policy")) + .filter(|policy| serde_json::from_value::((*policy).clone()).is_ok()) + { legacy_doc.insert("version".to_string(), json!(1)); legacy_doc.insert("policy".to_string(), policy.clone()); legacy_doc.insert("create_date".to_string(), json!("2025-03-07T12:00:00Z")); diff --git a/protocol/agent/v1/fixtures/auth/MANIFEST.sha256 b/protocol/agent/v1/fixtures/auth/MANIFEST.sha256 index f43d5f192..6df7052ef 100644 --- a/protocol/agent/v1/fixtures/auth/MANIFEST.sha256 +++ b/protocol/agent/v1/fixtures/auth/MANIFEST.sha256 @@ -1,5 +1,6 @@ 3d602080f7ca4c32ba9e37ad1a32665c78560726b30aeee08fd9e95eb2f36194 accept-vectors.json d3c19946288717088145592e0e8d6f2fa684443ba2f73d4c7bc49c415d6dd051 certificate-profile.json -060485263c51003274c056a0e04bec1b7d76157cf599ba79eebe040bc7cee71b error-codes.json +299a2ae34a8ca74bcf31deeb53a08f9eff279efa09d3f5358a0cf10866fe1a5d error-codes.json 43fe297ffb512b1b9f4af62f1832f3aa3905157893bfdc3dcc6d56f5a98aaef6 reject-vectors.json -b946175b094f4a8d75091b652fbe3d4327c9c795f28e02c96e1ab90a429e418d surface-separation.json +0cf26a7332fa6e3f57390e081f2cceead3236f6ca57f7038e3b55c2582af1733 rotation-proof.json +7c07100460fa23fca482466df9ca22f91c7f26b36087a7207a001f7d987f527e surface-separation.json diff --git a/protocol/agent/v1/fixtures/auth/error-codes.json b/protocol/agent/v1/fixtures/auth/error-codes.json index 6a934b139..1af792add 100644 --- a/protocol/agent/v1/fixtures/auth/error-codes.json +++ b/protocol/agent/v1/fixtures/auth/error-codes.json @@ -2,7 +2,7 @@ "protocolVersion": "v1", "fixtureSet": "auth", "fixture": "error-codes", - "description": "Frozen ErrorInfo reasons for agent authentication and negotiation. Clients branch on status and reason, never on message.", + "description": "Frozen ErrorInfo reasons for agent authentication, negotiation, and credential-rotation authorization. Clients branch on status and reason, never on message.", "domain": "rustfs.connect", "detailType": "type.googleapis.com/google.rpc.ErrorInfo", "disclosureRules": [ @@ -81,6 +81,24 @@ "httpStatus": 401, "status": "UNAUTHENTICATED", "meaning": "A browser session cookie was presented to an authenticated agent operation. The agent surface never accepts it." + }, + { + "reason": "ROTATION_CREDENTIAL_NOT_CURRENT", + "httpStatus": 409, + "status": "ABORTED", + "meaning": "The authenticated certificate is valid for ordinary agent operations but is not the device's current ACTIVE credential and therefore cannot authorize another rotation." + }, + { + "reason": "ROTATION_REQUEST_CONFLICT", + "httpStatus": 409, + "status": "ABORTED", + "meaning": "The requestId already belongs to a rotation with different transcript inputs and cannot be reused." + }, + { + "reason": "ROTATION_PROOF_INVALID", + "httpStatus": 401, + "status": "UNAUTHENTICATED", + "meaning": "The credential proof does not verify over the canonical rotation transcript under the presented certificate key." } ] } diff --git a/protocol/agent/v1/fixtures/auth/rotation-proof.json b/protocol/agent/v1/fixtures/auth/rotation-proof.json new file mode 100644 index 000000000..a65b83a30 --- /dev/null +++ b/protocol/agent/v1/fixtures/auth/rotation-proof.json @@ -0,0 +1,306 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "auth", + "fixture": "rotation-proof", + "description": "Frozen proof-of-possession transcript for rotating an online device credential. The current certificate key authorizes one new certificate request for one device and one idempotent request.", + "operation": { + "method": "POST", + "path": "/agent/clusterDevices/{device}:rotateCredential", + "operationId": "rotateClusterDeviceCredential", + "authenticatedSurface": "/agent/*", + "currentCredentialRequiredForNewRotation": true, + "outgoingOverlapCredentialMayAuthenticateOrdinaryOperations": true, + "outgoingOverlapCredentialMayRotate": false, + "outgoingOverlapCredentialMayReplayItsCompletedRotation": true + }, + "replayPolicy": { + "recordState": "COMPLETED", + "bindingFields": [ + "currentCertificateFingerprint", + "clusterDeviceName", + "requestId", + "certificateRequestSha256" + ], + "exactMatch": "returnStoredResult", + "mismatchReason": "ROTATION_REQUEST_CONFLICT", + "retentionLowerBound": "outgoingCredential.validUntil", + "sideEffects": "No issuer call, credential write, or overlap extension." + }, + "completedReplayRecord": { + "state": "COMPLETED", + "currentCertificateFingerprint": "1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09", + "clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3", + "certificateRequestSha256": "kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw", + "resultReference": "credential-rotation-result-01" + }, + "transcript": { + "domain": "RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1", + "domainTerminator": "0x0a", + "fieldSeparator": "0x3a", + "fieldTerminator": "0x0a", + "fieldCount": 4, + "encoding": "US-ASCII", + "normalisationPermitted": false, + "fieldOrder": [ + "currentCertificateFingerprint", + "clusterDeviceName", + "requestId", + "certificateRequestSha256" + ], + "fields": [ + { + "name": "currentCertificateFingerprint", + "position": 1, + "source": "the exact certificate accepted by trusted ingress and resolved by Connect", + "pattern": "^[0-9a-f]{64}$", + "binds": "the one currently active credential authorizing the rotation", + "absenceWouldAllow": "A proof captured from one current certificate to authorize another credential after the device rotated." + }, + { + "name": "clusterDeviceName", + "position": 2, + "source": "the authenticated identity after it is matched to the target resource", + "pattern": "^organizations/[0-9a-f-]{36}/clusters/[0-9a-f-]{36}/clusterDevices/[0-9a-f-]{36}$", + "binds": "the organization, cluster, and device being rotated", + "absenceWouldAllow": "A proof produced by one device to be presented against another device resource." + }, + { + "name": "requestId", + "position": 3, + "source": "the request body", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", + "binds": "the single idempotent rotation attempt", + "absenceWouldAllow": "A captured proof to be replayed as a fresh attempt instead of the same idempotent request." + }, + { + "name": "certificateRequestSha256", + "position": 4, + "source": "recomputed over the exact PKCS#10 DER octets in certificateRequest", + "pattern": "^[A-Za-z0-9_-]{43}$", + "binds": "the next device public key and certificate request", + "absenceWouldAllow": "A valid proof to be combined with an attacker certificate request." + } + ] + }, + "signature": { + "algorithmField": "proof.algorithm", + "algorithmEnumeration": ["ES256"], + "curve": "P-256", + "signatureEncoding": "fixed-width-r-s", + "signatureLengthBytes": 64, + "signatureTransferEncoding": "base64url-unpadded", + "signatureValuePattern": "^[A-Za-z0-9_-]{86}$", + "lowSRequired": true, + "verifyingKeySource": "the SubjectPublicKeyInfo of the exact presented certificate accepted by trusted ingress", + "newKeyPossession": "the PKCS#10 self-signature is verified separately under the public key inside certificateRequest", + "sharedEncodingContract": "protocol/agent/v1/registration-proof.md#the-signature" + }, + "verificationOrder": [ + { + "stage": "protocolVersion", + "rule": "protocolVersion must be the supported major version.", + "reason": "UNSUPPORTED_PROTOCOL" + }, + { + "stage": "encoding", + "rule": "proof.algorithm and proof.value obey the shared ES256 fixed-width low-S contract.", + "reasons": ["UNSUPPORTED_ALGORITHM", "SIGNATURE_MALFORMED", "SIGNATURE_NOT_CANONICAL"] + }, + { + "stage": "certificateRequest", + "rule": "certificateRequest is one self-signed PKCS#10 request whose public key is P-256.", + "reasons": ["CERTIFICATE_REQUEST_MALFORMED", "DEVICE_KEY_UNSUPPORTED"] + }, + { + "stage": "proof", + "rule": "Connect rebuilds the transcript and verifies proof.value under the presented certificate public key.", + "reason": "ROTATION_PROOF_INVALID" + }, + { + "stage": "replay", + "rule": "A completed record bound to the presented credential, device, requestId, and CSR digest returns its stored result without issuing again. A requestId bound to different transcript inputs is refused.", + "reason": "ROTATION_REQUEST_CONFLICT" + }, + { + "stage": "credential", + "rule": "After replay lookup misses, the authenticated certificate must be the device current ACTIVE credential, not a credential in the outgoing overlap.", + "reason": "ROTATION_CREDENTIAL_NOT_CURRENT" + } + ], + "reasonSources": { + "auth": ["UNSUPPORTED_PROTOCOL", "ROTATION_CREDENTIAL_NOT_CURRENT", "ROTATION_REQUEST_CONFLICT", "ROTATION_PROOF_INVALID"], + "sharedFromRegistration": [ + "UNSUPPORTED_ALGORITHM", + "SIGNATURE_MALFORMED", + "SIGNATURE_NOT_CANONICAL", + "CERTIFICATE_REQUEST_MALFORMED", + "DEVICE_KEY_UNSUPPORTED" + ] + }, + "example": { + "inputs": { + "currentCertificateFingerprint": "1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09", + "clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3", + "certificateRequestSha256": "kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw" + }, + "artifacts": { + "currentPublicKeySpki": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENEpjuFZTqp0Hxh/OWV3TGkluNjCo15dk+4CozuR6aT9Vaxhkb2M9nhaVGfk8+aSiSIiKFSCsYonKl8jh743Qow==", + "currentPublicKeyFingerprint": "28608e223c75ed89e72041f12afae4fc1cd3f1d3bcca38cbb62d1a41687610c2", + "certificateRequest": "MIIBHjCBxAIBADBiMRswGQYDVQQDDBJpZ25vcmVkLWJ5LWNvbm5lY3QxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQWPosvIKr5V3CpTUdLfMQxDx31B3SglLKyqRg/oH3J+PhUnqf7pDWW1sTkP2aRIDjAwRn0DpLrz405CcvGHvYEoAAwCgYIKoZIzj0EAwIDSQAwRgIhAMJzXo/CK4E9BfjOxP35he9LLlqENhK7HTzZQTuIgLX2AiEAwOZHibk5HEijTWcJ/UT117nssfJesWZVOWwz/KTIpi8=", + "otherCertificateRequest": "MIIBHTCBxAIBADBiMRswGQYDVQQDDBJpZ25vcmVkLWJ5LWNvbm5lY3QxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATi0KiZGcbPJixDiQN/aIVky7xqaYhjmwAn1aozrC2eanaDENYQ3D5zw5qoZNIp8YUk/CETZrba3C5KYR5ocdlLoAAwCgYIKoZIzj0EAwIDSAAwRQIhAL0ZtvWyTSu18RF5J4ZVIuOGjJpJwSdP+87CVxNKJJduAiAahIruc3FLtO3RI7B8Ome8IsVDUpSAQilThjdOeDMsYg==" + }, + "canonicalTranscript": "RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1\n64:1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09\n148:organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92\n36:7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3\n43:kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw\n", + "canonicalTranscriptLengthBytes": 346, + "canonicalTranscriptSha256": "e5f0e9cd0d5d420bc7e51217512de76e1cf4dceb1d67804b6518c4fb5d9fe434" + }, + "acceptVectors": [ + { + "name": "current credential authorizes one new certificate request", + "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3", + "proof": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ", + "expected": {"accepted": true, "reason": null, "stage": "credential", "replayed": false} + }, + { + "name": "the same certificate request under a new requestId needs its own proof", + "requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d", + "proof": "s-rkF9w41sqgdQNkOlhjcLYlrNHN_HL5xh4C3k_1a9ZQu7Uq0LoET_x7i4Rw3xQ75ec6B6tfqZ3XsHTS0FfN7Q", + "expected": {"accepted": true, "reason": null, "stage": "credential", "replayed": false} + }, + { + "name": "outgoing credential replays its completed byte-equivalent rotation", + "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3", + "proof": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ", + "replayRecord": "completedExample", + "mutation": {"currentCredential": false}, + "expected": {"accepted": true, "reason": null, "stage": "replay", "replayedResultReference": "credential-rotation-result-01"} + } + ], + "rejectVectors": [ + { + "name": "outgoing overlap credential attempts another rotation", + "stage": "credential", + "requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d", + "proof": "s-rkF9w41sqgdQNkOlhjcLYlrNHN_HL5xh4C3k_1a9ZQu7Uq0LoET_x7i4Rw3xQ75ec6B6tfqZ3XsHTS0FfN7Q", + "mutation": {"currentCredential": false}, + "expected": {"accepted": false, "reason": "ROTATION_CREDENTIAL_NOT_CURRENT"} + }, + { + "name": "rotation declares protocol v2", + "stage": "protocolVersion", + "mutation": {"protocolVersion": "v2"}, + "expected": {"accepted": false, "reason": "UNSUPPORTED_PROTOCOL"} + }, + { + "name": "proof declares ES384", + "stage": "encoding", + "mutation": {"proofAlgorithm": "ES384"}, + "expected": {"accepted": false, "reason": "UNSUPPORTED_ALGORITHM"} + }, + { + "name": "DER encoded proof", + "stage": "encoding", + "mutation": {"proofValue": "MEUCIQC-DPseb7FnklDsHeipQQpcHi1DETNZn3zyI_ECSVohkgIgLHeJ0k1L-WK4qhEucihh7yjoFj8lo764CNI9E_n2BLE"}, + "expected": {"accepted": false, "reason": "SIGNATURE_MALFORMED"} + }, + { + "name": "padded base64url proof", + "stage": "encoding", + "mutation": {"proofValue": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ=="}, + "expected": {"accepted": false, "reason": "SIGNATURE_MALFORMED"} + }, + { + "name": "protocol negotiation precedes proof encoding and stale credential state", + "stage": "protocolVersion", + "mutation": {"currentCredential": false, "protocolVersion": "v2", "proofValue": "not-a-signature"}, + "expected": {"accepted": false, "reason": "UNSUPPORTED_PROTOCOL"} + }, + { + "name": "declared algorithm precedes signature bytes", + "stage": "encoding", + "mutation": {"proofAlgorithm": "ES384", "proofValue": "not-a-signature"}, + "expected": {"accepted": false, "reason": "UNSUPPORTED_ALGORITHM"} + }, + { + "name": "malleated high-S proof", + "stage": "encoding", + "mutation": {"proofValue": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZLTiHYssrQGnkdV7tGN154Qk_7kboFz38zq542vAm0goA"}, + "expected": {"accepted": false, "reason": "SIGNATURE_NOT_CANONICAL"} + }, + { + "name": "proof over the rotation fields uses the registration domain", + "stage": "proof", + "mutation": {"proofValue": "as27GKXEKXtym-BU8NUl0BYhJkEUooZWPadrxwOWa40pAaN6p4VxKCCFggZl4ZQsl5CUtaMoMaxM_HzT1c1qUA"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "proof verification precedes stale credential state", + "stage": "proof", + "mutation": {"currentCredential": false, "proofValue": "as27GKXEKXtym-BU8NUl0BYhJkEUooZWPadrxwOWa40pAaN6p4VxKCCFggZl4ZQsl5CUtaMoMaxM_HzT1c1qUA"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "new certificate request key signs instead of the current credential key", + "stage": "proof", + "mutation": {"proofValue": "QEN8wMQQk5HjriDw2aeAxIfsRvrWgIjGR3KlpK81Y34kaTAacO-6_bOMAjQ0hMMIk-YYN6MDCw0Via_Ri0X0BA"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "proof is moved to another current certificate", + "stage": "proof", + "mutation": {"currentCertificateFingerprint": "2bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "proof is moved to another device", + "stage": "proof", + "mutation": {"clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e93"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "proof is replayed under another requestId", + "stage": "proof", + "mutation": {"requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "proof is combined with another certificate request", + "stage": "proof", + "mutation": {"certificateRequest": "otherCertificateRequest"}, + "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"} + }, + { + "name": "completed replay belongs to another certificate fingerprint", + "stage": "replay", + "replayRecord": "completedExample", + "replayRecordMutation": {"currentCertificateFingerprint": "2bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09"}, + "mutation": {"currentCredential": false}, + "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"} + }, + { + "name": "completed replay belongs to another device", + "stage": "replay", + "replayRecord": "completedExample", + "replayRecordMutation": {"clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e93"}, + "mutation": {"currentCredential": false}, + "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"} + }, + { + "name": "completed replay belongs to another requestId", + "stage": "replay", + "replayRecord": "completedExample", + "replayRecordMutation": {"requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d"}, + "mutation": {"currentCredential": false}, + "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"} + }, + { + "name": "completed replay belongs to another CSR digest", + "stage": "replay", + "replayRecord": "completedExample", + "replayRecordMutation": {"certificateRequestSha256": "PXxfNGWtIbrqVocag6lOLDDR41AjpHXJ7UcAWkbTSAs"}, + "mutation": {"currentCredential": false}, + "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"} + } + ] +} diff --git a/protocol/agent/v1/fixtures/auth/surface-separation.json b/protocol/agent/v1/fixtures/auth/surface-separation.json index 4d3f8a895..611a42cab 100644 --- a/protocol/agent/v1/fixtures/auth/surface-separation.json +++ b/protocol/agent/v1/fixtures/auth/surface-separation.json @@ -49,7 +49,7 @@ "schemeType": "mutualTLS", "forbiddenSecuritySchemes": ["sessionCookie"], "defaultSecurity": ["agentMutualTls"], - "publicOperations": ["getProtocolStatus"] + "publicOperations": ["getProtocolStatus", "exchangeRegistrationToken"] }, { "document": "openapi/control.json", diff --git a/protocol/agent/v1/fixtures/fixture-sets.json b/protocol/agent/v1/fixtures/fixture-sets.json index 5ded06740..72c4c1919 100644 --- a/protocol/agent/v1/fixtures/fixture-sets.json +++ b/protocol/agent/v1/fixtures/fixture-sets.json @@ -11,7 +11,7 @@ { "name": "auth", "status": "populated", - "purpose": "Client certificate profile, RFC 9440 header profile, authentication accept and reject vectors, surface separation, and the frozen error reason registry." + "purpose": "Client certificate profile, RFC 9440 header profile, authentication and credential-rotation proof vectors, surface separation, and the frozen error reason registry." }, { "name": "version", @@ -25,7 +25,7 @@ }, { "name": "heartbeat", - "status": "reserved", + "status": "populated", "purpose": "Heartbeat payloads, Connect receive time, and freshness window behavior." }, { diff --git a/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 b/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 new file mode 100644 index 000000000..7aa8e3191 --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 @@ -0,0 +1,5 @@ +975c1ca53eefeef6766a6fc0b3d3281f7408255342b0686e5e2aee5ad055414c duplicate.json +963529a38a02849c6c2acc6d72668dca9f63218b49c89fae41a451b584850411 overflow.json +e3adeee1c8a19aa17e70894896fb79c072e3785bea3611b93c11e79f039ed5af stale.json +35b9cebd8525389a701e8fe69fbe96407bcb31aa28392fe95babf4a4886985ad unknown.json +37941735dbd6ad3d238258a7b2cae6f0b3aa0ecaae1d8817817c3d718d11d633 valid.json diff --git a/protocol/agent/v1/fixtures/heartbeat/duplicate.json b/protocol/agent/v1/fixtures/heartbeat/duplicate.json new file mode 100644 index 000000000..11cad8ac1 --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/duplicate.json @@ -0,0 +1,9 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "duplicate", + "description": "An exact requestId replay returns the first result and creates no second heartbeat.", + "first": {"requestId": "550e8400-e29b-41d4-a716-446655440000", "sequence": 42}, + "replay": {"requestId": "550e8400-e29b-41d4-a716-446655440000", "sequence": 42}, + "expected": {"decision": "DUPLICATE", "heartbeatWrites": 1, "events": 1, "sameResponse": true} +} diff --git a/protocol/agent/v1/fixtures/heartbeat/overflow.json b/protocol/agent/v1/fixtures/heartbeat/overflow.json new file mode 100644 index 000000000..7b62b5b45 --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/overflow.json @@ -0,0 +1,11 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "overflow", + "description": "Values beyond frozen bounds are rejected before persistence.", + "vectors": [ + {"field": "sequence", "value": 9007199254740992, "maximum": 9007199254740991}, + {"field": "coarseNodeSummary.total", "value": 4097, "maximum": 4096} + ], + "expected": {"decision": "REJECT", "httpStatus": 422, "status": "INVALID_ARGUMENT"} +} diff --git a/protocol/agent/v1/fixtures/heartbeat/stale.json b/protocol/agent/v1/fixtures/heartbeat/stale.json new file mode 100644 index 000000000..aabe98a6f --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/stale.json @@ -0,0 +1,9 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "stale", + "description": "A lower heartbeat sequence is retained as history and cannot replace the current projection.", + "head": {"requestId": "550e8400-e29b-41d4-a716-446655440000", "sequence": 42}, + "late": {"requestId": "7c4d2e10-9f83-4a5b-b6c7-d8e9f0a1b2c3", "sequence": 9}, + "expected": {"decision": "ACCEPT_HISTORY", "currentSequence": 42, "historySequence": 9} +} diff --git a/protocol/agent/v1/fixtures/heartbeat/unknown.json b/protocol/agent/v1/fixtures/heartbeat/unknown.json new file mode 100644 index 000000000..6639cc0ff --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/unknown.json @@ -0,0 +1,18 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "unknown", + "description": "Unknown optional members and capabilities are accepted, discarded before hashing, and never stored or echoed.", + "requestAdditions": { + "telemetryProfile": "extended", + "authorization": "Bearer non-functional-example", + "capabilities": ["heartbeat", "future.capability"], + "coarseNodeSummary": {"rackNames": ["customer-rack"]} + }, + "expected": { + "decision": "ACCEPT", + "storedCapabilities": ["heartbeat"], + "discarded": ["authorization", "future.capability", "telemetryProfile", "coarseNodeSummary.rackNames"], + "echoed": [] + } +} diff --git a/protocol/agent/v1/fixtures/heartbeat/valid.json b/protocol/agent/v1/fixtures/heartbeat/valid.json new file mode 100644 index 000000000..ed3561b80 --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/valid.json @@ -0,0 +1,21 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "valid", + "description": "A bounded L0 heartbeat. clientTime is advisory; Connect's receivedAt is online authority.", + "request": { + "protocolVersion": "v1", + "requestId": "550e8400-e29b-41d4-a716-446655440000", + "agentVersion": "rustfs-agent/1.19.4", + "capabilities": ["heartbeat", "inventory"], + "sequence": 42, + "clientTime": "2026-08-22T01:02:03Z", + "coarseNodeSummary": {"total": 8, "healthy": 7, "degraded": 1} + }, + "expected": { + "decision": "ACCEPT", + "acceptedVersion": "v1", + "responseFields": ["serverTime", "acceptedVersion", "capabilityHints"], + "onlineAuthority": "serverTime" + } +} diff --git a/rustfs/Cargo.toml b/rustfs/Cargo.toml index 206d9dfcc..03afb5628 100644 --- a/rustfs/Cargo.toml +++ b/rustfs/Cargo.toml @@ -288,6 +288,7 @@ zstd.workspace = true # Cryptography and Security rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] } rustls-pki-types = { workspace = true } +x509-parser = { workspace = true } subtle = { workspace = true } jiff = { workspace = true, features = ["serde"] } time = { workspace = true, features = ["parsing", "formatting", "serde", "macros"] } @@ -321,7 +322,7 @@ thiserror = { workspace = true } tracing.workspace = true url = { workspace = true } urlencoding = { workspace = true } -uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] } +uuid = { workspace = true, features = ["v4", "v5", "fast-rng", "macro-diagnostics"] } zip = { workspace = true } libc = { workspace = true } rand = { workspace = true, features = ["serde"] } @@ -335,16 +336,16 @@ opentelemetry = { workspace = true } tracing-opentelemetry = { workspace = true } # Data structures hashbrown = { workspace = true, features = ["serde", "rayon"] } -mimalloc = { workspace = true } +rustfs-mimalloc = { workspace = true } [target.'cfg(target_os = "linux")'.dependencies] libsystemd.workspace = true [target.'cfg(not(target_os = "windows"))'.dependencies] -libmimalloc-sys.workspace = true +rustfs-mimalloc-sys.workspace = true [dev-dependencies] -uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] } +uuid = { workspace = true, features = ["v4", "v5", "fast-rng", "macro-diagnostics"] } serial_test = { workspace = true } tempfile = { workspace = true } aws-config = { workspace = true } diff --git a/rustfs/src/admin/AGENTS.md b/rustfs/src/admin/AGENTS.md index 4564baa2f..fc78c4ace 100644 --- a/rustfs/src/admin/AGENTS.md +++ b/rustfs/src/admin/AGENTS.md @@ -25,4 +25,3 @@ Applies to `rustfs/src/admin/`. ## Suggested Validation - Admin handler and routing tests under `rustfs/src/admin/` -- Full gate before commit: `make pre-commit` diff --git a/rustfs/src/admin/handlers/mod.rs b/rustfs/src/admin/handlers/mod.rs index f0a32f402..6822ccaa4 100644 --- a/rustfs/src/admin/handlers/mod.rs +++ b/rustfs/src/admin/handlers/mod.rs @@ -126,6 +126,7 @@ mod tests { let _list_remote_target_handler = replication::ListRemoteTargetHandler {}; let _remove_remote_target_handler = replication::RemoveRemoteTargetHandler {}; let _scanner_status_handler = scanner::ScannerStatusHandler {}; + let _scanner_cycle_state_reset_handler = scanner::ScannerCycleStateResetHandler {}; let _ilm_expiry_status_handler = scanner::IlmExpiryStatusHandler {}; let _manual_transition_handler = ilm_transition::ManualTransitionRunHandler {}; let _manual_transition_status_handler = ilm_transition::ManualTransitionJobStatusHandler {}; diff --git a/rustfs/src/admin/handlers/replication.rs b/rustfs/src/admin/handlers/replication.rs index 106c25790..229c3c9fd 100644 --- a/rustfs/src/admin/handlers/replication.rs +++ b/rustfs/src/admin/handlers/replication.rs @@ -73,6 +73,8 @@ enum TargetUpdateOp { /// Connection group: credentials plus endpoint, target bucket, and TLS settings. Credentials, Sync, + /// Per-target read-proxy opt-out (`disableProxy`). + Proxy, Bandwidth, Path, } @@ -81,12 +83,13 @@ fn parse_remote_target_update_ops(queries: &HashMap) -> S3Result const SUPPORTED_OPS: &[(&str, TargetUpdateOp)] = &[ ("creds", TargetUpdateOp::Credentials), ("sync", TargetUpdateOp::Sync), + ("proxy", TargetUpdateOp::Proxy), ("bandwidth", TargetUpdateOp::Bandwidth), ("path", TargetUpdateOp::Path), ]; // Present in the MinIO wire contract, but they drive target fields this // version rejects as unsupported — fail loudly instead of silently ignoring. - const UNSUPPORTED_OPS: &[&str] = &["proxy", "healthcheck", "edge", "edgeSyncBeforeExpiry"]; + const UNSUPPORTED_OPS: &[&str] = &["healthcheck", "edge", "edgeSyncBeforeExpiry"]; for key in UNSUPPORTED_OPS { if queries.get(*key).is_some_and(|value| value == "true") { @@ -312,11 +315,10 @@ impl RemoteTargetRequest { )); } - for (unsupported, configured) in - REMOTE_TARGET_UNSUPPORTED_FIELDS - .iter() - .copied() - .zip([self.disable_proxy, self.edge, self.edge_sync_before_expiry]) + for (unsupported, configured) in REMOTE_TARGET_UNSUPPORTED_FIELDS + .iter() + .copied() + .zip([self.edge, self.edge_sync_before_expiry]) { if configured { return Err(s3_error!( @@ -702,6 +704,7 @@ impl Operation for SetRemoteTargetHandler { target.deployment_id = remote_target.deployment_id.clone(); } TargetUpdateOp::Sync => target.replication_sync = remote_target.replication_sync, + TargetUpdateOp::Proxy => target.disable_proxy = remote_target.disable_proxy, TargetUpdateOp::Bandwidth => target.bandwidth_limit = remote_target.bandwidth_limit, TargetUpdateOp::Path => target.path = remote_target.path.clone(), } @@ -1520,6 +1523,7 @@ mod tests { ("update", "true"), ("creds", "true"), ("sync", "true"), + ("proxy", "true"), ("bandwidth", "true"), ("path", "true"), ])) @@ -1529,6 +1533,7 @@ mod tests { vec![ TargetUpdateOp::Credentials, TargetUpdateOp::Sync, + TargetUpdateOp::Proxy, TargetUpdateOp::Bandwidth, TargetUpdateOp::Path ] @@ -2070,7 +2075,6 @@ mod tests { ("credentials.session_token", serde_json::json!("session-token")), ("credentials.expiration", serde_json::json!("2026-01-01T00:00:00Z")), ("api", serde_json::json!("s3v2")), - ("disableProxy", serde_json::json!(true)), ("edge", serde_json::json!(true)), ("edgeSyncBeforeExpiry", serde_json::json!(true)), ] { @@ -2300,6 +2304,44 @@ mod tests { assert!(!REMOTE_TARGET_UNSUPPORTED_FIELDS.contains(&"healthCheckDuration")); } + #[test] + fn remote_target_disable_proxy_is_declared_writable_edge_stays_unsupported() { + assert!(REMOTE_TARGET_WRITABLE_FIELDS.contains(&"disableProxy")); + assert!(!REMOTE_TARGET_UNSUPPORTED_FIELDS.contains(&"disableProxy")); + // edge sync has no implementation behind it — it must stay rejected. + assert!(REMOTE_TARGET_UNSUPPORTED_FIELDS.contains(&"edge")); + assert!(REMOTE_TARGET_UNSUPPORTED_FIELDS.contains(&"edgeSyncBeforeExpiry")); + } + + #[test] + fn remote_target_create_accepts_disable_proxy() { + let mut request = valid_remote_target_request(); + request["disableProxy"] = serde_json::json!(true); + + let target = serde_json::from_value::(request) + .expect("request should deserialize") + .into_bucket_target() + .expect("disableProxy is a supported per-target read-proxy opt-out"); + + assert!(target.disable_proxy); + } + + #[test] + fn update_body_with_proxy_op_toggles_disable_proxy_without_credentials() { + // Mirrors the other partial-update groups: a proxy-only update body may + // omit the connection fields entirely. + let body = serde_json::json!({ + "arn": "arn:rustfs:replication:us-east-1:dep:target", + "type": "replication", + "disableProxy": true + }); + let request: RemoteTargetRequest = serde_json::from_value(body).expect("partial update body should deserialize"); + let target = request + .into_update_bucket_target(&[TargetUpdateOp::Proxy]) + .expect("proxy-only update must not require credentials"); + assert!(target.disable_proxy); + } + #[test] fn remote_target_capability_fields_do_not_overlap() { for field in REMOTE_TARGET_UNSUPPORTED_FIELDS { diff --git a/rustfs/src/admin/handlers/scanner.rs b/rustfs/src/admin/handlers/scanner.rs index ad500004d..fa8df2a69 100644 --- a/rustfs/src/admin/handlers/scanner.rs +++ b/rustfs/src/admin/handlers/scanner.rs @@ -13,8 +13,11 @@ // limitations under the License. use crate::admin::auth::authorize_admin_request; +use crate::admin::handlers::supervise_admin_mutation; use crate::admin::router::{AdminOperation, Operation, S3Router}; -use crate::admin::runtime_sources::current_scanner_metrics_report; +use crate::admin::runtime_sources::{ + app_context_from_req, current_object_store_handle_for_context, current_scanner_metrics_report, +}; use crate::module_switches::{ENV_SCANNER_ENABLED, scanner_enabled_from_env}; use crate::server::ADMIN_PREFIX; use chrono::Utc; @@ -22,11 +25,13 @@ use http::{HeaderMap, HeaderValue}; use hyper::{Method, StatusCode}; use matchit::Params; use rustfs_common::metrics::{ScannerLifecycleExpirySnapshot, ScannerMaintenanceControlSnapshot, ScannerMetricsReport}; +use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE; use rustfs_credentials::Credentials; use rustfs_policy::policy::action::{Action, AdminAction}; use s3s::header::CONTENT_TYPE; use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; -use serde::Serialize; +use serde::{Deserialize, Serialize}; +use tokio_util::sync::CancellationToken; const JSON_CONTENT_TYPE: &str = "application/json"; @@ -38,6 +43,13 @@ struct ScannerStatusResponse { metrics: ScannerMetricsReport, cycle_schedule: rustfs_scanner::ScannerCycleScheduleStatus, runtime_config: rustfs_scanner::runtime_config::ScannerRuntimeConfigStatus, + cycle_recovery: rustfs_scanner::ScannerCycleRecoveryStatus, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ScannerCycleResetRequest { + mode: String, } #[derive(Debug, Serialize)] @@ -117,6 +129,7 @@ fn scanner_status_response( metrics, cycle_schedule, runtime_config, + cycle_recovery: rustfs_scanner::scanner::scanner_cycle_recovery_status(), } } @@ -144,6 +157,11 @@ pub fn register_scanner_route(r: &mut S3Router) -> std::io::Resu format!("{ADMIN_PREFIX}/v3/scanner/status").as_str(), AdminOperation(&ScannerStatusHandler {}), )?; + r.insert( + Method::POST, + format!("{ADMIN_PREFIX}/v3/scanner/cycle-state/reset").as_str(), + AdminOperation(&ScannerCycleStateResetHandler {}), + )?; r.insert( Method::GET, format!("{ADMIN_PREFIX}/v3/ilm/expiry/status").as_str(), @@ -163,6 +181,13 @@ async fn validate_scanner_status_request(req: &S3Request) -> S3Result) -> S3Result { + if req.credentials.is_none() { + return Err(s3_error!(InvalidRequest, "missing credentials")); + } + authorize_admin_request(req, vec![Action::AdminAction(AdminAction::ConfigUpdateAdminAction)]).await +} + fn json_response(body: Vec) -> S3Result> { let mut headers = HeaderMap::new(); let content_type = HeaderValue::from_str(JSON_CONTENT_TYPE) @@ -192,6 +217,37 @@ impl Operation for ScannerStatusHandler { pub struct IlmExpiryStatusHandler {} +pub struct ScannerCycleStateResetHandler {} + +#[async_trait::async_trait] +impl Operation for ScannerCycleStateResetHandler { + async fn call(&self, mut req: S3Request, _params: Params<'_, '_>) -> S3Result> { + let _cred = validate_scanner_reset_request(&req).await?; + let body = req + .input + .store_all_limited(MAX_ADMIN_REQUEST_BODY_SIZE) + .await + .map_err(|err| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("invalid reset request body: {err}")))?; + let reset = serde_json::from_slice::(&body) + .map_err(|err| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("invalid reset request body: {err}")))?; + if reset.mode != "full-rescan" { + return Err(S3Error::with_message(S3ErrorCode::InvalidRequest, "reset mode must be full-rescan")); + } + let context = app_context_from_req(&req) + .ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "storage layer not initialized"))?; + let store = current_object_store_handle_for_context(Some(context.as_ref())) + .ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "storage layer not initialized"))?; + supervise_admin_mutation("scanner cycle state reset", async move { + rustfs_scanner::scanner::reset_scanner_cycle_recovery(CancellationToken::new(), store) + .await + .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?; + Ok::<_, S3Error>(()) + }) + .await?; + json_response(br#"{"status":"reset","mode":"full-rescan"}"#.to_vec()) + } +} + #[async_trait::async_trait] impl Operation for IlmExpiryStatusHandler { async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { @@ -237,6 +293,38 @@ mod tests { assert_eq!(err.message(), Some("missing credentials")); } + #[tokio::test] + async fn scanner_reset_gate_rejects_missing_credentials() { + let req = S3Request { + input: Body::from(String::new()), + method: Method::POST, + uri: http::Uri::from_static("/rustfs/admin/v3/scanner/cycle-state/reset"), + headers: HeaderMap::new(), + extensions: http::Extensions::new(), + credentials: None, + region: None, + service: None, + trailing_headers: None, + }; + + let err = validate_scanner_reset_request(&req) + .await + .expect_err("a reset request without credentials must be rejected"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + assert_eq!(err.message(), Some("missing credentials")); + } + + #[test] + fn admin_reset_requires_full_rescan_or_verified_cursor() { + let full_rescan: ScannerCycleResetRequest = + serde_json::from_str(r#"{"mode":"full-rescan"}"#).expect("full rescan must be accepted"); + assert_eq!(full_rescan.mode, "full-rescan"); + let cursor: ScannerCycleResetRequest = + serde_json::from_str(r#"{"mode":"cursor"}"#).expect("mode validation belongs to the handler"); + assert_ne!(cursor.mode, "full-rescan"); + assert!(serde_json::from_str::(r#"{"mode":"full-rescan","cursor":"untrusted"}"#).is_err()); + } + #[test] fn scanner_disabled_reason_reports_startup_env_key() { assert_eq!(scanner_disabled_reason(true), None); @@ -304,6 +392,11 @@ mod tests { assert_eq!(encoded["cycle_schedule"]["effective_interval_seconds"], 0); assert_eq!(encoded["cycle_schedule"]["clean_idle_backoff_enabled"], false); assert_eq!(encoded["cycle_schedule"]["clean_idle_backoff_multiplier"], 1); + assert_eq!(encoded["cycle_recovery"]["state"], "healthy"); + assert_eq!( + encoded["cycle_recovery"]["quarantine_path"], + rustfs_scanner::DATA_USAGE_BLOOM_RECOVERY_PATH.as_str() + ); } #[test] diff --git a/rustfs/src/admin/handlers/site_replication.rs b/rustfs/src/admin/handlers/site_replication.rs index 84807db6a..3ee3b8af5 100644 --- a/rustfs/src/admin/handlers/site_replication.rs +++ b/rustfs/src/admin/handlers/site_replication.rs @@ -41,7 +41,7 @@ use crate::admin::storage_api::config::save_admin_config; use crate::admin::storage_api::contract::bucket::{ BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions, SRBucketDeleteOp, }; -use crate::admin::storage_api::error::Error as StorageError; +use crate::admin::storage_api::error::{Error as StorageError, is_err_bucket_not_found}; use crate::admin::storage_api::runtime::ECStore; use crate::admin::utils::{encode_compatible_admin_payload, read_compatible_admin_body}; use crate::auth::constant_time_eq; @@ -55,6 +55,7 @@ use crate::storage::storage_api::{ use base64::Engine; use base64::engine::general_purpose::STANDARD as BASE64_STANDARD; use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use futures::StreamExt; use hmac::{Hmac, Mac}; use http::header::{CONTENT_TYPE, HOST}; use http::{HeaderMap, HeaderValue, Uri}; @@ -2109,6 +2110,18 @@ async fn remote_add_preflight_info(site: &PeerSite) -> S3Result Option { query_pairs(uri).get("bootstrapToken").cloned() } -fn bootstrap_bucket_make_op_path(bucket: &SRBucketInfo) -> String { +/// Query for a peer `make-with-versioning` bucket op. `versioningEnabled` +/// always travels so the outbound query matches MinIO's site-replication +/// make-bucket wire contract: MinIO's own create-bucket hook sends +/// `versioningEnabled=true` on this op. RustFS's inbound handler +/// force-enables versioning either way. +fn make_with_versioning_bucket_op_path(bucket: &str, created_at: Option<&str>, lock_enabled: bool) -> String { let mut query = form_urlencoded::Serializer::new(String::new()); - query.append_pair("bucket", &bucket.bucket); - query.append_pair("operation", "make-with-versioning"); - if let Some(created_at) = bucket - .created_at - .and_then(|value| value.format(&time::format_description::well_known::Rfc3339).ok()) - { - query.append_pair("createdAt", &created_at); + query.append_pair("bucket", bucket); + query.append_pair("operation", SITE_REPLICATION_BUCKET_OP_MAKE_WITH_VERSIONING); + query.append_pair("versioningEnabled", "true"); + if let Some(created_at) = created_at { + query.append_pair("createdAt", created_at); } - if bucket.object_lock_config.is_some() { + if lock_enabled { query.append_pair("lockEnabled", "true"); } - format!("/rustfs/admin/v3/site-replication/peer/bucket-ops?{}", query.finish()) + format!("{SITE_REPLICATION_PEER_BUCKET_OPS_PATH}?{}", query.finish()) +} + +fn bootstrap_bucket_make_op_path(bucket: &SRBucketInfo) -> String { + let created_at = bucket + .created_at + .and_then(|value| value.format(&time::format_description::well_known::Rfc3339).ok()); + make_with_versioning_bucket_op_path(&bucket.bucket, created_at.as_deref(), bucket.object_lock_config.is_some()) } fn bootstrap_bucket_meta_item(bucket: &SRBucketInfo, item_type: &str, updated_at: Option) -> SRBucketMeta { @@ -4280,16 +4303,7 @@ async fn broadcast_site_replication_make_bucket( .format(&time::format_description::well_known::Rfc3339) .unwrap_or_default(); - let path = { - let mut query = form_urlencoded::Serializer::new(String::new()); - query.append_pair("bucket", bucket); - query.append_pair("operation", "make-with-versioning"); - query.append_pair("createdAt", &created_at); - if lock_enabled { - query.append_pair("lockEnabled", "true"); - } - format!("/rustfs/admin/v3/site-replication/peer/bucket-ops?{}", query.finish()) - }; + let path = make_with_versioning_bucket_op_path(bucket, Some(&created_at), lock_enabled); let path = if let Some(token) = bootstrap_token { with_site_replication_bootstrap_token(&path, token) } else { @@ -10233,13 +10247,25 @@ impl Operation for SiteReplicationStatusHandler { } } +/// `POST /v3/site-replication/devnull` — peer link-check upload drain. +/// MinIO streams multi-megabyte probe bodies here during site netperf link +/// checks and expects an unbounded discard (its handler copies to io.Discard); +/// buffering through the 1MB admin body cap turned any larger probe into a +/// 400 and a false link failure. Stream and discard instead — no size cap. +async fn drain_site_replication_devnull(mut input: Body) -> S3Result<()> { + while let Some(chunk) = input.next().await { + chunk.map_err(|e| s3_error!(InvalidRequest, "failed to read devnull stream: {}", e))?; + } + Ok(()) +} + pub struct SiteReplicationDevNullHandler {} #[async_trait::async_trait] impl Operation for SiteReplicationDevNullHandler { async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { validate_site_replication_admin_request(&req, AdminAction::SiteReplicationOperationAction).await?; - let _ = read_plain_admin_body(req.input).await?; + drain_site_replication_devnull(req.input).await?; Ok(empty_response(StatusCode::NO_CONTENT)) } } @@ -10498,6 +10524,19 @@ impl Operation for SRPeerJoinHandler { } } +/// Outcome of a peer-driven `purge-deleted-bucket` replay. A bucket that is +/// already gone means the purge raced an earlier replay or a local delete — +/// that is success — but any other failure must reach the sender like the +/// sibling delete branches do: swallowing it answered 200 while the bucket +/// survived on this site. +fn purge_deleted_bucket_result(result: Result<(), StorageError>) -> S3Result<()> { + match result { + Ok(()) => Ok(()), + Err(err) if is_err_bucket_not_found(&err) => Ok(()), + Err(err) => Err(ApiError::from(err).into()), + } +} + pub struct SRPeerBucketOpsHandler {} #[async_trait::async_trait] @@ -10597,16 +10636,18 @@ impl Operation for SRPeerBucketOpsHandler { .map_err(ApiError::from)?; } "purge-deleted-bucket" => { - let _ = store - .delete_bucket( - &bucket, - &DeleteBucketOptions { - force: true, - srdelete_op: SRBucketDeleteOp::Purge, - ..Default::default() - }, - ) - .await; + purge_deleted_bucket_result( + store + .delete_bucket( + &bucket, + &DeleteBucketOptions { + force: true, + srdelete_op: SRBucketDeleteOp::Purge, + ..Default::default() + }, + ) + .await, + )?; } _ => return Err(s3_error!(InvalidRequest, "unsupported site replication bucket operation")), } @@ -13952,6 +13993,54 @@ mod tests { assert!(!query_flag(&uri, "missing")); } + /// A5 red-light: a `purge-deleted-bucket` replay must report success when + /// the bucket is already gone, and must propagate every other failure — + /// the swallowed error answered 200 while the bucket survived. + #[test] + fn test_purge_deleted_bucket_result_tolerates_only_missing_bucket() { + assert!(purge_deleted_bucket_result(Ok(())).is_ok()); + assert!(purge_deleted_bucket_result(Err(StorageError::BucketNotFound("photos".to_string()))).is_ok()); + assert!(purge_deleted_bucket_result(Err(StorageError::VolumeNotFound)).is_ok()); + let err = purge_deleted_bucket_result(Err(StorageError::StorageFull)) + .expect_err("non-not-found delete failures must propagate"); + assert_ne!(*err.code(), S3ErrorCode::NoSuchBucket); + } + + /// C5 red-light: the site-replication devnull drain must accept bodies + /// beyond the 1MB admin body cap — MinIO's link check streams large + /// probe bodies and treats a 400 as a broken link. + #[tokio::test] + async fn test_site_replication_devnull_drains_body_beyond_admin_cap() { + let body = Body::from(vec![0u8; MAX_ADMIN_REQUEST_BODY_SIZE + 1]); + drain_site_replication_devnull(body) + .await + .expect("devnull must drain bodies larger than the admin body cap"); + } + + /// A3 red-light: `versioningEnabled` must travel on every outbound + /// make-with-versioning bucket op so the query matches MinIO's + /// site-replication make-bucket wire contract (MinIO's own hook sends + /// `versioningEnabled=true` on this op). + #[test] + fn test_make_with_versioning_op_paths_send_versioning_enabled() { + let bucket = SRBucketInfo { + bucket: "photos".to_string(), + created_at: Some(OffsetDateTime::UNIX_EPOCH), + object_lock_config: Some(BASE64_STANDARD.encode("")), + ..Default::default() + }; + let bootstrap = bootstrap_bucket_make_op_path(&bucket); + assert!(bootstrap.contains("operation=make-with-versioning"), "{bootstrap}"); + assert!(bootstrap.contains("versioningEnabled=true"), "{bootstrap}"); + assert!(bootstrap.contains("createdAt="), "{bootstrap}"); + assert!(bootstrap.contains("lockEnabled=true"), "{bootstrap}"); + + // The broadcast path (create-bucket hook) shares the same builder. + let broadcast = make_with_versioning_bucket_op_path("photos", Some("1970-01-01T00:00:00Z"), false); + assert!(broadcast.contains("versioningEnabled=true"), "{broadcast}"); + assert!(!broadcast.contains("lockEnabled"), "{broadcast}"); + } + #[tokio::test] #[serial] async fn test_add_bootstrap_scope_only_allows_expected_bucket_setup_until_guard_drops() { diff --git a/rustfs/src/admin/handlers/system.rs b/rustfs/src/admin/handlers/system.rs index 9df481c11..981baaefb 100644 --- a/rustfs/src/admin/handlers/system.rs +++ b/rustfs/src/admin/handlers/system.rs @@ -1262,7 +1262,9 @@ mod tests { assert_eq!(response.summary.manual_transition_jobs.state, CapabilityState::Supported); assert_eq!(response.replication.contract_version, 1); assert_eq!(response.replication.bucket_replication.contract_version, 1); - assert_eq!(response.replication.remote_targets.contract_version, 1); + // v2: disableProxy moved from unsupported to writable (per-target + // read-proxy opt-out reached the admin API). + assert_eq!(response.replication.remote_targets.contract_version, 2); assert_eq!(response.replication.bucket_replication.status.state, CapabilityState::Supported); assert_eq!(response.replication.remote_targets.status.state, CapabilityState::Supported); assert_eq!( @@ -1293,7 +1295,15 @@ mod tests { .remote_targets .fields .iter() - .any(|field| field.name == "disableProxy" && field.state == super::ReplicationFieldState::Unsupported) + .any(|field| field.name == "disableProxy" && field.state == super::ReplicationFieldState::Supported) + ); + assert!( + response + .replication + .remote_targets + .fields + .iter() + .any(|field| field.name == "edge" && field.state == super::ReplicationFieldState::Unsupported) ); assert!( response @@ -1364,7 +1374,7 @@ mod tests { assert_eq!(value["summary"]["manual_transition_jobs"]["state"], "supported"); assert_eq!(value["replication"]["contract_version"], 1); assert_eq!(value["replication"]["bucket_replication"]["contract_version"], 1); - assert_eq!(value["replication"]["remote_targets"]["contract_version"], 1); + assert_eq!(value["replication"]["remote_targets"]["contract_version"], 2); assert_eq!(value["replication"]["bucket_replication"]["status"]["state"], "supported"); assert_eq!(value["replication"]["remote_targets"]["status"]["state"], "supported"); assert_eq!( @@ -1383,7 +1393,14 @@ mod tests { .as_array() .expect("remote target fields should be an array") .iter() - .any(|field| field["name"] == "disableProxy" && field["state"] == "unsupported") + .any(|field| field["name"] == "disableProxy" && field["state"] == "supported") + ); + assert!( + value["replication"]["remote_targets"]["fields"] + .as_array() + .expect("remote target fields should be an array") + .iter() + .any(|field| field["name"] == "edge" && field["state"] == "unsupported") ); assert!( value["replication"]["remote_targets"]["fields"] diff --git a/rustfs/src/admin/route_policy.rs b/rustfs/src/admin/route_policy.rs index ccb013211..e423c0b0e 100644 --- a/rustfs/src/admin/route_policy.rs +++ b/rustfs/src/admin/route_policy.rs @@ -428,6 +428,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[ admin(HttpMethod::Get, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High), admin(HttpMethod::Put, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High), admin(HttpMethod::Get, "/rustfs/admin/v3/scanner/status", SERVER_INFO, RouteRiskLevel::Sensitive), + admin( + HttpMethod::Post, + "/rustfs/admin/v3/scanner/cycle-state/reset", + CONFIG_UPDATE, + RouteRiskLevel::High, + ), admin( HttpMethod::Get, "/rustfs/admin/v3/ilm/expiry/status", @@ -2020,6 +2026,12 @@ mod tests { assert_not_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/expiry/status", SET_TIER); } + #[test] + fn route_policy_requires_config_update_for_scanner_cycle_reset() { + assert_action(HttpMethod::Post, "/rustfs/admin/v3/scanner/cycle-state/reset", CONFIG_UPDATE); + assert_not_action(HttpMethod::Post, "/rustfs/admin/v3/scanner/cycle-state/reset", SERVER_INFO); + } + #[test] fn route_policy_uses_tier_actions_for_transition_routes() { assert_action(HttpMethod::Post, "/rustfs/admin/v3/ilm/transition/run", SET_TIER); diff --git a/rustfs/src/admin/route_registration_test.rs b/rustfs/src/admin/route_registration_test.rs index e48e09c94..8dcb5d901 100644 --- a/rustfs/src/admin/route_registration_test.rs +++ b/rustfs/src/admin/route_registration_test.rs @@ -243,6 +243,7 @@ fn expected_admin_route_matrix() -> Vec { admin_route(Method::GET, "/v3/config"), admin_route(Method::PUT, "/v3/config"), admin_route(Method::GET, "/v3/scanner/status"), + admin_route(Method::POST, "/v3/scanner/cycle-state/reset"), admin_route(Method::GET, "/v3/audit/target/list"), admin_route_sample( Method::PUT, @@ -879,6 +880,7 @@ fn test_register_routes_cover_representative_admin_paths() { assert_route(&router, Method::GET, &admin_path("/v3/config")); assert_route(&router, Method::PUT, &admin_path("/v3/config")); assert_route(&router, Method::GET, &admin_path("/v3/scanner/status")); + assert_route(&router, Method::POST, &admin_path("/v3/scanner/cycle-state/reset")); assert_route(&router, Method::GET, &admin_path("/v3/ilm/expiry/status")); assert_route(&router, Method::POST, &admin_path("/v3/ilm/transition/run")); assert_route( @@ -1367,6 +1369,7 @@ fn test_admin_alias_paths_match_existing_admin_routes() { (Method::GET, compat_admin_alias_path("/v3/config")), (Method::PUT, compat_admin_alias_path("/v3/config")), (Method::GET, compat_admin_alias_path("/v3/scanner/status")), + (Method::POST, compat_admin_alias_path("/v3/scanner/cycle-state/reset")), (Method::GET, compat_admin_alias_path("/v3/ilm/expiry/status")), ] { assert!( diff --git a/rustfs/src/admin/site_replication_identity.rs b/rustfs/src/admin/site_replication_identity.rs index dc6440b4d..24784160b 100644 --- a/rustfs/src/admin/site_replication_identity.rs +++ b/rustfs/src/admin/site_replication_identity.rs @@ -13,9 +13,9 @@ // limitations under the License. use rustfs_madmin::{PeerInfo, SyncStatus}; -use std::collections::{BTreeMap, hash_map::DefaultHasher}; -use std::hash::{Hash, Hasher}; +use std::collections::BTreeMap; use url::Url; +use uuid::Uuid; fn has_http_scheme(endpoint: &str) -> bool { endpoint.get(..7).is_some_and(|prefix| prefix.eq_ignore_ascii_case("http://")) @@ -66,10 +66,12 @@ pub fn site_identity_key(endpoint: &str) -> String { .unwrap_or_else(|| trimmed.to_ascii_lowercase()) } +/// Fallback deployment ID for a peer that reported none. UUIDv5 over the +/// canonical endpoint: the ID is persisted in site-replication state and +/// broadcast to peers, so it must be identical across Rust toolchains +/// (`DefaultHasher` is not) and across spellings of the same endpoint. pub fn deployment_id_for_endpoint(endpoint: &str) -> String { - let mut hasher = DefaultHasher::new(); - endpoint.hash(&mut hasher); - format!("{:016x}", hasher.finish()) + Uuid::new_v5(&Uuid::NAMESPACE_URL, canonical_endpoint(endpoint).as_bytes()).to_string() } pub fn same_identity_endpoint(left: &str, right: &str) -> bool { @@ -174,6 +176,23 @@ mod tests { } } + /// B8 red-light: the fallback deployment ID must be a toolchain-stable + /// UUIDv5 over the canonical endpoint — `DefaultHasher` output is not + /// guaranteed stable across Rust releases, yet the ID is persisted in + /// site-replication state and broadcast to peers. + #[test] + fn deployment_id_for_endpoint_is_stable_uuid_v5_over_canonical_endpoint() { + let endpoint = "https://node-a.example.com:9000"; + let id = deployment_id_for_endpoint(endpoint); + let parsed = uuid::Uuid::parse_str(&id).expect("fallback deployment ID must be a UUID"); + assert_eq!(parsed.get_version_num(), 5, "fallback deployment ID must be UUIDv5"); + // Deterministic for the same endpoint and for spelling variants that + // share a canonical form; distinct endpoints stay distinct. + assert_eq!(id, deployment_id_for_endpoint(endpoint)); + assert_eq!(id, deployment_id_for_endpoint(" HTTPS://Node-A.Example.Com:9000/ ")); + assert_ne!(id, deployment_id_for_endpoint("https://node-b.example.com:9000")); + } + #[test] fn canonical_endpoint_accepts_case_insensitive_scheme() { assert_eq!( diff --git a/rustfs/src/admin/storage_api.rs b/rustfs/src/admin/storage_api.rs index 318718b49..bdf28f15e 100644 --- a/rustfs/src/admin/storage_api.rs +++ b/rustfs/src/admin/storage_api.rs @@ -51,7 +51,7 @@ mod ecstore_disk { } mod ecstore_error { - pub(crate) use crate::storage::storage_api::ecstore_error::StorageError; + pub(crate) use crate::storage::storage_api::ecstore_error::{StorageError, is_err_bucket_not_found}; } #[allow(unused_imports)] @@ -919,6 +919,7 @@ pub(crate) mod contract { } pub(crate) mod error { + pub(crate) use super::ecstore_error::is_err_bucket_not_found; pub(crate) use super::{Error, StorageError}; } diff --git a/rustfs/src/allocator_reclaim.rs b/rustfs/src/allocator_reclaim.rs index 0c31390c3..eba1a8948 100644 --- a/rustfs/src/allocator_reclaim.rs +++ b/rustfs/src/allocator_reclaim.rs @@ -369,14 +369,8 @@ pub fn allocator_reclaim_controller_snapshot(ctx: &CancellationToken) -> Allocat } #[cfg(not(target_os = "windows"))] -#[allow(unsafe_code)] fn collect_allocator_memory(force: bool) -> Result<(), String> { - // SAFETY: `mi_collect` is provided by the active global allocator backend - // on this target family. It is explicitly intended to reclaim retained - // pages/segments and does not require additional invariants from the caller. - unsafe { - libmimalloc_sys::mi_collect(force); - } + rustfs_mimalloc::MiMalloc::collect(force); Ok(()) } diff --git a/rustfs/src/app/bucket_usecase.rs b/rustfs/src/app/bucket_usecase.rs index 7d55035e9..b51254b23 100644 --- a/rustfs/src/app/bucket_usecase.rs +++ b/rustfs/src/app/bucket_usecase.rs @@ -1021,7 +1021,7 @@ fn build_list_objects_v2_metadata_output( object: Object { key: Some(encode_list_objects_v2_value(&object.name, encoding_type)), last_modified: object.mod_time.map(Timestamp::from), - size: Some(object.get_actual_size().unwrap_or_default()), + size: Some(object.get_actual_size_or_physical()), e_tag: object.etag.clone().map(|etag| to_s3s_etag(&etag)), storage_class: Some(ObjectStorageClass::from( object diff --git a/rustfs/src/app/object_usecase.rs b/rustfs/src/app/object_usecase.rs index a31ec1d19..d1978f2db 100644 --- a/rustfs/src/app/object_usecase.rs +++ b/rustfs/src/app/object_usecase.rs @@ -3969,6 +3969,55 @@ fn delete_creates_delete_marker(opts: &ObjectOptions) -> bool { opts.version_id.is_none() && opts.versioned && !opts.version_suspended } +fn delete_removes_current_object(opts: &ObjectOptions) -> bool { + delete_request_targets_current( + opts.version_id + .as_deref() + .and_then(|version_id| Uuid::parse_str(version_id).ok()), + ) +} + +fn delete_request_targets_current(version_id: Option) -> bool { + version_id.is_none() || version_id.is_some_and(|version_id| version_id.is_nil()) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum DeleteMemoryUpdate { + DeleteMarker, + Object { size: u64, removed_current_object: bool }, +} + +fn delete_memory_update( + creates_delete_marker: bool, + committed_delete_marker: bool, + requested_current: bool, + accounting_size: Option, + removed_current_object: bool, +) -> Option { + if creates_delete_marker || (committed_delete_marker && requested_current) { + return Some(DeleteMemoryUpdate::DeleteMarker); + } + + (!committed_delete_marker) + .then_some(accounting_size) + .flatten() + .map(|size| DeleteMemoryUpdate::Object { + size, + removed_current_object, + }) +} + +async fn apply_delete_memory_update(bucket: &str, update: Option) { + match update { + Some(DeleteMemoryUpdate::DeleteMarker) => record_bucket_delete_marker_memory(bucket).await, + Some(DeleteMemoryUpdate::Object { + size, + removed_current_object, + }) => record_bucket_object_delete_memory(bucket, size, removed_current_object).await, + None => {} + } +} + /// `DeleteObjects` is idempotent. A raw filesystem `NotFound` can cross the /// distributed delete path instead of its usual typed missing-object error. fn is_delete_objects_not_found(error: &EcstoreError) -> bool { @@ -8409,8 +8458,6 @@ impl DefaultObjectUsecase { object: ObjectToDelete, versioned: bool, version_suspended: bool, - size: i64, - existing: Option, } // Phase 2 (bounded concurrency, backlog#929 / HP-8): collect the @@ -8428,32 +8475,23 @@ impl DefaultObjectUsecase { skip_stat, } = prepared; let synthetic_version_id = object.version_id.is_none() && is_dir_object(&object.object_name); - let (goi, source_missing) = if skip_stat { - (ObjectInfo::default(), false) - } else { + if !skip_stat { match store_ref.get_object_info(bucket_ref, &object.object_name, &opts).await { - Ok(res) => (res, false), - Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => { - (ObjectInfo::default(), true) - } + Ok(_) => {} + Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {} Err(err) => return Err(ApiError::from(err)), } - }; - - let size = goi.size; + } if synthetic_version_id { object.version_id = Some(Uuid::nil()); } - let existing = (!skip_stat && !source_missing).then_some(goi); Ok::<_, ApiError>(AdmittedDelete { idx, object, versioned: opts.versioned, version_suspended: opts.version_suspended, - size, - existing, }) })) .buffered(DELETE_OBJECTS_PRE_STAT_CONCURRENCY) @@ -8464,15 +8502,11 @@ impl DefaultObjectUsecase { // per-key success/failure reporting is unchanged. let mut object_to_delete = Vec::new(); let mut object_to_delete_idx = Vec::new(); - let mut object_sizes = Vec::new(); - let mut existing_object_infos = Vec::new(); let mut object_versioning = Vec::new(); for admitted in admitted_deletes { - object_sizes.push(admitted.size); object_to_delete_idx.push(admitted.idx); object_versioning.push((admitted.versioned, admitted.version_suspended)); object_to_delete.push(admitted.object); - existing_object_infos.push(admitted.existing); } let cache_adapter = self.object_data_cache(); let cache_keys_before_delete = object_to_delete @@ -8489,8 +8523,8 @@ impl DefaultObjectUsecase { ..Default::default() }; apply_bucket_generation_guard(&req, &bucket, &mut storage_delete_opts)?; - let (dobjs, errs) = store - .delete_objects_with_tier_delete_journal(&bucket, object_to_delete.clone(), storage_delete_opts) + let (dobjs, errs, accounting) = store + .delete_objects_with_tier_delete_journal_and_accounting(&bucket, object_to_delete.clone(), storage_delete_opts) .await; let _manager = get_concurrency_manager(); @@ -8515,17 +8549,16 @@ impl DefaultObjectUsecase { delete_results[didx].delete_object = Some(deleted_object.clone()); let (versioned, version_suspended) = object_versioning[i]; let creates_delete_marker = object_to_delete[i].version_id.is_none() && versioned && !version_suspended; - if creates_delete_marker { - record_bucket_delete_marker_memory(&bucket).await; - } else { - let size = object_sizes[i].max(0) as u64; - record_bucket_object_delete_memory( - &bucket, - size, - existing_object_infos[i].is_some() && object_to_delete[i].version_id.is_none(), - ) - .await; - } + let committed_delete_marker = dobjs[i].delete_marker; + let delete_accounting = accounting.get(i).and_then(Option::as_ref); + let update = delete_memory_update( + creates_delete_marker, + committed_delete_marker, + delete_request_targets_current(object_to_delete[i].version_id), + delete_accounting.and_then(|value| value.size), + delete_accounting.is_some_and(|value| value.removed_current_object), + ); + apply_delete_memory_update(&bucket, update).await; } Err(error) => { delete_results[didx].error = Some(error); @@ -8803,12 +8836,24 @@ impl DefaultObjectUsecase { let _ = invalidate_object_data_cache_after_delete_success(&cache_adapter, &bucket, &key).await; } - // Fast in-memory update for immediate quota and admin usage consistency - if delete_creates_delete_marker(&opts) { - record_bucket_delete_marker_memory(&bucket).await; + // Fast in-memory update for immediate quota and admin usage consistency. + // Prefix/force deletes and synthetic directory entries do not carry one + // committed object identity; leave their cache delta to reconciliation. + let update = if force_delete || obj_info.name.is_empty() || synthetic_version_id { + None } else { - record_bucket_object_delete_memory(&bucket, obj_info.size.max(0) as u64, opts.version_id.is_none()).await; - } + // The storage commit returns this object's metadata while its + // generation lock is held. Never fall back to a pre-delete stat: + // an overwrite can commit between that stat and this delete. + delete_memory_update( + delete_creates_delete_marker(&opts), + obj_info.delete_marker, + opts.version_id.is_none(), + quota_object_size(&obj_info).ok(), + delete_removes_current_object(&opts), + ) + }; + apply_delete_memory_update(&bucket, update).await; if obj_info.name.is_empty() { if let Some((operation_id, target_arns, generation)) = force_delete_intent { @@ -17861,6 +17906,158 @@ mod tests { assert!(!can_skip_delete_objects_pre_stat(false, &delete_marker_creating_opts(), false)); } + #[test] + fn delete_accounting_recognizes_explicit_null_as_current_object() { + let opts = ObjectOptions { + version_id: Some(Uuid::nil().to_string()), + version_suspended: true, + ..Default::default() + }; + assert!(delete_removes_current_object(&opts)); + assert!(delete_request_targets_current(Some(Uuid::nil()))); + assert!(!delete_request_targets_current(Some(Uuid::new_v4()))); + assert!(!delete_removes_current_object(&ObjectOptions { + version_id: Some(Uuid::new_v4().to_string()), + ..Default::default() + })); + } + + #[test] + fn compressed_object_delete_restores_usage_baseline() { + let mut metadata = HashMap::new(); + insert_str(&mut metadata, SUFFIX_COMPRESSION, "klauspost/compress/s2".to_string()); + let object = ObjectInfo { + size: 400, + actual_size: 1000, + user_defined: Arc::new(metadata), + ..Default::default() + }; + let accounting_size = quota_object_size(&object).expect("logical compressed size should be canonical"); + + assert_eq!( + delete_memory_update(false, false, true, Some(accounting_size), true), + Some(DeleteMemoryUpdate::Object { + size: 1000, + removed_current_object: true, + }) + ); + } + + #[test] + fn invalid_accounting_metadata_is_reconciled_without_overflow() { + assert_eq!(delete_memory_update(false, false, true, None, true), None); + assert_eq!( + delete_memory_update(false, true, true, None, true), + Some(DeleteMemoryUpdate::DeleteMarker) + ); + } + + #[tokio::test] + #[serial_test::serial] + async fn compressed_delete_requests_restore_usage_baseline() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions}; + + let store = crate::app::gating_test_env::shared_gating_ecstore().await; + if current_app_context().is_none() { + crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await; + } + let bucket = format!("compressed-delete-request-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create compressed delete request bucket"); + + // Seed the process-local usage with the canonical logical bytes. The + // direct storage PUT below intentionally does not apply an app-layer + // usage delta; the two real DELETE requests must remove exactly this + // amount through their request-layer wiring. + crate::app::storage_api::test::data_usage::seed_bucket_usage_memory_for_test(&bucket, 2_000).await; + + for object in ["single", "batch"] { + let mut metadata = HashMap::new(); + insert_str(&mut metadata, SUFFIX_COMPRESSION, "klauspost/compress/s2".to_string()); + insert_str(&mut metadata, SUFFIX_ACTUAL_SIZE, "1000".to_string()); + let reader = HashReader::from_stream(std::io::Cursor::new(vec![0x5a; 400]), 400, 1000, None, None, false) + .expect("compressed fixture reader should be valid"); + let mut reader = PutObjReader::new(reader); + store + .put_object( + &bucket, + object, + &mut reader, + &ObjectOptions { + user_defined: metadata, + ..Default::default() + }, + ) + .await + .expect("compressed fixture object should be written"); + } + + let mut single_req = build_request( + DeleteObjectInput::builder() + .bucket(bucket.clone()) + .key("single".to_string()) + .build() + .expect("single delete input should build"), + Method::DELETE, + ); + single_req.extensions.insert(crate::storage::access::ReqInfo { + cred: Some(rustfs_credentials::Credentials::default()), + is_owner: true, + ..Default::default() + }); + DefaultObjectUsecase::from_global() + .execute_delete_object(single_req) + .await + .expect("single compressed delete should succeed"); + assert_eq!( + crate::app::storage_api::test::data_usage::get_bucket_usage_memory(&bucket).await, + Some(1_000), + "single delete must subtract the logical accounting size" + ); + + let mut batch_req = build_request( + DeleteObjectsInput::builder() + .bucket(bucket.clone()) + .delete(Delete { + objects: vec![ObjectIdentifier { + key: "batch".to_string(), + ..Default::default() + }], + quiet: None, + }) + .build() + .expect("batch delete input should build"), + Method::POST, + ); + batch_req.extensions.insert(crate::storage::access::ReqInfo { + cred: Some(rustfs_credentials::Credentials::default()), + is_owner: true, + ..Default::default() + }); + DefaultObjectUsecase::from_global() + .execute_delete_objects(batch_req) + .await + .expect("batch compressed delete should succeed"); + assert_eq!( + crate::app::storage_api::test::data_usage::get_bucket_usage_memory(&bucket).await, + Some(0), + "batch delete must subtract the committed logical accounting size" + ); + + store + .delete_bucket( + &bucket, + &DeleteBucketOptions { + force: true, + ..Default::default() + }, + ) + .await + .expect("clean up compressed delete request bucket"); + } + #[tokio::test] async fn execute_get_object_attributes_returns_internal_error_when_store_uninitialized() { let input = GetObjectAttributesInput::builder() diff --git a/rustfs/src/app/storage_api.rs b/rustfs/src/app/storage_api.rs index d90cfb6a8..a599cfde6 100644 --- a/rustfs/src/app/storage_api.rs +++ b/rustfs/src/app/storage_api.rs @@ -72,6 +72,11 @@ pub(crate) mod data_usage { compute_bucket_usage, live_bucket_usage_computations, seed_bucket_usage_memory_for_test, store_data_usage_in_backend, }; + #[cfg(test)] + pub(crate) async fn get_bucket_usage_memory(bucket: &str) -> Option { + crate::storage::storage_api::ecstore_data_usage::get_bucket_usage_memory(bucket).await + } + pub(crate) async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) { crate::storage::storage_api::ecstore_data_usage::record_bucket_object_delete_memory( bucket, @@ -1233,7 +1238,10 @@ pub(crate) mod test { pub(crate) use super::access::ReqInfo; pub(crate) use super::options::VERSIONING_CONFIG_LOOKUPS; - pub(crate) use super::{bucket, data_usage, ecfs, object_utils, runtime}; + pub(crate) use super::{bucket, ecfs, object_utils, runtime}; + pub(crate) mod data_usage { + pub(crate) use super::super::data_usage::*; + } pub(crate) use crate::storage::storage_api::test_consumer::{get_global_bucket_metadata_sys, set_bucket_metadata}; pub(crate) use crate::storage::storage_api::{ ECStore, Endpoint, Endpoints, PoolEndpoints, StorageObjectInfo, StorageObjectOptions, StoragePutObjReader, diff --git a/rustfs/src/bin/rustfs-cli.rs b/rustfs/src/bin/rustfs-cli.rs index 61c8bc795..49d04db72 100644 --- a/rustfs/src/bin/rustfs-cli.rs +++ b/rustfs/src/bin/rustfs-cli.rs @@ -17,6 +17,220 @@ //! This binary shares RustFS's existing subcommand dispatcher and provides the //! documented entry point for offline tooling such as `inspect bucket-meta`. -fn main() { +use std::fs; +use std::io::{Read as _, Write as _}; +use std::path::{Path, PathBuf}; +use std::process::ExitCode; +use std::time::{SystemTime, UNIX_EPOCH}; + +use rustfs::connect::offline::{OfflineEnrollment, OfflineKeyStore}; + +/// Owner read/write only. The response names the key being enrolled and the +/// challenge it answers; neither belongs to anyone else on the machine. +#[cfg(unix)] +const RESPONSE_MODE: u32 = 0o600; + +const USAGE: &str = "\ +Usage: rustfs-cli connect offline enroll --challenge --output [--key-dir ] + +Answers a Connect offline enrolment challenge without a network. Reads the +challenge from a file or from stdin when the path is `-`, verifies it against the +enrolment root compiled into this binary, mints the key being enrolled on first +use, and writes the signed response. + +No secret is ever accepted on the command line. +"; + +fn main() -> ExitCode { + let arguments: Vec = std::env::args().skip(1).collect(); + + // Offline enrolment is handled before the server dispatcher is reached, and + // the reason is the surface's whole point: `run_process` builds a Tokio + // runtime and enters the server's async main. An air-gapped enrolment must + // not start a runtime, a task, or anything that could open a socket, so the + // two paths cannot share an entry. + if matches!( + arguments.first().map(String::as_str), + Some("connect") if matches!(arguments.get(1).map(String::as_str), Some("offline")) + ) { + return match run_offline(&arguments[2..]) { + Ok(()) => ExitCode::SUCCESS, + Err(message) => { + eprintln!("rustfs-cli: {message}"); + ExitCode::FAILURE + } + }; + } + rustfs::startup_entrypoint::run_process(); + + ExitCode::SUCCESS +} + +fn run_offline(arguments: &[String]) -> Result<(), String> { + match arguments.first().map(String::as_str) { + Some("enroll") => enroll(&arguments[1..]), + Some(other) => Err(format!("unknown offline subcommand `{other}`\n\n{USAGE}")), + None => Err(format!("missing offline subcommand\n\n{USAGE}")), + } +} + +fn enroll(arguments: &[String]) -> Result<(), String> { + let mut challenge_path: Option = None; + let mut output_path: Option = None; + let mut key_directory: Option = None; + + let mut index = 0; + while index < arguments.len() { + let flag = arguments[index].as_str(); + let take_value = |name: &str| -> Result { + arguments + .get(index + 1) + .cloned() + .ok_or_else(|| format!("`{name}` needs a value\n\n{USAGE}")) + }; + + match flag { + "--challenge" => challenge_path = Some(take_value("--challenge")?), + "--output" => output_path = Some(take_value("--output")?), + "--key-dir" => key_directory = Some(take_value("--key-dir")?), + "-h" | "--help" => { + println!("{USAGE}"); + return Ok(()); + } + other => return Err(format!("unknown option `{other}`\n\n{USAGE}")), + } + + index += 2; + } + + let challenge_path = challenge_path.ok_or_else(|| format!("`--challenge` is required\n\n{USAGE}"))?; + let output_path = output_path.ok_or_else(|| format!("`--output` is required\n\n{USAGE}"))?; + let key_directory = key_directory.unwrap_or_else(|| ".".to_string()); + + let challenge = read_challenge(&challenge_path)?; + + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| "the system clock is before the Unix epoch".to_string())? + .as_secs() as i64; + + let verified = OfflineEnrollment::verify_challenge(&challenge, now).map_err(|error| error.to_string())?; + + // First use mints the key; a retry answers with the one already enrolled, + // because the operator may already be carrying a response naming it. + let key = OfflineKeyStore::new(&key_directory) + .load_or_create() + .map_err(|error| error.to_string())?; + + let mut device_nonce = [0u8; 32]; + getrandom(&mut device_nonce)?; + + let response = OfflineEnrollment::build_response(&verified, &key, &device_nonce, now).map_err(|error| error.to_string())?; + + write_response(Path::new(&output_path), &response)?; + + Ok(()) +} + +/// Reads the challenge from a file, or from stdin when the path is `-`. +/// +/// A challenge is not a secret — it is signed, public, and carried in by hand — +/// so accepting a path is safe. The response's key never arrives this way. +fn read_challenge(path: &str) -> Result, String> { + if path == "-" { + let mut buffer = Vec::new(); + std::io::stdin() + .read_to_end(&mut buffer) + .map_err(|error| format!("cannot read the challenge from stdin: {error}"))?; + return Ok(buffer); + } + + fs::read(path).map_err(|error| format!("cannot read the challenge at {path}: {error}")) +} + +/// Writes the response durably and atomically at mode 0600. +/// +/// Not the no-clobber publish `IdentityStore` performs for a key: an operator +/// who reruns an enrolment expects the response file to be replaced, whereas a +/// second key would strand the first. Same durability, deliberately different +/// publication rule. +fn write_response(path: &Path, response: &[u8]) -> Result<(), String> { + let parent = path.parent().filter(|parent| !parent.as_os_str().is_empty()); + let temporary: PathBuf = match parent { + Some(parent) => parent.join(format!(".{}.tmp", file_name(path))), + None => PathBuf::from(format!(".{}.tmp", file_name(path))), + }; + + let mut options = fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(RESPONSE_MODE); + } + + let write = (|| -> std::io::Result<()> { + let mut file = options.open(&temporary)?; + file.write_all(response)?; + + // The umask can only narrow the creation mode, so set the exact mode + // before the bytes become durable. + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(RESPONSE_MODE))?; + } + + file.sync_all() + })(); + + if let Err(error) = write { + let _ = fs::remove_file(&temporary); + return Err(format!("cannot write the response to {}: {error}", path.display())); + } + + fs::rename(&temporary, path).map_err(|error| { + let _ = fs::remove_file(&temporary); + format!("cannot publish the response at {}: {error}", path.display()) + })?; + + if let Some(parent) = parent { + sync_directory(parent); + } + + Ok(()) +} + +fn file_name(path: &Path) -> String { + path.file_name() + .map(|name| name.to_string_lossy().into_owned()) + .unwrap_or_else(|| "response".to_string()) +} + +/// Fsync the directory so the renamed entry survives power loss. Directories +/// cannot be opened for syncing on Windows, where this is a no-op. +fn sync_directory(directory: &Path) { + #[cfg(unix)] + { + if let Ok(handle) = fs::File::open(directory) { + let _ = handle.sync_all(); + } + } + #[cfg(not(unix))] + let _ = directory; +} + +/// Fills `buffer` with operating-system randomness. +/// +/// The device nonce must be unpredictable: it is what stops a captured response +/// being replayed as a fresh one. Sourced through p256's pinned rand_core 0.6 +/// rather than the workspace `rand` 0.10, matching `identity.rs`; the two are +/// different crate versions and only the pinned one is on p256's own path. +fn getrandom(buffer: &mut [u8]) -> Result<(), String> { + use p256::elliptic_curve::rand_core::{OsRng, RngCore as _}; + + OsRng + .try_fill_bytes(buffer) + .map_err(|error| format!("the operating system random source failed: {error}")) } diff --git a/rustfs/src/connect/client.rs b/rustfs/src/connect/client.rs new file mode 100644 index 000000000..a13ec953d --- /dev/null +++ b/rustfs/src/connect/client.rs @@ -0,0 +1,566 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::time::Duration; + +use base64::Engine as _; +use reqwest::{Client, StatusCode, Url}; +use rustls::RootCertStore; +use rustls::pki_types::{CertificateDer, pem::PemObject as _}; +use serde::Deserialize; +use uuid::Uuid; +use zeroize::Zeroizing; + +use super::credential_store::{ + CompletedRegistration, CredentialStore, CredentialStoreError, DeviceCredential, PendingRegistration, PendingRotation, +}; +use super::identity::{IdentityError, RegistrationTranscript}; +use super::identity_store::{IdentityStore, StoreError}; +use super::registration::{ + CredentialResponse, CredentialValidationError, ExpectedDevice, RegistrationRequest, RegistrationToken, RotationRequest, + certificate_fingerprint, certificate_request_matches, public_key_fingerprint, validate_credential, + validate_stored_credential, +}; + +const MAX_ATTEMPTS: usize = 3; +const MAX_RESPONSE_BYTES: usize = 1024 * 1024; +const ROTATION_THRESHOLD_SECONDS: i64 = 8 * 60 * 60; + +pub struct ConnectConfig<'a> { + pub endpoint: &'a str, + pub root_ca_pem: &'a [u8], + pub timeout: Duration, +} + +pub struct ConnectClient { + endpoint: Url, + roots: RootCertStore, + root_certificates: Vec>, + client: Client, + timeout: Duration, +} + +impl ConnectClient { + pub fn from_optional_config(config: Option>) -> Result, ClientError> { + config.map(Self::new).transpose() + } + + pub fn new(config: ConnectConfig<'_>) -> Result { + let mut endpoint = Url::parse(config.endpoint).map_err(|_| ClientError::Endpoint)?; + if endpoint.scheme() != "https" + || endpoint.cannot_be_a_base() + || !endpoint.username().is_empty() + || endpoint.password().is_some() + || endpoint.query().is_some() + || endpoint.fragment().is_some() + { + return Err(ClientError::Endpoint); + } + if !endpoint.path().ends_with('/') { + let path = format!("{}/", endpoint.path()); + endpoint.set_path(&path); + } + + let root_certificates = CertificateDer::pem_slice_iter(config.root_ca_pem) + .collect::, _>>() + .map_err(|_| ClientError::RootCertificate)?; + if root_certificates.is_empty() { + return Err(ClientError::RootCertificate); + } + let mut roots = RootCertStore::empty(); + let (accepted, rejected) = roots.add_parsable_certificates(root_certificates.clone()); + if accepted != root_certificates.len() || rejected != 0 { + return Err(ClientError::RootCertificate); + } + + let client = build_client(&root_certificates, config.timeout, None)?; + Ok(Self { + endpoint, + roots, + root_certificates, + client, + timeout: config.timeout, + }) + } + + pub async fn register( + &self, + identity_store: &IdentityStore, + credential_store: &CredentialStore, + token: &RegistrationToken, + ) -> Result { + let _lock = credential_store.lock().await?; + if let Some((credential, _)) = self.load_valid_credential(identity_store, credential_store)? { + ensure_credential_time(&credential, unix_now())?; + return Ok(credential); + } + + let identity = identity_store.load_or_create()?; + let candidate = PendingRegistration { + token_uid: token.registration_token_uid.clone(), + request_id: Uuid::new_v4().to_string(), + certificate_request: identity.certificate_request_base64()?, + previous_credential_fingerprint: None, + next_public_key_sha256: None, + }; + let pending = credential_store.claim_pending_registration(&candidate)?; + if pending.token_uid != token.registration_token_uid + || pending.previous_credential_fingerprint.is_some() + || pending.next_public_key_sha256.is_some() + || !is_request_id(&pending.request_id) + || !certificate_request_matches(&pending.certificate_request, &identity)? + { + return Err(ClientError::PendingRegistration); + } + + let credential = match self.exchange_registration(token, &pending, &identity).await { + Ok(credential) => credential, + Err(error @ (ClientError::AccessRevoked { .. } | ClientError::Rejected { .. })) => { + credential_store.clear_pending_registration()?; + return Err(error); + } + Err(error) => return Err(error), + }; + credential_store.save(&credential)?; + credential_store.clear_pending_registration()?; + Ok(credential) + } + + pub async fn reenroll( + &self, + identity_store: &IdentityStore, + credential_store: &CredentialStore, + token: &RegistrationToken, + ) -> Result { + let _lock = credential_store.lock().await?; + let (credential, _) = self + .load_valid_credential(identity_store, credential_store)? + .ok_or(ClientError::NotRegistered)?; + let fingerprint = certificate_fingerprint(&credential.certificate)?; + if credential_store.load_completed_registration()?.is_some_and(|completed| { + completed.token_uid == token.registration_token_uid && completed.credential_fingerprint == fingerprint + }) { + return Ok(credential); + } + credential_store.clear_pending_rotation()?; + let next = identity_store.load_or_create_next()?; + let next_fingerprint = public_key_fingerprint(&next); + let candidate = PendingRegistration { + token_uid: token.registration_token_uid.clone(), + request_id: Uuid::new_v4().to_string(), + certificate_request: next.certificate_request_base64()?, + previous_credential_fingerprint: Some(fingerprint.clone()), + next_public_key_sha256: Some(next_fingerprint.clone()), + }; + let pending = credential_store.claim_pending_registration(&candidate)?; + if pending.token_uid != token.registration_token_uid + || pending.previous_credential_fingerprint.as_deref() != Some(&fingerprint) + || pending.next_public_key_sha256.as_deref() != Some(&next_fingerprint) + || !is_request_id(&pending.request_id) + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRegistration); + } + let enrolled = match self.exchange_registration(token, &pending, &next).await { + Ok(credential) => credential, + Err(error @ (ClientError::AccessRevoked { .. } | ClientError::Rejected { .. })) => { + credential_store.clear_pending_registration()?; + identity_store.clear_next()?; + return Err(error); + } + Err(error) => return Err(error), + }; + credential_store.save(&enrolled)?; + identity_store.commit_next(&next)?; + credential_store.save_completed_registration(&CompletedRegistration { + token_uid: token.registration_token_uid.clone(), + credential_fingerprint: certificate_fingerprint(&enrolled.certificate)?, + })?; + credential_store.clear_pending_registration()?; + Ok(enrolled) + } + + async fn exchange_registration( + &self, + token: &RegistrationToken, + pending: &PendingRegistration, + identity: &super::identity::DeviceIdentity, + ) -> Result { + let csr_der = base64::engine::general_purpose::STANDARD + .decode(&pending.certificate_request) + .map_err(|_| ClientError::PendingRegistration)?; + let transcript = RegistrationTranscript::build( + &token.registration_token_uid, + &token.organization_uid, + &token.cluster_uid, + &pending.request_id, + &token.challenge_nonce, + token.expires_unix, + &csr_der, + )?; + let proof = identity.sign_registration(&transcript); + let body = RegistrationRequest::new(token, &pending.request_id, &pending.certificate_request, &proof); + let url = self.url("./registrationTokens:exchange")?; + let response = self + .send(StatusCode::CREATED, || self.client.post(url.clone()).json(&body)) + .await?; + let cluster = format!("organizations/{}/clusters/{}", token.organization_uid, token.cluster_uid); + let credential = validate_credential( + response, + identity, + &self.roots, + &self.root_certificates, + ExpectedDevice::Registration { cluster: &cluster }, + )?; + Ok(credential) + } + + pub async fn rotate_if_due( + &self, + identity_store: &IdentityStore, + credential_store: &CredentialStore, + now_unix: i64, + ) -> Result, ClientError> { + let _lock = credential_store.lock().await?; + let (credential, identity) = self + .load_valid_credential(identity_store, credential_store)? + .ok_or(ClientError::NotRegistered)?; + if credential_store.load_pending_registration()?.is_some() { + return Err(ClientError::PendingRegistration); + } + ensure_credential_time(&credential, now_unix)?; + if credential.not_after_unix - now_unix > ROTATION_THRESHOLD_SECONDS { + return Ok(None); + } + let fingerprint = certificate_fingerprint(&credential.certificate)?; + let next = identity_store.load_or_create_next()?; + let candidate = PendingRotation { + credential_fingerprint: fingerprint.clone(), + device_name: credential.name.clone(), + request_id: Uuid::new_v4().to_string(), + certificate_request: next.certificate_request_base64()?, + next_public_key_sha256: public_key_fingerprint(&next), + }; + let pending = credential_store.claim_pending_rotation(&candidate)?; + if pending.credential_fingerprint != fingerprint + || pending.device_name != credential.name + || !is_request_id(&pending.request_id) + || pending.next_public_key_sha256 != public_key_fingerprint(&next) + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRotation); + } + let body = RotationRequest::new( + &identity, + &fingerprint, + &credential.name, + &pending.request_id, + &pending.certificate_request, + )?; + let private_key = identity.to_pkcs8_pem()?; + let mut identity_pem = Zeroizing::new(Vec::with_capacity(credential.certificate_chain.len() + private_key.len() + 1)); + identity_pem.extend_from_slice(credential.certificate_chain.as_bytes()); + identity_pem.push(b'\n'); + identity_pem.extend_from_slice(private_key.as_bytes()); + let tls_identity = reqwest::Identity::from_pem(&identity_pem).map_err(|_| ClientError::IdentityCertificate)?; + let client = build_client(&self.root_certificates, self.timeout, Some(tls_identity))?; + let path = format!("clusterDevices/{}:rotateCredential", credential.uid); + let url = self.url(&path)?; + let response = self.send(StatusCode::OK, || client.post(url.clone()).json(&body)).await?; + let rotated = validate_credential( + response, + &next, + &self.roots, + &self.root_certificates, + ExpectedDevice::Rotation { name: &credential.name }, + )?; + if rotated.name != credential.name || rotated.uid != credential.uid { + return Err(ClientError::Credential(CredentialValidationError::Identity)); + } + credential_store.save(&rotated)?; + identity_store.commit_next(&next)?; + credential_store.clear_pending_rotation()?; + Ok(Some(rotated)) + } + + fn load_valid_credential( + &self, + identity_store: &IdentityStore, + credential_store: &CredentialStore, + ) -> Result, ClientError> { + let Some(credential) = credential_store.load()? else { + return Ok(None); + }; + let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?; + if let Some(pending) = credential_store.load_pending_registration()? { + let Some(previous) = pending.previous_credential_fingerprint.as_deref() else { + if pending.next_public_key_sha256.is_some() + || !is_request_id(&pending.request_id) + || !certificate_request_matches(&pending.certificate_request, ¤t)? + { + return Err(ClientError::PendingRegistration); + } + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + credential_store.clear_pending_registration()?; + return Ok(Some((credential, current))); + }; + let next_fingerprint = pending + .next_public_key_sha256 + .as_deref() + .ok_or(ClientError::PendingRegistration)?; + if !is_request_id(&pending.request_id) { + return Err(ClientError::PendingRegistration); + } + let fingerprint = certificate_fingerprint(&credential.certificate)?; + if fingerprint == previous { + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + let next = identity_store.load_next()?.ok_or(ClientError::PendingRegistration)?; + if public_key_fingerprint(&next) != next_fingerprint + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRegistration); + } + return Ok(Some((credential, current))); + } + if public_key_fingerprint(¤t) == next_fingerprint { + if !certificate_request_matches(&pending.certificate_request, ¤t)? { + return Err(ClientError::PendingRegistration); + } + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + } else { + let next = identity_store.load_next()?.ok_or(ClientError::PendingRegistration)?; + if public_key_fingerprint(&next) != next_fingerprint + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRegistration); + } + validate_stored_credential(&credential, &next, &self.roots, &self.root_certificates)?; + identity_store.commit_next(&next)?; + } + credential_store.save_completed_registration(&CompletedRegistration { + token_uid: pending.token_uid, + credential_fingerprint: fingerprint, + })?; + credential_store.clear_pending_registration()?; + let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?; + return Ok(Some((credential, current))); + } + let Some(pending) = credential_store.load_pending_rotation()? else { + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + return Ok(Some((credential, current))); + }; + let fingerprint = certificate_fingerprint(&credential.certificate)?; + if pending.device_name != credential.name || !is_request_id(&pending.request_id) { + return Err(ClientError::PendingRotation); + } + if fingerprint == pending.credential_fingerprint { + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + let next = identity_store.load_next()?.ok_or(ClientError::PendingRotation)?; + if pending.next_public_key_sha256 != public_key_fingerprint(&next) + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRotation); + } + return Ok(Some((credential, current))); + } + + if public_key_fingerprint(¤t) == pending.next_public_key_sha256 { + if !certificate_request_matches(&pending.certificate_request, ¤t)? { + return Err(ClientError::PendingRotation); + } + validate_stored_credential(&credential, ¤t, &self.roots, &self.root_certificates)?; + } else { + let next = identity_store.load_next()?.ok_or(ClientError::PendingRotation)?; + if public_key_fingerprint(&next) != pending.next_public_key_sha256 + || !certificate_request_matches(&pending.certificate_request, &next)? + { + return Err(ClientError::PendingRotation); + } + validate_stored_credential(&credential, &next, &self.roots, &self.root_certificates)?; + identity_store.commit_next(&next)?; + } + credential_store.clear_pending_rotation()?; + let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?; + Ok(Some((credential, current))) + } + + async fn send(&self, success: StatusCode, mut request: F) -> Result + where + F: FnMut() -> reqwest::RequestBuilder, + { + let mut last_status = None; + for attempt in 0..MAX_ATTEMPTS { + match request().send().await { + Ok(response) if response.status() == success => return decode_response(response).await, + Ok(response) if matches!(response.status(), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) => { + let status = response.status(); + let reason = decode_reason(response).await; + return Err(ClientError::AccessRevoked { status, reason }); + } + Ok(response) if matches!(response.status(), StatusCode::REQUEST_TIMEOUT | StatusCode::TOO_MANY_REQUESTS) => { + last_status = Some(response.status()); + } + Ok(response) if response.status().is_client_error() => { + let status = response.status(); + let reason = decode_reason(response).await; + return Err(ClientError::Rejected { status, reason }); + } + Ok(response) if response.status().is_server_error() => { + last_status = Some(response.status()); + } + Ok(response) => { + let status = response.status(); + let reason = decode_reason(response).await; + return Err(ClientError::Rejected { status, reason }); + } + Err(error) if !error.is_timeout() && !error.is_connect() => return Err(ClientError::Transport(error)), + Err(_) => {} + } + + if attempt + 1 < MAX_ATTEMPTS { + tokio::time::sleep(Duration::from_millis(50 * (attempt as u64 + 1))).await; + } + } + Err(ClientError::Unavailable { status: last_status }) + } + + fn url(&self, path: &str) -> Result { + self.endpoint.join(path).map_err(|_| ClientError::Endpoint) + } +} + +fn is_request_id(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| uuid.get_version() == Some(uuid::Version::Random) && uuid.to_string() == value) +} + +fn unix_now() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |duration| duration.as_secs() as i64) +} + +fn ensure_credential_time(credential: &DeviceCredential, now_unix: i64) -> Result<(), ClientError> { + if credential.not_before_unix > now_unix { + return Err(ClientError::CredentialNotYetValid); + } + if credential.not_after_unix <= now_unix { + return Err(ClientError::CredentialExpired); + } + Ok(()) +} + +fn build_client( + roots: &[CertificateDer<'static>], + timeout: Duration, + identity: Option, +) -> Result { + let certificates = roots + .iter() + .map(|root| reqwest::Certificate::from_der(root.as_ref())) + .collect::, _>>()?; + let mut builder = Client::builder() + .https_only(true) + .redirect(reqwest::redirect::Policy::none()) + .timeout(timeout) + .tls_certs_only(certificates); + if let Some(identity) = identity { + builder = builder.identity(identity); + } + builder.build().map_err(ClientError::Transport) +} + +async fn decode_response(mut response: reqwest::Response) -> Result { + let body = read_body(&mut response).await?; + serde_json::from_slice(&body).map_err(|_| ClientError::Response) +} + +async fn read_body(response: &mut reqwest::Response) -> Result, ClientError> { + let mut body = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(ClientError::Transport)? { + if body.len() + chunk.len() > MAX_RESPONSE_BYTES { + return Err(ClientError::ResponseTooLarge); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} + +#[derive(Deserialize)] +struct ErrorEnvelope { + #[serde(default)] + details: Vec, +} + +#[derive(Deserialize)] +struct ErrorDetail { + #[serde(default)] + reason: String, +} + +async fn decode_reason(mut response: reqwest::Response) -> Option { + let body = read_body(&mut response).await.ok()?; + serde_json::from_slice::(&body) + .ok()? + .details + .into_iter() + .find_map(|detail| (!detail.reason.is_empty()).then_some(detail.reason)) +} + +#[derive(Debug, thiserror::Error)] +pub enum ClientError { + #[error("Connect endpoint must be an HTTPS base URL without credentials, query, or fragment")] + Endpoint, + #[error("Connect root CA configuration is invalid")] + RootCertificate, + #[error( + "Connect registration has a pending attempt for a different token; restore the original protected token configuration" + )] + PendingRegistration, + #[error( + "Connect credential rotation has an unfinished attempt for a different current certificate; inspect the local credential store" + )] + PendingRotation, + #[error("RustFS is not registered with Connect")] + NotRegistered, + #[error("the Connect device private key is missing; restore device.key before using the stored certificate")] + IdentityMissing, + #[error("the Connect device certificate has expired; call ConnectClient::reenroll with a fresh registration token")] + CredentialExpired, + #[error("the Connect device certificate is not yet valid; fix local clock skew or call ConnectClient::reenroll")] + CredentialNotYetValid, + #[error("the stored Connect certificate and device private key cannot form a TLS identity")] + IdentityCertificate, + #[error( + "Connect rejected the device credential with HTTP {status}; reason={reason:?}; call ConnectClient::reenroll with a fresh registration token if revoked" + )] + AccessRevoked { status: StatusCode, reason: Option }, + #[error("Connect rejected the request with HTTP {status}; reason={reason:?}")] + Rejected { status: StatusCode, reason: Option }, + #[error("Connect remained unavailable after bounded retries; last_status={status:?}")] + Unavailable { status: Option }, + #[error("Connect response exceeded the 1 MiB credential-response limit")] + ResponseTooLarge, + #[error("Connect returned an invalid credential response")] + Response, + #[error(transparent)] + Transport(#[from] reqwest::Error), + #[error(transparent)] + Identity(#[from] IdentityError), + #[error(transparent)] + IdentityStore(#[from] StoreError), + #[error(transparent)] + CredentialStore(#[from] CredentialStoreError), + #[error(transparent)] + Credential(#[from] CredentialValidationError), +} diff --git a/rustfs/src/connect/config.rs b/rustfs/src/connect/config.rs new file mode 100644 index 000000000..391d4ef45 --- /dev/null +++ b/rustfs/src/connect/config.rs @@ -0,0 +1,173 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::ffi::OsString; +use std::fs; +use std::path::PathBuf; +use std::time::Duration; + +use super::{CredentialStore, IdentityStore}; + +pub const ENV_CONNECT_ENDPOINT: &str = "RUSTFS_CONNECT_ENDPOINT"; +pub const ENV_CONNECT_ROOT_CA_FILE: &str = "RUSTFS_CONNECT_ROOT_CA_FILE"; +pub const ENV_CONNECT_STATE_DIR: &str = "RUSTFS_CONNECT_STATE_DIR"; + +#[derive(Clone, Copy, Debug)] +pub struct HeartbeatSchedule { + pub cadence: Duration, + pub jitter: Duration, + pub timeout: Duration, + pub initial_backoff: Duration, + pub max_backoff: Duration, +} + +impl Default for HeartbeatSchedule { + fn default() -> Self { + Self { + cadence: Duration::from_secs(30), + jitter: Duration::from_secs(3), + timeout: Duration::from_secs(5), + initial_backoff: Duration::from_secs(1), + max_backoff: Duration::from_secs(5 * 60), + } + } +} + +#[derive(Clone, Debug)] +pub struct HeartbeatConfig { + pub endpoint: String, + pub root_ca_pem: Vec, + pub identity_store: IdentityStore, + pub credential_store: CredentialStore, + pub state_path: PathBuf, + pub schedule: HeartbeatSchedule, +} + +impl HeartbeatConfig { + pub fn new( + endpoint: impl Into, + root_ca_pem: impl Into>, + identity_store: IdentityStore, + credential_store: CredentialStore, + state_path: impl Into, + ) -> Self { + Self { + endpoint: endpoint.into(), + root_ca_pem: root_ca_pem.into(), + identity_store, + credential_store, + state_path: state_path.into(), + schedule: HeartbeatSchedule::default(), + } + } + + pub fn from_env() -> Result, HeartbeatConfigError> { + Self::from_env_values( + env::var_os(ENV_CONNECT_ENDPOINT), + env::var_os(ENV_CONNECT_ROOT_CA_FILE), + env::var_os(ENV_CONNECT_STATE_DIR), + ) + } + + fn from_env_values( + endpoint: Option, + root_ca_file: Option, + state_dir: Option, + ) -> Result, HeartbeatConfigError> { + let configured = endpoint.is_some() || root_ca_file.is_some() || state_dir.is_some(); + if !configured { + return Ok(None); + } + let (Some(endpoint), Some(root_ca_file), Some(state_dir)) = (endpoint, root_ca_file, state_dir) else { + return Err(HeartbeatConfigError::Partial); + }; + let endpoint = endpoint.into_string().map_err(|_| HeartbeatConfigError::EndpointEncoding)?; + let root_ca_file = PathBuf::from(root_ca_file); + let state_dir = PathBuf::from(state_dir); + if endpoint.is_empty() || root_ca_file.as_os_str().is_empty() || state_dir.as_os_str().is_empty() { + return Err(HeartbeatConfigError::Partial); + } + let root_ca_pem = fs::read(&root_ca_file).map_err(|source| HeartbeatConfigError::RootCertificate { + path: root_ca_file, + source, + })?; + Ok(Some(Self::new( + endpoint, + root_ca_pem, + IdentityStore::new(state_dir.join("identity")), + CredentialStore::new(state_dir.join("credential")), + state_dir.join("heartbeat/state.json"), + ))) + } +} + +#[derive(Debug, thiserror::Error)] +pub enum HeartbeatConfigError { + #[error( + "Connect heartbeat configuration requires RUSTFS_CONNECT_ENDPOINT, RUSTFS_CONNECT_ROOT_CA_FILE, and RUSTFS_CONNECT_STATE_DIR" + )] + Partial, + #[error("RUSTFS_CONNECT_ENDPOINT is not valid UTF-8")] + EndpointEncoding, + #[error("failed to read the Connect root CA at {path}: {source}")] + RootCertificate { + path: PathBuf, + #[source] + source: std::io::Error, + }, +} + +#[cfg(test)] +mod tests { + use super::{HeartbeatConfig, HeartbeatConfigError}; + use std::ffi::OsString; + + #[test] + fn absent_environment_is_disabled_without_side_effects() { + assert!( + HeartbeatConfig::from_env_values(None, None, None) + .expect("absent config") + .is_none() + ); + } + + #[test] + fn partial_environment_is_rejected() { + assert!(matches!( + HeartbeatConfig::from_env_values(Some(OsString::from("https://connect.example/agent/")), None, None), + Err(HeartbeatConfigError::Partial) + )); + } + + #[test] + fn complete_environment_builds_the_durable_paths() { + let temp = tempfile::tempdir().expect("tempdir"); + let root = temp.path().join("root.pem"); + std::fs::write(&root, b"root certificate").expect("root CA"); + let state = temp.path().join("state"); + let config = HeartbeatConfig::from_env_values( + Some(OsString::from("https://connect.example/agent/")), + Some(root.into_os_string()), + Some(state.clone().into_os_string()), + ) + .expect("complete config") + .expect("enabled config"); + + assert_eq!(config.endpoint, "https://connect.example/agent/"); + assert_eq!(config.root_ca_pem, b"root certificate"); + assert_eq!(config.state_path, state.join("heartbeat/state.json")); + assert!(!state.exists(), "parsing configuration must not create state"); + } +} diff --git a/rustfs/src/connect/credential_store.rs b/rustfs/src/connect/credential_store.rs new file mode 100644 index 000000000..863ae1096 --- /dev/null +++ b/rustfs/src/connect/credential_store.rs @@ -0,0 +1,370 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; + +use serde::{Deserialize, Serialize}; + +const CREDENTIAL_FILE: &str = "device.crt.json"; +const REGISTRATION_COMPLETED_FILE: &str = "registration.completed.json"; +const REGISTRATION_PENDING_FILE: &str = "registration.pending.json"; +const ROTATION_PENDING_FILE: &str = "rotation.pending.json"; +const LOCK_FILE: &str = ".state.lock"; + +#[cfg(unix)] +const FILE_MODE: u32 = 0o600; + +static STAGING_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +#[derive(Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct DeviceCredential { + pub name: String, + pub uid: String, + pub protocol_version: String, + pub key_id: String, + pub certificate_serial: String, + pub certificate: String, + pub certificate_chain: String, + pub not_before_unix: i64, + pub not_after_unix: i64, +} + +impl std::fmt::Debug for DeviceCredential { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("DeviceCredential") + .field("name", &self.name) + .field("key_id", &self.key_id) + .field("certificate_serial", &self.certificate_serial) + .field("not_after_unix", &self.not_after_unix) + .finish_non_exhaustive() + } +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub(crate) struct PendingRegistration { + pub token_uid: String, + pub request_id: String, + pub certificate_request: String, + #[serde(default)] + pub previous_credential_fingerprint: Option, + #[serde(default)] + pub next_public_key_sha256: Option, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub(crate) struct CompletedRegistration { + pub token_uid: String, + pub credential_fingerprint: String, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub(crate) struct PendingRotation { + pub credential_fingerprint: String, + pub device_name: String, + pub request_id: String, + pub certificate_request: String, + pub next_public_key_sha256: String, +} + +#[derive(Debug, thiserror::Error)] +pub enum CredentialStoreError { + #[error("connect credential store I/O failed at {path}: {source}")] + Io { + path: PathBuf, + #[source] + source: io::Error, + }, + + #[error("connect credential data at {path} is invalid: {source}")] + Invalid { + path: PathBuf, + #[source] + source: serde_json::Error, + }, + + #[cfg(unix)] + #[error("connect credential file at {path} has mode {mode:o}, expected {expected:o}")] + Permissions { path: PathBuf, mode: u32, expected: u32 }, +} + +#[derive(Clone, Debug)] +pub struct CredentialStore { + directory: PathBuf, +} + +pub(crate) struct CredentialLock { + _file: fs::File, +} + +impl CredentialStore { + pub fn new(directory: impl Into) -> Self { + Self { + directory: directory.into(), + } + } + + pub(crate) fn load(&self) -> Result, CredentialStoreError> { + self.read(CREDENTIAL_FILE) + } + + pub(crate) async fn lock(&self) -> Result { + let directory = self.directory.clone(); + tokio::task::spawn_blocking(move || { + fs::create_dir_all(&directory).map_err(|source| CredentialStoreError::Io { + path: directory.clone(), + source, + })?; + let path = directory.join(LOCK_FILE); + let mut options = fs::OpenOptions::new(); + options.create(true).truncate(false).read(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(FILE_MODE); + } + let file = options.open(&path).map_err(|source| CredentialStoreError::Io { + path: path.clone(), + source, + })?; + check_mode(&path)?; + file.lock().map_err(|source| CredentialStoreError::Io { path, source })?; + Ok(CredentialLock { _file: file }) + }) + .await + .map_err(|source| CredentialStoreError::Io { + path: self.directory.join(LOCK_FILE), + source: io::Error::other(source), + })? + } + + pub(crate) fn save(&self, credential: &DeviceCredential) -> Result<(), CredentialStoreError> { + self.write(CREDENTIAL_FILE, credential) + } + + pub(crate) fn claim_pending_registration( + &self, + pending: &PendingRegistration, + ) -> Result { + self.claim(REGISTRATION_PENDING_FILE, pending) + } + + pub(crate) fn load_pending_registration(&self) -> Result, CredentialStoreError> { + self.read(REGISTRATION_PENDING_FILE) + } + + pub(crate) fn clear_pending_registration(&self) -> Result<(), CredentialStoreError> { + self.remove(REGISTRATION_PENDING_FILE) + } + + pub(crate) fn load_completed_registration(&self) -> Result, CredentialStoreError> { + self.read(REGISTRATION_COMPLETED_FILE) + } + + pub(crate) fn save_completed_registration(&self, completed: &CompletedRegistration) -> Result<(), CredentialStoreError> { + self.write(REGISTRATION_COMPLETED_FILE, completed) + } + + pub(crate) fn load_pending_rotation(&self) -> Result, CredentialStoreError> { + self.read(ROTATION_PENDING_FILE) + } + + pub(crate) fn claim_pending_rotation(&self, pending: &PendingRotation) -> Result { + self.claim(ROTATION_PENDING_FILE, pending) + } + + pub(crate) fn clear_pending_rotation(&self) -> Result<(), CredentialStoreError> { + self.remove(ROTATION_PENDING_FILE) + } + + fn read Deserialize<'de>>(&self, file: &str) -> Result, CredentialStoreError> { + let path = self.directory.join(file); + let bytes = match fs::read(&path) { + Ok(bytes) => bytes, + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None), + Err(source) => return Err(CredentialStoreError::Io { path, source }), + }; + + check_mode(&path)?; + serde_json::from_slice(&bytes) + .map(Some) + .map_err(|source| CredentialStoreError::Invalid { path, source }) + } + + fn write(&self, file: &str, value: &T) -> Result<(), CredentialStoreError> { + let bytes = serde_json::to_vec(value).map_err(|source| CredentialStoreError::Invalid { + path: self.directory.join(file), + source, + })?; + + fs::create_dir_all(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + })?; + + let final_path = self.directory.join(file); + let temp_path = self.stage(file, &bytes)?; + let result = fs::rename(&temp_path, &final_path) + .map_err(|source| CredentialStoreError::Io { + path: final_path, + source, + }) + .and_then(|()| { + fsync_dir(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + }) + }); + + if result.is_err() { + let _ = fs::remove_file(&temp_path); + } + result + } + + fn claim(&self, file: &str, value: &T) -> Result + where + T: Clone + Serialize + for<'de> Deserialize<'de>, + { + if let Some(existing) = self.read(file)? { + return Ok(existing); + } + let bytes = serde_json::to_vec(value).map_err(|source| CredentialStoreError::Invalid { + path: self.directory.join(file), + source, + })?; + fs::create_dir_all(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + })?; + let final_path = self.directory.join(file); + let temp_path = self.stage(file, &bytes)?; + let published = fs::hard_link(&temp_path, &final_path); + let _ = fs::remove_file(&temp_path); + match published { + Ok(()) => { + fsync_dir(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + })?; + Ok(value.clone()) + } + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => { + fsync_dir(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + })?; + self.read(file)?.ok_or_else(|| CredentialStoreError::Io { + path: final_path, + source: io::Error::new(io::ErrorKind::NotFound, "pending state vanished after publication"), + }) + } + Err(source) => Err(CredentialStoreError::Io { + path: final_path, + source, + }), + } + } + + fn stage(&self, file: &str, bytes: &[u8]) -> Result { + loop { + let path = self.directory.join(format!( + ".{file}.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(FILE_MODE); + } + let mut staging = match options.open(&path) { + Ok(staging) => staging, + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => continue, + Err(source) => return Err(CredentialStoreError::Io { path, source }), + }; + let result = staging + .write_all(bytes) + .and_then(|()| staging.sync_all()) + .map_err(|source| CredentialStoreError::Io { + path: path.clone(), + source, + }) + .and_then(|()| check_mode(&path)); + if let Err(error) = result { + let _ = fs::remove_file(&path); + return Err(error); + } + return Ok(path); + } + } + + fn remove(&self, file: &str) -> Result<(), CredentialStoreError> { + let path = self.directory.join(file); + match fs::remove_file(&path) { + Ok(()) => fsync_dir(&self.directory).map_err(|source| CredentialStoreError::Io { + path: self.directory.clone(), + source, + }), + Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(()), + Err(source) => Err(CredentialStoreError::Io { path, source }), + } + } +} + +#[cfg(unix)] +fn check_mode(path: &Path) -> Result<(), CredentialStoreError> { + use std::os::unix::fs::PermissionsExt as _; + + let mode = fs::metadata(path) + .map_err(|source| CredentialStoreError::Io { + path: path.to_path_buf(), + source, + })? + .permissions() + .mode() + & 0o7777; + if mode != FILE_MODE { + return Err(CredentialStoreError::Permissions { + path: path.to_path_buf(), + mode, + expected: FILE_MODE, + }); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_mode(_path: &Path) -> Result<(), CredentialStoreError> { + Ok(()) +} + +fn fsync_dir(directory: &Path) -> io::Result<()> { + #[cfg(unix)] + fs::File::open(directory)?.sync_all()?; + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} diff --git a/rustfs/src/connect/heartbeat.rs b/rustfs/src/connect/heartbeat.rs new file mode 100644 index 000000000..2eeb8c135 --- /dev/null +++ b/rustfs/src/connect/heartbeat.rs @@ -0,0 +1,585 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use chrono::{DateTime, SecondsFormat, Utc}; +use reqwest::{Client, StatusCode, Url, header}; +use rustls::RootCertStore; +use rustls::pki_types::{CertificateDer, pem::PemObject as _}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; +use zeroize::Zeroizing; + +use super::config::HeartbeatConfig; +use super::credential_store::{CredentialStoreError, DeviceCredential}; +use super::identity::IdentityError; +use super::identity_store::StoreError; +use super::registration::{CredentialValidationError, validate_stored_credential}; + +const PROTOCOL_VERSION: &str = "v1"; +const AGENT_VERSION: &str = concat!("rustfs-agent/", env!("CARGO_PKG_VERSION")); +const MAX_SEQUENCE: u64 = 9_007_199_254_740_991; +const MAX_RESPONSE_BYTES: usize = 64 * 1024; +#[cfg(unix)] +const FILE_MODE: u32 = 0o600; +static STAGING_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct CoarseNodeSummary { + total: u16, + healthy: u16, + degraded: u16, +} + +impl CoarseNodeSummary { + pub fn new(total: u16, healthy: u16, degraded: u16) -> Result { + let summary = Self { + total, + healthy, + degraded, + }; + if !summary.is_valid() { + return Err(HeartbeatError::NodeSummary); + } + Ok(summary) + } + + fn is_valid(&self) -> bool { + self.total != 0 + && self.total <= 4096 + && self.healthy <= 4096 + && self.degraded <= 4096 + && self.healthy.saturating_add(self.degraded) <= self.total + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum HeartbeatStatus { + Starting, + Online { server_time: String }, + BackingOff { delay: Duration }, + AuthenticationStopped { status: u16, reason: Option }, + Failed { reason: String }, + Stopped, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub(crate) struct PendingHeartbeat { + protocol_version: String, + request_id: String, + agent_version: String, + capabilities: [String; 1], + sequence: u64, + client_time: String, + coarse_node_summary: CoarseNodeSummary, +} + +impl PendingHeartbeat { + fn is_valid(&self) -> bool { + self.protocol_version == PROTOCOL_VERSION + && self.agent_version == AGENT_VERSION + && self.capabilities[0] == "heartbeat" + && self.sequence <= MAX_SEQUENCE + && self.coarse_node_summary.is_valid() + && is_exact_utc_seconds(&self.client_time) + && Uuid::parse_str(&self.request_id) + .is_ok_and(|request_id| request_id.get_version_num() == 4 && request_id.to_string() == self.request_id) + } +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct HeartbeatResponse { + server_time: String, + accepted_version: String, + #[serde(default)] + capability_hints: Vec, +} + +pub(crate) enum Delivery { + Accepted { server_time: String }, + Retry { retry_after: Option }, + AuthenticationStopped { status: u16, reason: Option }, + Rejected { status: u16, reason: Option }, +} + +pub(crate) struct HeartbeatSender { + endpoint: Url, + root_store: RootCertStore, + roots: Vec>, + config: HeartbeatConfig, +} + +impl HeartbeatSender { + pub(crate) fn new(config: HeartbeatConfig) -> Result { + let mut endpoint = Url::parse(&config.endpoint).map_err(|_| HeartbeatError::Endpoint)?; + if endpoint.scheme() != "https" + || endpoint.cannot_be_a_base() + || !endpoint.username().is_empty() + || endpoint.password().is_some() + || endpoint.query().is_some() + || endpoint.fragment().is_some() + { + return Err(HeartbeatError::Endpoint); + } + if !endpoint.path().ends_with('/') { + endpoint.set_path(&format!("{}/", endpoint.path())); + } + let roots = CertificateDer::pem_slice_iter(&config.root_ca_pem) + .collect::, _>>() + .map_err(|_| HeartbeatError::RootCertificate)?; + if roots.is_empty() { + return Err(HeartbeatError::RootCertificate); + } + let mut root_store = RootCertStore::empty(); + let (accepted, rejected) = root_store.add_parsable_certificates(roots.clone()); + if accepted != roots.len() || rejected != 0 { + return Err(HeartbeatError::RootCertificate); + } + let schedule = config.schedule; + if schedule.cadence.is_zero() + || schedule.timeout.is_zero() + || schedule.timeout > Duration::from_secs(5) + || schedule.initial_backoff.is_zero() + || schedule.max_backoff < schedule.initial_backoff + || schedule.max_backoff > Duration::from_secs(5 * 60) + || schedule.jitter > schedule.cadence + { + return Err(HeartbeatError::Schedule); + } + Ok(Self { + endpoint, + root_store, + roots, + config, + }) + } + + pub(crate) async fn send(&self, heartbeat: &PendingHeartbeat) -> Result { + let (cluster_uid, client) = { + let _lock = self.config.credential_store.lock().await?; + let credential = self.config.credential_store.load()?.ok_or(HeartbeatError::NotRegistered)?; + let identity = self.config.identity_store.load()?.ok_or(HeartbeatError::IdentityMissing)?; + validate_stored_credential(&credential, &identity, &self.root_store, &self.roots)?; + let now = Utc::now().timestamp(); + if now < credential.not_before_unix || now >= credential.not_after_unix { + return Err(HeartbeatError::CredentialExpired); + } + let cluster_uid = cluster_uid(&credential)?.to_owned(); + let client = self.client(&credential, &identity.to_pkcs8_pem()?)?; + (cluster_uid, client) + }; + let url = self.endpoint.join(&format!("clusters/{cluster_uid}/heartbeats"))?; + let response = match client.post(url).json(heartbeat).send().await { + Ok(response) => response, + Err(error) if error.is_timeout() || error.is_connect() || error.is_request() => { + return Ok(Delivery::Retry { retry_after: None }); + } + Err(error) => return Err(error.into()), + }; + let status = response.status(); + if status == StatusCode::TOO_MANY_REQUESTS { + return Ok(Delivery::Retry { + retry_after: retry_after(response.headers(), Utc::now(), self.config.schedule.max_backoff), + }); + } + if status == StatusCode::REQUEST_TIMEOUT || status.is_server_error() { + return Ok(Delivery::Retry { retry_after: None }); + } + if matches!(status, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) { + return Ok(Delivery::AuthenticationStopped { + status: status.as_u16(), + reason: response_reason(response).await, + }); + } + if status != StatusCode::OK { + return Ok(Delivery::Rejected { + status: status.as_u16(), + reason: response_reason(response).await, + }); + } + let accepted: HeartbeatResponse = + serde_json::from_slice(&bounded_body(response).await?).map_err(|_| HeartbeatError::Response)?; + if accepted.accepted_version != PROTOCOL_VERSION + || accepted.capability_hints.len() > 32 + || accepted.capability_hints.iter().any(|hint| hint.len() > 32) + || !is_exact_utc_seconds(&accepted.server_time) + { + return Err(HeartbeatError::Response); + } + Ok(Delivery::Accepted { + server_time: accepted.server_time, + }) + } + + fn client(&self, credential: &DeviceCredential, key: &Zeroizing) -> Result { + let mut pem = Zeroizing::new(Vec::with_capacity(credential.certificate_chain.len() + key.len() + 1)); + pem.extend_from_slice(credential.certificate_chain.as_bytes()); + pem.push(b'\n'); + pem.extend_from_slice(key.as_bytes()); + let identity = reqwest::Identity::from_pem(&pem).map_err(|_| HeartbeatError::IdentityCertificate)?; + let roots = self + .roots + .iter() + .map(|root| reqwest::Certificate::from_der(root.as_ref())) + .collect::, _>>()?; + Client::builder() + .https_only(true) + .redirect(reqwest::redirect::Policy::none()) + .timeout(self.config.schedule.timeout) + .tls_certs_only(roots) + .identity(identity) + .build() + .map_err(Into::into) + } +} + +#[derive(Clone)] +pub(crate) struct HeartbeatStateStore { + path: PathBuf, +} + +#[derive(Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct HeartbeatState { + next_sequence: u64, + pending: Option, +} + +impl HeartbeatStateStore { + pub(crate) fn new(path: PathBuf) -> Self { + Self { path } + } + + pub(crate) fn try_runtime_lock(&self) -> Result { + let directory = parent(&self.path)?; + fs::create_dir_all(directory).map_err(|source| state_io(directory, source))?; + let name = filename(&self.path)?; + let path = directory.join(format!(".{name}.lock")); + let mut options = fs::OpenOptions::new(); + options.create(true).truncate(false).read(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(FILE_MODE); + } + let lock = options.open(&path).map_err(|source| state_io(&path, source))?; + check_mode(&path)?; + lock.try_lock().map_err(|_| HeartbeatError::AlreadyRunning)?; + Ok(lock) + } + + pub(crate) async fn prepare( + &self, + summary: CoarseNodeSummary, + now: DateTime, + ) -> Result { + let store = self.clone(); + tokio::task::spawn_blocking(move || store.prepare_sync(summary, now)) + .await + .map_err(|source| state_io(&self.path, io::Error::other(source)))? + } + + pub(crate) async fn mark_accepted(&self, accepted: &PendingHeartbeat) -> Result<(), HeartbeatError> { + let store = self.clone(); + let accepted = accepted.clone(); + tokio::task::spawn_blocking(move || store.mark_accepted_sync(&accepted)) + .await + .map_err(|source| state_io(&self.path, io::Error::other(source)))? + } + + fn prepare_sync(&self, summary: CoarseNodeSummary, now: DateTime) -> Result { + let mut state = self.read()?; + if let Some(pending) = state.pending { + return Ok(pending); + } + if state.next_sequence > MAX_SEQUENCE { + return Err(HeartbeatError::SequenceExhausted); + } + let pending = PendingHeartbeat { + protocol_version: PROTOCOL_VERSION.to_owned(), + request_id: Uuid::new_v4().to_string(), + agent_version: AGENT_VERSION.to_owned(), + capabilities: ["heartbeat".to_owned()], + sequence: state.next_sequence, + client_time: now.to_rfc3339_opts(SecondsFormat::Secs, true), + coarse_node_summary: summary, + }; + state.pending = Some(pending.clone()); + self.write(&state)?; + Ok(pending) + } + + fn mark_accepted_sync(&self, accepted: &PendingHeartbeat) -> Result<(), HeartbeatError> { + let mut state = self.read()?; + if state.pending.as_ref() != Some(accepted) { + return Err(HeartbeatError::StateConflict); + } + state.next_sequence = accepted.sequence.checked_add(1).ok_or(HeartbeatError::SequenceExhausted)?; + state.pending = None; + self.write(&state) + } + + fn read(&self) -> Result { + let bytes = match fs::read(&self.path) { + Ok(bytes) => bytes, + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(HeartbeatState::default()), + Err(source) => return Err(state_io(&self.path, source)), + }; + check_mode(&self.path)?; + let state: HeartbeatState = serde_json::from_slice(&bytes).map_err(|source| HeartbeatError::StateInvalid { + path: self.path.clone(), + source, + })?; + if state.next_sequence > MAX_SEQUENCE + 1 + || state + .pending + .as_ref() + .is_some_and(|pending| pending.sequence != state.next_sequence || !pending.is_valid()) + { + return Err(HeartbeatError::StateCorrupt { path: self.path.clone() }); + } + Ok(state) + } + + fn write(&self, state: &HeartbeatState) -> Result<(), HeartbeatError> { + let bytes = serde_json::to_vec(state).map_err(|source| HeartbeatError::StateInvalid { + path: self.path.clone(), + source, + })?; + let directory = parent(&self.path)?; + fs::create_dir_all(directory).map_err(|source| state_io(directory, source))?; + let temp = stage(directory, &self.path, &bytes)?; + let result = fs::rename(&temp, &self.path) + .map_err(|source| state_io(&self.path, source)) + .and_then(|()| fsync_dir(directory).map_err(|source| state_io(directory, source))); + if result.is_err() { + let _ = fs::remove_file(temp); + } + result + } +} + +fn cluster_uid(credential: &DeviceCredential) -> Result<&str, HeartbeatError> { + let mut parts = credential.name.split('/'); + let valid = parts.next() == Some("organizations"); + let organization_uid = parts.next(); + let valid = valid && parts.next() == Some("clusters"); + let cluster_uid = parts.next(); + let valid = valid && parts.next() == Some("clusterDevices"); + let device_uid = parts.next(); + if !valid + || organization_uid.is_none_or(str::is_empty) + || cluster_uid.is_none_or(str::is_empty) + || device_uid != Some(credential.uid.as_str()) + || parts.next().is_some() + { + return Err(HeartbeatError::CredentialName); + } + cluster_uid.ok_or(HeartbeatError::CredentialName) +} + +fn retry_after(headers: &header::HeaderMap, now: DateTime, maximum: Duration) -> Option { + let value = headers.get(header::RETRY_AFTER)?.to_str().ok()?; + let delay = value.parse::().ok().map(Duration::from_secs).or_else(|| { + DateTime::parse_from_rfc2822(value) + .ok() + .and_then(|at| (at.with_timezone(&Utc) - now).to_std().ok()) + })?; + Some(delay.min(maximum)) +} + +fn is_exact_utc_seconds(value: &str) -> bool { + DateTime::parse_from_rfc3339(value).is_ok_and(|time| { + time.offset().local_minus_utc() == 0 + && value.ends_with('Z') + && time.with_timezone(&Utc).to_rfc3339_opts(SecondsFormat::Secs, true) == value + }) +} + +async fn response_reason(response: reqwest::Response) -> Option { + #[derive(Deserialize)] + struct Envelope { + #[serde(default)] + details: Vec, + } + #[derive(Deserialize)] + struct Detail { + #[serde(default)] + reason: String, + } + + serde_json::from_slice::(&bounded_body(response).await.ok()?) + .ok()? + .details + .into_iter() + .find_map(|detail| (!detail.reason.is_empty()).then_some(detail.reason)) +} + +async fn bounded_body(mut response: reqwest::Response) -> Result, HeartbeatError> { + let mut body = Vec::new(); + while let Some(chunk) = response.chunk().await? { + if body.len().saturating_add(chunk.len()) > MAX_RESPONSE_BYTES { + return Err(HeartbeatError::ResponseTooLarge); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} + +fn parent(path: &Path) -> Result<&Path, HeartbeatError> { + path.parent() + .ok_or_else(|| state_io(path, io::Error::new(io::ErrorKind::InvalidInput, "state path has no parent"))) +} + +fn filename(path: &Path) -> Result<&str, HeartbeatError> { + path.file_name() + .and_then(|name| name.to_str()) + .ok_or_else(|| state_io(path, io::Error::new(io::ErrorKind::InvalidInput, "state filename is invalid"))) +} + +fn stage(directory: &Path, destination: &Path, bytes: &[u8]) -> Result { + let name = filename(destination)?; + loop { + let path = directory.join(format!( + ".{name}.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(FILE_MODE); + } + let mut file = match options.open(&path) { + Ok(file) => file, + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => continue, + Err(source) => return Err(state_io(&path, source)), + }; + if let Err(source) = file.write_all(bytes).and_then(|()| file.sync_all()) { + let _ = fs::remove_file(&path); + return Err(state_io(&path, source)); + } + return Ok(path); + } +} + +fn state_io(path: &Path, source: io::Error) -> HeartbeatError { + HeartbeatError::StateIo { + path: path.to_path_buf(), + source, + } +} + +#[cfg(unix)] +fn check_mode(path: &Path) -> Result<(), HeartbeatError> { + use std::os::unix::fs::PermissionsExt as _; + + let mode = fs::metadata(path) + .map_err(|source| state_io(path, source))? + .permissions() + .mode() + & 0o7777; + if mode != FILE_MODE { + return Err(HeartbeatError::StatePermissions { + path: path.to_path_buf(), + mode, + expected: FILE_MODE, + }); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_mode(_path: &Path) -> Result<(), HeartbeatError> { + Ok(()) +} + +fn fsync_dir(directory: &Path) -> io::Result<()> { + #[cfg(unix)] + fs::File::open(directory)?.sync_all()?; + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} + +#[derive(Debug, thiserror::Error)] +pub enum HeartbeatError { + #[error("Connect heartbeat endpoint must be an HTTPS base URL without credentials, query, or fragment")] + Endpoint, + #[error("Connect heartbeat root CA configuration is invalid")] + RootCertificate, + #[error("Connect heartbeat schedule is invalid")] + Schedule, + #[error("RustFS is not registered with Connect")] + NotRegistered, + #[error("the Connect device private key is missing")] + IdentityMissing, + #[error("the stored Connect certificate and device private key cannot form a TLS identity")] + IdentityCertificate, + #[error("the stored Connect credential name is invalid")] + CredentialName, + #[error("the stored Connect device certificate is not currently valid")] + CredentialExpired, + #[error("the Connect heartbeat node summary is outside protocol bounds")] + NodeSummary, + #[error("the Connect heartbeat sequence is exhausted")] + SequenceExhausted, + #[error("a Connect heartbeat runtime already owns this state")] + AlreadyRunning, + #[error("the persisted Connect heartbeat changed while delivery was in flight")] + StateConflict, + #[error("Connect heartbeat state I/O failed at {path}: {source}")] + StateIo { + path: PathBuf, + #[source] + source: io::Error, + }, + #[error("Connect heartbeat state at {path} is invalid: {source}")] + StateInvalid { + path: PathBuf, + #[source] + source: serde_json::Error, + }, + #[error("Connect heartbeat state at {path} violates the protocol invariants")] + StateCorrupt { path: PathBuf }, + #[cfg(unix)] + #[error("Connect heartbeat state at {path} has mode {mode:o}, expected {expected:o}")] + StatePermissions { path: PathBuf, mode: u32, expected: u32 }, + #[error("Connect heartbeat response exceeded 64 KiB")] + ResponseTooLarge, + #[error("Connect returned an invalid heartbeat response")] + Response, + #[error(transparent)] + Url(#[from] url::ParseError), + #[error(transparent)] + Transport(#[from] reqwest::Error), + #[error(transparent)] + Identity(#[from] IdentityError), + #[error(transparent)] + IdentityStore(#[from] StoreError), + #[error(transparent)] + CredentialStore(#[from] CredentialStoreError), + #[error(transparent)] + CredentialValidation(#[from] CredentialValidationError), +} diff --git a/rustfs/src/connect/identity.rs b/rustfs/src/connect/identity.rs index 64aa79266..2220f4e8c 100644 --- a/rustfs/src/connect/identity.rs +++ b/rustfs/src/connect/identity.rs @@ -24,7 +24,7 @@ use base64::Engine as _; use base64::engine::general_purpose::{STANDARD as BASE64_STANDARD, URL_SAFE_NO_PAD as BASE64_URL_NO_PAD}; use p256::ecdsa::signature::Signer as _; use p256::ecdsa::{Signature, SigningKey}; -use p256::pkcs8::{DecodePrivateKey as _, EncodePrivateKey as _}; +use p256::pkcs8::{DecodePrivateKey as _, EncodePrivateKey as _, LineEnding}; use sha2::{Digest as _, Sha256}; use zeroize::Zeroizing; @@ -213,6 +213,12 @@ impl DeviceIdentity { .map_err(|error| IdentityError::MalformedKey(error.to_string())) } + pub(crate) fn to_pkcs8_pem(&self) -> Result, IdentityError> { + self.signing_key + .to_pkcs8_pem(LineEnding::LF) + .map_err(|error| IdentityError::MalformedKey(error.to_string())) + } + /// Build the PKCS#10 certificate request Connect consumes. /// /// Connect reads the request for its SubjectPublicKeyInfo and its diff --git a/rustfs/src/connect/identity_store.rs b/rustfs/src/connect/identity_store.rs index 2f4e6724e..426c98352 100644 --- a/rustfs/src/connect/identity_store.rs +++ b/rustfs/src/connect/identity_store.rs @@ -32,6 +32,7 @@ use super::identity::{DeviceIdentity, IdentityError}; /// Name of the key file inside the store directory. const KEY_FILE: &str = "device.key"; +const NEXT_KEY_FILE: &str = "device.key.next"; /// Owner read/write only. The key is the device's whole identity. #[cfg(unix)] @@ -94,7 +95,15 @@ impl IdentityStore { /// been enrolled. Reading never creates anything, so an unconfigured /// server can ask without acquiring an identity as a side effect. pub fn load(&self) -> Result, StoreError> { - let path = self.key_path(); + self.load_file(KEY_FILE) + } + + pub(crate) fn load_next(&self) -> Result, StoreError> { + self.load_file(NEXT_KEY_FILE) + } + + fn load_file(&self, file: &str) -> Result, StoreError> { + let path = self.directory.join(file); let der = match fs::read(&path) { Ok(der) => Zeroizing::new(der), @@ -142,11 +151,15 @@ impl IdentityStore { let candidate = DeviceIdentity::generate(); let der = candidate.to_pkcs8_der()?; - match self.publish(&der) { + match self.publish(KEY_FILE, &der) { Ok(()) => Ok(candidate), // Another process published first. Its key is the identity; ours // was never written anywhere and simply goes out of scope. Err(StoreError::Io { source, .. }) if source.kind() == io::ErrorKind::AlreadyExists => { + fsync_dir(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + })?; self.load()?.ok_or_else(|| StoreError::Io { path: self.key_path(), source: io::Error::new( @@ -159,35 +172,106 @@ impl IdentityStore { } } + pub(crate) fn load_or_create_next(&self) -> Result { + if let Some(identity) = self.load_next()? { + return Ok(identity); + } + fs::create_dir_all(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + })?; + let candidate = DeviceIdentity::generate(); + let der = candidate.to_pkcs8_der()?; + match self.publish(NEXT_KEY_FILE, &der) { + Ok(()) => Ok(candidate), + Err(StoreError::Io { source, .. }) if source.kind() == io::ErrorKind::AlreadyExists => { + fsync_dir(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + })?; + self.load_next()?.ok_or_else(|| StoreError::Io { + path: self.directory.join(NEXT_KEY_FILE), + source: io::Error::new(io::ErrorKind::NotFound, "next device key vanished after publication"), + }) + } + Err(error) => Err(error), + } + } + + pub(crate) fn commit_next(&self, expected: &DeviceIdentity) -> Result<(), StoreError> { + let next_path = self.directory.join(NEXT_KEY_FILE); + match fs::rename(&next_path, self.key_path()) { + Ok(()) => fsync_dir(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + }), + Err(source) if source.kind() == io::ErrorKind::NotFound => { + let current = self.load()?.ok_or_else(|| StoreError::Io { + path: self.key_path(), + source, + })?; + if current.public_key_der() == expected.public_key_der() { + fsync_dir(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + }) + } else { + Err(StoreError::Io { + path: next_path, + source: io::Error::new(io::ErrorKind::NotFound, "next device key is missing"), + }) + } + } + Err(source) => Err(StoreError::Io { path: next_path, source }), + } + } + + pub(crate) fn clear_next(&self) -> Result<(), StoreError> { + let path = self.directory.join(NEXT_KEY_FILE); + match fs::remove_file(&path) { + Ok(()) => fsync_dir(&self.directory).map_err(|source| StoreError::Io { + path: self.directory.clone(), + source, + }), + Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(()), + Err(source) => Err(StoreError::Io { path, source }), + } + } + /// Write, seal, fsync, then link into place and fsync the directory. The /// key is durable before it is reachable, and it is reachable only once. - fn publish(&self, der: &[u8]) -> Result<(), StoreError> { + fn publish(&self, file: &str, der: &[u8]) -> Result<(), StoreError> { use std::io::Write as _; - let final_path = self.key_path(); - let temp_path = self.directory.join(format!( - "{KEY_FILE}.{}.{}.tmp", - std::process::id(), - STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) - )); + let final_path = self.directory.join(file); let io_at = |path: &Path| { let path = path.to_path_buf(); move |source| StoreError::Io { path, source } }; - let mut options = fs::OpenOptions::new(); - options.write(true).create(true).truncate(true); - #[cfg(unix)] - { - use std::os::unix::fs::OpenOptionsExt as _; - options.mode(KEY_MODE); - } - - let mut file = options.open(&temp_path).map_err(io_at(&temp_path))?; + let (temp_path, mut staging) = loop { + let temp_path = self.directory.join(format!( + ".{file}.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(KEY_MODE); + } + match options.open(&temp_path) { + Ok(staging) => break (temp_path, staging), + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => continue, + Err(source) => return Err(io_at(&temp_path)(source)), + } + }; let result = (|| -> Result<(), StoreError> { - file.write_all(der).map_err(io_at(&temp_path))?; + staging.write_all(der).map_err(io_at(&temp_path))?; // The umask can only narrow the creation mode, so set and verify // the exact mode before the bytes become durable. @@ -195,9 +279,10 @@ impl IdentityStore { { use std::os::unix::fs::PermissionsExt as _; - file.set_permissions(fs::Permissions::from_mode(KEY_MODE)) + staging + .set_permissions(fs::Permissions::from_mode(KEY_MODE)) .map_err(io_at(&temp_path))?; - let mode = file.metadata().map_err(io_at(&temp_path))?.permissions().mode() & 0o7777; + let mode = staging.metadata().map_err(io_at(&temp_path))?.permissions().mode() & 0o7777; if mode != KEY_MODE { return Err(StoreError::Permissions { path: temp_path.clone(), @@ -207,11 +292,11 @@ impl IdentityStore { } } - file.sync_all().map_err(io_at(&temp_path))?; + staging.sync_all().map_err(io_at(&temp_path))?; Ok(()) })(); - drop(file); + drop(staging); if let Err(error) = result { let _ = fs::remove_file(&temp_path); diff --git a/rustfs/src/connect/mod.rs b/rustfs/src/connect/mod.rs index 3972bf21c..3ce2ad975 100644 --- a/rustfs/src/connect/mod.rs +++ b/rustfs/src/connect/mod.rs @@ -21,12 +21,26 @@ //! canonical transcript frozen by //! `protocol/agent/v1/registration-proof.md`. //! -//! Nothing here contacts the network or starts a task. A deployment that has -//! not been enrolled into a Connect control plane never calls into it, so an -//! unconfigured server generates no key and holds no identity. +//! Enrolled deployments may start the optional outbound heartbeat runtime. +//! An unconfigured server starts no Connect task, generates no key, and holds +//! no Connect identity. +pub mod client; +pub mod config; +pub mod credential_store; +pub mod heartbeat; pub mod identity; pub mod identity_store; +pub mod offline; +pub mod registration; +pub mod runtime; +pub use client::{ClientError, ConnectClient, ConnectConfig}; +pub use config::{HeartbeatConfig, HeartbeatConfigError, HeartbeatSchedule}; +pub use credential_store::{CredentialStore, DeviceCredential}; +pub use heartbeat::{CoarseNodeSummary, HeartbeatError, HeartbeatStatus}; pub use identity::{DeviceIdentity, IdentityError, RegistrationProof, RegistrationTranscript}; pub use identity_store::{IdentityStore, StoreError}; +pub use offline::{EnrollmentError, OfflineEnrollment, OfflineKeyStore, VerifiedChallenge}; +pub use registration::{RegistrationToken, TokenError}; +pub use runtime::{HeartbeatRuntime, spawn_heartbeat_runtime}; diff --git a/rustfs/src/connect/offline/enrollment.rs b/rustfs/src/connect/offline/enrollment.rs new file mode 100644 index 000000000..a16fadb39 --- /dev/null +++ b/rustfs/src/connect/offline/enrollment.rs @@ -0,0 +1,684 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Challenge verification and response production for offline enrolment. +//! +//! Two invariants carry the security of this surface and both are easy to break +//! by accident: +//! +//! - Every signature is checked over the octets that arrived, never over a +//! re-serialised document. Parsing happens only to route the verification, and +//! nothing a parse yields is believed until the signature over those same +//! octets has verified. +//! - The enrolment root is the constant in this file. It is never taken from a +//! challenge, a configuration file, or an operator prompt, so there is no +//! trust-on-first-use path an operator could be talked into. +//! +//! The order of the checks in [`OfflineEnrollment::verify_challenge`] is frozen +//! by `verificationOrder.enrollmentChallenge` in +//! `protocol/agent/v1/fixtures/offline-enrollment/trust-model.json`, and the +//! signature encoding, the domain separation tags, and every rejection reason +//! are frozen beside it. Reordering the checks changes which reason a given +//! artifact produces, which is itself part of the contract. + +use base64::Engine as _; +use base64::engine::general_purpose::{STANDARD as BASE64_STANDARD, URL_SAFE_NO_PAD as BASE64_URL_NO_PAD}; +use p256::ecdsa::signature::{Signer as _, Verifier as _}; +use p256::ecdsa::{Signature, SigningKey, VerifyingKey}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use time::{Date, Month, OffsetDateTime, PrimitiveDateTime, Time}; + +use crate::connect::identity::DeviceIdentity; + +/// The hosted enrolment root, compiled in. Both halves are pinned: the +/// fingerprint identifies the root, and the point is what actually verifies the +/// first link, so a build cannot be pointed at a different key by supplying one. +const PINNED_ROOT_KEY_ID: &str = "df22e2806112debbe953672aafa186d699af0e97dd3fd2b09fa8359005fe348f"; +const PINNED_ROOT_PUBLIC_KEY: &str = "BFfx-K-FfEA5nK_Rz3IHacvRCkJyQ7JOd1geLyU6HKRZDgNezmVuKhvJ22VhemyjV__Gshk8JGGqOBzYPMD0p6s"; + +/// Domain separation tags. A document that verifies under one of these must not +/// be accepted for another artifact type, so the tag is part of the signature +/// input rather than a property of the caller. +const TAG_TRUST_LINK: &[u8] = b"rustfs-offline-trust-link-v1"; +const TAG_CHALLENGE: &[u8] = b"rustfs-offline-enrollment-challenge-v1"; +const TAG_RESPONSE: &[u8] = b"rustfs-offline-enrollment-response-v1"; + +/// The single octet between the tag and the signed document. +const DOMAIN_SEPARATOR: u8 = 0x00; + +const SIGNATURE_ALGORITHM: &str = "ES256"; +const PROTOCOL_VERSION: &str = "v1"; +const FORMAT_TRUST_LINK: &str = "rustfs.connect.offline.trustLink/1"; +const FORMAT_CHALLENGE: &str = "rustfs.connect.offline.enrollmentChallenge/1"; +const FORMAT_RESPONSE: &str = "rustfs.connect.offline.enrollmentResponse/1"; + +/// DER SubjectPublicKeyInfo header for an uncompressed P-256 point. A keyId is +/// the SHA-256 of this prefix followed by the 65 octet point, so the prefix is +/// also how a device public key is recovered from its own DER encoding. +const SPKI_PREFIX: [u8; 26] = [ + 0x30, 0x59, 0x30, 0x13, 0x06, 0x07, 0x2a, 0x86, 0x48, 0xce, 0x3d, 0x02, 0x01, 0x06, 0x08, 0x2a, 0x86, 0x48, 0xce, 0x3d, 0x03, + 0x01, 0x07, 0x03, 0x42, 0x00, +]; + +/// Order of the P-256 group, and half of it. `r` and `s` must lie in `[1, n)`, +/// and `s` additionally in `[1, n/2]`: ECDSA admits both `s` and `n - s`, and a +/// signature with two spellings cannot serve as an artifact identity. Every +/// ECDSA library accepts the malleated form, so the encoding layer rejects it. +const GROUP_ORDER: [u8; 32] = [ + 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xbc, 0xe6, 0xfa, 0xad, 0xa7, + 0x17, 0x9e, 0x84, 0xf3, 0xb9, 0xca, 0xc2, 0xfc, 0x63, 0x25, 0x51, +]; +const MAX_S: [u8; 32] = [ + 0x7f, 0xff, 0xff, 0xff, 0x80, 0x00, 0x00, 0x00, 0x7f, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xde, 0x73, 0x7d, 0x56, 0xd3, + 0x8b, 0xcf, 0x42, 0x79, 0xdc, 0xe5, 0x61, 0x7e, 0x31, 0x92, 0xa8, +]; + +const SCALAR_OCTETS: usize = 32; +const SIGNATURE_OCTETS: usize = 64; +/// 64 octets as unpadded base64url. The length is checked before decoding so +/// that `=` padding, the standard alphabet, DER, and a truncated value are all +/// refused rather than repaired. +const SIGNATURE_VALUE_CHARS: usize = 86; + +const PUBLIC_KEY_OCTETS: usize = 65; +const PUBLIC_KEY_CHARS: usize = 87; +/// SEC1 tag of an uncompressed point. Compressed and hybrid forms are refused. +const UNCOMPRESSED_POINT: u8 = 0x04; + +const TIMESTAMP_CHARS: usize = 20; + +/// The chain is exactly two links: a pinned root issues the intermediate, and +/// the intermediate issues the signing key. Roles are positional and the +/// enumeration is closed. +const CHAIN_LINK_COUNT: usize = 2; +const CHAIN_ROLES: [&str; CHAIN_LINK_COUNT] = ["intermediate", "signing"]; + +/// Skew allowed on the challenge window. A device may have no synchronised +/// clock at all, so its own reading of "now" is advisory. +const CLOCK_SKEW_TOLERANCE: i64 = 300; + +/// Longest life a challenge may claim. The issuer sets both ends of its own +/// window, so the protocol bound is applied on top of the declared expiry +/// rather than trusted from it. +const MAX_CHALLENGE_LIFETIME: i64 = 604_800; + +/// A challenge that verified, with the fields the response has to echo. +/// +/// Construction is the proof: a value of this type only exists after the chain +/// closed on the pinned root and the challenge signature verified over the +/// received octets. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct VerifiedChallenge { + pub challenge_id: String, + pub organization_name: String, + pub cluster_name: String, + pub nonce: String, + pub issued_at: String, + pub expires_at: String, + pub connect_key_id: String, + /// The signature value of the challenge, verbatim. It binds a response to + /// the one challenge it answers, so it is carried rather than recomputed. + pub challenge_proof: String, +} + +/// Why an offline enrolment artifact was refused. +/// +/// The variants are the frozen `reason` vocabulary of +/// `fixtures/offline-enrollment/error-codes.json`, which spans both halves of +/// the exchange. The device half implemented here produces the encoding, chain, +/// version, and freshness reasons; the reasons that describe a response being +/// evaluated against stored state — [`Self::ChallengeUnknown`], +/// [`Self::ChallengeProofInvalid`], [`Self::DeviceProofInvalid`], +/// [`Self::EnrollmentReplayed`], [`Self::OrganizationMismatch`], and +/// [`Self::ClusterMismatch`] — are Connect's to raise and are named here so the +/// two sides share one vocabulary. +/// +/// No variant carries a payload: a rejection must never disclose key material, +/// signature octets, nonces, or document bytes. +#[derive(Debug, PartialEq, Eq, thiserror::Error)] +pub enum EnrollmentError { + #[error("protocolVersion is missing, malformed, or names an unsupported major version")] + UnsupportedProtocol, + + #[error("formatVersion is not a supported offline enrollment format")] + UnsupportedFormat, + + #[error("the signature is not 64 octets of fixed-width r||s in unpadded base64url")] + SignatureMalformed, + + #[error("the signature is not in its canonical low-S form")] + SignatureNotCanonical, + + #[error("the signature does not verify over the received octets")] + SignatureInvalid, + + #[error("the trust chain is not issued by a root pinned in this build")] + EnrollmentRootUnknown, + + #[error("a trust link is invalid, misordered, or outside its validity at the challenge issuedAt")] + TrustChainInvalid, + + #[error("connectKeyId is not the subject of the last trust link")] + ConnectKeyUnchained, + + #[error("no issued challenge matches this challengeId")] + ChallengeUnknown, + + #[error("the challenge is not yet valid at the evaluation time")] + ChallengeNotYetValid, + + #[error("the challenge has expired at the evaluation time")] + ChallengeExpired, + + #[error("the response nonce or challengeProof is not the one issued for this challenge")] + ChallengeProofInvalid, + + #[error("the response does not prove possession of the device key it presents")] + DeviceProofInvalid, + + #[error("the challenge was already consumed")] + EnrollmentReplayed, + + #[error("the response names a different organization than the challenge it answers")] + OrganizationMismatch, + + #[error("the response names a different cluster than the challenge it answers")] + ClusterMismatch, + + /// The artifact could not be read as a signed enrolment document at all: the + /// envelope, the base64 of the signed octets, or a field the frozen order + /// reads before the signature verifies did not parse. The frozen reason set + /// has no code for a structurally unreadable document, so this variant maps + /// to none of them. + #[error("the offline enrollment document is not well formed")] + MalformedDocument, + + /// A fault on this side of the exchange rather than in the artifact: the + /// device key did not round-trip through its own PKCS#8 encoding, or the + /// caller named an instant outside the representable calendar. Fails closed + /// because a half-produced response must never reach removable media. + #[error("the enrollment response could not be produced on this device")] + ResponseNotProduced, +} + +impl EnrollmentError { + /// The frozen `reason` an operator and Connect both branch on. + /// + /// The `Display` message is prose and may be reworded; this is the stable + /// identifier, so nothing should parse the message instead. The two + /// variants with no frozen counterpart deliberately return codes outside + /// the frozen set rather than borrowing the nearest one, so a document that + /// simply failed to parse can never be reported as a signature or freshness + /// failure. + pub fn reason(&self) -> &'static str { + match self { + Self::UnsupportedProtocol => "UNSUPPORTED_PROTOCOL", + Self::UnsupportedFormat => "UNSUPPORTED_FORMAT", + Self::SignatureMalformed => "SIGNATURE_MALFORMED", + Self::SignatureNotCanonical => "SIGNATURE_NOT_CANONICAL", + Self::SignatureInvalid => "SIGNATURE_INVALID", + Self::EnrollmentRootUnknown => "ENROLLMENT_ROOT_UNKNOWN", + Self::TrustChainInvalid => "TRUST_CHAIN_INVALID", + Self::ConnectKeyUnchained => "CONNECT_KEY_UNCHAINED", + Self::ChallengeUnknown => "CHALLENGE_UNKNOWN", + Self::ChallengeNotYetValid => "CHALLENGE_NOT_YET_VALID", + Self::ChallengeExpired => "CHALLENGE_EXPIRED", + Self::ChallengeProofInvalid => "CHALLENGE_PROOF_INVALID", + Self::DeviceProofInvalid => "DEVICE_PROOF_INVALID", + Self::EnrollmentReplayed => "ENROLLMENT_REPLAYED", + Self::OrganizationMismatch => "ORGANIZATION_MISMATCH", + Self::ClusterMismatch => "CLUSTER_MISMATCH", + Self::MalformedDocument => "MALFORMED_DOCUMENT", + Self::ResponseNotProduced => "RESPONSE_NOT_PRODUCED", + } + } +} + +/// A signed document, in the shape both directions carry it. `bytes` is +/// standard padded base64 of the exact octets that were signed; nothing else is +/// ever used as the signature input. +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct SignedDocument { + bytes: String, + signature: DocumentSignature, +} + +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct DocumentSignature { + algorithm: String, + key_id: String, + value: String, +} + +/// The three fields the frozen order permits reading before anything verifies. +/// They route the verification and are not facts until it has. +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ChallengeRouting { + connect_key_id: String, + issued_at: String, + trust_chain: Vec, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ChallengeDocument { + format_version: String, + protocol_version: String, + challenge_id: String, + organization_name: String, + cluster_name: String, + nonce: String, + issued_at: String, + expires_at: String, + connect_key_id: String, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct TrustLink { + format_version: String, + protocol_version: String, + role: String, + issuer_key_id: String, + subject_key_id: String, + subject_public_key: String, + not_before: String, + not_after: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ResponseDocument<'a> { + format_version: &'a str, + protocol_version: &'a str, + challenge_id: &'a str, + organization_name: &'a str, + cluster_name: &'a str, + challenge_nonce: &'a str, + challenge_proof: &'a str, + device_key_id: String, + device_public_key: String, + device_nonce: String, + produced_at: String, +} + +/// The device half of the offline enrolment exchange: bytes in, bytes out. +pub struct OfflineEnrollment; + +impl OfflineEnrollment { + /// Verify an enrolment challenge and return what a response must echo. + /// + /// `now_unix` is the device's reading of the current time, which the clock + /// skew tolerance treats as advisory. + pub fn verify_challenge(document: &[u8], now_unix: i64) -> Result { + let envelope: SignedDocument = serde_json::from_slice(document).map_err(|_| EnrollmentError::MalformedDocument)?; + + // Step 1: the encoding is checked before anything is decoded from it, so + // a DER, padded, truncated, out-of-range, or high-S signature is refused + // on its spelling rather than handed to a library that would accept it. + let signature = decode_signature(&envelope.signature)?; + + // The octets that were transmitted. They are never re-serialised: every + // later step signs and parses this same buffer. + let bytes = BASE64_STANDARD + .decode(envelope.bytes.as_bytes()) + .map_err(|_| EnrollmentError::MalformedDocument)?; + + // Step 2: routing only. + let routing: ChallengeRouting = serde_json::from_slice(&bytes).map_err(|_| EnrollmentError::MalformedDocument)?; + let issued_at = parse_timestamp(&routing.issued_at)?; + + // Steps 3 to 5. + let connect_key = verify_trust_chain(&routing.trust_chain, &routing.connect_key_id, issued_at)?; + + // Step 6. The verification key comes from the chain, so `signature.keyId` + // is a label rather than an input: a value naming some other key simply + // fails to verify here. + if !verifies(&connect_key, TAG_CHALLENGE, &bytes, &signature) { + return Err(EnrollmentError::SignatureInvalid); + } + + // Step 7: only now is the document read as a fact. + let challenge: ChallengeDocument = serde_json::from_slice(&bytes).map_err(|_| EnrollmentError::MalformedDocument)?; + if challenge.protocol_version != PROTOCOL_VERSION { + return Err(EnrollmentError::UnsupportedProtocol); + } + if challenge.format_version != FORMAT_CHALLENGE { + return Err(EnrollmentError::UnsupportedFormat); + } + + // Step 8. + let expires_at = parse_timestamp(&challenge.expires_at)?; + check_challenge_window(issued_at, expires_at, now_unix)?; + + Ok(VerifiedChallenge { + challenge_id: challenge.challenge_id, + organization_name: challenge.organization_name, + cluster_name: challenge.cluster_name, + nonce: challenge.nonce, + issued_at: challenge.issued_at, + expires_at: challenge.expires_at, + connect_key_id: challenge.connect_key_id, + challenge_proof: envelope.signature.value, + }) + } + + /// Build the signed response an operator carries back to Connect. + /// + /// `device_nonce` is the response's own replay value and must come from a + /// cryptographic source. The private key never appears in the result: only + /// the public point, its fingerprint, and a signature over the document + /// that presents them, which is what makes presenting the key safe. + pub fn build_response( + challenge: &VerifiedChallenge, + key: &DeviceIdentity, + device_nonce: &[u8; 32], + produced_at_unix: i64, + ) -> Result, EnrollmentError> { + let issued_at = parse_timestamp(&challenge.issued_at)?; + let expires_at = parse_timestamp(&challenge.expires_at)?; + // Connect re-checks producedAt against the same window, so a response + // outside it is refused here rather than written to media and rejected + // after the operator has carried it out. + check_challenge_window(issued_at, expires_at, produced_at_unix)?; + + let point = device_public_point(key)?; + let produced_at = format_timestamp(produced_at_unix)?; + + let document = ResponseDocument { + format_version: FORMAT_RESPONSE, + protocol_version: PROTOCOL_VERSION, + challenge_id: &challenge.challenge_id, + organization_name: &challenge.organization_name, + cluster_name: &challenge.cluster_name, + challenge_nonce: &challenge.nonce, + challenge_proof: &challenge.challenge_proof, + device_key_id: key_id(&point), + device_public_key: BASE64_URL_NO_PAD.encode(point), + device_nonce: BASE64_URL_NO_PAD.encode(device_nonce), + produced_at, + }; + + // Serialised once. These octets are what is signed and what is carried, + // so no second serialisation can disagree with the signature. + let bytes = serde_json::to_vec(&document).map_err(|_| EnrollmentError::ResponseNotProduced)?; + let signature = sign(key, TAG_RESPONSE, &bytes)?; + + let envelope = SignedDocument { + bytes: BASE64_STANDARD.encode(&bytes), + signature: DocumentSignature { + algorithm: SIGNATURE_ALGORITHM.to_owned(), + key_id: document.device_key_id, + value: signature, + }, + }; + + serde_json::to_vec(&envelope).map_err(|_| EnrollmentError::ResponseNotProduced) + } +} + +/// Walk the chain from the pinned root to the signing key, returning the key +/// `connect_key_id` names once the chain vouches for it. +fn verify_trust_chain( + chain: &[SignedDocument], + connect_key_id: &str, + challenge_issued_at: i64, +) -> Result { + // The pinned root gate runs before the chain's shape is examined, so a + // chain that is internally consistent under a foreign root — exactly what + // trust on first use would have accepted — is refused for its root rather + // than for its length. + let first = chain.first().ok_or(EnrollmentError::EnrollmentRootUnknown)?; + let root = decode_trust_link(first)?; + if root.0.issuer_key_id != PINNED_ROOT_KEY_ID { + return Err(EnrollmentError::EnrollmentRootUnknown); + } + + let [_, second] = chain else { + return Err(EnrollmentError::TrustChainInvalid); + }; + let links = [root, decode_trust_link(second)?]; + + let mut issuer_key_id = PINNED_ROOT_KEY_ID.to_owned(); + let (mut issuer_key, _) = decode_public_key(PINNED_ROOT_PUBLIC_KEY).ok_or(EnrollmentError::EnrollmentRootUnknown)?; + + for (index, ((link, link_bytes), entry)) in links.iter().zip(chain).enumerate() { + if link.format_version != FORMAT_TRUST_LINK + || link.protocol_version != PROTOCOL_VERSION + || link.role != CHAIN_ROLES[index] + || link.issuer_key_id != issuer_key_id + // A link that names itself as its own issuer would let a stolen + // intermediate mint its own root. + || link.subject_key_id == link.issuer_key_id + { + return Err(EnrollmentError::TrustChainInvalid); + } + + let (subject_key, subject_point) = + decode_public_key(&link.subject_public_key).ok_or(EnrollmentError::TrustChainInvalid)?; + if key_id(&subject_point) != link.subject_key_id { + return Err(EnrollmentError::TrustChainInvalid); + } + + let signature = decode_signature(&entry.signature)?; + if !verifies(&issuer_key, TAG_TRUST_LINK, link_bytes, &signature) { + return Err(EnrollmentError::TrustChainInvalid); + } + + // The issuer controls both ends of a link's window, so it is evaluated + // with no skew tolerance, and against the challenge's issuedAt rather + // than against the device clock: a challenge carries the chain that was + // valid when it was issued. + let not_before = parse_timestamp(&link.not_before)?; + let not_after = parse_timestamp(&link.not_after)?; + if challenge_issued_at < not_before || challenge_issued_at > not_after { + return Err(EnrollmentError::TrustChainInvalid); + } + + issuer_key_id = link.subject_key_id.clone(); + issuer_key = subject_key; + } + + if issuer_key_id != connect_key_id { + return Err(EnrollmentError::ConnectKeyUnchained); + } + + Ok(issuer_key) +} + +/// Decode a link and keep the octets it was signed over: the signature is +/// checked against these, never against a re-encoding of the parsed link. +fn decode_trust_link(entry: &SignedDocument) -> Result<(TrustLink, Vec), EnrollmentError> { + let bytes = BASE64_STANDARD + .decode(entry.bytes.as_bytes()) + .map_err(|_| EnrollmentError::MalformedDocument)?; + let link = serde_json::from_slice(&bytes).map_err(|_| EnrollmentError::TrustChainInvalid)?; + Ok((link, bytes)) +} + +/// Check a signature's spelling and range, then admit it. +/// +/// `r` and `s` are compared against the group order here rather than left to +/// the ECDSA library, because a library that accepts high-S — every library +/// does — would let a malleated copy of an artifact pass as a second artifact. +fn decode_signature(signature: &DocumentSignature) -> Result { + if signature.algorithm != SIGNATURE_ALGORITHM { + return Err(EnrollmentError::SignatureMalformed); + } + + let value = signature.value.as_bytes(); + if value.len() != SIGNATURE_VALUE_CHARS || !value.iter().all(|byte| is_base64url(*byte)) { + return Err(EnrollmentError::SignatureMalformed); + } + + let decoded = BASE64_URL_NO_PAD + .decode(value) + .map_err(|_| EnrollmentError::SignatureMalformed)?; + let octets: [u8; SIGNATURE_OCTETS] = decoded + .as_slice() + .try_into() + .map_err(|_| EnrollmentError::SignatureMalformed)?; + + // Big-endian octets of equal length order lexicographically exactly as the + // integers they spell, so a slice comparison is the range check. + let (r, s) = octets.split_at(SCALAR_OCTETS); + let out_of_range = |scalar: &[u8]| scalar.iter().all(|byte| *byte == 0) || scalar >= &GROUP_ORDER[..]; + if out_of_range(r) || out_of_range(s) { + return Err(EnrollmentError::SignatureMalformed); + } + if s > &MAX_S[..] { + return Err(EnrollmentError::SignatureNotCanonical); + } + + Signature::from_slice(&octets).map_err(|_| EnrollmentError::SignatureMalformed) +} + +fn verifies(key: &VerifyingKey, tag: &[u8], bytes: &[u8], signature: &Signature) -> bool { + key.verify(&signature_input(tag, bytes), signature).is_ok() +} + +fn signature_input(tag: &[u8], bytes: &[u8]) -> Vec { + let mut input = Vec::with_capacity(tag.len() + 1 + bytes.len()); + input.extend_from_slice(tag); + input.push(DOMAIN_SEPARATOR); + input.extend_from_slice(bytes); + input +} + +fn sign(key: &DeviceIdentity, tag: &[u8], bytes: &[u8]) -> Result { + // `DeviceIdentity` publishes no general signing operation, so the key is + // rebuilt from its own PKCS#8 encoding; the encoding is wiped when the + // wrapper drops. + let pkcs8 = key.to_pkcs8_der().map_err(|_| EnrollmentError::ResponseNotProduced)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| EnrollmentError::ResponseNotProduced)?; + + let signature: Signature = signing_key.sign(&signature_input(tag, bytes)); + let canonical = signature.normalize_s().unwrap_or(signature); + + Ok(BASE64_URL_NO_PAD.encode(canonical.to_bytes())) +} + +/// The device's public point, recovered from the DER encoding the identity +/// publishes so that one prefix constant governs both the fingerprint and the +/// wire form. +fn device_public_point(key: &DeviceIdentity) -> Result<[u8; PUBLIC_KEY_OCTETS], EnrollmentError> { + key.public_key_der() + .strip_prefix(&SPKI_PREFIX) + .and_then(|point| <[u8; PUBLIC_KEY_OCTETS]>::try_from(point).ok()) + .ok_or(EnrollmentError::ResponseNotProduced) +} + +/// Decode an uncompressed SEC1 point and check that it is on the curve. +/// +/// The length and alphabet are checked before decoding so that a padded or +/// standard-alphabet spelling is refused, and the point tag is checked so that +/// the compressed and hybrid forms — which no keyId would match — cannot be +/// spelled at all. +fn decode_public_key(value: &str) -> Option<(VerifyingKey, [u8; PUBLIC_KEY_OCTETS])> { + let value = value.as_bytes(); + if value.len() != PUBLIC_KEY_CHARS || !value.iter().all(|byte| is_base64url(*byte)) { + return None; + } + + let point: [u8; PUBLIC_KEY_OCTETS] = BASE64_URL_NO_PAD.decode(value).ok()?.try_into().ok()?; + if point[0] != UNCOMPRESSED_POINT { + return None; + } + + VerifyingKey::from_sec1_bytes(&point).ok().map(|key| (key, point)) +} + +/// Lowercase SHA-256 hex of the DER SubjectPublicKeyInfo built from a 65 octet +/// uncompressed point. +fn key_id(point: &[u8]) -> String { + let mut digest = Sha256::new(); + digest.update(SPKI_PREFIX); + digest.update(point); + hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower) +} + +fn is_base64url(byte: u8) -> bool { + byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_' +} + +/// Parse `YYYY-MM-DDTHH:MM:SSZ` into a Unix instant. +/// +/// The shape is checked before the fields are read: offsets other than `Z` and +/// fractional seconds are refused rather than normalised, so two producers +/// cannot spell the same instant two ways. +fn parse_timestamp(value: &str) -> Result { + let octets = value.as_bytes(); + if octets.len() != TIMESTAMP_CHARS + || octets[4] != b'-' + || octets[7] != b'-' + || octets[10] != b'T' + || octets[13] != b':' + || octets[16] != b':' + || octets[19] != b'Z' + { + return Err(EnrollmentError::MalformedDocument); + } + + let field = |range: std::ops::Range| -> Result { + let text = &value[range]; + if !text.bytes().all(|byte| byte.is_ascii_digit()) { + return Err(EnrollmentError::MalformedDocument); + } + text.parse().map_err(|_| EnrollmentError::MalformedDocument) + }; + + let month = Month::try_from(field(5..7)? as u8).map_err(|_| EnrollmentError::MalformedDocument)?; + let date = Date::from_calendar_date(field(0..4)? as i32, month, field(8..10)? as u8) + .map_err(|_| EnrollmentError::MalformedDocument)?; + let clock = Time::from_hms(field(11..13)? as u8, field(14..16)? as u8, field(17..19)? as u8) + .map_err(|_| EnrollmentError::MalformedDocument)?; + + Ok(PrimitiveDateTime::new(date, clock).assume_utc().unix_timestamp()) +} + +fn format_timestamp(unix: i64) -> Result { + let moment = OffsetDateTime::from_unix_timestamp(unix).map_err(|_| EnrollmentError::ResponseNotProduced)?; + Ok(format!( + "{:04}-{:02}-{:02}T{:02}:{:02}:{:02}Z", + moment.year(), + u8::from(moment.month()), + moment.day(), + moment.hour(), + moment.minute(), + moment.second() + )) +} + +/// `at` must fall within `[issuedAt - 300, expiresAt + 300]`. +/// +/// The declared expiry is capped at the protocol's maximum challenge lifetime +/// because the issuer sets both ends of its own window; a challenge claiming a +/// longer life expires at the bound. +fn check_challenge_window(issued_at: i64, expires_at: i64, at: i64) -> Result<(), EnrollmentError> { + if at < issued_at.saturating_sub(CLOCK_SKEW_TOLERANCE) { + return Err(EnrollmentError::ChallengeNotYetValid); + } + + let effective_expiry = expires_at.min(issued_at.saturating_add(MAX_CHALLENGE_LIFETIME)); + if at > effective_expiry.saturating_add(CLOCK_SKEW_TOLERANCE) { + return Err(EnrollmentError::ChallengeExpired); + } + + Ok(()) +} diff --git a/rustfs/src/connect/offline/key_store.rs b/rustfs/src/connect/offline/key_store.rs new file mode 100644 index 000000000..1f67f7b7e --- /dev/null +++ b/rustfs/src/connect/offline/key_store.rs @@ -0,0 +1,72 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! On-disk home of the offline enrollment key. +//! +//! An air-gapped device enrols with a key that is not its online device +//! identity: the online key is minted during a registration exchange this +//! device cannot perform, and an operator who carries an enrolment response out +//! on removable media is enrolling exactly one key that Connect will pin. Losing +//! it means asking for a fresh challenge, so it is written durably and published +//! exactly once. +//! +//! The durability protocol is not reimplemented here. [`IdentityStore`] already +//! seals a P-256 key at mode 0600, fsyncs it, and publishes it through a +//! no-clobber link so a retry or a concurrent start converges on one key; it is +//! pointed at a directory of this key's own rather than generalised into a +//! key-store abstraction that would have to describe both lifecycles. + +use std::path::{Path, PathBuf}; + +use super::super::identity::DeviceIdentity; +use super::super::identity_store::{IdentityStore, StoreError}; + +/// Subdirectory holding the offline enrolment key, kept apart from the online +/// device identity so neither can be read in place of the other. +const OFFLINE_DIRECTORY: &str = "offline"; + +/// The offline enrolment key of one deployment. +#[derive(Clone, Debug)] +pub struct OfflineKeyStore { + inner: IdentityStore, +} + +impl OfflineKeyStore { + pub fn new(directory: impl AsRef) -> Self { + Self { + inner: IdentityStore::new(directory.as_ref().join(OFFLINE_DIRECTORY)), + } + } + + pub fn key_path(&self) -> PathBuf { + self.inner.key_path() + } + + /// Return the stored key, or `None` when this deployment has never enrolled + /// offline. Reading never creates one, so a deployment that only ever + /// registers online holds no offline key. + pub fn load(&self) -> Result, StoreError> { + self.inner.load() + } + + /// Return the stored key, generating and publishing one the first time. + /// + /// A second enrolment attempt returns the original key rather than minting a + /// replacement: the operator may already be carrying a response for it, and + /// two keys would mean the response and the device disagree about which one + /// Connect pinned. + pub fn load_or_create(&self) -> Result { + self.inner.load_or_create() + } +} diff --git a/rustfs/src/connect/offline/mod.rs b/rustfs/src/connect/offline/mod.rs new file mode 100644 index 000000000..c5364cb0b --- /dev/null +++ b/rustfs/src/connect/offline/mod.rs @@ -0,0 +1,35 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Offline enrolment: joining a Connect tenant without a network. +//! +//! An air-gapped cluster cannot perform the registration exchange, so an +//! operator carries a signed challenge in and a signed response out. The device +//! half of that exchange lives here: verifying the challenge against a root +//! whose fingerprint is compiled into this binary, minting the key being +//! enrolled, and signing the response. +//! +//! Nothing here opens a socket. That is the point of the surface, and it is +//! asserted rather than assumed: the enrolment path takes bytes and returns +//! bytes. +//! +//! The trust model, the signing convention, and every rejection reason are +//! frozen by `protocol/agent/v1/fixtures/offline-enrollment/` and by +//! `docs/adr/0009-offline-signing.md` on the Connect side. + +pub mod enrollment; +pub mod key_store; + +pub use enrollment::{EnrollmentError, OfflineEnrollment, VerifiedChallenge}; +pub use key_store::OfflineKeyStore; diff --git a/rustfs/src/connect/registration.rs b/rustfs/src/connect/registration.rs new file mode 100644 index 000000000..0dbdbe122 --- /dev/null +++ b/rustfs/src/connect/registration.rs @@ -0,0 +1,527 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::io::Read; +use std::sync::Arc; + +use base64::Engine as _; +use base64::engine::general_purpose::URL_SAFE_NO_PAD as BASE64_URL_NO_PAD; +use p256::ecdsa::signature::Signer as _; +use p256::ecdsa::{Signature, SigningKey}; +use p256::pkcs8::DecodePrivateKey as _; +use rustls::RootCertStore; +use rustls::pki_types::{CertificateDer, UnixTime, pem::PemObject as _}; +use rustls::server::WebPkiClientVerifier; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use time::OffsetDateTime; +use time::format_description::well_known::Rfc3339; +use uuid::{Uuid, Version}; +use x509_parser::extensions::GeneralName; +use x509_parser::oid_registry::OID_SIG_ECDSA_WITH_SHA256; +use x509_parser::prelude::{FromDer as _, X509Certificate, X509CertificationRequest}; +use zeroize::Zeroizing; + +use super::credential_store::DeviceCredential; +use super::identity::{DeviceIdentity, RegistrationProof}; + +pub const PROTOCOL_VERSION: &str = "v1"; + +const CERTIFICATE_LIFETIME_SECONDS: i64 = 86_400; +const ROTATION_DOMAIN: &[u8] = b"RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1"; +const MAX_TOKEN_BYTES: u64 = 16 * 1024; + +#[derive(Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct RegistrationTokenDocument { + registration_token_uid: String, + registration_token_secret: String, + organization_uid: String, + cluster_uid: String, + challenge_nonce: String, + expires_unix: i64, +} + +pub struct RegistrationToken { + pub registration_token_uid: String, + registration_token_secret: Zeroizing, + pub organization_uid: String, + pub cluster_uid: String, + pub challenge_nonce: String, + pub expires_unix: i64, +} + +impl std::fmt::Debug for RegistrationToken { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("RegistrationToken") + .field("registration_token_uid", &self.registration_token_uid) + .field("expires_unix", &self.expires_unix) + .finish_non_exhaustive() + } +} + +impl RegistrationToken { + pub fn from_reader(reader: impl Read) -> Result { + let mut bytes = Zeroizing::new(Vec::new()); + reader + .take(MAX_TOKEN_BYTES + 1) + .read_to_end(&mut bytes) + .map_err(TokenError::Read)?; + if bytes.len() as u64 > MAX_TOKEN_BYTES { + return Err(TokenError::TooLarge); + } + let document: RegistrationTokenDocument = serde_json::from_slice(&bytes).map_err(TokenError::Invalid)?; + let decoded = BASE64_URL_NO_PAD + .decode(&document.registration_token_secret) + .map(Zeroizing::new) + .map_err(|_| TokenError::SecretShape)?; + if decoded.len() != 32 || BASE64_URL_NO_PAD.encode(&decoded) != document.registration_token_secret { + return Err(TokenError::SecretShape); + } + if !is_uuid_v7(&document.registration_token_uid) + || !is_uuid_v7(&document.organization_uid) + || !is_uuid_v7(&document.cluster_uid) + || document.expires_unix < 0 + || document.challenge_nonce.len() != 64 + || !document + .challenge_nonce + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err(TokenError::Shape); + } + + Ok(Self { + registration_token_uid: document.registration_token_uid, + registration_token_secret: Zeroizing::new(document.registration_token_secret), + organization_uid: document.organization_uid, + cluster_uid: document.cluster_uid, + challenge_nonce: document.challenge_nonce, + expires_unix: document.expires_unix, + }) + } + + pub(crate) fn secret(&self) -> &str { + &self.registration_token_secret + } +} + +#[derive(Debug, thiserror::Error)] +pub enum TokenError { + #[error("failed to read the Connect registration token")] + Read(#[source] std::io::Error), + #[error("Connect registration token configuration is invalid")] + Invalid(#[source] serde_json::Error), + #[error("Connect registration token secret must be 32-byte unpadded base64url")] + SecretShape, + #[error("Connect registration token configuration exceeds 16 KiB")] + TooLarge, + #[error("Connect registration token fields do not match the protocol schema")] + Shape, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct RegistrationRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, + registration_token_uid: &'a str, + registration_token_secret: &'a str, + certificate_request: &'a str, + proof: ProofRef<'a>, +} + +#[derive(Serialize)] +struct ProofRef<'a> { + algorithm: &'a str, + value: &'a str, +} + +impl<'a> RegistrationRequest<'a> { + pub(crate) fn new( + token: &'a RegistrationToken, + request_id: &'a str, + certificate_request: &'a str, + proof: &'a RegistrationProof, + ) -> Self { + Self { + protocol_version: PROTOCOL_VERSION, + request_id, + registration_token_uid: &token.registration_token_uid, + registration_token_secret: token.secret(), + certificate_request, + proof: ProofRef { + algorithm: &proof.algorithm, + value: &proof.value, + }, + } + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct RotationRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, + certificate_request: &'a str, + proof: ProofOwned, +} + +#[derive(Serialize)] +struct ProofOwned { + algorithm: String, + value: String, +} + +impl<'a> RotationRequest<'a> { + pub(crate) fn new( + identity: &DeviceIdentity, + credential_fingerprint: &str, + device_name: &str, + request_id: &'a str, + certificate_request: &'a str, + ) -> Result { + let csr_der = base64::engine::general_purpose::STANDARD + .decode(certificate_request) + .map_err(|_| CredentialValidationError::CertificateRequest)?; + let csr_digest = BASE64_URL_NO_PAD.encode(Sha256::digest(&csr_der)); + let transcript = rotation_transcript(credential_fingerprint, device_name, request_id, &csr_digest)?; + let key = identity + .to_pkcs8_der() + .map_err(|_| CredentialValidationError::CertificateRequest)?; + let signing_key = SigningKey::from_pkcs8_der(&key).map_err(|_| CredentialValidationError::CertificateRequest)?; + let signature: Signature = signing_key.sign(&transcript); + let canonical = signature.normalize_s().unwrap_or(signature); + + Ok(Self { + protocol_version: PROTOCOL_VERSION, + request_id, + certificate_request, + proof: ProofOwned { + algorithm: "ES256".to_string(), + value: BASE64_URL_NO_PAD.encode(canonical.to_bytes()), + }, + }) + } +} + +fn rotation_transcript( + credential_fingerprint: &str, + device_name: &str, + request_id: &str, + csr_digest: &str, +) -> Result, CredentialValidationError> { + let fields = [credential_fingerprint, device_name, request_id, csr_digest]; + if fields + .iter() + .any(|field| !field.is_ascii() || field.as_bytes().contains(&b'\n')) + { + return Err(CredentialValidationError::RotationTranscript); + } + + let mut transcript = Vec::with_capacity(346); + transcript.extend_from_slice(ROTATION_DOMAIN); + transcript.push(b'\n'); + for field in fields { + transcript.extend_from_slice(field.len().to_string().as_bytes()); + transcript.push(b':'); + transcript.extend_from_slice(field.as_bytes()); + transcript.push(b'\n'); + } + Ok(transcript) +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct CredentialResponse { + pub name: String, + #[serde(default)] + pub uid: String, + #[serde(default)] + pub cluster: String, + pub protocol_version: String, + pub key_id: String, + pub certificate_serial: String, + pub certificate: String, + pub certificate_chain: String, + pub not_before: String, + pub not_after: String, +} + +pub(crate) enum ExpectedDevice<'a> { + Registration { cluster: &'a str }, + Rotation { name: &'a str }, + Stored, +} + +#[derive(Debug, thiserror::Error)] +pub enum CredentialValidationError { + #[error("Connect returned malformed certificate material")] + Certificate, + #[error("Connect returned a certificate chain that is not trusted")] + Chain, + #[error("Connect returned a certificate for the wrong device identity")] + Identity, + #[error("Connect returned a certificate for a different device key")] + Key, + #[error("Connect returned an invalid certificate validity window")] + Validity, + #[error("the device certificate request could not be prepared")] + CertificateRequest, + #[error("the credential rotation transcript contains an invalid field")] + RotationTranscript, +} + +pub(crate) fn validate_credential( + response: CredentialResponse, + identity: &DeviceIdentity, + roots: &RootCertStore, + root_certificates: &[CertificateDer<'static>], + expected: ExpectedDevice<'_>, +) -> Result { + validate_credential_at(response, identity, roots, root_certificates, expected, true) +} + +pub(crate) fn validate_stored_credential( + credential: &DeviceCredential, + identity: &DeviceIdentity, + roots: &RootCertStore, + root_certificates: &[CertificateDer<'static>], +) -> Result<(), CredentialValidationError> { + let not_before = OffsetDateTime::from_unix_timestamp(credential.not_before_unix) + .map_err(|_| CredentialValidationError::Validity)? + .format(&Rfc3339) + .map_err(|_| CredentialValidationError::Validity)?; + let not_after = OffsetDateTime::from_unix_timestamp(credential.not_after_unix) + .map_err(|_| CredentialValidationError::Validity)? + .format(&Rfc3339) + .map_err(|_| CredentialValidationError::Validity)?; + let response = CredentialResponse { + name: credential.name.clone(), + uid: credential.uid.clone(), + cluster: String::new(), + protocol_version: credential.protocol_version.clone(), + key_id: credential.key_id.clone(), + certificate_serial: credential.certificate_serial.clone(), + certificate: credential.certificate.clone(), + certificate_chain: credential.certificate_chain.clone(), + not_before, + not_after, + }; + validate_credential_at(response, identity, roots, root_certificates, ExpectedDevice::Stored, false).map(|_| ()) +} + +fn validate_credential_at( + response: CredentialResponse, + identity: &DeviceIdentity, + roots: &RootCertStore, + root_certificates: &[CertificateDer<'static>], + expected: ExpectedDevice<'_>, + verify_now: bool, +) -> Result { + if response.protocol_version != PROTOCOL_VERSION { + return Err(CredentialValidationError::Identity); + } + + let leaves = CertificateDer::pem_slice_iter(response.certificate.as_bytes()) + .collect::, _>>() + .map_err(|_| CredentialValidationError::Certificate)?; + if leaves.len() != 1 { + return Err(CredentialValidationError::Certificate); + } + let chain = CertificateDer::pem_slice_iter(response.certificate_chain.as_bytes()) + .collect::, _>>() + .map_err(|_| CredentialValidationError::Certificate)?; + if chain.is_empty() + || chain[0].as_ref() != leaves[0].as_ref() + || chain + .iter() + .skip(1) + .any(|certificate| root_certificates.iter().any(|root| root.as_ref() == certificate.as_ref())) + { + return Err(CredentialValidationError::Chain); + } + + let (remaining, certificate) = + X509Certificate::from_der(leaves[0].as_ref()).map_err(|_| CredentialValidationError::Certificate)?; + if !remaining.is_empty() { + return Err(CredentialValidationError::Certificate); + } + + let uid = match expected { + ExpectedDevice::Registration { cluster } => { + if response.uid.is_empty() + || response.cluster != cluster + || response.name != format!("{cluster}/clusterDevices/{}", response.uid) + { + return Err(CredentialValidationError::Identity); + } + response.uid.clone() + } + ExpectedDevice::Rotation { name } => { + if response.name != name || !response.uid.is_empty() || !response.cluster.is_empty() { + return Err(CredentialValidationError::Identity); + } + name.rsplit_once("/clusterDevices/") + .map(|(_, uid)| uid.to_string()) + .ok_or(CredentialValidationError::Identity)? + } + ExpectedDevice::Stored => { + let (cluster, name_uid) = response + .name + .rsplit_once("/clusterDevices/") + .ok_or(CredentialValidationError::Identity)?; + if response.uid != name_uid + || !response.cluster.is_empty() + || !valid_cluster_name(cluster) + || response.name.matches("/clusterDevices/").count() != 1 + { + return Err(CredentialValidationError::Identity); + } + response.uid.clone() + } + }; + let parsed_uid = Uuid::parse_str(&uid).map_err(|_| CredentialValidationError::Identity)?; + if parsed_uid.get_version() != Some(Version::SortRand) || parsed_uid.to_string() != uid { + return Err(CredentialValidationError::Identity); + } + let expected_uri = format!("urn:rustfs:connect:device:{uid}"); + let common_names = certificate + .subject() + .iter_common_name() + .map(|name| name.as_str()) + .collect::, _>>() + .map_err(|_| CredentialValidationError::Identity)?; + let san = certificate + .subject_alternative_name() + .map_err(|_| CredentialValidationError::Identity)? + .ok_or(CredentialValidationError::Identity)?; + let san_matches = matches!(san.value.general_names.as_slice(), [GeneralName::URI(uri)] if *uri == expected_uri); + + if common_names.as_slice() != [uid.as_str()] + || !san_matches + || certificate.subject().iter().count() != 1 + || certificate.subject().iter_attributes().count() != 1 + { + return Err(CredentialValidationError::Identity); + } + if certificate.public_key().raw != identity.public_key_der() { + return Err(CredentialValidationError::Key); + } + + let not_before = OffsetDateTime::parse(&response.not_before, &Rfc3339) + .map_err(|_| CredentialValidationError::Validity)? + .unix_timestamp(); + let not_after = OffsetDateTime::parse(&response.not_after, &Rfc3339) + .map_err(|_| CredentialValidationError::Validity)? + .unix_timestamp(); + let verify_unix = if verify_now { + UnixTime::now() + } else { + let midpoint = certificate.validity().not_before.timestamp() + + (certificate.validity().not_after.timestamp() - certificate.validity().not_before.timestamp()) / 2; + UnixTime::since_unix_epoch(std::time::Duration::from_secs( + midpoint.try_into().map_err(|_| CredentialValidationError::Validity)?, + )) + }; + let verifier = WebPkiClientVerifier::builder(Arc::new(roots.clone())) + .build() + .map_err(|_| CredentialValidationError::Chain)?; + verifier + .verify_client_cert(&leaves[0], &chain[1..], verify_unix) + .map_err(|_| CredentialValidationError::Chain)?; + if not_before != certificate.validity().not_before.timestamp() + || not_after != certificate.validity().not_after.timestamp() + || not_after - not_before != CERTIFICATE_LIFETIME_SECONDS + || certificate.signature_algorithm.algorithm != OID_SIG_ECDSA_WITH_SHA256 + || response.certificate_serial != canonical_serial(certificate.raw_serial())? + || response.key_id != format!("x509-{}", response.certificate_serial) + { + return Err(CredentialValidationError::Validity); + } + + Ok(DeviceCredential { + name: response.name, + uid, + protocol_version: response.protocol_version, + key_id: response.key_id, + certificate_serial: response.certificate_serial, + certificate: response.certificate, + certificate_chain: response.certificate_chain, + not_before_unix: not_before, + not_after_unix: not_after, + }) +} + +pub(crate) fn public_key_fingerprint(identity: &DeviceIdentity) -> String { + hex_lower(&Sha256::digest(identity.public_key_der())) +} + +pub(crate) fn certificate_request_matches(encoded: &str, identity: &DeviceIdentity) -> Result { + let der = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|_| CredentialValidationError::CertificateRequest)?; + let (remaining, request) = + X509CertificationRequest::from_der(&der).map_err(|_| CredentialValidationError::CertificateRequest)?; + Ok(remaining.is_empty() && request.certification_request_info.subject_pki.raw == identity.public_key_der()) +} + +fn canonical_serial(raw: &[u8]) -> Result { + let magnitude = match raw { + [0, first, rest @ ..] if first & 0x80 != 0 => { + if rest.len() + 1 > 16 { + return Err(CredentialValidationError::Validity); + } + &raw[1..] + } + [0] => raw, + [0, ..] | [] => return Err(CredentialValidationError::Validity), + [first, ..] if first & 0x80 != 0 => return Err(CredentialValidationError::Validity), + _ if raw.len() > 16 => return Err(CredentialValidationError::Validity), + _ => raw, + }; + let mut padded = [0u8; 16]; + padded[16 - magnitude.len()..].copy_from_slice(magnitude); + Ok(hex_lower(&padded)) +} + +fn is_uuid_v7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| uuid.get_version() == Some(Version::SortRand) && uuid.to_string() == value) +} + +fn valid_cluster_name(name: &str) -> bool { + let Some((organization, cluster)) = name + .strip_prefix("organizations/") + .and_then(|rest| rest.split_once("/clusters/")) + else { + return false; + }; + !cluster.contains('/') && is_uuid_v7(organization) && is_uuid_v7(cluster) +} + +pub(crate) fn certificate_fingerprint(certificate_pem: &str) -> Result { + let certificate = CertificateDer::pem_slice_iter(certificate_pem.as_bytes()) + .next() + .ok_or(CredentialValidationError::Certificate)? + .map_err(|_| CredentialValidationError::Certificate)?; + Ok(hex_lower(&Sha256::digest(certificate.as_ref()))) +} + +fn hex_lower(bytes: &[u8]) -> String { + bytes.iter().fold(String::with_capacity(bytes.len() * 2), |mut output, byte| { + use std::fmt::Write as _; + let _ = write!(output, "{byte:02x}"); + output + }) +} diff --git a/rustfs/src/connect/runtime.rs b/rustfs/src/connect/runtime.rs new file mode 100644 index 000000000..f0d919a07 --- /dev/null +++ b/rustfs/src/connect/runtime.rs @@ -0,0 +1,156 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::future::Future; +use std::time::Duration; + +use chrono::Utc; +use rand::RngExt as _; +use tokio::sync::watch; +use tokio::task::JoinHandle; +use tokio_util::sync::CancellationToken; + +use super::config::HeartbeatConfig; +use super::heartbeat::{CoarseNodeSummary, Delivery, HeartbeatError, HeartbeatSender, HeartbeatStateStore, HeartbeatStatus}; + +pub struct HeartbeatRuntime { + shutdown: CancellationToken, + status: watch::Receiver, + task: Option>, +} + +impl HeartbeatRuntime { + pub fn status(&self) -> watch::Receiver { + self.status.clone() + } + + pub async fn shutdown(mut self) { + self.shutdown.cancel(); + if let Some(task) = self.task.take() { + let _ = task.await; + } + } +} + +impl Drop for HeartbeatRuntime { + fn drop(&mut self) { + self.shutdown.cancel(); + } +} + +pub fn spawn_heartbeat_runtime( + config: Option, + parent_shutdown: &CancellationToken, + sample: F, +) -> Result, HeartbeatError> +where + F: Fn() -> CoarseNodeSummary + Send + Sync + 'static, +{ + let Some(config) = config else { + return Ok(None); + }; + let sender = HeartbeatSender::new(config.clone())?; + let store = HeartbeatStateStore::new(config.state_path.clone()); + let lock = store.try_runtime_lock()?; + let schedule = config.schedule; + let shutdown = parent_shutdown.child_token(); + let task_shutdown = shutdown.clone(); + let (status_tx, status_rx) = watch::channel(HeartbeatStatus::Starting); + let task = tokio::spawn(async move { + let _lock = lock; + let mut backoff = schedule.initial_backoff; + loop { + if task_shutdown.is_cancelled() { + break; + } + let pending = match store.prepare(sample(), Utc::now()).await { + Ok(pending) => pending, + Err(error) => return failed(&status_tx, error), + }; + let delivery = match cancellable(&task_shutdown, sender.send(&pending)).await { + Some(Ok(delivery)) => delivery, + Some(Err(error)) => return failed(&status_tx, error), + None => break, + }; + let delay = match delivery { + Delivery::Accepted { server_time } => { + if let Err(error) = store.mark_accepted(&pending).await { + return failed(&status_tx, error); + } + backoff = schedule.initial_backoff; + let _ = status_tx.send(HeartbeatStatus::Online { server_time }); + schedule.cadence.saturating_add(jitter(schedule.jitter)) + } + Delivery::Retry { retry_after } => { + let delay = retry_after + .unwrap_or(backoff) + .clamp(schedule.initial_backoff, schedule.max_backoff); + backoff = backoff.saturating_mul(2).min(schedule.max_backoff); + let _ = status_tx.send(HeartbeatStatus::BackingOff { delay }); + delay + } + Delivery::AuthenticationStopped { status, reason } => { + let _ = status_tx.send(HeartbeatStatus::AuthenticationStopped { status, reason }); + return; + } + Delivery::Rejected { status, reason } => { + let suffix = reason.map_or_else(String::new, |reason| format!("; reason={reason}")); + let _ = status_tx.send(HeartbeatStatus::Failed { + reason: format!("Connect rejected heartbeat with HTTP {status}{suffix}"), + }); + return; + } + }; + if sleep_or_cancel(&task_shutdown, delay).await { + break; + } + } + let _ = status_tx.send(HeartbeatStatus::Stopped); + }); + Ok(Some(HeartbeatRuntime { + shutdown, + status: status_rx, + task: Some(task), + })) +} + +fn failed(status: &watch::Sender, error: HeartbeatError) { + let _ = status.send(HeartbeatStatus::Failed { + reason: error.to_string(), + }); +} + +fn jitter(maximum: Duration) -> Duration { + if maximum.is_zero() { + Duration::ZERO + } else { + maximum.mul_f64(rand::rng().random_range(0.0..=1.0)) + } +} + +async fn cancellable(shutdown: &CancellationToken, future: impl Future) -> Option { + tokio::select! { + biased; + () = shutdown.cancelled() => None, + value = future => Some(value), + } +} + +async fn sleep_or_cancel(shutdown: &CancellationToken, delay: Duration) -> bool { + tokio::select! { + biased; + () = shutdown.cancelled() => true, + () = tokio::time::sleep(delay) => false, + } +} diff --git a/rustfs/src/main.rs b/rustfs/src/main.rs index 5e9d5075b..7f4bea1d2 100644 --- a/rustfs/src/main.rs +++ b/rustfs/src/main.rs @@ -26,22 +26,22 @@ struct MiMallocAllocator; unsafe impl GlobalAlloc for MiMallocAllocator { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { // SAFETY: the caller upholds GlobalAlloc's contract for layout. - unsafe { mimalloc::MiMalloc.alloc(layout) } + unsafe { rustfs_mimalloc::MiMalloc.alloc(layout) } } unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 { // SAFETY: the caller upholds GlobalAlloc's contract for layout. - unsafe { mimalloc::MiMalloc.alloc_zeroed(layout) } + unsafe { rustfs_mimalloc::MiMalloc.alloc_zeroed(layout) } } unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { // SAFETY: ptr and layout came from this allocator and are forwarded unchanged. - unsafe { mimalloc::MiMalloc.dealloc(ptr, layout) } + unsafe { rustfs_mimalloc::MiMalloc.dealloc(ptr, layout) } } unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { // SAFETY: ptr and layout came from this allocator and are forwarded unchanged. - unsafe { mimalloc::MiMalloc.realloc(ptr, layout, new_size) } + unsafe { rustfs_mimalloc::MiMalloc.realloc(ptr, layout, new_size) } } } @@ -51,7 +51,7 @@ static GLOBAL: hotpath::CountingAllocator = hotpath::Counting #[cfg(not(all(feature = "hotpath", feature = "hotpath-alloc")))] #[global_allocator] -static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc; +static GLOBAL: rustfs_mimalloc::MiMalloc = rustfs_mimalloc::MiMalloc; fn main() { let _hotpath_guard = hotpath::HotpathGuardBuilder::new("main").build(); @@ -71,8 +71,9 @@ mod tests { allocation.extend_from_slice(&[7_u8; 64]); assert_eq!(allocation.len(), 64); + let heap = rustfs_mimalloc::heap::Heap::main(); // SAFETY: the live Vec pointer is valid to inspect for heap ownership. - assert!(unsafe { libmimalloc_sys::mi_is_in_heap_region(allocation.as_ptr().cast()) }); + assert!(unsafe { heap.contains(allocation.as_ptr()) }); } #[test] @@ -85,12 +86,13 @@ mod tests { let layout = Layout::from_size_align(32, 8).expect("valid test allocation layout"); let grown_layout = Layout::from_size_align(64, 8).expect("valid grown test allocation layout"); let allocator = super::MiMallocAllocator; + let heap = rustfs_mimalloc::heap::Heap::main(); // SAFETY: The pointer is checked for null before use and later released // through the same allocator with the corresponding layout. let ptr = unsafe { allocator.alloc_zeroed(layout) }; assert!(!ptr.is_null()); - assert!(unsafe { libmimalloc_sys::mi_is_in_heap_region(ptr.cast()) }); + assert!(unsafe { heap.contains(ptr) }); assert!(unsafe { std::slice::from_raw_parts(ptr, 32).iter().all(|byte| *byte == 0) }); // SAFETY: `ptr` was allocated by `allocator` with `layout`; on failure @@ -102,7 +104,7 @@ mod tests { panic!("mimalloc realloc failed in allocator smoke test"); } - assert!(unsafe { libmimalloc_sys::mi_is_in_heap_region(grown_ptr.cast()) }); + assert!(unsafe { heap.contains(grown_ptr) }); // SAFETY: `grown_ptr` was reallocated by `allocator` and is released // with the matching grown layout. unsafe { allocator.dealloc(grown_ptr, grown_layout) }; diff --git a/rustfs/src/memory_observability.rs b/rustfs/src/memory_observability.rs index 3e29d24a0..12bd4b3c5 100644 --- a/rustfs/src/memory_observability.rs +++ b/rustfs/src/memory_observability.rs @@ -17,10 +17,7 @@ use rustfs_io_metrics::{ record_cpu_usage, record_memory_usage, record_process_memory_split, }; use serde::Serialize; -#[cfg(any(test, not(target_os = "windows")))] use serde_json::Value; -#[cfg(not(target_os = "windows"))] -use std::ffi::CStr; use std::path::Path; use std::sync::{Arc, Mutex, OnceLock}; use std::time::Duration; @@ -231,7 +228,18 @@ fn read_cgroup_memory_snapshot() -> Option { read_cgroup_v2().or_else(read_cgroup_v1) } -#[cfg(any(test, not(target_os = "windows")))] +fn read_allocator_memory_snapshot() -> Option { + let json = rustfs_mimalloc::MiMalloc::stats_json(); + if json.is_empty() { + return None; + } + let observation = parse_mimalloc_stats_json(&json)?; + Some(AllocatorMemorySnapshot { + backend: crate::allocator_reclaim::allocator_backend(), + observation, + }) +} + fn numeric_json_value(value: &Value) -> Option { match value { Value::Number(number) => number @@ -242,7 +250,6 @@ fn numeric_json_value(value: &Value) -> Option { } } -#[cfg(any(test, not(target_os = "windows")))] fn numeric_json_field(value: &Value, field: &str) -> Option { match value { Value::Object(fields) => fields @@ -254,7 +261,6 @@ fn numeric_json_field(value: &Value, field: &str) -> Option { } } -#[cfg(any(test, not(target_os = "windows")))] fn mimalloc_stat_field(value: &Value, metric: &str, field: &str) -> Option { match value { Value::Object(fields) => { @@ -271,12 +277,10 @@ fn mimalloc_stat_field(value: &Value, metric: &str, field: &str) -> Option } } -#[cfg(any(test, not(target_os = "windows")))] fn mimalloc_stat_current(value: &Value, metric: &str) -> Option { mimalloc_stat_field(value, metric, "current") } -#[cfg(any(test, not(target_os = "windows")))] fn mimalloc_stat_sum(value: &Value, metrics: &[&str], field: &str) -> Option { metrics .iter() @@ -285,7 +289,6 @@ fn mimalloc_stat_sum(value: &Value, metrics: &[&str], field: &str) -> Option 0) } -#[cfg(any(test, not(target_os = "windows")))] fn parse_mimalloc_stats_json(stats_json: &str) -> Option { let value = serde_json::from_str::(stats_json).ok()?; let malloc_metrics = ["malloc_normal", "malloc_huge"]; @@ -312,33 +315,6 @@ fn parse_mimalloc_stats_json(stats_json: &str) -> Option Option { - // SAFETY: `mi_stats_get_json` returns a null-terminated JSON buffer owned by - // mimalloc when called with a null input buffer. The mimalloc API requires - // freeing that buffer with `mi_free`; parsing finishes before the buffer is freed. - let observation = unsafe { - let stats_ptr = libmimalloc_sys::mi_stats_get_json(0, std::ptr::null_mut()); - if stats_ptr.is_null() { - return None; - } - - let observation = CStr::from_ptr(stats_ptr).to_str().ok().and_then(parse_mimalloc_stats_json); - libmimalloc_sys::mi_free(stats_ptr.cast()); - observation? - }; - Some(AllocatorMemorySnapshot { - backend: crate::allocator_reclaim::allocator_backend(), - observation, - }) -} - -#[cfg(target_os = "windows")] -fn read_allocator_memory_snapshot() -> Option { - None -} - fn configured_memory_observability_interval_secs() -> u64 { rustfs_utils::get_env_u64(ENV_MEMORY_OBSERVABILITY_INTERVAL_SECS, DEFAULT_MEMORY_OBSERVABILITY_INTERVAL_SECS).max(1) } @@ -566,6 +542,13 @@ mod tests { assert_eq!(parse_mimalloc_stats_json(r#"{ "allocator": "unknown" }"#), None); } + #[test] + fn read_allocator_memory_snapshot_uses_mimalloc_stats_json() { + let snapshot = super::read_allocator_memory_snapshot(); + #[cfg(not(target_os = "windows"))] + assert!(snapshot.is_some(), "allocator snapshot should be available on non-Windows"); + } + #[test] fn memory_observability_snapshot_reports_disabled_when_metrics_are_disabled() { let snapshot = build_memory_observability_status_snapshot(false, 15, false); diff --git a/rustfs/src/server/readiness.rs b/rustfs/src/server/readiness.rs index d3954b7a7..8a934e06e 100644 --- a/rustfs/src/server/readiness.rs +++ b/rustfs/src/server/readiness.rs @@ -20,6 +20,7 @@ use crate::storage_api::server::readiness::contract::admin::StorageAdminApi; use crate::storage_api::server::readiness::{Endpoint, EndpointServerPools, is_dist_erasure}; #[cfg(test)] use crate::storage_api::server::readiness::{Endpoints, PoolEndpoints}; +use crate::storage_api::startup::shutdown::mark_get_metadata_read_version_coalescing_service_ready; use bytes::Bytes; use http::HeaderValue; use http::{Request as HttpRequest, Response, StatusCode}; @@ -212,6 +213,9 @@ where if readiness_gate_blocks_path(path, &readiness) { return Ok(service_not_ready_response(readiness.current_stage())); } + if !is_probe_path(path) && readiness.is_ready() { + mark_get_metadata_read_version_coalescing_service_ready(); + } let resp = inner.call(req).await?; // System is ready, forward to the actual S3/RPC handlers // Transparently converts any response body into a BoxBody, and then Trace/Cors/Compression continues to work @@ -232,6 +236,7 @@ pub async fn publish_ready_when_runtime_ready( collect_node_readiness, |dependency_readiness| { readiness.mark_stage(rustfs_common::SystemStage::FullReady); + mark_get_metadata_read_version_coalescing_service_ready(); if let Some(state_manager) = state_manager { state_manager.update(ServiceState::Ready); } diff --git a/rustfs/src/startup_lifecycle.rs b/rustfs/src/startup_lifecycle.rs index c4d09872b..ad5681366 100644 --- a/rustfs/src/startup_lifecycle.rs +++ b/rustfs/src/startup_lifecycle.rs @@ -128,6 +128,7 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec } = lifecycle; let StartupServiceRuntime { optional_runtimes, + heartbeat, iam_bootstrap, enable_scanner, } = service_runtime; @@ -162,6 +163,9 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec shutdown_token, ) .await; + if let Some(heartbeat) = heartbeat { + heartbeat.shutdown().await; + } if let Err(err) = event_notifier_reconciler.await { tracing::warn!( target: "rustfs::main::run", diff --git a/rustfs/src/startup_services.rs b/rustfs/src/startup_services.rs index 585e9677a..c07db0f67 100644 --- a/rustfs/src/startup_services.rs +++ b/rustfs/src/startup_services.rs @@ -16,6 +16,7 @@ use crate::site_replication_reconcile::spawn_site_replication_reconcile_task; use crate::storage_api::startup::services::{ECStore, EndpointServerPools, ServerContextSlot}; use crate::{ config::Config, + connect::{CoarseNodeSummary, HeartbeatConfig, HeartbeatRuntime, spawn_heartbeat_runtime}, init::{init_buffer_profile_system, init_kms_system}, server::ServiceStateManager, startup_audit::init_audit_runtime, @@ -35,6 +36,7 @@ use tokio_util::sync::CancellationToken; pub(crate) struct StartupServiceRuntime { pub(crate) optional_runtimes: OptionalRuntimeServices, + pub(crate) heartbeat: Option, pub(crate) iam_bootstrap: IamBootstrapDisposition, pub(crate) enable_scanner: bool, } @@ -73,6 +75,8 @@ pub(crate) async fn init_startup_runtime_services( init_kms_system(config).await?; let optional_runtimes = init_optional_runtime_services().await?; + let heartbeat_config = HeartbeatConfig::from_env().map_err(std::io::Error::other)?; + let heartbeat_nodes = heartbeat_config.as_ref().map(|_| endpoint_pools.get_nodes().len()); init_buffer_profile_system(config); init_deadlock_detector_runtime(); @@ -92,10 +96,27 @@ pub(crate) async fn init_startup_runtime_services( init_notification_runtime(endpoint_pools, buckets).await?; let enable_scanner = init_background_service_runtime(store.clone()).await?; init_observability_runtime(store.clone(), ctx.clone()).await; + let heartbeat = start_heartbeat_runtime(heartbeat_config, heartbeat_nodes, &ctx)?; Ok(StartupServiceRuntime { optional_runtimes, + heartbeat, iam_bootstrap, enable_scanner, }) } + +fn start_heartbeat_runtime( + config: Option, + node_count: Option, + shutdown: &CancellationToken, +) -> Result> { + let Some(config) = config else { + return Ok(None); + }; + let summary = u16::try_from(node_count.unwrap_or_default()) + .ok() + .and_then(|total| CoarseNodeSummary::new(total, 0, 0).ok()) + .ok_or_else(|| std::io::Error::other("Connect heartbeat node count is outside protocol bounds"))?; + spawn_heartbeat_runtime(Some(config), shutdown, move || summary).map_err(std::io::Error::other) +} diff --git a/rustfs/src/storage/AGENTS.md b/rustfs/src/storage/AGENTS.md index 7817bad24..f9d115519 100644 --- a/rustfs/src/storage/AGENTS.md +++ b/rustfs/src/storage/AGENTS.md @@ -25,4 +25,3 @@ Applies to `rustfs/src/storage/`. ## Suggested Validation - Targeted module tests in `rustfs/src/storage/*_test.rs` -- Full gate before commit: `make pre-commit` diff --git a/rustfs/src/storage/rpc/node_service/disk.rs b/rustfs/src/storage/rpc/node_service/disk.rs index 5d9fbc969..617d0b80d 100644 --- a/rustfs/src/storage/rpc/node_service/disk.rs +++ b/rustfs/src/storage/rpc/node_service/disk.rs @@ -23,10 +23,12 @@ use bytes::Bytes; use rustfs_filemeta::FileInfo; use rustfs_io_metrics::internode_metrics::{ INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_REQUEST, - INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_OPERATION_GRPC_WRITE_ALL, - INTERNODE_STAGE_READ_VERSION_DISK_READ, INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE, - INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE, INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE, - INTERNODE_TRANSPORT_BACKEND_GRPC, global_internode_metrics, + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_VERSION, + INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_STAGE_BATCH_READ_VERSION_DISK_READ, + INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_DECODE, INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_JSON_ENCODE, + INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_MSGPACK_ENCODE, INTERNODE_STAGE_READ_VERSION_DISK_READ, + INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE, INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE, + INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE, INTERNODE_TRANSPORT_BACKEND_GRPC, global_internode_metrics, }; use rustfs_protos::proto_gen::node_service::*; use serde::de::DeserializeOwned; @@ -146,6 +148,29 @@ fn encode_file_info_msgpack(value: &FileInfo) -> std::result::Result, Di encode_msgpack_with_capacity(value, "FileInfo", FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT) } +fn encode_delete_versions_errors(disk_errors: Vec>) -> (Vec, Vec) { + let mut errors = Vec::with_capacity(disk_errors.len()); + let mut item_errors = Vec::with_capacity(disk_errors.len()); + for error in disk_errors { + match error { + Some(error) => { + let code = match &error { + DiskError::Io(source) if source.kind() == std::io::ErrorKind::NotFound => DiskError::FileNotFound.to_u32(), + _ => error.to_u32(), + }; + let error_info = error.to_string(); + errors.push(error_info.clone()); + item_errors.push(Error { code, error_info }); + } + None => { + errors.push(String::new()); + item_errors.push(Error::default()); + } + } + } + (errors, item_errors) +} + fn encode_msgpack_named(value: &T, value_name: &str) -> std::result::Result, DiskError> { let mut serializer = rmp_serde::Serializer::new(Vec::with_capacity(MSGPACK_ENCODE_CAPACITY_HINT)).with_struct_map(); value @@ -219,24 +244,42 @@ fn record_read_version_stage(stage: &'static str, started_at: Option) { } } +fn record_batch_read_version_stage(stage: &'static str, started_at: Option) { + if let Some(started_at) = started_at { + global_internode_metrics().record_stage_duration_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + stage, + started_at.elapsed(), + ); + } +} + fn encode_batch_read_version_response_payloads( batch_read_version_resps: &[BatchReadVersionResp], request_decoded_from_msgpack: bool, ) -> std::result::Result<(Vec, Vec), DiskError> { + let attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); let mut batch_read_version_resps_json = Vec::with_capacity(batch_read_version_resps.len()); - let mut batch_read_version_resps_bin = Vec::with_capacity(batch_read_version_resps.len()); - + let json_encode_started = internode_stage_timer(attribution_enabled); for batch_read_version_resp in batch_read_version_resps { batch_read_version_resps_json.push( compat_response_json(batch_read_version_resp, request_decoded_from_msgpack) .map_err(|err| DiskError::other(format!("encode BatchReadVersionResp json failed: {err}")))?, ); + } + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_JSON_ENCODE, json_encode_started); + + let mut batch_read_version_resps_bin = Vec::with_capacity(batch_read_version_resps.len()); + let msgpack_encode_started = internode_stage_timer(attribution_enabled); + for batch_read_version_resp in batch_read_version_resps { batch_read_version_resps_bin.push(Bytes::from(encode_msgpack_with_capacity( batch_read_version_resp, "BatchReadVersionResp", FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT, )?)); } + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_MSGPACK_ENCODE, msgpack_encode_started); Ok((batch_read_version_resps_json, batch_read_version_resps_bin)) } @@ -462,15 +505,37 @@ impl NodeService { &self, request: Request, ) -> Result, Status> { + let attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); let request = request.into_inner(); + if attribution_enabled { + let metrics = global_internode_metrics(); + metrics.record_incoming_request_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + ); + metrics.record_recv_bytes_for_operation_and_backend( + INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, + INTERNODE_TRANSPORT_BACKEND_GRPC, + request + .disk + .len() + .saturating_add(request.batch_read_version_req.len()) + .saturating_add(request.batch_read_version_req_bin.len()), + ); + } if let Some(disk) = self.find_disk(&request.disk).await { + let decode_started = internode_stage_timer(attribution_enabled); let decoded_batch_read_version_req: DecodedRpcPayload = match decode_msgpack_or_json_with_source( &request.batch_read_version_req_bin, &request.batch_read_version_req, "BatchReadVersionReq", ) { - Ok(batch_read_version_req) => batch_read_version_req, + Ok(batch_read_version_req) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_DECODE, decode_started); + batch_read_version_req + } Err(err) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_DECODE, decode_started); return Ok(Response::new(BatchReadVersionResponse { success: false, batch_read_version_resps: Vec::new(), @@ -491,8 +556,10 @@ impl NodeService { })); } + let disk_read_started = internode_stage_timer(attribution_enabled); match disk.batch_read_version(batch_read_version_req).await { Ok(batch_read_version_resps) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_DISK_READ, disk_read_started); let (batch_read_version_resps, batch_read_version_resps_bin) = match encode_batch_read_version_response_payloads(&batch_read_version_resps, request_decoded_from_msgpack) { @@ -514,12 +581,15 @@ impl NodeService { error: None, })) } - Err(err) => Ok(Response::new(BatchReadVersionResponse { - success: false, - batch_read_version_resps: Vec::new(), - batch_read_version_resps_bin: Vec::new(), - error: Some(err.into()), - })), + Err(err) => { + record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_DISK_READ, disk_read_started); + Ok(Response::new(BatchReadVersionResponse { + success: false, + batch_read_version_resps: Vec::new(), + batch_read_version_resps_bin: Vec::new(), + error: Some(err.into()), + })) + } } } else { Ok(Response::new(BatchReadVersionResponse { @@ -552,6 +622,7 @@ impl NodeService { success: false, errors: Vec::new(), error: Some(DiskError::other(format!("decode FileInfoVersions failed: {err}")).into()), + item_errors: Vec::new(), })); } }; @@ -563,30 +634,26 @@ impl NodeService { success: false, errors: Vec::new(), error: Some(DiskError::other(format!("decode DeleteOptions failed: {err}")).into()), + item_errors: Vec::new(), })); } }; - let errors = disk - .delete_versions(&request.volume, versions, opts) - .await - .into_iter() - .map(|error| match error { - Some(e) => e.to_string(), - None => "".to_string(), - }) - .collect(); + let (errors, item_errors) = + encode_delete_versions_errors(disk.delete_versions(&request.volume, versions, opts).await); Ok(Response::new(DeleteVersionsResponse { success: true, errors, error: None, + item_errors, })) } else { Ok(Response::new(DeleteVersionsResponse { success: false, errors: Vec::new(), error: Some(DiskError::other("cannot find disk".to_string()).into()), + item_errors: Vec::new(), })) } } @@ -702,9 +769,9 @@ impl NodeService { &self, request: Request, ) -> Result, Status> { - let request = request.into_inner(); let metrics = global_internode_metrics(); let read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); + let request = request.into_inner(); if read_version_attribution_enabled { metrics.record_incoming_request_for_operation_and_backend( INTERNODE_OPERATION_GRPC_READ_VERSION, @@ -1612,9 +1679,10 @@ impl NodeService { mod tests { use super::{ compat_response_json, decode_msgpack_or_json, decode_rename_data_request_file_info, - encode_batch_read_version_response_payloads, encode_file_info_msgpack, encode_msgpack, encode_msgpack_named, - encode_read_multiple_response_payloads, encode_rename_data_response_payloads, + encode_batch_read_version_response_payloads, encode_delete_versions_errors, encode_file_info_msgpack, encode_msgpack, + encode_msgpack_named, encode_read_multiple_response_payloads, encode_rename_data_response_payloads, }; + use crate::storage::DiskError; use crate::storage::rpc::node_service::make_server; use crate::storage::storage_api::ReadMultipleResp; use crate::storage::storage_api::RenameDataResp; @@ -1632,6 +1700,18 @@ mod tests { count: u32, } + #[test] + fn delete_versions_response_dual_writes_typed_item_errors() { + let raw_not_found = super::DiskError::Io(std::io::Error::from(std::io::ErrorKind::NotFound)); + let (errors, item_errors) = encode_delete_versions_errors(vec![Some(raw_not_found), None]); + + assert!(errors[0].starts_with("io error ")); + assert!(errors[1].is_empty()); + assert_eq!(item_errors[0].code, super::DiskError::FileNotFound.to_u32()); + assert_eq!(item_errors[0].error_info, errors[0]); + assert_eq!(item_errors[1].code, 0); + } + #[tokio::test] #[serial] async fn handle_read_version_records_attribution_for_missing_disk() { @@ -1996,7 +2076,8 @@ mod tests { path: "object-a".to_string(), version_id: "version-a".to_string(), success: false, - error: "file version not found".to_string(), + error: DiskError::FileVersionNotFound.to_string(), + error_code: DiskError::FileVersionNotFound.to_u32(), ..Default::default() }]; @@ -2012,8 +2093,43 @@ mod tests { .expect("msgpack batch read version response should decode"); assert_eq!(json_decoded.index, responses[0].index); + assert_eq!(json_decoded.error_code, responses[0].error_code); assert_eq!(msgpack_decoded.path, responses[0].path); assert_eq!(msgpack_decoded.error, responses[0].error); + assert_eq!(msgpack_decoded.error_code, responses[0].error_code); + } + + #[test] + fn batch_read_version_response_decode_accepts_legacy_payload_without_error_code() { + #[derive(Serialize)] + struct LegacyBatchReadVersionResp { + index: usize, + path: String, + version_id: String, + success: bool, + file_info: FileInfo, + error: String, + } + + let legacy = LegacyBatchReadVersionResp { + index: 2, + path: "object-legacy".to_string(), + version_id: "version-legacy".to_string(), + success: false, + file_info: FileInfo::default(), + error: "legacy error".to_string(), + }; + let legacy_json = serde_json::to_string(&legacy).expect("legacy json should encode"); + let legacy_msgpack = encode_msgpack(&legacy, "LegacyBatchReadVersionResp").expect("legacy msgpack should encode"); + + let json_decoded: BatchReadVersionResp = + decode_msgpack_or_json(&[], &legacy_json, "BatchReadVersionResp").expect("legacy json should decode"); + let msgpack_decoded: BatchReadVersionResp = + decode_msgpack_or_json(&legacy_msgpack, "", "BatchReadVersionResp").expect("legacy msgpack should decode"); + + assert_eq!(json_decoded.error_code, 0); + assert_eq!(msgpack_decoded.error_code, 0); + assert_eq!(msgpack_decoded.error, legacy.error); } #[test] diff --git a/rustfs/src/storage/s3_api/bucket.rs b/rustfs/src/storage/s3_api/bucket.rs index e105adba4..6599ca107 100644 --- a/rustfs/src/storage/s3_api/bucket.rs +++ b/rustfs/src/storage/s3_api/bucket.rs @@ -296,7 +296,10 @@ pub(crate) fn build_list_objects_v2_output( let mut obj = Object { key: Some(key), last_modified: v.mod_time.map(Timestamp::from), - size: Some(v.get_actual_size().unwrap_or_default()), + // Compressed legacy objects may retain an unknown (-1) + // logical-size sentinel; never expose that internal value in + // an S3 response. + size: Some(v.get_actual_size_or_physical()), e_tag: v.etag.clone().map(|etag| to_s3s_etag(&etag)), storage_class: v.storage_class.clone().map(ObjectStorageClass::from), ..Default::default() @@ -656,6 +659,45 @@ mod tests { assert_eq!(output.common_prefixes.as_ref().map(std::vec::Vec::len), Some(2)); } + #[test] + fn list_objects_never_exposes_compressed_unknown_size_sentinel() { + let mut metadata = std::collections::HashMap::new(); + rustfs_utils::http::insert_str( + &mut metadata, + rustfs_utils::http::SUFFIX_COMPRESSION, + "klauspost/compress/s2".to_string(), + ); + let output = build_list_objects_v2_output( + ListObjectsV2Info { + objects: vec![ObjectInfo { + name: "legacy-compressed".to_string(), + size: 128, + actual_size: -1, + user_defined: std::sync::Arc::new(metadata), + ..Default::default() + }], + ..Default::default() + }, + false, + 1000, + "bucket".to_string(), + String::new(), + None, + None, + None, + None, + ); + + assert_eq!( + output + .contents + .as_ref() + .and_then(|objects| objects.first()) + .and_then(|object| object.size), + Some(128) + ); + } + #[test] fn list_responses_report_standard_for_legacy_label_only_file_metadata() { let version_id = Uuid::parse_str("11111111-2222-3333-4444-555555555555").expect("fixture version ID should be valid"); diff --git a/rustfs/src/storage/storage_api.rs b/rustfs/src/storage/storage_api.rs index 31668783a..512e255ee 100644 --- a/rustfs/src/storage/storage_api.rs +++ b/rustfs/src/storage/storage_api.rs @@ -429,6 +429,8 @@ pub(crate) mod ecstore_config { } pub(crate) mod ecstore_data_usage { + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::data_usage::get_bucket_usage_memory; pub(crate) use rustfs_ecstore::api::data_usage::{ apply_bucket_usage_memory_overlay, init_compression_total_memory_from_backend, load_admin_data_usage_from_backend_cached, load_data_usage_from_backend, quota_object_size, record_bucket_delete_marker_memory, record_bucket_object_delete_memory, @@ -1098,6 +1100,10 @@ pub(crate) fn shutdown_background_monitors() { rustfs_ecstore::shutdown_background_monitors(); } +pub(crate) fn mark_get_metadata_read_version_coalescing_service_ready() { + rustfs_ecstore::mark_get_metadata_read_version_coalescing_service_ready(); +} + pub(crate) fn set_global_rustfs_port(value: u16) { ecstore_global::set_global_rustfs_port(value); } diff --git a/rustfs/src/storage_api.rs b/rustfs/src/storage_api.rs index e90c1f528..dea2923ac 100644 --- a/rustfs/src/storage_api.rs +++ b/rustfs/src/storage_api.rs @@ -284,7 +284,8 @@ pub(crate) mod startup { pub(crate) mod shutdown { pub(crate) use crate::storage::storage_api::{ - shutdown_background_monitors, shutdown_background_services, store_compression_total_in_backend, + mark_get_metadata_read_version_coalescing_service_ready, shutdown_background_monitors, shutdown_background_services, + store_compression_total_in_backend, }; } diff --git a/rustfs/tests/connect_heartbeat.rs b/rustfs/tests/connect_heartbeat.rs new file mode 100644 index 000000000..9414cce46 --- /dev/null +++ b/rustfs/tests/connect_heartbeat.rs @@ -0,0 +1,687 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::VecDeque; +use std::fs; +use std::path::Path; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use bytes::Bytes; +use http_body_util::{BodyExt as _, Full}; +use hyper::service::service_fn; +use hyper::{Request, Response, StatusCode}; +use hyper_util::rt::TokioIo; +use rcgen::{ + BasicConstraints, CertificateParams, DistinguishedName, DnType, ExtendedKeyUsagePurpose, IsCa, Issuer, KeyPair, + KeyUsagePurpose, SanType, +}; +use rustfs::connect::{ + CoarseNodeSummary, CredentialStore, DeviceCredential, HeartbeatConfig, HeartbeatSchedule, HeartbeatStatus, IdentityStore, + spawn_heartbeat_runtime, +}; +use rustls::RootCertStore; +use rustls::pki_types::{CertificateDer, PrivateKeyDer, PrivatePkcs8KeyDer}; +use rustls::server::WebPkiClientVerifier; +use serde_json::{Value, json}; +use time::OffsetDateTime; +use tokio::net::TcpListener; +use tokio::sync::watch; +use tokio_rustls::TlsAcceptor; +use tokio_util::sync::CancellationToken; + +const ORGANIZATION_UID: &str = "0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70"; +const CLUSTER_UID: &str = "0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81"; +const DEVICE_UID: &str = "0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92"; + +struct TestPki { + root_params: CertificateParams, + root_key: KeyPair, + root_der: CertificateDer<'static>, + root_pem: String, + server_der: CertificateDer<'static>, + server_key: PrivatePkcs8KeyDer<'static>, +} + +impl TestPki { + fn new() -> Self { + let now = OffsetDateTime::now_utc(); + let root_key = KeyPair::generate().expect("generate root key"); + let mut root_params = CertificateParams::default(); + root_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + root_params.not_before = now - time::Duration::days(30); + root_params.not_after = now + time::Duration::days(30); + root_params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::DigitalSignature]; + let root = root_params.self_signed(&root_key).expect("sign root"); + + let server_key = KeyPair::generate().expect("generate server key"); + let mut server_params = CertificateParams::default(); + server_params.not_before = now - time::Duration::hours(1); + server_params.not_after = now + time::Duration::days(2); + server_params + .subject_alt_names + .push(SanType::DnsName("localhost".try_into().expect("valid DNS name"))); + server_params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ServerAuth]; + let server = server_params + .signed_by(&server_key, &Issuer::from_params(&root_params, &root_key)) + .expect("sign server certificate"); + Self { + root_params, + root_key, + root_der: root.der().clone(), + root_pem: root.pem(), + server_der: server.der().clone(), + server_key: PrivatePkcs8KeyDer::from(server_key.serialize_der()), + } + } + + fn server_config(&self) -> rustls::ServerConfig { + let mut roots = RootCertStore::empty(); + roots.add(self.root_der.clone()).expect("add client root"); + let verifier = WebPkiClientVerifier::builder(Arc::new(roots)) + .build() + .expect("client verifier"); + rustls::ServerConfig::builder() + .with_client_cert_verifier(verifier) + .with_single_cert(vec![self.server_der.clone()], PrivateKeyDer::Pkcs8(self.server_key.clone_key())) + .expect("server TLS") + } + + fn stores(&self, temp: &tempfile::TempDir) -> (IdentityStore, CredentialStore) { + let now = OffsetDateTime::now_utc(); + self.stores_with_certificate(temp, now - time::Duration::hours(1), now + time::Duration::hours(23), true) + } + + fn stores_with_certificate( + &self, + temp: &tempfile::TempDir, + not_before: OffsetDateTime, + not_after: OffsetDateTime, + bind_identity: bool, + ) -> (IdentityStore, CredentialStore) { + let identity_store = IdentityStore::new(temp.path().join("identity")); + let identity = identity_store.load_or_create().expect("create identity"); + let private_key = PrivatePkcs8KeyDer::from(identity.to_pkcs8_der().expect("serialize key").to_vec()); + let device_key = if bind_identity { + KeyPair::from_pkcs8_der_and_sign_algo(&private_key, &rcgen::PKCS_ECDSA_P256_SHA256).expect("device key") + } else { + KeyPair::generate().expect("mismatched device key") + }; + let mut params = CertificateParams::default(); + params.not_before = not_before; + params.not_after = not_after; + params.serial_number = Some(vec![1; 16].into()); + params.key_usages = vec![KeyUsagePurpose::DigitalSignature]; + params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ClientAuth]; + params.distinguished_name = DistinguishedName::new(); + params.distinguished_name.push(DnType::CommonName, DEVICE_UID); + params.subject_alt_names.push(SanType::URI( + format!("urn:rustfs:connect:device:{DEVICE_UID}") + .try_into() + .expect("device URI"), + )); + let certificate = params + .signed_by(&device_key, &Issuer::from_params(&self.root_params, &self.root_key)) + .expect("device certificate"); + let cluster = format!("organizations/{ORGANIZATION_UID}/clusters/{CLUSTER_UID}"); + let credential = DeviceCredential { + name: format!("{cluster}/clusterDevices/{DEVICE_UID}"), + uid: DEVICE_UID.to_owned(), + protocol_version: "v1".to_owned(), + key_id: format!("x509-{}", "01".repeat(16)), + certificate_serial: "01".repeat(16), + certificate: certificate.pem(), + certificate_chain: certificate.pem(), + not_before_unix: not_before.unix_timestamp(), + not_after_unix: not_after.unix_timestamp(), + }; + let directory = temp.path().join("credential"); + fs::create_dir_all(&directory).expect("credential directory"); + let path = directory.join("device.crt.json"); + fs::write(&path, serde_json::to_vec(&credential).expect("credential JSON")).expect("write credential"); + private_mode(&path); + (identity_store, CredentialStore::new(directory)) + } +} + +#[derive(Clone)] +struct Reply { + status: StatusCode, + body: Value, + retry_after: Option<&'static str>, + delay: Duration, +} + +impl Reply { + fn ok(time: &str) -> Self { + Self { + status: StatusCode::OK, + body: json!({ + "serverTime": time, + "acceptedVersion": "v1", + "capabilityHints": [], + "futureField": true + }), + retry_after: None, + delay: Duration::ZERO, + } + } + + fn error(status: StatusCode) -> Self { + Self { + status, + body: json!({"details": []}), + retry_after: None, + delay: Duration::ZERO, + } + } +} + +struct TestServer { + endpoint: String, + seen: Arc>>, + task: tokio::task::JoinHandle<()>, +} + +impl Drop for TestServer { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn server(pki: &TestPki, replies: Vec) -> TestServer { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind server"); + let address = listener.local_addr().expect("server address"); + let acceptor = TlsAcceptor::from(Arc::new(pki.server_config())); + let replies = Arc::new(Mutex::new(VecDeque::from(replies))); + let seen = Arc::new(Mutex::new(Vec::new())); + let captured = seen.clone(); + let task = tokio::spawn(async move { + while let Ok((stream, _)) = listener.accept().await { + let acceptor = acceptor.clone(); + let replies = replies.clone(); + let seen = captured.clone(); + tokio::spawn(async move { + let Ok(stream) = acceptor.accept(stream).await else { return }; + let service = service_fn(move |request: Request| { + let replies = replies.clone(); + let seen = seen.clone(); + async move { + assert_eq!(request.uri().path(), format!("/agent/clusters/{CLUSTER_UID}/heartbeats")); + let body = request.into_body().collect().await.expect("request body").to_bytes(); + seen.lock() + .expect("seen lock") + .push(serde_json::from_slice(&body).expect("request JSON")); + let reply = replies + .lock() + .expect("reply lock") + .pop_front() + .unwrap_or_else(|| Reply::error(StatusCode::SERVICE_UNAVAILABLE)); + if !reply.delay.is_zero() { + tokio::time::sleep(reply.delay).await; + } + let mut builder = Response::builder() + .status(reply.status) + .header("content-type", "application/json"); + if let Some(value) = reply.retry_after { + builder = builder.header("retry-after", value); + } + Ok::<_, hyper::Error>( + builder + .body(Full::new(Bytes::from(serde_json::to_vec(&reply.body).expect("reply JSON")))) + .expect("reply"), + ) + } + }); + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection(TokioIo::new(stream), service) + .await; + }); + } + }); + TestServer { + endpoint: format!("https://localhost:{}/agent/", address.port()), + seen, + task, + } +} + +fn config(temp: &tempfile::TempDir, pki: &TestPki, server: &TestServer) -> HeartbeatConfig { + let (identity_store, credential_store) = pki.stores(temp); + config_with_stores(temp, pki, server, identity_store, credential_store) +} + +fn config_with_stores( + temp: &tempfile::TempDir, + pki: &TestPki, + server: &TestServer, + identity_store: IdentityStore, + credential_store: CredentialStore, +) -> HeartbeatConfig { + HeartbeatConfig { + endpoint: server.endpoint.clone(), + root_ca_pem: pki.root_pem.as_bytes().to_vec(), + identity_store, + credential_store, + state_path: temp.path().join("heartbeat/state.json"), + schedule: HeartbeatSchedule { + cadence: Duration::from_millis(40), + jitter: Duration::ZERO, + timeout: Duration::from_millis(200), + initial_backoff: Duration::from_millis(20), + max_backoff: Duration::from_millis(80), + }, + } +} + +fn rewrite_credential(temp: &tempfile::TempDir, update: impl FnOnce(&mut DeviceCredential)) { + let path = temp.path().join("credential/device.crt.json"); + let mut credential: DeviceCredential = + serde_json::from_slice(&fs::read(&path).expect("read credential")).expect("parse credential"); + update(&mut credential); + fs::write(&path, serde_json::to_vec(&credential).expect("credential JSON")).expect("rewrite credential"); + private_mode(&path); +} + +fn summary() -> CoarseNodeSummary { + CoarseNodeSummary::new(8, 7, 1).expect("node summary") +} + +async fn wait_for( + status: &mut watch::Receiver, + predicate: impl Fn(&HeartbeatStatus) -> bool, +) -> HeartbeatStatus { + tokio::time::timeout(Duration::from_secs(3), async { + loop { + let current = status.borrow_and_update().clone(); + if predicate(¤t) { + return current; + } + status.changed().await.expect("status channel"); + } + }) + .await + .expect("heartbeat status timeout") +} + +async fn assert_credential_failure(config: HeartbeatConfig, server: &TestServer, expected: &str) { + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + assert!(matches!( + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::Failed { .. })).await, + HeartbeatStatus::Failed { reason } if reason.contains(expected) + )); + assert!(server.seen.lock().expect("seen lock").is_empty()); + runtime.shutdown().await; +} + +#[tokio::test] +async fn connect_config_absent_starts_no_task() { + let shutdown = CancellationToken::new(); + let calls = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let sampled = calls.clone(); + let runtime = spawn_heartbeat_runtime(None, &shutdown, move || { + sampled.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + summary() + }) + .expect("absent config"); + + assert!(runtime.is_none()); + tokio::task::yield_now().await; + assert_eq!(calls.load(std::sync::atomic::Ordering::Relaxed), 0); +} + +#[tokio::test] +async fn duplicate_runtime_is_rejected_without_a_second_task() { + let pki = TestPki::new(); + let mut reply = Reply::ok("2026-08-22T01:02:03Z"); + reply.delay = Duration::from_secs(5); + let server = server(&pki, vec![reply]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let config = config(&temp, &pki, &server); + let runtime = spawn_heartbeat_runtime(Some(config.clone()), &shutdown, summary) + .expect("first runtime") + .expect("configured runtime"); + + assert!(matches!( + spawn_heartbeat_runtime(Some(config), &shutdown, summary), + Err(rustfs::connect::HeartbeatError::AlreadyRunning) + )); + runtime.shutdown().await; +} + +#[tokio::test(flavor = "current_thread")] +async fn dropped_runtime_keeps_the_lock_until_its_task_stops() { + let pki = TestPki::new(); + let server = server(&pki, vec![Reply::ok("2026-08-22T01:02:03Z")]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let config = config(&temp, &pki, &server); + let runtime = spawn_heartbeat_runtime(Some(config.clone()), &shutdown, summary) + .expect("first runtime") + .expect("configured runtime"); + + drop(runtime); + assert!(matches!( + spawn_heartbeat_runtime(Some(config.clone()), &shutdown, summary), + Err(rustfs::connect::HeartbeatError::AlreadyRunning) + )); + + let replacement = tokio::time::timeout(Duration::from_secs(3), async { + loop { + match spawn_heartbeat_runtime(Some(config.clone()), &shutdown, summary) { + Ok(Some(runtime)) => break runtime, + Err(rustfs::connect::HeartbeatError::AlreadyRunning) => tokio::task::yield_now().await, + Ok(None) => panic!("configured replacement returned no runtime"), + Err(error) => panic!("unexpected replacement error: {error}"), + } + } + }) + .await + .expect("dropped runtime releases its lock after stopping"); + replacement.shutdown().await; +} + +#[tokio::test] +async fn corrupt_persisted_state_is_rejected_before_network_delivery() { + let pki = TestPki::new(); + let server = server(&pki, vec![Reply::ok("2026-08-22T01:02:03Z")]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let config = config(&temp, &pki, &server); + let directory = config.state_path.parent().expect("state directory"); + fs::create_dir_all(directory).expect("create state directory"); + fs::write( + &config.state_path, + br#"{"nextSequence":0,"pending":{"protocolVersion":"v1","requestId":"550e8400-e29b-41d4-a716-446655440000","agentVersion":"rustfs-agent/1.0.0-rc.3","capabilities":["heartbeat"],"sequence":0,"clientTime":"2026-08-22T01:02:03Z","coarseNodeSummary":{"total":0,"healthy":0,"degraded":0}}}"#, + ) + .expect("write corrupt state"); + private_mode(&config.state_path); + let runtime = spawn_heartbeat_runtime(Some(config), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + + assert!(matches!( + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::Failed { .. })).await, + HeartbeatStatus::Failed { reason } if reason.contains("violates the protocol invariants") + )); + assert!(server.seen.lock().expect("seen lock").is_empty()); + runtime.shutdown().await; +} + +#[tokio::test] +async fn invalid_stored_resource_name_is_rejected_before_network_delivery() { + let pki = TestPki::new(); + let server = server(&pki, vec![]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let config = config(&temp, &pki, &server); + rewrite_credential(&temp, |credential| { + credential.name = format!("organizations/{ORGANIZATION_UID}/clusters/not-a-uuid/clusterDevices/{DEVICE_UID}"); + }); + + assert_credential_failure(config, &server, "wrong device identity").await; +} + +#[tokio::test] +async fn invalid_stored_protocol_is_rejected_before_network_delivery() { + let pki = TestPki::new(); + let server = server(&pki, vec![]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let config = config(&temp, &pki, &server); + rewrite_credential(&temp, |credential| credential.protocol_version = "v2".to_owned()); + + assert_credential_failure(config, &server, "wrong device identity").await; +} + +#[tokio::test] +async fn stored_certificate_key_mismatch_is_rejected_before_network_delivery() { + let pki = TestPki::new(); + let server = server(&pki, vec![]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let now = OffsetDateTime::now_utc(); + let (identity_store, credential_store) = + pki.stores_with_certificate(&temp, now - time::Duration::hours(1), now + time::Duration::hours(23), false); + let config = config_with_stores(&temp, &pki, &server, identity_store, credential_store); + + assert_credential_failure(config, &server, "different device key").await; +} + +#[tokio::test] +async fn expired_stored_certificate_is_rejected_before_network_delivery() { + let pki = TestPki::new(); + let server = server(&pki, vec![]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let now = OffsetDateTime::now_utc(); + let (identity_store, credential_store) = + pki.stores_with_certificate(&temp, now - time::Duration::days(2), now - time::Duration::days(1), true); + let config = config_with_stores(&temp, &pki, &server, identity_store, credential_store); + + assert_credential_failure(config, &server, "not currently valid").await; +} + +#[tokio::test] +async fn sends_only_l0_fields_and_accepts_additive_response_fields() { + let pki = TestPki::new(); + let server = server(&pki, vec![Reply::ok("2038-01-19T03:14:07Z")]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config(&temp, &pki, &server)), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + + assert_eq!( + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::Online { .. })).await, + HeartbeatStatus::Online { + server_time: "2038-01-19T03:14:07Z".to_owned() + } + ); + runtime.shutdown().await; + let seen = server.seen.lock().expect("seen lock"); + let request = &seen[0]; + let mut keys = request + .as_object() + .expect("heartbeat object") + .keys() + .map(String::as_str) + .collect::>(); + keys.sort_unstable(); + assert_eq!( + keys, + [ + "agentVersion", + "capabilities", + "clientTime", + "coarseNodeSummary", + "protocolVersion", + "requestId", + "sequence" + ] + ); + assert_eq!(request["capabilities"], json!(["heartbeat"])); + assert_eq!(request["coarseNodeSummary"], json!({"total": 8, "healthy": 7, "degraded": 1})); + assert_ne!(request["clientTime"], "2038-01-19T03:14:07Z"); + assert!(request.get("authorization").is_none()); +} + +#[tokio::test] +async fn restart_replays_pending_request_then_advances_sequence() { + let pki = TestPki::new(); + let first_server = server(&pki, vec![Reply::error(StatusCode::SERVICE_UNAVAILABLE)]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let first_config = config(&temp, &pki, &first_server); + let runtime = spawn_heartbeat_runtime(Some(first_config.clone()), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::BackingOff { .. })).await; + runtime.shutdown().await; + let first = first_server.seen.lock().expect("seen lock")[0].clone(); + drop(first_server); + + let second_server = server(&pki, vec![Reply::ok("2026-08-22T01:02:03Z"), Reply::ok("2026-08-22T01:02:04Z")]).await; + let mut second_config = first_config; + second_config.endpoint = second_server.endpoint.clone(); + let runtime = spawn_heartbeat_runtime(Some(second_config), &shutdown, summary) + .expect("restart runtime") + .expect("configured runtime"); + tokio::time::timeout(Duration::from_secs(3), async { + while second_server.seen.lock().expect("seen lock").len() < 2 { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("two heartbeats"); + runtime.shutdown().await; + + let seen = second_server.seen.lock().expect("seen lock"); + assert_eq!(seen[0]["requestId"], first["requestId"]); + assert_eq!(seen[0]["sequence"], first["sequence"]); + assert_ne!(seen[1]["requestId"], seen[0]["requestId"]); + assert_eq!(seen[1]["sequence"].as_u64(), seen[0]["sequence"].as_u64().map(|value| value + 1)); +} + +#[tokio::test] +async fn retry_after_is_respected_with_the_local_upper_bound() { + let pki = TestPki::new(); + let mut reply = Reply::error(StatusCode::TOO_MANY_REQUESTS); + reply.retry_after = Some("300"); + let server = server(&pki, vec![reply]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config(&temp, &pki, &server)), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + assert_eq!( + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::BackingOff { .. })).await, + HeartbeatStatus::BackingOff { + delay: Duration::from_millis(80) + } + ); + runtime.shutdown().await; +} + +#[tokio::test] +async fn disconnects_use_exponential_backoff_with_a_cap() { + let pki = TestPki::new(); + let server = server( + &pki, + vec![ + Reply::error(StatusCode::SERVICE_UNAVAILABLE), + Reply::error(StatusCode::SERVICE_UNAVAILABLE), + Reply::error(StatusCode::SERVICE_UNAVAILABLE), + ], + ) + .await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config(&temp, &pki, &server)), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + for delay in [20, 40, 80] { + assert_eq!( + wait_for(&mut status, |status| { + matches!(status, HeartbeatStatus::BackingOff { delay: observed } if *observed == Duration::from_millis(delay)) + }) + .await, + HeartbeatStatus::BackingOff { + delay: Duration::from_millis(delay) + } + ); + } + runtime.shutdown().await; +} + +#[tokio::test] +async fn revoked_credential_stops_and_exposes_local_status() { + let pki = TestPki::new(); + let mut reply = Reply::error(StatusCode::UNAUTHORIZED); + reply.body = json!({"details": [{"reason": "CREDENTIAL_REVOKED"}]}); + let server = server(&pki, vec![reply]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config(&temp, &pki, &server)), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + let mut status = runtime.status(); + assert_eq!( + wait_for(&mut status, |status| matches!(status, HeartbeatStatus::AuthenticationStopped { .. })).await, + HeartbeatStatus::AuthenticationStopped { + status: 401, + reason: Some("CREDENTIAL_REVOKED".to_owned()) + } + ); + tokio::time::sleep(Duration::from_millis(100)).await; + assert_eq!(server.seen.lock().expect("seen lock").len(), 1); + runtime.shutdown().await; +} + +#[tokio::test] +async fn shutdown_cancels_an_in_flight_request() { + let pki = TestPki::new(); + let mut reply = Reply::ok("2026-08-22T01:02:03Z"); + reply.delay = Duration::from_secs(5); + let server = server(&pki, vec![reply]).await; + let temp = tempfile::tempdir().expect("tempdir"); + let shutdown = CancellationToken::new(); + let runtime = spawn_heartbeat_runtime(Some(config(&temp, &pki, &server)), &shutdown, summary) + .expect("start runtime") + .expect("configured runtime"); + tokio::time::timeout(Duration::from_secs(3), async { + while server.seen.lock().expect("seen lock").is_empty() { + tokio::task::yield_now().await; + } + }) + .await + .expect("request reached server"); + tokio::time::timeout(Duration::from_millis(250), runtime.shutdown()) + .await + .expect("cancellable shutdown"); +} + +#[test] +fn consumes_the_frozen_heartbeat_fixtures() { + let registry: Value = + serde_json::from_str(include_str!("../../protocol/agent/v1/fixtures/fixture-sets.json")).expect("fixture registry"); + let heartbeat = registry["sets"] + .as_array() + .expect("fixture sets") + .iter() + .find(|set| set["name"] == "heartbeat") + .expect("heartbeat fixture set"); + assert_eq!(heartbeat["status"], "populated"); + let valid: Value = + serde_json::from_str(include_str!("../../protocol/agent/v1/fixtures/heartbeat/valid.json")).expect("valid fixture"); + assert_eq!(valid["request"]["protocolVersion"], "v1"); + let overflow: Value = + serde_json::from_str(include_str!("../../protocol/agent/v1/fixtures/heartbeat/overflow.json")).expect("overflow fixture"); + assert_eq!(overflow["expected"]["httpStatus"], 422); +} + +#[cfg(unix)] +fn private_mode(path: &Path) { + use std::os::unix::fs::PermissionsExt as _; + fs::set_permissions(path, fs::Permissions::from_mode(0o600)).expect("private mode"); +} + +#[cfg(not(unix))] +fn private_mode(_path: &Path) {} diff --git a/rustfs/tests/connect_offline_enrollment.rs b/rustfs/tests/connect_offline_enrollment.rs new file mode 100644 index 000000000..e1e9cf0a1 --- /dev/null +++ b/rustfs/tests/connect_offline_enrollment.rs @@ -0,0 +1,951 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Offline enrollment conformance against the frozen Connect fixtures. +//! +//! The device half of the air-gapped exchange verifies a challenge Connect +//! signed and produces a response Connect will verify. Neither side can talk to +//! the other while it does so, which means every disagreement about encoding, +//! trust, or clock windows surfaces as a failed enrollment in the field rather +//! than as an error at development time. The fixtures under +//! `protocol/agent/v1/fixtures/offline-enrollment/` are the shared statement of +//! what both sides must do, so this suite replays them rather than restating +//! them: accept vectors must be accepted with the fields the document carries, +//! reject vectors must fail with the single reason `error-codes.json` freezes, +//! and the signature encoding rules in `trust-model.json` must hold even where +//! the underlying ECDSA library is happy. + +use std::fs; +use std::path::PathBuf; + +use base64::Engine as _; +use base64::engine::general_purpose::STANDARD as BASE64_STANDARD; +use base64::engine::general_purpose::URL_SAFE_NO_PAD as BASE64_URL_NO_PAD; +use rustfs::connect::identity::DeviceIdentity; +use rustfs::connect::offline::{EnrollmentError, OfflineEnrollment, VerifiedChallenge}; +use serde_json::Value; +use sha2::{Digest as _, Sha256}; + +/// DER prefix of a P-256 `SubjectPublicKeyInfo`, frozen by +/// `trust-model.json` as `signature.subjectPublicKeyInfoDerPrefix`. The 65 +/// octet uncompressed point follows it, so a SEC1 point published in a fixture +/// becomes a decodable public key by concatenation. +const SPKI_PREFIX_HEX: &str = "3059301306072a8648ce3d020106082a8648ce3d030107034200"; + +/// `clockSkew.toleranceSeconds` in `trust-model.json`. +const SKEW_TOLERANCE_SECONDS: i64 = 300; + +// --------------------------------------------------------------------------- +// Fixture access +// --------------------------------------------------------------------------- + +fn fixture_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../protocol/agent/v1/fixtures/offline-enrollment") +} + +fn sha256_hex(bytes: &[u8]) -> String { + Sha256::digest(bytes).iter().map(|byte| format!("{byte:02x}")).collect() +} + +/// Read one fixture file and refuse it unless its bytes match the digest +/// `MANIFEST.sha256` freezes. +/// +/// Every vector in this suite arrives through here. A fixture edited on this +/// side therefore fails the tests that depend on it instead of quietly +/// redefining what conformance means, which is the failure mode a +/// fixture-driven suite is otherwise blind to. +fn read_fixture(name: &str) -> Vec { + let dir = fixture_dir(); + let manifest = fs::read_to_string(dir.join("MANIFEST.sha256")).expect("read MANIFEST.sha256"); + + let expected = manifest + .lines() + .filter(|line| !line.trim().is_empty()) + .find_map(|line| { + let (digest, file) = line + .split_once(" ") + .unwrap_or_else(|| panic!("malformed manifest line: {line}")); + (file == name).then(|| digest.to_string()) + }) + .unwrap_or_else(|| panic!("{name} is not listed in MANIFEST.sha256")); + + let bytes = fs::read(dir.join(name)).unwrap_or_else(|error| panic!("read {name}: {error}")); + assert_eq!(sha256_hex(&bytes), expected, "{name} does not match the digest MANIFEST.sha256 freezes"); + bytes +} + +fn fixture_json(name: &str) -> Value { + serde_json::from_slice(&read_fixture(name)).unwrap_or_else(|error| panic!("{name} parses: {error}")) +} + +fn accept_vectors() -> Value { + fixture_json("accept-vectors.json") +} + +fn reject_vectors() -> Value { + fixture_json("reject-vectors.json") +} + +fn trust_model() -> Value { + fixture_json("trust-model.json") +} + +fn vector_list(fixture: &Value) -> Vec { + fixture["vectors"].as_array().expect("fixture carries a vector list").clone() +} + +fn field<'a>(value: &'a Value, key: &str) -> &'a str { + value[key] + .as_str() + .unwrap_or_else(|| panic!("expected a string at '{key}' in {value}")) +} + +/// The octets an operator carries in on removable media. +/// +/// The fixture's `document` object *is* the transmitted artifact: a padded +/// base64 `bytes` field holding the raw signed octets, plus the detached +/// signature over them. Only `bytes` is covered by the signature, so +/// re-serialising the surrounding envelope here cannot change what a verifier +/// checks. +fn envelope(document: &Value) -> Vec { + serde_json::to_vec(document).expect("envelope serialises") +} + +/// The raw octets the signature covers, exactly as transmitted. +fn signed_octets(document: &Value) -> Vec { + BASE64_STANDARD + .decode(field(document, "bytes")) + .expect("document bytes are padded base64") +} + +/// The parsed signed document. Parsing is a convenience for the assertions +/// below; the implementation under test is required to verify before it parses. +fn signed_document(document: &Value) -> Value { + serde_json::from_slice(&signed_octets(document)).expect("signed document parses") +} + +fn unix(rfc3339: &str) -> i64 { + chrono::DateTime::parse_from_rfc3339(rfc3339) + .unwrap_or_else(|error| panic!("'{rfc3339}' is not RFC 3339: {error}")) + .timestamp() +} + +fn hex_to_bytes(hex: &str) -> Vec { + (0..hex.len()) + .step_by(2) + .map(|i| u8::from_str_radix(&hex[i..i + 2], 16).expect("valid hex")) + .collect() +} + +/// Turn a fixture's unpadded-base64url SEC1 point into a usable verifying key. +fn verifying_key(sec1_base64url: &str) -> p256::ecdsa::VerifyingKey { + let point = BASE64_URL_NO_PAD.decode(sec1_base64url).expect("public key is base64url"); + assert_eq!(point.len(), 65, "the protocol freezes a 65 octet uncompressed SEC1 point"); + + let mut der = hex_to_bytes(SPKI_PREFIX_HEX); + der.extend_from_slice(&point); + ::from_public_key_der(&der).expect("public key decodes") +} + +fn published_key(role_or_name: &str) -> Value { + fixture_json("trust-chain.json")["keys"] + .as_array() + .expect("trust chain publishes keys") + .iter() + .find(|key| field(key, "name") == role_or_name) + .unwrap_or_else(|| panic!("trust-chain.json publishes no key named '{role_or_name}'")) + .clone() +} + +/// `signatureInput = domainSeparationTag || 0x00 || the received octets`, the +/// rule `trust-model.json` freezes under `domainSeparation`. +fn signing_input(artifact_tag: &str, received: &[u8]) -> Vec { + let mut input = artifact_tag.as_bytes().to_vec(); + input.push(0x00); + input.extend_from_slice(received); + input +} + +fn domain_tag(artifact: &str) -> String { + let model = trust_model(); + assert_eq!( + field(&model["domainSeparation"], "separatorByte"), + "0x00", + "the separator byte this suite encodes is the one the trust model freezes" + ); + field(&model["domainSeparation"]["tags"], artifact).to_string() +} + +/// Locate an accept vector by the name other vectors reference it by. +fn accept_vector_named(name: &str) -> Value { + vector_list(&accept_vectors()) + .into_iter() + .find(|vector| field(vector, "name") == name) + .unwrap_or_else(|| panic!("accept-vectors.json carries no vector named '{name}'")) +} + +/// Verify the challenge a response vector answers, at that challenge's own +/// evaluation time. +fn answered_challenge(response_vector: &Value) -> (Value, VerifiedChallenge) { + let challenge_vector = accept_vector_named(field(response_vector, "answersChallenge")); + let now = unix(field(&challenge_vector, "evaluationTime")); + let verified = OfflineEnrollment::verify_challenge(&envelope(&challenge_vector["document"]), now) + .expect("the answered challenge is an accept vector and must verify"); + (challenge_vector, verified) +} + +fn device_nonce_of(document: &Value) -> [u8; 32] { + let raw = BASE64_URL_NO_PAD + .decode(field(&signed_document(document), "deviceNonce")) + .expect("deviceNonce is base64url"); + raw.try_into().expect("replay.nonceLengthBytes freezes a 32 octet nonce") +} + +// --------------------------------------------------------------------------- +// Accept vectors +// --------------------------------------------------------------------------- + +/// Every challenge accept vector must verify at its own evaluation time and +/// expose exactly what the signed document says. +/// +/// Two of these vectors sit on the skew boundary — 120 seconds before +/// `issuedAt` and 300 seconds after `expiresAt` — so a verifier that compares +/// against the raw window instead of the tolerated one fails here rather than +/// in an air-gapped data centre. `challenge_proof` is pinned to the challenge's +/// own detached signature value because that is what the response has to echo; +/// deriving it from anything else would silently break the binding. +#[test] +fn every_challenge_accept_vector_verifies_and_exposes_the_signed_fields() { + let mut verified_count = 0usize; + + for vector in vector_list(&accept_vectors()) { + if field(&vector, "artifact") != "challenge" { + continue; + } + + let name = field(&vector, "name"); + let document = &vector["document"]; + let now = unix(field(&vector, "evaluationTime")); + + let verified = OfflineEnrollment::verify_challenge(&envelope(document), now) + .unwrap_or_else(|error| panic!("accept vector '{name}' must verify: {}", error.reason())); + + let signed = signed_document(document); + assert_eq!(verified.challenge_id, field(&signed, "challengeId"), "vector '{name}' challengeId"); + assert_eq!( + verified.organization_name, + field(&signed, "organizationName"), + "vector '{name}' organizationName" + ); + assert_eq!(verified.cluster_name, field(&signed, "clusterName"), "vector '{name}' clusterName"); + assert_eq!(verified.nonce, field(&signed, "nonce"), "vector '{name}' nonce"); + assert_eq!(verified.issued_at, field(&signed, "issuedAt"), "vector '{name}' issuedAt"); + assert_eq!(verified.expires_at, field(&signed, "expiresAt"), "vector '{name}' expiresAt"); + assert_eq!(verified.connect_key_id, field(&signed, "connectKeyId"), "vector '{name}' connectKeyId"); + assert_eq!( + verified.challenge_proof, + field(&document["signature"], "value"), + "vector '{name}' must carry the challenge's own signature as the proof a response echoes" + ); + + verified_count += 1; + } + + assert_eq!( + verified_count, 3, + "accept-vectors.json publishes three challenge vectors; a fourth is a protocol change" + ); +} + +/// Connect's own producer wrote the response accept vectors. Rebuilding them +/// from the challenge they answer, with the device nonce and production time +/// they used, must reproduce every field that does not depend on which device +/// key signed — including the discarded-unknown-field vector, whose extra +/// `telemetryHint` must not survive into anything this side produces. +#[test] +fn response_accept_vectors_are_reproduced_field_for_field_by_build_response() { + let key = DeviceIdentity::generate(); + let mut reproduced = 0usize; + + for vector in vector_list(&accept_vectors()) { + if field(&vector, "artifact") != "response" { + continue; + } + + let name = field(&vector, "name"); + let published = signed_document(&vector["document"]); + let (_, challenge) = answered_challenge(&vector); + + let produced_at = unix(field(&published, "producedAt")); + let nonce = device_nonce_of(&vector["document"]); + + let built_envelope: Value = serde_json::from_slice( + &OfflineEnrollment::build_response(&challenge, &key, &nonce, produced_at) + .unwrap_or_else(|error| panic!("vector '{name}' must be reproducible: {}", error.reason())), + ) + .expect("the built response is JSON"); + let built = signed_document(&built_envelope); + + for shared in [ + "formatVersion", + "protocolVersion", + "challengeId", + "organizationName", + "clusterName", + "challengeNonce", + "challengeProof", + "deviceNonce", + ] { + assert_eq!( + field(&built, shared), + field(&published, shared), + "vector '{name}' field {shared} must match the response Connect published" + ); + } + assert_eq!( + unix(field(&built, "producedAt")), + produced_at, + "vector '{name}' producedAt must be the instant it was given" + ); + + // `versioning.additive` says an unknown optional field is discarded and + // never echoed back; a producer that copied the challenge or a previous + // response wholesale would carry it forward. + assert!( + built.get("telemetryHint").is_none(), + "vector '{name}' must not echo an unknown optional field" + ); + + reproduced += 1; + } + + assert_eq!( + reproduced, 2, + "accept-vectors.json publishes two response vectors; a third is a protocol change" + ); +} + +// --------------------------------------------------------------------------- +// Reject vectors +// --------------------------------------------------------------------------- + +/// Every challenge reject vector must fail, and fail for the one reason +/// `error-codes.json` freezes. +/// +/// Asserting only that verification failed would pass for an implementation +/// that rejects everything, and would let a tampered document be reported as an +/// expiry — a rejection reason is what an operator acts on, so it is part of the +/// contract rather than a diagnostic detail. +#[test] +fn every_challenge_reject_vector_fails_with_its_frozen_reason() { + let known_reasons: Vec = fixture_json("error-codes.json")["reasons"] + .as_array() + .expect("error-codes.json carries reasons") + .iter() + .map(|entry| field(entry, "reason").to_string()) + .collect(); + + let mut rejected = 0usize; + + for vector in vector_list(&reject_vectors()) { + if field(&vector, "artifact") != "challenge" { + continue; + } + + let name = field(&vector, "name"); + let expected = field(&vector["expected"], "reason"); + assert!( + known_reasons.iter().any(|reason| reason == expected), + "vector '{name}' names reason {expected}, which error-codes.json does not freeze" + ); + + let now = unix(field(&vector, "evaluationTime")); + let error = OfflineEnrollment::verify_challenge(&envelope(&vector["document"]), now) + .expect_err(&format!("reject vector '{name}' must not verify")); + + assert_eq!(error.reason(), expected, "vector '{name}' must fail as {expected}"); + rejected += 1; + } + + assert_eq!( + rejected, 8, + "reject-vectors.json publishes eight challenge vectors; losing one silently narrows the suite" + ); +} + +/// The response reject vectors are artifacts Connect refuses. This side never +/// verifies a response, so the device-side statement is the stronger one: given +/// the challenge each vector answers, `build_response` must not be capable of +/// emitting that artifact in the first place. +/// +/// Each arm pins the specific field a compromised or careless producer would +/// have to get wrong, so an implementation that copied values out of the wrong +/// place — the response's own document, an operator-supplied argument, a +/// previous exchange — fails here. +#[test] +fn response_reject_vectors_are_artifacts_build_response_cannot_emit() { + let key = DeviceIdentity::generate(); + let mut covered = 0usize; + + for vector in vector_list(&reject_vectors()) { + if field(&vector, "artifact") != "response" { + continue; + } + + let name = field(&vector, "name"); + let refused = signed_document(&vector["document"]); + let (_, challenge) = answered_challenge(&vector); + let produced_at = unix(field(&refused, "producedAt")); + let nonce = device_nonce_of(&vector["document"]); + + let outcome = OfflineEnrollment::build_response(&challenge, &key, &nonce, produced_at); + + match field(&vector["expected"], "reason") { + // `responseWindow` in trust-model.json: a device that emits a + // response outside the tolerated challenge window has produced an + // artifact Connect will refuse, so the refusal belongs here rather + // than at the far end of a courier run. + "CHALLENGE_EXPIRED" => { + let error = outcome.expect_err(&format!("vector '{name}': producing this response must be refused")); + assert_eq!(error.reason(), "CHALLENGE_EXPIRED", "vector '{name}' must refuse as CHALLENGE_EXPIRED"); + covered += 1; + continue; + } + reason => { + let built_envelope: Value = serde_json::from_slice( + &outcome.unwrap_or_else(|error| panic!("vector '{name}' baseline must build: {}", error.reason())), + ) + .expect("the built response is JSON"); + let built = signed_document(&built_envelope); + + match reason { + "ORGANIZATION_MISMATCH" => { + assert_ne!( + field(&refused, "organizationName"), + challenge.organization_name, + "vector '{name}' is only a mismatch if it names another organization" + ); + assert_eq!( + field(&built, "organizationName"), + challenge.organization_name, + "vector '{name}': the organization must come from the challenge, never from elsewhere" + ); + } + "CLUSTER_MISMATCH" => { + assert_ne!( + field(&refused, "clusterName"), + challenge.cluster_name, + "vector '{name}' is only a mismatch if it names another cluster" + ); + assert_eq!( + field(&built, "clusterName"), + challenge.cluster_name, + "vector '{name}': the cluster must come from the challenge, never from elsewhere" + ); + } + "CHALLENGE_PROOF_INVALID" => { + // Two distinct vectors land here: a nonce the challenge + // never carried, and a proof lifted from another + // challenge. Both must be impossible to produce. + assert_eq!( + field(&built, "challengeNonce"), + challenge.nonce, + "vector '{name}': the echoed nonce must be the challenge's own" + ); + assert_eq!( + field(&built, "challengeProof"), + challenge.challenge_proof, + "vector '{name}': the proof must be the answered challenge's signature" + ); + assert!( + field(&refused, "challengeNonce") != challenge.nonce + || field(&refused, "challengeProof") != challenge.challenge_proof, + "vector '{name}' must differ from the challenge in nonce or proof to be rejectable" + ); + } + "DEVICE_PROOF_INVALID" => { + // The refused vector presents one key and is signed by + // another; hold the fixture to that claim, then require + // the built response to be the opposite. Proof of + // possession is the only thing that makes presenting a + // key in an unauthenticated document safe. + use p256::ecdsa::signature::Verifier as _; + + let presented = verifying_key(field(&refused, "devicePublicKey")); + let raw = BASE64_URL_NO_PAD + .decode(field(&vector["document"]["signature"], "value")) + .expect("signature is base64url"); + let signature = p256::ecdsa::Signature::from_slice(&raw).expect("signature parses"); + assert!( + presented + .verify( + &signing_input(&domain_tag("enrollmentResponse"), &signed_octets(&vector["document"])), + &signature + ) + .is_err(), + "vector '{name}' is only a possession failure if it does not verify under the key it presents" + ); + + assert_response_proves_possession(&built_envelope, name); + } + "UNSUPPORTED_FORMAT" => { + assert_ne!( + field(&refused, "formatVersion"), + field(&built, "formatVersion"), + "vector '{name}' is only unsupported if it names another format version" + ); + assert_eq!( + field(&built, "formatVersion"), + "rustfs.connect.offline.enrollmentResponse/1", + "vector '{name}': the format version is frozen" + ); + } + "UNSUPPORTED_PROTOCOL" => { + assert_ne!( + field(&refused, "protocolVersion"), + field(&built, "protocolVersion"), + "vector '{name}' is only unsupported if it names another protocol major" + ); + assert_eq!(field(&built, "protocolVersion"), "v1", "vector '{name}': the protocol major is frozen"); + } + "ENROLLMENT_REPLAYED" => { + // The vector claims to be a byte-identical replay of an + // accepted response; hold it to that, because a replay + // vector that is not byte identical proves nothing about + // single use. + let accepted = accept_vector_named("response binding the device public key and the challenge proof"); + assert_eq!( + signed_octets(&vector["document"]), + signed_octets(&accepted["document"]), + "vector '{name}' must be the accepted response octet for octet" + ); + assert_eq!( + field(&vector["document"]["signature"], "value"), + field(&accepted["document"]["signature"], "value"), + "vector '{name}' must carry the accepted response's signature" + ); + + // A fresh device nonce is a different artifact, so a + // second enrollment is never mistaken for a replay of + // the first. + let other = OfflineEnrollment::build_response(&challenge, &key, &[0x5a; 32], produced_at) + .expect("a second response builds"); + assert_ne!( + signed_octets(&built_envelope), + signed_octets(&serde_json::from_slice::(&other).expect("JSON")), + "vector '{name}': a different device nonce must yield a different artifact" + ); + } + other => panic!("vector '{name}' names an unhandled reason {other}; extend this test"), + } + } + } + + covered += 1; + } + + assert_eq!( + covered, 9, + "reject-vectors.json publishes nine response vectors; losing one silently narrows the suite" + ); +} + +// --------------------------------------------------------------------------- +// Signature encoding +// --------------------------------------------------------------------------- + +/// The high-S malleation is the rejection the whole encoding rule exists for. +/// +/// `(r, n - s)` is a second valid signature over the same document under the +/// same key. Every mainstream ECDSA library verifies it, so an implementation +/// that hands the decoded octets straight to `p256` accepts a forged-looking +/// duplicate of a genuine challenge — and because the 64 octets differ, that +/// duplicate is a distinct artifact identity that slips past any deduplication +/// keyed on the signature. This test proves the rejection came from the +/// encoding rule and not from a failed verification: it first shows the +/// malleated signature verifying mathematically, then requires +/// `verify_challenge` to refuse it as SIGNATURE_NOT_CANONICAL. +#[test] +fn malleated_high_s_signature_is_refused_although_it_verifies_mathematically() { + use p256::ecdsa::signature::Verifier as _; + + let model = trust_model(); + let malleated = model["rejectedSignatureEncodings"] + .as_array() + .expect("trust-model.json publishes rejected encodings") + .iter() + .find(|entry| field(entry, "reason") == "SIGNATURE_NOT_CANONICAL") + .expect("trust-model.json publishes the high-S malleation") + .clone(); + assert!( + malleated["acceptedByALenientVerifier"].as_bool() == Some(true), + "this vector is only interesting because a lenient verifier accepts it" + ); + + let vector = accept_vector_named("challenge signed by a chained signing key under the pinned root"); + let genuine_value = field(&vector["document"]["signature"], "value").to_string(); + let malleated_value = field(&malleated, "value").to_string(); + assert_ne!(genuine_value, malleated_value, "the malleation must be a different encoding"); + + let genuine = BASE64_URL_NO_PAD.decode(&genuine_value).expect("signature is base64url"); + let raw = BASE64_URL_NO_PAD.decode(&malleated_value).expect("signature is base64url"); + assert_eq!(raw.len(), 64, "the malleation is well formed at 64 octets"); + assert_eq!(raw[..32], genuine[..32], "the malleation shares r with the genuine signature"); + assert_ne!(raw[32..], genuine[32..], "the malleation replaces s with n - s"); + + // Step one: the malleated pair really does verify under the signing key, so + // a verifier cannot be excused for accepting it on mathematical grounds. + let signature = p256::ecdsa::Signature::from_slice(&raw).expect("the malleated signature parses"); + assert!(signature.normalize_s().is_some(), "the malleated signature must be the high-S form"); + let key = verifying_key(field(&published_key("signing"), "publicKey")); + let input = signing_input(&domain_tag("enrollmentChallenge"), &signed_octets(&vector["document"])); + key.verify(&input, &signature) + .expect("the malleated signature must verify mathematically, or this test proves nothing"); + + // Step two: the implementation must refuse it anyway, and say why. + let mut tampered = vector["document"].clone(); + tampered["signature"]["value"] = Value::String(malleated_value); + + let now = unix(field(&vector, "evaluationTime")); + let error = OfflineEnrollment::verify_challenge(&envelope(&tampered), now) + .expect_err("a high-S signature must be refused even though it verifies"); + assert_eq!( + error.reason(), + "SIGNATURE_NOT_CANONICAL", + "a malleated signature is a canonicality failure, not a verification failure" + ); +} + +/// Every encoding `trust-model.json` names as rejected must fail with the +/// reason it names — DER, padded base64url, truncation, and out-of-range +/// scalars alongside the malleation. Three of the five are accepted by a +/// lenient verifier, so a single blanket "signature did not verify" answer would +/// be both wrong and undiagnosable. +#[test] +fn every_rejected_signature_encoding_fails_with_its_frozen_reason() { + let vector = accept_vector_named("challenge signed by a chained signing key under the pinned root"); + let now = unix(field(&vector, "evaluationTime")); + let model = trust_model(); + let encodings = model["rejectedSignatureEncodings"] + .as_array() + .expect("trust-model.json publishes rejected encodings"); + + for entry in encodings { + let name = field(entry, "name"); + let mut tampered = vector["document"].clone(); + tampered["signature"]["value"] = Value::String(field(entry, "value").to_string()); + + let error: EnrollmentError = OfflineEnrollment::verify_challenge(&envelope(&tampered), now) + .err() + .unwrap_or_else(|| panic!("rejected encoding '{name}' must not verify")); + + assert_eq!(error.reason(), field(entry, "reason"), "rejected encoding '{name}'"); + } + + assert_eq!(encodings.len(), 5, "trust-model.json freezes five rejected encodings"); +} + +// --------------------------------------------------------------------------- +// Clock window +// --------------------------------------------------------------------------- + +/// The tolerated window is `[issuedAt - 300, expiresAt + 300]`, inclusive at +/// both ends. An air-gapped device has no synchronised clock, so an +/// off-by-one here either strands a legitimate enrollment or widens the window +/// a stolen challenge stays usable in. Both ends are checked at the exact bound +/// and one second past it, and the reason distinguishes the two directions. +#[test] +fn challenge_is_accepted_at_the_exact_skew_bound_and_refused_one_second_past_it() { + let vector = accept_vector_named("challenge signed by a chained signing key under the pinned root"); + let document = envelope(&vector["document"]); + let signed = signed_document(&vector["document"]); + + let issued_at = unix(field(&signed, "issuedAt")); + let expires_at = unix(field(&signed, "expiresAt")); + + let earliest = issued_at - SKEW_TOLERANCE_SECONDS; + OfflineEnrollment::verify_challenge(&document, earliest).expect("the earliest tolerated instant is inside the window"); + let error = + OfflineEnrollment::verify_challenge(&document, earliest - 1).expect_err("one second earlier is outside the window"); + assert_eq!(error.reason(), "CHALLENGE_NOT_YET_VALID"); + + let latest = expires_at + SKEW_TOLERANCE_SECONDS; + OfflineEnrollment::verify_challenge(&document, latest).expect("the latest tolerated instant is inside the window"); + let error = OfflineEnrollment::verify_challenge(&document, latest + 1).expect_err("one second later is outside the window"); + assert_eq!(error.reason(), "CHALLENGE_EXPIRED"); +} + +// --------------------------------------------------------------------------- +// Response production +// --------------------------------------------------------------------------- + +/// Assert a built response proves possession of the key it presents: the +/// fingerprint matches the presented key, and the detached signature is a +/// canonical low-S ES256 signature that verifies under that key over the exact +/// octets transmitted. +fn assert_response_proves_possession(built_envelope: &Value, label: &str) { + use p256::ecdsa::signature::Verifier as _; + + let raw = signed_octets(built_envelope); + let built = signed_document(built_envelope); + let signature_block = &built_envelope["signature"]; + + assert_eq!(field(signature_block, "algorithm"), "ES256", "{label}: the algorithm is frozen"); + + let value = field(signature_block, "value"); + assert_eq!(value.len(), 86, "{label}: the transfer encoding is 86 unpadded base64url characters"); + assert!( + value.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_'), + "{label}: the signature must use the base64url alphabet with no padding" + ); + + let bytes = BASE64_URL_NO_PAD.decode(value).expect("signature is base64url"); + assert_eq!(bytes.len(), 64, "{label}: the signature is a fixed-width r || s"); + let signature = p256::ecdsa::Signature::from_slice(&bytes).expect("signature parses"); + assert!( + signature.normalize_s().is_none(), + "{label}: this side must never emit the malleated high-S form it refuses to accept" + ); + + let presented = field(&built, "devicePublicKey"); + let key = verifying_key(presented); + key.verify(&signing_input(&domain_tag("enrollmentResponse"), &raw), &signature) + .unwrap_or_else(|error| panic!("{label}: the response must verify under the key it presents: {error}")); + + // `signature.keyIdAlgorithm`: the lowercase SHA-256 of the DER + // SubjectPublicKeyInfo, not of the bare point and not of the transfer + // encoding. + let mut spki = hex_to_bytes(SPKI_PREFIX_HEX); + spki.extend_from_slice(&BASE64_URL_NO_PAD.decode(presented).expect("public key is base64url")); + let fingerprint = sha256_hex(&spki); + assert_eq!( + field(&built, "deviceKeyId"), + fingerprint, + "{label}: deviceKeyId must be the fingerprint of the key the document presents" + ); + assert_eq!( + field(signature_block, "keyId"), + fingerprint, + "{label}: the detached signature must name the same key" + ); +} + +/// A response is the only thing Connect will ever see from this device, so it +/// has to carry the whole binding on its own: the challenge it answers, the +/// proof that challenge was genuine, the key being enrolled, and possession of +/// that key. +#[test] +fn built_response_binds_the_challenge_proof_and_proves_possession_of_the_device_key() { + let vector = accept_vector_named("response binding the device public key and the challenge proof"); + let (challenge_vector, challenge) = answered_challenge(&vector); + let key = DeviceIdentity::generate(); + let produced_at = unix(field(&signed_document(&vector["document"]), "producedAt")); + + let bytes = OfflineEnrollment::build_response(&challenge, &key, &[0x11; 32], produced_at).expect("the response builds"); + let built_envelope: Value = serde_json::from_slice(&bytes).expect("the response is JSON"); + let built = signed_document(&built_envelope); + + assert_response_proves_possession(&built_envelope, "built response"); + + // The proof is the challenge's own detached signature. A producer that + // echoed the nonce alone, or hashed something, would let a response be + // built from an unverified challenge. + assert_eq!( + field(&built, "challengeProof"), + field(&challenge_vector["document"]["signature"], "value"), + "the proof must be the signature of the challenge being answered" + ); + assert_eq!(field(&built, "challengeNonce"), challenge.nonce); + assert_eq!(field(&built, "challengeId"), challenge.challenge_id); + + assert_eq!( + field(&built, "devicePublicKey"), + BASE64_URL_NO_PAD.encode(&key.public_key_der()[hex_to_bytes(SPKI_PREFIX_HEX).len()..]), + "the presented key must be the key that was passed in" + ); + assert_eq!( + field(&built, "deviceNonce"), + BASE64_URL_NO_PAD.encode([0x11; 32]), + "the device nonce must be the one that was passed in" + ); + assert!(field(&built, "producedAt").ends_with('Z'), "producedAt is a UTC RFC 3339 instant"); +} + +/// The response leaves the air gap on removable media and is read by anyone who +/// handles it. A producer that serialised the key pair instead of the public +/// key, or logged a debug rendering into the document, would put the enrolled +/// private key on that medium — and the enrollment would still succeed, so +/// nothing else in this suite would notice. +#[test] +fn built_response_carries_no_private_key_material() { + let vector = accept_vector_named("response binding the device public key and the challenge proof"); + let (_, challenge) = answered_challenge(&vector); + let key = DeviceIdentity::generate(); + let produced_at = unix(field(&signed_document(&vector["document"]), "producedAt")); + + let response = OfflineEnrollment::build_response(&challenge, &key, &[0x22; 32], produced_at).expect("the response builds"); + + // The envelope carries the signed document base64-encoded, so a needle + // present in the document is not present in the envelope octets. Both + // layers are searched: an operator handling the medium can read either. + let envelope_value: Value = serde_json::from_slice(&response).expect("the response is JSON"); + let mut haystack = response; + haystack.extend_from_slice(&signed_octets(&envelope_value)); + + let pkcs8 = key.to_pkcs8_der().expect("serialise the key"); + let secret = ::from_pkcs8_der(&pkcs8).expect("the key parses"); + let scalar = secret.to_bytes(); + + // Every spelling the scalar could plausibly reach a document in: raw, and + // the three encodings this protocol already uses elsewhere. + let scalar_hex: String = scalar.iter().map(|byte| format!("{byte:02x}")).collect(); + for (description, needle) in [ + ("the PKCS#8 encoding", pkcs8.to_vec()), + ("the raw private scalar", scalar.to_vec()), + ("the scalar in base64url", BASE64_URL_NO_PAD.encode(scalar).into_bytes()), + ("the scalar in standard base64", BASE64_STANDARD.encode(scalar).into_bytes()), + ("the scalar in hex", scalar_hex.into_bytes()), + ] { + assert!( + !haystack.windows(needle.len()).any(|window| window == needle.as_slice()), + "the response must not contain {description}" + ); + } + + // The public half must be there, so the absence above is a statement about + // what was excluded rather than about a haystack that would not have found + // the private half either. + let point = BASE64_URL_NO_PAD.encode(&key.public_key_der()[hex_to_bytes(SPKI_PREFIX_HEX).len()..]); + assert!( + haystack.windows(point.len()).any(|window| window == point.as_bytes()), + "the response must still present the public key" + ); +} + +// --------------------------------------------------------------------------- +// The offline invariant +// --------------------------------------------------------------------------- + +/// The whole surface exists because there is no network. This asserts that +/// three different ways, because no single one of them is conclusive on its own. +/// +/// 1. The process opens no descriptor across a full verify-and-respond cycle. A +/// socket, a DNS resolver, a pooled HTTP client, or a revocation-list fetch +/// all show up here — including one that is opened and cached rather than +/// opened and closed, which is what a lazily built client does. +/// 2. The cycle is a pure byte transform: the same inputs produce the same +/// verified fields, and the evaluation instant is an argument rather than an +/// ambient read, so nothing about the outcome can depend on reachability. +/// 3. Repeating the cycle changes nothing observable, so a first call cannot be +/// quietly initialising shared state that a later one reuses. +#[cfg(unix)] +#[test] +fn enrollment_opens_no_descriptor_and_is_a_pure_byte_transform() { + let vector = accept_vector_named("challenge signed by a chained signing key under the pinned root"); + let document = envelope(&vector["document"]); + let now = unix(field(&vector, "evaluationTime")); + let key = DeviceIdentity::generate(); + + // Warm anything the test harness itself lazily opens before the baseline. + let _ = open_descriptors(); + let baseline = open_descriptors(); + assert!( + !baseline.is_empty(), + "the descriptor table must be readable for this test to mean anything" + ); + + let mut fields = Vec::new(); + for _ in 0..2 { + let challenge = OfflineEnrollment::verify_challenge(&document, now).expect("the challenge verifies"); + let response = OfflineEnrollment::build_response(&challenge, &key, &[0x33; 32], now).expect("the response builds"); + fields.push(( + challenge.challenge_id.clone(), + challenge.nonce.clone(), + challenge.challenge_proof.clone(), + signed_octets(&serde_json::from_slice::(&response).expect("JSON")), + )); + } + + assert_eq!( + open_descriptors(), + baseline, + "the enrollment path must not open a descriptor: no socket, no resolver, no cached client" + ); + + let (first, second) = (&fields[0], &fields[1]); + assert_eq!(first.0, second.0, "verification must be deterministic"); + assert_eq!(first.1, second.1, "verification must be deterministic"); + assert_eq!(first.2, second.2, "verification must be deterministic"); + assert_eq!( + first.3, second.3, + "the signed response octets are a function of the challenge, the key, the nonce, and the instant" + ); +} + +#[cfg(unix)] +fn open_descriptors() -> Vec { + // Linux publishes the table at /proc/self/fd; the BSDs and macOS at /dev/fd. + let path = if PathBuf::from("/proc/self/fd").is_dir() { + "/proc/self/fd" + } else { + "/dev/fd" + }; + + let mut entries: Vec = fs::read_dir(path) + .unwrap_or_else(|error| panic!("read {path}: {error}")) + .map(|entry| entry.expect("read dir entry").file_name().to_string_lossy().into_owned()) + .collect(); + entries.sort(); + entries +} + +/// A descriptor count taken around a call cannot see a socket that was opened +/// and closed inside it, so the invariant is also asserted where it can be +/// stated absolutely: the implementation names no network API at all. +/// +/// This is the shape the regression actually takes — someone adds a +/// revocation-list fetch, a time-server check, or a "just confirm the challenge +/// with Connect" call — and it is caught at the source rather than by observing +/// its effects. +#[test] +fn enrollment_implementation_names_no_network_api() { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("src/connect/offline/enrollment.rs"); + let source = fs::read_to_string(&path).unwrap_or_else(|error| panic!("read {}: {error}", path.display())); + + // Prose is allowed to discuss the invariant it is documenting, so only code + // is scanned. + let code: String = source + .lines() + .filter(|line| !line.trim_start().starts_with("//")) + .collect::>() + .join("\n"); + + for forbidden in [ + "std::net", + "tokio::net", + "TcpStream", + "TcpListener", + "UdpSocket", + "UnixStream", + "ToSocketAddrs", + "reqwest", + "hyper", + "tonic", + ] { + assert!( + !code.contains(forbidden), + "offline enrollment must not reach the network, but the implementation names {forbidden}" + ); + } +} diff --git a/rustfs/tests/connect_registration.rs b/rustfs/tests/connect_registration.rs new file mode 100644 index 000000000..a1e94030d --- /dev/null +++ b/rustfs/tests/connect_registration.rs @@ -0,0 +1,911 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::VecDeque; +use std::fs; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use base64::Engine as _; +use base64::engine::general_purpose::{STANDARD as BASE64_STANDARD, URL_SAFE_NO_PAD as BASE64_URL_NO_PAD}; +use bytes::Bytes; +use http_body_util::{BodyExt as _, Full}; +use hyper::service::service_fn; +use hyper::{Request, Response, StatusCode}; +use hyper_util::rt::TokioIo; +use p256::ecdsa::signature::Verifier as _; +use p256::ecdsa::{Signature, VerifyingKey}; +use p256::pkcs8::DecodePublicKey as _; +use rcgen::{ + BasicConstraints, CertificateParams, DistinguishedName, DnType, ExtendedKeyUsagePurpose, IsCa, Issuer, KeyPair, + KeyUsagePurpose, SanType, SerialNumber, +}; +use rustfs::connect::{ClientError, ConnectClient, ConnectConfig, CredentialStore, IdentityStore, RegistrationToken, TokenError}; +use rustls::RootCertStore; +use rustls::pki_types::{CertificateDer, PrivateKeyDer, PrivatePkcs8KeyDer, pem::PemObject as _}; +use rustls::server::WebPkiClientVerifier; +use serde_json::{Value, json}; +use sha2::{Digest as _, Sha256}; +use time::OffsetDateTime; +use time::format_description::well_known::Rfc3339; +use tokio::net::TcpListener; +use tokio_rustls::TlsAcceptor; + +const ORGANIZATION_UID: &str = "0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70"; +const CLUSTER_UID: &str = "0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81"; +const DEVICE_UID: &str = "0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92"; +const TOKEN_UID: &str = "0198f4b0-6f00-7b60-9271-7d8e9fa0b1c5"; +const FRESH_TOKEN_UID: &str = "0198f4b0-7f00-7c70-a381-8e9fa0b1c2d6"; +const SECOND_TOKEN_UID: &str = "0198f4b0-8f00-7d80-b491-9fa0b1c2d3e7"; + +struct TestPki { + root_params: CertificateParams, + root_key: KeyPair, + root_der: CertificateDer<'static>, + root_pem: String, + server_der: CertificateDer<'static>, + server_key: PrivatePkcs8KeyDer<'static>, +} + +impl TestPki { + fn new() -> Self { + let now = OffsetDateTime::now_utc(); + let root_key = KeyPair::generate().expect("generate root key"); + let mut root_params = CertificateParams::default(); + root_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + root_params.not_before = now - time::Duration::days(1); + root_params.not_after = now + time::Duration::days(30); + root_params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::DigitalSignature]; + root_params.distinguished_name.push(DnType::CommonName, "Connect test root"); + let root = root_params.self_signed(&root_key).expect("sign root"); + + let server_key = KeyPair::generate().expect("generate server key"); + let mut server_params = CertificateParams::default(); + server_params.not_before = now - time::Duration::hours(1); + server_params.not_after = now + time::Duration::days(2); + server_params + .subject_alt_names + .push(SanType::DnsName("localhost".try_into().expect("valid DNS name"))); + server_params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ServerAuth]; + let issuer = Issuer::from_params(&root_params, &root_key); + let server = server_params + .signed_by(&server_key, &issuer) + .expect("sign server certificate"); + + Self { + root_params, + root_key, + root_der: root.der().clone(), + root_pem: root.pem(), + server_der: server.der().clone(), + server_key: PrivatePkcs8KeyDer::from(server_key.serialize_der()), + } + } + + fn credential(&self, identity: &rustfs::connect::DeviceIdentity, uri: &str, serial_byte: u8) -> Value { + let now = OffsetDateTime::now_utc().replace_nanosecond(0).expect("whole second"); + self.credential_window(identity, uri, serial_byte, now, now + time::Duration::days(1)) + } + + fn credential_window( + &self, + identity: &rustfs::connect::DeviceIdentity, + uri: &str, + serial_byte: u8, + not_before: OffsetDateTime, + not_after: OffsetDateTime, + ) -> Value { + let mut params = CertificateParams::default(); + params.not_before = not_before; + params.not_after = not_after; + params.serial_number = Some(SerialNumber::from(vec![serial_byte; 16])); + params.key_usages = vec![KeyUsagePurpose::DigitalSignature]; + params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ClientAuth]; + params.distinguished_name = DistinguishedName::new(); + params.distinguished_name.push(DnType::CommonName, DEVICE_UID); + params + .subject_alt_names + .push(SanType::URI(uri.try_into().expect("valid URI SAN"))); + let private_key = identity.to_pkcs8_der().expect("serialize device key"); + let private_key = PrivatePkcs8KeyDer::from(private_key.to_vec()); + let device_key = + KeyPair::from_pkcs8_der_and_sign_algo(&private_key, &rcgen::PKCS_ECDSA_P256_SHA256).expect("parse device key"); + let issuer = Issuer::from_params(&self.root_params, &self.root_key); + let certificate = params.signed_by(&device_key, &issuer).expect("sign device certificate"); + let serial = format!("{serial_byte:02x}").repeat(16); + let cluster = format!("organizations/{ORGANIZATION_UID}/clusters/{CLUSTER_UID}"); + + json!({ + "name": format!("{cluster}/clusterDevices/{DEVICE_UID}"), + "uid": DEVICE_UID, + "cluster": cluster, + "protocolVersion": "v1", + "keyId": format!("x509-{serial}"), + "certificateSerial": serial, + "certificate": certificate.pem(), + "certificateChain": certificate.pem(), + "notBefore": not_before.format(&Rfc3339).expect("format notBefore"), + "notAfter": not_after.format(&Rfc3339).expect("format notAfter"), + }) + } + + fn server_config(&self, require_client: bool) -> rustls::ServerConfig { + let mut roots = RootCertStore::empty(); + roots.add(self.root_der.clone()).expect("add client root"); + let verifier = WebPkiClientVerifier::builder(Arc::new(roots)); + let verifier = if require_client { + verifier.build() + } else { + verifier.allow_unauthenticated().build() + } + .expect("build client verifier"); + rustls::ServerConfig::builder() + .with_client_cert_verifier(verifier) + .with_single_cert(vec![self.server_der.clone()], PrivateKeyDer::Pkcs8(self.server_key.clone_key())) + .expect("build server TLS") + } +} + +#[derive(Clone)] +enum Reply { + Json(StatusCode, Value), + DelayedClose(Duration), + VerifiedRotation { + response: Value, + current_public_key: Vec, + current_certificate_fingerprint: String, + device_name: String, + }, +} + +struct TestServer { + endpoint: String, + seen: Arc>>, + task: tokio::task::JoinHandle<()>, +} + +impl Drop for TestServer { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn server(pki: &TestPki, replies: Vec) -> TestServer { + server_with_client_auth(pki, replies, false).await +} + +async fn server_with_client_auth(pki: &TestPki, replies: Vec, require_client: bool) -> TestServer { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind test server"); + let address = listener.local_addr().expect("server address"); + let acceptor = TlsAcceptor::from(Arc::new(pki.server_config(require_client))); + let replies = Arc::new(Mutex::new(VecDeque::from(replies))); + let seen = Arc::new(Mutex::new(Vec::new())); + let captured = seen.clone(); + let task = tokio::spawn(async move { + loop { + let Ok((stream, _)) = listener.accept().await else { + return; + }; + let acceptor = acceptor.clone(); + let replies = replies.clone(); + let seen = captured.clone(); + tokio::spawn(async move { + let Ok(stream) = acceptor.accept(stream).await else { + return; + }; + let service = service_fn(move |request: Request| { + let replies = replies.clone(); + let seen = seen.clone(); + async move { + let body = request.into_body().collect().await.expect("read request body").to_bytes(); + let value: Value = serde_json::from_slice(&body).expect("request JSON"); + seen.lock().expect("seen lock").push(value.clone()); + let reply = replies.lock().expect("reply lock").pop_front().expect("planned reply"); + match reply { + Reply::Json(status, value) => Ok::<_, hyper::Error>( + Response::builder() + .status(status) + .header("content-type", "application/json") + .body(Full::new(Bytes::from(serde_json::to_vec(&value).expect("reply JSON")))) + .expect("response"), + ), + Reply::DelayedClose(delay) => { + tokio::time::sleep(delay).await; + Ok(Response::builder() + .status(StatusCode::SERVICE_UNAVAILABLE) + .body(Full::new(Bytes::new())) + .expect("response")) + } + Reply::VerifiedRotation { + response, + current_public_key, + current_certificate_fingerprint, + device_name, + } => { + verify_rotation_request( + &value, + ¤t_public_key, + ¤t_certificate_fingerprint, + &device_name, + ); + Ok(Response::builder() + .status(StatusCode::OK) + .header("content-type", "application/json") + .body(Full::new(Bytes::from(serde_json::to_vec(&response).expect("reply JSON")))) + .expect("response")) + } + } + } + }); + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection(TokioIo::new(stream), service) + .await; + }); + } + }); + TestServer { + endpoint: format!("https://localhost:{}/agent/", address.port()), + seen, + task, + } +} + +fn verify_rotation_request(request: &Value, current_public_key: &[u8], fingerprint: &str, device_name: &str) { + assert_eq!(request["protocolVersion"], "v1"); + assert_eq!(request["proof"]["algorithm"], "ES256"); + let csr = BASE64_STANDARD + .decode(request["certificateRequest"].as_str().expect("certificateRequest")) + .expect("CSR base64"); + let csr_digest = BASE64_URL_NO_PAD.encode(Sha256::digest(&csr)); + let request_id = request["requestId"].as_str().expect("requestId"); + let transcript = rebuilt_rotation_transcript( + b"RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1", + [fingerprint, device_name, request_id, &csr_digest], + ); + let encoded = request["proof"]["value"].as_str().expect("proof value"); + assert_eq!(encoded.len(), 86); + let raw = BASE64_URL_NO_PAD.decode(encoded).expect("proof base64url"); + let signature = Signature::from_slice(&raw).expect("fixed-width signature"); + assert!(signature.normalize_s().is_none(), "rotation proof must be low-S"); + let verifying = VerifyingKey::from_public_key_der(current_public_key).expect("current public key"); + verifying.verify(&transcript, &signature).expect("rotation proof verifies"); + + let wrong_domain = rebuilt_rotation_transcript( + b"RUSTFS-CONNECT-CREDENTIAL-ROTATION-V2", + [fingerprint, device_name, request_id, &csr_digest], + ); + assert!(verifying.verify(&wrong_domain, &signature).is_err()); + let wrong_order = rebuilt_rotation_transcript( + b"RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1", + [device_name, fingerprint, request_id, &csr_digest], + ); + assert!(verifying.verify(&wrong_order, &signature).is_err()); +} + +fn rebuilt_rotation_transcript(domain: &[u8], fields: [&str; 4]) -> Vec { + let mut transcript = Vec::new(); + transcript.extend_from_slice(domain); + transcript.push(b'\n'); + for field in fields { + transcript.extend_from_slice(field.len().to_string().as_bytes()); + transcript.push(b':'); + transcript.extend_from_slice(field.as_bytes()); + transcript.push(b'\n'); + } + transcript +} + +fn certificate_fingerprint(pem: &str) -> String { + let certificate = CertificateDer::pem_slice_iter(pem.as_bytes()) + .next() + .expect("leaf certificate") + .expect("certificate PEM"); + Sha256::digest(certificate.as_ref()) + .iter() + .map(|byte| format!("{byte:02x}")) + .collect() +} + +fn token_document() -> Value { + json!({ + "registrationTokenUid": TOKEN_UID, + "registrationTokenSecret": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "organizationUid": ORGANIZATION_UID, + "clusterUid": CLUSTER_UID, + "challengeNonce": "a3f1c07d9b2e4856af0c1d3b5e7f9012c4a6b8d0e2f4061738495a6b7c8d9e0f", + "expiresUnix": OffsetDateTime::now_utc().unix_timestamp() + 3600, + }) +} + +fn token() -> RegistrationToken { + token_with_uid(TOKEN_UID) +} + +fn token_with_uid(uid: &str) -> RegistrationToken { + let document = token_document(); + let mut document = document; + document["registrationTokenUid"] = json!(uid); + RegistrationToken::from_reader(serde_json::to_vec(&document).expect("token JSON").as_slice()).expect("token parses") +} + +fn stores(temp: &tempfile::TempDir) -> (IdentityStore, CredentialStore) { + ( + IdentityStore::new(temp.path().join("identity")), + CredentialStore::new(temp.path().join("credential")), + ) +} + +fn client(server: &TestServer, pki: &TestPki, timeout: Duration) -> ConnectClient { + ConnectClient::new(ConnectConfig { + endpoint: &server.endpoint, + root_ca_pem: pki.root_pem.as_bytes(), + timeout, + }) + .expect("build Connect client") +} + +fn rotation_response(pki: &TestPki, identity: &rustfs::connect::DeviceIdentity, serial: u8) -> (Value, Value) { + let stored = pki.credential(identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), serial); + let mut wire = stored.clone(); + wire.as_object_mut().expect("response object").remove("uid"); + wire.as_object_mut().expect("response object").remove("cluster"); + (wire, stored) +} + +fn write_stored_credential(path: &std::path::Path, response: &Value) { + let not_before = OffsetDateTime::parse(response["notBefore"].as_str().expect("notBefore"), &Rfc3339) + .expect("parse notBefore") + .unix_timestamp(); + let not_after = OffsetDateTime::parse(response["notAfter"].as_str().expect("notAfter"), &Rfc3339) + .expect("parse notAfter") + .unix_timestamp(); + let stored = json!({ + "name": response["name"], + "uid": DEVICE_UID, + "protocolVersion": response["protocolVersion"], + "keyId": response["keyId"], + "certificateSerial": response["certificateSerial"], + "certificate": response["certificate"], + "certificateChain": response["certificateChain"], + "notBeforeUnix": not_before, + "notAfterUnix": not_after, + }); + fs::write(path, serde_json::to_vec(&stored).expect("stored credential JSON")).expect("write credential"); + set_owner_only(path); +} + +#[cfg(unix)] +fn set_owner_only(path: &std::path::Path) { + use std::os::unix::fs::PermissionsExt as _; + fs::set_permissions(path, fs::Permissions::from_mode(0o600)).expect("set owner-only mode"); +} + +#[cfg(not(unix))] +fn set_owner_only(_path: &std::path::Path) {} + +#[tokio::test] +async fn registration_reuses_request_and_csr_after_timeout_and_restart() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let identity = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let response = pki.credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 0x80); + let server = server( + &pki, + vec![ + Reply::DelayedClose(Duration::from_millis(200)), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::CREATED, response), + ], + ) + .await; + let first = client(&server, &pki, Duration::from_millis(80)); + assert!(matches!( + first.register(&identity_store, &credential_store, &token()).await, + Err(ClientError::Unavailable { .. }) + )); + + let restarted = client(&server, &pki, Duration::from_secs(2)); + let credential = restarted + .register(&identity_store, &credential_store, &token()) + .await + .expect("restart replays completed exchange"); + assert_eq!(credential.uid, DEVICE_UID); + assert_eq!(credential.certificate_serial, "80".repeat(16)); + + let seen = server.seen.lock().expect("seen lock"); + assert_eq!(seen.len(), 4); + for request in &seen[1..] { + assert_eq!(request["requestId"], seen[0]["requestId"]); + assert_eq!(request["certificateRequest"], seen[0]["certificateRequest"]); + } +} + +#[tokio::test] +async fn registration_rejects_untrusted_or_misbound_credentials() { + for case in ["san", "chain", "key", "key_id", "cluster", "name"] { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let identity = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let mut response = match case { + "san" => pki.credential(&identity, "urn:rustfs:connect:device:0198f4b0-3c00-7e30-8f41-4a5b6c7d8e93", 2), + "chain" => TestPki::new().credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 2), + "key" => pki.credential( + &rustfs::connect::DeviceIdentity::generate(), + &format!("urn:rustfs:connect:device:{DEVICE_UID}"), + 2, + ), + _ => pki.credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 2), + }; + match case { + "key_id" => response["keyId"] = json!("x509-deadbeef"), + "cluster" => response["cluster"] = json!(format!("organizations/{ORGANIZATION_UID}/clusters/other")), + "name" => { + response["name"] = json!(format!("organizations/{ORGANIZATION_UID}/clusters/other/clusterDevices/{DEVICE_UID}")) + } + _ => {} + } + let server = server(&pki, vec![Reply::Json(StatusCode::CREATED, response)]).await; + let error = client(&server, &pki, Duration::from_secs(2)) + .register(&identity_store, &credential_store, &token()) + .await + .expect_err("invalid returned identity must fail closed"); + assert!(matches!(error, ClientError::Credential(_))); + assert!(!temp.path().join("credential/device.crt.json").exists()); + } +} + +#[tokio::test] +async fn stored_credential_is_revalidated_before_reuse() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let identity = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 4); + let server = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + let client = client(&server, &pki, Duration::from_secs(2)); + client + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + + let path = temp.path().join("credential/device.crt.json"); + let mut stored: Value = serde_json::from_slice(&fs::read(&path).expect("read credential")).expect("credential JSON"); + stored["certificateSerial"] = json!("00".repeat(16)); + fs::write(&path, serde_json::to_vec(&stored).expect("credential JSON")).expect("tamper credential"); + let error = client + .register(&identity_store, &credential_store, &token()) + .await + .expect_err("tampered stored credential must fail closed"); + assert!(matches!(error, ClientError::Credential(_))); +} + +#[tokio::test] +async fn register_rejects_expired_and_not_yet_valid_stored_credentials() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let identity = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 9); + let registration = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + let client = client(®istration, &pki, Duration::from_secs(2)); + client + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + + let now = OffsetDateTime::now_utc().replace_nanosecond(0).expect("whole second"); + let path = temp.path().join("credential/device.crt.json"); + let expired = pki.credential_window( + &identity, + &format!("urn:rustfs:connect:device:{DEVICE_UID}"), + 10, + now - time::Duration::days(2), + now - time::Duration::days(1), + ); + write_stored_credential(&path, &expired); + assert!(matches!( + client.register(&identity_store, &credential_store, &token()).await, + Err(ClientError::CredentialExpired) + )); + + let future = pki.credential_window( + &identity, + &format!("urn:rustfs:connect:device:{DEVICE_UID}"), + 11, + now + time::Duration::hours(1), + now + time::Duration::hours(25), + ); + write_stored_credential(&path, &future); + assert!(matches!( + client.register(&identity_store, &credential_store, &token()).await, + Err(ClientError::CredentialNotYetValid) + )); +} + +#[tokio::test] +async fn concurrent_rotation_retries_converge_and_promote_the_next_key() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let current = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let mut issued = pki.credential(¤t, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 5); + issued["certificateChain"] = json!( + issued["certificateChain"] + .as_str() + .expect("certificate chain") + .trim_end_matches('\n') + ); + let registration = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + let registered = client(®istration, &pki, Duration::from_secs(2)) + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + + let retries = server( + &pki, + vec![ + Reply::DelayedClose(Duration::from_millis(200)), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})), + ], + ) + .await; + let retry_client = client(&retries, &pki, Duration::from_millis(80)); + let due = registered.not_after_unix - 8 * 60 * 60; + let (first, second) = tokio::join!( + retry_client.rotate_if_due(&identity_store, &credential_store, due), + retry_client.rotate_if_due(&identity_store, &credential_store, due) + ); + assert!(matches!(first, Err(ClientError::Unavailable { .. }))); + assert!(matches!(second, Err(ClientError::Unavailable { .. }))); + let (request_id, certificate_request) = { + let seen = retries.seen.lock().expect("seen lock"); + assert!(seen.len() >= 3, "bounded retries must reach the server"); + for request in &seen[1..] { + assert_eq!(request["requestId"], seen[0]["requestId"]); + assert_eq!(request["certificateRequest"], seen[0]["certificateRequest"]); + } + (seen[0]["requestId"].clone(), seen[0]["certificateRequest"].clone()) + }; + + let next_der = fs::read(temp.path().join("identity/device.key.next")).expect("read staged next key"); + let next = rustfs::connect::DeviceIdentity::from_pkcs8_der(&next_der).expect("parse next key"); + assert_ne!(current.public_key_der(), next.public_key_der()); + assert_eq!( + identity_store + .load() + .expect("load current key") + .expect("current key") + .public_key_der(), + current.public_key_der() + ); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + let mode = fs::metadata(temp.path().join("identity/device.key.next")) + .expect("next key metadata") + .permissions() + .mode() + & 0o7777; + assert_eq!(mode, 0o600); + } + let (rotated, _) = rotation_response(&pki, &next, 6); + let success = server_with_client_auth( + &pki, + vec![Reply::VerifiedRotation { + response: rotated, + current_public_key: current.public_key_der(), + current_certificate_fingerprint: certificate_fingerprint(®istered.certificate), + device_name: registered.name.clone(), + }], + true, + ) + .await; + let success_client = client(&success, &pki, Duration::from_secs(2)); + let (due_result, current_result) = tokio::join!( + success_client.rotate_if_due(&identity_store, &credential_store, due), + success_client.rotate_if_due(&identity_store, &credential_store, OffsetDateTime::now_utc().unix_timestamp()) + ); + let credential = due_result + .expect("retry rotation") + .or(current_result.expect("concurrent current-state check")) + .expect("exactly one rotation is due"); + assert_eq!(credential.certificate_serial, "06".repeat(16)); + let success_seen = success.seen.lock().expect("seen lock"); + assert_eq!(success_seen.len(), 1, "the post-commit actor must not publish stale state"); + assert_eq!(success_seen[0]["requestId"], request_id); + assert_eq!(success_seen[0]["certificateRequest"], certificate_request); + drop(success_seen); + assert_eq!( + identity_store + .load() + .expect("load key") + .expect("current key") + .public_key_der(), + next.public_key_der() + ); + assert!(!temp.path().join("identity/device.key.next").exists()); + assert!(!temp.path().join("credential/rotation.pending.json").exists()); +} + +#[tokio::test] +async fn rotation_commit_recovers_after_each_durable_step() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let current = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(¤t, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 7); + let registration = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + let registered = client(®istration, &pki, Duration::from_secs(2)) + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + let failed = server(&pki, vec![Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})); 3]).await; + let due = registered.not_after_unix - 8 * 60 * 60; + assert!(matches!( + client(&failed, &pki, Duration::from_secs(2)) + .rotate_if_due(&identity_store, &credential_store, due) + .await, + Err(ClientError::Unavailable { .. }) + )); + + let pending_path = temp.path().join("credential/rotation.pending.json"); + let pending = fs::read(&pending_path).expect("read pending state"); + let next_der = fs::read(temp.path().join("identity/device.key.next")).expect("read next key"); + let next = rustfs::connect::DeviceIdentity::from_pkcs8_der(&next_der).expect("parse next key"); + let (_, stored) = rotation_response(&pki, &next, 8); + write_stored_credential(&temp.path().join("credential/device.crt.json"), &stored); + + let idle = server(&pki, vec![]).await; + assert!( + client(&idle, &pki, Duration::from_secs(2)) + .rotate_if_due(&identity_store, &credential_store, OffsetDateTime::now_utc().unix_timestamp()) + .await + .expect("recover after credential save") + .is_none() + ); + assert_eq!( + identity_store + .load() + .expect("load key") + .expect("current key") + .public_key_der(), + next.public_key_der() + ); + + fs::write(&pending_path, pending).expect("restore pending after key commit"); + set_owner_only(&pending_path); + assert!( + client(&idle, &pki, Duration::from_secs(2)) + .rotate_if_due(&identity_store, &credential_store, OffsetDateTime::now_utc().unix_timestamp()) + .await + .expect("recover after key commit") + .is_none() + ); + assert!(!pending_path.exists()); +} + +#[tokio::test] +async fn pending_reenrollment_blocks_rotation_and_resumes_original_exchange() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let current = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(¤t, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 13); + let registration = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + let registered = client(®istration, &pki, Duration::from_secs(2)) + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + + let failed = server(&pki, vec![Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})); 3]).await; + assert!(matches!( + client(&failed, &pki, Duration::from_secs(2)) + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await, + Err(ClientError::Unavailable { .. }) + )); + + let pending_path = temp.path().join("credential/registration.pending.json"); + let pending = fs::read(&pending_path).expect("read pending reenrollment"); + let pending_document: Value = serde_json::from_slice(&pending).expect("pending JSON"); + let next_der = fs::read(temp.path().join("identity/device.key.next")).expect("read next key"); + let next = rustfs::connect::DeviceIdentity::from_pkcs8_der(&next_der).expect("parse next key"); + let enrolled = pki.credential(&next, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 14); + + let rotation = server(&pki, vec![]).await; + let error = client(&rotation, &pki, Duration::from_secs(2)) + .rotate_if_due(&identity_store, &credential_store, registered.not_after_unix - 8 * 60 * 60) + .await + .expect_err("pending reenrollment blocks rotation"); + assert!(matches!(error, ClientError::PendingRegistration)); + assert!(rotation.seen.lock().expect("seen lock").is_empty()); + + let resumed = server(&pki, vec![Reply::Json(StatusCode::CREATED, enrolled)]).await; + let credential = client(&resumed, &pki, Duration::from_secs(2)) + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await + .expect("resume reenrollment"); + assert_eq!(credential.certificate_serial, "0e".repeat(16)); + let resumed_seen = resumed.seen.lock().expect("seen lock"); + assert_eq!(resumed_seen.len(), 1); + assert_eq!(resumed_seen[0]["requestId"], pending_document["requestId"]); + assert_eq!(resumed_seen[0]["certificateRequest"], pending_document["certificateRequest"]); +} + +#[tokio::test] +async fn reenrollment_commit_recovers_after_each_durable_step() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let current = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(¤t, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 13); + let registration = server(&pki, vec![Reply::Json(StatusCode::CREATED, issued)]).await; + client(®istration, &pki, Duration::from_secs(2)) + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + + let failed = server(&pki, vec![Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})); 3]).await; + assert!(matches!( + client(&failed, &pki, Duration::from_secs(2)) + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await, + Err(ClientError::Unavailable { .. }) + )); + + let pending_path = temp.path().join("credential/registration.pending.json"); + let pending = fs::read(&pending_path).expect("read pending reenrollment"); + let next_der = fs::read(temp.path().join("identity/device.key.next")).expect("read next key"); + let next = rustfs::connect::DeviceIdentity::from_pkcs8_der(&next_der).expect("parse next key"); + let enrolled = pki.credential(&next, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 14); + write_stored_credential(&temp.path().join("credential/device.crt.json"), &enrolled); + + let idle = server(&pki, vec![]).await; + let idle_client = client(&idle, &pki, Duration::from_secs(2)); + let recovered = idle_client + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await + .expect("recover after reenrollment credential save"); + assert_eq!(recovered.certificate_serial, "0e".repeat(16)); + assert_eq!( + identity_store + .load() + .expect("load key") + .expect("current key") + .public_key_der(), + next.public_key_der() + ); + + fs::remove_file(temp.path().join("credential/registration.completed.json")).expect("remove completed receipt"); + fs::write(&pending_path, pending).expect("restore pending after key commit"); + set_owner_only(&pending_path); + let recovered = idle_client + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await + .expect("recover after reenrollment key commit"); + assert_eq!(recovered.certificate_serial, "0e".repeat(16)); + assert!(!pending_path.exists()); + let recovered = idle_client + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await + .expect("completed reenrollment is idempotent after pending cleanup"); + assert_eq!(recovered.certificate_serial, "0e".repeat(16)); + assert!(idle.seen.lock().expect("seen lock").is_empty()); + + let different = server(&pki, vec![Reply::Json(StatusCode::SERVICE_UNAVAILABLE, json!({})); 3]).await; + assert!(matches!( + client(&different, &pki, Duration::from_secs(2)) + .reenroll(&identity_store, &credential_store, &token_with_uid(SECOND_TOKEN_UID)) + .await, + Err(ClientError::Unavailable { .. }) + )); + assert_eq!(different.seen.lock().expect("seen lock").len(), 3); +} + +#[test] +fn registration_token_schema_is_strict_and_bounded() { + let mut document = serde_json::to_value(token_document()).expect("token document"); + document["unexpected"] = json!(true); + assert!(matches!( + RegistrationToken::from_reader(serde_json::to_vec(&document).expect("token JSON").as_slice()), + Err(TokenError::Invalid(_)) + )); + assert!(matches!( + RegistrationToken::from_reader(vec![b' '; 16 * 1024 + 1].as_slice()), + Err(TokenError::TooLarge) + )); + let mut malformed = token_document(); + malformed["challengeNonce"] = json!("A".repeat(64)); + assert!(matches!( + RegistrationToken::from_reader(serde_json::to_vec(&malformed).expect("token JSON").as_slice()), + Err(TokenError::Shape) + )); +} + +#[tokio::test] +async fn rotation_waits_for_threshold_and_stops_on_revocation() { + let temp = tempfile::tempdir().expect("temp dir"); + let (identity_store, credential_store) = stores(&temp); + let identity = identity_store.load_or_create().expect("create identity"); + let pki = TestPki::new(); + let issued = pki.credential(&identity, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 3); + let rotation_server = server( + &pki, + vec![ + Reply::Json(StatusCode::CREATED, issued), + Reply::Json(StatusCode::UNAUTHORIZED, json!({"details": [{"reason": "DEVICE_REVOKED"}]})), + ], + ) + .await; + let connect = client(&rotation_server, &pki, Duration::from_secs(2)); + let current = connect + .register(&identity_store, &credential_store, &token()) + .await + .expect("register"); + assert!( + connect + .rotate_if_due(&identity_store, &credential_store, current.not_before_unix) + .await + .expect("not due") + .is_none() + ); + let error = connect + .rotate_if_due(&identity_store, &credential_store, current.not_after_unix - 8 * 60 * 60) + .await + .expect_err("revocation must stop rotation"); + assert!(matches!(error, ClientError::AccessRevoked { .. })); + assert!(error.to_string().contains("ConnectClient::reenroll")); + assert_eq!(rotation_server.seen.lock().expect("seen lock").len(), 2); + let stored: Value = + serde_json::from_slice(&fs::read(temp.path().join("credential/device.crt.json")).expect("read stored credential")) + .expect("stored credential JSON"); + assert_eq!(stored["certificateSerial"], current.certificate_serial); + + let next_der = fs::read(temp.path().join("identity/device.key.next")).expect("read staged next key"); + let next = rustfs::connect::DeviceIdentity::from_pkcs8_der(&next_der).expect("parse staged next key"); + let enrolled = pki.credential(&next, &format!("urn:rustfs:connect:device:{DEVICE_UID}"), 12); + let reenrollment = server(&pki, vec![Reply::Json(StatusCode::CREATED, enrolled)]).await; + let fresh = client(&reenrollment, &pki, Duration::from_secs(2)) + .reenroll(&identity_store, &credential_store, &token_with_uid(FRESH_TOKEN_UID)) + .await + .expect("fresh token reenrolls revoked credential"); + assert_eq!(fresh.certificate_serial, "0c".repeat(16)); + assert_eq!( + identity_store + .load() + .expect("load identity") + .expect("identity") + .public_key_der(), + next.public_key_der() + ); +} + +#[test] +fn unconfigured_connect_has_no_side_effects() { + let temp = tempfile::tempdir().expect("temp dir"); + let directory = temp.path().join("connect"); + assert!( + ConnectClient::from_optional_config(None) + .expect("unconfigured is valid") + .is_none() + ); + assert!(!directory.exists()); +} diff --git a/scripts/check_scheduled_validation_freshness.py b/scripts/check_scheduled_validation_freshness.py new file mode 100644 index 000000000..d9bdf0597 --- /dev/null +++ b/scripts/check_scheduled_validation_freshness.py @@ -0,0 +1,263 @@ +#!/usr/bin/env python3 +"""Fail when a critical scheduled validation has not started recently.""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timedelta, timezone +import json +import os +from pathlib import Path +import re +import sys +import tempfile +import unittest +from unittest import mock +from urllib.parse import quote, urlencode +from urllib.request import Request, urlopen + + +ROOT = Path(__file__).resolve().parents[1] + + +def load_validations(path: Path) -> list[tuple[str, int]]: + data = json.loads(path.read_text()) + if not isinstance(data, list) or not data: + raise ValueError("scheduled validation config must be a non-empty list") + + validations: list[tuple[str, int]] = [] + seen: set[str] = set() + for item in data: + if not isinstance(item, dict): + raise ValueError("scheduled validation entries must be objects") + workflow = item.get("workflow") + max_age_hours = item.get("max_age_hours") + if not isinstance(workflow, str) or not re.fullmatch( + r"\.github/workflows/[a-z0-9-]+\.yml", workflow + ): + raise ValueError(f"invalid scheduled validation workflow: {workflow!r}") + if workflow in seen: + raise ValueError(f"duplicate scheduled validation workflow: {workflow}") + if ( + not isinstance(max_age_hours, int) + or isinstance(max_age_hours, bool) + or max_age_hours <= 0 + ): + raise ValueError(f"invalid max_age_hours for {workflow}: {max_age_hours!r}") + seen.add(workflow) + validations.append((workflow, max_age_hours)) + return validations + + +def parse_timestamp(value: object) -> datetime: + if not isinstance(value, str): + raise ValueError(f"invalid run timestamp: {value!r}") + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + if parsed.tzinfo is None: + raise ValueError(f"run timestamp has no timezone: {value!r}") + return parsed.astimezone(timezone.utc) + + +def stale_reason( + run: dict[str, object] | None, now: datetime, max_age_hours: int +) -> str | None: + if run is None: + return "no scheduled run has been recorded" + created_at = parse_timestamp(run.get("created_at")) + age = now - created_at + if age > timedelta(hours=max_age_hours): + return f"last scheduled run is {age.total_seconds() / 3600:.1f}h old" + return None + + +def fetch_latest_scheduled_run( + repository: str, workflow: str, token: str, api_url: str +) -> dict[str, object] | None: + owner, repo = repository.split("/", 1) + workflow_name = Path(workflow).name + endpoint = ( + f"{api_url.rstrip('/')}/repos/{quote(owner, safe='')}/{quote(repo, safe='')}" + f"/actions/workflows/{quote(workflow_name, safe='')}/runs?" + + urlencode({"event": "schedule", "per_page": 1}) + ) + request = Request( + endpoint, + headers={ + "Accept": "application/vnd.github+json", + "Authorization": f"Bearer {token}", + "X-GitHub-Api-Version": "2022-11-28", + }, + ) + with urlopen(request, timeout=30) as response: + payload = json.load(response) + runs = payload.get("workflow_runs") + if not isinstance(runs, list): + raise ValueError(f"GitHub returned no workflow_runs list for {workflow}") + if not runs: + return None + if not isinstance(runs[0], dict): + raise ValueError(f"GitHub returned an invalid workflow run for {workflow}") + return runs[0] + + +def write_report(path: Path, failures: list[tuple[str, int, str, str]]) -> None: + lines = ["## Scheduled validation freshness"] + if not failures: + lines.append("") + lines.append("All critical scheduled validations have a recent scheduled run.") + else: + lines.extend( + [ + "", + "The following critical validations are stale or could not be inspected:", + "", + "| Workflow | Limit | Result | Last run |", + "| --- | ---: | --- | --- |", + ] + ) + for workflow, max_age_hours, reason, run_url in failures: + link = f"[open]({run_url})" if run_url else "—" + lines.append(f"| `{workflow}` | {max_age_hours}h | {reason} | {link} |") + path.write_text("\n".join(lines) + "\n") + + +def check_freshness( + config: Path, report: Path, repository: str, token: str, api_url: str +) -> int: + now = datetime.now(timezone.utc) + failures: list[tuple[str, int, str, str]] = [] + for workflow, max_age_hours in load_validations(config): + try: + run = fetch_latest_scheduled_run(repository, workflow, token, api_url) + reason = stale_reason(run, now, max_age_hours) + if reason is not None: + run_url = str(run.get("html_url", "")) if run else "" + failures.append((workflow, max_age_hours, reason, run_url)) + except Exception as error: + failures.append( + (workflow, max_age_hours, f"inspection failed: {error}", "") + ) + write_report(report, failures) + return 1 if failures else 0 + + +class SelfTests(unittest.TestCase): + NOW = datetime(2026, 8, 22, 12, tzinfo=timezone.utc) + + def test_freshness_boundaries(self) -> None: + at_limit = {"created_at": "2026-08-21T00:00:00Z"} + past_limit = {"created_at": "2026-08-20T23:59:59Z"} + self.assertIsNone(stale_reason(at_limit, self.NOW, 36)) + self.assertIsNotNone(stale_reason(past_limit, self.NOW, 36)) + self.assertIsNotNone(stale_reason(None, self.NOW, 36)) + + def test_config_rejects_duplicate_and_invalid_entries(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "validations.json" + path.write_text( + json.dumps( + [ + {"workflow": ".github/workflows/ci.yml", "max_age_hours": 36}, + {"workflow": ".github/workflows/ci.yml", "max_age_hours": 0}, + ] + ) + ) + with self.assertRaises(ValueError): + load_validations(path) + path.write_text( + json.dumps( + [{"workflow": ".github/workflows/ci.yml", "max_age_hours": 0}] + ) + ) + with self.assertRaises(ValueError): + load_validations(path) + + def test_check_reports_missing_runs(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + config = root / "validations.json" + report = root / "report.md" + config.write_text( + json.dumps( + [ + {"workflow": ".github/workflows/ci.yml", "max_age_hours": 36}, + {"workflow": ".github/workflows/fuzz.yml", "max_age_hours": 36}, + {"workflow": ".github/workflows/mint.yml", "max_age_hours": 36}, + ] + ) + ) + with mock.patch( + __name__ + ".fetch_latest_scheduled_run", + side_effect=[ + {"created_at": "2999-01-01T00:00:00Z"}, + None, + RuntimeError("API unavailable"), + ], + ): + self.assertEqual( + check_freshness( + config, + report, + "rustfs/rustfs", + "token", + "https://api.github.test", + ), + 1, + ) + contents = report.read_text() + self.assertIn(".github/workflows/fuzz.yml", contents) + self.assertIn("inspection failed: API unavailable", contents) + self.assertNotIn(".github/workflows/ci.yml`", contents) + + config.write_text( + json.dumps( + [{"workflow": ".github/workflows/ci.yml", "max_age_hours": 36}] + ) + ) + with mock.patch( + __name__ + ".fetch_latest_scheduled_run", + return_value={"created_at": "2999-01-01T00:00:00Z"}, + ): + self.assertEqual( + check_freshness( + config, + report, + "rustfs/rustfs", + "token", + "https://api.github.test", + ), + 0, + ) + self.assertIn("All critical scheduled validations", report.read_text()) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--config", type=Path, default=ROOT / ".github/scheduled-validations.json" + ) + parser.add_argument("--report", type=Path) + parser.add_argument("--self-test", action="store_true") + args = parser.parse_args() + + if args.self_test: + load_validations(args.config) + suite = unittest.defaultTestLoader.loadTestsFromTestCase(SelfTests) + return ( + 0 if unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful() else 1 + ) + if args.report is None: + parser.error("--report is required unless --self-test is used") + + repository = os.environ.get("GITHUB_REPOSITORY", "") + token = os.environ.get("GH_TOKEN", "") + api_url = os.environ.get("GITHUB_API_URL", "https://api.github.com") + if not re.fullmatch(r"[^/\s]+/[^/\s]+", repository): + parser.error("GITHUB_REPOSITORY must be owner/repository") + if not token: + parser.error("GH_TOKEN is required") + return check_freshness(args.config, args.report, repository, token, api_url) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check_test_wiring.py b/scripts/check_test_wiring.py new file mode 100755 index 000000000..aa97dc944 --- /dev/null +++ b/scripts/check_test_wiring.py @@ -0,0 +1,1004 @@ +#!/usr/bin/env python3 +"""Fail when committed tests silently fall out of their execution wiring.""" + +from __future__ import annotations + +import hashlib +import json +import re +import sys +import tempfile +import tomllib +import unittest +from datetime import datetime, timezone +from unittest import mock +from pathlib import Path +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + + +ROOT = Path(__file__).resolve().parents[1] +SCHEDULED_ALERT_WORKFLOWS = tuple( + item["workflow"] + for item in json.loads((ROOT / ".github/scheduled-validations.json").read_text()) +) + + +def words(value: str) -> set[str]: + return {item.strip() for item in value.split(",") if item.strip()} + + +def rust_code_only(source: str) -> str: + """Blank Rust comments and literals while preserving byte positions.""" + def quoted_end(quote_index: int, delimiter: str) -> int: + end = quote_index + 1 + while end < len(source): + if source[end] == "\\": + end += 2 + elif source[end] == delimiter: + return end + 1 + else: + end += 1 + return end + + code = list(source) + index = 0 + while index < len(source): + if source.startswith("//", index): + end = source.find("\n", index) + end = len(source) if end < 0 else end + elif source.startswith("/*", index): + depth = 1 + end = index + 2 + while end < len(source) and depth: + if source.startswith("/*", end): + depth += 1 + end += 2 + elif source.startswith("*/", end): + depth -= 1 + end += 2 + else: + end += 1 + else: + raw = re.match(r'(?:br|cr|r)(?P#{0,})"', source[index:]) + if raw: + marker = '"' + raw.group("hashes") + end = source.find(marker, index + raw.end()) + end = len(source) if end < 0 else end + len(marker) + elif source[index] == '"' or source.startswith(('b"', 'c"'), index): + start_quote = index if source[index] == '"' else index + 1 + end = quoted_end(start_quote, '"') + elif source[index] == "'" and index + 2 < len(source) and ( + source[index + 1] == "\\" or source[index + 2] == "'" + ): + end = quoted_end(index, "'") + elif source.startswith("b'", index): + end = quoted_end(index + 1, "'") + else: + index += 1 + continue + + for offset in range(index, end): + if code[offset] != "\n": + code[offset] = " " + index = end + return "".join(code) + + +def declared(parent: Path, module: str) -> bool: + source = parent.read_text() + code = rust_code_only(source) + pattern = re.compile(rf"^\s*(?:pub(?:\([^)]*\))?\s+)?mod\s+{re.escape(module)}\s*;", re.MULTILINE) + allowed_preambles = { + "#[cfg(test)]": "#[cfg(test)]", + "#[cfg(all(test,target_os=))]": '#[cfg(all(test,target_os="linux"))]', + } + for match in pattern.finditer(code): + prefix = code[: match.start()] + depths = {"(": 0, "[": 0, "{": 0} + pairs = {")": "(", "]": "[", "}": "{"} + for char in prefix: + if char in depths: + depths[char] += 1 + elif char in pairs: + depths[pairs[char]] -= 1 + if any(depths.values()): + continue + + boundary = max(prefix.rfind(";"), prefix.rfind("{"), prefix.rfind("}")) + code_preamble = re.sub(r"\s+", "", prefix[boundary + 1 :]) + if not code_preamble: + return True + if code_preamble in allowed_preambles: + attr_start = prefix.rfind("#[cfg", boundary + 1) + if attr_start >= 0 and re.sub(r"\s+", "", source[attr_start : match.start()]) == allowed_preambles[code_preamble]: + return True + return False + + +def module_source(src: Path, directory: Path) -> Path | None: + if not directory.parts: + return src / "lib.rs" + + mod_file = src / directory / "mod.rs" + if mod_file.is_file(): + return mod_file + + sibling = src.joinpath(*directory.parts[:-1], f"{directory.name}.rs") + return sibling if sibling.is_file() else None + + +def check_e2e_modules(root: Path) -> list[str]: + src = root / "crates/e2e_test/src" + errors: list[str] = [] + for test_file in sorted(src.rglob("*_test.rs")): + relative = test_file.relative_to(root).as_posix() + directory = test_file.relative_to(src).parent + parent = module_source(src, directory) + if parent is None: + errors.append(f"{relative}: no canonical parent module") + continue + + if not declared(parent, test_file.stem): + errors.append(f"{relative}: not declared by {parent.relative_to(root).as_posix()}") + while directory.parts: + module = directory.name + directory = directory.parent + parent = module_source(src, directory) + if parent is None: + errors.append(f"{relative}: module {module} has no canonical parent") + break + if not declared(parent, module): + errors.append(f"{relative}: module {module} not declared by {parent.relative_to(root).as_posix()}") + return errors + + +def check_fuzz_targets(root: Path) -> list[str]: + manifest = tomllib.loads((root / "fuzz/Cargo.toml").read_text()) + expected = {item["name"] for item in manifest.get("bin", []) if "name" in item} + errors: list[str] = [] + if not expected: + return ["fuzz/Cargo.toml: no [[bin]] fuzz targets found"] + + runner = (root / "scripts/fuzz/run.sh").read_text() + match = re.search(r'^targets="([^"]+)"', runner, re.MULTILINE) + runner_targets = set(match.group(1).split()) if match else set() + if runner_targets != expected: + errors.append(f"scripts/fuzz/run.sh targets {sorted(runner_targets)} != manifest {sorted(expected)}") + + workflow = (root / ".github/workflows/fuzz.yml").read_text() + matrices = [words(value) for value in re.findall(r"^\s*target:\s*\[([^]]+)]", workflow, re.MULTILINE)] + if len(matrices) != 2: + errors.append(f".github/workflows/fuzz.yml: expected smoke and nightly target matrices, found {len(matrices)}") + for index, matrix in enumerate(matrices, start=1): + if matrix != expected: + errors.append(f".github/workflows/fuzz.yml matrix {index} {sorted(matrix)} != manifest {sorted(expected)}") + + runtime_targets = re.findall(r"^\s*FUZZ_TARGET:\s*(\S.*?)\s*$", workflow, re.MULTILINE) + if runtime_targets != ["${{ matrix.target }}", "${{ matrix.target }}"]: + errors.append(".github/workflows/fuzz.yml: smoke and nightly jobs must pass matrix.target to FUZZ_TARGET") + + dependency_paths = { + f"{path.removeprefix('../')}/**" + for dependency in manifest.get("dependencies", {}).values() + if isinstance(dependency, dict) + and isinstance(path := dependency.get("path"), str) + and path.startswith("../crates/") + } + missing_paths = sorted(path for path in dependency_paths if f'"{path}"' not in workflow) + if missing_paths: + errors.append(f".github/workflows/fuzz.yml missing direct dependency paths: {', '.join(missing_paths)}") + + staged_matches = re.findall(r"^\s*for target in ([^;]+); do", workflow, re.MULTILINE) + staged = set(staged_matches[0].split()) if staged_matches else set() + if staged != expected: + errors.append(f".github/workflows/fuzz.yml staged binaries {sorted(staged)} != manifest {sorted(expected)}") + + return errors + + +def check_runner_selection(root: Path) -> list[str]: + runner = (root / "scripts/run_e2e_tests.sh").read_text() + errors: list[str] = [] + if "--include-ignored" not in runner: + errors.append("scripts/run_e2e_tests.sh: runner must include default and ignored tests") + if "--test-threads=1" not in runner: + errors.append("scripts/run_e2e_tests.sh: runner must serialize fixed-port protocol tests") + if re.search(r"(? list[str]: + runner = (root / "scripts/s3-tests/run.sh").read_text() + if "--showlocals" in runner: + return ["scripts/s3-tests/run.sh: pytest failure diagnostics must not dump local values"] + return [] + + +def profile_selection(root: Path, profile: str) -> str: + if not re.fullmatch(r"e2e-[a-z0-9-]+", profile): + raise ValueError(f"invalid e2e profile name: {profile}") + path = root / f".config/{profile}-selection.txt" + lines = [line for line in path.read_text().splitlines() if line.strip()] + values = dict(line.split("=", 1) for line in lines if "=" in line) + if len(values) != len(lines) or any(not re.fullmatch(r"sha256(?:-[a-z0-9]+)?", key) for key in values): + raise ValueError(f"{path.relative_to(root).as_posix()}: invalid sha256 entry") + key = f"sha256-{sys.platform}" + digest = values.get(key, values.get("sha256", "")) + if not re.fullmatch(r"[0-9a-f]{64}", digest): + raise ValueError(f"{path.relative_to(root).as_posix()}: missing sha256 for {sys.platform}") + return digest + + +def check_profile_definitions(root: Path) -> list[str]: + config = tomllib.loads((root / ".config/nextest.toml").read_text()) + profiles = { + profile + for profile in config.get("profile", {}) + if profile.startswith("e2e-") + } + selection_profiles = { + path.name.removesuffix("-selection.txt") for path in (root / ".config").glob("e2e-*-selection.txt") + } + errors: list[str] = [] + for profile in sorted(profiles | selection_profiles): + if profile not in profiles: + errors.append(f".config/nextest.toml: missing profile.{profile}") + if profile not in selection_profiles: + errors.append(f".config/{profile}-selection.txt: missing expected profile selection") + continue + try: + profile_selection(root, profile) + except (FileNotFoundError, ValueError) as error: + errors.append(str(error)) + return errors + + +def yaml_block(lines: list[str], key: str, indent: int) -> list[str] | None: + try: + start = lines.index(f"{' ' * indent}{key}:") + 1 + except ValueError: + return None + end = next( + ( + index + for index in range(start, len(lines)) + if lines[index].strip() + and not lines[index].lstrip().startswith("#") + and len(lines[index]) - len(lines[index].lstrip()) <= indent + ), + len(lines), + ) + return lines[start:end] + + +def workflow_step_block(job_lines: list[str], action: str) -> tuple[int, list[str]] | None: + uses_index = next( + ( + index + for index, line in enumerate(job_lines) + if ( + line.split("#", 1)[0].strip() == f"- uses: {action}" + and len(line) - len(line.lstrip()) == 6 + ) + or ( + line.split("#", 1)[0].strip() == f"uses: {action}" + and len(line) - len(line.lstrip()) == 8 + ) + ), + None, + ) + if uses_index is None: + return None + start = next( + ( + index + for index in range(uses_index, -1, -1) + if job_lines[index].lstrip().startswith("- ") + ), + uses_index, + ) + indent = len(job_lines[start]) - len(job_lines[start].lstrip()) + end = next( + ( + index + for index in range(start + 1, len(job_lines)) + if len(job_lines[index]) - len(job_lines[index].lstrip()) == indent + and job_lines[index].lstrip().startswith("- ") + ), + len(job_lines), + ) + return start, job_lines[start:end] + + +def alert_step_errors( + job_lines: list[str], + expected_action_if: str | None, + required_permissions: tuple[str, ...], + required_action_tokens: tuple[str, ...], +) -> list[str]: + checkout = workflow_step_block(job_lines, "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0") + action = workflow_step_block(job_lines, "./.github/actions/schedule-failure-issue") + errors: list[str] = [] + permissions = yaml_block(job_lines, "permissions", 4) + permission_text = "\n".join(line.split("#", 1)[0] for line in permissions or []) + missing_permissions = [token for token in required_permissions if token not in permission_text] + if missing_permissions: + errors.append("alert job permissions missing " + ", ".join(missing_permissions)) + if checkout is None: + errors.append("checkout step is missing") + if action is None: + errors.append("local alert action step is missing") + if checkout is None or action is None: + return errors + + if checkout[0] >= action[0]: + errors.append("checkout must run before the local alert action") + checkout_ifs = [line.strip() for line in checkout[1] if line.strip().startswith("if:")] + if checkout_ifs: + errors.append("checkout step must not be conditional") + action_ifs = [line.strip() for line in action[1] if line.strip().startswith("if:")] + expected_ifs = [] if expected_action_if is None else [expected_action_if] + if action_ifs != expected_ifs: + errors.append("alert action has an invalid step condition") + action_text = "\n".join(line.split("#", 1)[0] for line in action[1]) + missing_action_tokens = [token for token in required_action_tokens if token not in action_text] + if missing_action_tokens: + errors.append("alert action inputs missing " + ", ".join(missing_action_tokens)) + return errors + + +def schedule_utc_slots(hour: int, minute: int, timezone_name: str | None) -> set[tuple[int, int]]: + if timezone_name is None: + return {(hour, minute)} + zone = ZoneInfo(timezone_name) + return { + (utc.hour, utc.minute) + for year in (2025, 2026) + for month in range(1, 13) + for utc in [datetime(year, month, 1, hour, minute, tzinfo=zone).astimezone(timezone.utc)] + } + + +def check_scheduled_alerts(root: Path) -> list[str]: + errors: list[str] = [] + schedule_slots: dict[tuple[int, int], list[str]] = {} + for relative in SCHEDULED_ALERT_WORKFLOWS: + path = root / relative + try: + lines = path.read_text().splitlines() + except FileNotFoundError: + errors.append(f"{relative}: missing scheduled validation workflow") + continue + + on_block = yaml_block(lines, "on", 0) + schedule_block = yaml_block(on_block or [], "schedule", 2) + schedule_lines = schedule_block or [] + cron_indices = [index for index, line in enumerate(schedule_lines) if re.match(r"^\s*-\s+cron:", line)] + if not cron_indices: + errors.append(f"{relative}: missing simple numeric schedule") + else: + for position, cron_index in enumerate(cron_indices): + cron_line = schedule_lines[cron_index] + schedule = re.match(r"^\s*-\s+cron:\s*[\"']?(\d+)\s+(\d+)\s+", cron_line) + if not schedule: + errors.append(f"{relative}: missing simple numeric schedule") + continue + minute, hour = map(int, schedule.groups()) + if minute == 0: + errors.append(f"{relative}: scheduled validation must avoid minute zero") + entry_end = cron_indices[position + 1] if position + 1 < len(cron_indices) else len(schedule_lines) + entry = "\n".join(schedule_lines[cron_index + 1 : entry_end]) + timezone_match = re.search(r"^\s*timezone:\s*[\"']?([^\"'\s]+)", entry, re.MULTILINE) + timezone_name = timezone_match.group(1) if timezone_match else None + try: + utc_slots = schedule_utc_slots(hour, minute, timezone_name) + except ZoneInfoNotFoundError: + errors.append(f"{relative}: unknown schedule timezone {timezone_name}") + continue + for slot in utc_slots: + schedule_slots.setdefault(slot, []).append(relative) + + job_lines = yaml_block(lines, "alert-on-failure", 2) + if job_lines is None: + errors.append(f"{relative}: missing alert-on-failure job") + continue + job = "\n".join(line.split("#", 1)[0] for line in job_lines) + required = ( + "always()", + "github.event_name == 'schedule'", + "contains(needs.*.result, 'failure')", + "issues: write", + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: ./.github/actions/schedule-failure-issue", + "github-token: ${{ secrets.GITHUB_TOKEN }}", + ) + missing = [token for token in required if token not in job] + if missing: + errors.append(f"{relative}: alert-on-failure missing {', '.join(missing)}") + else: + errors.extend( + f"{relative}: {error}" + for error in alert_step_errors(job_lines, None, ("issues: write",), ("github-token: ${{ secrets.GITHUB_TOKEN }}",)) + ) + + for (hour, minute), workflows in schedule_slots.items(): + if len(workflows) > 1: + errors.append( + f"scheduled validations share {hour:02d}:{minute:02d} UTC: {', '.join(workflows)}" + ) + + watchdog_path = root / ".github/workflows/scheduled-validation-watchdog.yml" + try: + watchdog_lines = watchdog_path.read_text().splitlines() + except FileNotFoundError: + errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing completion watchdog") + return errors + watchdog_on = yaml_block(watchdog_lines, "on", 0) + watchdog_run = yaml_block(watchdog_on or [], "workflow_run", 2) + watchdog_workflows = yaml_block(watchdog_run or [], "workflows", 4) + if watchdog_workflows is None: + errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing workflow_run workflows") + return errors + watchdog_sources = "\n".join(line.split("#", 1)[0] for line in watchdog_workflows) + for relative in SCHEDULED_ALERT_WORKFLOWS: + path = root / relative + if not path.is_file(): + continue + source = path.read_text() + match = re.search(r"^name:\s*[\"']?([^\"'\n]+)", source, re.MULTILINE) + if not match: + errors.append(f"{relative}: missing workflow name") + elif f'- "{match.group(1).strip()}"' not in watchdog_sources: + errors.append(f"{relative}: missing from scheduled completion watchdog") + watchdog_job_lines = yaml_block(watchdog_lines, "alert-on-incomplete-run", 2) + if watchdog_job_lines is None: + errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing alert-on-incomplete-run job") + return errors + watchdog_job = "\n".join(line.split("#", 1)[0] for line in watchdog_job_lines) + required = ( + "github.event.workflow_run.event == 'schedule'", + "github.event.workflow_run.conclusion != 'success'", + "github.event.workflow_run.conclusion != 'failure'", + "actions: read", + "issues: write", + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: ./.github/actions/schedule-failure-issue", + "github-token: ${{ secrets.GITHUB_TOKEN }}", + "workflow-name: ${{ github.event.workflow_run.name }}", + "source-run-id: ${{ github.event.workflow_run.id }}", + "source-run-attempt: ${{ github.event.workflow_run.run_attempt }}", + "source-event: ${{ github.event.workflow_run.event }}", + "source-ref-name: ${{ github.event.workflow_run.head_branch }}", + "source-sha: ${{ github.event.workflow_run.head_sha }}", + ) + missing = [token for token in required if token not in watchdog_job] + if missing: + errors.append( + ".github/workflows/scheduled-validation-watchdog.yml: missing " + ", ".join(missing) + ) + else: + errors.extend( + ".github/workflows/scheduled-validation-watchdog.yml: " + error + for error in alert_step_errors( + watchdog_job_lines, + None, + ("actions: read", "issues: write"), + ( + "github-token: ${{ secrets.GITHUB_TOKEN }}", + "workflow-name: ${{ github.event.workflow_run.name }}", + "source-run-id: ${{ github.event.workflow_run.id }}", + "source-run-attempt: ${{ github.event.workflow_run.run_attempt }}", + "source-event: ${{ github.event.workflow_run.event }}", + "source-ref-name: ${{ github.event.workflow_run.head_branch }}", + "source-sha: ${{ github.event.workflow_run.head_sha }}", + ), + ) + ) + + freshness_path = root / ".github/workflows/scheduled-validation-freshness.yml" + try: + freshness_lines = freshness_path.read_text().splitlines() + except FileNotFoundError: + errors.append(".github/workflows/scheduled-validation-freshness.yml: missing freshness check") + return errors + freshness_job_lines = yaml_block(freshness_lines, "check-freshness", 2) + if freshness_job_lines is None: + errors.append(".github/workflows/scheduled-validation-freshness.yml: missing check-freshness job") + return errors + freshness_job = "\n".join(line.split("#", 1)[0] for line in freshness_job_lines) + required = ( + "python3 scripts/check_scheduled_validation_freshness.py", + "actions: read", + "issues: write", + "if: failure()", + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: ./.github/actions/schedule-failure-issue", + "github-token: ${{ secrets.GITHUB_TOKEN }}", + "details-file: ${{ runner.temp }}/scheduled-validation-freshness.md", + ) + missing = [token for token in required if token not in freshness_job] + if missing: + errors.append( + ".github/workflows/scheduled-validation-freshness.yml: missing " + ", ".join(missing) + ) + else: + errors.extend( + ".github/workflows/scheduled-validation-freshness.yml: " + error + for error in alert_step_errors( + freshness_job_lines, + "if: failure()", + ("actions: read", "issues: write"), + ( + "github-token: ${{ secrets.GITHUB_TOKEN }}", + "details-file: ${{ runner.temp }}/scheduled-validation-freshness.md", + ), + ) + ) + if not (root / "scripts/check_scheduled_validation_freshness.py").is_file(): + errors.append("scripts/check_scheduled_validation_freshness.py: missing freshness checker") + return errors + + +def check_profile_listing(root: Path, profile: str, listing: Path) -> list[str]: + try: + expected_digest = profile_selection(root, profile) + data = json.loads(listing.read_text()) + selected = sorted( + f"{suite_id}::{test_name}" + for suite_id, suite in data["rust-suites"].items() + for test_name, testcase in suite["testcases"].items() + if testcase.get("filter-match", {}).get("status") == "matches" + ) + digest = hashlib.sha256(("\n".join(selected) + "\n").encode()).hexdigest() + except (FileNotFoundError, KeyError, TypeError, ValueError, json.JSONDecodeError) as error: + return [f"cannot read {profile} nextest listing: {error}"] + if digest != expected_digest: + return [ + f"{profile} selection changed: count={len(selected)} sha256={digest}; " + f"expected sha256={expected_digest}" + ] + print(f"{profile} selection OK: {len(selected)} tests, sha256={digest}") + return [] + + +def validate(root: Path) -> list[str]: + errors: list[str] = [] + errors.extend(check_e2e_modules(root)) + errors.extend(check_fuzz_targets(root)) + errors.extend(check_runner_selection(root)) + errors.extend(check_s3_tests_runner(root)) + errors.extend(check_profile_definitions(root)) + errors.extend(check_scheduled_alerts(root)) + return errors + + +class SelfTests(unittest.TestCase): + def test_e2e_requires_registration(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + src = root / "crates/e2e_test/src" + src.mkdir(parents=True) + (src / "lib.rs").write_text("") + test_file = src / "boundary_test.rs" + test_file.write_text("#[test]\nfn boundary() {}\n") + self.assertEqual(len(check_e2e_modules(root)), 1) + (src / "lib.rs").write_text("mod boundary_test;\n") + self.assertEqual(check_e2e_modules(root), []) + + nested = src / "protocols" + nested.mkdir() + (nested / "mod.rs").write_text("mod fixed_port_test;\n") + (nested / "fixed_port_test.rs").write_text("#[test]\nfn fixed_port() {}\n") + self.assertEqual(len(check_e2e_modules(root)), 1) + (src / "lib.rs").write_text("mod boundary_test;\nmod protocols;\n") + self.assertEqual(check_e2e_modules(root), []) + + (src / "lib.rs").write_text("#[cfg(any())]\nmod boundary_test;\nmod protocols;\n") + self.assertEqual(len(check_e2e_modules(root)), 1) + + (src / "lib.rs").write_text( + "#[cfg(any())]\n/// hidden module\nmod boundary_test;\n#[cfg_attr(test, cfg(any()))]\nmod protocols;\n" + ) + self.assertEqual(len(check_e2e_modules(root)), 2) + + (src / "lib.rs").write_text( + 'const PHANTOM: &str = r#"{\nmod boundary_test;\n"#;\ndiscard! { mod protocols; }\n' + ) + self.assertEqual(len(check_e2e_modules(root)), 2) + + (src / "lib.rs").write_text( + '#[cfg(all(test, target_os = r"windows" /* target_os = "linux" */))]\n' + "mod boundary_test;\nmod protocols;\n" + ) + self.assertEqual(len(check_e2e_modules(root)), 1) + + (src / "lib.rs").write_text( + '#[cfg(all(test, target_os = r"windows"))] // #[cfg(all(test, target_os = "linux"))]\n' + "mod boundary_test;\nmod protocols;\n" + ) + self.assertEqual(len(check_e2e_modules(root)), 1) + + def test_fuzz_runtime_uses_matrix_target(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "fuzz").mkdir() + (root / "scripts/fuzz").mkdir(parents=True) + (root / ".github/workflows").mkdir(parents=True) + (root / "fuzz/Cargo.toml").write_text( + 'dep = { path = "../crates/dep" }\n[[bin]]\nname = "one"\n' + ) + (root / "scripts/fuzz/run.sh").write_text('targets="one"\n') + (root / ".github/workflows/fuzz.yml").write_text( + 'paths:\n - "crates/dep/**"\n' + "target: [one]\nFUZZ_TARGET: fixed\n" + "target: [one]\nFUZZ_TARGET: ${{ matrix.target }}\n" + "for target in one; do\n" + " fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/one\n" + ) + self.assertEqual(len(check_fuzz_targets(root)), 1) + + def test_s3_runner_rejects_unbounded_failure_locals(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + runner = root / "scripts/s3-tests/run.sh" + runner.parent.mkdir(parents=True) + runner.write_text("tox -- -vv -ra --tb=long\n") + self.assertEqual(check_s3_tests_runner(root), []) + runner.write_text("tox -- -vv -ra --showlocals --tb=long\n") + self.assertEqual(len(check_s3_tests_runner(root)), 1) + with ( + mock.patch(__name__ + ".check_e2e_modules", return_value=[]), + mock.patch(__name__ + ".check_fuzz_targets", return_value=[]), + mock.patch(__name__ + ".check_runner_selection", return_value=[]), + mock.patch(__name__ + ".check_profile_definitions", return_value=[]), + mock.patch(__name__ + ".check_scheduled_alerts", return_value=[]), + ): + self.assertEqual(len(validate(root)), 1) + + def test_profile_listing_enforces_selection(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / ".config").mkdir() + digest = hashlib.sha256(b"suite::two\n").hexdigest() + (root / ".config/e2e-smoke-selection.txt").write_text(f"sha256={digest}\n") + listing = root / "listing.json" + listing.write_text( + json.dumps( + { + "rust-suites": { + "suite": { + "testcases": { + "one": {"filter-match": {"status": "matches"}}, + "two": {"filter-match": {"status": "mismatch"}}, + } + } + } + } + ) + ) + self.assertEqual(len(check_profile_listing(root, "e2e-smoke", listing)), 1) + + def test_profile_listing_binds_platform_digest(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / ".config").mkdir() + darwin_digest = hashlib.sha256(b"suite::darwin\n").hexdigest() + linux_digest = hashlib.sha256(b"suite::linux\n").hexdigest() + (root / ".config/e2e-full-selection.txt").write_text( + f"sha256-darwin={darwin_digest}\nsha256-linux={linux_digest}\n" + ) + listing = root / "listing.json" + listing.write_text( + json.dumps( + { + "rust-suites": { + "suite": { + "testcases": {"darwin": {"filter-match": {"status": "matches"}}} + } + } + } + ) + ) + with mock.patch.object(sys, "platform", "linux"): + self.assertEqual(len(check_profile_listing(root, "e2e-full", listing)), 1) + + def test_scheduled_alerts_require_completion_watchdog(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + alert = ( + " alert-on-failure:\n" + " if: always() && github.event_name == 'schedule' && " + "contains(needs.*.result, 'failure')\n" + " permissions:\n" + " issues: write\n" + " steps:\n" + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " - uses: ./.github/actions/schedule-failure-issue\n" + " with:\n" + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + ) + names: list[str] = [] + for index, relative in enumerate(SCHEDULED_ALERT_WORKFLOWS, start=1): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + names.append(path.stem) + path.write_text( + f'name: "{path.stem}"\n' + f'on:\n schedule:\n - cron: "{index} {index} * * *"\n' + f'jobs:\n{alert}' + ) + watchdog = root / ".github/workflows/scheduled-validation-watchdog.yml" + watchdog.write_text( + "on:\n workflow_run:\n workflows:\n" + + "\n".join(f' - "{name}"' for name in names) + + "\njobs:\n" + + " alert-on-incomplete-run:\n" + + " github.event.workflow_run.event == 'schedule'\n" + + " github.event.workflow_run.conclusion != 'success'\n" + + " github.event.workflow_run.conclusion != 'failure'\n" + + " permissions:\n" + + " actions: read\n" + + " issues: write\n" + + " steps:\n" + + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + + " - uses: ./.github/actions/schedule-failure-issue\n" + + " with:\n" + + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + + " workflow-name: ${{ github.event.workflow_run.name }}\n" + + " source-run-id: ${{ github.event.workflow_run.id }}\n" + + " source-run-attempt: ${{ github.event.workflow_run.run_attempt }}\n" + + " source-event: ${{ github.event.workflow_run.event }}\n" + + " source-ref-name: ${{ github.event.workflow_run.head_branch }}\n" + + " source-sha: ${{ github.event.workflow_run.head_sha }}\n" + ) + freshness = root / ".github/workflows/scheduled-validation-freshness.yml" + freshness.write_text( + "jobs:\n" + " check-freshness:\n" + " permissions:\n" + " actions: read\n" + " issues: write\n" + " steps:\n" + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " - run: python3 scripts/check_scheduled_validation_freshness.py\n" + " - uses: ./.github/actions/schedule-failure-issue\n" + " if: failure()\n" + " with:\n" + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + " details-file: ${{ runner.temp }}/scheduled-validation-freshness.md\n" + ) + checker = root / "scripts/check_scheduled_validation_freshness.py" + checker.parent.mkdir() + checker.write_text("") + self.assertEqual(check_scheduled_alerts(root), []) + + first = root / SCHEDULED_ALERT_WORKFLOWS[0] + mutations = ( + ("contains(needs.*.result, 'failure')", "false"), + ("issues: write", "issues: read"), + ( + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: actions/checkout@missing", + ), + ( + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n", + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " if: github.event_name == 'workflow_dispatch'\n", + ), + ( + " - uses: ./.github/actions/schedule-failure-issue\n", + " - uses: ./.github/actions/schedule-failure-issue\n" + " if: github.event_name == 'workflow_dispatch'\n", + ), + ("uses: ./.github/actions/schedule-failure-issue", "uses: actions/checkout@v7"), + ("github-token: ${{ secrets.GITHUB_TOKEN }}", "github-token: missing"), + ) + for required, replacement in mutations: + original = first.read_text() + first.write_text(original.replace(required, replacement)) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(original) + + first_original = first.read_text() + real_steps = ( + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " - uses: ./.github/actions/schedule-failure-issue\n" + " with:\n" + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + ) + first.write_text( + first_original.replace( + real_steps, + " - run: |\n" + " : <<'MARKER'\n" + " uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " MARKER\n" + " - run: |\n" + " : <<'MARKER'\n" + " uses: ./.github/actions/schedule-failure-issue\n" + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + " MARKER\n", + ) + ) + self.assertTrue(check_scheduled_alerts(root)) + first.write_text( + first_original.replace( + real_steps, + " - uses: ./.github/actions/schedule-failure-issue\n" + " with:\n" + " github-token: ${{ secrets.GITHUB_TOKEN }}\n" + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n", + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(first_original) + + watchdog_mutations = ( + ("actions: read", "actions: none"), + ("issues: write", "issues: read"), + ( + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: actions/checkout@missing", + ), + ( + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n", + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " if: github.event_name == 'workflow_dispatch'\n", + ), + ( + " - uses: ./.github/actions/schedule-failure-issue\n", + " - uses: ./.github/actions/schedule-failure-issue\n" + " if: github.event_name == 'workflow_dispatch'\n", + ), + ("uses: ./.github/actions/schedule-failure-issue", "uses: actions/checkout@v7"), + ("github-token: ${{ secrets.GITHUB_TOKEN }}", "github-token: missing"), + ("source-event: ${{ github.event.workflow_run.event }}", "source-event: watchdog"), + ( + "source-ref-name: ${{ github.event.workflow_run.head_branch }}", + "source-ref-name: main", + ), + ("source-sha: ${{ github.event.workflow_run.head_sha }}", "source-sha: missing"), + ) + for required, replacement in watchdog_mutations: + original = watchdog.read_text() + watchdog.write_text(original.replace(required, replacement)) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + watchdog.write_text(original) + + watchdog_original = watchdog.read_text() + watchdog.write_text( + watchdog_original.replace("issues: write", "issues: read") + + " decoy:\n permissions:\n issues: write\n" + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + watchdog.write_text(watchdog_original) + + first_original = first.read_text() + first.write_text( + first_original.replace(' schedule:\n - cron: "1 1 * * *"\n', "") + + ' decoy:\n strategy:\n matrix:\n cron:\n - "1 1 * * *"\n' + + ' runs-on: ubuntu-latest\n steps:\n - run: true\n' + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(first_original) + + first.write_text( + first_original.replace( + ' - cron: "1 1 * * *"\n', + ' - cron: "1 1 * * *"\n - cron: "0 5 * * *"\n', + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text( + first_original.replace( + ' - cron: "1 1 * * *"\n', + ' - cron: "1 1 * * *"\n - cron: "2 2 * * *"\n', + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(first_original) + + watchdog.write_text( + watchdog_original.replace(f' - "{names[0]}"\n', "") + + f' decoy:\n strategy:\n matrix:\n workflow:\n - "{names[0]}"\n' + + ' runs-on: ubuntu-latest\n steps:\n - run: true\n' + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + watchdog.write_text(watchdog_original) + + watchdog.write_text(watchdog_original.replace(f' - "{names[0]}"\n', "")) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + watchdog.write_text(watchdog_original) + original = first.read_text() + first.write_text(re.sub(r'- cron: "\d+ \d+', '- cron: "0 0', original, count=1)) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(original) + + second = root / SCHEDULED_ALERT_WORKFLOWS[1] + second_original = second.read_text() + second.write_text(re.sub(r'- cron: "\d+ \d+', '- cron: "1 1', second_original, count=1)) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + second.write_text(second_original) + + first.write_text( + first_original.replace( + ' - cron: "1 1 * * *"\n', + ' - cron: "7 0 * * *"\n timezone: "Asia/Shanghai"\n', + ) + ) + second.write_text( + second_original.replace( + ' - cron: "2 2 * * *"\n', + ' - cron: "2 2 * * *"\n - cron: "7 16 * * *"\n', + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + first.write_text(first_original) + second.write_text(second_original) + + freshness_original = freshness.read_text() + freshness.write_text(freshness_original.replace("details-file:", "report-file:")) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + freshness.write_text( + freshness_original.replace("github-token: ${{ secrets.GITHUB_TOKEN }}", "github-token: missing") + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + freshness.write_text( + freshness_original.replace( + "uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", + "uses: actions/checkout@missing", + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + freshness.write_text( + freshness_original.replace( + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n", + " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n" + " if: github.event_name == 'workflow_dispatch'\n", + ) + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + freshness.write_text( + freshness_original.replace("if: failure()", "if: github.event_name == 'workflow_dispatch'") + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + freshness.write_text( + freshness_original.replace("issues: write", "issues: read") + + " decoy:\n permissions:\n issues: write\n" + ) + self.assertEqual(len(check_scheduled_alerts(root)), 1) + +def main() -> int: + if sys.argv[1:] == ["--self-test"]: + suite = unittest.defaultTestLoader.loadTestsFromTestCase(SelfTests) + return 0 if unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful() else 1 + if len(sys.argv) == 4 and sys.argv[1] == "--check-profile": + errors = check_profile_listing(ROOT, sys.argv[2], Path(sys.argv[3])) + if errors: + for error in errors: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + return 0 + if sys.argv[1:]: + print( + "usage: check_test_wiring.py [--self-test | --check-profile PROFILE LISTING]", + file=sys.stderr, + ) + return 2 + + errors = validate(ROOT) + if errors: + for error in errors: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print("OK: e2e modules, runner selection, fuzz matrices, profiles, and scheduled alerts are wired") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/run_e2e_tests.sh b/scripts/run_e2e_tests.sh index 810aec5fd..8e98b202c 100755 --- a/scripts/run_e2e_tests.sh +++ b/scripts/run_e2e_tests.sh @@ -20,6 +20,8 @@ DATA_DIR="$TARGET_DIR/rustfs_test_data" RUSTFS_PID="" TEST_FILTER="" TEST_TYPE="all" +RUSTFS_BUILD_FEATURES="${RUSTFS_BUILD_FEATURES:-ftps,webdav,sftp}" +export RUSTFS_BUILD_FEATURES # Function to print colored output print_info() { @@ -92,7 +94,7 @@ build_rustfs() { print_info "Building RustFS..." cd "$PROJECT_ROOT" - if ! cargo build --bin rustfs; then + if ! cargo build --bin rustfs --features "$RUSTFS_BUILD_FEATURES"; then print_error "Failed to build RustFS" exit 1 fi @@ -219,27 +221,28 @@ start_rustfs() { run_tests() { print_info "Running e2e tests..." cd "$PROJECT_ROOT" - - local test_cmd="cargo test --package e2e_test --lib" - + + local test_cmd=(cargo test --package e2e_test --lib) + case "$TEST_TYPE" in "specific") - test_cmd="$test_cmd -- $TEST_FILTER --exact --show-output --ignored" + test_cmd+=(-- "$TEST_FILTER") print_info "Running specific test: $TEST_FILTER" ;; "file") - test_cmd="$test_cmd -- $TEST_FILTER --show-output --ignored" + test_cmd+=(-- "$TEST_FILTER") print_info "Running tests in file/module: $TEST_FILTER" ;; "all") - test_cmd="$test_cmd -- --show-output --ignored" + test_cmd+=(--) print_info "Running all e2e tests" ;; esac - - print_info "Test command: $test_cmd" - - if eval "$test_cmd"; then + test_cmd+=(--show-output --include-ignored --test-threads=1) + + print_info "Test command: ${test_cmd[*]}" + + if "${test_cmd[@]}"; then print_success "All tests passed!" return 0 else diff --git a/scripts/s3-tests/README.md b/scripts/s3-tests/README.md index dab4de556..fe06d679c 100644 --- a/scripts/s3-tests/README.md +++ b/scripts/s3-tests/README.md @@ -253,6 +253,9 @@ Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (defau - `junit.xml`: Test results in JUnit format (compatible with CI/CD systems) - `pytest.log`: Detailed pytest logs with full test output +- `all-collected-nodeids.txt`: Exact node IDs in the pinned upstream suite +- `selected-nodeids.txt`: Exact node IDs expected in this run +- `unsharded-selected-nodeids.txt`: Exact node IDs before deterministic sharding - `compat-report.md`: Classification report generated by `report_compat.py` — regressions against `implemented_tests.txt`, promotion candidates (tests that pass but are still listed as unimplemented/excluded), and tests missing @@ -449,9 +452,11 @@ RustFS. Two GitHub Actions workflows delegate to it: - **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`) against a Docker deployment — single node or a real 4-node distributed - cluster behind HAProxy. The sweep fails only on regressions in the - implemented whitelist; everything else is reported by `report_compat.py` - as promotion candidates or unclassified tests. + cluster behind HAProxy. Regressions, unclassified tests, incomplete + execution, and infrastructure errors fail the sweep; classified unsupported + behavior remains informational. Scheduled topology runs are split into four + deterministic exact-node-ID shards, and every case has a five-minute timeout, + so one stalled case cannot erase the entire sweep's evidence. Keeping both workflows on this script means local runs, the PR gate, and the scheduled sweep always execute tests the same way (same pinned s3-tests @@ -466,8 +471,9 @@ pass/fail table in the job summary. ## Companion Tools - `report_compat.py` — diffs a junit.xml result against the classification - lists; run automatically at the end of `run.sh`, and used by the weekly - sweep to gate on whitelist regressions only (`--fail-on-regression`). + lists and the exact pytest collection; run before execution to reject stale + or missing classifications, then after execution to detect regressions and + incomplete parameterized cases. - `api_coverage.py` — quantifies S3 API surface coverage by comparing the s3s `S3` trait (at the revision pinned in Cargo.toml) against the methods RustFS overrides in `impl S3 for FS`: diff --git a/scripts/s3-tests/excluded_tests.txt b/scripts/s3-tests/excluded_tests.txt index 637aaff8c..49401c91b 100644 --- a/scripts/s3-tests/excluded_tests.txt +++ b/scripts/s3-tests/excluded_tests.txt @@ -307,3 +307,10 @@ test_object_acl_write test_object_acl_writeacp test_put_bucket_acl_grant_group_read test_object_raw_get_bucket_acl + +# Require upstream cloud-storage or IAM account services +test_bucket_logging_requester_assumed_role +test_lifecycle_cloud_transition_target_by_bucket +test_lifecycle_cloud_transition_target_by_bucket_multiple_buckets +test_list_object_versions_restore_status +test_list_objects_restore_status diff --git a/scripts/s3-tests/implemented_tests.txt b/scripts/s3-tests/implemented_tests.txt index 0f29756e8..83b6f8513 100644 --- a/scripts/s3-tests/implemented_tests.txt +++ b/scripts/s3-tests/implemented_tests.txt @@ -521,9 +521,13 @@ test_atomic_dual_conditional_write_1mb test_atomic_write_bucket_gone test_bucket_acl_canned_private_to_private test_bucket_concurrent_set_canned_acl +test_bucket_create_delete +test_bucket_policy test_bucket_policy_acl test_bucket_policy_put_obj_acl test_bucketv2_policy_acl +test_copy_enc +test_copy_part_enc test_copy_object_ifmatch_failed test_copy_object_ifnonematch_good test_cors_presigned_put_object_tenant_with_acl diff --git a/scripts/s3-tests/report_compat.py b/scripts/s3-tests/report_compat.py index 078c22c1d..a3c44a74a 100755 --- a/scripts/s3-tests/report_compat.py +++ b/scripts/s3-tests/report_compat.py @@ -21,14 +21,17 @@ Classifies every executed test into: - unclassified passes: passed but not present in any list (new upstream tests) - unclassified failures: failed and not present in any list (new upstream tests) -Writes a markdown report and prints a summary to stdout. Exit code is 0 unless ---fail-on-regression is given and at least one regression was found. +Writes a markdown report and prints a summary to stdout. Optional gates reject +regressions, unclassified tests, stale classifications, and incomplete node-ID +execution. """ from __future__ import annotations import argparse +from collections import Counter import pathlib +import re import sys import xml.etree.ElementTree as ET @@ -45,33 +48,47 @@ LIST_FILES = { } -def load_list(path: pathlib.Path) -> set[str]: - names: set[str] = set() +def load_entries(path: pathlib.Path) -> list[str]: + names: list[str] = [] if not path.is_file(): return names for line in path.read_text(encoding="utf-8").splitlines(): line = line.strip() if line and not line.startswith("#"): - names.add(line) + names.append(line) return names +def classification_errors(entries: dict[str, list[str]]) -> list[str]: + errors: list[str] = [] + lists = {key: set(names) for key, names in entries.items()} + for key, names in entries.items(): + duplicates = sorted(name for name, count in Counter(names).items() if count > 1) + if duplicates: + errors.append(f"{LIST_FILES[key]} has duplicates: {', '.join(duplicates)}") + keys = tuple(lists) + for index, left in enumerate(keys): + for right in keys[index + 1 :]: + overlap = sorted(lists[left] & lists[right]) + if overlap: + errors.append(f"{left}/{right} classifications overlap: {', '.join(overlap)}") + return errors + + def base_name(testcase_name: str) -> str: """Strip pytest parametrization (test_foo[param]) to match list entries.""" return testcase_name.split("[", 1)[0] -def parse_junit(path: pathlib.Path) -> dict[str, str]: - """Return {test name: status} with status in passed/failed/error/skipped. - - Parametrized cases collapse onto their base name; any failing variant marks - the whole test failed. - """ +def parse_junit(path: pathlib.Path) -> tuple[dict[str, str], list[str], list[tuple[str, str, str, str]]]: + """Return exact statuses, pytest-timeout cases, and failure summaries.""" results: dict[str, str] = {} + timed_out: list[str] = [] + failures: list[tuple[str, str, str, str]] = [] severity = {"skipped": 0, "passed": 1, "failed": 2, "error": 2} root = ET.parse(path).getroot() for case in root.iter("testcase"): - name = base_name(case.get("name", "")) + name = case.get("name", "") if not name: continue if case.find("failure") is not None: @@ -85,7 +102,35 @@ def parse_junit(path: pathlib.Path) -> dict[str, str]: prev = results.get(name) if prev is None or severity[status] > severity[prev]: results[name] = status - return results + node = case.find("failure") if status == "failed" else case.find("error") + if node is not None: + details = " ".join(filter(None, [node.get("message", ""), node.text or ""])) + message = node.get("message") or next(iter((node.text or "").strip().splitlines()), "") + failures.append((case.get("classname", ""), name, case.get("time", "0"), message)) + if re.search(r"\bTimeout\s*(?:>|\()", details, re.IGNORECASE): + timed_out.append(name) + return results, timed_out, failures + + +def collapse_results(results: dict[str, str]) -> dict[str, str]: + """Collapse parametrized cases for classification-level reporting.""" + collapsed: dict[str, str] = {} + severity = {"skipped": 0, "passed": 1, "failed": 2, "error": 2} + for exact_name, status in results.items(): + name = base_name(exact_name) + previous = collapsed.get(name) + if previous is None or severity[status] > severity[previous]: + collapsed[name] = status + return collapsed + + +def load_collected_nodeids(path: pathlib.Path) -> set[str]: + names: set[str] = set() + for line in path.read_text(encoding="utf-8").splitlines(): + nodeid = line.strip() + if nodeid: + names.add(nodeid.rsplit("::", 1)[-1]) + return names def render_section(title: str, rows: list[str], hint: str = "") -> list[str]: @@ -102,7 +147,7 @@ def render_section(title: str, rows: list[str], hint: str = "") -> list[str]: def main() -> int: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--junit", required=True, type=pathlib.Path, help="junit.xml produced by pytest") + parser.add_argument("--junit", type=pathlib.Path, help="junit.xml produced by pytest") parser.add_argument( "--lists-dir", type=pathlib.Path, @@ -115,14 +160,60 @@ def main() -> int: action="store_true", help="exit non-zero when a test from implemented_tests.txt failed", ) + parser.add_argument( + "--fail-on-unclassified", + action="store_true", + help="exit non-zero when an executed test is absent from every classification", + ) + parser.add_argument( + "--collected-nodeids", + type=pathlib.Path, + help="exact pytest node IDs from the pinned suite's collect-only pass", + ) + parser.add_argument( + "--check-classifications-only", + action="store_true", + help="validate classification names against collected node IDs without reading JUnit", + ) args = parser.parse_args() - if not args.junit.is_file(): + entries = {key: load_entries(args.lists_dir / fname) for key, fname in LIST_FILES.items()} + lists = {key: set(names) for key, names in entries.items()} + invalid_classifications = classification_errors(entries) + collected: set[str] = set() + if args.collected_nodeids: + collected = load_collected_nodeids(args.collected_nodeids) + collected_base = {base_name(name) for name in collected} + classified = set().union(*lists.values()) + missing_classifications = sorted(collected_base - classified) + stale_classifications = sorted(classified - collected_base) + else: + missing_classifications = [] + stale_classifications = [] + + if args.check_classifications_only: + if not args.collected_nodeids: + parser.error("--check-classifications-only requires --collected-nodeids") + for error in invalid_classifications: + print(f"[INVALID] {error}") + for name in missing_classifications: + print(f"[UNCLASSIFIED] {name}") + for name in stale_classifications: + print(f"[STALE] {name}") + return 1 if invalid_classifications or missing_classifications or stale_classifications else 0 + + if invalid_classifications: + for error in invalid_classifications: + print(f"[ERROR] {error}", file=sys.stderr) + return 2 + + if not args.junit or not args.junit.is_file(): print(f"[ERROR] junit file not found: {args.junit}", file=sys.stderr) return 2 - lists = {key: load_list(args.lists_dir / fname) for key, fname in LIST_FILES.items()} - results = parse_junit(args.junit) + exact_results, timed_out, failures = parse_junit(args.junit) + results = collapse_results(exact_results) + missing_results = sorted(collected - exact_results.keys()) if collected else [] regressions: list[str] = [] promotions: dict[str, list[str]] = {"unimplemented": [], "excluded": []} @@ -155,7 +246,9 @@ def main() -> int: lines = [ "# S3 compatibility report", "", - f"Executed: {len(results)} tests — " + f"Executed: {len(exact_results)} exact cases across {len(results)} classified tests.", + "", + "Classification status — " f"{counts['passed']} passed, {counts['failed']} failed, " f"{counts['error']} errored, {counts['skipped']} skipped.", "", @@ -191,6 +284,16 @@ def main() -> int: unclassified_failed, "Failing and absent from every list — triage into `unimplemented_tests.txt` or `excluded_tests.txt`.", ) + lines += render_section( + "Missing results", + missing_results, + "Present in the pinned upstream suite but absent from JUnit — the sweep was incomplete.", + ) + lines += render_section( + "Timed out", + timed_out, + "Per-test timeout is an infrastructure failure regardless of compatibility classification.", + ) report = "\n".join(lines) if args.output: @@ -201,13 +304,28 @@ def main() -> int: print( f"[INFO] {len(regressions)} regression(s), " f"{len(promotions['unimplemented']) + len(promotions['excluded']) + len(unclassified_passed)} promotion candidate(s), " - f"{len(unclassified_failed)} unclassified failure(s)" + f"{len(unclassified_failed)} unclassified failure(s), " + f"{len(missing_results)} missing result(s), " + f"{len(timed_out)} timeout(s)" ) for name in sorted(regressions): print(f"[REGRESSION] {name}") + if failures: + print("[ERROR] s3-tests failed testcase summary:") + for classname, name, duration, message in failures[:20]: + nodeid = f"{classname}::{name}" if classname else name + print(f"[ERROR] - {nodeid} ({duration}s): {message}") + if len(failures) > 20: + print(f"[ERROR] - ... {len(failures) - 20} additional failed testcases omitted") if args.fail_on_regression and regressions: return 1 + if args.fail_on_unclassified and (unclassified_passed or unclassified_failed): + return 1 + if args.collected_nodeids and missing_results: + return 1 + if timed_out: + return 1 return 0 diff --git a/scripts/s3-tests/run.sh b/scripts/s3-tests/run.sh index d9fd15040..74e2289f6 100755 --- a/scripts/s3-tests/run.sh +++ b/scripts/s3-tests/run.sh @@ -58,6 +58,19 @@ if [[ "${TEST_SCOPE}" != "implemented" && "${TEST_SCOPE}" != "all" ]]; then echo "[ERROR] Invalid TEST_SCOPE: ${TEST_SCOPE} (must be \"implemented\" or \"all\")" >&2 exit 1 fi +S3_SHARD_COUNT="${S3_SHARD_COUNT:-1}" +S3_SHARD_INDEX="${S3_SHARD_INDEX:-0}" +TEST_TIMEOUT="${TEST_TIMEOUT:-300}" +if [[ ! "${S3_SHARD_COUNT}" =~ ^[1-9][0-9]*$ ]] \ + || [[ ! "${S3_SHARD_INDEX}" =~ ^[0-9]+$ ]] \ + || (( S3_SHARD_INDEX >= S3_SHARD_COUNT )); then + echo "[ERROR] Invalid S3 shard ${S3_SHARD_INDEX}/${S3_SHARD_COUNT}" >&2 + exit 1 +fi +if [[ ! "${TEST_TIMEOUT}" =~ ^[1-9][0-9]*$ ]]; then + echo "[ERROR] Invalid TEST_TIMEOUT: ${TEST_TIMEOUT}" >&2 + exit 1 +fi # Upstream ceph/s3-tests suite, pinned for reproducible runs. # Bump S3TESTS_REV deliberately: upstream changes can rename tests or change @@ -96,55 +109,6 @@ log_error() { echo -e "${RED}[ERROR]${NC} $*" } -summarize_junit_failures() { - local junit_path="$1" - - if [ ! -f "${junit_path}" ]; then - log_warn "JUnit report not found: ${junit_path}" - return 0 - fi - - python3 - "${junit_path}" <<'PY' -import sys -import xml.etree.ElementTree as ET - -junit_path = sys.argv[1] -try: - root = ET.parse(junit_path).getroot() -except Exception as exc: - print(f"[WARN] Failed to parse JUnit report {junit_path}: {exc}") - raise SystemExit(0) - -failures = [] -for case in root.iter("testcase"): - failure = case.find("failure") - error = case.find("error") - node = failure if failure is not None else error - if node is None: - continue - - classname = case.attrib.get("classname", "") - name = case.attrib.get("name", "") - duration = case.attrib.get("time", "0") - message = node.attrib.get("message") or (node.text or "").strip().splitlines()[0:1] - if isinstance(message, list): - message = message[0] if message else "" - failures.append((classname, name, duration, message)) - -if not failures: - print("[INFO] No failed testcases found in JUnit report") - raise SystemExit(0) - -print("[ERROR] s3-tests failed testcase summary:") -for classname, name, duration, message in failures[:20]: - nodeid = f"{classname}::{name}" if classname else name - print(f"[ERROR] - {nodeid} ({duration}s): {message}") - -if len(failures) > 20: - print(f"[ERROR] - ... {len(failures) - 20} additional failed testcases omitted") -PY -} - # ============================================================================= # Test Classification Files # ============================================================================= @@ -322,6 +286,9 @@ Environment Variables: MAXFAIL - Stop after N failures, 0 = never stop (default: 1) XDIST - Enable parallel execution with N workers (default: 0) TEST_SCOPE - "implemented" (whitelist, default) or "all" (entire upstream suite) + S3_SHARD_COUNT - Number of deterministic exact-node-ID shards (default: 1) + S3_SHARD_INDEX - Zero-based shard index (default: 0) + TEST_TIMEOUT - Per-test timeout in seconds (default: 300) S3TESTS_REPO - s3-tests repository URL (default: https://github.com/ceph/s3-tests.git) S3TESTS_REV - Pinned s3-tests commit; bump deliberately and reclassify test lists MARKEXPR - pytest marker expression (default: no marker filtering) @@ -982,9 +949,10 @@ mkdir -p "${ARTIFACTS_DIR}" XDIST_ARGS="" if [ "${XDIST}" != "0" ]; then # Add pytest-xdist to requirements.txt so tox installs it inside its virtualenv - echo "pytest-xdist" >> requirements.txt + grep -qxF "pytest-xdist" requirements.txt || echo "pytest-xdist" >> requirements.txt XDIST_ARGS="-n ${XDIST} --dist=loadgroup" fi +grep -qxF "pytest-timeout" requirements.txt || echo "pytest-timeout" >> requirements.txt # Resolve config path (absolute path for tox) if [[ "${S3TESTS_CONF}" = /* ]]; then @@ -1003,12 +971,70 @@ else PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}") fi +collect_nodeids() { + local output_path="$1" + shift + local collect_log="${output_path%.txt}.log" + local collect_rc=0 + local node_prefix="${S3_TEST_FILE//./\\.}::" + + set +e + S3TEST_CONF="${CONF_OUTPUT_PATH}" tox -- -q --collect-only "$@" 2>&1 | tee "${collect_log}" + collect_rc=${PIPESTATUS[0]} + set -e + if [ "${collect_rc}" -ne 0 ]; then + log_error "pytest collection failed with exit code ${collect_rc}" + return "${collect_rc}" + fi + grep -E "^${node_prefix}" "${collect_log}" > "${output_path}" || true + if [ ! -s "${output_path}" ]; then + log_error "pytest collection produced no S3 test node IDs" + return 1 + fi +} + +ALL_COLLECTED_NODEIDS="${ARTIFACTS_DIR}/all-collected-nodeids.txt" +UNSHARDED_SELECTED_NODEIDS="${ARTIFACTS_DIR}/unsharded-selected-nodeids.txt" +SELECTED_NODEIDS="${ARTIFACTS_DIR}/selected-nodeids.txt" +collect_nodeids "${ALL_COLLECTED_NODEIDS}" "${S3_TEST_FILE}" -m "not rustfs_never_marker" +python3 "${SCRIPT_DIR}/report_compat.py" \ + --lists-dir "${SCRIPT_DIR}" \ + --collected-nodeids "${ALL_COLLECTED_NODEIDS}" \ + --check-classifications-only || { + log_error "S3 test classifications do not match pinned revision ${S3TESTS_REV}" + exit 1 +} +if [[ "${TEST_SCOPE}" == "all" && -z "${TESTEXPR}" && "${MARKEXPR}" == "not rustfs_never_marker" ]]; then + cp "${ALL_COLLECTED_NODEIDS}" "${UNSHARDED_SELECTED_NODEIDS}" +else + collect_nodeids "${UNSHARDED_SELECTED_NODEIDS}" "${PYTEST_SELECTION_ARGS[@]}" -m "${MARKEXPR}" +fi + +if (( S3_SHARD_COUNT > 1 )); then + awk -v count="${S3_SHARD_COUNT}" -v shard_index="${S3_SHARD_INDEX}" \ + '((NR - 1) % count) == shard_index' \ + "${UNSHARDED_SELECTED_NODEIDS}" > "${SELECTED_NODEIDS}" + if [[ ! -s "${SELECTED_NODEIDS}" ]]; then + log_error "Shard ${S3_SHARD_INDEX}/${S3_SHARD_COUNT} selected no tests" + exit 1 + fi + PYTEST_SELECTION_ARGS=() + while IFS= read -r nodeid; do + PYTEST_SELECTION_ARGS+=("${nodeid}") + done < "${SELECTED_NODEIDS}" + log_info "Selected shard ${S3_SHARD_INDEX}/${S3_SHARD_COUNT}: ${#PYTEST_SELECTION_ARGS[@]} exact cases" +else + cp "${UNSHARDED_SELECTED_NODEIDS}" "${SELECTED_NODEIDS}" +fi + # Run tests from s3tests/functional +# Failure locals can contain multi-MiB request bodies; keep tracebacks without expanding local values. set +e S3TEST_CONF="${CONF_OUTPUT_PATH}" \ tox -- \ - -vv -ra --showlocals --tb=long \ + -vv -ra --tb=long \ --maxfail="${MAXFAIL}" \ + --timeout="${TEST_TIMEOUT}" \ --junitxml="${ARTIFACTS_DIR}/junit.xml" \ ${XDIST_ARGS} \ "${PYTEST_SELECTION_ARGS[@]}" \ @@ -1033,19 +1059,22 @@ elif [ "${DEPLOY_MODE}" = "existing" ]; then echo "{\"host\": \"${S3_HOST}\", \"port\": ${S3_PORT}, \"mode\": \"existing\"}" > "${ARTIFACTS_DIR}/rustfs-${TEST_MODE}/inspect.json" || true fi -# Step 11: Classification report (informational, never fails the run) +# Step 11: Classification report and gate REPORT_SCRIPT="${SCRIPT_DIR}/report_compat.py" -if [ -f "${REPORT_SCRIPT}" ] && [ -f "${ARTIFACTS_DIR}/junit.xml" ]; then - python3 "${REPORT_SCRIPT}" \ - --junit "${ARTIFACTS_DIR}/junit.xml" \ - --lists-dir "${SCRIPT_DIR}" \ - --output "${ARTIFACTS_DIR}/compat-report.md" \ - || log_warn "Compatibility report generation failed" -fi - -if [ ${TEST_EXIT_CODE} -ne 0 ]; then - summarize_junit_failures "${ARTIFACTS_DIR}/junit.xml" +REPORT_ARGS=( + --junit "${ARTIFACTS_DIR}/junit.xml" + --lists-dir "${SCRIPT_DIR}" + --collected-nodeids "${SELECTED_NODEIDS}" + --output "${ARTIFACTS_DIR}/compat-report.md" + --fail-on-regression +) +if [[ "${TEST_SCOPE}" == "all" ]]; then + REPORT_ARGS+=(--fail-on-unclassified) fi +set +e +python3 "${REPORT_SCRIPT}" "${REPORT_ARGS[@]}" +REPORT_EXIT_CODE=$? +set -e # Summary if [ ${TEST_EXIT_CODE} -eq 0 ]; then @@ -1059,4 +1088,10 @@ else log_info "Check RustFS logs: ${ARTIFACTS_DIR}/rustfs-${TEST_MODE}/rustfs.log" fi -exit ${TEST_EXIT_CODE} +if [[ "${TEST_EXIT_CODE}" -ne 0 && "${TEST_EXIT_CODE}" -ne 1 ]]; then + exit "${TEST_EXIT_CODE}" +fi +if [[ "${TEST_SCOPE}" == "implemented" && "${TEST_EXIT_CODE}" -ne 0 ]]; then + exit "${TEST_EXIT_CODE}" +fi +exit "${REPORT_EXIT_CODE}" diff --git a/scripts/s3-tests/test_report_compat.py b/scripts/s3-tests/test_report_compat.py new file mode 100644 index 000000000..acae360d0 --- /dev/null +++ b/scripts/s3-tests/test_report_compat.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""Regression tests for the S3 compatibility report.""" + +from __future__ import annotations + +import importlib.util +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +REPORT_PATH = Path(__file__).with_name("report_compat.py") +SPEC = importlib.util.spec_from_file_location("report_compat", REPORT_PATH) +assert SPEC and SPEC.loader +REPORT = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(REPORT) + + +class ReportCompatTests(unittest.TestCase): + def test_upstream_names_expose_incomplete_junit(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + directory = Path(tmp) + collected = directory / "collected.txt" + collected.write_text("s3tests/functional/test_s3.py::test_one[a]\ns3tests/functional/test_s3.py::test_one[b]\n") + junit = directory / "junit.xml" + junit.write_text('') + + expected = REPORT.load_collected_nodeids(collected) + results, _, _ = REPORT.parse_junit(junit) + + self.assertEqual(expected - results.keys(), {"test_one[b]"}) + + def test_cli_fails_an_incomplete_sweep(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + directory = Path(tmp) + collected = directory / "collected.txt" + collected.write_text("s3tests/functional/test_s3.py::test_one\ns3tests/functional/test_s3.py::test_two\n") + junit = directory / "junit.xml" + junit.write_text('') + for filename in REPORT.LIST_FILES.values(): + (directory / filename).write_text("") + (directory / "implemented_tests.txt").write_text("test_one\n") + + result = subprocess.run( + [ + sys.executable, + str(REPORT_PATH), + "--junit", + str(junit), + "--lists-dir", + str(directory), + "--collected-nodeids", + str(collected), + "--fail-on-regression", + "--fail-on-unclassified", + ], + check=False, + capture_output=True, + text=True, + ) + + self.assertEqual(result.returncode, 1) + self.assertIn("1 missing result(s)", result.stdout) + + def test_preflight_rejects_missing_and_stale_classifications(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + directory = Path(tmp) + collected = directory / "collected.txt" + collected.write_text("s3tests/functional/test_s3.py::test_known[a]\ntest_new\n") + for filename in REPORT.LIST_FILES.values(): + (directory / filename).write_text("") + (directory / "implemented_tests.txt").write_text("test_known\ntest_stale\ntest_stale\n") + + result = subprocess.run( + [ + sys.executable, + str(REPORT_PATH), + "--lists-dir", + str(directory), + "--collected-nodeids", + str(collected), + "--check-classifications-only", + ], + check=False, + capture_output=True, + text=True, + ) + + self.assertEqual(result.returncode, 1) + self.assertIn("[UNCLASSIFIED] test_new", result.stdout) + self.assertIn("[STALE] test_stale", result.stdout) + self.assertIn("[INVALID] implemented_tests.txt has duplicates: test_stale", result.stdout) + + def test_timeout_fails_even_when_test_is_excluded(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + directory = Path(tmp) + junit = directory / "junit.xml" + junit.write_text( + '' + ) + for filename in REPORT.LIST_FILES.values(): + (directory / filename).write_text("") + (directory / "excluded_tests.txt").write_text("test_slow\n") + + result = subprocess.run( + [sys.executable, str(REPORT_PATH), "--junit", str(junit), "--lists-dir", str(directory)], + check=False, + capture_output=True, + text=True, + ) + + self.assertEqual(result.returncode, 1) + self.assertIn("1 timeout(s)", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/s3-tests/unimplemented_tests.txt b/scripts/s3-tests/unimplemented_tests.txt index 5db63e3a7..cb7f3d644 100644 --- a/scripts/s3-tests/unimplemented_tests.txt +++ b/scripts/s3-tests/unimplemented_tests.txt @@ -11,8 +11,12 @@ # Failed tests test_bucket_create_delete_bucket_ownership +test_bucket_logging_request_id test_create_bucket_no_ownership_controls test_bucket_logging_owner +test_head_object_404_with_policy_prefix +test_multipart_reupload_checksum_and_etag +test_multipart_upload_complete_without_create test_object_copy_not_owned_bucket test_bucket_policy_multipart test_post_object_upload_checksum diff --git a/scripts/security/check_cache_save_if.sh b/scripts/security/check_cache_save_if.sh index 3e63bdc8a..27abc6545 100755 --- a/scripts/security/check_cache_save_if.sh +++ b/scripts/security/check_cache_save_if.sh @@ -64,3 +64,25 @@ if [ "$status" -ne 0 ]; then fi echo "OK: every ./.github/actions/setup call states cache-save-if explicitly" + +# rust-cache hashes every CARGO*, CC*, CFLAGS*, CXX*, CMAKE*, and RUST* +# variable that is present when the setup action runs. The dedicated writer +# and the CI readers therefore need identical workflow-level compiler env. +compiler_env() { + awk ' + /^env:[[:space:]]*$/ { in_env = 1; next } + in_env && /^[^[:space:]]/ { exit } + in_env && /^ (CARGO|CC|CFLAGS|CXX|CMAKE|RUST)[A-Z0-9_]*:/ { print } + ' "$1" | sort +} + +ci_env="$(compiler_env .github/workflows/ci.yml)" +warm_env="$(compiler_env .github/workflows/cache-warm.yml)" + +if [ "$ci_env" != "$warm_env" ]; then + echo "CI and cache-warm compiler environments differ; rust-cache keys will not match:" >&2 + diff -u <(printf '%s\n' "$ci_env") <(printf '%s\n' "$warm_env") >&2 || true + exit 1 +fi + +echo "OK: cache-warm and CI compiler environments match" diff --git a/scripts/security/check_performance_ab_workflow.sh b/scripts/security/check_performance_ab_workflow.sh index 5465e4fdf..09b97cc0d 100755 --- a/scripts/security/check_performance_ab_workflow.sh +++ b/scripts/security/check_performance_ab_workflow.sh @@ -13,8 +13,29 @@ require_absent_pattern() { fi } +require_present_pattern() { + local pattern="$1" + local description="$2" + + if ! grep -Eq -- "$pattern" "$workflow"; then + echo "invalid performance A/B workflow contract: $description" >&2 + exit 1 + fi +} + require_absent_pattern '(^|[^[:alnum:]_])pull_request(_target)?([^[:alnum:]_]|$)' "the workflow must not contain PR event handling" require_absent_pattern 'pull-requests[[:space:]]*:[[:space:]]*write' "the workflow must not receive PR write permission" require_absent_pattern 'permissions[[:space:]]*:[[:space:]]*write-all' "the workflow must not receive broad write permission" +require_absent_pattern '^[[:space:]]*push:' "the workflow must not spend a release build on every main push" +require_present_pattern 'listWorkflowRuns' "the scheduled baseline must come from workflow history" +require_present_pattern 'status:[[:space:]]*"success"' "the scheduled baseline must be a successful run" +require_present_pattern 'SCHEDULED_BASELINE_SHA' "the resolved scheduled baseline must reach the comparison" +require_present_pattern "SCHEDULED_BASELINE_SHA:-\\\$candidate_sha" "the first scheduled run must seed from its verified candidate" +require_present_pattern 'git merge-base --is-ancestor' "the scheduled baseline must stay on candidate history" +require_present_pattern 'Cache successful candidate baseline' "a successful candidate must become the next cached baseline" +if ! sed -n '/^ warp-ab:/,/^ alert-on-failure:/p' "$workflow" | grep -Eq '^ timeout-minutes:[[:space:]]*180([[:space:]]|$)'; then + echo "invalid performance A/B workflow contract: the cold-cache path must fit both builds, the A/B run, and evidence publication" >&2 + exit 1 +fi -echo "Performance A/B workflow trust boundary ok." +echo "Performance A/B workflow contract ok."