mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-28 23:07:39 +00:00
Compare commits
24 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9601a7c4bc | |||
| d16e3cf7b6 | |||
| 3e056ffb4d | |||
| 72b4f6182f | |||
| a9d3022f37 | |||
| c232f1552d | |||
| 7581a70412 | |||
| 8cb9296f20 | |||
| b1ba6cdfdc | |||
| 0bfc7612d4 | |||
| 4c997519e8 | |||
| 95c5d108d1 | |||
| f6ad79dd58 | |||
| 82947d3595 | |||
| aef67b6499 | |||
| 8e88419f3a | |||
| 0197a4b538 | |||
| 7a3bf8cdbd | |||
| 3d3957a248 | |||
| b4026b43b2 | |||
| 851c295753 | |||
| f879946ead | |||
| d88f524d9f | |||
| 54ac37ec0a |
@@ -27,7 +27,4 @@ Risk and review shape:
|
||||
and list verified and unverified target classes in the PR Impact section.
|
||||
|
||||
Available domain lenses are security, concurrency/durability, compatibility,
|
||||
and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
|
||||
explicit adversarial request, a high-risk change, or a substantial PR review;
|
||||
then read only its matching role references. A routine standard pass does not
|
||||
load the playbook unless the reviewer needs a RustFS-specific probe.
|
||||
and performance; select only those matched by the change.
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
name: adversarial-validation
|
||||
description: Review RustFS diffs or designs for explicit adversarial requests, high-risk changes under the repository review policy, or substantial PR reviews. Skip ordinary questions, diagnosis, planning, status, routine low-risk implementation, and prose with no execution effect.
|
||||
---
|
||||
|
||||
# RustFS Adversarial Validation
|
||||
|
||||
Use the [repository risk tiers and review shape](../../references/adversarial-validation.md). This skill
|
||||
routes a review to RustFS-specific probes without loading unrelated domains.
|
||||
|
||||
## Select Lenses
|
||||
|
||||
Read only the references required by the diff:
|
||||
|
||||
| Lens | When to read |
|
||||
|---|---|
|
||||
| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
|
||||
| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
|
||||
| [Test coverage](references/test-coverage.md) | Behavior or test changes |
|
||||
| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
|
||||
| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
|
||||
| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
|
||||
| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
|
||||
|
||||
Do not read all references as a precaution. A path name alone is insufficient;
|
||||
the changed behavior must touch the lens's domain.
|
||||
|
||||
For a dedicated security audit or advisory analysis, use
|
||||
`security-advisory-lessons` instead of loading it automatically during every
|
||||
adversarial review.
|
||||
|
||||
## Review Protocol
|
||||
|
||||
1. Freeze the exact final diff/head (or the design under review) and list the
|
||||
selected lenses.
|
||||
2. Run the review shape required by the repository risk tier.
|
||||
3. For each selected lens, either report a concrete finding or a null verdict
|
||||
naming the attacks performed.
|
||||
4. Apply root `AGENTS.md`'s finding standard. Test each candidate against callers,
|
||||
existing coverage, and invariants before accepting it; an adversarial role
|
||||
does not have to produce a defect.
|
||||
5. Fix or rebut supported findings with code-path, test, or invariant evidence.
|
||||
6. After a non-trivial edit, rerun only lenses affected by that edit against the
|
||||
new exact diff.
|
||||
|
||||
Do not turn a null verdict into a long checklist. Record concise evidence that
|
||||
the relevant failure classes were attacked, then stop under the root completion
|
||||
rule. Keep the required per-lens verdicts for high-risk PRs.
|
||||
@@ -1,38 +0,0 @@
|
||||
# Compatibility Lens
|
||||
|
||||
- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
|
||||
including mixed casing and removal of both twins.
|
||||
- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
|
||||
remote tiers receive no `versionId`; versioned purge requests retain the real
|
||||
version ID.
|
||||
- `xl.meta` changes preserve supported header/meta versions, recompute
|
||||
signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
|
||||
- Foreign/corrupt metadata validates parallel array lengths and missing fields;
|
||||
it returns a decode error rather than indexing, panicking, or fabricating data.
|
||||
- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
|
||||
Bitrot framing, shard math, distribution, and inline prefixes are contracts.
|
||||
- Client-visible metadata/events strip both internal prefixes
|
||||
case-insensitively.
|
||||
- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
|
||||
compatibly and absent new fields fail closed where authorization/quorum is
|
||||
involved.
|
||||
- Replay real client request shapes and exact pagination boundaries for S3
|
||||
handler changes.
|
||||
- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
|
||||
fixtures and encrypted migration data.
|
||||
- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
|
||||
condition, and default toward reading old data safely.
|
||||
|
||||
## Outbound targets
|
||||
|
||||
- A change to what the replication or migration client sends by default
|
||||
(checksum policy, payload framing, headers, version-id addressing) is judged
|
||||
against every target class, not the one it fixes. Name each target-side rule
|
||||
the current default satisfies — checksum required with Object Lock
|
||||
parameters, `aws-chunked` decoding, version-id adoption, ETag equals content
|
||||
MD5 — and show which cell of
|
||||
`crates/e2e_test/src/replication_target_matrix_test.rs` covers each.
|
||||
- A test that asserts the fix ("no trailer header") is not evidence; the
|
||||
matrix cell that asserts the target accepted and stored the object is.
|
||||
- Every new environment escape hatch appears in
|
||||
`docs/operations/replication-outbound-transport.md` in the same diff.
|
||||
@@ -1,27 +0,0 @@
|
||||
# Concurrency and Durability Lens
|
||||
|
||||
- For every changed lock, enumerate overlapping lock sets and construct the
|
||||
ABBA interleaving. Multiple-lock order must be documented and consistent.
|
||||
- Mark guard lifetimes and every `.await`, disk,
|
||||
and RPC call; estimate contention/timeouts. Tokio read/write guards across
|
||||
`.await` need bounded hold time; long-lived reads can wedge writers (#4195).
|
||||
Keep `std::sync::Mutex` holds brief and never across `.await`.
|
||||
- Use direct `fetch_*` for unconditional atomic read-modify-write and
|
||||
`compare_exchange` for conditional updates.
|
||||
- Object commits remain fenced if the distributed lock is lost after shard
|
||||
writes and before metadata rename.
|
||||
- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
|
||||
-> sync required ancestors`; simulate a crash after each step and honor the
|
||||
configured durability gate.
|
||||
- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
|
||||
heal remains best-effort per target where that is the established contract.
|
||||
- At every new cancellable await between mutation and cleanup/commit, drop the
|
||||
future and inspect leftover files, counters, permits, and replay state.
|
||||
- Multipart operations on the same upload ID are serialized where required;
|
||||
abort/complete/list races cannot delete parts before durable commit.
|
||||
- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
|
||||
committed write or delete the last surviving copy.
|
||||
- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
|
||||
and duplicate delivery has an idempotency contract.
|
||||
- Streaming reconstruction failures after partial output surface as errors, not
|
||||
successful EOF.
|
||||
@@ -1,31 +0,0 @@
|
||||
# Correctness Lens
|
||||
|
||||
Attack the changed behavior, not every subsystem in the repository.
|
||||
|
||||
- Trace new error paths to the caller. Inject the ignored/wildcard variants and
|
||||
verify they cannot become success, not-found, or a plausible default.
|
||||
- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
|
||||
changed count, size, index, page limit, or optional value.
|
||||
- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
|
||||
mixed disk errors and nil/placeholder entries.
|
||||
- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
|
||||
continuation markers, and object/prefix name collisions.
|
||||
- For EC/read/streaming changes, inject failure after partial output and verify
|
||||
the client receives an error rather than a clean truncated body. Assert exact
|
||||
bytes and length.
|
||||
- For multipart/object commits, fail before/after rename and cleanup; committed
|
||||
data must remain readable and pre-commit cleanup must not destroy parts.
|
||||
- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
|
||||
tree and cache traversals safely.
|
||||
- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
|
||||
versions, and deterministic tie-breaking.
|
||||
- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
|
||||
below the layer that sees trailing slashes are dead.
|
||||
- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
|
||||
nil/empty `versionId` to an unversioned tier.
|
||||
- For agent rules/skill routers, test a trigger matrix covering ordinary
|
||||
inquiry, low-risk implementation, explicit review, high-risk code, PR
|
||||
creation, release, and post-PR monitoring. Each case must select only the
|
||||
intended workflow and retain required safety/authorization boundaries.
|
||||
|
||||
Null verdicts name only the probes relevant to the diff.
|
||||
@@ -1,34 +0,0 @@
|
||||
# Performance Lens
|
||||
|
||||
- For added clones/allocations on request/object/block paths, quantify copied
|
||||
data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
|
||||
capacity reservation only for a concrete repeated cost.
|
||||
- Route every new sync/flush through the durability-mode and bucket override
|
||||
gates; mode `none` must not pay the new fsync.
|
||||
- Keep blocking filesystem/CPU work off async runtime threads, but do not split
|
||||
one small operation into many `spawn_blocking` round trips; coalesce `open`,
|
||||
`read`/`write`, `fsync`, and `close` into one blocking closure per disk. When
|
||||
coalescing steps across execution lanes, route the combined closure to the
|
||||
dedicated fsync lane whenever any step performs file or directory sync so
|
||||
stalled syncs cannot starve foreground metadata reads.
|
||||
- Check for per-disk syscall amplification (`disks × parts/files`): pin a
|
||||
directory once (`openat2` on Linux) and reuse the handle across sibling reads
|
||||
or same-directory renames instead of re-walking ancestor components per file
|
||||
or re-opening the same parent directory twice.
|
||||
- When the file size is already known from `fstat` metadata, pre-size the
|
||||
destination buffer (`len + 1`) before reading to EOF so small metadata reads
|
||||
complete in a single `read()` + EOF check rather than exponential-growth
|
||||
reallocations and syscalls.
|
||||
- Avoid redundant disk re-reads or existence checks under the same exclusive
|
||||
lock when the state or raw bytes were just loaded or persisted.
|
||||
- Measure lock hold time across I/O and compare acquisition order for ABBA.
|
||||
- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
|
||||
section when they need not be there.
|
||||
- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
|
||||
use a gate or bounded concurrency when the concrete fan-out warrants it.
|
||||
- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
|
||||
balance and avoid repeated metadata decode/fetch per object.
|
||||
- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
|
||||
require an existing gate.
|
||||
- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
|
||||
A/B evidence, especially for 4 KiB objects.
|
||||
@@ -1,31 +0,0 @@
|
||||
# Security Lens
|
||||
|
||||
Use `security-advisory-lessons` only for a dedicated advisory/security audit.
|
||||
For an ordinary matched diff, attack these boundaries:
|
||||
|
||||
- Admin routes: route registration, whitelist, handler authn, and the exact
|
||||
`AdminAction` must agree. Read-only diagnostics still require admin authz.
|
||||
- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
|
||||
names as attacker-controlled; prove ownership/root authority before writes.
|
||||
- Protocol frontends: every changed/sibling command authorizes the matching S3
|
||||
action before reaching storage.
|
||||
- Secrets/signatures: use constant-time comparison, normalize public failures,
|
||||
keep RPC/root/STS keys independent, and fail closed when secrets are absent.
|
||||
- RPC: bind signatures to the exact method/path and timestamp; reject replay,
|
||||
stale, malformed, truncated, and invalid-enum payloads without panic.
|
||||
- Paths/object/archive entries: reject traversal, absolute/platform escapes,
|
||||
and normalization differences between authz and storage.
|
||||
- Copy/multipart/presigned POST: enforce source, destination, version-aware
|
||||
actions, copy-source conditions, and every signed policy condition.
|
||||
- Logging/errors: never expose credentials, tokens, expected signatures, raw
|
||||
secret-bearing input, or merged configs—including via `Debug` and parse errors.
|
||||
- Untrusted serde: reject unknown fields where compatible and validate
|
||||
security-critical defaults/ranges before numeric conversion.
|
||||
- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
|
||||
isolate user content; never reflect credentialed arbitrary origins or trust
|
||||
forwarded identity from direct clients.
|
||||
- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
|
||||
foreground, lifecycle, scanner, and force-delete paths.
|
||||
|
||||
Security findings distinguish unauthenticated compromise from a
|
||||
low-privileged authenticated bypass.
|
||||
@@ -1,22 +0,0 @@
|
||||
# Simplicity Lens
|
||||
|
||||
- Compare the production diff with the smallest equivalent local edit. Fewer
|
||||
lines alone are not evidence; the replacement must preserve correctness,
|
||||
compatibility, readability, and real boundaries.
|
||||
- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
|
||||
relevant dependencies for each new helper, constant, wrapper, or fixture.
|
||||
- Reject forced reuse when normalization, error, deadline, or durability
|
||||
semantics differ.
|
||||
- Require a concrete trigger for every new defensive branch. Keep boundary
|
||||
checks for disk/RPC/version data and checks immediately before destructive
|
||||
actions.
|
||||
- Flag one-caller helpers only when they merely forward or split a short linear
|
||||
flow without adding domain naming, invariant isolation, or useful context.
|
||||
- Ensure a replacement removes the superseded in-scope path or keeps one
|
||||
canonical core behind a documented compatibility adapter.
|
||||
- Remove narration/change-history comments; preserve concise safety, lock,
|
||||
durability, and compatibility invariants.
|
||||
- Treat tests, fixtures, generated code, and documentation separately from
|
||||
production growth. Do not optimize away meaningful regression coverage.
|
||||
|
||||
A finding must include a concrete smaller design, not a style preference.
|
||||
@@ -1,26 +0,0 @@
|
||||
# Test-Coverage Lens
|
||||
|
||||
- For every behavior claim, name the focused test/check that fails if the
|
||||
changed hunk is reverted. If none is practical, require the reason and
|
||||
residual risk.
|
||||
- Confirm tests exercise the real production path and distinguish the intended
|
||||
behavior from the named regression. A success, `is_err()`, or no-panic check
|
||||
can be sufficient when that is the actual contract; require exact values,
|
||||
bytes, state, or error variants when those distinctions matter to the change.
|
||||
- For new flags/modes, verify each branch and ask which test fails if the branch
|
||||
is inverted.
|
||||
- For new error propagation, inject the failure and assert the caller observes
|
||||
it; mentally replacing `?`/`return Err` with success must break a test.
|
||||
- Streaming GET tests assert the complete body and length under degraded reads.
|
||||
- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
|
||||
round-trips are insufficient for compatibility.
|
||||
- Concurrency tests use readiness polling, isolate global state, and avoid fixed
|
||||
sleeps or unrealistically short timeouts. Use nextest groups when process-level
|
||||
serialization is required.
|
||||
- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
|
||||
through a helper that prefers one key.
|
||||
- Avoid duplicate tests of the same production path and poison-value class.
|
||||
Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
|
||||
absent vs empty vs nil UUID.
|
||||
- A focused test proves only the targets/features it builds. Add compilation or
|
||||
Clippy only for uncovered changed targets.
|
||||
@@ -13,7 +13,7 @@ description: Review a GitHub PR using its exact base/head and repository risk po
|
||||
```
|
||||
|
||||
Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
|
||||
2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
|
||||
2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
|
||||
3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
|
||||
4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
|
||||
5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
name: security-advisory-lessons
|
||||
description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
|
||||
---
|
||||
|
||||
# RustFS Security Advisory Lessons
|
||||
|
||||
Use this skill as the deep security lens. For a normal adversarial review with a
|
||||
matched security surface, the concise security reference under
|
||||
`adversarial-validation` is sufficient.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Freeze the exact diff/head and identify the changed trust boundaries.
|
||||
2. Inspect the headings in [advisory-patterns.md](references/advisory-patterns.md),
|
||||
then read the matching sections. Read the full map only for a broad security
|
||||
audit. Useful headings are
|
||||
auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
|
||||
secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
|
||||
3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
|
||||
and default-config cases. Security decisions must fail closed.
|
||||
4. Require a focused negative regression test for the bypass/exploit form, not
|
||||
only the intended success path. State residual risk when a test is impractical.
|
||||
5. Report proven vulnerabilities separately from defense-in-depth hardening.
|
||||
|
||||
When advisory currentness matters, fetch the live inventory instead of treating
|
||||
the reference as a status mirror:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
||||
```
|
||||
|
||||
Fetch an individual advisory only when the live summary indicates a new or
|
||||
changed lesson.
|
||||
|
||||
## Finding Standard
|
||||
|
||||
Every candidate names the lower-trust principal, the boundary it crosses, the
|
||||
affected resource, and the observable result. A best-practice gap or an impact
|
||||
confined to the caller's own principal is hardening, not a candidate.
|
||||
|
||||
Report each candidate under exactly one verdict:
|
||||
|
||||
- `confirmed`: a complete `file:line` source trace and a bounded observed
|
||||
result. Only confirmed findings carry severity; each also states attacker
|
||||
prerequisites, concrete input/path, impact, smallest safe fix, and a
|
||||
regression check.
|
||||
- `needs_validation`: the trace is blocked on one exact unresolved fact, such
|
||||
as deployment or infrastructure behavior. State that fact; assign no severity
|
||||
and do not substitute an assumption.
|
||||
- `rejected`: a disproved candidate, with the caller, invariant, or test that
|
||||
disproves it.
|
||||
|
||||
Do not exaggerate unauthenticated impact when the actual issue requires
|
||||
authenticated low privilege.
|
||||
@@ -1,158 +0,0 @@
|
||||
# RustFS Advisory Pattern Map
|
||||
|
||||
This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
|
||||
|
||||
When current advisory state, severity, URLs, or full text matters, fetch it live:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
||||
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
|
||||
```
|
||||
|
||||
Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
|
||||
|
||||
## Pattern Index
|
||||
|
||||
### Admin authorization and route exposure
|
||||
|
||||
- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
|
||||
- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
|
||||
- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
|
||||
- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
|
||||
- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
|
||||
- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
|
||||
- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
|
||||
|
||||
### IAM import, service accounts, and privilege boundaries
|
||||
|
||||
- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
|
||||
- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
|
||||
- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
|
||||
- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
|
||||
- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
|
||||
|
||||
### STS, OIDC, and federation flows
|
||||
|
||||
- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
|
||||
- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
|
||||
- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
|
||||
- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
|
||||
- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
|
||||
|
||||
### IAM policy conditions and external policy plugins
|
||||
|
||||
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
|
||||
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
|
||||
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
|
||||
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
|
||||
|
||||
### S3 object actions, copy, multipart, and upload policy validation
|
||||
|
||||
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
|
||||
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
|
||||
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
|
||||
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
|
||||
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
|
||||
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
|
||||
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
|
||||
- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
|
||||
|
||||
### Protocol frontends and IAM parity
|
||||
|
||||
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
|
||||
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
|
||||
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
|
||||
- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
|
||||
|
||||
### Filesystem paths and object key traversal
|
||||
|
||||
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
|
||||
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
|
||||
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
|
||||
- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
|
||||
|
||||
### Secrets, defaults, and cryptographic misuse
|
||||
|
||||
- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
|
||||
- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
|
||||
- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
|
||||
- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
|
||||
- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
|
||||
- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
|
||||
|
||||
### Sensitive logging and debug output
|
||||
|
||||
- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
|
||||
- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
|
||||
- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
|
||||
|
||||
### RPC input validation and panic safety
|
||||
|
||||
- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
|
||||
- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
|
||||
- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
|
||||
|
||||
### Browser, CORS, and console isolation
|
||||
|
||||
- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
|
||||
- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
|
||||
- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
|
||||
|
||||
### Trusted proxy and source IP conditions
|
||||
|
||||
- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
|
||||
|
||||
### SSE and on-disk storage invariants
|
||||
|
||||
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
|
||||
- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
|
||||
|
||||
### Object Lock and retention invariants
|
||||
|
||||
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
|
||||
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
|
||||
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
|
||||
|
||||
### Serde deserialization and input validation
|
||||
|
||||
- Reject unknown fields in untrusted S3 API XML/JSON, lifecycle, policy, and replication input where compatibility permits. Check the current type and supported payload fixtures; do not infer the repository's current coverage from an older audit. Where extra fields are part of the compatibility contract, validate security-critical values explicitly.
|
||||
- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
|
||||
- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` with a typed error, or clamp only when the domain explicitly requires saturation.
|
||||
- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
|
||||
|
||||
## Useful Search Seeds
|
||||
|
||||
Use these targeted searches when a diff touches security-sensitive code:
|
||||
|
||||
```bash
|
||||
rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
|
||||
rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
|
||||
rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|presign|SignedHeaders|x-amz-copy-source|collect_signed_headers|content-length-range|starts-with" rustfs crates
|
||||
rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
|
||||
rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
|
||||
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
|
||||
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
|
||||
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
|
||||
rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
|
||||
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
|
||||
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
|
||||
```
|
||||
|
||||
## Minimum Regression Test Expectations
|
||||
|
||||
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
|
||||
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
|
||||
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
|
||||
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
|
||||
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
|
||||
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
|
||||
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
|
||||
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
|
||||
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
|
||||
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
|
||||
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
|
||||
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
|
||||
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
|
||||
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
|
||||
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
|
||||
@@ -1,2 +1,2 @@
|
||||
sha256-darwin=a67a347f26be186c598b5beaf68e6bcbe1855350a35e1423dd5b8a81bcba06a5
|
||||
sha256-linux=ad4fabe811c6cca1aba89fad34ccd404b5832a6a955875c2826f6e0fa1ea4f1f
|
||||
sha256-darwin=67251cb7d9978d38e899268cac4d7898297ff7e5b5c48c9c1546dc760b93568d
|
||||
sha256-linux=673da76c2e5710c4622fe2fa32d2b297730895769487fcf1152ee1c2366f32f4
|
||||
|
||||
@@ -1 +1 @@
|
||||
sha256=f065e7071aac9d48a7c1b301f328fa7cc14f11c4e7dc46da3e55057c4ea29bd5
|
||||
sha256=e9dc896774122a304e2b8b90f28c82d4d4e0af8ded36a82b147b30945ec994e3
|
||||
|
||||
@@ -128,7 +128,7 @@ setup = 'ecstore-large-stack'
|
||||
# path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need
|
||||
# 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI.
|
||||
[[profile.default.scripts]]
|
||||
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
|
||||
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
|
||||
setup = 'ecstore-base-stack'
|
||||
|
||||
[[profile.default.scripts]]
|
||||
@@ -419,7 +419,7 @@ setup = 'ecstore-large-stack'
|
||||
|
||||
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
|
||||
[[profile.ci.scripts]]
|
||||
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
|
||||
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
|
||||
setup = 'ecstore-base-stack'
|
||||
|
||||
[[profile.ci.scripts]]
|
||||
|
||||
@@ -16,7 +16,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|
||||
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
|
||||
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
|
||||
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
|
||||
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
|
||||
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
|
||||
| `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
|
||||
|
||||
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools.
|
||||
@@ -27,11 +27,15 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
|
||||
|
||||
1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB.
|
||||
2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes).
|
||||
3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines.
|
||||
3. **Hardware or environment.** Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
|
||||
4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below).
|
||||
|
||||
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
|
||||
|
||||
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection stays on the operator-managed `smoke-testing` runner as part of the automation environment. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
|
||||
|
||||
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
|
||||
|
||||
A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost.
|
||||
|
||||
## Cache rule
|
||||
|
||||
@@ -337,7 +337,7 @@ jobs:
|
||||
name: Build Docker Images
|
||||
needs: build-check
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: dind-sm-standard-2
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
|
||||
@@ -44,7 +44,7 @@ jobs:
|
||||
check-ephemerality:
|
||||
if: github.repository == 'rustfs/rustfs'
|
||||
name: Check runner ephemerality
|
||||
runs-on: sm-standard-2
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
||||
@@ -42,7 +42,7 @@ concurrency:
|
||||
jobs:
|
||||
prepare:
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
|
||||
runs-on: smoke-testing
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
manifest: ${{ steps.candidate.outputs.manifest }}
|
||||
@@ -188,7 +188,7 @@ jobs:
|
||||
complete-chain:
|
||||
needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
|
||||
if: ${{ always() && needs.prepare.result != 'skipped' }}
|
||||
runs-on: smoke-testing
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
|
||||
@@ -90,7 +90,7 @@ env:
|
||||
jobs:
|
||||
runner-check:
|
||||
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
|
||||
runs-on: sm-standard-2
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
|
||||
@@ -158,9 +158,7 @@ requested adversarial/design reviews, and agent-instruction changes that alter
|
||||
execution. Ordinary questions, diagnoses, status reports, non-adversarial code
|
||||
reviews, and low-risk planning do not trigger it.
|
||||
|
||||
For applicable work and substantial PR reviews, read the [risk tiers and review shape](.agents/references/adversarial-validation.md).
|
||||
Load only the matching domain probes; ordinary reviews do not become adversarial
|
||||
merely because this reference exists.
|
||||
For applicable work and substantial PR reviews, follow the [risk tiers and review shape](.agents/references/adversarial-validation.md) and check changed trust boundaries and relevant correctness, security, concurrency, durability, compatibility, performance, and negative regression coverage.
|
||||
|
||||
A review has no finding quota; `No findings` is a complete outcome. A request to
|
||||
find problems is not evidence that a defect exists. Before reporting a candidate,
|
||||
|
||||
@@ -26,7 +26,7 @@ rustfs-io-core = { workspace = true }
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
|
||||
# Async runtime
|
||||
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
|
||||
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time"] }
|
||||
|
||||
# Logging
|
||||
tracing = { workspace = true }
|
||||
|
||||
@@ -100,6 +100,7 @@ when restoring `/rustfs/console` for a custom OEM build.
|
||||
## S3 API environment variables
|
||||
|
||||
- `RUSTFS_API_OBJECT_MAX_VERSIONS` caps the number of retained versions for a single object. It defaults to `9223372036854775807`, matching MinIO's practical-unlimited default. Set a positive integer to enforce a lower per-object metadata bound.
|
||||
- `RUSTFS_VEEAM_SOSAPI_ENABLE` serves Veeam SOSAPI `system.xml` and `capacity.xml` for an authenticated GET or HEAD of those exact bucket-root keys. It defaults to `true`. Explicit version reads use the normal object path. Capacity requests return `ServiceUnavailable` until the scanner publishes a complete snapshot certified for the current bucket incarnation and authoritative quota metadata can be read. Legacy or mixed-version snapshots without an incarnation proof cannot serve capacity until a compatible scanner publishes a certified snapshot. Capacity also returns `ServiceUnavailable` when cluster or disk observations are missing, stale, or degraded. Retry after scanner convergence and storage or metadata recovery. Capacity assumes shard paths have independent backing space; shared physical storage can make the estimate optimistic. Set the flag to `false` to restore the normal object path, including deletion of a real object previously stored at that key.
|
||||
- `MINIO_API_OBJECT_MAX_VERSIONS` is accepted as a compatibility alias when the canonical RustFS variable is not set.
|
||||
|
||||
## Distributed endpoint locality
|
||||
|
||||
@@ -222,6 +222,16 @@ pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
|
||||
/// Environment variable for server KMS enable.
|
||||
pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
|
||||
|
||||
/// Environment variable for Veeam SOSAPI virtual system objects.
|
||||
///
|
||||
/// When true (the default), authenticated GET and HEAD of the two Veeam
|
||||
/// SOSAPI keys return generated XML. When false, those keys use the normal
|
||||
/// object path so an operator can remove a real object stored there earlier.
|
||||
pub const ENV_VEEAM_SOSAPI_ENABLE: &str = "RUSTFS_VEEAM_SOSAPI_ENABLE";
|
||||
|
||||
/// Default Veeam SOSAPI enablement.
|
||||
pub const DEFAULT_VEEAM_SOSAPI_ENABLE: bool = true;
|
||||
|
||||
/// Default KMS enable for server-side encryption
|
||||
/// This is the default value for enabling KMS encryption for server-side encryption.
|
||||
/// Default value: false
|
||||
|
||||
@@ -466,6 +466,9 @@ pub struct BucketTargetUsageInfo {
|
||||
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct BucketUsageInfo {
|
||||
pub size: u64,
|
||||
/// Bucket identity certified by every set contributing to this snapshot.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub bucket_incarnation: Option<String>,
|
||||
// Following five fields suffixed with V1 are here for backward compatibility
|
||||
// Total Size for objects that have not yet been replicated
|
||||
pub replication_pending_size_v1: u64,
|
||||
@@ -2316,6 +2319,9 @@ impl BucketUsageInfo {
|
||||
/// Add size summary to this bucket usage
|
||||
/// Merge another BucketUsageInfo into this one
|
||||
pub fn merge(&mut self, other: &BucketUsageInfo) {
|
||||
if self.bucket_incarnation != other.bucket_incarnation {
|
||||
self.bucket_incarnation = None;
|
||||
}
|
||||
self.size += other.size;
|
||||
self.objects_count += other.objects_count;
|
||||
self.versions_count += other.versions_count;
|
||||
@@ -3299,6 +3305,31 @@ mod tests {
|
||||
assert_eq!(usage1.versions_count, 15);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_usage_merge_discards_incompatible_incarnation_proofs() {
|
||||
let id = "00000000-0000-4000-8000-000000000001";
|
||||
let mut same = BucketUsageInfo {
|
||||
bucket_incarnation: Some(id.into()),
|
||||
..Default::default()
|
||||
};
|
||||
same.merge(&BucketUsageInfo {
|
||||
bucket_incarnation: Some(id.into()),
|
||||
..Default::default()
|
||||
});
|
||||
assert_eq!(same.bucket_incarnation.as_deref(), Some(id));
|
||||
same.merge(&BucketUsageInfo::default());
|
||||
assert!(same.bucket_incarnation.is_none());
|
||||
let mut different = BucketUsageInfo {
|
||||
bucket_incarnation: Some(id.into()),
|
||||
..Default::default()
|
||||
};
|
||||
different.merge(&BucketUsageInfo {
|
||||
bucket_incarnation: Some("00000000-0000-4000-8000-000000000002".into()),
|
||||
..Default::default()
|
||||
});
|
||||
assert!(different.bucket_incarnation.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn size_summary_add_saturates_instead_of_overflowing() {
|
||||
// The scanner folds one summary per object into a per-prefix total, so a
|
||||
|
||||
@@ -1129,9 +1129,198 @@ async fn phase_quorum_matrix(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn list_parts_disk_loss_after_metadata_read(
|
||||
cluster: &RustFSTestClusterEnvironment,
|
||||
proxies: &[RpcFaultProxy],
|
||||
clients: &[Client],
|
||||
bucket: &str,
|
||||
) -> TestResult {
|
||||
let key = "list-parts-after-metadata-loss";
|
||||
let body = payload_for("list-parts-acknowledged", 4096);
|
||||
let upload = clients[0].create_multipart_upload().bucket(bucket).key(key).send().await?;
|
||||
let upload_id = upload.upload_id().ok_or("missing upload ID")?.to_owned();
|
||||
let part = clients[0]
|
||||
.upload_part()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.upload_id(&upload_id)
|
||||
.part_number(1)
|
||||
.body(ByteStream::from(body.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let etag = part.e_tag().ok_or("missing part ETag")?.to_owned();
|
||||
let path = upload_path_scope(bucket, key, &upload_id)?;
|
||||
let root = Path::new(&cluster.nodes[0].data_dirs[0])
|
||||
.join(RUSTFS_META_MULTIPART_BUCKET)
|
||||
.join(&path);
|
||||
let data_dirs = std::fs::read_dir(&root)?
|
||||
.filter_map(Result::ok)
|
||||
.map(|entry| entry.path())
|
||||
.filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(data_dirs.len(), 1, "the coordinator's actual part directory must be identified");
|
||||
let data_dir = &data_dirs[0];
|
||||
let dir_name = data_dir
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.ok_or("non-UTF8 data directory")?;
|
||||
let scoped_path = format!("{path}/{dir_name}");
|
||||
let held_dir = data_dir.with_extension("e2e-held");
|
||||
std::fs::rename(data_dir, &held_dir)?;
|
||||
let pauses = proxies[1..]
|
||||
.iter()
|
||||
.map(|proxy| proxy.pause(&["ListDir"], usize::MAX, Some(&scoped_path)))
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
let request = clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id);
|
||||
let task = tokio::spawn(async move { request.send().await });
|
||||
// ListDir is dispatched only after the upload metadata read succeeded.
|
||||
// Require all three scoped remote requests; unrelated failures do not count.
|
||||
wait_for_faults(&pauses, &["ListDir"]).await?;
|
||||
let result = tokio::time::timeout(DEADLINE, task).await??;
|
||||
for pause in &pauses {
|
||||
pause.discard_pending();
|
||||
}
|
||||
drop(pauses);
|
||||
std::fs::rename(&held_dir, data_dir)?;
|
||||
let error = result.expect_err("one missing directory and three unavailable participants cannot prove an empty upload");
|
||||
let response = error.raw_response().expect("ListParts must return an S3 service response");
|
||||
assert_eq!(response.status().as_u16(), 503);
|
||||
assert_eq!(error.as_service_error().and_then(|err| err.meta().code()), Some("SlowDownRead"));
|
||||
assert!(response.headers().get("x-rustfs-error-cause").is_none());
|
||||
assert!(response.headers().get("retry-after").is_none());
|
||||
assert!(
|
||||
response
|
||||
.headers()
|
||||
.get("content-type")
|
||||
.is_some_and(|value| value.starts_with("application/xml"))
|
||||
);
|
||||
assert!(
|
||||
response
|
||||
.headers()
|
||||
.get("x-amz-request-id")
|
||||
.is_some_and(|value| !value.is_empty())
|
||||
);
|
||||
let listed = recover_after_fault("ListParts after directory restoration", || {
|
||||
clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id).send()
|
||||
})
|
||||
.await??;
|
||||
assert_eq!(listed.parts().len(), 1);
|
||||
assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
|
||||
clients[0]
|
||||
.complete_multipart_upload()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.upload_id(&upload_id)
|
||||
.multipart_upload(
|
||||
CompletedMultipartUpload::builder()
|
||||
.parts(CompletedPart::builder().part_number(1).e_tag(&etag).build())
|
||||
.build(),
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
for client in clients {
|
||||
assert_object_bytes_after_fault(client, bucket, key, &body).await?;
|
||||
}
|
||||
eprintln!(
|
||||
"MPU list proof: metadata read completed, three scoped ListDir RPCs held, missing local directory rejected; restored part and all readers verified"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn concurrent_capped_parts_respect_limit_on_distinct_nodes(clients: &[Client], bucket: &str) -> TestResult {
|
||||
let key = "concurrent-capped-parts";
|
||||
let upload = clients[0]
|
||||
.create_multipart_upload()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.customize()
|
||||
.mutate_request(|request| {
|
||||
let separator = if request.uri().contains('?') { "&" } else { "?" };
|
||||
let uri = format!("{}{separator}x-rustfs-max-total-object-size=10", request.uri());
|
||||
request
|
||||
.set_uri(uri)
|
||||
.expect("the signed multipart size query must be a valid URI");
|
||||
})
|
||||
.send()
|
||||
.await?;
|
||||
let upload_id = upload.upload_id().ok_or("missing capped upload ID")?.to_owned();
|
||||
let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(3));
|
||||
let mut tasks = Vec::new();
|
||||
for (index, number, byte) in [(0_usize, 1_i32, 0x41_u8), (1, 2, 0x42)] {
|
||||
let client = clients[index].clone();
|
||||
let bucket = bucket.to_owned();
|
||||
let upload_id = upload_id.clone();
|
||||
let barrier = std::sync::Arc::clone(&barrier);
|
||||
tasks.push(tokio::spawn(async move {
|
||||
barrier.wait().await;
|
||||
let result = client
|
||||
.upload_part()
|
||||
.bucket(&bucket)
|
||||
.key(key)
|
||||
.upload_id(&upload_id)
|
||||
.part_number(number)
|
||||
.body(ByteStream::from(vec![byte; 6]))
|
||||
.send()
|
||||
.await;
|
||||
(number, byte, result)
|
||||
}));
|
||||
}
|
||||
barrier.wait().await;
|
||||
let mut winner = None;
|
||||
let mut rejected = 0;
|
||||
for task in tasks {
|
||||
let (number, byte, result) = tokio::time::timeout(DEADLINE, task).await??;
|
||||
match result {
|
||||
Ok(part) => {
|
||||
assert!(winner.is_none(), "both six-byte parts cannot fit the ten-byte limit");
|
||||
winner = Some((number, byte, part.e_tag().ok_or("missing accepted ETag")?.to_owned()));
|
||||
}
|
||||
Err(error) => {
|
||||
assert_eq!(
|
||||
error.as_service_error().and_then(|err| err.meta().code()),
|
||||
Some("EntityTooLarge"),
|
||||
"{error:?}"
|
||||
);
|
||||
rejected += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
let (number, byte, etag) = winner.ok_or("no part was admitted")?;
|
||||
assert_eq!(rejected, 1);
|
||||
let listed = clients[0]
|
||||
.list_parts()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.upload_id(&upload_id)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(listed.parts().len(), 1);
|
||||
assert_eq!(listed.parts()[0].part_number(), Some(number));
|
||||
assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
|
||||
clients[0]
|
||||
.complete_multipart_upload()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.upload_id(&upload_id)
|
||||
.multipart_upload(
|
||||
CompletedMultipartUpload::builder()
|
||||
.parts(CompletedPart::builder().part_number(number).e_tag(&etag).build())
|
||||
.build(),
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
for client in clients {
|
||||
assert_object_bytes_after_fault(client, bucket, key, &[byte; 6]).await?;
|
||||
}
|
||||
eprintln!("MPU size limit: two coordinators raced; exactly one six-byte part admitted under ten-byte limit");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_delay_preserves_write_quorum_and_rejects_quorum_minus_one() -> TestResult {
|
||||
let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
|
||||
list_parts_disk_loss_after_metadata_read(&cluster, &proxies, &clients, &bucket).await?;
|
||||
concurrent_capped_parts_respect_limit_on_distinct_nodes(&clients, &bucket).await?;
|
||||
phase_quorum_matrix(&cluster, &proxies, &clients, &bucket, false, 1, 3).await
|
||||
}
|
||||
|
||||
@@ -1214,6 +1403,22 @@ async fn mixed_version_failed_complete_requires_current_coordinator_for_abort()
|
||||
cluster.start_node(0).await?;
|
||||
crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness(&clients, &bucket, "mixed-mpu-current-writer", DEADLINE)
|
||||
.await?;
|
||||
// Read-only ListParts remains available during the documented mutation pause.
|
||||
// Compare the actual old and current coordinators on the same legacy upload.
|
||||
for (version, client) in [("current", &clients[0]), ("previous", &clients[1])] {
|
||||
let listed = client
|
||||
.list_parts()
|
||||
.bucket(&bucket)
|
||||
.key(key)
|
||||
.upload_id(upload_id)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(listed.parts().len(), 1, "{version} coordinator must list the legacy part");
|
||||
assert_eq!(listed.parts()[0].part_number(), Some(1));
|
||||
assert_eq!(listed.parts()[0].size(), Some(9 * 1024 * 1024));
|
||||
assert_eq!(listed.parts()[0].e_tag(), part.e_tag());
|
||||
}
|
||||
eprintln!("MPU mixed version: old and current coordinators agree on legacy ListParts identity and size");
|
||||
let scope = upload_scope(upload_id)?;
|
||||
let observers = proxies[1..]
|
||||
.iter()
|
||||
@@ -1344,7 +1549,7 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
|
||||
capability_sha256: Option<[u8; 32]>,
|
||||
abort: Option<AbortRef>,
|
||||
}
|
||||
let (cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
|
||||
let (mut cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
|
||||
let key = "partial-abort";
|
||||
let committed = payload_for("committed-before-abort", 4096);
|
||||
clients[0]
|
||||
@@ -1475,6 +1680,48 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
|
||||
.raw_response()
|
||||
.and_then(|response| response.headers().get("x-amz-request-id"))
|
||||
);
|
||||
assert!(
|
||||
retained.values().all(|source| source.participant.0 != 3),
|
||||
"the selected offline node must have no surviving root"
|
||||
);
|
||||
cluster.stop_node(3)?;
|
||||
let offline = tokio::time::timeout(
|
||||
DEADLINE,
|
||||
clients[0]
|
||||
.abort_multipart_upload()
|
||||
.bucket(&bucket)
|
||||
.key(key)
|
||||
.upload_id(upload_id)
|
||||
.send(),
|
||||
)
|
||||
.await?
|
||||
.expect_err("an unavailable original participant cannot be replaced by the certificate");
|
||||
assert_unavailable(&offline, "storage_write_quorum", None);
|
||||
assert!(
|
||||
multipart_sources(&cluster, Some(&hash))? == retained,
|
||||
"an unavailable participant must not authorize deletion or change retained evidence"
|
||||
);
|
||||
cluster.start_node(3).await?;
|
||||
wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
|
||||
eprintln!("MPU certified residual: unavailable original returned storage_write_quorum; all surviving bytes retained");
|
||||
|
||||
// Kill the coordinator and a node owning a real surviving source. The
|
||||
// retained bytes and full-N certificate must survive both process restarts
|
||||
// before a fresh coordinator can authorize the retry.
|
||||
let survivor_node = retained.values().next().ok_or("missing surviving source")?.participant.0;
|
||||
let mut restarted = vec![0, survivor_node];
|
||||
restarted.sort_unstable();
|
||||
restarted.dedup();
|
||||
for node in restarted {
|
||||
cluster.stop_node(node)?;
|
||||
cluster.start_node(node).await?;
|
||||
wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
|
||||
assert!(
|
||||
multipart_sources(&cluster, Some(&hash))? == retained,
|
||||
"restart must preserve every surviving source and certificate byte"
|
||||
);
|
||||
}
|
||||
eprintln!("MPU certified residual: coordinator and surviving owner restarted; all retained bytes unchanged");
|
||||
let delete_observers = proxies
|
||||
.iter()
|
||||
.map(|proxy| proxy.pause(&["RecoverMultipartSource"], usize::MAX, Some(&path)))
|
||||
|
||||
@@ -47,6 +47,9 @@ struct UpgradeSeed {
|
||||
history_bucket: String,
|
||||
history_key: &'static str,
|
||||
history_body: Vec<u8>,
|
||||
multipart_upload: String,
|
||||
multipart_etag: String,
|
||||
multipart_body: Vec<u8>,
|
||||
versioned_bucket: String,
|
||||
versioned_key: &'static str,
|
||||
version1: String,
|
||||
@@ -241,6 +244,25 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
|
||||
let history_body = b"written by the previous 4-node release".to_vec();
|
||||
put_object(&root, &history_bucket, history_key, history_body.clone()).await?;
|
||||
|
||||
let multipart_body = vec![0x5a; 4096];
|
||||
let multipart = root
|
||||
.create_multipart_upload()
|
||||
.bucket(&history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.send()
|
||||
.await?;
|
||||
let multipart_upload = multipart.upload_id().ok_or("legacy multipart upload ID missing")?.to_owned();
|
||||
let part = root
|
||||
.upload_part()
|
||||
.bucket(&history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&multipart_upload)
|
||||
.part_number(1)
|
||||
.body(aws_sdk_s3::primitives::ByteStream::from(multipart_body.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let multipart_etag = part.e_tag().ok_or("legacy multipart part ETag missing")?.to_owned();
|
||||
|
||||
let versioned_key = "versioned-history.txt";
|
||||
let version1_body = b"version-one-before-upgrade".to_vec();
|
||||
let version1 = root
|
||||
@@ -278,6 +300,9 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
|
||||
history_bucket,
|
||||
history_key,
|
||||
history_body,
|
||||
multipart_upload,
|
||||
multipart_etag,
|
||||
multipart_body,
|
||||
versioned_bucket,
|
||||
versioned_key,
|
||||
version1,
|
||||
@@ -304,6 +329,21 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
|
||||
)
|
||||
.await?;
|
||||
assert_object_bytes(&root_a, &seed.history_bucket, seed.history_key, &seed.history_body).await?;
|
||||
// During mixed-version operation this is read-only; mutation resumes only
|
||||
// after the whole fleet is upgraded.
|
||||
for client in [&root_a, &root_b] {
|
||||
let listed = client
|
||||
.list_parts()
|
||||
.bucket(&seed.history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&seed.multipart_upload)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(listed.parts().len(), 1, "{context}: legacy part listing");
|
||||
assert_eq!(listed.parts()[0].part_number(), Some(1));
|
||||
assert_eq!(listed.parts()[0].size(), Some(4096));
|
||||
assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
|
||||
}
|
||||
|
||||
let v1 = root_b
|
||||
.get_object()
|
||||
@@ -387,17 +427,146 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn compare_degraded_legacy_list_parts(dist: &mut DistCluster, seed: &UpgradeSeed, previous: &Path) -> TestResult {
|
||||
let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(seed.multipart_upload.as_bytes())?)?;
|
||||
let (_, upload_dir) = decoded.split_once('.').ok_or("legacy upload ID lacks deployment delimiter")?;
|
||||
let hash = sha256_hex(format!("{}/legacy-multipart.bin", seed.history_bucket).as_bytes());
|
||||
let root = format!("{hash}/{upload_dir}");
|
||||
let mut held = Vec::new();
|
||||
let mut original_files = Vec::new();
|
||||
for (index, node) in dist.cluster.nodes.iter().enumerate() {
|
||||
let path = Path::new(&node.data_dirs[0])
|
||||
.join(crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET)
|
||||
.join(&root);
|
||||
let entries = std::fs::read_dir(path)?.collect::<Result<Vec<_>, _>>()?;
|
||||
let dirs = entries
|
||||
.into_iter()
|
||||
.map(|entry| entry.path())
|
||||
.filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(dirs.len(), 1, "each configured disk must hold the original part before injection");
|
||||
let source_root = dirs[0].parent().ok_or("part directory has no upload root")?;
|
||||
for path in [
|
||||
source_root.join("xl.meta"),
|
||||
dirs[0].join("part.1"),
|
||||
dirs[0].join("part.1.meta"),
|
||||
] {
|
||||
original_files.push((path.clone(), std::fs::read(path)?));
|
||||
}
|
||||
if index < 3 {
|
||||
held.push((dirs[0].clone(), dirs[0].with_extension("held-for-mixed-test")));
|
||||
}
|
||||
}
|
||||
dist.cluster.stop_node(3)?;
|
||||
for (original, temporary) in &held {
|
||||
std::fs::rename(original, temporary)?;
|
||||
}
|
||||
|
||||
let clients = dist.clients()?;
|
||||
let current = tokio::time::timeout(
|
||||
CREDENTIAL_TIMEOUT,
|
||||
clients[0]
|
||||
.list_parts()
|
||||
.bucket(&seed.history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&seed.multipart_upload)
|
||||
.send(),
|
||||
)
|
||||
.await?;
|
||||
let legacy = tokio::time::timeout(
|
||||
CREDENTIAL_TIMEOUT,
|
||||
clients[1]
|
||||
.list_parts()
|
||||
.bucket(&seed.history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&seed.multipart_upload)
|
||||
.send(),
|
||||
)
|
||||
.await?;
|
||||
for (original, temporary) in &held {
|
||||
std::fs::rename(temporary, original)?;
|
||||
}
|
||||
// Three online disks retain metadata quorum but have no part directory;
|
||||
// the fourth disk is offline. A legacy empty response is not a proof that
|
||||
// the unavailable participant has no part.
|
||||
let legacy = legacy.expect("previous coordinator should expose its legacy empty-list behavior");
|
||||
assert!(legacy.parts().is_empty());
|
||||
let current = current.expect_err("current coordinator must not infer empty from missing directories and offline disks");
|
||||
assert_eq!(current.raw_response().expect("current S3 response").status().as_u16(), 503);
|
||||
assert_eq!(current.as_service_error().and_then(|error| error.meta().code()), Some("SlowDownRead"));
|
||||
dist.cluster.start_node_from_binary(3, previous).await?;
|
||||
wait_for_upgrade_write_readiness(&clients, &seed.history_bucket, "mixed-list-restored", CREDENTIAL_TIMEOUT).await?;
|
||||
for (path, before) in original_files {
|
||||
assert_eq!(
|
||||
std::fs::read(&path)?,
|
||||
before,
|
||||
"the fault must preserve every original source file: {path:?}"
|
||||
);
|
||||
}
|
||||
for client in [&clients[0], &clients[1]] {
|
||||
let listed = client
|
||||
.list_parts()
|
||||
.bucket(&seed.history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&seed.multipart_upload)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(listed.parts().len(), 1);
|
||||
assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
|
||||
}
|
||||
eprintln!(
|
||||
"MPU mixed degraded: one offline disk, three missing part directories; previous returned empty, current returned 503 SlowDownRead; original part restored on both"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn complete_legacy_multipart(dist: &DistCluster, seed: &UpgradeSeed) -> TestResult {
|
||||
// Native legacy-source upgrade needs every original participant, not just
|
||||
// the ordinary three-of-four write quorum proven by the earlier probes.
|
||||
let clients = dist.clients()?;
|
||||
crate::upgrade_compatibility_test::wait_for_drive_readmission(
|
||||
&clients,
|
||||
&seed.history_bucket,
|
||||
"legacy-multipart-all-drives",
|
||||
CREDENTIAL_TIMEOUT,
|
||||
)
|
||||
.await?;
|
||||
let root = &clients[0];
|
||||
root.complete_multipart_upload()
|
||||
.bucket(&seed.history_bucket)
|
||||
.key("legacy-multipart.bin")
|
||||
.upload_id(&seed.multipart_upload)
|
||||
.multipart_upload(
|
||||
aws_sdk_s3::types::CompletedMultipartUpload::builder()
|
||||
.parts(
|
||||
aws_sdk_s3::types::CompletedPart::builder()
|
||||
.part_number(1)
|
||||
.e_tag(&seed.multipart_etag)
|
||||
.build(),
|
||||
)
|
||||
.build(),
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
for client in &clients {
|
||||
assert_object_bytes(client, &seed.history_bucket, "legacy-multipart.bin", &seed.multipart_body).await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> TestResult {
|
||||
init_logging();
|
||||
let previous = source_binary()?;
|
||||
let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
|
||||
dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
|
||||
capture_upgrade_logs(&mut dist, "direct-upgrade")?;
|
||||
dist.start_from_binary(&previous).await?;
|
||||
|
||||
let seed = seed_history_and_iam(&dist).await?;
|
||||
dist.restart_with_current_binary().await?;
|
||||
assert_history_and_iam(&dist, &seed, "direct").await?;
|
||||
complete_legacy_multipart(&dist, &seed).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -419,6 +588,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
|
||||
init_logging();
|
||||
let previous = source_binary()?;
|
||||
let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
|
||||
dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
|
||||
capture_upgrade_logs(&mut dist, "rolling-upgrade")?;
|
||||
dist.start_from_binary(&previous).await?;
|
||||
|
||||
@@ -426,6 +596,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
|
||||
|
||||
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
|
||||
assert_history_and_iam(&dist, &seed, "one-current-node").await?;
|
||||
compare_degraded_legacy_list_parts(&mut dist, &seed, &previous).await?;
|
||||
|
||||
for node_idx in [1, 2] {
|
||||
replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
|
||||
@@ -434,5 +605,6 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
|
||||
|
||||
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
|
||||
assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
|
||||
complete_legacy_multipart(&dist, &seed).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1716,6 +1716,15 @@ mod tests {
|
||||
cluster.set_env("RUSTFS_HEAL_AUTO_HEAL_ENABLE", background_enabled.to_string());
|
||||
cluster.set_env("RUSTFS_HEAL_MRF_ENABLE", background_enabled.to_string());
|
||||
cluster.set_env("RUSTFS_SCANNER_ENABLED", background_enabled.to_string());
|
||||
let bucket = "heal-restart-during-rebuild";
|
||||
let outage_key_prefix = "cluster/written-while-node-down-";
|
||||
if !background_enabled {
|
||||
// With MRF disabled, an outage PUT still hands its missing shard to a
|
||||
// best-effort repair on the PUT coordinator. That repair retries until
|
||||
// the target rejoins, so suppress it to keep the explicit root heal
|
||||
// the only owner of the outage object.
|
||||
cluster.set_env("RUSTFS_E2E_SUPPRESS_RENAME_TAIL_HEAL_PREFIX", format!("{bucket}/{outage_key_prefix}"));
|
||||
}
|
||||
if background_enabled {
|
||||
// Only the scanner cadence is accelerated. Keep normal Heal
|
||||
// concurrency and every automatic recovery owner enabled.
|
||||
@@ -1748,7 +1757,6 @@ mod tests {
|
||||
cluster.start_with_binary(&server_binary).await?;
|
||||
let clients = cluster.create_all_clients()?;
|
||||
|
||||
let bucket = "heal-restart-during-rebuild";
|
||||
clients[0].create_bucket().bucket(bucket).send().await?;
|
||||
|
||||
let ReplacementDriveSelection {
|
||||
@@ -1876,7 +1884,7 @@ mod tests {
|
||||
let mut replacement_outage_erasure_index = None;
|
||||
let mut rejected_outage_keys = Vec::new();
|
||||
for attempt in 0..max_outage_write_attempts {
|
||||
let candidate_key = format!("cluster/written-while-node-down-{attempt:04}.bin");
|
||||
let candidate_key = format!("{outage_key_prefix}{attempt:04}.bin");
|
||||
let put_result = timeout(
|
||||
Duration::from_secs(30),
|
||||
clients[2]
|
||||
@@ -1986,25 +1994,6 @@ mod tests {
|
||||
}
|
||||
|
||||
let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
|
||||
if !background_enabled {
|
||||
// The PUT path may have admitted a direct Internal object repair while
|
||||
// node 1 was offline. Cancel the isolated bucket path before the target
|
||||
// returns; otherwise it could rebuild the outage object and invalidate
|
||||
// the explicit-root ownership assertion below.
|
||||
let cancel_outage_heal_path = format!("/rustfs/admin/v3/heal/{bucket}?forceStop=true");
|
||||
let (cancel_status, cancel_body) = admin_request(
|
||||
&cluster.nodes[0].url,
|
||||
Method::POST,
|
||||
&cancel_outage_heal_path,
|
||||
Some(heal_body.to_string()),
|
||||
&cluster.access_key,
|
||||
&cluster.secret_key,
|
||||
)
|
||||
.await?;
|
||||
if !cancel_status.is_success() {
|
||||
return Err(format!("cancel outage heal failed: {cancel_status} {cancel_body}").into());
|
||||
}
|
||||
}
|
||||
|
||||
// Keep the partial-repair checkpoint stable across readiness and admin
|
||||
// requests. Endpoint-blackhole tests must prove their own network stall.
|
||||
|
||||
@@ -38,6 +38,8 @@ mod tests {
|
||||
use tracing::info;
|
||||
|
||||
const PAGINATION_TIMEOUT: Duration = Duration::from_secs(120);
|
||||
// Bound queued durable commits on the single fixture disk under CI I/O pressure.
|
||||
const FIXTURE_PUT_CONCURRENCY: usize = 4;
|
||||
|
||||
// Fixture population can require thousands of durable PUTs. Bound the
|
||||
// listing traversal itself with one deadline shared by every page.
|
||||
@@ -564,7 +566,7 @@ mod tests {
|
||||
let object_count = 1002;
|
||||
// Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
|
||||
stream::iter(0..object_count)
|
||||
.for_each_concurrent(16, |i| {
|
||||
.for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |i| {
|
||||
let client = &client;
|
||||
async move {
|
||||
let key = format!("object{i:04}.txt");
|
||||
@@ -575,7 +577,7 @@ mod tests {
|
||||
.body(ByteStream::from_static(b"test content"))
|
||||
.send()
|
||||
.await
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
|
||||
}
|
||||
})
|
||||
.await;
|
||||
@@ -822,7 +824,7 @@ mod tests {
|
||||
.flat_map(|dir| (0..10).map(move |file| format!("dir-{dir:03}/file{file:02}.txt")))
|
||||
.collect();
|
||||
stream::iter(&all_keys)
|
||||
.for_each_concurrent(16, |key| {
|
||||
.for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |key| {
|
||||
let client = &client;
|
||||
async move {
|
||||
client
|
||||
@@ -832,7 +834,7 @@ mod tests {
|
||||
.body(ByteStream::from_static(b"x"))
|
||||
.send()
|
||||
.await
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
|
||||
}
|
||||
})
|
||||
.await;
|
||||
@@ -1055,7 +1057,7 @@ mod tests {
|
||||
let files_per_dir = 100;
|
||||
let fixture_started = std::time::Instant::now();
|
||||
stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
|
||||
.for_each_concurrent(16, |(d, f)| {
|
||||
.for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |(d, f)| {
|
||||
let client = &client;
|
||||
async move {
|
||||
let key = format!("dir-{d:02}/file{f:03}.txt");
|
||||
@@ -1066,7 +1068,7 @@ mod tests {
|
||||
.body(ByteStream::from_static(b"x"))
|
||||
.send()
|
||||
.await
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
|
||||
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
|
||||
}
|
||||
})
|
||||
.await;
|
||||
|
||||
@@ -261,6 +261,61 @@ mod tests {
|
||||
archive
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn snowball_auto_extract_rejects_veeam_virtual_members() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_VEEAM_SOSAPI_ENABLE", "true")])
|
||||
.await?;
|
||||
let client = env.create_s3_client();
|
||||
let bucket = "snowball-veeam-reserved";
|
||||
client.create_bucket().bucket(bucket).send().await?;
|
||||
let prefix = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c";
|
||||
for (member, destination, use_prefix) in [
|
||||
(format!("{prefix}/system.xml"), format!("{prefix}/system.xml"), false),
|
||||
("capacity.xml".to_string(), format!("{prefix}/capacity.xml"), true),
|
||||
] {
|
||||
let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
|
||||
append_typed_entry(&mut builder, &member, tokio_tar::EntryType::Regular, b"blocked").await?;
|
||||
let archive = builder.into_inner().await?.into_inner();
|
||||
let mut request = client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("fixture.tar")
|
||||
.metadata("Snowball-Auto-Extract", "true")
|
||||
.body(ByteStream::from(archive));
|
||||
if use_prefix {
|
||||
request = request
|
||||
.metadata("Minio-Snowball-Prefix", prefix)
|
||||
.metadata("Minio-Snowball-Ignore-Errors", "true");
|
||||
}
|
||||
let error = request.send().await.expect_err("virtual archive members must be rejected");
|
||||
assert_eq!(error.code(), Some("MethodNotAllowed"));
|
||||
let listed = client.list_objects_v2().bucket(bucket).prefix(&destination).send().await?;
|
||||
assert!(listed.contents().is_empty(), "rejected virtual member must not be stored");
|
||||
}
|
||||
let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
|
||||
append_typed_entry(&mut builder, "ordinary.xml", tokio_tar::EntryType::Regular, b"ordinary").await?;
|
||||
client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("fixture.tar")
|
||||
.metadata("Snowball-Auto-Extract", "true")
|
||||
.metadata("Minio-Snowball-Prefix", prefix)
|
||||
.body(ByteStream::from(builder.into_inner().await?.into_inner()))
|
||||
.send()
|
||||
.await?;
|
||||
let ordinary = client
|
||||
.get_object()
|
||||
.bucket(bucket)
|
||||
.key(format!("{prefix}/ordinary.xml"))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(ordinary.body.collect().await?.into_bytes().as_ref(), b"ordinary");
|
||||
env.stop_server();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
@@ -13,9 +13,13 @@
|
||||
// limitations under the License.
|
||||
|
||||
use crate::common::{RustFSTestClusterEnvironment, init_logging};
|
||||
use crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET;
|
||||
use aws_sdk_s3::error::SdkError;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use tokio::time::{Duration, sleep};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::time::{Duration, Instant, sleep};
|
||||
use tracing::info;
|
||||
use uuid::Uuid;
|
||||
|
||||
@@ -84,6 +88,126 @@ async fn wait_for_cleanup_on_all_nodes(
|
||||
Err("stale multipart upload was not cleaned up on all nodes within timeout".into())
|
||||
}
|
||||
|
||||
type PhysicalTree = BTreeMap<PathBuf, Option<Vec<u8>>>;
|
||||
|
||||
fn physical_tree(root: &Path) -> Result<PhysicalTree, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let mut result = BTreeMap::new();
|
||||
let mut pending = vec![root.to_path_buf()];
|
||||
while let Some(directory) = pending.pop() {
|
||||
for entry in std::fs::read_dir(directory)? {
|
||||
let entry = entry?;
|
||||
let path = entry.path();
|
||||
let relative = path.strip_prefix(root)?.to_path_buf();
|
||||
let kind = entry.file_type()?;
|
||||
if kind.is_dir() {
|
||||
result.insert(relative, None);
|
||||
pending.push(path);
|
||||
} else if kind.is_file() {
|
||||
result.insert(relative, Some(std::fs::read(path)?));
|
||||
} else {
|
||||
return Err(format!("unexpected staging entry type: {relative:?}").into());
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
async fn assert_low_quorum_residuals_are_diagnosed_and_preserved(
|
||||
cluster: &RustFSTestClusterEnvironment,
|
||||
clients: &[aws_sdk_s3::Client],
|
||||
log_paths: &[PathBuf],
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let mut residuals = Vec::new();
|
||||
for copies in [1_usize, 2] {
|
||||
let key = format!("multipart/partial-{copies}-{}", Uuid::new_v4().simple());
|
||||
let upload = clients[0]
|
||||
.create_multipart_upload()
|
||||
.bucket(CLEANUP_BUCKET)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await?;
|
||||
let upload_id = upload.upload_id().ok_or("create response missing upload ID")?;
|
||||
clients[0]
|
||||
.upload_part()
|
||||
.bucket(CLEANUP_BUCKET)
|
||||
.key(&key)
|
||||
.upload_id(upload_id)
|
||||
.part_number(1)
|
||||
.body(ByteStream::from(vec![0x41; 4096]))
|
||||
.send()
|
||||
.await?;
|
||||
let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes())?)?;
|
||||
let (_, upload_dir) = decoded.split_once('.').ok_or("upload ID lacks deployment delimiter")?;
|
||||
let digest = Sha256::digest(format!("{CLEANUP_BUCKET}/{key}").as_bytes());
|
||||
let hash = hex_simd::encode_to_string(digest, hex_simd::AsciiCase::Lower);
|
||||
let relative = format!("{hash}/{upload_dir}");
|
||||
let roots = cluster
|
||||
.nodes
|
||||
.iter()
|
||||
.map(|node| {
|
||||
Path::new(&node.data_dirs[0])
|
||||
.join(RUSTFS_META_MULTIPART_BUCKET)
|
||||
.join(&relative)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(roots.len(), 4);
|
||||
let mut preserved = Vec::new();
|
||||
for root in &roots {
|
||||
assert!(
|
||||
root.join("xl.meta").is_file(),
|
||||
"all four original metadata files must exist before injection"
|
||||
);
|
||||
preserved.push(physical_tree(root)?);
|
||||
}
|
||||
for root in roots.iter().skip(copies) {
|
||||
std::fs::remove_dir_all(root)?;
|
||||
}
|
||||
residuals.push((relative, roots, preserved, copies));
|
||||
}
|
||||
|
||||
let deadline = Instant::now() + Duration::from_secs(45);
|
||||
loop {
|
||||
let logs = log_paths
|
||||
.iter()
|
||||
.filter_map(|path| std::fs::read_to_string(path).ok())
|
||||
.collect::<Vec<_>>();
|
||||
let diagnosed = residuals.iter().all(|(path, _, _, _)| {
|
||||
logs.iter().any(|log| {
|
||||
log.lines()
|
||||
.any(|line| line.contains(path) && line.contains("multipart_cleanup_lock_or_recheck_failed"))
|
||||
})
|
||||
});
|
||||
if diagnosed {
|
||||
break;
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
return Err("stale low-quorum residuals were not diagnosed before the deadline".into());
|
||||
}
|
||||
sleep(Duration::from_millis(200)).await;
|
||||
}
|
||||
for (path, roots, preserved, copies) in &residuals {
|
||||
for slot in 0..*copies {
|
||||
assert_eq!(
|
||||
physical_tree(&roots[slot])?,
|
||||
preserved[slot],
|
||||
"retained source evidence changed: {path}, slot {slot}"
|
||||
);
|
||||
}
|
||||
for root in roots.iter().skip(*copies) {
|
||||
assert!(!root.exists(), "cleanup must not recreate missing source roots: {path}");
|
||||
}
|
||||
}
|
||||
for path in log_paths {
|
||||
let log = std::fs::read_to_string(path)?;
|
||||
assert!(
|
||||
!log.contains("op=\"upload_part\"") && !log.contains("\"op\":\"upload_part\""),
|
||||
"background cleanup must not be logged as a client UploadPart"
|
||||
);
|
||||
}
|
||||
eprintln!("Stale MPU low-quorum: EC:2 one/two-copy roots diagnosed; full surviving file inventories preserved");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
|
||||
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
@@ -92,6 +216,22 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
|
||||
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
|
||||
cluster.set_env("RUSTFS_API_STALE_UPLOADS_EXPIRY", "5s");
|
||||
cluster.set_env("RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL", "1s");
|
||||
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
|
||||
cluster.set_env(
|
||||
"RUST_LOG",
|
||||
concat!("warn,rustfs_ecstore", "::bucket::lifecycle::bucket_lifecycle_ops=debug"),
|
||||
);
|
||||
let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")
|
||||
.map(PathBuf::from)
|
||||
.map(|root| root.join(format!("stale-multipart-{}", Uuid::new_v4().simple())))
|
||||
.unwrap_or_else(|| PathBuf::from(&cluster.temp_dir));
|
||||
std::fs::create_dir_all(&log_dir)?;
|
||||
let log_paths = (0..cluster.nodes.len())
|
||||
.map(|index| log_dir.join(format!("stale-node-{index}.log")))
|
||||
.collect::<Vec<_>>();
|
||||
for (index, path) in log_paths.iter().enumerate() {
|
||||
cluster.set_node_capture_log_path(index, path.to_string_lossy().into_owned())?;
|
||||
}
|
||||
cluster.start().await?;
|
||||
cluster.create_test_bucket(CLEANUP_BUCKET).await?;
|
||||
|
||||
@@ -137,6 +277,7 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
|
||||
);
|
||||
|
||||
wait_for_cleanup_on_all_nodes(&clients, CLEANUP_BUCKET, &key, &upload_id).await?;
|
||||
assert_low_quorum_residuals_are_diagnosed_and_preserved(&cluster, &clients, &log_paths).await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1066,7 +1066,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness(
|
||||
/// reduced-redundancy class then fails with 503 `ServiceUnavailable` until the
|
||||
/// serving node writes through every drive, on the previous release and on
|
||||
/// the current build alike.
|
||||
async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
|
||||
pub(crate) async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
|
||||
wait_for_upgrade_writes(
|
||||
clients,
|
||||
bucket,
|
||||
|
||||
@@ -600,6 +600,154 @@ mod tests {
|
||||
info!("✅ PASSED: Veeam backup workflow simulation completed successfully");
|
||||
}
|
||||
|
||||
/// Veeam B&R discovers SOSAPI by GET/HEAD of two well-known keys.
|
||||
#[tokio::test]
|
||||
async fn test_veeam_sosapi_system_and_capacity() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
use crate::common::local_http_client;
|
||||
|
||||
const SYSTEM_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml";
|
||||
const CAPACITY_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/capacity.xml";
|
||||
|
||||
init_logging();
|
||||
info!("🧪 TEST: Veeam SOSAPI system.xml and capacity.xml");
|
||||
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
env.start_rustfs_server_with_env(vec![], crate::common::FAST_DATA_USAGE_SCANNER_ENV)
|
||||
.await?;
|
||||
let client = create_s3_client(&env);
|
||||
let bucket = "veeam-sosapi";
|
||||
create_bucket(&client, bucket).await?;
|
||||
|
||||
let system = client.get_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
|
||||
let system_etag = system.e_tag().map(str::to_string);
|
||||
let system_len = system.content_length();
|
||||
assert_eq!(system.content_type(), Some("application/xml"));
|
||||
let system_body = system.body.collect().await?.into_bytes();
|
||||
let system_xml = std::str::from_utf8(&system_body)?;
|
||||
assert!(system_xml.contains("<ProtocolVersion>\"1.0\"</ProtocolVersion>"), "{system_xml}");
|
||||
assert!(
|
||||
system_xml.contains("<ModelName>\"RustFS ") && system_xml.contains("\"</ModelName>"),
|
||||
"{system_xml}"
|
||||
);
|
||||
assert!(system_xml.contains("<CapacityInfo>true</CapacityInfo>"));
|
||||
assert!(system_xml.contains("<UploadSessions>false</UploadSessions>"));
|
||||
assert!(system_xml.contains("<IAMSTS>false</IAMSTS>"));
|
||||
assert!(!system_xml.contains("SystemRecommendations"));
|
||||
assert!(!system_xml.contains("APIEndpoints"));
|
||||
assert!(!system_xml.contains("""));
|
||||
|
||||
let head = client.head_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
|
||||
assert_eq!(head.content_type(), Some("application/xml"));
|
||||
assert_eq!(head.content_length(), system_len);
|
||||
assert_eq!(head.e_tag().map(str::to_string), system_etag);
|
||||
assert!(head.last_modified().is_some());
|
||||
|
||||
let ranged = client
|
||||
.get_object()
|
||||
.bucket(bucket)
|
||||
.key(SYSTEM_KEY)
|
||||
.range("bytes=0-3")
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(ranged.content_length(), Some(4));
|
||||
let ranged_header = ranged.content_range().unwrap_or("");
|
||||
assert!(ranged_header.starts_with("bytes 0-3/"), "{ranged_header}");
|
||||
let ranged_body = ranged.body.collect().await?.into_bytes();
|
||||
assert_eq!(&ranged_body[..], &system_body[..4]);
|
||||
|
||||
let capacity = tokio::time::timeout(std::time::Duration::from_secs(180), async {
|
||||
loop {
|
||||
match client.get_object().bucket(bucket).key(CAPACITY_KEY).send().await {
|
||||
Ok(capacity) => break Ok(capacity),
|
||||
Err(err) if err.code() == Some("ServiceUnavailable") => {
|
||||
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
|
||||
}
|
||||
Err(err) => break Err(err),
|
||||
}
|
||||
}
|
||||
})
|
||||
.await??;
|
||||
assert_eq!(capacity.content_type(), Some("application/xml"));
|
||||
let capacity_body = capacity.body.collect().await?.into_bytes();
|
||||
let capacity_xml = std::str::from_utf8(&capacity_body)?;
|
||||
let capacity_value = xml_u64(capacity_xml, "Capacity")?;
|
||||
let available_value = xml_u64(capacity_xml, "Available")?;
|
||||
let used_value = xml_u64(capacity_xml, "Used")?;
|
||||
assert!(used_value <= capacity_value, "{capacity_xml}");
|
||||
assert!(available_value <= capacity_value.saturating_sub(used_value), "{capacity_xml}");
|
||||
|
||||
let put_err = client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key(SYSTEM_KEY)
|
||||
.body(ByteStream::from_static(b"overwrite"))
|
||||
.send()
|
||||
.await
|
||||
.expect_err("SOSAPI put must fail");
|
||||
assert_eq!(put_err.code(), Some("MethodNotAllowed"));
|
||||
let delete_err = client
|
||||
.delete_object()
|
||||
.bucket(bucket)
|
||||
.key(CAPACITY_KEY)
|
||||
.send()
|
||||
.await
|
||||
.expect_err("SOSAPI delete must fail");
|
||||
assert_eq!(delete_err.code(), Some("MethodNotAllowed"));
|
||||
|
||||
let normal_key = "Veeam/Backup/job.dat";
|
||||
client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key(normal_key)
|
||||
.body(ByteStream::from_static(b"backup-bytes"))
|
||||
.send()
|
||||
.await?;
|
||||
let normal = client.get_object().bucket(bucket).key(normal_key).send().await?;
|
||||
let normal_body = normal.body.collect().await?.into_bytes();
|
||||
assert_eq!(&normal_body[..], b"backup-bytes");
|
||||
|
||||
let anon = local_http_client()
|
||||
.get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(anon.status(), reqwest::StatusCode::FORBIDDEN);
|
||||
let anon_body = anon.text().await?;
|
||||
assert!(!anon_body.contains("ProtocolVersion"), "{anon_body}");
|
||||
|
||||
let now = time::OffsetDateTime::now_utc();
|
||||
let scope_date = format!("{:04}{:02}{:02}", now.year(), u8::from(now.month()), now.day());
|
||||
let amz_date = format!("{scope_date}T{:02}{:02}{:02}Z", now.hour(), now.minute(), now.second());
|
||||
let bad = local_http_client()
|
||||
.get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
|
||||
.header("x-amz-date", &amz_date)
|
||||
.header("x-amz-content-sha256", "UNSIGNED-PAYLOAD")
|
||||
.header(
|
||||
"authorization",
|
||||
format!(
|
||||
"AWS4-HMAC-SHA256 Credential=bad/{scope_date}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000"
|
||||
),
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(bad.status(), reqwest::StatusCode::FORBIDDEN);
|
||||
let bad_body = bad.text().await?;
|
||||
assert!(!bad_body.contains("ProtocolVersion"), "{bad_body}");
|
||||
|
||||
info!("✅ PASSED: Veeam SOSAPI system.xml and capacity.xml");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn xml_u64(xml: &str, element: &str) -> Result<u64, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let start = format!("<{element}>");
|
||||
let end = format!("</{element}>");
|
||||
let start_idx = xml.find(&start).ok_or_else(|| format!("missing {element} in {xml}"))? + start.len();
|
||||
let end_idx = xml[start_idx..]
|
||||
.find(&end)
|
||||
.ok_or_else(|| format!("unclosed {element} in {xml}"))?
|
||||
+ start_idx;
|
||||
Ok(xml[start_idx..end_idx].parse()?)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_terraform_put_after_delete() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
@@ -252,7 +252,7 @@ pub mod bucket {
|
||||
pub mod quota {
|
||||
pub use crate::bucket::quota::{
|
||||
BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
|
||||
QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration,
|
||||
QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration, QuotaType,
|
||||
};
|
||||
|
||||
pub mod checker {
|
||||
@@ -371,15 +371,24 @@ pub mod compression {
|
||||
|
||||
pub mod config {
|
||||
pub mod com {
|
||||
/// Preserve empty existing objects so callers can distinguish corruption from absence.
|
||||
pub async fn read_config_preserve_empty_with_metadata_opts(
|
||||
api: std::sync::Arc<crate::store::ECStore>,
|
||||
file: &str,
|
||||
opts: &crate::object_api::ObjectOptions,
|
||||
) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
|
||||
crate::config::com::read_config_preserve_empty_with_metadata_opts(api, file, opts).await
|
||||
}
|
||||
|
||||
pub use crate::config::com::{
|
||||
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
|
||||
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
|
||||
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
|
||||
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
|
||||
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
|
||||
save_server_config, save_server_config_no_lock, save_server_config_snapshot,
|
||||
save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
|
||||
with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
|
||||
read_server_config_snapshot, read_server_config_with_generation, save_config, save_config_no_lock,
|
||||
save_config_with_opts, save_config_with_opts_quiet, save_server_config, save_server_config_no_lock,
|
||||
save_server_config_snapshot, save_server_config_snapshot_with_generation, server_config_path,
|
||||
try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
|
||||
with_server_config_write_lock,
|
||||
};
|
||||
}
|
||||
@@ -416,9 +425,10 @@ pub mod data_usage {
|
||||
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
|
||||
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
|
||||
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
|
||||
load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size,
|
||||
record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
|
||||
record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
|
||||
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
|
||||
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
|
||||
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
|
||||
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
|
||||
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
|
||||
remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
|
||||
store_data_usage_in_backend,
|
||||
|
||||
@@ -63,6 +63,7 @@ use crate::services::tier::{
|
||||
};
|
||||
use crate::set_disk::{
|
||||
MAX_PARTS_COUNT, RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY, SetDisks, get_lock_acquire_timeout,
|
||||
multipart_part_logical_size, should_persist_encryption_original_size,
|
||||
};
|
||||
use crate::storage_api_contracts::{
|
||||
lifecycle::ExpirationOptions,
|
||||
@@ -119,6 +120,7 @@ const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
|
||||
const EVENT_LIFECYCLE_WORKER_STATE: &str = "lifecycle_worker_state";
|
||||
const EVENT_LIFECYCLE_TRANSITION_COMPENSATION: &str = "lifecycle_transition_compensation";
|
||||
const EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP: &str = "lifecycle_stale_multipart_cleanup";
|
||||
const METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL: &str = "rustfs_stale_multipart_cleanup_blocked_total";
|
||||
const EVENT_LIFECYCLE_SCAN_SKIPPED: &str = "lifecycle_scan_skipped";
|
||||
const EVENT_LIFECYCLE_EVALUATION_FAILED: &str = "lifecycle_evaluation_failed";
|
||||
const EVENT_LIFECYCLE_EXPIRED_DETECTED: &str = "lifecycle_expired_detected";
|
||||
@@ -1086,8 +1088,9 @@ impl ExpiryState {
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
#[doc(hidden)]
|
||||
pub fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
|
||||
let (tx, rx) = mpsc::channel(capacity);
|
||||
Arc::new(RwLock::new(Self {
|
||||
tasks_tx: vec![tx],
|
||||
@@ -3510,19 +3513,29 @@ async fn stale_upload_current_size_with_opts(
|
||||
&ObjectOptions {
|
||||
data_movement,
|
||||
no_lock,
|
||||
strict_multipart_part_sizes: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.ok()?;
|
||||
|
||||
Some(
|
||||
parts
|
||||
.parts
|
||||
.iter()
|
||||
.map(|part| part.actual_size.max(part.size as i64).max(0) as usize)
|
||||
.sum(),
|
||||
)
|
||||
stale_upload_size_from_parts(&parts)
|
||||
}
|
||||
|
||||
fn stale_upload_size_from_parts(parts: &crate::storage_api_contracts::multipart::ListPartsInfo) -> Option<usize> {
|
||||
if parts.is_truncated {
|
||||
return None;
|
||||
}
|
||||
let transformed = rustfs_utils::http::contains_key_str(&parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION)
|
||||
|| should_persist_encryption_original_size(&parts.user_defined);
|
||||
parts.parts.iter().try_fold(0usize, |total, part| {
|
||||
// ListParts represents a quorum-confirmed missing part with an empty ETag.
|
||||
if part.etag.as_deref().is_none_or(str::is_empty) {
|
||||
return None;
|
||||
}
|
||||
total.checked_add(multipart_part_logical_size(part.size, part.actual_size, transformed)?)
|
||||
})
|
||||
}
|
||||
|
||||
async fn stale_upload_lifecycle_due(
|
||||
@@ -3539,18 +3552,43 @@ async fn stale_upload_lifecycle_due(
|
||||
let bucket = metadata.get(RUSTFS_MULTIPART_BUCKET_KEY)?;
|
||||
let object = metadata.get(RUSTFS_MULTIPART_OBJECT_KEY)?;
|
||||
|
||||
let lifecycle = match metadata_boundary::get_lifecycle_config(bucket).await {
|
||||
let mut lifecycle = match metadata_boundary::get_lifecycle_config(bucket).await {
|
||||
Ok((lifecycle, _)) => lifecycle,
|
||||
Err(_) => return None,
|
||||
};
|
||||
|
||||
let has_size_filter = |filter: &s3s::dto::LifecycleRuleFilter| {
|
||||
filter.object_size_greater_than.is_some()
|
||||
|| filter.object_size_less_than.is_some()
|
||||
|| filter
|
||||
.and
|
||||
.as_ref()
|
||||
.is_some_and(|and| and.object_size_greater_than.is_some() || and.object_size_less_than.is_some())
|
||||
};
|
||||
let needs_size = lifecycle
|
||||
.rules
|
||||
.iter()
|
||||
.any(|rule| rule.abort_incomplete_multipart_upload.is_some() && rule.filter.as_ref().is_some_and(has_size_filter));
|
||||
let size = if needs_size {
|
||||
match stale_upload_current_size_with_opts(set, metadata, upload_dir, no_lock).await {
|
||||
Some(size) => size,
|
||||
None => {
|
||||
// An unknown size cannot authorize a size-filtered abort.
|
||||
// Unrelated rules can still be evaluated independently.
|
||||
lifecycle
|
||||
.rules
|
||||
.retain(|rule| !rule.filter.as_ref().is_some_and(has_size_filter));
|
||||
0
|
||||
}
|
||||
}
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let object_opts = ObjectOpts {
|
||||
name: object.clone(),
|
||||
user_tags: metadata.get(AMZ_OBJECT_TAGGING).cloned().unwrap_or_default(),
|
||||
mod_time: Some(initiated),
|
||||
size: stale_upload_current_size_with_opts(set, metadata, upload_dir, no_lock)
|
||||
.await
|
||||
.unwrap_or_default(),
|
||||
size,
|
||||
is_latest: true,
|
||||
delete_marker: false,
|
||||
user_defined: metadata.clone(),
|
||||
@@ -3707,6 +3745,23 @@ async fn cleanup_empty_multipart_sha_dirs_on_local_disks(set: &Arc<SetDisks>) {
|
||||
}
|
||||
}
|
||||
|
||||
fn record_stale_multipart_cleanup_blocked(stage: &'static str) {
|
||||
metrics::counter!(METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, "stage" => stage).increment(1);
|
||||
}
|
||||
|
||||
fn record_stale_multipart_cleanup_lock_lost(path: &str) {
|
||||
record_stale_multipart_cleanup_blocked("lock_lost");
|
||||
debug!(
|
||||
event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
|
||||
state = "skipped",
|
||||
path = %path,
|
||||
reason = "multipart_cleanup_lock_lost",
|
||||
"Skipped stale multipart cleanup"
|
||||
);
|
||||
}
|
||||
|
||||
async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: OffsetDateTime, default_expiry: StdDuration) -> usize {
|
||||
let mut deleted = 0usize;
|
||||
let mut candidates = HashMap::new();
|
||||
@@ -3803,7 +3858,8 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
|
||||
for candidate in candidates.into_values() {
|
||||
let upload_dir = candidate.path.rsplit('/').next().unwrap_or_default().to_string();
|
||||
let mut due = stale_upload_default_due(candidate.initiated, default_expiry);
|
||||
if let Some(metadata) = candidate.metadata.as_ref()
|
||||
if now < due
|
||||
&& let Some(metadata) = candidate.metadata.as_ref()
|
||||
&& let Some(lifecycle_due) = stale_upload_lifecycle_due(set, metadata, candidate.initiated, &upload_dir, false).await
|
||||
&& lifecycle_due < due
|
||||
{
|
||||
@@ -3817,6 +3873,7 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
|
||||
let cleanup_guard = match set.lock_stale_multipart_cleanup(&candidate.path).await {
|
||||
Ok(guard) => guard,
|
||||
Err(err) => {
|
||||
record_stale_multipart_cleanup_blocked("recheck");
|
||||
debug!(
|
||||
event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -3832,13 +3889,18 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
|
||||
let current_metadata = cleanup_guard.file_info().metadata.clone();
|
||||
let current_initiated = initiated_from_upload_dir(&upload_dir, cleanup_guard.file_info().mod_time);
|
||||
let mut current_due = stale_upload_default_due(current_initiated, default_expiry);
|
||||
if let Some(lifecycle_due) =
|
||||
stale_upload_lifecycle_due(set, ¤t_metadata, current_initiated, &upload_dir, true).await
|
||||
if now < current_due
|
||||
&& let Some(lifecycle_due) =
|
||||
stale_upload_lifecycle_due(set, ¤t_metadata, current_initiated, &upload_dir, true).await
|
||||
&& lifecycle_due < current_due
|
||||
{
|
||||
current_due = lifecycle_due;
|
||||
}
|
||||
if now < current_due || cleanup_guard.is_lock_lost() {
|
||||
if now < current_due {
|
||||
continue;
|
||||
}
|
||||
if cleanup_guard.is_lock_lost() {
|
||||
record_stale_multipart_cleanup_lock_lost(&candidate.path);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -3858,15 +3920,18 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
|
||||
"Removed stale multipart upload"
|
||||
);
|
||||
}
|
||||
Err(err) => debug!(
|
||||
event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
|
||||
path = %candidate.path,
|
||||
error = ?err,
|
||||
reason = "multipart_remove_failed",
|
||||
"Failed to remove stale multipart upload"
|
||||
),
|
||||
Err(err) => {
|
||||
record_stale_multipart_cleanup_blocked("delete");
|
||||
debug!(
|
||||
event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
|
||||
path = %candidate.path,
|
||||
error = ?err,
|
||||
reason = "multipart_remove_failed",
|
||||
"Failed to remove stale multipart upload"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15868,6 +15933,530 @@ mod tests {
|
||||
assert!(ecstore.is_ok(), "fresh local ECStore boot should succeed, got {ecstore:?}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
#[serial]
|
||||
async fn stale_multipart_cleanup_preserves_partial_metadata_without_upload_part_warning() {
|
||||
use tracing::instrument::WithSubscriber as _;
|
||||
#[derive(Clone, Default)]
|
||||
struct CapturedLogs(Arc<StdMutex<Vec<u8>>>);
|
||||
struct LogWriter(Arc<StdMutex<Vec<u8>>>);
|
||||
impl std::io::Write for LogWriter {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.0.lock().unwrap().extend_from_slice(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLogs {
|
||||
type Writer = LogWriter;
|
||||
fn make_writer(&'a self) -> Self::Writer {
|
||||
LogWriter(self.0.clone())
|
||||
}
|
||||
}
|
||||
|
||||
let (paths, ecstore) = setup_test_env().await;
|
||||
let bucket = format!("stale-partial-{}", Uuid::new_v4().simple());
|
||||
create_test_bucket(&ecstore, &bucket).await;
|
||||
for copies in [1, 2] {
|
||||
let object = format!("partial-{copies}.bin");
|
||||
let initiated = OffsetDateTime::now_utc() - time::Duration::hours(30);
|
||||
let upload = ecstore
|
||||
.new_multipart_upload(
|
||||
&bucket,
|
||||
&object,
|
||||
&ObjectOptions {
|
||||
mod_time: Some(initiated),
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("multipart upload should be created");
|
||||
let decoded = base64_simd::URL_SAFE_NO_PAD
|
||||
.decode_to_vec(upload.upload_id.as_bytes())
|
||||
.unwrap();
|
||||
let decoded = String::from_utf8(decoded).unwrap();
|
||||
let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload id");
|
||||
let upload_path = format!("{}/{upload_dir}", multipart_sha_dir(&bucket, &object));
|
||||
let physical_paths: Vec<_> = paths
|
||||
.iter()
|
||||
.map(|path| path.join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path))
|
||||
.collect();
|
||||
let original: Vec<_> = physical_paths
|
||||
.iter()
|
||||
.take(copies)
|
||||
.map(|path| std::fs::read(path.join(STORAGE_FORMAT_FILE)).unwrap())
|
||||
.collect();
|
||||
let info = super::get_file_info(
|
||||
&original[0],
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_path,
|
||||
"",
|
||||
super::FileInfoOpts {
|
||||
data: false,
|
||||
include_free_versions: false,
|
||||
include_part_checksums: false,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!((info.erasure.data_blocks, info.erasure.parity_blocks), (2, 2));
|
||||
for path in physical_paths.iter().skip(copies) {
|
||||
fs::remove_dir_all(path).await.expect("remove selected metadata copies");
|
||||
}
|
||||
if copies == 2 {
|
||||
ecstore
|
||||
.get_multipart_info(&bucket, &object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("two metadata copies must meet read quorum");
|
||||
}
|
||||
|
||||
let logs = CapturedLogs::default();
|
||||
let subscriber = tracing_subscriber::fmt()
|
||||
.with_max_level(tracing::Level::DEBUG)
|
||||
.with_writer(logs.clone())
|
||||
.with_ansi(false)
|
||||
.without_time()
|
||||
.finish();
|
||||
let recorder = crate::test_metrics::CapturingRecorder::default();
|
||||
let recorder_guard = metrics::set_default_local_recorder(&recorder);
|
||||
cleanup_stale_multipart_uploads_once_at(
|
||||
ecstore.clone(),
|
||||
OffsetDateTime::now_utc(),
|
||||
StdDuration::from_secs(24 * 60 * 60),
|
||||
)
|
||||
.with_subscriber(subscriber)
|
||||
.await;
|
||||
drop(recorder_guard);
|
||||
for (path, expected) in physical_paths.iter().zip(&original) {
|
||||
assert_eq!(fs::read(path.join(STORAGE_FORMAT_FILE)).await.unwrap(), *expected);
|
||||
}
|
||||
let log_output = String::from_utf8(logs.0.lock().unwrap().clone()).unwrap();
|
||||
let rechecks = log_output
|
||||
.lines()
|
||||
.filter(|line| line.contains("multipart_cleanup_lock_or_recheck_failed"));
|
||||
let matching_path = rechecks.clone().any(|line| line.contains(&upload_path));
|
||||
assert!(
|
||||
matching_path,
|
||||
"missing path-specific background diagnostic for {upload_path}: {log_output}"
|
||||
);
|
||||
assert_eq!(
|
||||
recorder.counter_value(super::METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, &[("stage", "recheck")]),
|
||||
u64::try_from(rechecks.count()).expect("diagnostic count fits u64"),
|
||||
"each observed recheck failure must increment the metric exactly once"
|
||||
);
|
||||
assert!(!log_output.contains("op=\"upload_part\""), "unexpected upload-part warning: {log_output}");
|
||||
for path in physical_paths.iter().take(copies) {
|
||||
fs::remove_dir_all(path).await.unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
let logs = CapturedLogs::default();
|
||||
let subscriber = tracing_subscriber::fmt()
|
||||
.with_max_level(tracing::Level::DEBUG)
|
||||
.with_writer(logs.clone())
|
||||
.with_ansi(false)
|
||||
.without_time()
|
||||
.finish();
|
||||
let recorder = crate::test_metrics::CapturingRecorder::default();
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
tracing::subscriber::with_default(subscriber, || {
|
||||
super::record_stale_multipart_cleanup_lock_lost("test-hash/test-upload");
|
||||
})
|
||||
});
|
||||
let output = String::from_utf8(logs.0.lock().unwrap().clone()).unwrap();
|
||||
assert!(
|
||||
output.contains("multipart_cleanup_lock_lost") && output.contains("test-hash/test-upload"),
|
||||
"{output}"
|
||||
);
|
||||
assert!(!output.contains("op=\"upload_part\""), "{output}");
|
||||
assert_eq!(
|
||||
recorder.counter_value(super::METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, &[("stage", "lock_lost")]),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
#[serial]
|
||||
async fn stale_multipart_cleanup_lists_parts_only_when_size_is_needed() {
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use tracing::instrument::WithSubscriber as _;
|
||||
use tracing_subscriber::prelude::*;
|
||||
|
||||
struct PartListCounter {
|
||||
calls: Arc<AtomicUsize>,
|
||||
bucket: String,
|
||||
object: &'static str,
|
||||
}
|
||||
struct PartListFields<'a> {
|
||||
bucket: &'a str,
|
||||
object: &'a str,
|
||||
matching_bucket: bool,
|
||||
matching_object: bool,
|
||||
}
|
||||
impl tracing::field::Visit for PartListFields<'_> {
|
||||
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
|
||||
match field.name() {
|
||||
"bucket" => self.matching_bucket = value == self.bucket,
|
||||
"object" => self.matching_object = value == self.object,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
|
||||
let value = format!("{value:?}");
|
||||
match field.name() {
|
||||
"bucket" => self.matching_bucket = value == format!("{:?}", self.bucket),
|
||||
"object" => self.matching_object = value == format!("{:?}", self.object),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for PartListCounter {
|
||||
fn on_new_span(
|
||||
&self,
|
||||
attrs: &tracing::span::Attributes<'_>,
|
||||
_id: &tracing::span::Id,
|
||||
_ctx: tracing_subscriber::layer::Context<'_, S>,
|
||||
) {
|
||||
if attrs.metadata().name() != "list_object_parts" {
|
||||
return;
|
||||
}
|
||||
let mut fields = PartListFields {
|
||||
bucket: &self.bucket,
|
||||
object: self.object,
|
||||
matching_bucket: false,
|
||||
matching_object: false,
|
||||
};
|
||||
attrs.record(&mut fields);
|
||||
if fields.matching_bucket && fields.matching_object {
|
||||
self.calls.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let (_, ecstore) = setup_test_env().await;
|
||||
let bucket = format!("stale-list-parts-{}", Uuid::new_v4().simple());
|
||||
create_test_bucket(&ecstore, &bucket).await;
|
||||
set_abort_incomplete_lifecycle(&bucket, "logs/", 0).await;
|
||||
let now = OffsetDateTime::now_utc();
|
||||
for (object, age, size_filter, should_list) in [
|
||||
("logs/default-expired", time::Duration::hours(30), false, false),
|
||||
("logs/lifecycle-only", time::Duration::minutes(5), false, false),
|
||||
("logs/size-filter", time::Duration::minutes(5), true, true),
|
||||
] {
|
||||
if size_filter {
|
||||
set_abort_incomplete_lifecycle_with_size(&bucket, "logs/", 0, 0).await;
|
||||
}
|
||||
let upload = ecstore
|
||||
.new_multipart_upload(
|
||||
&bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
mod_time: Some(now - age),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("create multipart upload");
|
||||
if size_filter {
|
||||
let mut data = PutObjReader::from_vec(vec![1, 2, 3, 4]);
|
||||
ecstore
|
||||
.put_object_part(&bucket, object, &upload.upload_id, 1, &mut data, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("upload a part for size filtering");
|
||||
}
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let subscriber = tracing_subscriber::registry().with(PartListCounter {
|
||||
calls: Arc::clone(&calls),
|
||||
bucket: bucket.clone(),
|
||||
object,
|
||||
});
|
||||
cleanup_stale_multipart_uploads_once_at(ecstore.clone(), now, StdDuration::from_secs(24 * 60 * 60))
|
||||
.with_subscriber(subscriber)
|
||||
.await;
|
||||
let listed = calls.load(Ordering::Relaxed);
|
||||
assert_eq!(listed > 0, should_list, "unexpected ListParts calls for {object}: {listed}");
|
||||
let result = ecstore
|
||||
.get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await;
|
||||
assert!(
|
||||
matches!(result, Err(ref err) if is_err_invalid_upload_id(err)),
|
||||
"upload must be removed: {result:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_multipart_cleanup_rejects_incomplete_or_overflowing_part_sizes() {
|
||||
use crate::storage_api_contracts::multipart::{ListPartsInfo, PartInfo};
|
||||
|
||||
let part = |size, actual_size, etag: &str| PartInfo {
|
||||
size,
|
||||
actual_size,
|
||||
etag: Some(etag.to_owned()),
|
||||
..Default::default()
|
||||
};
|
||||
let mut parts = ListPartsInfo {
|
||||
parts: vec![part(6, 6, "valid")],
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
|
||||
parts.is_truncated = true;
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
parts.is_truncated = false;
|
||||
parts.parts = vec![part(0, 0, "")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
parts.parts = vec![part(1, -1, "valid")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
parts.parts = vec![part(8, 6, "valid")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
parts.parts = vec![part(4, 6, "valid")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
rustfs_utils::http::insert_str(&mut parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION, "test".to_owned());
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
|
||||
parts.parts = vec![part(8, 6, "valid")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
|
||||
rustfs_utils::http::remove_str(&mut parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION);
|
||||
parts
|
||||
.user_defined
|
||||
.insert("x-amz-server-side-encryption".to_owned(), "AES256".to_owned());
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
|
||||
parts.user_defined.clear();
|
||||
parts.parts = vec![part(usize::MAX, 0, "valid"), part(1, 1, "valid")];
|
||||
assert_eq!(super::stale_upload_size_from_parts(&parts), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn stale_multipart_cleanup_handles_unknown_size_without_false_abort() {
|
||||
let (_, ecstore) = setup_test_env().await;
|
||||
let bucket = format!("stale-unknown-size-{}", Uuid::new_v4().simple());
|
||||
let object = "logs/object";
|
||||
create_test_bucket(&ecstore, &bucket).await;
|
||||
let size_rule = r#"<Rule><ID>size</ID><Status>Enabled</Status><Filter><ObjectSizeLessThan>5</ObjectSizeLessThan></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule>"#;
|
||||
let independent_rule = r#"<Rule><ID>prefix</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule>"#;
|
||||
let lifecycle = |rules: &str| format!("<LifecycleConfiguration>{rules}</LifecycleConfiguration>").into_bytes();
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle(size_rule))
|
||||
.await
|
||||
.expect("install upper-size lifecycle rule");
|
||||
let metadata = HashMap::from([
|
||||
(super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
|
||||
(super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
|
||||
]);
|
||||
let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
|
||||
let set = &ecstore.pools[0].disk_set[0];
|
||||
let upload = ecstore
|
||||
.new_multipart_upload(&bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("create empty multipart upload");
|
||||
let decoded = base64_simd::URL_SAFE_NO_PAD
|
||||
.decode_to_vec(upload.upload_id.as_bytes())
|
||||
.expect("decode upload ID");
|
||||
let decoded = String::from_utf8(decoded).expect("upload ID must be UTF-8");
|
||||
let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
|
||||
.await
|
||||
.is_some(),
|
||||
"a known empty upload must match the upper-size rule"
|
||||
);
|
||||
let missing_upload = format!("{}x{}", Uuid::new_v4(), initiated.unix_timestamp_nanos());
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, &missing_upload, false)
|
||||
.await
|
||||
.is_none(),
|
||||
"an unreadable size must not match the upper-size rule"
|
||||
);
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle(&format!("{size_rule}{independent_rule}")))
|
||||
.await
|
||||
.expect("install size-independent lifecycle rule");
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, &missing_upload, false)
|
||||
.await
|
||||
.is_some(),
|
||||
"a size-independent rule remains eligible when size is unknown"
|
||||
);
|
||||
ecstore
|
||||
.abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("remove this test's remaining upload from the shared fixture");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn strict_multipart_lifecycle_size_skips_unavailable_participant_and_recovers() {
|
||||
let (_, ecstore) = setup_test_env().await;
|
||||
let bucket = format!("stale-strict-size-{}", Uuid::new_v4().simple());
|
||||
let object = "logs/object";
|
||||
create_test_bucket(&ecstore, &bucket).await;
|
||||
let rule = br#"<LifecycleConfiguration><Rule><ID>size</ID><Status>Enabled</Status><Filter><ObjectSizeLessThan>5</ObjectSizeLessThan></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>"#;
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule.to_vec())
|
||||
.await
|
||||
.expect("install upper-size rule");
|
||||
let upload = ecstore
|
||||
.new_multipart_upload(&bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("create empty upload");
|
||||
let decoded = base64_simd::URL_SAFE_NO_PAD
|
||||
.decode_to_vec(upload.upload_id.as_bytes())
|
||||
.expect("decode upload ID");
|
||||
let decoded = String::from_utf8(decoded).expect("upload ID is UTF-8");
|
||||
let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
|
||||
let metadata = HashMap::from([
|
||||
(super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
|
||||
(super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
|
||||
]);
|
||||
let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
|
||||
let set = &ecstore.pools[0].disk_set[0];
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
|
||||
.await
|
||||
.is_some()
|
||||
);
|
||||
let disk = set.disks.write().await[3].take();
|
||||
assert!(disk.is_some());
|
||||
let due = super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false).await;
|
||||
set.disks.write().await[3] = disk;
|
||||
assert!(due.is_none(), "unknown size must not match an upper-size rule");
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
|
||||
.await
|
||||
.is_some()
|
||||
);
|
||||
ecstore
|
||||
.abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("remove this test's remaining upload from the shared fixture");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn strict_multipart_lifecycle_does_not_treat_a_minority_part_as_empty() {
|
||||
let (paths, ecstore) = setup_test_env().await;
|
||||
let bucket = format!("stale-part-minority-{}", Uuid::new_v4().simple());
|
||||
let object = "logs/object";
|
||||
create_test_bucket(&ecstore, &bucket).await;
|
||||
let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
|
||||
let upload = ecstore
|
||||
.new_multipart_upload(
|
||||
&bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
mod_time: Some(initiated),
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("create upload");
|
||||
let mut body = PutObjReader::from_vec(vec![7; 6]);
|
||||
ecstore
|
||||
.put_object_part(&bucket, object, &upload.upload_id, 1, &mut body, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("write a known six-byte part");
|
||||
let decoded = base64_simd::URL_SAFE_NO_PAD
|
||||
.decode_to_vec(upload.upload_id.as_bytes())
|
||||
.expect("decode upload ID");
|
||||
let decoded = String::from_utf8(decoded).expect("upload ID is UTF-8");
|
||||
let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
|
||||
let upload_path = format!("{}/{upload_dir}", multipart_sha_dir(&bucket, object));
|
||||
let root = paths[0].join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path);
|
||||
let bytes = fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("read source metadata");
|
||||
let info = super::get_file_info(
|
||||
&bytes,
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_path,
|
||||
"",
|
||||
super::FileInfoOpts {
|
||||
data: false,
|
||||
include_free_versions: false,
|
||||
include_part_checksums: false,
|
||||
},
|
||||
)
|
||||
.expect("decode source metadata");
|
||||
let data_dir = info.data_dir.expect("upload data directory").to_string();
|
||||
let metadata = HashMap::from([
|
||||
(super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
|
||||
(super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
|
||||
]);
|
||||
let set = &ecstore.pools[0].disk_set[0];
|
||||
let rule = |filter: &str| {
|
||||
format!("<LifecycleConfiguration><Rule><ID>size</ID><Status>Enabled</Status><Filter>{filter}</Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>").into_bytes()
|
||||
};
|
||||
let filters = [
|
||||
"<ObjectSizeLessThan>10</ObjectSizeLessThan>",
|
||||
"<ObjectSizeGreaterThan>5</ObjectSizeGreaterThan>",
|
||||
];
|
||||
for filter in filters {
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
|
||||
.await
|
||||
.expect("install baseline size filter");
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
|
||||
.await
|
||||
.is_some(),
|
||||
"known six-byte part must satisfy {filter} before the fault"
|
||||
);
|
||||
}
|
||||
let mut saved = Vec::new();
|
||||
for path in paths.iter().take(3) {
|
||||
for file in ["part.1", "part.1.meta"] {
|
||||
let path = path
|
||||
.join(RUSTFS_META_MULTIPART_BUCKET)
|
||||
.join(&upload_path)
|
||||
.join(&data_dir)
|
||||
.join(file);
|
||||
let bytes = fs::read(&path).await.expect("snapshot original part");
|
||||
fs::remove_file(&path).await.expect("simulate a missing part on this disk");
|
||||
saved.push((path, bytes));
|
||||
}
|
||||
}
|
||||
for filter in filters {
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
|
||||
.await
|
||||
.expect("install size filter");
|
||||
super::metadata_boundary::get_lifecycle_config(&bucket)
|
||||
.await
|
||||
.expect("size filter must be readable during the fault");
|
||||
assert!(
|
||||
super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
|
||||
.await
|
||||
.is_none(),
|
||||
"a minority part must not satisfy {filter}"
|
||||
);
|
||||
}
|
||||
cleanup_stale_multipart_uploads_once_at(
|
||||
ecstore.clone(),
|
||||
OffsetDateTime::now_utc(),
|
||||
StdDuration::from_secs(7 * 24 * 60 * 60),
|
||||
)
|
||||
.await;
|
||||
ecstore
|
||||
.get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("unknown size must preserve the upload during cleanup");
|
||||
for (path, bytes) in saved {
|
||||
fs::write(path, bytes).await.expect("restore original part");
|
||||
}
|
||||
let size = super::stale_upload_current_size_with_opts(set, &metadata, upload_dir, false).await;
|
||||
assert_eq!(size, Some(6), "restored part must have its original logical size");
|
||||
for filter in filters {
|
||||
metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
|
||||
.await
|
||||
.expect("restore size filter");
|
||||
let config = super::metadata_boundary::get_lifecycle_config(&bucket)
|
||||
.await
|
||||
.expect("read lifecycle");
|
||||
let due = super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false).await;
|
||||
assert!(due.is_some(), "restored part must satisfy {filter}; size={size:?}, config={config:?}");
|
||||
}
|
||||
ecstore
|
||||
.abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("remove the restored test upload from the shared fixture");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn stale_multipart_cleanup_uses_default_expiry_without_lifecycle() {
|
||||
|
||||
@@ -63,14 +63,7 @@ pub(crate) async fn get_expiry_configs_for_options(
|
||||
bucket: &str,
|
||||
opts: &crate::object_api::ObjectOptions,
|
||||
) -> Result<LifecycleExpiryConfigs> {
|
||||
let bucket_incarnation_id = match opts
|
||||
.object_lock_config_snapshot
|
||||
.as_ref()
|
||||
.and_then(|snapshot| snapshot.validated_bucket_incarnation_for(api.id, bucket, opts.expected_bucket_incarnation_id))
|
||||
{
|
||||
Some(incarnation_id) => incarnation_id,
|
||||
None => metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?,
|
||||
};
|
||||
let bucket_incarnation_id = metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?;
|
||||
let metadata = get_authoritative_metadata(api, bucket, bucket_incarnation_id).await?;
|
||||
let table_bucket_enabled = metadata.table_bucket_enabled();
|
||||
|
||||
|
||||
@@ -564,12 +564,39 @@ pub(crate) async fn validate_bucket_reconstruction_in(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate against disk while retaining an already-held Object Lock fence.
|
||||
/// Resolve the authoritative bucket incarnation id for `opts`:
|
||||
/// 1. When namespace locking is enabled and `opts.object_lock_config_snapshot` is
|
||||
/// present, validate and return the snapshot's incarnation under its held
|
||||
/// lifecycle and metadata-transaction fences.
|
||||
/// 2. When namespace locking is enabled and `opts.bucket_lifecycle_lock_fence` is
|
||||
/// present and not lost, reuse the store's `BucketFenceRegistry` memo if a
|
||||
/// registered lifecycle read guard is currently held without lock loss.
|
||||
/// 3. Otherwise, acquire the bucket metadata transaction read lock (unless
|
||||
/// already held on `opts`) and read the incarnation from disk.
|
||||
pub(crate) async fn get_bucket_incarnation_id_for_options_in(
|
||||
ctx: &crate::runtime::instance::InstanceContext,
|
||||
bucket: &str,
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<Uuid> {
|
||||
if !ctx.lock_manager().is_disabled() {
|
||||
if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
|
||||
let store = object_store_in(ctx).await?;
|
||||
return snapshot
|
||||
.validated_bucket_incarnation_for(store.id, bucket, opts.expected_bucket_incarnation_id)
|
||||
.ok_or_else(|| {
|
||||
Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
|
||||
});
|
||||
}
|
||||
if opts
|
||||
.bucket_lifecycle_lock_fence
|
||||
.as_ref()
|
||||
.is_some_and(|fence| !fence.is_lock_lost())
|
||||
&& let Some(store) = object_store_if_initialized_in(ctx).await
|
||||
&& let Some(incarnation) = store.bucket_fence_registry.validated_incarnation(bucket)
|
||||
{
|
||||
return Ok(incarnation);
|
||||
}
|
||||
}
|
||||
let sys = bucket_metadata_sys_of(ctx)?;
|
||||
let sys = sys.read().await.clone();
|
||||
let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
|
||||
|
||||
@@ -488,6 +488,18 @@ where
|
||||
Ok(data)
|
||||
}
|
||||
|
||||
/// Read config bytes and metadata while preserving an existing empty payload.
|
||||
pub(crate) async fn read_config_preserve_empty_with_metadata_opts<S>(
|
||||
api: Arc<S>,
|
||||
file: &str,
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<(Vec<u8>, ObjectInfo)>
|
||||
where
|
||||
S: EcstoreObjectIO,
|
||||
{
|
||||
read_config_with_metadata_inner(api, file, opts, true, None).await
|
||||
}
|
||||
|
||||
pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
|
||||
where
|
||||
S: ObjectIO<
|
||||
@@ -2605,6 +2617,15 @@ where
|
||||
save_server_config_snapshot(api, cfg, &snapshot).await.map(|_| ())
|
||||
}
|
||||
|
||||
/// Read config and generation from the same object without taking the transaction write lock.
|
||||
pub async fn read_server_config_with_generation(api: Arc<ECStore>) -> Result<(Config, Option<Uuid>)> {
|
||||
match read_config_with_metadata_inner(api.clone(), &server_config_path(), &ObjectOptions::default(), false, None).await {
|
||||
Ok((raw, info)) => Ok((decode_persisted_server_config(&raw)?.merge(), info.data_dir.filter(|id| !id.is_nil()))),
|
||||
Err(Error::ConfigNotFound) => read_config_without_migrate(api).await.map(|config| (config, None)),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
|
||||
/// One serialized read of the persisted server config used as the seed and
|
||||
/// compare-and-swap fence for an admin update.
|
||||
#[derive(Debug)]
|
||||
|
||||
@@ -28300,6 +28300,167 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// One version commits, then every in-place attempt to copy the next one
|
||||
/// times out on the target namespace lock. The entry is deferred, not
|
||||
/// failed, with both source versions retained; the resumed entry finds the
|
||||
/// committed version and moves both with their exact bytes.
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_defers_target_lookup_lock_timeout_and_retries_without_losing_source() {
|
||||
use tokio::io::AsyncReadExt as _;
|
||||
|
||||
let fixture = NullVersionDecommission::new(NullVersionTop::NewerVersion).await;
|
||||
let object = fixture.object;
|
||||
let injected = Arc::new(AtomicUsize::new(0));
|
||||
let attempted = Arc::new(AtomicUsize::new(0));
|
||||
let fault = DecommissionTestFaultGuard::install_with_error(
|
||||
Arc::new({
|
||||
let injected = Arc::clone(&injected);
|
||||
let attempted = Arc::clone(&attempted);
|
||||
move |stage, _, faulted, _, success| {
|
||||
let target = stage == "before_target_commit" && faulted == object && success;
|
||||
let should_inject = target && attempted.fetch_add(1, Ordering::SeqCst) > 0;
|
||||
if should_inject {
|
||||
injected.fetch_add(1, Ordering::SeqCst);
|
||||
}
|
||||
should_inject
|
||||
}
|
||||
}),
|
||||
Some(Arc::new(|_, bucket, object| {
|
||||
data_movement::data_movement_stage_error_for_test(
|
||||
"decommission_object",
|
||||
"prepare_put_object",
|
||||
bucket,
|
||||
object,
|
||||
Error::Lock(rustfs_lock::LockError::timeout(
|
||||
format!("{bucket}/{object}@latest"),
|
||||
std::time::Duration::from_secs(5),
|
||||
)),
|
||||
)
|
||||
})),
|
||||
);
|
||||
let (outcome, _) = fixture.run_entry().await;
|
||||
assert_eq!(injected.load(Ordering::SeqCst), DECOMMISSION_VERSION_COPY_ATTEMPTS);
|
||||
fixture.assert_source_retained().await;
|
||||
assert_eq!(
|
||||
fixture
|
||||
.target_versions()
|
||||
.await
|
||||
.expect("the first version was copied")
|
||||
.versions
|
||||
.len(),
|
||||
1,
|
||||
"one version must commit before the next version times out"
|
||||
);
|
||||
assert!(
|
||||
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
|
||||
"{outcome:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
fixture.store.pool_meta.read().await.pools[0]
|
||||
.decommission
|
||||
.as_ref()
|
||||
.expect("active decommission")
|
||||
.items_decommission_failed,
|
||||
0,
|
||||
"a deferred entry must not be counted as a permanent failure"
|
||||
);
|
||||
drop(fault);
|
||||
let (outcome, _) = fixture.run_entry().await;
|
||||
assert!(matches!(outcome, DecommissionEntryOutcome::Complete), "{outcome:?}");
|
||||
fixture.assert_moved().await;
|
||||
let mut reader = fixture.store.pools[1]
|
||||
.get_object_reader(
|
||||
fixture.bucket,
|
||||
fixture.object,
|
||||
None,
|
||||
HeaderMap::new(),
|
||||
&ObjectOptions {
|
||||
version_id: Some(fixture.seeded.versions[0].version_id.expect("newer version").to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("the newer version must remain readable");
|
||||
let mut actual = Vec::new();
|
||||
reader.stream.read_to_end(&mut actual).await.expect("read the newer version");
|
||||
assert_eq!(actual, b"written after versioning was enabled");
|
||||
}
|
||||
|
||||
/// A namespace lock still held on the target key, as by a copy whose node
|
||||
/// was killed mid-write, makes target selection fail with a typed
|
||||
/// object-lock timeout. A real held lock, not an injected error, must defer
|
||||
/// the entry rather than fail it; once the lock is released selection
|
||||
/// recovers and the resumed entry moves both versions.
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_target_selection_returns_timeout_for_a_held_object_lock() {
|
||||
temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("1"))], async {
|
||||
let fixture = NullVersionDecommission::new(NullVersionTop::NewerVersion).await;
|
||||
let target_set = fixture.store.pools[1].get_disks_by_key(fixture.object);
|
||||
let lock = target_set
|
||||
.new_ns_lock(fixture.bucket, fixture.object)
|
||||
.await
|
||||
.expect("target lock");
|
||||
let guard = lock
|
||||
.get_write_lock(std::time::Duration::from_secs(30))
|
||||
.await
|
||||
.expect("hold target lock");
|
||||
let opts = ObjectOptions {
|
||||
data_movement: true,
|
||||
src_pool_idx: 0,
|
||||
version_id: Some(uuid::Uuid::nil().to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let select = || {
|
||||
fixture
|
||||
.store
|
||||
.select_data_movement_pool_idx(fixture.bucket, fixture.object, 1, &opts, false)
|
||||
};
|
||||
let result = tokio::time::timeout(get_lock_acquire_timeout() + std::time::Duration::from_secs(15), select())
|
||||
.await
|
||||
.expect("target selection must terminate");
|
||||
match result {
|
||||
Err(Error::Lock(rustfs_lock::LockError::Timeout { resource, .. })) => {
|
||||
assert!(resource.contains(fixture.bucket) && resource.contains(fixture.object), "{resource}");
|
||||
}
|
||||
other => panic!("expected an object lock timeout, got {other:?}"),
|
||||
}
|
||||
|
||||
let (outcome, _) = fixture.run_entry().await;
|
||||
assert!(
|
||||
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
|
||||
"{outcome:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
fixture.store.pool_meta.read().await.pools[0]
|
||||
.decommission
|
||||
.as_ref()
|
||||
.expect("active decommission")
|
||||
.items_decommission_failed,
|
||||
0,
|
||||
"a held target lock must not be counted as a permanent failure"
|
||||
);
|
||||
fixture.assert_source_retained().await;
|
||||
assert!(
|
||||
fixture.target_versions().await.is_none(),
|
||||
"no version may commit while the target lock is held"
|
||||
);
|
||||
|
||||
drop(guard);
|
||||
assert_eq!(
|
||||
select()
|
||||
.await
|
||||
.expect("target selection must recover once the lock is released"),
|
||||
1
|
||||
);
|
||||
let (outcome, _) = fixture.run_entry().await;
|
||||
assert!(matches!(outcome, DecommissionEntryOutcome::Complete), "{outcome:?}");
|
||||
fixture.assert_moved().await;
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_object_attempt_metric_keeps_the_peak() {
|
||||
let mut attempts = 0;
|
||||
|
||||
@@ -1493,6 +1493,48 @@ pub async fn load_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<
|
||||
project_data_usage_from_backend_cached(store, DataUsageInfo::clone).await
|
||||
}
|
||||
|
||||
/// Read one bucket's persisted usage without cloning the complete snapshot.
|
||||
pub async fn load_bucket_usage_from_backend_cached(
|
||||
store: Arc<ECStore>,
|
||||
bucket: &str,
|
||||
incarnation: uuid::Uuid,
|
||||
) -> Result<Option<u64>, Error> {
|
||||
project_data_usage_from_backend_cached(store, |info| certified_bucket_usage(info, bucket, incarnation)).await
|
||||
}
|
||||
|
||||
fn certified_bucket_usage(info: &DataUsageInfo, bucket: &str, incarnation: uuid::Uuid) -> Option<u64> {
|
||||
if incarnation.is_nil()
|
||||
|| !info.is_complete_bucket_usage_snapshot()
|
||||
|| info.usage_snapshot_observational
|
||||
|| info.scanner_cycle.is_none()
|
||||
|| info.scanner_epoch.is_none()
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let pending_count = u64::try_from(info.usage_snapshot_pending_buckets.len()).ok()?;
|
||||
if pending_count != info.usage_snapshot_pending_bucket_count {
|
||||
return None;
|
||||
}
|
||||
match info.usage_snapshot_converged {
|
||||
Some(true) if pending_count == 0 => {}
|
||||
Some(false)
|
||||
if pending_count > 0
|
||||
&& info.usage_snapshot_pending_buckets.windows(2).all(|pair| pair[0] < pair[1])
|
||||
&& info
|
||||
.usage_snapshot_pending_buckets
|
||||
.binary_search_by(|pending| pending.as_str().cmp(bucket))
|
||||
.is_err() => {}
|
||||
_ => return None,
|
||||
}
|
||||
let usage = info.buckets_usage.get(bucket)?;
|
||||
let certified = usage
|
||||
.bucket_incarnation
|
||||
.as_deref()
|
||||
.and_then(|id| uuid::Uuid::parse_str(id).ok())
|
||||
.filter(|id| !id.is_nil())?;
|
||||
(certified == incarnation).then_some(usage.size)
|
||||
}
|
||||
|
||||
// Project under the cache guard so scalar readers never copy the snapshot's
|
||||
// bucket maps. The owned API shares the same TTL, refresh and invalidation path.
|
||||
async fn project_data_usage_from_backend_cached<T>(
|
||||
|
||||
@@ -129,6 +129,41 @@ fn projected_publication_moves_bucket_maps_into_cache() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn certified_bucket_usage_rejects_unproven_and_pending_data() {
|
||||
let id = uuid::Uuid::from_u128(7);
|
||||
let mut info = data_usage_info_for_test("known", 1, 42, SystemTime::UNIX_EPOCH);
|
||||
info.usage_snapshot_converged = Some(true);
|
||||
info.scanner_cycle = Some(1);
|
||||
info.scanner_epoch = Some(1);
|
||||
info.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some(id.to_string());
|
||||
assert_eq!(certified_bucket_usage(&info, "known", id), Some(42));
|
||||
assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::from_u128(8)), None);
|
||||
assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::nil()), None);
|
||||
let mut legacy = info.clone();
|
||||
legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = None;
|
||||
assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
|
||||
legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some("corrupt".into());
|
||||
assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
|
||||
let mut pending = info.clone();
|
||||
pending.usage_snapshot_converged = Some(false);
|
||||
pending.usage_snapshot_pending_buckets = vec!["other".into()];
|
||||
pending.usage_snapshot_pending_bucket_count = 1;
|
||||
assert_eq!(certified_bucket_usage(&pending, "known", id), Some(42));
|
||||
pending.usage_snapshot_pending_buckets = vec!["known".into()];
|
||||
assert_eq!(certified_bucket_usage(&pending, "known", id), None);
|
||||
pending.usage_snapshot_pending_bucket_count = 0;
|
||||
assert_eq!(certified_bucket_usage(&pending, "known", id), None);
|
||||
pending.usage_snapshot_pending_buckets = vec!["z".into(), "a".into()];
|
||||
pending.usage_snapshot_pending_bucket_count = 2;
|
||||
assert_eq!(certified_bucket_usage(&pending, "known", id), None);
|
||||
let mut observational = info.clone();
|
||||
observational.usage_snapshot_observational = true;
|
||||
assert_eq!(certified_bucket_usage(&observational, "known", id), None);
|
||||
info.usage_snapshot_converged = None;
|
||||
assert_eq!(certified_bucket_usage(&info, "known", id), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
|
||||
@@ -140,6 +175,13 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
|
||||
info.buckets_usage.insert("empty".to_string(), BucketUsageInfo::default());
|
||||
info.bucket_sizes.insert("empty".to_string(), 0);
|
||||
info.buckets_count = 2;
|
||||
info.usage_snapshot_converged = Some(true);
|
||||
info.scanner_cycle = Some(1);
|
||||
info.scanner_epoch = Some(1);
|
||||
let incarnation = uuid::Uuid::from_u128(7);
|
||||
for usage in info.buckets_usage.values_mut() {
|
||||
usage.bucket_incarnation = Some(incarnation.to_string());
|
||||
}
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
@@ -151,11 +193,29 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
|
||||
|
||||
for (bucket, expected) in [("known", Some(42)), ("empty", Some(0)), ("missing", None)] {
|
||||
assert_eq!(persisted_bucket_usage(store.clone(), bucket).await, expected);
|
||||
assert_eq!(
|
||||
load_bucket_usage_from_backend_cached(store.clone(), bucket, incarnation)
|
||||
.await
|
||||
.expect("project bucket usage"),
|
||||
expected
|
||||
);
|
||||
let owned = load_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("owned cached snapshot");
|
||||
assert_eq!(expected, owned.buckets_usage.get(bucket).map(|usage| usage.size));
|
||||
}
|
||||
assert_eq!(
|
||||
load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::from_u128(8))
|
||||
.await
|
||||
.unwrap(),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::nil())
|
||||
.await
|
||||
.unwrap(),
|
||||
None
|
||||
);
|
||||
info.buckets_usage.get_mut("known").expect("known bucket").size = 84;
|
||||
info.bucket_sizes.insert("known".to_string(), 84);
|
||||
save_config(
|
||||
|
||||
@@ -8450,6 +8450,23 @@ impl LocalDisk {
|
||||
return Err(std::io::Error::other("metadata read worker cancelled"));
|
||||
}
|
||||
let _io_permit = io_permit;
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
{
|
||||
let opened = root.open_readonly(&path);
|
||||
if opened.is_ok() {
|
||||
os::prepared_publication_test_hooks::run(
|
||||
os::prepared_publication_test_hooks::Stage::MetadataReadAfterOpen,
|
||||
&path,
|
||||
);
|
||||
}
|
||||
let result = read_fn(opened);
|
||||
os::prepared_publication_test_hooks::run(
|
||||
os::prepared_publication_test_hooks::Stage::MetadataReadAfterRead,
|
||||
&path,
|
||||
);
|
||||
Ok(result)
|
||||
}
|
||||
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
|
||||
Ok(read_fn(root.open_readonly(&path)))
|
||||
})
|
||||
.await
|
||||
@@ -11309,34 +11326,64 @@ impl LocalDisk {
|
||||
#[cfg(unix)]
|
||||
pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize, usize)>) -> Vec<Result<Bytes>> {
|
||||
let n = requests.len();
|
||||
let metrics = rustfs_io_metrics::LocalBatchReadMetrics::capture();
|
||||
let submitted = metrics.as_ref().map(|_| std::time::Instant::now());
|
||||
tokio::task::spawn_blocking(move || {
|
||||
use std::os::unix::fs::FileExt;
|
||||
use rustfs_io_metrics::LocalBatchReadStage;
|
||||
use std::os::unix::fs::{FileExt, OpenOptionsExt};
|
||||
|
||||
let execution_started = metrics.as_ref().map(|_| std::time::Instant::now());
|
||||
if let (Some(metrics), Some(submitted)) = (&metrics, submitted) {
|
||||
metrics.record_stage(LocalBatchReadStage::QueueWait, true, submitted.elapsed());
|
||||
metrics.record_batch(n);
|
||||
}
|
||||
let mut results = Vec::with_capacity(n);
|
||||
for (file_path, offset, length) in requests {
|
||||
let mut bytes_read = 0;
|
||||
let r = (|| -> Result<Bytes> {
|
||||
let meta = std::fs::metadata(&file_path).map_err(DiskError::from)?;
|
||||
let end = offset.checked_add(length).ok_or(DiskError::FileCorrupt)?;
|
||||
if meta.len() < u64::try_from(end).unwrap_or(u64::MAX) {
|
||||
let end = u64::try_from(end).map_err(|_| DiskError::FileCorrupt)?;
|
||||
// Bind validation and reads to one descriptor. Nonblocking open
|
||||
// lets corrupt non-regular entries fail without waiting on a FIFO.
|
||||
let file = measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Open, || {
|
||||
std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.custom_flags(libc::O_NONBLOCK)
|
||||
.open(&file_path)
|
||||
.map_err(DiskError::from)
|
||||
})?;
|
||||
let meta = measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Metadata, || {
|
||||
file.metadata().map_err(DiskError::from)
|
||||
})?;
|
||||
if !meta.is_file() || meta.len() < end {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
|
||||
let file = std::fs::File::open(&file_path).map_err(DiskError::from)?;
|
||||
let mut buf = vec![0u8; length];
|
||||
let mut total = 0usize;
|
||||
while total < length {
|
||||
let nbytes = file
|
||||
.read_at(&mut buf[total..], u64::try_from(offset + total).unwrap_or(u64::MAX))
|
||||
.map_err(DiskError::from)?;
|
||||
if nbytes == 0 {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
total += nbytes;
|
||||
if length > 0 {
|
||||
measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Read, || {
|
||||
while bytes_read < length {
|
||||
let nbytes = file
|
||||
.read_at(&mut buf[bytes_read..], u64::try_from(offset + bytes_read).unwrap_or(u64::MAX))
|
||||
.map_err(DiskError::from)?;
|
||||
if nbytes == 0 {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
bytes_read += nbytes;
|
||||
}
|
||||
Ok(())
|
||||
})?;
|
||||
}
|
||||
Ok(Bytes::from(buf))
|
||||
})();
|
||||
if let Some(metrics) = &metrics {
|
||||
metrics.record_shard(r.is_ok(), length, bytes_read);
|
||||
}
|
||||
results.push(r);
|
||||
}
|
||||
if let (Some(metrics), Some(started)) = (&metrics, execution_started) {
|
||||
metrics.record_stage(LocalBatchReadStage::Execution, results.iter().all(Result::is_ok), started.elapsed());
|
||||
}
|
||||
results
|
||||
})
|
||||
.await
|
||||
@@ -11346,6 +11393,20 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn measure_local_batch_stage<T>(
|
||||
metrics: Option<&rustfs_io_metrics::LocalBatchReadMetrics>,
|
||||
stage: rustfs_io_metrics::LocalBatchReadStage,
|
||||
operation: impl FnOnce() -> Result<T>,
|
||||
) -> Result<T> {
|
||||
let started = metrics.map(|_| std::time::Instant::now());
|
||||
let result = operation();
|
||||
if let (Some(metrics), Some(started)) = (metrics, started) {
|
||||
metrics.record_stage(stage, result.is_ok(), started.elapsed());
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl DiskAPI for LocalDisk {
|
||||
async fn supports_guarded_delete_versions(&self) -> Result<()> {
|
||||
@@ -30957,6 +31018,173 @@ mod test {
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn local_batch_read_metrics_cover_success_and_unattempted_failure_stages() {
|
||||
use crate::test_metrics::CapturingRecorder;
|
||||
|
||||
let dir = tempfile::tempdir().expect("batch metrics directory");
|
||||
let good = dir.path().join("good.bin");
|
||||
std::fs::write(&good, b"good data").expect("write batch metrics payload");
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.expect("batch metrics runtime");
|
||||
let recorder = CapturingRecorder::default();
|
||||
let previous = rustfs_io_metrics::get_stage_metrics_enabled();
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
|
||||
let results = metrics::with_local_recorder(&recorder, || {
|
||||
runtime.block_on(batch_shard_pread(vec![
|
||||
(good.clone(), 0, 9),
|
||||
(dir.path().join("missing.bin"), 0, 4),
|
||||
(good.clone(), 0, 1024),
|
||||
(good, 0, 0),
|
||||
]))
|
||||
});
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(previous);
|
||||
|
||||
assert_eq!(results[0].as_ref().expect("valid shard read").as_ref(), b"good data");
|
||||
assert!(results[1].is_err(), "missing shard must remain an error");
|
||||
assert!(matches!(results[2], Err(DiskError::FileCorrupt)), "range failure must preserve its error");
|
||||
assert!(results[3].as_ref().expect("empty range succeeds").is_empty());
|
||||
let counter = "rustfs_io_get_object_local_batch_shards_total";
|
||||
assert_eq!(recorder.counter_value(counter, &[("outcome", "success")]), 2);
|
||||
assert_eq!(recorder.counter_value(counter, &[("outcome", "error")]), 2);
|
||||
assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 9);
|
||||
let stages = "rustfs_io_get_object_local_batch_stage_seconds";
|
||||
for (stage, success, errors) in [("open", 3, 1), ("metadata", 3, 0), ("read", 1, 0), ("execution", 0, 1)] {
|
||||
assert_eq!(
|
||||
recorder
|
||||
.histogram_values(stages, &[("stage", stage), ("outcome", "success")])
|
||||
.len(),
|
||||
success
|
||||
);
|
||||
assert_eq!(
|
||||
recorder
|
||||
.histogram_values(stages, &[("stage", stage), ("outcome", "error")])
|
||||
.len(),
|
||||
errors
|
||||
);
|
||||
}
|
||||
assert_eq!(recorder.histogram_values(stages, &[("stage", "queue_wait")]).len(), 1);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn local_batch_read_metrics_survive_caller_cancellation() {
|
||||
use crate::test_metrics::CapturingRecorder;
|
||||
|
||||
let dir = tempfile::tempdir().expect("cancelled batch directory");
|
||||
let path = dir.path().join("shard");
|
||||
std::fs::write(&path, b"data").expect("write cancellation fixture");
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.max_blocking_threads(1)
|
||||
.enable_all()
|
||||
.build()
|
||||
.expect("single blocking-worker runtime");
|
||||
let recorder = CapturingRecorder::default();
|
||||
let previous = rustfs_io_metrics::get_stage_metrics_enabled();
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
runtime.block_on(async {
|
||||
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
|
||||
let (release_tx, release_rx) = std::sync::mpsc::channel();
|
||||
let blocker = tokio::task::spawn_blocking(move || {
|
||||
started_tx.send(()).expect("signal occupied blocking worker");
|
||||
release_rx.recv().expect("release occupied blocking worker");
|
||||
});
|
||||
started_rx.await.expect("blocking worker has started");
|
||||
let mut batch = Box::pin(batch_shard_pread(vec![(path, 0, 4)]));
|
||||
let pending = futures::FutureExt::now_or_never(&mut batch).is_none();
|
||||
drop(batch);
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(false);
|
||||
release_tx.send(()).expect("release worker after dropping caller");
|
||||
blocker.await.expect("blocking worker finishes");
|
||||
assert!(pending, "batch must have been submitted while the worker was occupied");
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
while recorder
|
||||
.histogram_values("rustfs_io_get_object_local_batch_stage_seconds", &[("stage", "execution")])
|
||||
.is_empty()
|
||||
{
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("detached batch must still record completion");
|
||||
});
|
||||
});
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(previous);
|
||||
assert_eq!(
|
||||
recorder.counter_value("rustfs_io_get_object_local_batch_shards_total", &[("outcome", "success")]),
|
||||
1
|
||||
);
|
||||
assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 4);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn test_batch_shard_pread_rejects_nonregular_files_without_blocking() {
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
|
||||
let dir = tempfile::tempdir().expect("nonregular shard fixture");
|
||||
let fifo = dir.path().join("fifo");
|
||||
assert!(
|
||||
std::process::Command::new("mkfifo")
|
||||
.arg(&fifo)
|
||||
.status()
|
||||
.expect("run POSIX mkfifo for the shard fixture")
|
||||
.success()
|
||||
);
|
||||
let good = dir.path().join("good");
|
||||
std::fs::write(&good, b"data").expect("write regular shard");
|
||||
let mut batch = Box::pin(batch_shard_pread(vec![
|
||||
(dir.path().to_owned(), 0, 0),
|
||||
(fifo.clone(), 0, 0),
|
||||
(fifo.clone(), 0, 4),
|
||||
(good, 0, 4),
|
||||
]));
|
||||
let results = match tokio::time::timeout(Duration::from_secs(2), &mut batch).await {
|
||||
Ok(results) => results,
|
||||
Err(_) => {
|
||||
// Release a regressed blocking open before failing, so runtime
|
||||
// shutdown does not leave the test waiting on its worker.
|
||||
let _release = std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.custom_flags(libc::O_NONBLOCK)
|
||||
.open(&fifo)
|
||||
.expect("release blocked FIFO reader");
|
||||
batch.await;
|
||||
panic!("a nonregular shard must not block its batch");
|
||||
}
|
||||
};
|
||||
for result in &results[..3] {
|
||||
assert!(matches!(result, Err(DiskError::FileCorrupt)), "nonregular shard: {result:?}");
|
||||
}
|
||||
assert_eq!(results[3].as_ref().expect("later regular shard remains readable").as_ref(), b"data");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn test_batch_shard_pread_validates_descriptor_ranges() {
|
||||
let dir = tempfile::tempdir().expect("range validation fixture");
|
||||
let path = dir.path().join("shard");
|
||||
std::fs::write(&path, b"01234567").expect("write range fixture");
|
||||
let results = batch_shard_pread(vec![
|
||||
(path.clone(), 3, 5),
|
||||
(path.clone(), 8, 0),
|
||||
(path.clone(), 8, 1),
|
||||
(path.clone(), 9, 0),
|
||||
(path, usize::MAX, 1),
|
||||
])
|
||||
.await;
|
||||
assert_eq!(results[0].as_ref().expect("range ending at EOF").as_ref(), b"34567");
|
||||
assert!(results[1].as_ref().expect("empty range at EOF").is_empty());
|
||||
for result in &results[2..] {
|
||||
assert!(matches!(result, Err(DiskError::FileCorrupt)), "invalid range: {result:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn test_batch_shard_pread_basic() {
|
||||
|
||||
@@ -333,6 +333,8 @@ pub(crate) mod prepared_publication_test_hooks {
|
||||
PreparedRename,
|
||||
Rename,
|
||||
Remove,
|
||||
MetadataReadAfterOpen,
|
||||
MetadataReadAfterRead,
|
||||
#[cfg(test)]
|
||||
Rollback,
|
||||
#[cfg(test)]
|
||||
@@ -457,6 +459,8 @@ pub enum LocalPublicationStage {
|
||||
PreparedRename,
|
||||
Rename,
|
||||
Remove,
|
||||
MetadataReadAfterOpen,
|
||||
MetadataReadAfterRead,
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "test-util", not(windows)))]
|
||||
@@ -469,6 +473,8 @@ impl LocalPublicationPause {
|
||||
LocalPublicationStage::PreparedRename => prepared_publication_test_hooks::Stage::PreparedRename,
|
||||
LocalPublicationStage::Rename => prepared_publication_test_hooks::Stage::Rename,
|
||||
LocalPublicationStage::Remove => prepared_publication_test_hooks::Stage::Remove,
|
||||
LocalPublicationStage::MetadataReadAfterOpen => prepared_publication_test_hooks::Stage::MetadataReadAfterOpen,
|
||||
LocalPublicationStage::MetadataReadAfterRead => prepared_publication_test_hooks::Stage::MetadataReadAfterRead,
|
||||
};
|
||||
let (entered_tx, entered) = oneshot::channel();
|
||||
let (release, release_rx) = std::sync::mpsc::channel::<()>();
|
||||
|
||||
@@ -959,6 +959,10 @@ pub struct ObjectOptions {
|
||||
/// Persisted bucket incarnation observed before authorization.
|
||||
pub expected_bucket_incarnation_id: Option<Uuid>,
|
||||
pub no_lock: bool,
|
||||
/// Require complete participant observations for internal multipart size decisions.
|
||||
/// This transient flag is never derived from client input or stored on disk.
|
||||
#[doc(hidden)]
|
||||
pub strict_multipart_part_sizes: bool,
|
||||
/// A genuine target write guard; loss fences alone do not authorize a borrowed write.
|
||||
#[doc(hidden)]
|
||||
pub write_lock_context: Option<WriteLockContext>,
|
||||
|
||||
@@ -1782,6 +1782,285 @@ mod tests {
|
||||
fixture.assert_moved().await;
|
||||
}
|
||||
|
||||
/// Each drive's copy of `version_id` of `object` on `pool`.
|
||||
#[cfg(feature = "test-util")]
|
||||
async fn drive_copies(store: &ECStore, pool: usize, bucket: &str, object: &str, version_id: uuid::Uuid) -> Vec<FileInfo> {
|
||||
use crate::disk::DiskAPI as _;
|
||||
let set = store.pools[pool].get_disks_by_key(object);
|
||||
let mut copies = Vec::new();
|
||||
for disk in set.disks.read().await.iter().flatten() {
|
||||
let path = disk.path().join(bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE);
|
||||
let Ok(bytes) = tokio::fs::read(&path).await else {
|
||||
continue;
|
||||
};
|
||||
let meta = FileMeta::load(&bytes).expect("drive metadata should decode");
|
||||
if meta.find_version(Some(version_id)).is_ok() {
|
||||
copies.push(
|
||||
meta.into_fileinfo(bucket, object, &version_id.to_string(), false, false, false)
|
||||
.expect("drive copy should decode"),
|
||||
);
|
||||
}
|
||||
}
|
||||
copies
|
||||
}
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
fn drive_space(total: u64, used: u64) -> crate::disk::DiskInfo {
|
||||
crate::disk::DiskInfo {
|
||||
total,
|
||||
used,
|
||||
free: total - used,
|
||||
free_inodes: 1 << 20,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
const DANGLING_COPY_REBALANCE_ID: &str = "rebalance-dangling-target-copy";
|
||||
|
||||
/// A delete marker over one object version on pool 0 of three four-drive
|
||||
/// pools, plus what an earlier attempt that lost write quorum left on one
|
||||
/// drive of pool 1: both version ids, with older modification times and
|
||||
/// different bytes. Object reads report such copies as absent. Pool 2
|
||||
/// reports far more free space than pool 1, so a free-space choice picks
|
||||
/// pool 2 (the kill-resume e2e case ended with the marker on two pools).
|
||||
#[cfg(feature = "test-util")]
|
||||
struct DanglingTargetCopy {
|
||||
_temp_dirs: Vec<tempfile::TempDir>,
|
||||
store: Arc<ECStore>,
|
||||
_space: crate::store::PoolSpaceForTest,
|
||||
bucket: &'static str,
|
||||
object: &'static str,
|
||||
incarnation: uuid::Uuid,
|
||||
source_set: Arc<SetDisks>,
|
||||
marker: FileInfo,
|
||||
version: FileInfo,
|
||||
payload: Vec<u8>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
impl DanglingTargetCopy {
|
||||
async fn new(holder_space: crate::disk::DiskInfo) -> Self {
|
||||
let (temp_dirs, store, _peer) =
|
||||
crate::services::rebalance::test_three_pool_stores_with_four_disk_sets_with_isolated_node_contexts(None).await;
|
||||
let space = crate::store::PoolSpaceForTest::install(store.id, &[(1, holder_space), (2, drive_space(1 << 50, 0))]);
|
||||
let bucket = "rebalance-dangling-target";
|
||||
let object = "versioned/06.bin";
|
||||
Box::pin(store.make_bucket(bucket, &MakeBucketOptions::default()))
|
||||
.await
|
||||
.expect("create bucket");
|
||||
let incarnation = store.bucket_incarnation_id(bucket).await.expect("bucket incarnation");
|
||||
let mut rebalance_meta = active_rebalance_meta(DANGLING_COPY_REBALANCE_ID);
|
||||
rebalance_meta.pool_stats.push(RebalanceStats::default());
|
||||
rebalance_meta
|
||||
.save(Arc::clone(&store.pools[0]))
|
||||
.await
|
||||
.expect("persist the active rebalance run");
|
||||
*store.rebalance_meta.write().await = Some(rebalance_meta);
|
||||
let versioned = ObjectOptions {
|
||||
versioned: true,
|
||||
expected_bucket_incarnation_id: Some(incarnation),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let payload = vec![0x5a; 2048];
|
||||
let source_set = store.pools[0].get_disks_by_key(object);
|
||||
Box::pin(source_set.put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &versioned))
|
||||
.await
|
||||
.expect("seed the object version");
|
||||
let marker = Box::pin(source_set.delete_object(bucket, object, versioned.clone()))
|
||||
.await
|
||||
.expect("seed the delete marker");
|
||||
assert!(marker.delete_marker);
|
||||
let seeded = source_set
|
||||
.load_file_info_versions_exact(bucket, object)
|
||||
.await
|
||||
.expect("read seeded versions")
|
||||
.expect("seeded versions must exist");
|
||||
let [marker, version] = seeded.versions.as_slice() else {
|
||||
panic!("expected a delete marker over one version: {seeded:?}");
|
||||
};
|
||||
let (marker, version) = (marker.clone(), version.clone());
|
||||
let marker_id = marker.version_id.expect("the marker has a version id");
|
||||
let version_id = version.version_id.expect("the object version has a version id");
|
||||
let earlier = |time: Option<OffsetDateTime>| time.map(|time| time - time::Duration::hours(1));
|
||||
|
||||
let holder = store.pools[1].get_disks_by_key(object);
|
||||
let mut marker_opts =
|
||||
super::super::migration::rebalance_delete_marker_opts(&marker, Some(marker_id.to_string()), 0, Some(incarnation));
|
||||
marker_opts.mod_time = earlier(marker.mod_time);
|
||||
Box::pin(holder.delete_object(bucket, object, marker_opts))
|
||||
.await
|
||||
.expect("publish the earlier marker copy");
|
||||
Box::pin(holder.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(vec![0x11; 4096]),
|
||||
&ObjectOptions {
|
||||
version_id: Some(version_id.to_string()),
|
||||
mod_time: earlier(version.mod_time),
|
||||
..versioned
|
||||
},
|
||||
))
|
||||
.await
|
||||
.expect("publish the earlier object version copy");
|
||||
{
|
||||
use crate::disk::DiskAPI as _;
|
||||
for disk in holder.disks.read().await.iter().flatten().skip(1) {
|
||||
tokio::fs::remove_dir_all(disk.path().join(bucket).join(object))
|
||||
.await
|
||||
.expect("drop the copies from the drives the attempt missed");
|
||||
}
|
||||
}
|
||||
for version_id in [marker_id, version_id] {
|
||||
assert_eq!(drive_copies(&store, 1, bucket, object, version_id).await.len(), 1);
|
||||
}
|
||||
|
||||
Self {
|
||||
_temp_dirs: temp_dirs,
|
||||
store,
|
||||
_space: space,
|
||||
bucket,
|
||||
object,
|
||||
incarnation,
|
||||
source_set,
|
||||
marker,
|
||||
version,
|
||||
payload,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_entry(&self) -> RebalanceEntryOutcome {
|
||||
let entry = metacache_entry_from_source(&self.source_set, self.bucket, self.object).await;
|
||||
Box::pin(Arc::clone(&self.store).rebalance_entry(
|
||||
RebalanceEntryTarget {
|
||||
bucket: self.bucket.to_string(),
|
||||
pool_index: 0,
|
||||
},
|
||||
entry,
|
||||
Arc::clone(&self.source_set),
|
||||
Arc::new(RebalanceBucketConfigs {
|
||||
bucket_incarnation_id: Some(self.incarnation),
|
||||
..Default::default()
|
||||
}),
|
||||
Arc::from(DANGLING_COPY_REBALANCE_ID),
|
||||
CancellationToken::new(),
|
||||
))
|
||||
.await
|
||||
.expect("rebalance entry")
|
||||
}
|
||||
|
||||
/// Pools whose drives hold `version_id`, with the number of drives.
|
||||
async fn holders(&self, version_id: uuid::Uuid) -> Vec<(usize, usize)> {
|
||||
let mut holders = Vec::new();
|
||||
for pool in 0..self.store.pools.len() {
|
||||
let drives = drive_copies(&self.store, pool, self.bucket, self.object, version_id)
|
||||
.await
|
||||
.len();
|
||||
if drives > 0 {
|
||||
holders.push((pool, drives));
|
||||
}
|
||||
}
|
||||
holders
|
||||
}
|
||||
|
||||
/// The source entry is gone and `pool` serves both versions as the
|
||||
/// source wrote them, on every drive.
|
||||
async fn assert_moved_to(&self, pool: usize) {
|
||||
assert!(
|
||||
self.source_set
|
||||
.load_file_info_versions_exact(self.bucket, self.object)
|
||||
.await
|
||||
.expect("read source versions")
|
||||
.is_none(),
|
||||
"the source entry must be cleaned up"
|
||||
);
|
||||
let target = self.store.pools[pool]
|
||||
.get_disks_by_key(self.object)
|
||||
.load_file_info_versions_exact(self.bucket, self.object)
|
||||
.await
|
||||
.expect("read target versions")
|
||||
.expect("both versions must reach the target pool");
|
||||
let moved = |id| {
|
||||
target
|
||||
.versions
|
||||
.iter()
|
||||
.find(|version| version.version_id == id)
|
||||
.unwrap_or_else(|| panic!("version {id:?} must reach pool {pool}: {target:?}"))
|
||||
};
|
||||
let marker = moved(self.marker.version_id);
|
||||
assert!(marker.deleted, "the marker must stay a delete marker");
|
||||
assert_eq!(marker.mod_time, self.marker.mod_time);
|
||||
let version = moved(self.version.version_id);
|
||||
assert!(!version.deleted);
|
||||
assert_eq!(version.mod_time, self.version.mod_time);
|
||||
|
||||
let version_id = self.version.version_id.expect("object version id");
|
||||
let copies = drive_copies(&self.store, pool, self.bucket, self.object, version_id).await;
|
||||
assert_eq!(copies.len(), 4, "the object version must be on every drive of pool {pool}");
|
||||
for copy in &copies {
|
||||
assert_eq!(copy.mod_time, self.version.mod_time, "an earlier copy must be replaced, not kept");
|
||||
assert_eq!(copy.data_dir, copies[0].data_dir, "every drive must hold the same data");
|
||||
}
|
||||
let marker_id = self.marker.version_id.expect("marker version id");
|
||||
let markers = drive_copies(&self.store, pool, self.bucket, self.object, marker_id).await;
|
||||
assert_eq!(markers.len(), 4, "the marker must be on every drive of pool {pool}");
|
||||
assert!(markers.iter().all(|copy| copy.deleted));
|
||||
|
||||
let mut reader = self.store.pools[pool]
|
||||
.get_object_reader(
|
||||
self.bucket,
|
||||
self.object,
|
||||
None,
|
||||
HeaderMap::new(),
|
||||
&ObjectOptions {
|
||||
version_id: Some(version_id.to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("the moved version must be readable");
|
||||
let mut actual = Vec::new();
|
||||
reader.stream.read_to_end(&mut actual).await.expect("drain the moved version");
|
||||
assert_eq!(actual, self.payload, "the moved version must carry the source bytes");
|
||||
}
|
||||
}
|
||||
|
||||
/// A retry reuses the pool that holds an earlier attempt's copy, even
|
||||
/// though free space favours another pool, and rewrites that copy from
|
||||
/// the source instead of leaving the version in two pools.
|
||||
#[cfg(feature = "test-util")]
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn real_rebalance_retry_reuses_the_pool_holding_a_dangling_copy() {
|
||||
run_large_stack_async_test("rebalance-dangling-target-copy", || async {
|
||||
let fixture = DanglingTargetCopy::new(drive_space(64 << 20, 0)).await;
|
||||
assert_eq!(fixture.run_entry().await, RebalanceEntryOutcome::Completed);
|
||||
for version in [&fixture.marker, &fixture.version] {
|
||||
let id = version.version_id.expect("version id");
|
||||
assert_eq!(fixture.holders(id).await, vec![(1, 4)], "version {id} must be in one pool");
|
||||
}
|
||||
fixture.assert_moved_to(1).await;
|
||||
});
|
||||
}
|
||||
|
||||
/// A full pool holding an earlier copy must not pin the retry: placement
|
||||
/// falls back to the free-space choice, and the stale copy stays for heal.
|
||||
#[cfg(feature = "test-util")]
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn real_rebalance_retry_skips_a_full_pool_holding_a_dangling_copy() {
|
||||
run_large_stack_async_test("rebalance-dangling-full-holder", || async {
|
||||
let fixture = DanglingTargetCopy::new(drive_space(1 << 30, 1 << 30)).await;
|
||||
assert_eq!(fixture.run_entry().await, RebalanceEntryOutcome::Completed);
|
||||
for version in [&fixture.marker, &fixture.version] {
|
||||
let id = version.version_id.expect("version id");
|
||||
assert_eq!(fixture.holders(id).await, vec![(1, 1), (2, 4)], "version {id}");
|
||||
}
|
||||
fixture.assert_moved_to(2).await;
|
||||
});
|
||||
}
|
||||
|
||||
/// An `ExpiredObjectAllVersions` expiry is a prefix delete that takes the
|
||||
/// bucket lifecycle write lock, while the entry holds that lock for read
|
||||
/// through its bucket incarnation fence. Applied inline, it waited out the
|
||||
|
||||
@@ -280,7 +280,7 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
|
||||
continue;
|
||||
}
|
||||
|
||||
if !parts[part_idx].etag.is_empty() {
|
||||
if !parts[part_idx].etag.is_empty() && parts[part_idx].error.is_none() {
|
||||
present_count += 1;
|
||||
let part = &parts[part_idx];
|
||||
let key = (part.etag.as_str(), part.number, part.size, part.actual_size, part.integrity.as_ref());
|
||||
@@ -1118,9 +1118,9 @@ pub(in crate::set_disk) fn fill_deferred_bitrot_readers(
|
||||
// offset can be advanced to the failing stripe. A parity reader that was
|
||||
// opened eagerly during setup is pinned at the stripe-0 stream position
|
||||
// and could never be engaged mid-object, so swap it for an unopened
|
||||
// deferred reader carrying a stripe handle. The eager open already proved
|
||||
// the shard is reachable; no shard bytes were read from it, and the
|
||||
// ready/error bookkeeping that quorum decisions rely on is left untouched.
|
||||
// deferred reader carrying a stripe handle. The eager setup already proved
|
||||
// the shard is reachable; the ready/error bookkeeping that quorum decisions
|
||||
// rely on is left untouched.
|
||||
// Gate off (default): keep the eagerly opened parity readers exactly as
|
||||
// before — the lockstep path reads them on every stripe.
|
||||
if !demand_bound_lockstep {
|
||||
@@ -1217,8 +1217,8 @@ pub(in crate::set_disk) fn record_bitrot_reader_setup_strategy(
|
||||
}
|
||||
|
||||
/// When all online shards are local and mmap-read is enabled, materialize
|
||||
/// shard bytes with one `batch_shard_pread` instead of per-shard
|
||||
/// `spawn_blocking` via `open_disk_reader`.
|
||||
/// shard bytes in batches instead of per-shard `spawn_blocking` via
|
||||
/// `open_disk_reader`. Demand-bound reads add parity batches only as needed.
|
||||
#[cfg(unix)]
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn try_create_bitrot_readers_via_batch_pread(
|
||||
@@ -1232,18 +1232,29 @@ async fn try_create_bitrot_readers_via_batch_pread(
|
||||
shard_size: usize,
|
||||
checksum_algo: HashAlgorithm,
|
||||
skip_verify_bitrot: bool,
|
||||
data_shards: usize,
|
||||
parity_shards: usize,
|
||||
mode: BitrotReaderSetupMode,
|
||||
strategy: BitrotReaderSetupStrategy,
|
||||
) -> Option<BitrotReaderSetup> {
|
||||
use crate::disk::local::batch_shard_pread;
|
||||
use rustfs_io_metrics::{LocalBatchReadDecision, record_local_batch_read_decision};
|
||||
use std::io::Cursor;
|
||||
|
||||
let prepare_started = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
|
||||
let record_decision = |decision| {
|
||||
record_local_batch_read_decision(decision, prepare_started.map(|started| started.elapsed()));
|
||||
};
|
||||
let (adj_off, adj_len) = adjust_shard_read_params(read_offset, read_length, shard_size, &checksum_algo);
|
||||
if adj_len > object_mmap_read_max_length() {
|
||||
record_decision(LocalBatchReadDecision::ReadCapExceeded);
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut batch_items: Vec<(usize, std::path::PathBuf, usize, usize)> = Vec::new();
|
||||
for (idx, disk_op) in disks.iter().enumerate() {
|
||||
if files.get(idx).is_some_and(|fi| fi.data.is_some()) {
|
||||
record_decision(LocalBatchReadDecision::InlineData);
|
||||
return None;
|
||||
}
|
||||
if let Some(disk) = disk_op.as_ref() {
|
||||
@@ -1251,50 +1262,79 @@ async fn try_create_bitrot_readers_via_batch_pread(
|
||||
let path_str = format!("{object}/{data_dir}/part.{part_number}");
|
||||
match disk.get_object_path_for_io_if_local(bucket, &path_str) {
|
||||
Some(Ok(p)) => batch_items.push((idx, p, adj_off, adj_len)),
|
||||
_ => return None,
|
||||
Some(Err(_)) => {
|
||||
record_decision(LocalBatchReadDecision::InvalidPath);
|
||||
return None;
|
||||
}
|
||||
None => {
|
||||
record_decision(LocalBatchReadDecision::NonLocalShard);
|
||||
return None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if batch_items.is_empty() {
|
||||
record_decision(LocalBatchReadDecision::NoLocalShards);
|
||||
return None;
|
||||
}
|
||||
|
||||
let requests: Vec<_> = batch_items.iter().map(|(_, p, off, len)| (p.clone(), *off, *len)).collect();
|
||||
#[cfg(test)]
|
||||
for (disk_index, _, _, _) in &batch_items {
|
||||
disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, *disk_index);
|
||||
}
|
||||
let batch_results = batch_shard_pread(requests).await;
|
||||
|
||||
// Legacy lockstep consumes every shard, so keep its coalesced batch even
|
||||
// when setup prefers data blocks. Only demand-bound decode can leave the
|
||||
// parity readers unopened; preading them here would discard their bytes
|
||||
// when fill_deferred_bitrot_readers installs stripe-aligned reserves.
|
||||
let data_first = strategy != BitrotReaderSetupStrategy::AllShards
|
||||
&& crate::erasure::coding::decode::get_lockstep_data_shards_only_enabled();
|
||||
let mut setup = BitrotReaderSetup::new(disks.len());
|
||||
for (i, (idx, _, _, _)) in batch_items.iter().enumerate() {
|
||||
setup.mark_scheduled(*idx);
|
||||
match &batch_results[i] {
|
||||
Ok(bytes) => {
|
||||
let reader = BitrotReader::new(
|
||||
ShardReader::InMemory(Cursor::new(bytes.clone())),
|
||||
shard_size,
|
||||
checksum_algo.clone(),
|
||||
skip_verify_bitrot,
|
||||
);
|
||||
setup.apply_reader_result(*idx, Ok(Some(reader)));
|
||||
}
|
||||
Err(e) => {
|
||||
setup.apply_reader_result(*idx, Err(e.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (idx, disk_op) in disks.iter().enumerate() {
|
||||
if setup.scheduled[idx] {
|
||||
continue;
|
||||
}
|
||||
setup.mark_scheduled(idx);
|
||||
if disk_op.is_none() {
|
||||
for (idx, disk) in disks.iter().enumerate() {
|
||||
if disk.is_none() {
|
||||
setup.mark_scheduled(idx);
|
||||
setup.apply_reader_result(idx, Ok(None));
|
||||
}
|
||||
}
|
||||
let mut batch_size = if data_first {
|
||||
batch_items.iter().take_while(|(idx, _, _, _)| *idx < data_shards).count()
|
||||
} else {
|
||||
batch_items.len()
|
||||
};
|
||||
let mut pending = batch_items.into_iter();
|
||||
record_decision(LocalBatchReadDecision::Selected);
|
||||
loop {
|
||||
let mut indices = Vec::with_capacity(batch_size.min(pending.len()));
|
||||
let requests: Vec<_> = pending
|
||||
.by_ref()
|
||||
.take(batch_size)
|
||||
.map(|(idx, path, offset, length)| {
|
||||
setup.mark_scheduled(idx);
|
||||
indices.push(idx);
|
||||
#[cfg(test)]
|
||||
disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, idx);
|
||||
(path, offset, length)
|
||||
})
|
||||
.collect();
|
||||
if !requests.is_empty() {
|
||||
for (idx, result) in indices.into_iter().zip(batch_shard_pread(requests).await) {
|
||||
let result = result.map(|bytes| {
|
||||
Some(BitrotReader::new(
|
||||
ShardReader::InMemory(Cursor::new(bytes)),
|
||||
shard_size,
|
||||
checksum_algo.clone(),
|
||||
skip_verify_bitrot,
|
||||
))
|
||||
});
|
||||
setup.apply_reader_result(idx, result);
|
||||
}
|
||||
}
|
||||
if !data_first || setup.has_setup_quorum(data_shards, parity_shards, mode) || pending.len() == 0 {
|
||||
break;
|
||||
}
|
||||
// A missing/corrupt data file needs replacement parity. Reuse the
|
||||
// canonical target, including the extra verification source required
|
||||
// for reconstruction, and retry only the outstanding deficit.
|
||||
batch_size = setup
|
||||
.scheduling_target(data_shards, parity_shards, mode)
|
||||
.saturating_sub(setup.available_shards());
|
||||
}
|
||||
|
||||
Some(setup)
|
||||
}
|
||||
@@ -1312,7 +1352,12 @@ async fn try_create_bitrot_readers_via_batch_pread(
|
||||
_shard_size: usize,
|
||||
_checksum_algo: HashAlgorithm,
|
||||
_skip_verify_bitrot: bool,
|
||||
_data_shards: usize,
|
||||
_parity_shards: usize,
|
||||
_mode: BitrotReaderSetupMode,
|
||||
_strategy: BitrotReaderSetupStrategy,
|
||||
) -> Option<BitrotReaderSetup> {
|
||||
rustfs_io_metrics::record_local_batch_read_decision(rustfs_io_metrics::LocalBatchReadDecision::UnsupportedPlatform, None);
|
||||
None
|
||||
}
|
||||
|
||||
@@ -1476,11 +1521,19 @@ pub(in crate::set_disk) async fn create_bitrot_readers_until_quorum_with_prefere
|
||||
) -> BitrotReaderSetup {
|
||||
let strategy = get_bitrot_reader_setup_strategy(mode, prefer_data_blocks_first);
|
||||
|
||||
if use_mmap_read
|
||||
&& !matches!(
|
||||
crate::set_disk::get_object_read_policy(),
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource
|
||||
)
|
||||
let batch_read_enabled = if !use_mmap_read {
|
||||
rustfs_io_metrics::record_local_batch_read_decision(rustfs_io_metrics::LocalBatchReadDecision::MmapDisabled, None);
|
||||
false
|
||||
} else if matches!(
|
||||
crate::set_disk::get_object_read_policy(),
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource
|
||||
) {
|
||||
rustfs_io_metrics::record_local_batch_read_decision(rustfs_io_metrics::LocalBatchReadDecision::CopySource, None);
|
||||
false
|
||||
} else {
|
||||
true
|
||||
};
|
||||
if batch_read_enabled
|
||||
&& let Some(mut setup) = try_create_bitrot_readers_via_batch_pread(
|
||||
files,
|
||||
disks,
|
||||
@@ -1492,6 +1545,10 @@ pub(in crate::set_disk) async fn create_bitrot_readers_until_quorum_with_prefere
|
||||
shard_size,
|
||||
checksum_algo.clone(),
|
||||
skip_verify_bitrot,
|
||||
data_shards,
|
||||
parity_shards,
|
||||
mode,
|
||||
strategy,
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -7864,6 +7921,575 @@ mod tests {
|
||||
(dirs, disks)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
async fn local_batch_quorum_fixture(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> (Vec<TempDir>, Vec<Option<DiskStore>>, Vec<FileInfo>, Vec<u8>) {
|
||||
let payload: Vec<_> = (0..1024 * 1024)
|
||||
.map(|index| u8::try_from(index % 251).expect("bounded byte"))
|
||||
.collect();
|
||||
let (dirs, disks) = call_counter_local_disks(bucket, 8).await;
|
||||
let mut files = inline_metadata_fanout_fileinfos_with_geometry(bucket, object, &payload, false, 4, 4).await;
|
||||
for (file, disk) in files.iter_mut().zip(&disks) {
|
||||
let path = format!("{object}/{}/part.1", file.data_dir.expect("fixture data directory"));
|
||||
disk.as_ref()
|
||||
.expect("fixture disk")
|
||||
.write_all(bucket, &path, file.data.take().expect("encoded fixture shard"))
|
||||
.await
|
||||
.expect("external shard fixture should write");
|
||||
}
|
||||
(dirs, disks, files, payload)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
async fn local_batch_quorum_setup(
|
||||
files: &[FileInfo],
|
||||
disks: &[Option<DiskStore>],
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
mode: BitrotReaderSetupMode,
|
||||
) -> BitrotReaderSetup {
|
||||
let erasure = coding::Erasure::new_with_options(4, 4, files[0].erasure.block_size, false);
|
||||
create_bitrot_readers_until_quorum_with_preference(
|
||||
files,
|
||||
disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
erasure.shard_file_offset(0, 1024 * 1024, 1024 * 1024),
|
||||
erasure.shard_size(),
|
||||
HashAlgorithm::HighwayHash256S,
|
||||
false,
|
||||
true,
|
||||
4,
|
||||
4,
|
||||
mode,
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
async fn local_batch_quorum_decode(setup: BitrotReaderSetup, block_size: usize) -> (Vec<u8>, Option<std::io::Error>) {
|
||||
let erasure = coding::Erasure::new_with_options(4, 4, block_size, false);
|
||||
let mut output = Vec::new();
|
||||
let (_, error) = erasure
|
||||
.decode_with_stripe_handles_and_reopeners(
|
||||
&mut output,
|
||||
setup.readers,
|
||||
0,
|
||||
1024 * 1024,
|
||||
1024 * 1024,
|
||||
None,
|
||||
setup.deferred_stripe_handles,
|
||||
setup.deferred_reopeners,
|
||||
)
|
||||
.await;
|
||||
(output, error)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_healthy_reads_only_data_when_demand_bound() {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("false")),
|
||||
],
|
||||
crate::erasure::coding::decode::with_decode_read_policy(
|
||||
crate::erasure::coding::decode::DecodeReadPolicy::DemandBound,
|
||||
async {
|
||||
let bucket = "batch-data-only";
|
||||
let object = "healthy-one-mib";
|
||||
let (_dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
|
||||
let calls = disk_call_counters::observe(object);
|
||||
let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
|
||||
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
|
||||
assert_eq!(setup.available_shards(), 4);
|
||||
assert_eq!(setup.deferred_shards(), 4);
|
||||
for index in 4..8 {
|
||||
assert_eq!(calls.for_disk(disk_call_counters::KIND_DISK_SHARD_OPEN, index), 0);
|
||||
assert!(!setup.attempted[index], "parity must stay unopened until needed");
|
||||
}
|
||||
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
|
||||
assert!(error.is_none(), "healthy decode failed: {error:?}");
|
||||
assert_eq!(output, payload);
|
||||
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
|
||||
},
|
||||
),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_preserves_legacy_and_all_shards_override() {
|
||||
for (data_first, demand_bound) in [("true", "false"), ("false", "true")] {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some(data_first)),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some(demand_bound)),
|
||||
],
|
||||
async {
|
||||
let bucket = "batch-legacy";
|
||||
let object = "all-shards";
|
||||
let (_dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
|
||||
let calls = disk_call_counters::observe(object);
|
||||
let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
|
||||
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 8);
|
||||
assert_eq!(setup.available_shards(), 8);
|
||||
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
|
||||
assert!(error.is_none(), "legacy decode failed: {error:?}");
|
||||
assert_eq!(output, payload);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_recovers_missing_truncated_and_bitrot_data() {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
|
||||
],
|
||||
async {
|
||||
for fault in ["missing", "truncated", "bitrot"] {
|
||||
let bucket = "batch-degraded";
|
||||
let object = format!("degraded-{fault}");
|
||||
let (dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, &object).await;
|
||||
let path = dirs[0]
|
||||
.path()
|
||||
.join(bucket)
|
||||
.join(&object)
|
||||
.join(files[0].data_dir.expect("fixture directory").to_string())
|
||||
.join("part.1");
|
||||
match fault {
|
||||
"missing" => tokio::fs::remove_file(&path).await.expect("remove data shard"),
|
||||
"truncated" => tokio::fs::write(&path, b"short").await.expect("truncate data shard"),
|
||||
_ => {
|
||||
let mut bytes = tokio::fs::read(&path).await.expect("read bitrot fixture");
|
||||
bytes[32] ^= 0xff;
|
||||
tokio::fs::write(&path, bytes)
|
||||
.await
|
||||
.expect("corrupt data without changing length");
|
||||
}
|
||||
}
|
||||
let setup =
|
||||
local_batch_quorum_setup(&files, &disks, bucket, &object, BitrotReaderSetupMode::ReadQuorum).await;
|
||||
assert_eq!(setup.available_shards(), 4);
|
||||
if fault == "bitrot" {
|
||||
assert_eq!(setup.attempted_shards(), 4, "bitrot is detected on decode, not pread");
|
||||
} else {
|
||||
assert_eq!(setup.attempted_shards(), 5, "setup reads only one replacement parity shard");
|
||||
}
|
||||
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
|
||||
assert!(error.is_none(), "{fault} should recover: {error:?}");
|
||||
assert_eq!(output, payload, "{fault} must reconstruct the exact body");
|
||||
}
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_keeps_extra_reconstruction_verification_source() {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_CODEC_STREAMING_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
|
||||
],
|
||||
async {
|
||||
let bucket = "batch-verification";
|
||||
let object = "missing-data";
|
||||
let (_dirs, mut disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
|
||||
disks[0] = None;
|
||||
let calls = disk_call_counters::observe(object);
|
||||
let setup =
|
||||
local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::VerifyReconstruction).await;
|
||||
assert_eq!(setup.available_shards(), 5, "degraded reconstruction needs quorum plus verification");
|
||||
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 5);
|
||||
assert!(setup.ready[4] && setup.ready[5], "two parity sources must be available");
|
||||
assert!(!setup.attempted[6] && !setup.attempted[7], "unneeded parity stays deferred");
|
||||
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
|
||||
assert!(error.is_none(), "verified reconstruction failed: {error:?}");
|
||||
assert_eq!(output, payload);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_fails_without_read_quorum() {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
|
||||
],
|
||||
async {
|
||||
let bucket = "batch-no-quorum";
|
||||
let object = "all-data-offline";
|
||||
let (_dirs, mut disks, files, _payload) = local_batch_quorum_fixture(bucket, object).await;
|
||||
disks[..5].fill(None);
|
||||
let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
|
||||
assert_eq!(setup.available_shards(), 3);
|
||||
assert!(!setup.has_setup_quorum(4, 4, BitrotReaderSetupMode::ReadQuorum));
|
||||
assert_eq!(setup.deferred_shards(), 0);
|
||||
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
|
||||
assert!(error.is_some(), "insufficient shards must fail closed");
|
||||
assert!(output.is_empty(), "no unverified bytes may be emitted");
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn local_batch_quorum_recovers_late_bitrot_after_range_offset() {
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
|
||||
("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
|
||||
],
|
||||
async {
|
||||
let bucket = "batch-ranged";
|
||||
let object = "late-bitrot";
|
||||
let (_dirs, disks) = call_counter_local_disks(bucket, 8).await;
|
||||
let erasure = coding::Erasure::new_with_options(4, 4, 64, false);
|
||||
let payload: Vec<u8> = (0..192).collect();
|
||||
let mut shards = vec![Vec::new(); 8];
|
||||
for block in payload.chunks(64) {
|
||||
for (index, shard) in erasure
|
||||
.encode_data(block)
|
||||
.expect("encode range stripe")
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
let mut writer = coding::BitrotWriterWrapper::new(
|
||||
coding::CustomWriter::new_inline_buffer(),
|
||||
erasure.shard_size(),
|
||||
HashAlgorithm::HighwayHash256S,
|
||||
);
|
||||
writer.write(&shard).await.expect("write range stripe");
|
||||
writer.shutdown().await.expect("finish range stripe");
|
||||
shards[index].extend(writer.into_inline_data().expect("range stripe bytes"));
|
||||
}
|
||||
}
|
||||
// Corrupt the third stripe; the first requested stripe is healthy.
|
||||
shards[0][2 * (32 + erasure.shard_size()) + 32] ^= 0xff;
|
||||
let mut files = vec![FileInfo::new(object, 4, 4); 8];
|
||||
let data_dir = Uuid::new_v4();
|
||||
for ((file, disk), bytes) in files.iter_mut().zip(&disks).zip(shards) {
|
||||
file.data_dir = Some(data_dir);
|
||||
disk.as_ref()
|
||||
.expect("fixture disk")
|
||||
.write_all(bucket, &format!("{object}/{data_dir}/part.1"), Bytes::from(bytes))
|
||||
.await
|
||||
.expect("write multi-stripe shard");
|
||||
}
|
||||
let calls = disk_call_counters::observe(object);
|
||||
let setup = create_bitrot_readers_until_quorum_with_preference(
|
||||
&files,
|
||||
&disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
erasure.shard_size(),
|
||||
2 * erasure.shard_size(),
|
||||
erasure.shard_size(),
|
||||
HashAlgorithm::HighwayHash256S,
|
||||
false,
|
||||
true,
|
||||
4,
|
||||
4,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
|
||||
let mut output = Vec::new();
|
||||
let (written, error) = erasure
|
||||
.decode_with_stripe_handles_and_reopeners(
|
||||
&mut output,
|
||||
setup.readers,
|
||||
64,
|
||||
128,
|
||||
payload.len(),
|
||||
None,
|
||||
setup.deferred_stripe_handles,
|
||||
setup.deferred_reopeners,
|
||||
)
|
||||
.await;
|
||||
assert!(error.is_none(), "late bitrot must recover from stripe-aligned parity: {error:?}");
|
||||
assert_eq!(written, 128);
|
||||
assert_eq!(output, payload[64..]);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn local_batch_read_decisions_follow_the_actual_reader_setup_entry() {
|
||||
use crate::test_metrics::CapturingRecorder;
|
||||
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.expect("reader setup validation runtime");
|
||||
let recorder = CapturingRecorder::default();
|
||||
let previous = rustfs_io_metrics::get_stage_metrics_enabled();
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
runtime.block_on(temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_OBJECT_MMAP_READ_MAX_LENGTH, Some("4")),
|
||||
(rustfs_config::ENV_INTERNODE_PREWARM, Some("false")),
|
||||
],
|
||||
async {
|
||||
let bucket = "batch-path-attribution";
|
||||
let object = "object";
|
||||
let (_dirs, disks) = call_counter_local_disks(bucket, 2).await;
|
||||
let data_dir = Uuid::new_v4();
|
||||
let mut files = vec![FileInfo::new(object, 1, 1); 2];
|
||||
for (file, disk) in files.iter_mut().zip(&disks) {
|
||||
file.data_dir = Some(data_dir);
|
||||
disk.as_ref()
|
||||
.expect("local disk")
|
||||
.write_all(bucket, &format!("{object}/{data_dir}/part.1"), Bytes::from_static(b"data"))
|
||||
.await
|
||||
.expect("write batch fixture");
|
||||
}
|
||||
assert_eq!(object_mmap_read_max_length(), 4, "cap must be controlled by this test");
|
||||
for length in [3, 4] {
|
||||
let setup = create_bitrot_readers_until_quorum_with_preference(
|
||||
&files,
|
||||
&disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
length,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
true,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(setup.readers.iter().flatten().count(), 2);
|
||||
}
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&files,
|
||||
&disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
5,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none(),
|
||||
"a read above the adjusted-length cap must fall back"
|
||||
);
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&files,
|
||||
&disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::HighwayHash256S,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none(),
|
||||
"checksum overhead must be included in the read cap"
|
||||
);
|
||||
let mut inline_files = files.clone();
|
||||
inline_files[0].data = Some(Bytes::from_static(b"data"));
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&inline_files,
|
||||
&disks,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
let mut endpoint = Endpoint::try_from("http://127.0.0.1:1/disk").expect("remote placeholder endpoint");
|
||||
endpoint.is_local = false;
|
||||
let remote = new_disk(
|
||||
&endpoint,
|
||||
&DiskOption {
|
||||
cleanup: false,
|
||||
health_check: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("remote placeholder must initialize without a connection");
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&files,
|
||||
&[Some(remote), None],
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none(),
|
||||
"any remote shard must keep the setup out of the local batch path"
|
||||
);
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&files,
|
||||
&disks,
|
||||
bucket,
|
||||
"../outside",
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
let missing = vec![None, None];
|
||||
assert!(
|
||||
try_create_bitrot_readers_via_batch_pread(
|
||||
&files,
|
||||
&missing,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
BitrotReaderSetupStrategy::AllShards,
|
||||
)
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
for (use_mmap, policy) in [
|
||||
(false, crate::set_disk::GetObjectReadPolicy::Default),
|
||||
(true, crate::set_disk::GetObjectReadPolicy::CopySource),
|
||||
] {
|
||||
crate::set_disk::with_get_object_read_policy(policy, async {
|
||||
create_bitrot_readers_until_quorum_with_preference(
|
||||
&files,
|
||||
&missing,
|
||||
bucket,
|
||||
object,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
4,
|
||||
HashAlgorithm::None,
|
||||
false,
|
||||
use_mmap,
|
||||
1,
|
||||
1,
|
||||
BitrotReaderSetupMode::ReadQuorum,
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
})
|
||||
.await;
|
||||
}
|
||||
},
|
||||
));
|
||||
});
|
||||
rustfs_io_metrics::set_get_stage_metrics_enabled(previous);
|
||||
let metric = "rustfs_io_get_object_local_batch_decisions_total";
|
||||
for (decision, expected) in [
|
||||
("selected", 2),
|
||||
("read_cap_exceeded", 2),
|
||||
("inline_data", 1),
|
||||
("invalid_path", 1),
|
||||
("non_local_shard", 1),
|
||||
("no_local_shards", 1),
|
||||
("mmap_disabled", 1),
|
||||
("copy_source", 1),
|
||||
] {
|
||||
assert_eq!(recorder.counter_value(metric, &[("decision", decision)]), expected, "{decision}");
|
||||
}
|
||||
assert_eq!(
|
||||
recorder.counter_value("rustfs_io_get_object_local_batch_shards_total", &[("outcome", "success")]),
|
||||
4
|
||||
);
|
||||
assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 14);
|
||||
assert_eq!(recorder.histogram_sample_count("rustfs_io_get_object_local_batch_prepare_seconds"), 8);
|
||||
}
|
||||
|
||||
async fn reopen_local_disk(dir: &TempDir) -> DiskStore {
|
||||
let endpoint =
|
||||
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
|
||||
@@ -12576,6 +13202,20 @@ mod tests {
|
||||
assert_eq!(err, DiskError::ErasureReadQuorum);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_read_part_does_not_count_error_response_with_etag_as_a_vote() {
|
||||
let healthy = read_part_test_part(1, "same-etag");
|
||||
let mut failed = healthy.clone();
|
||||
failed.error = Some("metadata read failed".to_owned());
|
||||
let responses = vec![Some(vec![healthy.clone()]), Some(vec![healthy]), Some(vec![failed])];
|
||||
let err = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, 3)
|
||||
.expect_err("an error-bearing response must not supply the third vote");
|
||||
assert_eq!(err, DiskError::ErasureReadQuorum);
|
||||
let part = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, 2)
|
||||
.expect("the two healthy matching responses remain a valid quorum");
|
||||
assert!(part.error.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_read_part_accepts_alternate_missing_error_markers() {
|
||||
let responses = vec![
|
||||
|
||||
@@ -338,17 +338,26 @@ fn abort_slot(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn abort_certificate_cohort(
|
||||
pub(in crate::set_disk) fn abort_certificate_cohort(
|
||||
certificate: &MultipartAbortCertificate,
|
||||
ids: &[Option<Uuid>],
|
||||
authority: &ProofVerificationContext,
|
||||
) -> Result<()> {
|
||||
certificate.verify_with_authority(authority)?;
|
||||
if ids.len() != certificate.disk_ids.len()
|
||||
|| ids.iter().copied().collect::<Option<Vec<_>>>().as_ref() != Some(&certificate.disk_ids)
|
||||
{
|
||||
if ids.len() != certificate.disk_ids.len() {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
let mut missing = false;
|
||||
for (actual, expected) in ids.iter().zip(&certificate.disk_ids) {
|
||||
match actual {
|
||||
Some(actual) if actual != expected => return Err(DiskError::FileCorrupt),
|
||||
Some(_) => {}
|
||||
None => missing = true,
|
||||
}
|
||||
}
|
||||
if missing {
|
||||
return Err(DiskError::ErasureWriteQuorum);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -283,6 +283,23 @@ fn ensure_decommission_tier_free_version_commit_fence(bucket: &str, object: &str
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Names a `bucket/object` key prefix whose best-effort rename-tail repairs
|
||||
/// are not submitted. A root-heal fixture arms it so the explicit admin heal
|
||||
/// remains the only owner of objects written while their target was offline.
|
||||
#[cfg(feature = "e2e-test-hooks")]
|
||||
const RENAME_TAIL_HEAL_SUPPRESS_PREFIX_ENV: &str = "RUSTFS_E2E_SUPPRESS_RENAME_TAIL_HEAL_PREFIX";
|
||||
|
||||
#[cfg(feature = "e2e-test-hooks")]
|
||||
fn rename_tail_heal_suppressed_for_e2e(bucket: &str, object: Option<&str>) -> bool {
|
||||
let Some(prefix) = std::env::var(RENAME_TAIL_HEAL_SUPPRESS_PREFIX_ENV)
|
||||
.ok()
|
||||
.filter(|prefix| !prefix.is_empty())
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
object.is_some_and(|object| format!("{bucket}/{object}").starts_with(&prefix))
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
pub(super) async fn require_current_restore_operation_id(
|
||||
&self,
|
||||
@@ -486,10 +503,16 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
|
||||
rustfs_utils::http::metadata_compat::remove_str(metadata, SUFFIX_BUCKET_INCARNATION_ID);
|
||||
}
|
||||
|
||||
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
|
||||
pub(crate) fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
|
||||
metadata.keys().any(|key| is_object_encryption_marker(key))
|
||||
}
|
||||
|
||||
/// Reject ambiguous part sizes unless the upload records a transformation.
|
||||
pub(crate) fn multipart_part_logical_size(stored_size: usize, actual_size: i64, transformed: bool) -> Option<usize> {
|
||||
let actual_size = usize::try_from(actual_size).ok()?;
|
||||
(transformed || actual_size == stored_size).then_some(actual_size)
|
||||
}
|
||||
|
||||
/// Per-set memoized capacity dirty scope.
|
||||
///
|
||||
/// The disk endpoint/path identity of each slot in a set is immutable for the
|
||||
@@ -1024,6 +1047,19 @@ impl GetObjectFileInfo {
|
||||
self.as_parts().0
|
||||
}
|
||||
|
||||
/// Marks an exact-version selection as the object's latest version after
|
||||
/// the versions above it were skipped as dangling.
|
||||
fn mark_selected_as_latest(&mut self) {
|
||||
if let Some(OwnedGetObjectFileInfo {
|
||||
payload: OwnedGetObjectFileInfoPayload::Direct { fi, .. },
|
||||
..
|
||||
}) = self.owned.as_mut()
|
||||
{
|
||||
fi.is_latest = true;
|
||||
fi.successor_mod_time = None;
|
||||
}
|
||||
}
|
||||
|
||||
fn parts_metadata(&self) -> &[FileInfo] {
|
||||
self.as_parts().1
|
||||
}
|
||||
@@ -4232,14 +4268,23 @@ pub(super) struct MultipartWriteQuorumContext<'a> {
|
||||
part_number: Option<usize>,
|
||||
}
|
||||
|
||||
fn record_multipart_write_quorum_failure(
|
||||
stage: &'static str,
|
||||
errs: &[Option<DiskError>],
|
||||
write_quorum: usize,
|
||||
) -> crate::disk::error_reduce::WriteQuorumFailureSummary {
|
||||
let summary = build_write_quorum_failure_summary(errs, OBJECT_OP_IGNORED_ERRS, write_quorum);
|
||||
runtime_sources::record_erasure_write_quorum_failure(stage, summary.dominant_error_label);
|
||||
summary
|
||||
}
|
||||
|
||||
fn log_multipart_write_quorum_failure(
|
||||
context: MultipartWriteQuorumContext<'_>,
|
||||
errs: &[Option<DiskError>],
|
||||
write_quorum: usize,
|
||||
returned_error: &DiskError,
|
||||
) {
|
||||
let summary = build_write_quorum_failure_summary(errs, OBJECT_OP_IGNORED_ERRS, write_quorum);
|
||||
runtime_sources::record_erasure_write_quorum_failure(context.stage, summary.dominant_error_label);
|
||||
let summary = record_multipart_write_quorum_failure(context.stage, errs, write_quorum);
|
||||
warn!(
|
||||
target: "rustfs_ecstore::set_disk",
|
||||
event = EVENT_SET_DISK_MULTIPART,
|
||||
@@ -5121,6 +5166,11 @@ impl SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "e2e-test-hooks")]
|
||||
if rename_tail_heal_suppressed_for_e2e(&request.bucket, request.object_prefix.as_deref()) {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let _ = rustfs_heal_contracts::heal_channel::send_heal_request(request).await;
|
||||
Ok(())
|
||||
}
|
||||
@@ -6841,6 +6891,17 @@ impl SetDisks {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Whether any drive holds `version_id` of the key, including a copy on
|
||||
/// fewer drives than read quorum that object reads report as absent.
|
||||
pub(crate) async fn holds_version_on_any_drive(&self, bucket: &str, object: &str, version_id: Uuid) -> bool {
|
||||
let disks = self.disk_inventory().await;
|
||||
let (copies, _) = Self::read_all_raw_file_info(&disks, bucket, object, false).await;
|
||||
copies
|
||||
.iter()
|
||||
.flatten()
|
||||
.any(|copy| FileMeta::load(©.buf).is_ok_and(|meta| meta.find_version(Some(version_id)).is_ok()))
|
||||
}
|
||||
|
||||
/// Publish an internal tier free-version record without changing its
|
||||
/// delete-marker shape or remote-tier identity. The caller holds the
|
||||
/// source and target object locks; a write quorum is required before the
|
||||
|
||||
@@ -40,9 +40,35 @@ pub(crate) mod test_hooks {
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
type Hook = Arc<dyn Fn() -> BoxFuture<'static, ()> + Send + Sync>;
|
||||
type TransactionLockHook = Arc<dyn Fn(&rustfs_lock::NamespaceLockGuard) + Send + Sync>;
|
||||
|
||||
static BEFORE_GC_FINALIZE: Mutex<Vec<(String, String, Hook)>> = Mutex::new(Vec::new());
|
||||
static CLEANUP_ATTEMPTS: Mutex<Vec<(String, String)>> = Mutex::new(Vec::new());
|
||||
static AFTER_TRANSACTION_LOCK: Mutex<Vec<(String, String, TransactionLockHook)>> = Mutex::new(Vec::new());
|
||||
|
||||
pub(crate) fn after_transaction_lock_acquired(bucket: &str, object: &str, hook: TransactionLockHook) {
|
||||
AFTER_TRANSACTION_LOCK
|
||||
.lock()
|
||||
.expect("transaction lock hooks")
|
||||
.push((bucket.to_owned(), object.to_owned(), hook));
|
||||
}
|
||||
|
||||
pub(in crate::set_disk) fn run_after_transaction_lock_acquired(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
guard: &rustfs_lock::NamespaceLockGuard,
|
||||
) {
|
||||
let hook = {
|
||||
let mut hooks = AFTER_TRANSACTION_LOCK.lock().expect("transaction lock hooks");
|
||||
hooks
|
||||
.iter()
|
||||
.position(|(hook_bucket, hook_object, _)| hook_bucket == bucket && hook_object == object)
|
||||
.map(|index| hooks.remove(index).2)
|
||||
};
|
||||
if let Some(hook) = hook {
|
||||
hook(guard);
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn record_cleanup_attempt(bucket: &str, object: &str) {
|
||||
CLEANUP_ATTEMPTS
|
||||
@@ -112,6 +138,7 @@ pub(crate) struct DeleteRecoveryContext<'a> {
|
||||
/// Only the locked heal path may repair a live marker-creation source.
|
||||
pub repair_marker_source: bool,
|
||||
pub namespace_owner: Option<Arc<dyn Send + Sync>>,
|
||||
pub transaction_guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
|
||||
pub recovery_ticket: Option<crate::store::delete_recovery_runtime::DeleteRecoveryTicket>,
|
||||
pub published_hint: Option<(Uuid, &'a [Result<DeleteRecoveryInspection>])>,
|
||||
pub can_commit: &'a (dyn Fn() -> bool + Sync),
|
||||
@@ -125,6 +152,7 @@ impl<'a> DeleteRecoveryContext<'a> {
|
||||
explicit_delete: false,
|
||||
repair_marker_source: false,
|
||||
namespace_owner: None,
|
||||
transaction_guard: None,
|
||||
recovery_ticket: None,
|
||||
published_hint: None,
|
||||
can_commit,
|
||||
@@ -373,9 +401,22 @@ impl SetDisks {
|
||||
write_quorum: usize,
|
||||
mut context: DeleteRecoveryContext<'_>,
|
||||
) -> Result<()> {
|
||||
if context.namespace_owner.is_none() {
|
||||
context.namespace_owner = Some(self.ctx.begin_namespace_commit());
|
||||
}
|
||||
let transaction_guard = match context.transaction_guard.take() {
|
||||
Some(guard) => guard,
|
||||
None => self
|
||||
.acquire_delete_transaction_lock(bucket, object)
|
||||
.await
|
||||
.map_err(DiskError::other)?,
|
||||
};
|
||||
let outer_can_commit = context.can_commit;
|
||||
let can_commit = || outer_can_commit() && !transaction_guard.is_lock_lost();
|
||||
context.can_commit = &can_commit;
|
||||
context.transaction_guard = Some(Arc::clone(&transaction_guard));
|
||||
let owner = context
|
||||
.namespace_owner
|
||||
.take()
|
||||
.unwrap_or_else(|| self.ctx.begin_namespace_commit());
|
||||
context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
|
||||
let can_commit = context.can_commit;
|
||||
let states = inspect(disks, bucket, object, transaction).await;
|
||||
let untouched_absence = |state: &Result<DeleteRecoveryInspection>| matches!(state, Ok(state) if state.state == DeleteRecoveryState::Absent && state.current_digest.is_none());
|
||||
@@ -547,8 +588,25 @@ impl SetDisks {
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
context: DeleteRecoveryContext<'_>,
|
||||
mut context: DeleteRecoveryContext<'_>,
|
||||
) -> Result<DeleteRecoverySummary> {
|
||||
let transaction_guard = match context.transaction_guard.take() {
|
||||
Some(guard) => guard,
|
||||
None => self
|
||||
.acquire_delete_transaction_lock(bucket, object)
|
||||
.await
|
||||
.map_err(DiskError::other)?,
|
||||
};
|
||||
let outer_can_commit = context.can_commit;
|
||||
let can_commit = || outer_can_commit() && !transaction_guard.is_lock_lost();
|
||||
context.can_commit = &can_commit;
|
||||
if !context.dry_run {
|
||||
let owner = context
|
||||
.namespace_owner
|
||||
.take()
|
||||
.unwrap_or_else(|| self.ctx.begin_namespace_commit());
|
||||
context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
|
||||
}
|
||||
let DeleteRecoveryContext {
|
||||
incarnation,
|
||||
dry_run,
|
||||
@@ -558,6 +616,7 @@ impl SetDisks {
|
||||
published_hint,
|
||||
can_commit,
|
||||
recovery_ticket: _,
|
||||
transaction_guard: _,
|
||||
} = context;
|
||||
let disks = self.disk_inventory().await;
|
||||
let complete_inventory = disks.len() == self.set_drive_count && disks.iter().all(Option::is_some);
|
||||
@@ -1257,6 +1316,135 @@ mod tests {
|
||||
assert_eq!(body(&store, bucket, None).await, original);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_recovery_deferred_batch_discards_released_alias_fence_and_preserves_caller_fence() {
|
||||
use crate::bucket::metadata_sys::ObjectLockConfigState;
|
||||
use crate::object_api::{NamespaceLockFence, NamespaceLockSignalTestFence, ObjectLockConfigSnapshot};
|
||||
use crate::storage_api_contracts::object::ObjectToDelete;
|
||||
use std::sync::Mutex;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
for lose_caller in [false, true] {
|
||||
let bucket = if lose_caller {
|
||||
"deferred-batch-lost-caller"
|
||||
} else {
|
||||
"deferred-batch-released-alias"
|
||||
};
|
||||
let (_dirs, store, disks) = fixture(bucket).await;
|
||||
put(&store, bucket, &[27; 4096], false).await;
|
||||
let transaction = applied(&store, &disks, bucket, None, disks.len()).await;
|
||||
let before = join_all(
|
||||
disks
|
||||
.iter()
|
||||
.map(|disk| disk.inspect_delete_transaction(bucket, "object", transaction)),
|
||||
)
|
||||
.await
|
||||
.into_iter()
|
||||
.collect::<Result<Vec<_>>>()
|
||||
.expect("capture pending recovery evidence");
|
||||
let mut set = (*store.pools[0].disk_set[0]).clone();
|
||||
set.lockers = vec![Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(
|
||||
rustfs_lock::GlobalLockManager::Enabled(Arc::new(rustfs_lock::FastObjectLockManager::new())),
|
||||
)))];
|
||||
set.ctx
|
||||
.update_erasure_type(crate::layout::endpoints::SetupType::DistErasure)
|
||||
.await;
|
||||
|
||||
let (caller_fence, caller_lost) = NamespaceLockFence::loss_handle_for_test();
|
||||
let released_alias_lost = Arc::new(AtomicBool::new(false));
|
||||
let signal_fence = Arc::new(Mutex::new(None));
|
||||
let captured_signal_fence = Arc::clone(&signal_fence);
|
||||
let captured_alias_lost = Arc::clone(&released_alias_lost);
|
||||
test_hooks::after_transaction_lock_acquired(
|
||||
bucket,
|
||||
"object",
|
||||
Arc::new(move |guard| {
|
||||
let signal = guard.lock_lost_signal().expect("distributed alias signal");
|
||||
let fence = NamespaceLockSignalTestFence::install_with_loss_handle(&signal, Arc::clone(&captured_alias_lost));
|
||||
// Retain the signal so a later allocation cannot reuse its
|
||||
// registry identity after the original guard is dropped.
|
||||
*captured_signal_fence.lock().expect("captured alias fence") = Some((fence, signal));
|
||||
}),
|
||||
);
|
||||
let replay_started = Arc::new(AtomicBool::new(false));
|
||||
let captured_replay_started = Arc::clone(&replay_started);
|
||||
let expired_alias = Arc::clone(&released_alias_lost);
|
||||
test_hooks::after_transaction_lock_acquired(
|
||||
bucket,
|
||||
"object",
|
||||
Arc::new(move |_| {
|
||||
// Acquiring the same transaction key again proves the first
|
||||
// lease was released. Only its copied fence becomes stale.
|
||||
expired_alias.store(true, Ordering::Release);
|
||||
caller_lost.store(lose_caller, Ordering::Release);
|
||||
captured_replay_started.store(true, Ordering::Release);
|
||||
}),
|
||||
);
|
||||
|
||||
let (deleted, errors) = set
|
||||
.delete_objects(
|
||||
bucket,
|
||||
vec![ObjectToDelete {
|
||||
object_name: "object".into(),
|
||||
..Default::default()
|
||||
}],
|
||||
ObjectOptions {
|
||||
namespace_lock_fence: Some(caller_fence),
|
||||
object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(
|
||||
ObjectLockConfigState::ConfirmedAbsent,
|
||||
))),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
replay_started.load(Ordering::Acquire),
|
||||
"missing metadata must take the deferred replay path"
|
||||
);
|
||||
assert!(
|
||||
released_alias_lost.load(Ordering::Acquire),
|
||||
"the completed batch alias must be marked lost"
|
||||
);
|
||||
if lose_caller {
|
||||
assert!(errors[0].is_some(), "losing the original caller fence must reject deferred replay");
|
||||
for (disk, original) in disks.iter().zip(before) {
|
||||
let after = disk
|
||||
.inspect_delete_transaction(bucket, "object", transaction)
|
||||
.await
|
||||
.expect("retained proof");
|
||||
assert_eq!(
|
||||
after.fingerprint, original.fingerprint,
|
||||
"lost caller authority must preserve every prepared journal"
|
||||
);
|
||||
}
|
||||
} else {
|
||||
assert!(
|
||||
errors.iter().all(Option::is_none),
|
||||
"released alias loss must not reject fresh recovery: {errors:?}"
|
||||
);
|
||||
assert!(
|
||||
!deleted[0].found,
|
||||
"fresh unversioned recovery commits absence before the idempotent batch retry"
|
||||
);
|
||||
for disk in &disks {
|
||||
assert!(
|
||||
discover(disk, bucket, Some("object"))
|
||||
.await
|
||||
.expect("final recovery evidence")
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(
|
||||
disk.read_version("", bucket, "object", "", &ReadOptions::default())
|
||||
.await
|
||||
.expect_err("replayed delete must remove the source"),
|
||||
DiskError::FileNotFound
|
||||
);
|
||||
}
|
||||
}
|
||||
drop(signal_fence);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_recovery_corrupt_bucket_does_not_prevent_other_bucket_rollback() {
|
||||
let bad_bucket = "recover-corrupt-bucket";
|
||||
|
||||
@@ -3634,6 +3634,18 @@ impl SetDisks {
|
||||
return Ok((result, err.map(|e| e.into())));
|
||||
}
|
||||
|
||||
// Keep the transaction key through both replay and the metadata heal
|
||||
// based on that replay. The raw target key always precedes this key.
|
||||
let transaction_guard =
|
||||
if _write_lock_guard.is_some() && !opts.read_repair && !crate::bucket::utils::is_meta_bucketname(bucket) {
|
||||
Some(self.acquire_delete_transaction_lock(bucket, object).await?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let locks_lost = || {
|
||||
_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
};
|
||||
let mut recovery_removed_version = false;
|
||||
if !opts.read_repair
|
||||
&& _write_lock_guard.is_some()
|
||||
@@ -3655,12 +3667,17 @@ impl SetDisks {
|
||||
dry_run: opts.dry_run,
|
||||
explicit_delete,
|
||||
repair_marker_source: true,
|
||||
transaction_guard: transaction_guard.clone(),
|
||||
namespace_owner: (!opts.dry_run).then(|| {
|
||||
Arc::new((scope.clone(), self.ctx.begin_namespace_commit(), _write_lock_guard.clone()))
|
||||
as Arc<dyn Send + Sync>
|
||||
Arc::new((
|
||||
scope.clone(),
|
||||
self.ctx.begin_namespace_commit(),
|
||||
_write_lock_guard.clone(),
|
||||
transaction_guard.clone(),
|
||||
)) as Arc<dyn Send + Sync>
|
||||
}),
|
||||
..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &|| {
|
||||
scope.check().is_ok() && !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
|
||||
scope.check().is_ok() && !locks_lost()
|
||||
})
|
||||
},
|
||||
)
|
||||
@@ -3752,7 +3769,7 @@ impl SetDisks {
|
||||
&& errs
|
||||
.iter()
|
||||
.all(|err| matches!(err, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound)))
|
||||
&& !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
|
||||
&& !locks_lost()
|
||||
{
|
||||
*absence = Some(HealedObjectAbsence {
|
||||
pool_index: self.pool_index,
|
||||
@@ -3767,8 +3784,10 @@ impl SetDisks {
|
||||
// Pass no_lock=true since we already obtained write lock (or are already called with no_lock=true)
|
||||
let mut inner_opts = *opts;
|
||||
inner_opts.no_lock = true;
|
||||
let commit_fence = || !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|
||||
let namespace_owner = _write_lock_guard.as_ref().map(|guard| guard.clone() as Arc<dyn Send + Sync>);
|
||||
let commit_fence = || !locks_lost();
|
||||
let namespace_owner = _write_lock_guard
|
||||
.as_ref()
|
||||
.map(|guard| Arc::new((Arc::clone(guard), transaction_guard.clone())) as Arc<dyn Send + Sync>);
|
||||
let (result, err) = pin_object_heal_future(|| {
|
||||
self.heal_object_with_explicit_version_regen(
|
||||
bucket,
|
||||
@@ -3815,7 +3834,7 @@ impl SetDisks {
|
||||
result,
|
||||
err.map(Into::into),
|
||||
absence,
|
||||
_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()),
|
||||
locks_lost(),
|
||||
bucket,
|
||||
object,
|
||||
));
|
||||
@@ -3827,7 +3846,7 @@ impl SetDisks {
|
||||
result,
|
||||
err.map(Into::into),
|
||||
absence,
|
||||
_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()),
|
||||
locks_lost(),
|
||||
bucket,
|
||||
object,
|
||||
))
|
||||
|
||||
@@ -44,6 +44,21 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
|
||||
|
||||
#[tracing::instrument(level = "trace", skip(self))]
|
||||
async fn new_ns_lock(&self, bucket: &str, object: &str) -> Result<NamespaceLockWrapper> {
|
||||
// Lock the file the disk resolves, not the spelling: on Windows `a\b`
|
||||
// and `a/b` share one xl.meta, so they must share one lock. The key is
|
||||
// folded by the requesting node only; lock servers compare it verbatim.
|
||||
let resource = ObjectKey {
|
||||
bucket: Arc::from(bucket),
|
||||
object: Arc::from(object_name_disk_identity(object).as_ref()),
|
||||
version: None,
|
||||
};
|
||||
|
||||
Ok(self.namespace_lock_for_key(resource).await)
|
||||
}
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
async fn namespace_lock_for_key(&self, resource: ObjectKey) -> NamespaceLockWrapper {
|
||||
// Resolved from this set's own instance context (backlog#1052), not the
|
||||
// ambient facade: the facade tracks whichever context is currently
|
||||
// published process-wide, so a second instance (or, in tests, another
|
||||
@@ -69,20 +84,33 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
|
||||
NamespaceLock::with_local_manager_shared(self.set_lock_namespace.clone(), self.local_lock_manager.clone())
|
||||
};
|
||||
|
||||
// Lock the file the disk resolves, not the spelling: on Windows `a\b`
|
||||
// and `a/b` share one xl.meta, so they must share one lock. The key is
|
||||
// folded by the requesting node only; lock servers compare it verbatim.
|
||||
let resource = ObjectKey {
|
||||
bucket: Arc::from(bucket),
|
||||
object: Arc::from(object_name_disk_identity(object).as_ref()),
|
||||
version: None,
|
||||
};
|
||||
|
||||
Ok(NamespaceLockWrapper::new(set_lock, resource, self.locker_owner.clone()))
|
||||
NamespaceLockWrapper::new(set_lock, resource, self.locker_owner.clone())
|
||||
}
|
||||
|
||||
/// Raw object locks precede this set-local transaction lock. Its versioned
|
||||
/// resource cannot collide with normal namespace keys, which use None.
|
||||
/// Keeping the target set's clients avoids nesting overlapping raw quorums.
|
||||
pub(in crate::set_disk) async fn acquire_delete_transaction_lock(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
|
||||
let resource = ObjectKey::with_version(
|
||||
bucket,
|
||||
object_name_disk_identity(object).as_ref(),
|
||||
format!("rustfs:delete-transaction:{}:{}", self.pool_index, self.set_index),
|
||||
);
|
||||
let guard = self
|
||||
.namespace_lock_for_key(resource)
|
||||
.await
|
||||
.get_write_lock(super::super::get_lock_acquire_timeout())
|
||||
.await
|
||||
.map_err(|error| self.map_namespace_lock_error(bucket, object, "delete_transaction", error))?;
|
||||
#[cfg(test)]
|
||||
super::delete_recovery::test_hooks::run_after_transaction_lock_acquired(bucket, object, &guard);
|
||||
Ok(Arc::new(guard))
|
||||
}
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
#[allow(dead_code, reason = "lock diagnostics formatter with no caller in this port (backlog#1823)")]
|
||||
pub(in crate::set_disk) fn format_lock_error(&self, bucket: &str, object: &str, mode: &str, err: &LockResult) -> String {
|
||||
match err {
|
||||
|
||||
@@ -43,8 +43,9 @@ use super::super::{
|
||||
complete_multipart_part_error, complete_multipart_part_error_result, complete_part_checksum, contains_key_str,
|
||||
create_bitrot_writer, debug, disk, error, file_info_is_valid_for_metadata, get_complete_multipart_md5, get_header_map,
|
||||
get_str, insert_str, is_err_object_not_found, is_err_version_not_found, is_min_allowed_part_size,
|
||||
log_multipart_write_quorum_failure, parts_after_marker, path_join_buf, prepare_complete_multipart_parts,
|
||||
record_compression_total_memory, reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
|
||||
log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
|
||||
prepare_complete_multipart_parts, record_compression_total_memory, record_multipart_write_quorum_failure,
|
||||
reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
|
||||
restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata,
|
||||
to_object_err, warn,
|
||||
};
|
||||
@@ -233,6 +234,19 @@ fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, PartialEq, Eq)]
|
||||
enum MultipartMetadataQuorum {
|
||||
Read,
|
||||
Write,
|
||||
Cleanup,
|
||||
}
|
||||
|
||||
impl From<bool> for MultipartMetadataQuorum {
|
||||
fn from(write: bool) -> Self {
|
||||
if write { Self::Write } else { Self::Read }
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) struct StaleMultipartCleanupGuard {
|
||||
file_info: FileInfo,
|
||||
upload_path: String,
|
||||
@@ -410,6 +424,8 @@ impl StaleMultipartCleanupGuard {
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum MultipartCommitPause {
|
||||
NewUploadBeforeLockLost,
|
||||
#[cfg(test)]
|
||||
ListPartsAfterMetadata,
|
||||
PutPartBeforeLockAcquire,
|
||||
PutPartBeforeLockLost,
|
||||
PutPartAfterCapacityAdmission,
|
||||
@@ -824,7 +840,8 @@ where
|
||||
}
|
||||
|
||||
if successful_responses < read_quorum {
|
||||
if empty_upload_fallback_possible(successful_responses, &errs) {
|
||||
// A single missing directory cannot speak for unavailable disks.
|
||||
if !errs.is_empty() && errs.iter().all(|err| matches!(err, Some(DiskError::FileNotFound))) {
|
||||
return Err(DiskError::FileNotFound);
|
||||
}
|
||||
|
||||
@@ -834,6 +851,51 @@ where
|
||||
Ok((errs, object_parts))
|
||||
}
|
||||
|
||||
// Size-dependent decisions must not discard a part merely because only a
|
||||
// minority of its files was observed. All participants must answer, and each
|
||||
// observed part must have a complete file/metadata pair on a read quorum.
|
||||
fn validate_part_size_observations(
|
||||
errs: &[Option<DiskError>],
|
||||
object_parts: &[Vec<String>],
|
||||
read_quorum: usize,
|
||||
) -> disk::error::Result<Vec<usize>> {
|
||||
if errs.iter().any(Option::is_some) {
|
||||
return Err(DiskError::ErasureReadQuorum);
|
||||
}
|
||||
let mut counts: HashMap<usize, usize> = HashMap::new();
|
||||
for entries in object_parts {
|
||||
let mut observed: HashMap<usize, u8> = HashMap::new();
|
||||
for entry in entries {
|
||||
let Some(name) = entry.strip_prefix("part.") else {
|
||||
continue;
|
||||
};
|
||||
let (number, flag) = match name.strip_suffix(".meta") {
|
||||
Some(number) => (number, 2),
|
||||
None => (name, 1),
|
||||
};
|
||||
let parsed = number.parse::<usize>().map_err(|_| DiskError::FileCorrupt)?;
|
||||
if !(1..=MAX_PARTS_COUNT).contains(&parsed)
|
||||
|| !number.as_bytes().first().is_some_and(|digit| (b'1'..=b'9').contains(digit))
|
||||
{
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
*observed.entry(parsed).or_default() |= flag;
|
||||
}
|
||||
for (number, flags) in observed {
|
||||
if flags != 3 {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
*counts.entry(number).or_default() += 1;
|
||||
}
|
||||
}
|
||||
if counts.values().any(|&count| count < read_quorum) {
|
||||
return Err(DiskError::ErasureReadQuorum);
|
||||
}
|
||||
let mut part_numbers: Vec<_> = counts.into_keys().collect();
|
||||
part_numbers.sort_unstable();
|
||||
Ok(part_numbers)
|
||||
}
|
||||
|
||||
fn reduce_quorum_part_numbers(object_parts: Vec<Vec<String>>, read_quorum: usize) -> Vec<usize> {
|
||||
let mut part_quorum_map: HashMap<usize, usize> = HashMap::new();
|
||||
|
||||
@@ -1004,7 +1066,7 @@ impl SetDisks {
|
||||
]),
|
||||
SLASH_SEPARATOR
|
||||
);
|
||||
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
|
||||
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum, true).await {
|
||||
Ok(parts) => parts,
|
||||
Err(DiskError::FileNotFound) => return Ok(0),
|
||||
Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
|
||||
@@ -1016,28 +1078,45 @@ impl SetDisks {
|
||||
.iter()
|
||||
.map(|number| format!("{part_path}part.{number}.meta"))
|
||||
.collect::<Vec<_>>();
|
||||
let existing_parts =
|
||||
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
|
||||
let existing_parts = Self::read_parts(
|
||||
&online_disks,
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&part_meta_paths,
|
||||
&part_numbers,
|
||||
fi.write_quorum(self.default_write_quorum()),
|
||||
)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
|
||||
|
||||
existing_parts.into_iter().try_fold(0_u64, |total, part| {
|
||||
if part.error.is_some() || part.number == replacing_part {
|
||||
return if part.error.is_some() {
|
||||
Err(Error::PartMissingOrCorrupt)
|
||||
} else {
|
||||
Ok(total)
|
||||
};
|
||||
}
|
||||
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
if existing_parts.len() != part_numbers.len() {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
}
|
||||
let transformed = fi.is_compressed() || should_persist_encryption_original_size(&fi.metadata);
|
||||
existing_parts
|
||||
.into_iter()
|
||||
.zip(part_numbers)
|
||||
.try_fold(0_u64, |total, (part, number)| {
|
||||
if part.number != number {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
}
|
||||
if part.error.is_some() || part.number == replacing_part {
|
||||
return if part.error.is_some() {
|
||||
Err(Error::PartMissingOrCorrupt)
|
||||
} else {
|
||||
Ok(total)
|
||||
};
|
||||
}
|
||||
let part_size = multipart_part_logical_size(part.size, part.actual_size, transformed)
|
||||
.and_then(|size| u64::try_from(size).ok())
|
||||
.ok_or(Error::PartMissingOrCorrupt)?;
|
||||
total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async fn discover_multipart_upload_paths(
|
||||
@@ -1139,7 +1218,7 @@ impl SetDisks {
|
||||
}
|
||||
let upload_id = runtime_sources::deployment_upload_id(raw_upload_id);
|
||||
let file_info = match self
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload_id, &upload_path, false, true)
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload_id, &upload_path, MultipartMetadataQuorum::Read, true)
|
||||
.await
|
||||
{
|
||||
Ok((file_info, _)) => file_info,
|
||||
@@ -1226,7 +1305,7 @@ impl SetDisks {
|
||||
.acquire_write_lock_diag("abort_data_movement_multipart", RUSTFS_META_MULTIPART_BUCKET, &upload_path)
|
||||
.await?;
|
||||
let file_info = match self
|
||||
.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_path, true, true)
|
||||
.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_path, MultipartMetadataQuorum::Write, true)
|
||||
.await
|
||||
{
|
||||
Ok((file_info, _)) => file_info,
|
||||
@@ -1269,6 +1348,7 @@ impl SetDisks {
|
||||
disks: &[Option<DiskStore>],
|
||||
part_path: &str,
|
||||
read_quorum: usize,
|
||||
strict_size: bool,
|
||||
) -> disk::error::Result<Vec<usize>> {
|
||||
let mut futures = Vec::with_capacity(disks.len());
|
||||
let part_path = part_path.to_string();
|
||||
@@ -1277,21 +1357,24 @@ impl SetDisks {
|
||||
let part_path = part_path.clone();
|
||||
futures.push(async move {
|
||||
if let Some(disk) = disk {
|
||||
disk.list_dir(RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_MULTIPART_BUCKET, part_path.as_str(), -1)
|
||||
match disk
|
||||
.list_dir(RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_MULTIPART_BUCKET, part_path.as_str(), -1)
|
||||
.await
|
||||
{
|
||||
Err(DiskError::FileNotFound) if strict_size => Ok(Vec::new()),
|
||||
result => result,
|
||||
}
|
||||
} else {
|
||||
Err(DiskError::DiskNotFound)
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
let mut errs = Vec::with_capacity(disks.len());
|
||||
let mut object_parts = Vec::with_capacity(disks.len());
|
||||
|
||||
let (collected_errs, collected_parts) = collect_list_parts_results(futures, read_quorum).await?;
|
||||
errs.extend(collected_errs);
|
||||
object_parts.extend(collected_parts);
|
||||
let (errs, object_parts) = collect_list_parts_results(futures, read_quorum).await?;
|
||||
|
||||
if strict_size {
|
||||
return validate_part_size_observations(&errs, &object_parts, read_quorum);
|
||||
}
|
||||
if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum) {
|
||||
return Err(err);
|
||||
}
|
||||
@@ -1333,7 +1416,7 @@ impl SetDisks {
|
||||
data_movement: bool,
|
||||
) -> Result<(FileInfo, Vec<FileInfo>)> {
|
||||
let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, data_movement)?;
|
||||
self.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, write, false)
|
||||
self.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, write.into(), false)
|
||||
.await
|
||||
}
|
||||
|
||||
@@ -1373,7 +1456,7 @@ impl SetDisks {
|
||||
object: &str,
|
||||
upload_id: &str,
|
||||
upload_id_path: &str,
|
||||
write: bool,
|
||||
quorum_mode: MultipartMetadataQuorum,
|
||||
include_recovery: bool,
|
||||
) -> Result<(FileInfo, Vec<FileInfo>)> {
|
||||
// Slow metadata I/O must not hold the topology registry's read lock.
|
||||
@@ -1489,22 +1572,26 @@ impl SetDisks {
|
||||
}
|
||||
|
||||
let mut quorum = read_quorum as usize;
|
||||
if write {
|
||||
if quorum_mode != MultipartMetadataQuorum::Read {
|
||||
quorum = write_quorum as usize;
|
||||
|
||||
if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, quorum) {
|
||||
log_multipart_write_quorum_failure(
|
||||
MultipartWriteQuorumContext {
|
||||
stage: MULTIPART_WRITE_QUORUM_UPLOAD_METADATA,
|
||||
bucket,
|
||||
object,
|
||||
upload_id,
|
||||
part_number: None,
|
||||
},
|
||||
&errs,
|
||||
quorum,
|
||||
&err,
|
||||
);
|
||||
if quorum_mode == MultipartMetadataQuorum::Write {
|
||||
log_multipart_write_quorum_failure(
|
||||
MultipartWriteQuorumContext {
|
||||
stage: MULTIPART_WRITE_QUORUM_UPLOAD_METADATA,
|
||||
bucket,
|
||||
object,
|
||||
upload_id,
|
||||
part_number: None,
|
||||
},
|
||||
&errs,
|
||||
quorum,
|
||||
&err,
|
||||
);
|
||||
} else {
|
||||
record_multipart_write_quorum_failure(MULTIPART_WRITE_QUORUM_UPLOAD_METADATA, &errs, quorum);
|
||||
}
|
||||
return Err(map_error(err));
|
||||
}
|
||||
} else if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, quorum) {
|
||||
@@ -1523,7 +1610,7 @@ impl SetDisks {
|
||||
.acquire_write_lock_diag("stale_multipart_cleanup", RUSTFS_META_MULTIPART_BUCKET, upload_path)
|
||||
.await?;
|
||||
let (file_info, _) = self
|
||||
.check_multipart_upload_path_exists("", "", "", upload_path, true, true)
|
||||
.check_multipart_upload_path_exists("", "", "", upload_path, MultipartMetadataQuorum::Cleanup, true)
|
||||
.await?;
|
||||
let write_quorum = file_info.write_quorum(self.default_write_quorum());
|
||||
Ok(StaleMultipartCleanupGuard {
|
||||
@@ -2634,6 +2721,8 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
|
||||
.await?;
|
||||
#[cfg(test)]
|
||||
pause_multipart_commit(bucket, object, MultipartCommitPause::ListPartsAfterMetadata).await;
|
||||
|
||||
let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, opts.data_movement)?;
|
||||
|
||||
@@ -2682,16 +2771,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
SLASH_SEPARATOR
|
||||
);
|
||||
|
||||
let mut part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
|
||||
Ok(parts) => parts,
|
||||
Err(err) => {
|
||||
if err == DiskError::FileNotFound {
|
||||
return Ok(ret);
|
||||
}
|
||||
let mut part_numbers =
|
||||
match Self::list_parts(&online_disks, &part_path, read_quorum, opts.strict_multipart_part_sizes).await {
|
||||
Ok(parts) => parts,
|
||||
Err(err) => {
|
||||
if err == DiskError::FileNotFound {
|
||||
return Ok(ret);
|
||||
}
|
||||
|
||||
return Err(to_object_err(err.into(), vec![bucket, object]));
|
||||
}
|
||||
};
|
||||
return Err(to_object_err(err.into(), vec![bucket, object]));
|
||||
}
|
||||
};
|
||||
|
||||
if part_numbers.is_empty() {
|
||||
return Ok(ret);
|
||||
@@ -2708,11 +2798,29 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
.map(|v| format!("{part_path}part.{v}.meta"))
|
||||
.collect::<Vec<String>>();
|
||||
|
||||
let object_parts =
|
||||
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e.into(), vec![bucket, object, upload_id]))?;
|
||||
let object_parts = Self::read_parts(
|
||||
&online_disks,
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&part_meta_paths,
|
||||
&part_numbers,
|
||||
if opts.strict_multipart_part_sizes {
|
||||
fi.write_quorum(self.default_write_quorum())
|
||||
} else {
|
||||
read_quorum
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e.into(), vec![bucket, object, upload_id]))?;
|
||||
|
||||
if opts.strict_multipart_part_sizes
|
||||
&& (object_parts.len() != part_numbers.len()
|
||||
|| object_parts
|
||||
.iter()
|
||||
.zip(&part_numbers)
|
||||
.any(|(part, number)| part.number != *number || part.error.is_some() || part.etag.is_empty()))
|
||||
{
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
}
|
||||
let mut count = max_parts;
|
||||
|
||||
for part in object_parts.iter() {
|
||||
@@ -3032,7 +3140,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
// its full write quorum and the upload's incarnation/lifecycle fences.
|
||||
let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, opts.data_movement)?;
|
||||
let (fi, _) = self
|
||||
.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, false, true)
|
||||
.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, MultipartMetadataQuorum::Read, true)
|
||||
.await?;
|
||||
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
|
||||
@@ -3192,7 +3300,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
}
|
||||
SourceRecovery::Missing => {
|
||||
return match self
|
||||
.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, false, true)
|
||||
.check_multipart_upload_path_exists(
|
||||
bucket,
|
||||
object,
|
||||
upload_id,
|
||||
&upload_id_path,
|
||||
MultipartMetadataQuorum::Read,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => Err(Error::ErasureWriteQuorum),
|
||||
@@ -7492,7 +7607,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn collect_list_parts_results_returns_file_not_found_for_empty_upload_dirs() {
|
||||
async fn collect_list_parts_results_rejects_unavailable_empty_upload_dirs() {
|
||||
let tasks: Vec<_> = vec![
|
||||
(5_u64, Err(DiskError::FileNotFound)),
|
||||
(10, Err(DiskError::DiskNotFound)),
|
||||
@@ -7507,11 +7622,372 @@ mod tests {
|
||||
|
||||
let err = collect_list_parts_results(tasks, 2)
|
||||
.await
|
||||
.expect_err("missing multipart directories should be treated as empty uploads");
|
||||
.expect_err("an unavailable disk cannot prove an empty upload");
|
||||
|
||||
assert_eq!(err, DiskError::ErasureReadQuorum);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
#[serial]
|
||||
async fn list_parts_rejects_disk_loss_after_upload_metadata_was_read() {
|
||||
let (dirs, disks, set) = hermetic_set_disks(4).await;
|
||||
let bucket = "multipart-list-loss-after-metadata";
|
||||
let object = "object";
|
||||
make_bucket_on_all(&disks, bucket).await;
|
||||
let opts = ObjectOptions {
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (upload_id, _) = stage_upload_with_create_opts(&set, bucket, object, &[1; 32], &opts).await;
|
||||
let (fi, _) = set
|
||||
.check_upload_id_exists(bucket, object, &upload_id, false)
|
||||
.await
|
||||
.expect("read initial metadata");
|
||||
let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
|
||||
let part_dir = dirs[0]
|
||||
.path()
|
||||
.join(RUSTFS_META_MULTIPART_BUCKET)
|
||||
.join(upload_path)
|
||||
.join(fi.data_dir.expect("upload data directory").to_string());
|
||||
let held_dir = part_dir.with_extension("held-for-test");
|
||||
let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::ListPartsAfterMetadata);
|
||||
let query_set = Arc::clone(&set);
|
||||
let query_id = upload_id.clone();
|
||||
let query = tokio::spawn(async move {
|
||||
query_set
|
||||
.list_object_parts(bucket, object, &query_id, None, 100, &ObjectOptions::default())
|
||||
.await
|
||||
});
|
||||
barrier.wait_until_paused().await;
|
||||
tokio::fs::rename(&part_dir, &held_dir)
|
||||
.await
|
||||
.expect("simulate the surviving disk missing the part directory");
|
||||
{
|
||||
let mut live = set.disks.write().await;
|
||||
for slot in 1..4 {
|
||||
live[slot] = None;
|
||||
}
|
||||
}
|
||||
barrier.release();
|
||||
let result = query.await.expect("list task must finish");
|
||||
drop(barrier);
|
||||
tokio::fs::rename(&held_dir, &part_dir)
|
||||
.await
|
||||
.expect("restore the original directory");
|
||||
{
|
||||
let mut live = set.disks.write().await;
|
||||
for slot in 1..4 {
|
||||
live[slot] = Some(disks[slot].clone());
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
matches!(result, Err(StorageError::InsufficientReadQuorum(..))),
|
||||
"disk loss must fail rather than return an empty part list: {result:?}"
|
||||
);
|
||||
let recovered = set
|
||||
.list_object_parts(bucket, object, &upload_id, None, 100, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("listing resumes after the original disks return");
|
||||
assert_eq!(recovered.parts.len(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn collect_list_parts_results_proves_all_directories_missing() {
|
||||
let tasks = (0..4)
|
||||
.map(|_| async { Err::<Vec<String>, _>(DiskError::FileNotFound) })
|
||||
.collect();
|
||||
let err = collect_list_parts_results(tasks, 2)
|
||||
.await
|
||||
.expect_err("all missing directories are empty");
|
||||
assert_eq!(err, DiskError::FileNotFound);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn strict_multipart_part_sizes_reject_incomplete_observations() {
|
||||
let pair = || vec!["part.1".to_string(), "part.1.meta".to_string()];
|
||||
let all_online = vec![None; 4];
|
||||
assert!(validate_part_size_observations(&all_online, &[vec![], vec![], vec![], vec![]], 2).is_ok());
|
||||
assert_eq!(
|
||||
validate_part_size_observations(&all_online, &[pair(), pair(), vec![], vec![]], 2),
|
||||
Ok(vec![1])
|
||||
);
|
||||
let unordered = vec!["part.9.meta", "part.1", "part.9", "part.1.meta", "part.1"]
|
||||
.into_iter()
|
||||
.map(str::to_owned)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(
|
||||
validate_part_size_observations(&all_online, &[unordered.clone(), unordered, vec![], vec![]], 2),
|
||||
Ok(vec![1, 9])
|
||||
);
|
||||
assert_eq!(
|
||||
validate_part_size_observations(&all_online, &[pair(), vec![], vec![], vec![]], 2),
|
||||
Err(DiskError::ErasureReadQuorum)
|
||||
);
|
||||
assert_eq!(
|
||||
validate_part_size_observations(&all_online, &[vec!["part.1.meta".to_string()], pair(), pair(), vec![]], 2),
|
||||
Err(DiskError::FileCorrupt)
|
||||
);
|
||||
for invalid in ["part.0", "part.10001", "part.01", "part.+1", "part.1x"] {
|
||||
let entries = vec![invalid.to_string(), format!("{invalid}.meta")];
|
||||
assert_eq!(
|
||||
validate_part_size_observations(&all_online, &[entries.clone(), entries.clone(), entries.clone(), entries], 2),
|
||||
Err(DiskError::FileCorrupt),
|
||||
"noncanonical part names are not valid size evidence even on every disk"
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
validate_part_size_observations(
|
||||
&[None, None, None, Some(DiskError::DiskNotFound)],
|
||||
&[pair(), pair(), pair(), vec![]],
|
||||
2
|
||||
),
|
||||
Err(DiskError::ErasureReadQuorum)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn strict_multipart_capped_upload_rejects_unavailable_participant_and_recovers() {
|
||||
let (_dirs, disks, set) = hermetic_set_disks(4).await;
|
||||
let bucket = "strict-multipart-size-bucket";
|
||||
let object = "object";
|
||||
make_bucket_on_all(&disks, bucket).await;
|
||||
let mut metadata = HashMap::new();
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "10".to_string());
|
||||
let opts = ObjectOptions {
|
||||
max_parity: true,
|
||||
user_defined: metadata,
|
||||
..Default::default()
|
||||
};
|
||||
let (upload_id, _) = stage_upload_with_create_opts(&set, bucket, object, &[1; 6], &opts).await;
|
||||
|
||||
set.disks.write().await[3] = None;
|
||||
let mut part = PutObjReader::from_vec(vec![2; 4]);
|
||||
let err = set
|
||||
.put_object_part(bucket, object, &upload_id, 2, &mut part, &ObjectOptions::default())
|
||||
.await;
|
||||
assert!(err.is_err(), "an unavailable disk must not produce a size admission: {err:?}");
|
||||
set.disks.write().await[3] = Some(disks[3].clone());
|
||||
|
||||
// Two distinct part identities at read quorum cannot prove which size
|
||||
// belongs to the acknowledged write; require the write quorum instead.
|
||||
let (other_id, _) = stage_upload_with_create_opts(&set, bucket, "other", &[9; 2], &ObjectOptions::default()).await;
|
||||
let (fi, _) = set
|
||||
.check_upload_id_exists(bucket, object, &upload_id, false)
|
||||
.await
|
||||
.expect("read capped upload metadata");
|
||||
let (other_fi, _) = set
|
||||
.check_upload_id_exists(bucket, "other", &other_id, false)
|
||||
.await
|
||||
.expect("read other upload metadata");
|
||||
let path = format!(
|
||||
"{}/{}/part.1.meta",
|
||||
SetDisks::get_upload_id_dir(bucket, object, &upload_id),
|
||||
fi.data_dir.expect("data dir")
|
||||
);
|
||||
let other_path = format!(
|
||||
"{}/{}/part.1.meta",
|
||||
SetDisks::get_upload_id_dir(bucket, "other", &other_id),
|
||||
other_fi.data_dir.expect("data dir")
|
||||
);
|
||||
let mut originals = Vec::new();
|
||||
for disk in disks.iter().take(2) {
|
||||
originals.push(
|
||||
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
|
||||
.await
|
||||
.expect("save original part metadata"),
|
||||
);
|
||||
let conflicting = disk
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &other_path)
|
||||
.await
|
||||
.expect("read conflicting part metadata");
|
||||
disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, conflicting)
|
||||
.await
|
||||
.expect("install split part metadata");
|
||||
}
|
||||
let mut split = PutObjReader::from_vec(vec![5; 4]);
|
||||
let split_result = set
|
||||
.put_object_part(bucket, object, &upload_id, 2, &mut split, &ObjectOptions::default())
|
||||
.await;
|
||||
for (disk, original) in disks.iter().zip(originals) {
|
||||
disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
|
||||
.await
|
||||
.expect("restore original part metadata");
|
||||
}
|
||||
assert!(
|
||||
split_result.is_err(),
|
||||
"split part identities must not authorize a size decision: {split_result:?}"
|
||||
);
|
||||
|
||||
let mut originals = Vec::new();
|
||||
for disk in disks.iter().take(3) {
|
||||
let original = disk
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
|
||||
.await
|
||||
.expect("save original part metadata");
|
||||
let mut stale = ObjectPartInfo::unmarshal(&original).expect("decode original part metadata");
|
||||
stale.actual_size = 1;
|
||||
disk.write_all(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&path,
|
||||
Bytes::from(stale.marshal_msg().expect("encode stale part")),
|
||||
)
|
||||
.await
|
||||
.expect("install stale size on write quorum");
|
||||
originals.push(original);
|
||||
}
|
||||
let mut undercounted = PutObjReader::from_vec(vec![6; 4]);
|
||||
let result = set
|
||||
.put_object_part(bucket, object, &upload_id, 2, &mut undercounted, &ObjectOptions::default())
|
||||
.await;
|
||||
for (disk, original) in disks.iter().zip(originals) {
|
||||
disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
|
||||
.await
|
||||
.expect("restore original part metadata");
|
||||
}
|
||||
assert!(
|
||||
matches!(result, Err(StorageError::PartMissingOrCorrupt)),
|
||||
"smaller logical size must fail closed: {result:?}"
|
||||
);
|
||||
|
||||
let mut originals = Vec::new();
|
||||
for disk in disks.iter().take(3) {
|
||||
let original = disk
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
|
||||
.await
|
||||
.expect("save original part metadata");
|
||||
let mut wrong_identity = ObjectPartInfo::unmarshal(&original).expect("decode original part metadata");
|
||||
wrong_identity.number = 2;
|
||||
disk.write_all(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&path,
|
||||
Bytes::from(wrong_identity.marshal_msg().expect("encode wrong identity")),
|
||||
)
|
||||
.await
|
||||
.expect("install mismatched part identity on write quorum");
|
||||
originals.push(original);
|
||||
}
|
||||
let list_result = set
|
||||
.list_object_parts(
|
||||
bucket,
|
||||
object,
|
||||
&upload_id,
|
||||
None,
|
||||
100,
|
||||
&ObjectOptions {
|
||||
strict_multipart_part_sizes: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let mut replacement = PutObjReader::from_vec(vec![7; 4]);
|
||||
let result = set
|
||||
.put_object_part(bucket, object, &upload_id, 2, &mut replacement, &ObjectOptions::default())
|
||||
.await;
|
||||
for (disk, original) in disks.iter().zip(originals) {
|
||||
disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
|
||||
.await
|
||||
.expect("restore original part metadata");
|
||||
}
|
||||
assert!(
|
||||
matches!(list_result, Err(StorageError::PartMissingOrCorrupt)),
|
||||
"strict listing must reject mismatched part identity: {list_result:?}"
|
||||
);
|
||||
assert!(
|
||||
matches!(result, Err(StorageError::PartMissingOrCorrupt)),
|
||||
"mismatched part identity must not bypass the cap: {result:?}"
|
||||
);
|
||||
|
||||
let mut too_large = PutObjReader::from_vec(vec![3; 5]);
|
||||
let err = set
|
||||
.put_object_part(bucket, object, &upload_id, 2, &mut too_large, &ObjectOptions::default())
|
||||
.await;
|
||||
assert!(
|
||||
matches!(err, Err(StorageError::EntityTooLarge(11, 10))),
|
||||
"existing part must count: {err:?}"
|
||||
);
|
||||
let mut fits = PutObjReader::from_vec(vec![4; 4]);
|
||||
set.put_object_part(bucket, object, &upload_id, 2, &mut fits, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("the upload should resume at the exact limit after all disks recover");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn capped_multipart_concurrent_parts_cannot_exceed_logical_limit() {
|
||||
let (_dirs, disks, set) = hermetic_set_disks(4).await;
|
||||
let bucket = "concurrent-capped-multipart";
|
||||
let object = "object";
|
||||
make_bucket_on_all(&disks, bucket).await;
|
||||
let mut metadata = HashMap::new();
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "10".to_owned());
|
||||
let opts = ObjectOptions {
|
||||
max_parity: true,
|
||||
user_defined: metadata,
|
||||
..Default::default()
|
||||
};
|
||||
let upload = set
|
||||
.new_multipart_upload(bucket, object, &opts)
|
||||
.await
|
||||
.expect("create capped upload");
|
||||
let barrier = Arc::new(tokio::sync::Barrier::new(3));
|
||||
let mut tasks = Vec::new();
|
||||
for (number, byte) in [(1_usize, 0x41_u8), (2, 0x42)] {
|
||||
let set = Arc::clone(&set);
|
||||
let barrier = Arc::clone(&barrier);
|
||||
let upload_id = upload.upload_id.clone();
|
||||
tasks.push(tokio::spawn(async move {
|
||||
let mut body = PutObjReader::from_vec(vec![byte; 6]);
|
||||
barrier.wait().await;
|
||||
set.put_object_part(bucket, object, &upload_id, number, &mut body, &ObjectOptions::default())
|
||||
.await
|
||||
}));
|
||||
}
|
||||
barrier.wait().await;
|
||||
let mut accepted = Vec::new();
|
||||
let mut rejected = 0;
|
||||
for task in tasks {
|
||||
match tokio::time::timeout(Duration::from_secs(30), task)
|
||||
.await
|
||||
.expect("concurrent part deadline")
|
||||
.expect("part task")
|
||||
{
|
||||
Ok(part) => accepted.push(part),
|
||||
Err(StorageError::EntityTooLarge(12, 10)) => rejected += 1,
|
||||
Err(other) => panic!("unexpected concurrent admission result: {other:?}"),
|
||||
}
|
||||
}
|
||||
assert_eq!(accepted.len(), 1, "only one six-byte part fits the ten-byte limit");
|
||||
assert_eq!(rejected, 1);
|
||||
let listed = set
|
||||
.list_object_parts(bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("read final part listing");
|
||||
assert_eq!(listed.parts.len(), 1);
|
||||
assert_eq!(listed.parts[0].part_num, accepted[0].part_num);
|
||||
let winner = accepted.remove(0);
|
||||
let part_number = winner.part_num;
|
||||
set.clone()
|
||||
.complete_multipart_upload(
|
||||
bucket,
|
||||
object,
|
||||
&upload.upload_id,
|
||||
vec![CompletePart {
|
||||
part_num: part_number,
|
||||
etag: winner.etag,
|
||||
..Default::default()
|
||||
}],
|
||||
&ObjectOptions::default(),
|
||||
)
|
||||
.await
|
||||
.expect("complete the admitted part");
|
||||
let mut reader = set
|
||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("read completed object");
|
||||
let mut actual = Vec::new();
|
||||
reader.stream.read_to_end(&mut actual).await.expect("read whole object");
|
||||
assert_eq!(actual, vec![if part_number == 1 { 0x41 } else { 0x42 }; 6]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn collect_list_parts_results_fails_early_when_file_not_found_fallback_is_impossible() {
|
||||
let started = Instant::now();
|
||||
@@ -8131,9 +8607,7 @@ mod tests {
|
||||
assert!(listed.parts.is_empty(), "part lock loss before rename must not publish the part");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn put_object_part_fences_upload_lock_loss_before_rename() {
|
||||
async fn put_object_part_fences_upload_lock_loss_before_rename_case(capped: bool) {
|
||||
let target = Arc::new(std::sync::RwLock::new(None));
|
||||
let refresh_calls = Arc::new(AtomicUsize::new(0));
|
||||
let lockers: Vec<Arc<dyn LockClient>> = (0..4)
|
||||
@@ -8145,10 +8619,12 @@ mod tests {
|
||||
let bucket = "multipart-put-part-lock-loss-bucket";
|
||||
let object = "object";
|
||||
make_bucket_on_all(&disk_stores, bucket).await;
|
||||
let mut create_opts = ObjectOptions::default();
|
||||
if capped {
|
||||
insert_str(&mut create_opts.user_defined, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "8192".to_owned());
|
||||
}
|
||||
let upload = temp_env::async_with_vars([(ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, Some("false"))], async {
|
||||
set_disks
|
||||
.new_multipart_upload(bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
set_disks.new_multipart_upload(bucket, object, &create_opts).await
|
||||
})
|
||||
.await
|
||||
.expect("multipart upload should be created");
|
||||
@@ -8199,6 +8675,18 @@ mod tests {
|
||||
assert!(listed.parts.is_empty(), "lock loss before rename must not publish the part");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn put_object_part_fences_upload_lock_loss_before_rename() {
|
||||
put_object_part_fences_upload_lock_loss_before_rename_case(false).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn capped_multipart_part_fences_upload_lock_loss_before_rename() {
|
||||
put_object_part_fences_upload_lock_loss_before_rename_case(true).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn complete_encrypted_multipart_marks_quorum_validated_layout() {
|
||||
temp_env::async_with_vars([(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("true"))], async {
|
||||
@@ -11133,7 +11621,7 @@ mod tests {
|
||||
*disk = None;
|
||||
}
|
||||
let (info, _) = set
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
|
||||
.await
|
||||
.expect("four primary plus four recovery copies reach metadata read quorum");
|
||||
assert_eq!(info.name, path);
|
||||
@@ -11154,7 +11642,7 @@ mod tests {
|
||||
*disk = None;
|
||||
}
|
||||
assert!(
|
||||
set.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
|
||||
set.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
|
||||
.await
|
||||
.is_err(),
|
||||
"two files on each of four disks must not count as eight voters"
|
||||
@@ -14471,7 +14959,7 @@ mod tests {
|
||||
object,
|
||||
&u_new,
|
||||
&SetDisks::get_upload_id_dir(bucket, object, &u_new),
|
||||
true,
|
||||
MultipartMetadataQuorum::Write,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
@@ -14516,7 +15004,7 @@ mod tests {
|
||||
object,
|
||||
&upload,
|
||||
&SetDisks::get_upload_id_dir(bucket, object, &upload),
|
||||
true,
|
||||
MultipartMetadataQuorum::Write,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
@@ -14599,7 +15087,7 @@ mod tests {
|
||||
assert_completion_files_preserved(&disk_stores, RUSTFS_META_MULTIPART_BUCKET, &source_files).await;
|
||||
set_disks
|
||||
.check_multipart_upload_path_exists(
|
||||
bucket, object, &u_new, &SetDisks::get_upload_id_dir(bucket, object, &u_new), true, true,
|
||||
bucket, object, &u_new, &SetDisks::get_upload_id_dir(bucket, object, &u_new), MultipartMetadataQuorum::Write, true,
|
||||
)
|
||||
.await
|
||||
.expect("upload authority must survive the post-commit crash");
|
||||
|
||||
@@ -363,6 +363,202 @@ mod native_abort_closure_tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stale_cleanup_reclaims_certified_one_and_two_disk_residuals_after_reopen() {
|
||||
for survivors in [1, 2] {
|
||||
let (dirs, disks, set) = hermetic_set_disks(4).await;
|
||||
let bucket = "native-stale-certified";
|
||||
let object = format!("survivors-{survivors}");
|
||||
make_bucket_on_all(&disks, bucket).await;
|
||||
let opts = ObjectOptions {
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (upload, _) = stage_upload_with_create_opts(&set, bucket, &object, &[0x61; 4096], &opts).await;
|
||||
let path = SetDisks::get_upload_id_dir(bucket, &object, &upload);
|
||||
let before = originals(&disks, &path).await;
|
||||
{
|
||||
let physical = disks.iter().cloned().map(Some).collect::<Vec<_>>();
|
||||
let SourceCleanupAuthority::Native { certificate, snapshots } =
|
||||
prepare_source_cleanup(&physical, bucket, &object, &path, true, &set.multipart_proof_verification_context())
|
||||
.await
|
||||
.expect("close every original and persist the certificate")
|
||||
else {
|
||||
panic!("native upload must retain a native certificate")
|
||||
};
|
||||
for slot in 0..(4 - survivors) {
|
||||
let response = operation(
|
||||
&disks[slot],
|
||||
&path,
|
||||
MultipartSourceOperation::DeleteAborted {
|
||||
expected: snapshots[slot].clone().expect("actual certified source"),
|
||||
certificate: certificate.as_ref().clone(),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(response, MultipartSourceResponse::AbortDeleted(_)));
|
||||
}
|
||||
}
|
||||
let (reopened, cold) = cold_reopen(disks, set).await;
|
||||
for slot in (4 - survivors)..4 {
|
||||
assert_eq!(
|
||||
retained_native_multipart_original(&reopened[slot], slot, bucket, &object, &path).await,
|
||||
before[slot]
|
||||
);
|
||||
}
|
||||
cold.lock_stale_multipart_cleanup(&path)
|
||||
.await
|
||||
.expect("complete certificate authorizes the background retry")
|
||||
.delete(&cold)
|
||||
.await
|
||||
.expect("background retry removes certified survivors");
|
||||
for dir in &dirs {
|
||||
assert!(!dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
|
||||
}
|
||||
assert!(
|
||||
cold.lock_stale_multipart_cleanup(&path).await.is_err(),
|
||||
"a repeated pass must not recreate the retired upload"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn stale_certified_residual_survives_lock_loss_then_converges() {
|
||||
let target = Arc::new(std::sync::RwLock::new(None));
|
||||
let refresh_calls = Arc::new(AtomicUsize::new(0));
|
||||
let lockers: Vec<Arc<dyn LockClient>> = (0..4)
|
||||
.map(|_| {
|
||||
Arc::new(SelectiveLockLossClient::new(Arc::clone(&target), Arc::clone(&refresh_calls))) as Arc<dyn LockClient>
|
||||
})
|
||||
.collect();
|
||||
let (dirs, disks, set) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
|
||||
let bucket = "stale-certified-lock-loss";
|
||||
let object = "object";
|
||||
make_bucket_on_all(&disks, bucket).await;
|
||||
let opts = ObjectOptions {
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (upload, _) = stage_upload_with_create_opts(&set, bucket, object, &[0x63; 4096], &opts).await;
|
||||
let path = SetDisks::get_upload_id_dir(bucket, object, &upload);
|
||||
let (fi, _) = set
|
||||
.check_upload_id_exists(bucket, object, &upload, false)
|
||||
.await
|
||||
.expect("source metadata before closure");
|
||||
let part_path = format!("{path}/{}/part.1", fi.data_dir.expect("source data directory"));
|
||||
let original = originals(&disks, &path).await;
|
||||
let surviving_part = disks[3]
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
|
||||
.await
|
||||
.expect("snapshot surviving shard");
|
||||
{
|
||||
let physical = disks.iter().cloned().map(Some).collect::<Vec<_>>();
|
||||
let SourceCleanupAuthority::Native { certificate, snapshots } =
|
||||
prepare_source_cleanup(&physical, bucket, object, &path, true, &set.multipart_proof_verification_context())
|
||||
.await
|
||||
.expect("all original participants close and persist the certificate")
|
||||
else {
|
||||
panic!("native certificate required")
|
||||
};
|
||||
let authority = set.multipart_proof_verification_context();
|
||||
let mut ids = certificate.disk_ids.iter().copied().map(Some).collect::<Vec<_>>();
|
||||
ids[0] = None;
|
||||
assert_eq!(
|
||||
crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
|
||||
Err(DiskError::ErasureWriteQuorum),
|
||||
"an unavailable participant is a retryable proof gap"
|
||||
);
|
||||
ids[1] = Some(Uuid::new_v4());
|
||||
assert_eq!(
|
||||
crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
|
||||
Err(DiskError::FileCorrupt),
|
||||
"a missing participant cannot hide a known identity conflict"
|
||||
);
|
||||
ids[0] = Some(Uuid::new_v4());
|
||||
ids[1] = Some(certificate.disk_ids[1]);
|
||||
assert_eq!(
|
||||
crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
|
||||
Err(DiskError::FileCorrupt),
|
||||
"a known different identity remains a certificate conflict"
|
||||
);
|
||||
for slot in 0..3 {
|
||||
let response = operation(
|
||||
&disks[slot],
|
||||
&path,
|
||||
MultipartSourceOperation::DeleteAborted {
|
||||
expected: snapshots[slot].clone().expect("certified source"),
|
||||
certificate: certificate.as_ref().clone(),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(response, MultipartSourceResponse::AbortDeleted(_)));
|
||||
}
|
||||
}
|
||||
assert!(dirs[3].path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
|
||||
let offline = set.disks.write().await[0]
|
||||
.take()
|
||||
.expect("take an original participant offline");
|
||||
let rejected = set.lock_stale_multipart_cleanup(&path).await;
|
||||
set.disks.write().await[0] = Some(offline);
|
||||
let err = rejected
|
||||
.err()
|
||||
.expect("a certificate must not substitute for an unavailable participant");
|
||||
assert!(
|
||||
matches!(err, StorageError::ErasureWriteQuorum | StorageError::InsufficientWriteQuorum(..)),
|
||||
"the missing participant must be reported as a quorum failure: {err:?}"
|
||||
);
|
||||
assert_eq!(retained_native_multipart_original(&disks[3], 3, bucket, object, &path).await, original[3]);
|
||||
assert_eq!(
|
||||
disks[3]
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
|
||||
.await
|
||||
.expect("surviving shard while a participant was offline"),
|
||||
surviving_part
|
||||
);
|
||||
tokio::time::pause();
|
||||
*target.write().expect("lock-loss target") =
|
||||
Some(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, path.clone()));
|
||||
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
|
||||
let guard = set
|
||||
.lock_stale_multipart_cleanup(&path)
|
||||
.await
|
||||
.expect("certificate admits the one-disk residual");
|
||||
tokio::time::advance(Duration::from_secs(11)).await;
|
||||
for _ in 0..20 {
|
||||
if guard.is_lock_lost() {
|
||||
break;
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
assert!(refresh_calls.load(Ordering::Acquire) > 0, "the real lock refresh must fail");
|
||||
assert!(guard.is_lock_lost(), "the cleanup guard must observe lock loss");
|
||||
// Real disk I/O must not run under Tokio's auto-advancing paused clock:
|
||||
// a pending blocking operation could otherwise exhaust its timeout.
|
||||
tokio::time::resume();
|
||||
let err = guard.delete(&set).await.expect_err("lock loss must fence physical deletion");
|
||||
assert!(matches!(err, StorageError::NamespaceLockQuorumUnavailable { .. }), "{err:?}");
|
||||
assert_eq!(retained_native_multipart_original(&disks[3], 3, bucket, object, &path).await, original[3]);
|
||||
assert_eq!(
|
||||
disks[3]
|
||||
.read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
|
||||
.await
|
||||
.expect("surviving shard after loss"),
|
||||
surviving_part
|
||||
);
|
||||
|
||||
*target.write().expect("restore lock service") = None;
|
||||
set.lock_stale_multipart_cleanup(&path)
|
||||
.await
|
||||
.expect("reacquire the cleanup lock")
|
||||
.delete(&set)
|
||||
.await
|
||||
.expect("the certified residual converges after lock recovery");
|
||||
for dir in &dirs {
|
||||
assert!(!dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn native_abort_partial_delete_surviving_producer_confirms_current_and_historical_exact_target() {
|
||||
use crate::set_disk::core::io_primitives::{rename_fault_injection, rollback_fault_injection};
|
||||
@@ -507,7 +703,7 @@ mod native_abort_closure_tests {
|
||||
let path = SetDisks::get_upload_id_dir(bucket, object, &upload);
|
||||
pin_historical_unmarked_upload(&disks, bucket, &path).await;
|
||||
let before = originals(&disks, &path).await;
|
||||
set.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
|
||||
set.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
|
||||
.await
|
||||
.expect("readonly legacy cleanup identity lookup");
|
||||
assert_eq!(
|
||||
@@ -786,7 +982,7 @@ mod native_abort_closure_tests {
|
||||
"known orphan native evidence remains unknown: {inspected:?}"
|
||||
);
|
||||
let readonly = set
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
|
||||
.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
|
||||
.await;
|
||||
assert!(
|
||||
matches!(
|
||||
|
||||
@@ -73,6 +73,7 @@ struct DeleteCommitContext<'a> {
|
||||
incarnation: Option<Uuid>,
|
||||
options: Option<&'a ObjectOptions>,
|
||||
guard: Option<Arc<super::super::ObjectLockDiagGuard>>,
|
||||
transaction_guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
|
||||
}
|
||||
|
||||
impl<'a> DeleteCommitContext<'a> {
|
||||
@@ -81,17 +82,37 @@ impl<'a> DeleteCommitContext<'a> {
|
||||
incarnation: options.expected_bucket_incarnation_id,
|
||||
options: Some(options),
|
||||
guard: guard.cloned(),
|
||||
transaction_guard: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn with_transaction_guard(mut self, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Self {
|
||||
self.transaction_guard = guard.cloned();
|
||||
self
|
||||
}
|
||||
|
||||
fn ensure_transaction_lock_held(&self, bucket: &str, object: &str) -> Result<()> {
|
||||
if self.transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
|
||||
return Err(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_transaction",
|
||||
bucket: bucket.to_owned(),
|
||||
object: object.to_owned(),
|
||||
required: 1,
|
||||
achieved: 0,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn can_commit(&self, bucket: &str, object: &str) -> bool {
|
||||
self.options.is_none_or(|options| {
|
||||
ensure_delete_commit_locks_held(self.guard.as_deref(), bucket, object, options).is_ok()
|
||||
&& options
|
||||
.scanner_publication_commit_scope
|
||||
.as_ref()
|
||||
.is_none_or(|scope| scope.can_commit())
|
||||
})
|
||||
!self.transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
&& self.options.is_none_or(|options| {
|
||||
ensure_delete_commit_locks_held(self.guard.as_deref(), bucket, object, options).is_ok()
|
||||
&& options
|
||||
.scanner_publication_commit_scope
|
||||
.as_ref()
|
||||
.is_none_or(|scope| scope.can_commit())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3852,6 +3873,7 @@ impl SetDisks {
|
||||
object: &str,
|
||||
opts: &ObjectOptions,
|
||||
lock_guard: Option<&Arc<super::super::ObjectLockDiagGuard>>,
|
||||
transaction_guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
|
||||
) -> Result<bool> {
|
||||
if is_meta_bucketname(bucket)
|
||||
|| opts.version_id.is_some()
|
||||
@@ -3888,6 +3910,7 @@ impl SetDisks {
|
||||
object,
|
||||
DeleteRecoveryContext {
|
||||
explicit_delete: true,
|
||||
transaction_guard: transaction_guard.cloned(),
|
||||
namespace_owner: Some(Arc::new((
|
||||
self.ctx.begin_namespace_commit(),
|
||||
lock_guard.cloned(),
|
||||
@@ -8839,7 +8862,7 @@ impl SetDisks {
|
||||
// quorum, failing write quorum on update_object_meta (backlog#872).
|
||||
let read_opts = opts.as_read_opts();
|
||||
let (mut fi, _, disks) = self
|
||||
.get_object_fileinfo_gated(bucket, object, &read_opts, false, false)
|
||||
.get_object_fileinfo_for_metadata_update(bucket, object, &read_opts)
|
||||
.await?
|
||||
.into_owned();
|
||||
|
||||
@@ -8877,8 +8900,11 @@ impl SetDisks {
|
||||
fi: &FileInfo,
|
||||
force_del_marker: bool,
|
||||
delete_marker_purge: Option<MrfDeleteMarkerPurge>,
|
||||
commit_context: DeleteCommitContext<'_>,
|
||||
mut commit_context: DeleteCommitContext<'_>,
|
||||
) -> Result<()> {
|
||||
if !is_meta_bucketname(bucket) && commit_context.transaction_guard.is_none() {
|
||||
commit_context.transaction_guard = Some(self.acquire_delete_transaction_lock(bucket, object).await?);
|
||||
}
|
||||
let source_bucket_incarnation_id = commit_context.incarnation;
|
||||
let transported = delete_file_info_with_replication_transport_metadata(fi);
|
||||
let fi = &transported;
|
||||
@@ -8887,6 +8913,7 @@ impl SetDisks {
|
||||
Arc::new((
|
||||
self.ctx.begin_namespace_commit(),
|
||||
commit_context.guard.clone(),
|
||||
commit_context.transaction_guard.clone(),
|
||||
commit_context
|
||||
.options
|
||||
.and_then(|options| options.object_lock_config_snapshot.clone()),
|
||||
@@ -8926,7 +8953,11 @@ impl SetDisks {
|
||||
|
||||
for disk in disks.iter() {
|
||||
let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
|
||||
let transaction_guard = commit_context.transaction_guard.as_ref();
|
||||
futures.push(async move {
|
||||
if transaction_guard.is_some_and(|guard| guard.is_lock_lost()) {
|
||||
return Err(DiskError::DiskOngoingReq);
|
||||
}
|
||||
if let Some(disk) = disk {
|
||||
let options = DeleteOptions {
|
||||
old_data_dir: Some(rollback_dir),
|
||||
@@ -8976,6 +9007,7 @@ impl SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
commit_context.ensure_transaction_lock_held(bucket, object)?;
|
||||
let mut quorum_result = resolve_tiered_decommission_write_quorum_result(&errs, write_quorum, bucket, object);
|
||||
if !is_meta_bucketname(bucket) && errs.iter().flatten().any(super::delete_recovery::unsupported_recovery) {
|
||||
quorum_result = Err(DiskError::UnsupportedDisk.into());
|
||||
@@ -9008,6 +9040,7 @@ impl SetDisks {
|
||||
write_quorum,
|
||||
DeleteRecoveryContext {
|
||||
recovery_ticket,
|
||||
transaction_guard: commit_context.transaction_guard.clone(),
|
||||
namespace_owner: namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>),
|
||||
..DeleteRecoveryContext::new(source_bucket_incarnation_id, &|| commit_context.can_commit(bucket, object))
|
||||
},
|
||||
@@ -9017,6 +9050,7 @@ impl SetDisks {
|
||||
}
|
||||
#[cfg(test)]
|
||||
pause_delete_before_cleanup(bucket).await;
|
||||
commit_context.ensure_transaction_lock_held(bucket, object)?;
|
||||
let mut rollback_futures = Vec::new();
|
||||
for (index, err) in errs.iter().enumerate() {
|
||||
if structured_cleanup {
|
||||
@@ -9042,7 +9076,11 @@ impl SetDisks {
|
||||
let object = object.to_string();
|
||||
let fi = fi.clone();
|
||||
let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
|
||||
let transaction_guard = commit_context.transaction_guard.as_ref();
|
||||
rollback_futures.push(async move {
|
||||
if transaction_guard.is_some_and(|guard| guard.is_lock_lost()) {
|
||||
return;
|
||||
}
|
||||
if should_rollback {
|
||||
// The dedicated undo path never forwards the marker-only creation flag.
|
||||
if let Err(err) = disk
|
||||
@@ -9099,6 +9137,7 @@ impl SetDisks {
|
||||
}
|
||||
|
||||
join_all(rollback_futures).await;
|
||||
commit_context.ensure_transaction_lock_held(bucket, object)?;
|
||||
drop(namespace_owner);
|
||||
if quorum_result.is_ok()
|
||||
&& errs.iter().any(Option::is_some)
|
||||
@@ -9294,36 +9333,56 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
};
|
||||
|
||||
let (read_quorum, write_quorum) = match Self::object_quorum_from_meta(&metas, &errs, self.default_parity_count) {
|
||||
Ok((r, w)) => (
|
||||
usize::try_from(r)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureReadQuorum.into(), vec![src_bucket, src_object]))?,
|
||||
usize::try_from(w)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![src_bucket, src_object]))?,
|
||||
),
|
||||
let src_version_id = src_opts.version_id.as_deref().unwrap_or_default();
|
||||
let quorum = Self::object_quorum_from_meta(&metas, &errs, self.default_parity_count);
|
||||
let selection = match &quorum {
|
||||
Ok((read_quorum, write_quorum)) => {
|
||||
let read_quorum = usize::try_from(*read_quorum)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureReadQuorum.into(), vec![src_bucket, src_object]))?;
|
||||
let write_quorum = usize::try_from(*write_quorum)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![src_bucket, src_object]))?;
|
||||
Self::select_fileinfo_for_read(src_bucket, &disks, &metas, &errs, src_version_id, read_quorum, write_quorum)
|
||||
.map(|(online_disks, fi)| (online_disks, fi, write_quorum))
|
||||
}
|
||||
Err(err) => Err(err.clone()),
|
||||
};
|
||||
let (online_disks, mut fi, write_quorum) = match selection {
|
||||
Ok(selection) => selection,
|
||||
Err(mut err) => {
|
||||
if err == DiskError::ErasureReadQuorum
|
||||
&& !src_bucket.starts_with(RUSTFS_META_BUCKET)
|
||||
&& self
|
||||
.delete_if_dangling(src_bucket, src_object, &metas, &errs, &HashMap::new(), src_opts.clone())
|
||||
.await
|
||||
.is_ok()
|
||||
// The merged latest version can be one a failed PUT left below
|
||||
// quorum; resolve it like a latest GET before the dangling
|
||||
// cleanup below can report a key with a committed version absent.
|
||||
if let Some(snapshot) = self
|
||||
.select_latest_past_dangling_versions(src_bucket, src_object, src_opts, true, &disks, &err)
|
||||
.await
|
||||
{
|
||||
if src_opts.version_id.is_some() {
|
||||
err = DiskError::FileVersionNotFound
|
||||
} else {
|
||||
err = DiskError::FileNotFound
|
||||
// A version-only copy writes the served version's own
|
||||
// copies, so they and its write quorum replace the latest
|
||||
// read's.
|
||||
let (fi, parts_metadata, online_disks) = snapshot.into_owned();
|
||||
metas = parts_metadata;
|
||||
let write_quorum = fi.write_quorum(self.default_write_quorum());
|
||||
(online_disks, fi, write_quorum)
|
||||
} else {
|
||||
if quorum.is_err()
|
||||
&& err == DiskError::ErasureReadQuorum
|
||||
&& !src_bucket.starts_with(RUSTFS_META_BUCKET)
|
||||
&& self
|
||||
.delete_if_dangling(src_bucket, src_object, &metas, &errs, &HashMap::new(), src_opts.clone())
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
if src_opts.version_id.is_some() {
|
||||
err = DiskError::FileVersionNotFound
|
||||
} else {
|
||||
err = DiskError::FileNotFound
|
||||
}
|
||||
}
|
||||
return Err(to_object_err(err.into(), vec![src_bucket, src_object]));
|
||||
}
|
||||
return Err(to_object_err(err.into(), vec![src_bucket, src_object]));
|
||||
}
|
||||
};
|
||||
|
||||
let src_version_id = src_opts.version_id.as_deref().unwrap_or_default();
|
||||
let (online_disks, mut fi) =
|
||||
Self::select_fileinfo_for_read(src_bucket, &disks, &metas, &errs, src_version_id, read_quorum, write_quorum)
|
||||
.map_err(|e| to_object_err(e.into(), vec![src_bucket, src_object]))?;
|
||||
|
||||
if fi.deleted {
|
||||
if src_opts.version_id.is_none() {
|
||||
return Err(to_object_err(Error::FileNotFound, vec![src_bucket, src_object]));
|
||||
@@ -9461,6 +9520,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
// A version-only copy writes and then reverts on a quorum miss;
|
||||
// refuse it before writing when the served version's disks cannot
|
||||
// reach its write quorum even if every write succeeds.
|
||||
let reachable = online_disks
|
||||
.iter()
|
||||
.map(|disk| disk.is_none().then_some(DiskError::DiskNotFound))
|
||||
.collect::<Vec<_>>();
|
||||
if let Some(err) = reduce_write_quorum_errs(&reachable, OBJECT_OP_IGNORED_ERRS, write_quorum) {
|
||||
return Err(to_object_err(err.into(), vec![src_bucket, src_object]));
|
||||
}
|
||||
Self::write_unique_file_info(&online_disks, "", src_bucket, src_object, &metas, write_quorum)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e.into(), vec![src_bucket, src_object]))?;
|
||||
@@ -9528,9 +9597,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&self,
|
||||
bucket: &str,
|
||||
objects: Vec<ObjectToDelete>,
|
||||
opts: ObjectOptions,
|
||||
mut opts: ObjectOptions,
|
||||
) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
|
||||
let current_generations = opts.conditional_delete_generations.as_deref();
|
||||
let mut del_objects = vec![DeletedObject::default(); objects.len()];
|
||||
let mut accounting = vec![None; objects.len()];
|
||||
let delete_config_snapshot = opts
|
||||
@@ -9625,6 +9693,30 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
let outer_namespace_lock_fence = opts.namespace_lock_fence.clone();
|
||||
let mut transaction_guards = std::collections::BTreeMap::new();
|
||||
if !is_meta_bucketname(bucket) {
|
||||
// Every raw batch lock is acquired first. Canonical sorting also
|
||||
// gives aliases of one disk key exactly one transaction owner.
|
||||
let keys: std::collections::BTreeSet<_> = locked_objects.iter().cloned().collect();
|
||||
for key in keys {
|
||||
match self.acquire_delete_transaction_lock(bucket, &key).await {
|
||||
Ok(guard) => {
|
||||
opts.add_namespace_lock_guard(&guard);
|
||||
transaction_guards.insert(key, guard);
|
||||
}
|
||||
Err(error) => {
|
||||
for (index, lock_name) in lock_names.iter().enumerate() {
|
||||
if lock_name.as_ref() == key {
|
||||
del_errs[index] = Some(error.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let transaction_guards = Arc::new(transaction_guards);
|
||||
let current_generations = opts.conditional_delete_generations.as_deref();
|
||||
let object_lock_config = if is_meta_bucketname(bucket) {
|
||||
Arc::new(ObjectLockConfigSnapshot::new(ObjectLockConfigState::ConfirmedAbsent))
|
||||
} else {
|
||||
@@ -9722,6 +9814,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&dobj.object_name,
|
||||
DeleteRecoveryContext {
|
||||
dry_run: !opts.no_lock,
|
||||
transaction_guard: transaction_guards.get(lock_names[i].as_ref()).cloned(),
|
||||
namespace_owner: opts.delete_mutation_owner.clone(),
|
||||
..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| {
|
||||
ensure_delete_commit_locks_held(None, bucket, &dobj.object_name, &opts).is_ok()
|
||||
@@ -10003,6 +10096,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
let namespace_owner = (!is_meta_bucketname(bucket)).then(|| {
|
||||
Arc::new((
|
||||
self.ctx.begin_namespace_commit(),
|
||||
Arc::clone(&transaction_guards),
|
||||
opts.delete_mutation_owner.clone(),
|
||||
opts.object_lock_config_snapshot.clone(),
|
||||
opts.scanner_publication_commit_scope.clone(),
|
||||
@@ -10178,6 +10272,24 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
|
||||
let results = join_all(futures).await;
|
||||
if transaction_guards.values().any(|guard| guard.is_lock_lost()) {
|
||||
for (index, object) in objects.iter().enumerate() {
|
||||
if del_errs[index].is_none() {
|
||||
accounting[index] = None;
|
||||
del_errs[index] = Some(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_transaction",
|
||||
bucket: bucket.to_owned(),
|
||||
object: object.object_name.clone(),
|
||||
required: 1,
|
||||
achieved: 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
if dist_erasure {
|
||||
self.release_dist_delete_object_locks_batch(dist_batch_lock_ids).await;
|
||||
}
|
||||
return (del_objects, del_errs, accounting);
|
||||
}
|
||||
|
||||
let mut del_obj_errs: Vec<Vec<Option<DiskError>>> = vec![vec![None; objects.len()]; disks.len()];
|
||||
|
||||
@@ -10273,6 +10385,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
let group = &vers[index];
|
||||
let recovery_ticket = recovery_tickets[index].clone();
|
||||
let namespace_owner = namespace_owner.clone();
|
||||
let transaction_guard = transaction_guards
|
||||
.get(object_name_disk_identity(&group.name).as_ref())
|
||||
.cloned();
|
||||
let disks = &disks;
|
||||
let opts = &opts;
|
||||
async move {
|
||||
@@ -10293,6 +10408,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
DeleteRecoveryContext {
|
||||
recovery_ticket,
|
||||
namespace_owner,
|
||||
transaction_guard,
|
||||
..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &can_commit)
|
||||
},
|
||||
)
|
||||
@@ -10339,7 +10455,11 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
let object = fi_vers.name.clone();
|
||||
let versions = fi_vers.clone();
|
||||
let disk_namespace_owner = namespace_owner.clone();
|
||||
let transaction_guard = transaction_guards.get(object_name_disk_identity(&object).as_ref()).cloned();
|
||||
rollback_futures.push(async move {
|
||||
if transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
|
||||
return;
|
||||
}
|
||||
if should_rollback {
|
||||
let errs = disk
|
||||
.delete_versions_with_namespace_owner(
|
||||
@@ -10415,7 +10535,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
if del_errs[index].is_none() {
|
||||
let object = &objects[index].object_name;
|
||||
let result = match ensure_delete_commit_locks_held(None, bucket, object, &opts) {
|
||||
Ok(()) => self.reclaim_missing_delete_residue(bucket, object, &opts, None).await,
|
||||
Ok(()) => {
|
||||
self.reclaim_missing_delete_residue(
|
||||
bucket,
|
||||
object,
|
||||
&opts,
|
||||
None,
|
||||
transaction_guards.get(object_name_disk_identity(object).as_ref()),
|
||||
)
|
||||
.await
|
||||
}
|
||||
Err(err) => Err(err),
|
||||
};
|
||||
if let Err(err) = result {
|
||||
@@ -10424,6 +10553,13 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
// Deferred recovery takes fresh raw and transaction locks. Discard the
|
||||
// completed leases' deadlines while retaining every caller-owned fence.
|
||||
opts.namespace_lock_fence = outer_namespace_lock_fence;
|
||||
// Release inner owners before reacquiring any raw key lock.
|
||||
drop(namespace_owner);
|
||||
drop(transaction_guards);
|
||||
|
||||
// TODO(backlog): support partial object deletion for multi-part objects
|
||||
|
||||
if dist_erasure {
|
||||
@@ -10533,6 +10669,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
// Lock order: the caller's raw object lock, then this target set's
|
||||
// transaction key. The inner key remains distinct even when pool
|
||||
// client quorums overlap, and no_lock never bypasses journal ownership.
|
||||
let transaction_guard = if !is_meta_bucketname(bucket) && (!opts.delete_prefix || opts.delete_prefix_object) {
|
||||
let guard = self.acquire_delete_transaction_lock(bucket, object).await?;
|
||||
opts.add_namespace_lock_guard(&guard);
|
||||
Some(guard)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if opts.delete_prefix {
|
||||
let destructive_dispatch_phase = opts.lifecycle_delete_all.as_ref().is_none_or(|request| {
|
||||
matches!(
|
||||
@@ -10633,7 +10779,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&delete_request,
|
||||
false,
|
||||
None,
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref())
|
||||
.with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await?;
|
||||
if let Some((_, deleted_object)) = replication_delete {
|
||||
@@ -10667,7 +10814,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&delete_request,
|
||||
false,
|
||||
None,
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref())
|
||||
.with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
@@ -10687,7 +10835,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&delete_request,
|
||||
false,
|
||||
None,
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref())
|
||||
.with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
@@ -10776,6 +10925,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
bucket,
|
||||
object,
|
||||
DeleteRecoveryContext {
|
||||
transaction_guard: transaction_guard.clone(),
|
||||
namespace_owner: Some(Arc::new((
|
||||
self.ctx.begin_namespace_commit(),
|
||||
_lock_guard.clone(),
|
||||
@@ -10816,7 +10966,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
if is_err_object_not_found(err) {
|
||||
ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?;
|
||||
if self
|
||||
.reclaim_missing_delete_residue(bucket, object, &opts, _lock_guard.as_ref())
|
||||
.reclaim_missing_delete_residue(bucket, object, &opts, _lock_guard.as_ref(), transaction_guard.as_ref())
|
||||
.await?
|
||||
{
|
||||
if let Some(scope) = scanner_publication_commit_scope.as_ref() {
|
||||
@@ -10866,7 +11016,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&dfi,
|
||||
false,
|
||||
None,
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e, vec![bucket, object]))?;
|
||||
@@ -10968,7 +11118,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&fi,
|
||||
should_force_delete_marker_for_missing_version(&opts),
|
||||
delete_marker_purge.clone(),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e, vec![bucket, object]))?;
|
||||
@@ -11033,7 +11183,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
&dfi,
|
||||
opts.delete_marker,
|
||||
delete_marker_purge,
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
|
||||
DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| to_object_err(e, vec![bucket, object]))?;
|
||||
@@ -11147,45 +11297,63 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
};
|
||||
|
||||
let (read_quorum, write_quorum) = match Self::object_quorum_from_meta(&metas, &errs, self.default_parity_count) {
|
||||
Ok((read_quorum, write_quorum)) => (read_quorum, write_quorum),
|
||||
let version_id = opts.version_id.as_deref().unwrap_or_default();
|
||||
let quorum = Self::object_quorum_from_meta(&metas, &errs, self.default_parity_count);
|
||||
let selection = match &quorum {
|
||||
Ok((read_quorum, write_quorum)) => {
|
||||
let read_quorum = usize::try_from(*read_quorum)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]))?;
|
||||
let write_quorum = usize::try_from(*write_quorum)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![bucket, object]))?;
|
||||
Self::select_fileinfo_for_read(bucket, &disks, &metas, &errs, version_id, read_quorum, write_quorum)
|
||||
}
|
||||
Err(err) => Err(err.clone()),
|
||||
};
|
||||
let (online_disks, mut fi) = match selection {
|
||||
Ok(selection) => selection,
|
||||
Err(mut err) => {
|
||||
// A replication terminal CAS is a validation/write-back
|
||||
// attempt, never a heal operation. In particular,
|
||||
// ValidateOnly must remain zero-write, and a worker whose
|
||||
// outer replication lease is already lost must not delete the
|
||||
// surviving xl.meta copies before the namespace fence below
|
||||
// can reject it. Leave dangling cleanup to heal/scanner.
|
||||
if opts.replication_status_writeback.is_some() {
|
||||
// The merged latest version can be one a failed PUT left below
|
||||
// quorum; resolve it like a latest GET before the dangling
|
||||
// cleanup below can report a key with a committed version absent.
|
||||
// A replication terminal CAS stays side-effect free: the walk
|
||||
// queues heal for the versions it skips, so it does not run.
|
||||
let resolved = if opts.replication_status_writeback.is_none() {
|
||||
self.select_latest_past_dangling_versions(bucket, object, opts, false, &disks, &err)
|
||||
.await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(snapshot) = resolved {
|
||||
let (fi, _, online_disks) = snapshot.into_owned();
|
||||
(online_disks, fi)
|
||||
} else {
|
||||
// A replication terminal CAS is a validation/write-back
|
||||
// attempt, never a heal operation. In particular,
|
||||
// ValidateOnly must remain zero-write, and a worker whose
|
||||
// outer replication lease is already lost must not delete
|
||||
// the surviving xl.meta copies before the namespace fence
|
||||
// below can reject it. Leave dangling cleanup to
|
||||
// heal/scanner.
|
||||
if quorum.is_err()
|
||||
&& opts.replication_status_writeback.is_none()
|
||||
&& err == DiskError::ErasureReadQuorum
|
||||
&& !bucket.starts_with(RUSTFS_META_BUCKET)
|
||||
&& self
|
||||
.delete_if_dangling(bucket, object, &metas, &errs, &HashMap::new(), opts.clone())
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
if opts.version_id.is_some() {
|
||||
err = DiskError::FileVersionNotFound
|
||||
} else {
|
||||
err = DiskError::FileNotFound
|
||||
}
|
||||
}
|
||||
return Err(to_object_err(err.into(), vec![bucket, object]));
|
||||
}
|
||||
if err == DiskError::ErasureReadQuorum
|
||||
&& !bucket.starts_with(RUSTFS_META_BUCKET)
|
||||
&& self
|
||||
.delete_if_dangling(bucket, object, &metas, &errs, &HashMap::new(), opts.clone())
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
if opts.version_id.is_some() {
|
||||
err = DiskError::FileVersionNotFound
|
||||
} else {
|
||||
err = DiskError::FileNotFound
|
||||
}
|
||||
}
|
||||
return Err(to_object_err(err.into(), vec![bucket, object]));
|
||||
}
|
||||
};
|
||||
|
||||
let read_quorum =
|
||||
usize::try_from(read_quorum).map_err(|_| to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]))?;
|
||||
let write_quorum = usize::try_from(write_quorum)
|
||||
.map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![bucket, object]))?;
|
||||
|
||||
let version_id = opts.version_id.as_deref().unwrap_or_default();
|
||||
let (online_disks, mut fi) =
|
||||
Self::select_fileinfo_for_read(bucket, &disks, &metas, &errs, version_id, read_quorum, write_quorum)
|
||||
.map_err(|e| to_object_err(e.into(), vec![bucket, object]))?;
|
||||
|
||||
if fi.deleted {
|
||||
return Err(to_object_err(Error::MethodNotAllowed, vec![bucket, object]));
|
||||
}
|
||||
@@ -11790,13 +11958,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
let bucket_lifecycle_guard = if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
|
||||
&& opts.bucket_lifecycle_lock_fence.is_none()
|
||||
{
|
||||
let guard = metadata_sys::object_store_in(&self.ctx)
|
||||
.await?
|
||||
.acquire_bucket_lifecycle_read_lock(bucket)
|
||||
.await?;
|
||||
self.validate_bucket_incarnation(bucket, expected_incarnation_id, opts)
|
||||
.await?;
|
||||
Some(guard)
|
||||
Some(
|
||||
metadata_sys::object_store_in(&self.ctx)
|
||||
.await?
|
||||
.acquire_bucket_incarnation_fence_with_options(bucket, expected_incarnation_id, Some(opts))
|
||||
.await?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -25358,6 +25525,7 @@ mod delete_objects_lock_gating_tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn delete_object_aborts_when_outer_lock_is_lost_at_commit() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
let bucket = "single-lost-outer-lock";
|
||||
@@ -25578,6 +25746,13 @@ mod delete_objects_lock_gating_tests {
|
||||
.as_fast_lock_manager()
|
||||
.expect("isolated fixture should use the local lock manager");
|
||||
let lock_metrics = lock_manager.get_metrics().shard_metrics;
|
||||
let raw_key = ObjectKey::new(bucket, "obj-a");
|
||||
let raw_holder = lock_manager.get_lock_info(&raw_key).expect("outer holder must be visible");
|
||||
let transaction_key = ObjectKey::with_version(
|
||||
bucket,
|
||||
"obj-a",
|
||||
format!("rustfs:delete-transaction:{}:{}", set_disks.pool_index, set_disks.set_index),
|
||||
);
|
||||
assert_eq!(lock_manager.current_lock_counts(), (1, 0));
|
||||
let mut delete = Box::pin(set_disks.delete_objects(
|
||||
bucket,
|
||||
@@ -25594,23 +25769,43 @@ mod delete_objects_lock_gating_tests {
|
||||
// than a second on CI without waiting for the namespace lock at all.
|
||||
let deletion = futures::future::poll_fn(|cx| {
|
||||
let result = std::future::Future::poll(delete.as_mut(), cx);
|
||||
let current_holder = lock_manager.get_lock_info(&raw_key).expect("outer holder must remain held");
|
||||
assert_eq!(
|
||||
(current_holder.mode, ¤t_holder.owner, current_holder.acquired_at),
|
||||
(raw_holder.mode, &raw_holder.owner, raw_holder.acquired_at),
|
||||
"no_lock must preserve the original raw object holder"
|
||||
);
|
||||
let transaction_holder = lock_manager.get_lock_info(&transaction_key);
|
||||
if let Some(holder) = &transaction_holder {
|
||||
assert_eq!(holder.mode, rustfs_lock::LockMode::Exclusive);
|
||||
}
|
||||
assert_eq!(
|
||||
lock_manager.current_lock_counts(),
|
||||
(1, 0),
|
||||
"no_lock batch delete must preserve the outer holder without adding lock waiters"
|
||||
(1 + u64::from(transaction_holder.is_some()), 0),
|
||||
"only the distinct transaction owner may be added, without raw-key waiters"
|
||||
);
|
||||
let current = lock_manager.get_metrics().shard_metrics;
|
||||
assert_eq!(
|
||||
(current.total_acquisitions(), current.timeouts),
|
||||
(lock_metrics.total_acquisitions(), lock_metrics.timeouts),
|
||||
"no_lock batch delete must not acquire or time out on another namespace lock"
|
||||
assert!(
|
||||
current.total_acquisitions() <= lock_metrics.total_acquisitions() + 1,
|
||||
"the transaction owner must be reused by inline recovery"
|
||||
);
|
||||
assert_eq!(current.timeouts, lock_metrics.timeouts, "the raw key must never be reacquired");
|
||||
result
|
||||
});
|
||||
let (deleted, errs) = tokio::time::timeout(Duration::from_secs(30), deletion)
|
||||
.await
|
||||
.expect("no_lock batch delete disk work should complete");
|
||||
|
||||
assert_eq!(lock_manager.current_lock_counts(), (1, 0));
|
||||
assert!(
|
||||
lock_manager.get_lock_info(&transaction_key).is_none(),
|
||||
"the transaction owner must be released"
|
||||
);
|
||||
assert_eq!(
|
||||
lock_manager.get_metrics().shard_metrics.total_acquisitions(),
|
||||
lock_metrics.total_acquisitions() + 1,
|
||||
"no_lock still requires exactly one distinct transaction owner"
|
||||
);
|
||||
assert!(errs[0].is_none(), "no_lock batch delete should not fail with a lock error: {:?}", errs[0]);
|
||||
assert!(deleted[0].found, "existing object must still be deleted");
|
||||
for disk in &disk_stores {
|
||||
|
||||
@@ -264,6 +264,11 @@ enum MetadataReadPurpose {
|
||||
GetObjectReader,
|
||||
}
|
||||
|
||||
/// Upper bound on exact-version reads a latest read spends looking past
|
||||
/// dangling versions before it keeps the read-quorum error; each read is one
|
||||
/// metadata fan-out.
|
||||
const MAX_DANGLING_LATEST_VERSIONS: usize = 8;
|
||||
|
||||
/// How a metadata read-quorum failure is reported when the replies prove the
|
||||
/// object or version dangling. Only the object read entry points
|
||||
/// (`get_object_info`, `get_object_reader` and prepared GET metadata) report
|
||||
@@ -273,6 +278,10 @@ enum MetadataReadPurpose {
|
||||
#[derive(Clone, Copy, PartialEq, Eq)]
|
||||
enum DanglingRead {
|
||||
KeepQuorumError,
|
||||
/// Metadata updates of the latest version keep the quorum error but, like
|
||||
/// object reads, look past newer versions proven dangling
|
||||
/// (`read_latest_past_dangling_versions`).
|
||||
SkipDanglingVersions,
|
||||
ReportNotFound,
|
||||
}
|
||||
|
||||
@@ -724,9 +733,10 @@ impl SetDisks {
|
||||
}
|
||||
|
||||
/// Like `get_object_fileinfo`, but `allow_early_stop=false` forces the full
|
||||
/// quorum fanout. Read-before-write callers (object tagging) must use this:
|
||||
/// the returned online-disk set is the write target, and the early-stop
|
||||
/// subset would fail write quorum (backlog#872 regression fix).
|
||||
/// quorum fanout. Read-before-write callers must use this or
|
||||
/// `get_object_fileinfo_for_metadata_update`: the returned online-disk set
|
||||
/// is the write target, and the early-stop subset would fail write quorum
|
||||
/// (backlog#872 regression fix).
|
||||
#[tracing::instrument(level = "debug", skip(self))]
|
||||
pub(in crate::set_disk) async fn get_object_fileinfo_gated(
|
||||
&self,
|
||||
@@ -748,6 +758,64 @@ impl SetDisks {
|
||||
.await
|
||||
}
|
||||
|
||||
/// The full-fanout read behind a metadata update such as object tagging.
|
||||
/// A latest read whose newest versions never committed resolves to the
|
||||
/// version a latest GET serves, while a dangling key keeps the quorum
|
||||
/// error. The returned online disks are the write target.
|
||||
#[tracing::instrument(level = "debug", skip(self))]
|
||||
pub(in crate::set_disk) async fn get_object_fileinfo_for_metadata_update(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<GetObjectFileInfo> {
|
||||
self.get_object_fileinfo_gated_inner(
|
||||
bucket,
|
||||
object,
|
||||
opts,
|
||||
false,
|
||||
false,
|
||||
MetadataReadPurpose::FullLayout,
|
||||
DanglingRead::SkipDanglingVersions,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Recovers a failed latest selection in read-before-write callers that
|
||||
/// read and select metadata themselves (`put_object_metadata` and the
|
||||
/// CopyObject source) the way a latest GET does. `None` keeps the caller's
|
||||
/// error: a failure other than read quorum, an explicit version, the
|
||||
/// system bucket's commit-proof rule, or a walk that finds no readable
|
||||
/// version or cannot read every drive. The selected metadata keeps its
|
||||
/// part checksums because a version-only copy carries them into the new
|
||||
/// version.
|
||||
pub(in crate::set_disk) async fn select_latest_past_dangling_versions(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
opts: &ObjectOptions,
|
||||
read_data: bool,
|
||||
disks: &[Option<DiskStore>],
|
||||
err: &DiskError,
|
||||
) -> Option<GetObjectFileInfo> {
|
||||
if *err != DiskError::ErasureReadQuorum || opts.version_id.is_some() || crate::bucket::utils::is_meta_bucketname(bucket) {
|
||||
return None;
|
||||
}
|
||||
let opts = ObjectOptions {
|
||||
include_part_checksums: true,
|
||||
..opts.clone()
|
||||
};
|
||||
Box::pin(self.read_latest_past_dangling_versions(
|
||||
bucket,
|
||||
object,
|
||||
&opts,
|
||||
read_data,
|
||||
MetadataReadPurpose::FullLayout,
|
||||
disks,
|
||||
))
|
||||
.await
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn get_object_fileinfo_gated_inner(
|
||||
&self,
|
||||
@@ -971,6 +1039,18 @@ impl SetDisks {
|
||||
{
|
||||
Ok(selection) => selection,
|
||||
Err(err) => {
|
||||
if err == DiskError::ErasureReadQuorum
|
||||
&& vid.is_empty()
|
||||
&& dangling_read != DanglingRead::KeepQuorumError
|
||||
&& metadata_fanout_complete
|
||||
&& !crate::bucket::utils::is_meta_bucketname(bucket)
|
||||
&& let Some(snapshot) =
|
||||
Box::pin(self.read_latest_past_dangling_versions(bucket, object, opts, read_data, purpose, &disks))
|
||||
.await
|
||||
{
|
||||
rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, "reached");
|
||||
return Ok(snapshot);
|
||||
}
|
||||
let err = StorageError::from(err);
|
||||
rustfs_io_metrics::record_get_object_metadata_quorum_result(
|
||||
metadata_metrics_path,
|
||||
@@ -1051,6 +1131,168 @@ impl SetDisks {
|
||||
Ok(snapshot)
|
||||
}
|
||||
|
||||
/// A latest read whose split cannot reach read quorum may still have a
|
||||
/// readable older version under newer versions that never committed;
|
||||
/// MinIO resolves this in `pickLatestQuorumFilesInfo`. With every drive
|
||||
/// answering, try versions newest first through exact-version reads and
|
||||
/// serve the readable one with the newest selected modification time. A
|
||||
/// version whose exact read proves it dangling (the NoSuchVersion an
|
||||
/// explicit GET reports) is skipped and queued for heal, which owns its
|
||||
/// cleanup; any other outcome keeps the read-quorum error.
|
||||
async fn read_latest_past_dangling_versions(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
opts: &ObjectOptions,
|
||||
read_data: bool,
|
||||
purpose: MetadataReadPurpose,
|
||||
disks: &[Option<DiskStore>],
|
||||
) -> Option<GetObjectFileInfo> {
|
||||
let disk_object = rustfs_utils::path::encode_dir_object(object);
|
||||
let (raw, errs) = Self::read_all_raw_file_info(disks, bucket, &disk_object, false).await;
|
||||
if errs
|
||||
.iter()
|
||||
.any(|err| err.as_ref().is_some_and(|err| !matches!(err, DiskError::FileNotFound)))
|
||||
{
|
||||
return None;
|
||||
}
|
||||
// Stacks are newest first, so only each drive's first few headers can
|
||||
// matter. Keep a version's newest modification time across drives: a
|
||||
// rewritten null version can carry different times on different drives.
|
||||
let mut newest = std::collections::HashMap::<Option<uuid::Uuid>, Option<time::OffsetDateTime>>::new();
|
||||
let mut truncated = false;
|
||||
let mut metadata = Vec::with_capacity(raw.len());
|
||||
for raw in raw.into_iter().flatten() {
|
||||
let mut meta = rustfs_filemeta::FileMeta::load(&raw.buf).ok()?;
|
||||
let mut versions = meta
|
||||
.versions
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, version)| !version.header.free_version());
|
||||
let mut retained_end = 0;
|
||||
for (index, version) in versions.by_ref().take(MAX_DANGLING_LATEST_VERSIONS) {
|
||||
let mod_time = newest.entry(version.header.version_id).or_insert(version.header.mod_time);
|
||||
*mod_time = (*mod_time).max(version.header.mod_time);
|
||||
retained_end = index + 1;
|
||||
}
|
||||
truncated |= versions.next().is_some();
|
||||
meta.versions.truncate(retained_end);
|
||||
metadata.push(meta);
|
||||
}
|
||||
let mut candidates = newest.into_iter().collect::<Vec<_>>();
|
||||
candidates.sort_by(|left, right| right.1.cmp(&left.1).then_with(|| left.0.cmp(&right.0)));
|
||||
|
||||
// Use the stored header for equal-time ordering: re-encoding a selected
|
||||
// FileInfo can change legacy signatures or omit part-checksum metadata.
|
||||
let selected_header = |snapshot: &GetObjectFileInfo| {
|
||||
let mut selected = snapshot.fi().clone();
|
||||
selected.data = None;
|
||||
let version_id = selected.version_id.unwrap_or_else(uuid::Uuid::nil).to_string();
|
||||
let include_part_checksums =
|
||||
opts.include_part_checksums || opts.part_number.is_some() || opts.data_movement || opts.raw_data_movement_read;
|
||||
metadata.iter().find_map(|meta| {
|
||||
let version = meta
|
||||
.versions
|
||||
.iter()
|
||||
.filter(|version| !version.header.free_version())
|
||||
.take(MAX_DANGLING_LATEST_VERSIONS)
|
||||
.find(|version| {
|
||||
version.header.version_id.unwrap_or_else(uuid::Uuid::nil)
|
||||
== selected.version_id.unwrap_or_else(uuid::Uuid::nil)
|
||||
&& version.header.mod_time == selected.mod_time
|
||||
})?;
|
||||
let mut original = if include_part_checksums {
|
||||
meta.into_fileinfo(&selected.volume, &selected.name, &version_id, false, opts.incl_free_versions, true)
|
||||
} else {
|
||||
meta.into_fileinfo_without_part_checksums(
|
||||
&selected.volume,
|
||||
&selected.name,
|
||||
&version_id,
|
||||
false,
|
||||
opts.incl_free_versions,
|
||||
)
|
||||
}
|
||||
.ok()?;
|
||||
if !include_part_checksums {
|
||||
rustfs_utils::http::remove_str(&mut original.metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS);
|
||||
}
|
||||
original.data = None;
|
||||
original.is_latest = selected.is_latest;
|
||||
original.successor_mod_time = selected.successor_mod_time;
|
||||
original.num_versions = selected.num_versions;
|
||||
(original == selected).then_some(version.header.clone())
|
||||
})
|
||||
};
|
||||
|
||||
let mut best: Option<(Option<time::OffsetDateTime>, GetObjectFileInfo)> = None;
|
||||
let mut skipped = Vec::new();
|
||||
let mut settled = false;
|
||||
for (reads, (version_id, newest_mod_time)) in candidates.into_iter().enumerate() {
|
||||
if best.as_ref().is_some_and(|(selected, _)| newest_mod_time < *selected) {
|
||||
settled = true;
|
||||
break;
|
||||
}
|
||||
if reads == MAX_DANGLING_LATEST_VERSIONS {
|
||||
return None;
|
||||
}
|
||||
// The nil UUID addresses the null version slot at this layer.
|
||||
let version_id = version_id.unwrap_or_else(uuid::Uuid::nil).to_string();
|
||||
let exact = ObjectOptions {
|
||||
version_id: Some(version_id.clone()),
|
||||
..opts.clone()
|
||||
};
|
||||
match Box::pin(self.get_object_fileinfo_gated_inner(
|
||||
bucket,
|
||||
object,
|
||||
&exact,
|
||||
read_data,
|
||||
false,
|
||||
purpose,
|
||||
DanglingRead::ReportNotFound,
|
||||
))
|
||||
.await
|
||||
{
|
||||
Ok(snapshot) => {
|
||||
let selected = snapshot.fi().mod_time;
|
||||
let replace = match best.as_ref() {
|
||||
Some((current, previous)) if selected == *current => {
|
||||
selected_header(&snapshot)?.sorts_before(&selected_header(previous)?)
|
||||
}
|
||||
Some((current, _)) => selected > *current,
|
||||
None => true,
|
||||
};
|
||||
if replace {
|
||||
best = Some((selected, snapshot));
|
||||
}
|
||||
}
|
||||
Err(StorageError::VersionNotFound(..)) => skipped.push(version_id),
|
||||
Err(_) => return None,
|
||||
}
|
||||
}
|
||||
// Headers past each drive's first few were not compared; only a walk
|
||||
// that proved no remaining candidate can be newer may stop early.
|
||||
if !settled && truncated {
|
||||
return None;
|
||||
}
|
||||
let (_, mut snapshot) = best?;
|
||||
snapshot.mark_selected_as_latest();
|
||||
if !opts.suppress_read_repair {
|
||||
for version_id in &skipped {
|
||||
submit_read_repair_heal(
|
||||
bucket,
|
||||
object,
|
||||
Some(version_id),
|
||||
self.pool_index,
|
||||
self.set_index,
|
||||
None,
|
||||
"dangling_latest_version",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
Some(snapshot)
|
||||
}
|
||||
|
||||
#[hotpath::measure(impl_type = "SetDisks")]
|
||||
pub(super) async fn get_object_info_fileinfo_and_quorum(
|
||||
&self,
|
||||
@@ -9751,6 +9993,7 @@ mod dangling_read_tests {
|
||||
use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions};
|
||||
use crate::storage_api_contracts::object::ObjectOperations;
|
||||
use http::HeaderMap;
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
use tempfile::TempDir;
|
||||
use tokio::io::AsyncReadExt;
|
||||
@@ -9761,9 +10004,13 @@ mod dangling_read_tests {
|
||||
const PARITY: usize = 2;
|
||||
|
||||
async fn dangling_test_set(bucket: &str) -> (Vec<TempDir>, Arc<SetDisks>) {
|
||||
dangling_test_set_with(bucket, DRIVES, PARITY).await
|
||||
}
|
||||
|
||||
async fn dangling_test_set_with(bucket: &str, drives: usize, parity: usize) -> (Vec<TempDir>, Arc<SetDisks>) {
|
||||
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
|
||||
ctx.update_erasure_type(crate::layout::endpoints::SetupType::Erasure).await;
|
||||
let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(DRIVES, PARITY, ctx).await;
|
||||
let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(drives, parity, ctx).await;
|
||||
set.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("test bucket should be created on every drive");
|
||||
@@ -9777,6 +10024,9 @@ mod dangling_read_tests {
|
||||
&mut PutObjReader::from_vec(body.to_vec()),
|
||||
&ObjectOptions {
|
||||
versioned,
|
||||
// Wait for every drive so tests that edit one drive's metadata
|
||||
// do not race the PUT's background tail writes.
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
@@ -9796,6 +10046,25 @@ mod dangling_read_tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Drops `version_id` from `disks`, modelling drives a failed PUT never
|
||||
/// reached while the older versions below it stay intact.
|
||||
async fn drop_version_from(set: &SetDisks, bucket: &str, object: &str, version_id: Uuid, disks: std::ops::Range<usize>) {
|
||||
let all = set.disks.read().await.clone();
|
||||
for disk in &all[disks] {
|
||||
let disk = disk.as_ref().expect("every drive should be online");
|
||||
let mut remove = FileInfo {
|
||||
volume: bucket.to_string(),
|
||||
name: object.to_string(),
|
||||
version_id: Some(version_id),
|
||||
..Default::default()
|
||||
};
|
||||
remove.set_skip_tier_free_version();
|
||||
disk.delete_version(bucket, object, remove, false, DeleteOptions::default())
|
||||
.await
|
||||
.expect("dropping the version should model a drive that never received it");
|
||||
}
|
||||
}
|
||||
|
||||
fn read_opts(version_id: Option<&str>) -> ObjectOptions {
|
||||
ObjectOptions {
|
||||
version_id: version_id.map(str::to_owned),
|
||||
@@ -9955,12 +10224,642 @@ mod dangling_read_tests {
|
||||
head(&set, bucket, object, Some(&v1))
|
||||
.await
|
||||
.expect("HEAD of the committed version must succeed");
|
||||
// The latest read sees two generations (version two on three drives,
|
||||
// version one on the rest), which is not proof of absence.
|
||||
let Err(err) = head(&set, bucket, object, None).await else {
|
||||
panic!("a split latest version must not resolve without quorum");
|
||||
// The latest read sees version two on three drives and version one on
|
||||
// the rest. Version two's exact read proves it dangling, so the latest
|
||||
// read serves version one, the newest version an exact read can serve.
|
||||
let latest = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("the latest read must serve the newest readable version");
|
||||
assert_eq!(latest.version_id.map(|id| id.to_string()).as_deref(), Some(v1.as_str()));
|
||||
assert_eq!(get(&set, bucket, object, None).await.expect("latest GET"), b"version one");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_skips_dangling_versions_above_a_committed_version() {
|
||||
// EC 12+4: a failed PUT that left version two on 5..=11 drives keeps
|
||||
// version one, present in every stack, below its read quorum of twelve
|
||||
// when only the top of each stack is read. Up to seven copies the
|
||||
// listing quorum already shows version one; from eight it lists the
|
||||
// dangling version two.
|
||||
let bucket = "dangling-latest-12-4";
|
||||
let (_dirs, set) = dangling_test_set_with(bucket, 16, 4).await;
|
||||
for copies in 5..=11 {
|
||||
let object = format!("torn/latest-{copies:02}");
|
||||
let v1 = put(&set, bucket, &object, b"committed version", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
let v2 = put(&set, bucket, &object, b"failed overwrite", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
drop_version_from(&set, bucket, &object, v2, copies..16).await;
|
||||
|
||||
let context = format!("version two on {copies} of 16 drives");
|
||||
let latest = head(&set, bucket, &object, None)
|
||||
.await
|
||||
.unwrap_or_else(|err| panic!("{context}: HEAD must serve version one: {err:?}"));
|
||||
assert_eq!(latest.version_id, Some(v1), "{context}");
|
||||
assert!(latest.is_latest, "{context}: the served version is the latest one");
|
||||
assert_eq!(
|
||||
get(&set, bucket, &object, None)
|
||||
.await
|
||||
.unwrap_or_else(|err| panic!("{context}: GET must serve version one: {err:?}")),
|
||||
b"committed version",
|
||||
"{context}"
|
||||
);
|
||||
let Err(err) = head(&set, bucket, &object, Some(&v2.to_string())).await else {
|
||||
panic!("{context}: the dangling version must not be served");
|
||||
};
|
||||
assert!(matches!(err, StorageError::VersionNotFound(..)), "{context}: {err:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_keeps_quorum_error_when_the_newest_version_is_uncertain() {
|
||||
// Version two is on every drive, but a torn metadata update left two
|
||||
// identities at one modification time, three copies each. Its exact
|
||||
// read fails with a quorum error rather than NoSuchVersion, so the
|
||||
// walk must not fall through to version one.
|
||||
let bucket = "dangling-latest-uncertain";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
put(&set, bucket, object, b"version one", true).await;
|
||||
let v2 = put(&set, bucket, object, b"version two", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id")
|
||||
.to_string();
|
||||
let disks = set.disks.read().await.clone();
|
||||
for disk in disks.iter().take(3) {
|
||||
let disk = disk.as_ref().expect("every drive should be online");
|
||||
let mut fi = disk
|
||||
.read_version("", bucket, object, &v2, &ReadOptions::default())
|
||||
.await
|
||||
.expect("version two should be on every drive");
|
||||
fi.metadata.insert("x-amz-meta-torn".to_string(), "half-applied".to_string());
|
||||
disk.update_metadata(bucket, object, fi, &crate::disk::UpdateMetadataOpts::default())
|
||||
.await
|
||||
.expect("a half-applied metadata update should be stored");
|
||||
}
|
||||
|
||||
assert_get_and_head_keep_read_quorum_error(&set, bucket, object, "torn metadata update on version two").await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_serves_a_committed_null_version_below_a_dangling_version() {
|
||||
// A null version written while versioning was suspended sits between
|
||||
// version one and a dangling version three. The walk reaches the null
|
||||
// slot through the nil UUID and serves it, not version one.
|
||||
let bucket = "dangling-latest-null";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
let v1 = put(&set, bucket, object, b"version one", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(b"null version".to_vec()),
|
||||
&ObjectOptions {
|
||||
version_suspended: true,
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("a suspended-versioning PUT should commit the null version");
|
||||
let v3 = put(&set, bucket, object, b"version three", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
drop_version_from(&set, bucket, object, v3, 3..DRIVES).await;
|
||||
|
||||
let latest = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("the committed null version must be served");
|
||||
assert_ne!(latest.version_id, Some(v1));
|
||||
assert_ne!(latest.version_id, Some(v3));
|
||||
assert!(latest.is_latest);
|
||||
assert_eq!(get(&set, bucket, object, None).await.expect("latest GET"), b"null version");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_orders_versions_by_their_newest_modification_time() {
|
||||
// Drive 0 missed the suspended-versioning rewrite of the null version,
|
||||
// so its null copy predates version one while every other drive holds
|
||||
// the committed rewrite. Ordering by the first drive's time would try
|
||||
// version one before the null version and serve it.
|
||||
let bucket = "dangling-latest-order";
|
||||
let object = "torn/latest";
|
||||
let (dirs, set) = dangling_test_set(bucket).await;
|
||||
let suspended = ObjectOptions {
|
||||
version_suspended: true,
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
};
|
||||
assert!(is_read_quorum_error(&err), "a split latest version keeps the quorum error: {err:?}");
|
||||
set.put_object(bucket, object, &mut PutObjReader::from_vec(b"stale null".to_vec()), &suspended)
|
||||
.await
|
||||
.expect("the first null version should commit");
|
||||
let v1 = put(&set, bucket, object, b"version one", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
let drive_zero = object_dir(&dirs, 0, bucket, object).join(STORAGE_FORMAT_FILE);
|
||||
let stale = tokio::fs::read(&drive_zero)
|
||||
.await
|
||||
.expect("drive zero metadata should be readable");
|
||||
set.put_object(bucket, object, &mut PutObjReader::from_vec(b"committed null".to_vec()), &suspended)
|
||||
.await
|
||||
.expect("the null rewrite should commit");
|
||||
tokio::fs::write(&drive_zero, stale)
|
||||
.await
|
||||
.expect("drive zero should keep the pre-rewrite stack");
|
||||
let v5 = put(&set, bucket, object, b"version five", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
drop_version_from(&set, bucket, object, v5, 3..DRIVES).await;
|
||||
|
||||
let latest = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("the committed null rewrite must be served");
|
||||
assert_ne!(latest.version_id, Some(v1));
|
||||
assert_eq!(get(&set, bucket, object, None).await.expect("latest GET"), b"committed null");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_preserves_object_before_equal_time_marker() {
|
||||
let bucket = "dangling-latest-equal-time-marker";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
let same_time = time::OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixed source mtime");
|
||||
for suspended in [false, true] {
|
||||
let object = if suspended {
|
||||
"torn/equal-time-null"
|
||||
} else {
|
||||
"torn/equal-time-versioned"
|
||||
};
|
||||
let object_id = if suspended { Uuid::nil() } else { Uuid::from_u128(2) };
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(b"committed object".to_vec()),
|
||||
&ObjectOptions {
|
||||
versioned: !suspended,
|
||||
version_suspended: suspended,
|
||||
replication_request: true,
|
||||
version_id: Some(object_id.to_string()),
|
||||
mod_time: Some(same_time),
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("commit object version");
|
||||
set.delete_object(
|
||||
bucket,
|
||||
object,
|
||||
ObjectOptions {
|
||||
versioned: true,
|
||||
version_id: Some(Uuid::from_u128(1).to_string()),
|
||||
mod_time: Some(same_time),
|
||||
data_movement: true,
|
||||
delete_marker: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("recreate source delete marker during data movement");
|
||||
let expected = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("canonical latest before dangling write");
|
||||
assert_eq!(expected.version_id, (!suspended).then_some(object_id));
|
||||
assert!(!expected.delete_marker, "canonical header order prefers object to equal-time marker");
|
||||
assert_eq!(get(&set, bucket, object, None).await.expect("canonical GET"), b"committed object");
|
||||
let dangling_id = Uuid::from_u128(3);
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(b"failed newer version".to_vec()),
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
replication_request: true,
|
||||
version_id: Some(dangling_id.to_string()),
|
||||
mod_time: Some(same_time + time::Duration::seconds(1)),
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("commit newer version before modeling failed rollback");
|
||||
drop_version_from(&set, bucket, object, dangling_id, 3..DRIVES).await;
|
||||
let actual = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("latest HEAD past dangling version");
|
||||
let actual_get = get(&set, bucket, object, None).await;
|
||||
assert_eq!(
|
||||
actual.version_id, expected.version_id,
|
||||
"{object}: unreadable newer version must preserve canonical equal-time latest; GET result: {actual_get:?}"
|
||||
);
|
||||
assert!(
|
||||
!actual.delete_marker,
|
||||
"{object}: fallback must preserve the object before its equal-time marker"
|
||||
);
|
||||
assert_eq!(actual_get.expect("latest GET"), b"committed object");
|
||||
|
||||
let checksum_opts = ObjectOptions {
|
||||
include_part_checksums: true,
|
||||
..Default::default()
|
||||
};
|
||||
let checksum_head = set
|
||||
.get_object_info(bucket, object, &checksum_opts)
|
||||
.await
|
||||
.expect("latest HEAD with part checksums");
|
||||
assert_eq!(checksum_head.version_id, expected.version_id, "{object}: checksum HEAD version");
|
||||
assert!(!checksum_head.delete_marker, "{object}: checksum HEAD must return the object");
|
||||
let mut checksum_reader = set
|
||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &checksum_opts)
|
||||
.await
|
||||
.expect("latest GET with part checksums");
|
||||
let mut checksum_body = Vec::new();
|
||||
checksum_reader
|
||||
.stream
|
||||
.read_to_end(&mut checksum_body)
|
||||
.await
|
||||
.expect("checksum GET should stream the committed bytes");
|
||||
assert_eq!(checksum_body, b"committed object", "{object}: checksum GET bytes");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_read_past_dangling_versions_requires_every_drive_to_answer() {
|
||||
let bucket = "dangling-latest-offline";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
put(&set, bucket, object, b"version one", true).await;
|
||||
let v2 = put(&set, bucket, object, b"version two", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
drop_version_from(&set, bucket, object, v2, 3..DRIVES).await;
|
||||
// The offline drive may hold the copy that makes version two readable.
|
||||
let offline = set.disks.write().await[5].take().expect("drive five should start online");
|
||||
|
||||
assert_get_and_head_keep_read_quorum_error(&set, bucket, object, "one drive offline").await;
|
||||
|
||||
set.disks.write().await[5] = Some(offline);
|
||||
let latest = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("with every drive answering the dangling version is skipped");
|
||||
assert_ne!(latest.version_id, Some(v2));
|
||||
}
|
||||
|
||||
/// Version one on every drive under a version two that a failed PUT left
|
||||
/// on drives `0..3`, below the read quorum of four.
|
||||
async fn put_committed_below_dangling(set: &SetDisks, bucket: &str, object: &str) -> (Uuid, Uuid) {
|
||||
let v1 = put(set, bucket, object, b"version one", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
let v2 = put(set, bucket, object, b"version two", true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id");
|
||||
drop_version_from(set, bucket, object, v2, 3..DRIVES).await;
|
||||
(v1, v2)
|
||||
}
|
||||
|
||||
/// Each drive's copy of `version_id`, or `None` where the drive lacks it.
|
||||
async fn version_metadata(
|
||||
set: &SetDisks,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Uuid,
|
||||
) -> Vec<Option<HashMap<String, String>>> {
|
||||
let disks = set.disks.read().await.clone();
|
||||
let mut copies = Vec::with_capacity(disks.len());
|
||||
for disk in &disks {
|
||||
let disk = disk.as_ref().expect("every drive should be online");
|
||||
copies.push(
|
||||
disk.read_version("", bucket, object, &version_id.to_string(), &ReadOptions::default())
|
||||
.await
|
||||
.ok()
|
||||
.map(|fi| fi.metadata),
|
||||
);
|
||||
}
|
||||
copies
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_tagging_skips_dangling_versions_above_a_committed_version() {
|
||||
let bucket = "dangling-latest-tagging";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
let (v1, v2) = put_committed_below_dangling(&set, bucket, object).await;
|
||||
let tag_of = |metadata: &Option<HashMap<String, String>>| {
|
||||
metadata
|
||||
.as_ref()
|
||||
.and_then(|metadata| metadata.get(rustfs_utils::http::headers::AMZ_OBJECT_TAGGING).cloned())
|
||||
};
|
||||
|
||||
let tagged = set
|
||||
.put_object_tags(bucket, object, "stage=committed", &ObjectOptions::default())
|
||||
.await
|
||||
.expect("PutObjectTagging must update the version a latest GET serves");
|
||||
assert_eq!(tagged.version_id, Some(v1));
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
assert_eq!(
|
||||
tag_of(metadata).as_deref(),
|
||||
Some("stage=committed"),
|
||||
"drive {disk} must carry the tag on version one"
|
||||
);
|
||||
}
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v2).await.iter().enumerate() {
|
||||
assert_eq!(tag_of(metadata), None, "drive {disk} must leave the dangling version untouched");
|
||||
}
|
||||
assert_eq!(*head(&set, bucket, object, None).await.expect("latest HEAD").user_tags, "stage=committed");
|
||||
|
||||
let untagged = set
|
||||
.delete_object_tags(bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("DeleteObjectTagging must update the version a latest GET serves");
|
||||
assert_eq!(untagged.version_id, Some(v1));
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
assert_eq!(tag_of(metadata).as_deref(), Some(""), "drive {disk} must clear the tag on version one");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_metadata_update_skips_dangling_versions_above_a_committed_version() {
|
||||
let bucket = "dangling-latest-metadata";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
let (v1, v2) = put_committed_below_dangling(&set, bucket, object).await;
|
||||
|
||||
let updated = set
|
||||
.put_object_metadata(
|
||||
bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
eval_metadata: Some(HashMap::from([("x-amz-meta-probe".to_string(), "updated".to_string())])),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("a latest metadata update must target the version a latest GET serves");
|
||||
assert_eq!(updated.version_id, Some(v1));
|
||||
assert!(updated.is_latest, "the updated version is the latest one");
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
let metadata = metadata.as_ref().expect("version one is on every drive");
|
||||
assert_eq!(metadata.get("x-amz-meta-probe").map(String::as_str), Some("updated"), "drive {disk}");
|
||||
}
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v2).await.iter().enumerate() {
|
||||
assert!(
|
||||
metadata
|
||||
.as_ref()
|
||||
.is_none_or(|metadata| !metadata.contains_key("x-amz-meta-probe")),
|
||||
"drive {disk} must leave the dangling version untouched"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_copy_source_skips_dangling_versions_above_a_committed_version() {
|
||||
let bucket = "dangling-latest-copy";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
|
||||
// A metadata-only self-copy rewrites the latest version in place.
|
||||
let object = "torn/in-place";
|
||||
let (v1, _) = put_committed_below_dangling(&set, bucket, object).await;
|
||||
let mut source = head(&set, bucket, object, None).await.expect("latest HEAD");
|
||||
assert_eq!(source.version_id, Some(v1));
|
||||
source.metadata_only = true;
|
||||
source.user_defined = Arc::new(HashMap::from([("x-amz-meta-copied".to_string(), "in-place".to_string())]));
|
||||
let copied = set
|
||||
.copy_object(
|
||||
bucket,
|
||||
object,
|
||||
bucket,
|
||||
object,
|
||||
&mut source,
|
||||
&ObjectOptions::default(),
|
||||
&ObjectOptions::default(),
|
||||
)
|
||||
.await
|
||||
.expect("a CopyObject source without a version id must resolve like a latest GET");
|
||||
assert_eq!(copied.version_id, Some(v1));
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
let metadata = metadata.as_ref().expect("version one is on every drive");
|
||||
assert_eq!(metadata.get("x-amz-meta-copied").map(String::as_str), Some("in-place"), "drive {disk}");
|
||||
}
|
||||
|
||||
// A version-only self-copy commits a new version that shares the
|
||||
// served version's data.
|
||||
let object = "torn/new-version";
|
||||
let (_, v2) = put_committed_below_dangling(&set, bucket, object).await;
|
||||
let v3 = Uuid::new_v4();
|
||||
let mut source = head(&set, bucket, object, None).await.expect("latest HEAD");
|
||||
source.version_only = true;
|
||||
let copied = set
|
||||
.copy_object(
|
||||
bucket,
|
||||
object,
|
||||
bucket,
|
||||
object,
|
||||
&mut source,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
},
|
||||
&ObjectOptions {
|
||||
version_id: Some(v3.to_string()),
|
||||
versioned: true,
|
||||
object_lock_config_snapshot: Some(Arc::new(crate::object_api::ObjectLockConfigSnapshot::new(
|
||||
crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent,
|
||||
))),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("a version-only CopyObject source without a version id must resolve like a latest GET");
|
||||
assert_eq!(copied.version_id, Some(v3));
|
||||
let latest = head(&set, bucket, object, None).await.expect("latest HEAD");
|
||||
assert_eq!(latest.version_id, Some(v3));
|
||||
assert_eq!(get(&set, bucket, object, None).await.expect("latest GET"), b"version one");
|
||||
assert!(
|
||||
version_metadata(&set, bucket, object, v2).await[..3]
|
||||
.iter()
|
||||
.all(Option::is_some),
|
||||
"the copy must not remove the dangling version; heal owns its cleanup"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_metadata_update_past_dangling_versions_keeps_write_quorum() {
|
||||
// EC 3+3: read quorum three, write quorum four. Version one reached
|
||||
// three drives, so it is readable but below write quorum; versions two
|
||||
// and three are dangling on two drives each. A latest read serves
|
||||
// version one, and a metadata update of it is refused before writing.
|
||||
let bucket = "dangling-latest-write-quorum";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set_with(bucket, 6, 3).await;
|
||||
let mut versions = Vec::new();
|
||||
for body in [b"version zero".as_slice(), b"version one", b"version two", b"version three"] {
|
||||
versions.push(
|
||||
put(&set, bucket, object, body, true)
|
||||
.await
|
||||
.version_id
|
||||
.expect("versioned PUT should assign a version id"),
|
||||
);
|
||||
}
|
||||
let [_, v1, v2, v3] = versions[..] else {
|
||||
unreachable!("four versions were written");
|
||||
};
|
||||
drop_version_from(&set, bucket, object, v1, 3..6).await;
|
||||
drop_version_from(&set, bucket, object, v2, 2..6).await;
|
||||
drop_version_from(&set, bucket, object, v3, 0..3).await;
|
||||
drop_version_from(&set, bucket, object, v3, 5..6).await;
|
||||
|
||||
let latest = head(&set, bucket, object, None)
|
||||
.await
|
||||
.expect("version one is readable at read quorum");
|
||||
assert_eq!(latest.version_id, Some(v1));
|
||||
|
||||
let is_write_quorum_error =
|
||||
|err: &StorageError| matches!(err, StorageError::ErasureWriteQuorum | StorageError::InsufficientWriteQuorum(_, _));
|
||||
let Err(err) = set
|
||||
.put_object_tags(bucket, object, "stage=refused", &ObjectOptions::default())
|
||||
.await
|
||||
else {
|
||||
panic!("tagging a version below write quorum must be refused");
|
||||
};
|
||||
assert!(is_write_quorum_error(&err), "tagging keeps the write-quorum error: {err:?}");
|
||||
let Err(err) = set
|
||||
.put_object_metadata(
|
||||
bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
eval_metadata: Some(HashMap::from([("x-amz-meta-probe".to_string(), "refused".to_string())])),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
else {
|
||||
panic!("a metadata update of a version below write quorum must be refused");
|
||||
};
|
||||
assert!(is_write_quorum_error(&err), "the metadata update keeps the write-quorum error: {err:?}");
|
||||
let v4 = Uuid::new_v4();
|
||||
let mut source = head(&set, bucket, object, None).await.expect("latest HEAD");
|
||||
source.version_only = true;
|
||||
let Err(err) = set
|
||||
.copy_object(
|
||||
bucket,
|
||||
object,
|
||||
bucket,
|
||||
object,
|
||||
&mut source,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
},
|
||||
&ObjectOptions {
|
||||
version_id: Some(v4.to_string()),
|
||||
versioned: true,
|
||||
object_lock_config_snapshot: Some(Arc::new(crate::object_api::ObjectLockConfigSnapshot::new(
|
||||
crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent,
|
||||
))),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
else {
|
||||
panic!("a version-only copy of a version below write quorum must be refused");
|
||||
};
|
||||
assert!(is_write_quorum_error(&err), "the version-only copy keeps the write-quorum error: {err:?}");
|
||||
assert!(
|
||||
version_metadata(&set, bucket, object, v4).await.iter().all(Option::is_none),
|
||||
"a refused version-only copy must not write the new version to any drive"
|
||||
);
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
assert!(
|
||||
metadata.as_ref().is_none_or(|metadata| {
|
||||
!metadata.contains_key(rustfs_utils::http::headers::AMZ_OBJECT_TAGGING)
|
||||
&& !metadata.contains_key("x-amz-meta-probe")
|
||||
}),
|
||||
"drive {disk} must not be written by a refused update"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn replication_status_writeback_never_walks_past_dangling_versions() {
|
||||
// A replication terminal CAS stays side-effect free: it keeps the
|
||||
// read-quorum error instead of resolving the latest version, so it
|
||||
// neither queues heal for the dangling version nor writes version one.
|
||||
let bucket = "dangling-latest-replication-cas";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
let (v1, _) = put_committed_below_dangling(&set, bucket, object).await;
|
||||
|
||||
let Err(err) = set
|
||||
.put_object_metadata(
|
||||
bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
eval_metadata: Some(HashMap::from([("x-amz-meta-probe".to_string(), "written".to_string())])),
|
||||
replication_status_writeback: Some(Box::new(crate::object_api::ReplicationStatusWritebackCondition {
|
||||
mode: crate::object_api::ReplicationStatusWritebackMode::ValidateOnly,
|
||||
..Default::default()
|
||||
})),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
else {
|
||||
panic!("a replication terminal CAS must not resolve past a dangling version");
|
||||
};
|
||||
assert!(is_read_quorum_error(&err), "the CAS keeps the read-quorum error: {err:?}");
|
||||
for (disk, metadata) in version_metadata(&set, bucket, object, v1).await.iter().enumerate() {
|
||||
let metadata = metadata.as_ref().expect("version one is on every drive");
|
||||
assert!(!metadata.contains_key("x-amz-meta-probe"), "drive {disk} must not be written");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn latest_metadata_updates_past_dangling_versions_require_every_drive_to_answer() {
|
||||
let bucket = "dangling-latest-update-offline";
|
||||
let object = "torn/latest";
|
||||
let (_dirs, set) = dangling_test_set(bucket).await;
|
||||
put_committed_below_dangling(&set, bucket, object).await;
|
||||
let offline = set.disks.write().await[5].take().expect("drive five should start online");
|
||||
|
||||
let Err(err) = set
|
||||
.put_object_tags(bucket, object, "stage=uncertain", &ObjectOptions::default())
|
||||
.await
|
||||
else {
|
||||
panic!("tagging must not resolve past a version an offline drive may commit");
|
||||
};
|
||||
assert!(is_read_quorum_error(&err), "tagging keeps the read-quorum error: {err:?}");
|
||||
let Err(err) = set
|
||||
.put_object_metadata(
|
||||
bucket,
|
||||
object,
|
||||
&ObjectOptions {
|
||||
versioned: true,
|
||||
eval_metadata: Some(HashMap::from([("x-amz-meta-probe".to_string(), "uncertain".to_string())])),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
else {
|
||||
panic!("a metadata update must not resolve past a version an offline drive may commit");
|
||||
};
|
||||
assert!(is_read_quorum_error(&err), "the metadata update keeps the read-quorum error: {err:?}");
|
||||
set.disks.write().await[5] = Some(offline);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -448,6 +448,14 @@ impl ECStore {
|
||||
await_orphan_bucket_recovery(bucket, &publication_guard, &lifecycle_guard, &metadata_guard, &ns_guard, recover).await
|
||||
}
|
||||
|
||||
/// Read quota and bucket identity from one authoritative metadata snapshot.
|
||||
pub async fn bucket_quota_and_incarnation_from_disk(
|
||||
&self,
|
||||
bucket: &str,
|
||||
) -> Result<(Option<crate::bucket::quota::BucketQuota>, Uuid, OffsetDateTime)> {
|
||||
metadata_sys::get_quota_config_and_incarnation_from_disk_in(&self.ctx, bucket).await
|
||||
}
|
||||
|
||||
pub async fn get_bucket_metadata(&self, bucket: &str) -> Result<Arc<BucketMetadata>> {
|
||||
let sys = metadata_sys::require_bucket_metadata_sys_in(&self.ctx)?;
|
||||
sys.read().await.get(bucket).await
|
||||
@@ -564,13 +572,83 @@ impl ECStore {
|
||||
.await
|
||||
}
|
||||
|
||||
pub(super) async fn acquire_bucket_incarnation_fence_with_options(
|
||||
pub(crate) async fn acquire_bucket_incarnation_fence_with_options(
|
||||
&self,
|
||||
bucket: &str,
|
||||
expected: Uuid,
|
||||
opts: Option<&ObjectOptions>,
|
||||
) -> Result<super::BucketIncarnationFenceGuard> {
|
||||
Ok(self
|
||||
.acquire_bucket_incarnation_fence_for_options(bucket, Some(expected), opts)
|
||||
.await?
|
||||
.0)
|
||||
}
|
||||
|
||||
pub(crate) async fn acquire_bucket_incarnation_fence_for_options(
|
||||
&self,
|
||||
bucket: &str,
|
||||
expected: Option<Uuid>,
|
||||
opts: Option<&ObjectOptions>,
|
||||
) -> Result<(super::BucketIncarnationFenceGuard, Uuid)> {
|
||||
let inner = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
|
||||
self.validate_bucket_incarnation_fence_guard_with_options(bucket, inner, expected, opts)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Acquire lexicographically ordered bucket lifecycle read locks for a copy
|
||||
/// operation and validate both bucket incarnations through [`super::BucketFenceRegistry`].
|
||||
pub async fn acquire_copy_bucket_incarnation_fences(
|
||||
&self,
|
||||
source_bucket: &str,
|
||||
expected_source: Option<Uuid>,
|
||||
destination_bucket: &str,
|
||||
expected_destination: Option<Uuid>,
|
||||
) -> Result<(super::BucketIncarnationFenceGuard, Uuid, Option<super::BucketIncarnationFenceGuard>, Uuid)> {
|
||||
let (source_guard, destination_guard) = if source_bucket == destination_bucket {
|
||||
(self.acquire_bucket_lifecycle_read_lock(source_bucket).await?, None)
|
||||
} else if source_bucket < destination_bucket {
|
||||
let source_guard = self.acquire_bucket_lifecycle_read_lock(source_bucket).await?;
|
||||
let destination_guard = self.acquire_bucket_lifecycle_read_lock(destination_bucket).await?;
|
||||
(source_guard, Some(destination_guard))
|
||||
} else {
|
||||
let destination_guard = self.acquire_bucket_lifecycle_read_lock(destination_bucket).await?;
|
||||
let source_guard = self.acquire_bucket_lifecycle_read_lock(source_bucket).await?;
|
||||
(source_guard, Some(destination_guard))
|
||||
};
|
||||
let (source_guard, current_source_incarnation_id) = self
|
||||
.validate_bucket_incarnation_fence_guard_with_options(source_bucket, source_guard, expected_source, None)
|
||||
.await?;
|
||||
let (destination_guard, current_destination_incarnation_id) = if let Some(destination_guard) = destination_guard {
|
||||
let (destination_guard, destination_incarnation_id) = self
|
||||
.validate_bucket_incarnation_fence_guard_with_options(
|
||||
destination_bucket,
|
||||
destination_guard,
|
||||
expected_destination,
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
(Some(destination_guard), destination_incarnation_id)
|
||||
} else {
|
||||
if expected_destination.is_some_and(|expected| expected != current_source_incarnation_id) {
|
||||
return Err(StorageError::BucketNotFound(destination_bucket.to_owned()));
|
||||
}
|
||||
(None, current_source_incarnation_id)
|
||||
};
|
||||
Ok((
|
||||
source_guard,
|
||||
current_source_incarnation_id,
|
||||
destination_guard,
|
||||
current_destination_incarnation_id,
|
||||
))
|
||||
}
|
||||
|
||||
async fn validate_bucket_incarnation_fence_guard_with_options(
|
||||
&self,
|
||||
bucket: &str,
|
||||
inner: rustfs_lock::NamespaceLockGuard,
|
||||
expected: Option<Uuid>,
|
||||
opts: Option<&ObjectOptions>,
|
||||
) -> Result<(super::BucketIncarnationFenceGuard, Uuid)> {
|
||||
let mut guard = super::BucketIncarnationFenceGuard::new(
|
||||
Arc::clone(&self.bucket_fence_registry),
|
||||
bucket,
|
||||
@@ -586,10 +664,10 @@ impl ECStore {
|
||||
}
|
||||
})
|
||||
.await?;
|
||||
if current != expected {
|
||||
if expected.is_some_and(|expected| current != expected) {
|
||||
return Err(StorageError::BucketNotFound(bucket.to_owned()));
|
||||
}
|
||||
Ok(guard)
|
||||
Ok((guard, current))
|
||||
}
|
||||
|
||||
/// Hold this guard through recursive-delete authorization and mutation so
|
||||
|
||||
@@ -149,6 +149,18 @@ impl BucketFenceRegistry {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the memoized incarnation id for `bucket` when at least one
|
||||
/// registered lifecycle read guard is currently held without lock loss.
|
||||
pub(crate) fn validated_incarnation(&self, bucket: &str) -> Option<Uuid> {
|
||||
let mut entries = self.entries.lock().expect("bucket fence registry poisoned");
|
||||
let entry = entries.get_mut(bucket)?;
|
||||
if entry.guards.iter().any(|guard| guard.loss_probe.is_lost()) {
|
||||
entry.validated = None;
|
||||
return None;
|
||||
}
|
||||
entry.validated
|
||||
}
|
||||
}
|
||||
|
||||
/// Owns continuous lifecycle coverage and its incarnation validation memo.
|
||||
@@ -221,7 +233,7 @@ impl BucketIncarnationFenceGuard {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn is_lock_lost(&self) -> bool {
|
||||
pub fn is_lock_lost(&self) -> bool {
|
||||
self.inner.as_ref().is_some_and(NamespaceLockGuard::is_lock_lost)
|
||||
}
|
||||
|
||||
@@ -310,9 +322,16 @@ mod tests {
|
||||
let (first_probe, first_lost) = controllable_probe();
|
||||
let first = reg.enter("b", first_probe);
|
||||
assert_eq!(first.memoized, None);
|
||||
assert_eq!(reg.validated_incarnation("b"), None);
|
||||
reg.memoize("b", uuid(7));
|
||||
assert_eq!(reg.validated_incarnation("b"), Some(uuid(7)));
|
||||
|
||||
first_lost.store(true, std::sync::atomic::Ordering::SeqCst);
|
||||
assert_eq!(
|
||||
reg.validated_incarnation("b"),
|
||||
None,
|
||||
"live lost guard must clear and block validated_incarnation"
|
||||
);
|
||||
|
||||
let second = reg.enter("b", live_probe());
|
||||
assert_eq!(second.memoized, None, "live lost guard must force disk revalidation");
|
||||
@@ -320,9 +339,11 @@ mod tests {
|
||||
|
||||
let third = reg.enter("b", live_probe());
|
||||
assert_eq!(third.memoized, None, "memo remains blocked while the lost guard is live");
|
||||
assert_eq!(reg.validated_incarnation("b"), None);
|
||||
|
||||
reg.exit("b", first.token, true);
|
||||
reg.memoize("b", uuid(8));
|
||||
assert_eq!(reg.validated_incarnation("b"), Some(uuid(8)));
|
||||
let fourth = reg.enter("b", live_probe());
|
||||
assert_eq!(fourth.memoized, Some(uuid(8)), "memo resumes after lost coverage leaves");
|
||||
|
||||
|
||||
@@ -21384,6 +21384,141 @@ mod tests {
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn put_object_metadata_and_copy_object_reuse_overlapping_incarnation_fence() {
|
||||
let temp = tempfile::tempdir().expect("create metadata/copy fence store");
|
||||
let (_ctx, store, shutdown) =
|
||||
without_storage_class_env(build_isolated_test_store(temp.path(), "metadata-copy-fence-reuse", &[4])).await;
|
||||
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
let bucket = format!("metadata-copy-fence-{}", Uuid::new_v4());
|
||||
store
|
||||
.make_bucket(&bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create bucket");
|
||||
let mut seed_reader = PutObjReader::from_vec(b"seed payload".to_vec());
|
||||
let versioned_opts = ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
let mut src_info = store
|
||||
.put_object(&bucket, "src.bin", &mut seed_reader, &versioned_opts)
|
||||
.await
|
||||
.expect("seed source object");
|
||||
let src_version_id = src_info
|
||||
.version_id
|
||||
.expect("versioned PUT should return a version ID")
|
||||
.to_string();
|
||||
let expected = store.bucket_incarnation_id(&bucket).await.expect("read incarnation");
|
||||
let coverage = store
|
||||
.acquire_bucket_incarnation_fence(&bucket, expected)
|
||||
.await
|
||||
.expect("validate continuous coverage");
|
||||
let metadata_writer = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
|
||||
.await
|
||||
.expect("hold metadata writer during covered metadata and copy operations");
|
||||
|
||||
let updated = tokio::time::timeout(
|
||||
Duration::from_secs(10),
|
||||
store.put_object_metadata(
|
||||
&bucket,
|
||||
"src.bin",
|
||||
&ObjectOptions {
|
||||
version_id: Some(src_version_id.clone()),
|
||||
versioned: true,
|
||||
eval_metadata: Some(HashMap::from([("x-amz-meta-fenced".to_string(), "true".to_string())])),
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect("covered put_object_metadata must not reacquire the metadata transaction lock")
|
||||
.expect("update object metadata using authoritative memo");
|
||||
assert_eq!(updated.user_defined.get("x-amz-meta-fenced").map(String::as_str), Some("true"));
|
||||
|
||||
Arc::make_mut(&mut src_info.user_defined).insert("x-amz-meta-copied".to_string(), "true".to_string());
|
||||
let src_copy_opts = ObjectOptions {
|
||||
version_id: Some(src_version_id),
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
let copied = tokio::time::timeout(
|
||||
Duration::from_secs(10),
|
||||
store.copy_object(&bucket, "src.bin", &bucket, "src.bin", &mut src_info, &src_copy_opts, &versioned_opts),
|
||||
)
|
||||
.await
|
||||
.expect("covered versioned copy_object must not reacquire the metadata transaction lock")
|
||||
.expect("versioned copy using authoritative memo");
|
||||
assert_eq!(copied.user_defined.get("x-amz-meta-copied").map(String::as_str), Some("true"));
|
||||
|
||||
let (src_guard, src_incarnation, dst_guard, dst_incarnation) = tokio::time::timeout(
|
||||
Duration::from_secs(10),
|
||||
store.acquire_copy_bucket_incarnation_fences(&bucket, Some(expected), &bucket, Some(expected)),
|
||||
)
|
||||
.await
|
||||
.expect("covered acquire_copy_bucket_incarnation_fences must not reacquire the metadata transaction lock")
|
||||
.expect("validate same-bucket copy fences using authoritative memo");
|
||||
assert!(dst_guard.is_none());
|
||||
assert_eq!(src_incarnation, expected);
|
||||
assert_eq!(dst_incarnation, expected);
|
||||
let mut borrowed_fence_opts = ObjectOptions {
|
||||
expected_bucket_incarnation_id: Some(expected),
|
||||
..Default::default()
|
||||
};
|
||||
src_guard.attach_to_object_options(&mut borrowed_fence_opts);
|
||||
let fenced_incarnation = tokio::time::timeout(
|
||||
Duration::from_secs(10),
|
||||
crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&store.ctx, &bucket, &borrowed_fence_opts),
|
||||
)
|
||||
.await
|
||||
.expect("borrowed fence under active coverage must not reacquire the metadata transaction lock")
|
||||
.expect("resolve incarnation under borrowed fence");
|
||||
assert_eq!(fenced_incarnation, expected);
|
||||
assert!(
|
||||
matches!(
|
||||
store
|
||||
.acquire_copy_bucket_incarnation_fences(&bucket, Some(expected), &bucket, Some(Uuid::new_v4()))
|
||||
.await,
|
||||
Err(StorageError::BucketNotFound(_))
|
||||
),
|
||||
"same-bucket copy fence must reject a mismatched destination incarnation"
|
||||
);
|
||||
drop(src_guard);
|
||||
|
||||
let stale_opts = ObjectOptions {
|
||||
expected_bucket_incarnation_id: Some(Uuid::new_v4()),
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
matches!(
|
||||
store.put_object_metadata(&bucket, "src.bin", &stale_opts).await,
|
||||
Err(StorageError::BucketNotFound(_))
|
||||
),
|
||||
"put_object_metadata must reject a stale expected incarnation even under warm coverage"
|
||||
);
|
||||
assert!(
|
||||
matches!(
|
||||
store
|
||||
.copy_object(
|
||||
&bucket,
|
||||
"src.bin",
|
||||
&bucket,
|
||||
"src.bin",
|
||||
&mut src_info,
|
||||
&ObjectOptions::default(),
|
||||
&stale_opts,
|
||||
)
|
||||
.await,
|
||||
Err(StorageError::BucketNotFound(_))
|
||||
),
|
||||
"copy_object must reject a stale expected incarnation even under warm coverage"
|
||||
);
|
||||
drop(metadata_writer);
|
||||
drop(coverage);
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn multipart_snapshot_bypasses_initializer_queued_behind_metadata_writer() {
|
||||
@@ -21469,10 +21604,25 @@ mod tests {
|
||||
.acquire_bucket_incarnation_fence(&bucket, old)
|
||||
.await
|
||||
.expect("validate with no-op lock");
|
||||
let snapshot = store
|
||||
.object_lock_config_snapshot_under_lifecycle_fence(&bucket, &crate::object_api::NamespaceLockFence::new())
|
||||
.await
|
||||
.expect("capture snapshot under no-op lock");
|
||||
let current = Uuid::new_v4();
|
||||
crate::bucket::metadata::save_bucket_incarnation(store.clone(), &bucket, current)
|
||||
.await
|
||||
.expect("replace authority while no-op guard provides no exclusion");
|
||||
let snapshot_opts = ObjectOptions {
|
||||
expected_bucket_incarnation_id: Some(old),
|
||||
object_lock_config_snapshot: Some(snapshot),
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(
|
||||
crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&store.ctx, &bucket, &snapshot_opts)
|
||||
.await
|
||||
.expect("disabled lock manager must re-read incarnation from disk instead of trusting snapshot"),
|
||||
current
|
||||
);
|
||||
assert!(
|
||||
matches!(
|
||||
store.acquire_bucket_incarnation_fence(&bucket, old).await,
|
||||
|
||||
@@ -1030,16 +1030,19 @@ impl ECStore {
|
||||
// Lock order matches writes: bucket lifecycle, metadata validation,
|
||||
// then object namespace. Borrow fences supplied by an outer operation.
|
||||
let bucket_guard = if opts.bucket_lifecycle_lock_fence.is_none() {
|
||||
let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
|
||||
opts.add_bucket_lifecycle_lock_guard(&guard);
|
||||
let guard = self
|
||||
.acquire_bucket_incarnation_fence_with_options(bucket, bucket_incarnation_id, Some(&opts))
|
||||
.await?;
|
||||
guard.attach_to_object_options(&mut opts);
|
||||
Some(guard)
|
||||
} else {
|
||||
let incarnation =
|
||||
crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
|
||||
if incarnation != bucket_incarnation_id {
|
||||
return Err(Error::BucketNotFound(bucket.to_owned()));
|
||||
}
|
||||
None
|
||||
};
|
||||
let incarnation = crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
|
||||
if incarnation != bucket_incarnation_id {
|
||||
return Err(Error::BucketNotFound(bucket.to_owned()));
|
||||
}
|
||||
let metadata_sys = crate::bucket::metadata_sys::bucket_metadata_sys_of(&self.ctx)?;
|
||||
let metadata_sys = metadata_sys.read().await.clone();
|
||||
let (versioning, _) = metadata_sys.get_versioning_config(bucket).await?;
|
||||
|
||||
@@ -452,6 +452,8 @@ pub use object::{
|
||||
};
|
||||
mod peer;
|
||||
mod rebalance;
|
||||
#[cfg(test)]
|
||||
pub(crate) use rebalance::PoolSpaceForTest;
|
||||
pub(crate) mod utils;
|
||||
|
||||
use peer::init_local_peer;
|
||||
|
||||
@@ -412,7 +412,7 @@ impl ECStore {
|
||||
|
||||
/// Multipart lock order is bucket lifecycle, generation validation, then
|
||||
/// object/upload locks in the selected set.
|
||||
async fn guard_multipart_bucket_incarnation(
|
||||
pub(super) async fn guard_multipart_bucket_incarnation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
opts: &ObjectOptions,
|
||||
|
||||
@@ -30,8 +30,8 @@ use crate::bucket::lifecycle::{
|
||||
},
|
||||
};
|
||||
use crate::bucket::metadata_sys::{
|
||||
acquire_bucket_metadata_transaction_read_lock_in, get_bucket_incarnation_id_for_options_in,
|
||||
get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in, get_object_lock_config_and_incarnation_from_disk_in,
|
||||
acquire_bucket_metadata_transaction_read_lock_in, get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in,
|
||||
get_object_lock_config_and_incarnation_from_disk_in,
|
||||
};
|
||||
use crate::bucket::object_lock::objectlock_sys::{
|
||||
check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state,
|
||||
@@ -4291,8 +4291,40 @@ impl ECStore {
|
||||
.select_decommission_capacity_target_pool(owner, expected_data_bytes)
|
||||
.await;
|
||||
}
|
||||
let holder = self
|
||||
.data_movement_pool_holding_version(bucket, object, opts.version_id.as_deref(), opts.src_pool_idx)
|
||||
.await;
|
||||
self.get_available_pool_idx_preferring(bucket, object, size, None, holder)
|
||||
.await
|
||||
.ok_or(Error::DiskFull)
|
||||
}
|
||||
|
||||
self.get_available_pool_idx(bucket, object, size).await.ok_or(Error::DiskFull)
|
||||
/// The target pool that already holds a copy of the exact version being
|
||||
/// moved. An attempt that lost write quorum can leave that copy on fewer
|
||||
/// drives than read quorum, which the pool lookup reports as absent; a
|
||||
/// retry that placed the version on another pool would leave it in two.
|
||||
/// Callers still subject the holder to the free-space checks, so a full
|
||||
/// holder cannot pin every retry of a deferred entry.
|
||||
async fn data_movement_pool_holding_version(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<&str>,
|
||||
src_pool_idx: usize,
|
||||
) -> Option<usize> {
|
||||
let version_id = Uuid::parse_str(version_id?).ok()?;
|
||||
let holders = join_all(self.pools.iter().enumerate().map(|(pool_idx, pool)| async move {
|
||||
// Match the pools the free-space choice can return.
|
||||
if pool_idx == src_pool_idx || self.is_suspended(pool_idx).await || self.is_pool_rebalancing(pool_idx).await {
|
||||
return None;
|
||||
}
|
||||
pool.get_disks_by_key(object)
|
||||
.holds_version_on_any_drive(bucket, object, version_id)
|
||||
.await
|
||||
.then_some(pool_idx)
|
||||
}))
|
||||
.await;
|
||||
holders.into_iter().flatten().next()
|
||||
}
|
||||
|
||||
async fn find_data_movement_target_info(
|
||||
@@ -4851,24 +4883,20 @@ impl ECStore {
|
||||
if !is_meta_bucketname(dst_bucket) && dst_opts.expected_bucket_incarnation_id.is_none() {
|
||||
dst_opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(dst_bucket).await?);
|
||||
}
|
||||
let _bucket_lifecycle_guard = if is_meta_bucketname(dst_bucket) || dst_opts.bucket_lifecycle_lock_fence.is_some() {
|
||||
None
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(dst_bucket).await?)
|
||||
};
|
||||
let current_bucket_incarnation_id = if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
|
||||
dst_opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, dst_bucket, &dst_opts).await?;
|
||||
if dst_opts
|
||||
.expected_bucket_incarnation_id
|
||||
.is_some_and(|expected| expected != current_incarnation_id)
|
||||
{
|
||||
return Err(StorageError::BucketNotFound(dst_bucket.to_string()));
|
||||
}
|
||||
Some(current_incarnation_id)
|
||||
} else {
|
||||
dst_opts.expected_bucket_incarnation_id
|
||||
};
|
||||
let (_bucket_lifecycle_guard, current_bucket_incarnation_id) =
|
||||
if is_meta_bucketname(dst_bucket) || dst_opts.bucket_lifecycle_lock_fence.is_some() {
|
||||
(None, dst_opts.expected_bucket_incarnation_id)
|
||||
} else {
|
||||
let (guard, current_incarnation_id) = self
|
||||
.acquire_bucket_incarnation_fence_for_options(
|
||||
dst_bucket,
|
||||
dst_opts.expected_bucket_incarnation_id,
|
||||
Some(&dst_opts),
|
||||
)
|
||||
.await?;
|
||||
guard.attach_to_object_options(&mut dst_opts);
|
||||
(Some(guard), Some(current_incarnation_id))
|
||||
};
|
||||
if dst_opts
|
||||
.bucket_lifecycle_lock_fence
|
||||
.as_ref()
|
||||
@@ -5420,7 +5448,10 @@ impl ECStore {
|
||||
if let Some(owner) = DecommissionCapacityOwner::from_options(&opts) {
|
||||
self.select_decommission_capacity_target_pool(owner, 0).await?
|
||||
} else {
|
||||
self.get_available_pool_idx_excluding(bucket, object, 0, opts.src_pool_idx)
|
||||
let holder = self
|
||||
.data_movement_pool_holding_version(bucket, object, opts.version_id.as_deref(), opts.src_pool_idx)
|
||||
.await;
|
||||
self.get_available_pool_idx_preferring(bucket, object, 0, Some(opts.src_pool_idx), holder)
|
||||
.await
|
||||
.ok_or(Error::DiskFull)?
|
||||
}
|
||||
@@ -6469,24 +6500,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<()> {
|
||||
let object = encode_dir_object(object);
|
||||
let mut opts = transition_restore_pool_opts(opts);
|
||||
if !is_meta_bucketname(bucket) && opts.expected_bucket_incarnation_id.is_none() {
|
||||
opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(bucket).await?);
|
||||
}
|
||||
let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
|
||||
};
|
||||
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
|
||||
opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
if !is_meta_bucketname(bucket) {
|
||||
let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
|
||||
if opts.expected_bucket_incarnation_id != Some(current_incarnation_id) {
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
}
|
||||
let opts = transition_restore_pool_opts(opts);
|
||||
let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, &opts).await?;
|
||||
if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
|
||||
let expected_incarnation_id = opts
|
||||
.expected_bucket_incarnation_id
|
||||
@@ -6554,16 +6569,11 @@ impl ECStore {
|
||||
let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
|
||||
let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
|
||||
if opts
|
||||
.expected_bucket_incarnation_id
|
||||
.is_some_and(|expected| expected != current_incarnation_id)
|
||||
{
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
let (guard, current_incarnation_id) = self
|
||||
.acquire_bucket_incarnation_fence_for_options(bucket, opts.expected_bucket_incarnation_id, Some(&opts))
|
||||
.await?;
|
||||
opts.expected_bucket_incarnation_id = Some(current_incarnation_id);
|
||||
opts.add_bucket_lifecycle_lock_guard(&guard);
|
||||
guard.attach_to_object_options(&mut opts);
|
||||
if guard.is_lock_lost() {
|
||||
return Err(Error::other("bucket lifecycle lock was lost before the metadata update"));
|
||||
}
|
||||
@@ -13132,6 +13142,202 @@ mod tests {
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn assert_delete_excludes_recovery_across_pool_lock_domains(topology: &str, batch: bool) {
|
||||
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
|
||||
let (_first_dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
|
||||
let (_second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
|
||||
let mut store = new_prepared_reader_test_store_with_ctx(&[first_set, second_set], Arc::clone(&ctx)).await;
|
||||
let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..4)
|
||||
.map(|_| {
|
||||
Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(
|
||||
Arc::new(rustfs_lock::FastObjectLockManager::new()),
|
||||
)))) as Arc<dyn rustfs_lock::LockClient>
|
||||
})
|
||||
.collect();
|
||||
let domains = match topology {
|
||||
"disjoint" => [vec![clients[0].clone()], vec![clients[1].clone()]],
|
||||
"identical" => [clients[..3].to_vec(), clients[..3].to_vec()],
|
||||
"overlapping" => [clients[..3].to_vec(), clients[1..].to_vec()],
|
||||
_ => panic!("unknown lock topology"),
|
||||
};
|
||||
for (pool, lockers) in store.pools.iter_mut().zip(domains) {
|
||||
let mut set = (*pool.disk_set[0]).clone();
|
||||
set.lockers = lockers;
|
||||
Arc::make_mut(pool).disk_set[0] = Arc::new(set);
|
||||
}
|
||||
let store = Arc::new(store);
|
||||
crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
|
||||
let bucket = "delete-nonzero-pool-lock";
|
||||
let object = "object.bin";
|
||||
store
|
||||
.handle_make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create the bucket in both pools");
|
||||
if batch {
|
||||
// Latest-marker routing sends this batch only to the nonzero pool,
|
||||
// so the cleanup barrier observes its prepared journal directly.
|
||||
store
|
||||
.update_bucket_metadata_config(
|
||||
bucket,
|
||||
crate::bucket::metadata::BUCKET_VERSIONING_CONFIG,
|
||||
b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
|
||||
)
|
||||
.await
|
||||
.expect("enable versioning for the routed batch");
|
||||
}
|
||||
let target = store.pools[1].get_disks_by_key(object);
|
||||
let source = target
|
||||
.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(b"delete lock regression".to_vec()),
|
||||
&ObjectOptions {
|
||||
versioned: batch,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("seed an object only in the nonzero pool");
|
||||
ctx.update_erasure_type(SetupType::DistErasure).await;
|
||||
|
||||
let object_barrier = (!batch).then(|| crate::set_disk::DeleteObjectCommitBarrier::install(bucket, object));
|
||||
let batch_barrier = batch.then(|| crate::set_disk::DeleteCleanupBarrier::install(bucket));
|
||||
let delete_store = Arc::clone(&store);
|
||||
let delete = tokio::spawn(async move {
|
||||
if batch {
|
||||
let (_, errors) = delete_store
|
||||
.delete_objects(
|
||||
bucket,
|
||||
vec![ObjectToDelete {
|
||||
object_name: object.to_owned(),
|
||||
..Default::default()
|
||||
}],
|
||||
ObjectOptions::default(),
|
||||
)
|
||||
.await;
|
||||
match errors.into_iter().flatten().next() {
|
||||
Some(error) => Err(error),
|
||||
None => Ok(()),
|
||||
}
|
||||
} else {
|
||||
delete_store
|
||||
.delete_object(bucket, object, ObjectOptions::default())
|
||||
.await
|
||||
.map(|_| ())
|
||||
}
|
||||
});
|
||||
if let Some(barrier) = &object_barrier {
|
||||
barrier.wait_until_paused().await;
|
||||
}
|
||||
if let Some(barrier) = &batch_barrier {
|
||||
barrier.wait_until_paused().await;
|
||||
}
|
||||
let recovery_store = Arc::clone(&store);
|
||||
let mut recovery = tokio::spawn(async move {
|
||||
recovery_store
|
||||
.recover_delete_transactions_at_incarnation(
|
||||
bucket,
|
||||
Some(object),
|
||||
None,
|
||||
&HealOpts {
|
||||
pool: Some(1),
|
||||
set: Some(0),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
});
|
||||
let early_recovery = tokio::time::timeout(Duration::from_millis(100), &mut recovery).await;
|
||||
let recovery_was_blocked = early_recovery.is_err();
|
||||
if let Some(barrier) = &object_barrier {
|
||||
barrier.release();
|
||||
}
|
||||
if let Some(barrier) = &batch_barrier {
|
||||
barrier.release();
|
||||
}
|
||||
tokio::time::timeout(Duration::from_secs(10), delete)
|
||||
.await
|
||||
.expect("the delete should finish after releasing its commit barrier")
|
||||
.expect("the delete task should join")
|
||||
.expect("the delete should succeed");
|
||||
let recovered = match early_recovery {
|
||||
Ok(result) => result,
|
||||
Err(_) => tokio::time::timeout(Duration::from_secs(10), recovery)
|
||||
.await
|
||||
.expect("recovery should finish after the frontend delete"),
|
||||
};
|
||||
recovered
|
||||
.expect("the recovery task should join")
|
||||
.expect("recovery should succeed");
|
||||
assert!(
|
||||
recovery_was_blocked,
|
||||
"recovery must not inspect an in-flight frontend delete with {topology} pool lock clients"
|
||||
);
|
||||
if batch {
|
||||
let marker = target
|
||||
.get_object_info(bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("the versioned batch must publish a current delete marker");
|
||||
assert!(marker.delete_marker && marker.is_latest);
|
||||
assert_eq!(marker.num_versions, 2, "the source version must remain beside the new marker");
|
||||
assert_ne!(marker.version_id, source.version_id);
|
||||
let mut reader = target
|
||||
.get_object_reader(
|
||||
bucket,
|
||||
object,
|
||||
None,
|
||||
HeaderMap::new(),
|
||||
&ObjectOptions {
|
||||
version_id: Some(source.version_id.expect("versioned source identity").to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("the original source version must remain readable");
|
||||
let mut body = Vec::new();
|
||||
reader
|
||||
.stream
|
||||
.read_to_end(&mut body)
|
||||
.await
|
||||
.expect("read the retained source version");
|
||||
assert_eq!(body, b"delete lock regression");
|
||||
} else {
|
||||
let error = target
|
||||
.get_object_info(bucket, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect_err("the deleted object must be absent");
|
||||
assert!(is_err_object_not_found(&error));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn batch_delete_excludes_recovery_on_nonzero_pool() {
|
||||
assert_delete_excludes_recovery_across_pool_lock_domains("disjoint", true).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn delete_excludes_recovery_with_disjoint_pool_lock_clients() {
|
||||
assert_delete_excludes_recovery_across_pool_lock_domains("disjoint", false).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn delete_excludes_recovery_with_identical_pool_lock_clients() {
|
||||
assert_delete_excludes_recovery_across_pool_lock_domains("identical", false).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
async fn delete_excludes_recovery_with_overlapping_pool_lock_clients() {
|
||||
assert_delete_excludes_recovery_across_pool_lock_domains("overlapping", false).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -122,6 +122,51 @@ async fn pause_prepared_pool_read_before_refetch(object: &str) {
|
||||
state.refetch_release.notified().await;
|
||||
}
|
||||
}
|
||||
/// Capacity that every online drive of one pool reports to free-space target
|
||||
/// selection, per store, until the guard drops.
|
||||
#[cfg(test)]
|
||||
static POOL_SPACE_FOR_TEST: std::sync::Mutex<Vec<(Uuid, usize, crate::disk::DiskInfo)>> = std::sync::Mutex::new(Vec::new());
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct PoolSpaceForTest(Uuid);
|
||||
|
||||
#[cfg(test)]
|
||||
impl PoolSpaceForTest {
|
||||
pub(crate) fn install(store: Uuid, pools: &[(usize, crate::disk::DiskInfo)]) -> Self {
|
||||
POOL_SPACE_FOR_TEST
|
||||
.lock()
|
||||
.expect("pool space override must not be poisoned")
|
||||
.extend(pools.iter().map(|(pool, info)| (store, *pool, info.clone())));
|
||||
Self(store)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for PoolSpaceForTest {
|
||||
fn drop(&mut self) {
|
||||
POOL_SPACE_FOR_TEST
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.retain(|(store, _, _)| *store != self.0);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn pool_space_for_test(
|
||||
store: Uuid,
|
||||
pool: usize,
|
||||
infos: Vec<Option<crate::disk::DiskInfo>>,
|
||||
) -> Vec<Option<crate::disk::DiskInfo>> {
|
||||
let overrides = POOL_SPACE_FOR_TEST.lock().expect("pool space override must not be poisoned");
|
||||
let Some((_, _, info)) = overrides
|
||||
.iter()
|
||||
.find(|(override_store, override_pool, _)| *override_store == store && *override_pool == pool)
|
||||
else {
|
||||
return infos;
|
||||
};
|
||||
infos.into_iter().map(|disk| disk.map(|_| info.clone())).collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
use support::resolve_latest_object_info_candidates;
|
||||
use support::{
|
||||
@@ -438,30 +483,7 @@ impl ECStore {
|
||||
}
|
||||
|
||||
pub(super) async fn get_available_pool_idx(&self, bucket: &str, object: &str, size: i64) -> Option<usize> {
|
||||
// // Return a random one first
|
||||
|
||||
let mut server_pools = self.get_server_pools_available_space(bucket, object, size).await;
|
||||
server_pools.filter_max_used(100 - (100_f64 * DISK_RESERVE_FRACTION) as u64);
|
||||
let total = server_pools.total_available();
|
||||
|
||||
if total == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut rng = rand::rng();
|
||||
let random_u64: u64 = rng.random_range(0..total);
|
||||
|
||||
let choose = random_u64 % total;
|
||||
let mut at_total = 0;
|
||||
|
||||
for pool in server_pools.iter() {
|
||||
at_total += pool.available;
|
||||
if at_total > choose && pool.available > 0 {
|
||||
return Some(pool.index);
|
||||
}
|
||||
}
|
||||
|
||||
None
|
||||
self.get_available_pool_idx_preferring(bucket, object, size, None, None).await
|
||||
}
|
||||
|
||||
pub(super) async fn get_available_pool_idx_excluding(
|
||||
@@ -470,13 +492,33 @@ impl ECStore {
|
||||
object: &str,
|
||||
size: i64,
|
||||
excluded_pool_idx: usize,
|
||||
) -> Option<usize> {
|
||||
self.get_available_pool_idx_preferring(bucket, object, size, Some(excluded_pool_idx), None)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Free-space choice among the pools that can take `size` more bytes,
|
||||
/// weighted by available space. `preferred_pool_idx` is returned whenever
|
||||
/// it passes the same checks, so a full or excluded pool is never forced.
|
||||
pub(super) async fn get_available_pool_idx_preferring(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
size: i64,
|
||||
excluded_pool_idx: Option<usize>,
|
||||
preferred_pool_idx: Option<usize>,
|
||||
) -> Option<usize> {
|
||||
let mut server_pools = self.get_server_pools_available_space(bucket, object, size).await;
|
||||
server_pools.filter_max_used(100 - (100_f64 * DISK_RESERVE_FRACTION) as u64);
|
||||
|
||||
if let Some(pool) = server_pools.0.get_mut(excluded_pool_idx) {
|
||||
if let Some(pool) = excluded_pool_idx.and_then(|idx| server_pools.0.get_mut(idx)) {
|
||||
pool.available = 0;
|
||||
}
|
||||
if let Some(pool_idx) = preferred_pool_idx
|
||||
&& server_pools.0.get(pool_idx).is_some_and(|pool| pool.available > 0)
|
||||
{
|
||||
return Some(pool_idx);
|
||||
}
|
||||
|
||||
let total = server_pools.total_available();
|
||||
if total == 0 {
|
||||
@@ -509,6 +551,8 @@ impl ECStore {
|
||||
|
||||
let disks = pool.get_disks_by_key(object).disk_inventory().await;
|
||||
let disk_infos = get_disk_infos(&disks).await;
|
||||
#[cfg(test)]
|
||||
let disk_infos = pool_space_for_test(self.id, idx, disk_infos);
|
||||
|
||||
(idx, pool.set_count, disk_infos)
|
||||
}))
|
||||
|
||||
@@ -16,7 +16,7 @@ use rustfs_io_metrics::internode_metrics::INTERNODE_OPERATION_PUT_FILE_STREAM;
|
||||
use rustfs_rio::{InternodeHttpError, InternodeHttpErrorKind};
|
||||
use thiserror::Error;
|
||||
|
||||
use super::heal::{DiskError, EcstoreError};
|
||||
use super::heal::{DiskError, EcstoreError, storage_api::owner::EcstorePoolMetadataFailure};
|
||||
|
||||
const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
|
||||
pub(crate) const MRF_MISSING_PROOF_PREFIX: &str = "Missing exact storage proof for durable MRF repair ";
|
||||
@@ -236,6 +236,14 @@ pub(crate) fn classify_heal_error(error: &Error) -> HealErrorClass {
|
||||
/// with object absence, the conflation that once let an offline drive be
|
||||
/// recorded as absent and skipped (backlog#856 / #799 B7).
|
||||
pub(crate) fn classify_storage_error(error: &EcstoreError) -> HealErrorClass {
|
||||
if let Some(failure) = error.pool_metadata_failure() {
|
||||
return match failure.kind {
|
||||
EcstorePoolMetadataFailure::ReadUnavailable | EcstorePoolMetadataFailure::TransactionUnknown => {
|
||||
HealErrorClass::Transient
|
||||
}
|
||||
EcstorePoolMetadataFailure::RecoveryRequired | EcstorePoolMetadataFailure::FenceLost => HealErrorClass::Permanent,
|
||||
};
|
||||
}
|
||||
if let EcstoreError::Lock(lock_error) = error {
|
||||
return if lock_error.is_fatal() {
|
||||
HealErrorClass::Permanent
|
||||
@@ -435,6 +443,34 @@ mod tests {
|
||||
use super::Error;
|
||||
use crate::heal::{DiskError, EcstoreError};
|
||||
|
||||
#[test]
|
||||
fn pool_metadata_failures_keep_their_typed_recovery_classification() {
|
||||
use crate::heal::storage_api::owner::{EcstorePoolMetadataError, EcstorePoolMetadataFailure};
|
||||
|
||||
for (kind, retryable) in [
|
||||
(EcstorePoolMetadataFailure::ReadUnavailable, true),
|
||||
(EcstorePoolMetadataFailure::TransactionUnknown, true),
|
||||
(EcstorePoolMetadataFailure::RecoveryRequired, false),
|
||||
(EcstorePoolMetadataFailure::FenceLost, false),
|
||||
] {
|
||||
let error = EcstoreError::other(EcstorePoolMetadataError {
|
||||
kind,
|
||||
operation: "bucket heal".to_owned(),
|
||||
phase: "native_heal",
|
||||
since: std::time::UNIX_EPOCH.into(),
|
||||
source: Some(std::sync::Arc::new(EcstoreError::SlowDown)),
|
||||
});
|
||||
let text = error.to_string();
|
||||
for error in [error.clone(), error] {
|
||||
assert_eq!(Error::Storage(error).is_recoverable_heal(), retryable, "{kind:?}");
|
||||
}
|
||||
assert!(
|
||||
!Error::Storage(EcstoreError::other(text)).is_recoverable_heal(),
|
||||
"display text must not grant recovery authority"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn namespace_lock_quorum_is_recoverable_across_io_error_conversions() {
|
||||
for (required, achieved) in [(2, 1), (1, 0)] {
|
||||
|
||||
@@ -26,7 +26,11 @@ pub(crate) use rustfs_ecstore::api::disk::{
|
||||
RUSTFS_META_BUCKET as ECSTORE_RUSTFS_META_BUCKET, ReplacementExecutionLease as EcstoreReplacementExecutionLease,
|
||||
};
|
||||
pub(crate) use rustfs_ecstore::api::disk::{DiskOption as EcstoreDiskOption, new_disk as ecstore_new_disk};
|
||||
pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreErrorType, StorageError as EcstoreStorageError};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::error::PoolMetadataError as EcstorePoolMetadataError;
|
||||
pub(crate) use rustfs_ecstore::api::error::{
|
||||
Error as EcstoreErrorType, PoolMetadataFailure as EcstorePoolMetadataFailure, StorageError as EcstoreStorageError,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::layout::{
|
||||
EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints,
|
||||
@@ -53,6 +57,9 @@ pub(crate) use rustfs_ecstore::api::storage::{
|
||||
use rustfs_storage_api as storage_contracts;
|
||||
|
||||
pub(crate) mod owner {
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::EcstorePoolMetadataError;
|
||||
pub(crate) use super::EcstorePoolMetadataFailure;
|
||||
pub(crate) use super::storage_contracts::{ObjectIO, ObjectOperations};
|
||||
|
||||
pub(crate) use super::{
|
||||
|
||||
@@ -306,9 +306,7 @@ impl HealTask {
|
||||
let mut progress = self.progress.write().await;
|
||||
progress.update_stage(3, 3);
|
||||
}
|
||||
Err(Error::TaskExecutionFailed {
|
||||
message: format!("Failed to heal bucket {bucket}: {e}"),
|
||||
})
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -520,9 +518,7 @@ impl HealTask {
|
||||
error = %err,
|
||||
"Heal cluster pool metadata failed"
|
||||
);
|
||||
Err(Error::TaskExecutionFailed {
|
||||
message: format!("Failed to heal cluster pool metadata: {err}"),
|
||||
})
|
||||
Err(err)
|
||||
}
|
||||
Err(Error::TaskCancelled) => Err(Error::TaskCancelled),
|
||||
Err(Error::TaskTimeout) => Err(Error::TaskTimeout),
|
||||
@@ -539,9 +535,7 @@ impl HealTask {
|
||||
error = %err,
|
||||
"Heal cluster pool metadata failed"
|
||||
);
|
||||
Err(Error::TaskExecutionFailed {
|
||||
message: format!("Failed to heal cluster pool metadata: {err}"),
|
||||
})
|
||||
Err(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ use super::super::{DiskOption, DiskStore, Endpoint, new_disk};
|
||||
use super::*;
|
||||
use crate::heal::EcstoreError;
|
||||
use crate::heal::storage::HealStorageObjectResult;
|
||||
use crate::heal::storage_api::owner::{EcstorePoolMetadataError, EcstorePoolMetadataFailure};
|
||||
|
||||
mod absent_verdict;
|
||||
mod concurrent_delete;
|
||||
@@ -784,7 +785,10 @@ mod canonical_outcome {
|
||||
async fn untraversable_bucket_does_not_claim_complete_cluster_coverage() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["metadata unavailable"]))])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([(
|
||||
"bucket-a".to_string(),
|
||||
VecDeque::from([Error::other("metadata unavailable")]),
|
||||
)])),
|
||||
..Default::default()
|
||||
});
|
||||
let task = HealTask::from_request(
|
||||
@@ -1566,7 +1570,7 @@ struct MockStorage {
|
||||
include_object_dir_candidate: Mutex<bool>,
|
||||
listed_buckets: Mutex<Option<Vec<String>>>,
|
||||
bucket_discovery_incomplete: bool,
|
||||
bucket_heal_errors: Mutex<HashMap<String, VecDeque<&'static str>>>,
|
||||
bucket_heal_errors: Mutex<HashMap<String, VecDeque<Error>>>,
|
||||
bucket_heal_calls: Mutex<Vec<String>>,
|
||||
delete_recovery_results: Mutex<HashMap<String, Result<Option<HealResultItem>>>>,
|
||||
delete_recovery_calls: Mutex<Vec<String>>,
|
||||
@@ -1990,6 +1994,7 @@ fn replacement_identity(
|
||||
|
||||
#[derive(Clone)]
|
||||
enum MockHealObjectOutcome {
|
||||
PoolMetadata { kind: EcstorePoolMetadataFailure, outer: bool },
|
||||
LegacyAmbiguous { outer: bool },
|
||||
MissingObject { outer: bool },
|
||||
MissingVersion,
|
||||
@@ -2009,6 +2014,16 @@ enum MockHealObjectOutcome {
|
||||
PermanentOther(&'static str),
|
||||
}
|
||||
|
||||
fn pool_metadata_error(kind: EcstorePoolMetadataFailure) -> Error {
|
||||
Error::Storage(EcstoreError::other(EcstorePoolMetadataError {
|
||||
kind,
|
||||
operation: "bucket heal".to_owned(),
|
||||
phase: "native_heal",
|
||||
since: std::time::UNIX_EPOCH.into(),
|
||||
source: None,
|
||||
}))
|
||||
}
|
||||
|
||||
fn unavailable_drive_heal_result(state: DriveState) -> (HealResultItem, Option<Error>) {
|
||||
(
|
||||
HealResultItem {
|
||||
@@ -2180,6 +2195,14 @@ impl HealStorageAPI for MockStorage {
|
||||
.and_then(VecDeque::pop_front)
|
||||
{
|
||||
return match outcome {
|
||||
MockHealObjectOutcome::PoolMetadata { kind, outer } => {
|
||||
let error = pool_metadata_error(kind);
|
||||
if outer {
|
||||
Err(error)
|
||||
} else {
|
||||
Ok((HealResultItem::default(), Some(error)))
|
||||
}
|
||||
}
|
||||
MockHealObjectOutcome::LegacyAmbiguous { outer } => {
|
||||
let error = Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous);
|
||||
if outer {
|
||||
@@ -2256,6 +2279,14 @@ impl HealStorageAPI for MockStorage {
|
||||
}
|
||||
if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
|
||||
return match outcome {
|
||||
MockHealObjectOutcome::PoolMetadata { kind, outer } => {
|
||||
let error = pool_metadata_error(kind);
|
||||
if outer {
|
||||
Err(error)
|
||||
} else {
|
||||
Ok((HealResultItem::default(), Some(error)))
|
||||
}
|
||||
}
|
||||
MockHealObjectOutcome::LegacyAmbiguous { outer } => {
|
||||
let error = Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous);
|
||||
if outer {
|
||||
@@ -2387,14 +2418,14 @@ impl HealStorageAPI for MockStorage {
|
||||
async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
|
||||
self.bucket_heal_calls.lock().unwrap().push(bucket.to_string());
|
||||
self.bucket_heal_opts.lock().unwrap().push(*opts);
|
||||
if let Some(message) = self
|
||||
if let Some(error) = self
|
||||
.bucket_heal_errors
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get_mut(bucket)
|
||||
.and_then(VecDeque::pop_front)
|
||||
{
|
||||
return Err(Error::other(message));
|
||||
return Err(error);
|
||||
}
|
||||
Ok(HealResultItem::default())
|
||||
}
|
||||
@@ -3429,13 +3460,70 @@ async fn cluster_recreate_fails_when_pool_metadata_heal_fails() {
|
||||
.await
|
||||
.expect_err("cluster recreate heal must not hide pool metadata failure");
|
||||
|
||||
assert!(matches!(err, Error::TaskExecutionFailed { .. }));
|
||||
assert!(matches!(err, Error::Other(ref message) if message == "pool metadata missing"));
|
||||
assert_eq!(
|
||||
storage.heal_object_calls.lock().expect("object calls").as_slice(),
|
||||
["object-a".to_string(), "object-b".to_string(), POOL_META_NAME.to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cluster_pool_metadata_failures_preserve_recovery_classification_in_both_response_shapes() {
|
||||
for outer in [false, true] {
|
||||
for (kind, retryable) in [
|
||||
(EcstorePoolMetadataFailure::ReadUnavailable, true),
|
||||
(EcstorePoolMetadataFailure::TransactionUnknown, true),
|
||||
(EcstorePoolMetadataFailure::RecoveryRequired, false),
|
||||
(EcstorePoolMetadataFailure::FenceLost, false),
|
||||
] {
|
||||
let storage = Arc::new(MockStorage {
|
||||
listed_buckets: Mutex::new(Some(Vec::new())),
|
||||
..Default::default()
|
||||
});
|
||||
storage.heal_object_outcomes.lock().expect("object outcomes").insert(
|
||||
POOL_META_NAME.to_string(),
|
||||
VecDeque::from([MockHealObjectOutcome::PoolMetadata { kind, outer }]),
|
||||
);
|
||||
let mut request = HealRequest::new(
|
||||
HealType::Cluster,
|
||||
HealOptions {
|
||||
recreate_missing: true,
|
||||
timeout: None,
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
request.source = HealRequestSource::Admin;
|
||||
let task = HealTask::from_request(request, storage.clone());
|
||||
let error = task
|
||||
.execute()
|
||||
.await
|
||||
.expect_err("pool metadata failure must reach the scheduler");
|
||||
assert!(
|
||||
matches!(&error, Error::Storage(error) if error.pool_metadata_failure().is_some_and(|failure| failure.kind == kind)),
|
||||
"the original typed pool metadata state must survive either response shape: {error}"
|
||||
);
|
||||
assert_eq!(error.is_recoverable_heal(), retryable, "{kind:?}, outer={outer}");
|
||||
assert_eq!(storage.heal_object_calls.lock().expect("object calls").as_slice(), [POOL_META_NAME]);
|
||||
if retryable {
|
||||
let retry = task
|
||||
.retry_request_with_remaining_timeout()
|
||||
.await
|
||||
.expect("existing retry request");
|
||||
assert_eq!(retry.id, task.id, "recovery retains the accepted task token");
|
||||
HealTask::from_request(retry, storage.clone())
|
||||
.execute()
|
||||
.await
|
||||
.expect("the same task should complete after pool metadata recovery");
|
||||
assert_eq!(
|
||||
storage.heal_object_calls.lock().expect("object calls").as_slice(),
|
||||
[POOL_META_NAME, POOL_META_NAME]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cluster_dry_run_does_not_heal_pool_metadata() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
@@ -3799,7 +3887,10 @@ async fn test_recursive_bucket_heal_reports_typed_exhausted_and_permanent_failur
|
||||
async fn test_cluster_heal_continues_after_bucket_failure() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["metadata unavailable"]))])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([(
|
||||
"bucket-a".to_string(),
|
||||
VecDeque::from([Error::other("metadata unavailable")]),
|
||||
)])),
|
||||
..Default::default()
|
||||
});
|
||||
let request = HealRequest::new(
|
||||
@@ -3831,7 +3922,10 @@ async fn test_cluster_heal_continues_after_bucket_failure() {
|
||||
async fn test_cluster_heal_retries_only_recoverable_bucket() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["lock acquisition timeout"]))])),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([(
|
||||
"bucket-a".to_string(),
|
||||
VecDeque::from([Error::other("lock acquisition timeout")]),
|
||||
)])),
|
||||
..Default::default()
|
||||
});
|
||||
let request = HealRequest::new(
|
||||
@@ -3855,6 +3949,47 @@ async fn test_cluster_heal_retries_only_recoverable_bucket() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cluster_bucket_heal_retries_only_recoverable_pool_metadata_failures() {
|
||||
for (kind, retryable) in [
|
||||
(EcstorePoolMetadataFailure::ReadUnavailable, true),
|
||||
(EcstorePoolMetadataFailure::TransactionUnknown, true),
|
||||
(EcstorePoolMetadataFailure::RecoveryRequired, false),
|
||||
(EcstorePoolMetadataFailure::FenceLost, false),
|
||||
] {
|
||||
let storage = Arc::new(MockStorage {
|
||||
bucket_incarnation_id: Mutex::new(Some(Uuid::new_v4())),
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_owned(), VecDeque::from([pool_metadata_error(kind)]))])),
|
||||
..Default::default()
|
||||
});
|
||||
let mut request = HealRequest::new(
|
||||
HealType::Cluster,
|
||||
HealOptions {
|
||||
recursive: true,
|
||||
timeout: None,
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
request.source = HealRequestSource::Admin;
|
||||
let task = HealTask::from_request(request, storage.clone());
|
||||
let result = task.execute().await;
|
||||
if retryable {
|
||||
result.expect("a recoverable pool metadata state must not terminate the root heal");
|
||||
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
|
||||
assert_eq!(
|
||||
storage.bucket_heal_calls.lock().expect("bucket calls").as_slice(),
|
||||
["bucket-a", "bucket-a"]
|
||||
);
|
||||
} else {
|
||||
result.expect_err("operator recovery and lost fences must remain terminal");
|
||||
let failure = task.take_batch_failure().await.expect("permanent cluster failure");
|
||||
assert_eq!((failure.retryable, failure.permanent), (0, 1));
|
||||
assert_eq!(storage.bucket_heal_calls.lock().expect("bucket calls").as_slice(), ["bucket-a"]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_recursive_bucket_heal_does_not_remove_bucket_metadata() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
@@ -4617,7 +4752,7 @@ async fn erasure_set_bucket_prepass_failure_stops_before_object_heal() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
bucket_heal_errors: Mutex::new(HashMap::from([(
|
||||
"bucket-a".to_string(),
|
||||
VecDeque::from(["injected bucket prepass failure"]),
|
||||
VecDeque::from([Error::other("injected bucket prepass failure")]),
|
||||
)])),
|
||||
resume_disk: Mutex::new(Some(disk)),
|
||||
..Default::default()
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Durable restrictions owned by an authenticated user. A record is not an IAM
|
||||
//! grant or credential; issuing credentials still requires current IAM checks.
|
||||
use crate::{error::is_err_config_not_found, store::Store};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
const PREFIX: &str = "config/iam/plugin-delegations/v1";
|
||||
|
||||
#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Restrictions {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none", deserialize_with = "present_value")]
|
||||
pub version: Option<u32>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none", deserialize_with = "present_value")]
|
||||
pub output_bucket: Option<String>,
|
||||
pub deployment: String,
|
||||
pub bucket: String,
|
||||
pub input_prefix: String,
|
||||
pub output_prefix: String,
|
||||
pub expires_at_unix_seconds: u64,
|
||||
}
|
||||
|
||||
// An absent v1 field is compatible; an explicitly null v2 field is invalid.
|
||||
fn present_value<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>(deserializer: D) -> Result<Option<T>, D::Error> {
|
||||
T::deserialize(deserializer).map(Some)
|
||||
}
|
||||
|
||||
impl Restrictions {
|
||||
pub fn output_bucket(&self) -> Result<&str, DelegationError> {
|
||||
match (self.version, self.output_bucket.as_deref()) {
|
||||
(None, None) => Ok(&self.bucket),
|
||||
(Some(2), Some(output)) => Ok(output),
|
||||
_ => Err(DelegationError::Invalid),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn validate(&self, now: u64) -> Result<(), DelegationError> {
|
||||
let output_bucket = self.output_bucket()?;
|
||||
let names: [&str; 5] = [
|
||||
&self.deployment,
|
||||
&self.bucket,
|
||||
&self.input_prefix,
|
||||
&self.output_prefix,
|
||||
output_bucket,
|
||||
];
|
||||
if names
|
||||
.iter()
|
||||
.any(|s| s.is_empty() || s.len() > 1024 || s.contains("${") || s.chars().any(char::is_control))
|
||||
|| self.bucket.contains('/')
|
||||
|| self.bucket.contains('*')
|
||||
|| self.bucket.contains('?')
|
||||
|| output_bucket.contains('/')
|
||||
|| output_bucket.contains('*')
|
||||
|| output_bucket.contains('?')
|
||||
|| self.input_prefix.contains('*')
|
||||
|| self.input_prefix.contains('?')
|
||||
|| self.output_prefix.contains('*')
|
||||
|| self.output_prefix.contains('?')
|
||||
|| self.expires_at_unix_seconds <= now
|
||||
|| self.expires_at_unix_seconds.saturating_sub(now) > 86400
|
||||
{
|
||||
return Err(DelegationError::Invalid);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Delegation {
|
||||
pub version: u32,
|
||||
pub id: String,
|
||||
pub revision: String,
|
||||
pub owner_access_key: String,
|
||||
pub created_at_unix_seconds: u64,
|
||||
pub restrictions: Restrictions,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct Tombstone {
|
||||
version: u32,
|
||||
id: String,
|
||||
revision: String,
|
||||
owner_access_key: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum DelegationError {
|
||||
#[error("invalid delegation restriction")]
|
||||
Invalid,
|
||||
#[error("delegation not found")]
|
||||
NotFound,
|
||||
#[error("delegation owner rejected")]
|
||||
OwnerRejected,
|
||||
#[error("delegation unavailable")]
|
||||
Unavailable,
|
||||
#[error("delegation revoked")]
|
||||
Revoked,
|
||||
#[error("delegation expired")]
|
||||
Expired,
|
||||
}
|
||||
|
||||
fn record_path(id: &str) -> Result<String, DelegationError> {
|
||||
if id.len() != 32 || !id.bytes().all(|c| c.is_ascii_hexdigit()) {
|
||||
return Err(DelegationError::Invalid);
|
||||
}
|
||||
Ok(format!("{PREFIX}/{id}/record.json"))
|
||||
}
|
||||
|
||||
async fn load<T: Store>(store: &T, id: &str) -> Result<Delegation, DelegationError> {
|
||||
let record: Delegation = store.load_iam_config_strict(record_path(id)?).await.map_err(|error| {
|
||||
if is_err_config_not_found(&error) {
|
||||
DelegationError::NotFound
|
||||
} else {
|
||||
DelegationError::Unavailable
|
||||
}
|
||||
})?;
|
||||
if !matches!(record.version, 1 | 2)
|
||||
|| record.version != record.restrictions.version.unwrap_or(1)
|
||||
|| record.id != id
|
||||
|| record.revision != id
|
||||
|| record.owner_access_key.is_empty()
|
||||
{
|
||||
return Err(DelegationError::Unavailable);
|
||||
}
|
||||
record
|
||||
.restrictions
|
||||
.validate(record.created_at_unix_seconds)
|
||||
.map_err(|_| DelegationError::Unavailable)?;
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
pub(crate) async fn create<T: Store>(
|
||||
store: &T,
|
||||
owner: &str,
|
||||
restrictions: Restrictions,
|
||||
now: u64,
|
||||
) -> Result<Delegation, DelegationError> {
|
||||
restrictions.validate(now)?;
|
||||
if owner.is_empty() {
|
||||
return Err(DelegationError::OwnerRejected);
|
||||
}
|
||||
// IDs are generated exclusively here; no update operation accepts an ID.
|
||||
let id = format!("{:032x}", rand::random::<u128>());
|
||||
let path = record_path(&id)?;
|
||||
match store.load_iam_config_strict::<Delegation>(&path).await {
|
||||
Err(error) if is_err_config_not_found(&error) => {}
|
||||
_ => return Err(DelegationError::Unavailable),
|
||||
}
|
||||
let record = Delegation {
|
||||
version: restrictions.version.unwrap_or(1),
|
||||
revision: id.clone(),
|
||||
id,
|
||||
owner_access_key: owner.to_owned(),
|
||||
created_at_unix_seconds: now,
|
||||
restrictions,
|
||||
};
|
||||
store
|
||||
.save_iam_config(&record, path)
|
||||
.await
|
||||
.map_err(|_| DelegationError::Unavailable)?;
|
||||
if load(store, &record.id).await? != record {
|
||||
return Err(DelegationError::Unavailable);
|
||||
}
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
pub(crate) async fn revoke<T: Store>(store: &T, owner: &str, id: &str) -> Result<(), DelegationError> {
|
||||
let record = load(store, id).await?;
|
||||
if record.owner_access_key != owner {
|
||||
return Err(DelegationError::OwnerRejected);
|
||||
}
|
||||
let tombstone = Tombstone {
|
||||
version: 1,
|
||||
id: record.id,
|
||||
revision: record.revision,
|
||||
owner_access_key: record.owner_access_key,
|
||||
};
|
||||
let path = format!("{PREFIX}/{id}/revoked.json");
|
||||
// Every retry writes exactly the same revocation; active records never change.
|
||||
store
|
||||
.save_iam_config(&tombstone, &path)
|
||||
.await
|
||||
.map_err(|_| DelegationError::Unavailable)?;
|
||||
let saved: Tombstone = store
|
||||
.load_iam_config_strict(path)
|
||||
.await
|
||||
.map_err(|_| DelegationError::Unavailable)?;
|
||||
if saved != tombstone {
|
||||
return Err(DelegationError::Unavailable);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn read_active<T: Store>(store: &T, owner: &str, id: &str, now: u64) -> Result<Delegation, DelegationError> {
|
||||
let record = load(store, id).await?;
|
||||
if record.owner_access_key != owner {
|
||||
return Err(DelegationError::OwnerRejected);
|
||||
}
|
||||
check_active(store, record, now).await
|
||||
}
|
||||
|
||||
pub(crate) async fn read_owned<T: Store>(store: &T, owner: &str, id: &str) -> Result<Delegation, DelegationError> {
|
||||
let record = load(store, id).await?;
|
||||
if record.owner_access_key != owner {
|
||||
return Err(DelegationError::OwnerRejected);
|
||||
}
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
pub(crate) async fn read_active_for_broker<T: Store>(store: &T, id: &str, now: u64) -> Result<Delegation, DelegationError> {
|
||||
let record = load(store, id).await?;
|
||||
check_active(store, record, now).await
|
||||
}
|
||||
async fn check_active<T: Store>(store: &T, record: Delegation, now: u64) -> Result<Delegation, DelegationError> {
|
||||
let id = &record.id;
|
||||
match store
|
||||
.load_iam_config_strict::<Tombstone>(format!("{PREFIX}/{id}/revoked.json"))
|
||||
.await
|
||||
{
|
||||
Ok(_) => return Err(DelegationError::Revoked),
|
||||
Err(error) if is_err_config_not_found(&error) => {}
|
||||
Err(_) => return Err(DelegationError::Unavailable),
|
||||
}
|
||||
if now >= record.restrictions.expires_at_unix_seconds {
|
||||
return Err(DelegationError::Expired);
|
||||
}
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn delegation_v1_json_remains_compatible() {
|
||||
let legacy = serde_json::json!({"deployment":"runner-a","bucket":"photos",
|
||||
"input_prefix":"inputs/","output_prefix":"outputs/","expires_at_unix_seconds":200});
|
||||
let restriction: Restrictions = serde_json::from_value(legacy.clone()).expect("legacy restriction");
|
||||
restriction.validate(100).expect("legacy validation");
|
||||
assert_eq!(restriction.output_bucket().expect("v1 output"), "photos");
|
||||
assert_eq!(serde_json::to_value(restriction).expect("legacy serialization"), legacy);
|
||||
let record = serde_json::json!({"version":1,"id":"a".repeat(32),"revision":"a".repeat(32),
|
||||
"owner_access_key":"owner","created_at_unix_seconds":100,"restrictions":legacy});
|
||||
let decoded: Delegation = serde_json::from_value(record.clone()).expect("legacy record");
|
||||
assert_eq!(serde_json::to_value(decoded).expect("legacy record serialization"), record);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delegation_v2_requires_explicit_output_bucket() {
|
||||
let valid = serde_json::json!({"version":2,"deployment":"runner-a","bucket":"photos",
|
||||
"input_prefix":"inputs/","output_bucket":"artifacts","output_prefix":"outputs/",
|
||||
"expires_at_unix_seconds":200});
|
||||
let restriction: Restrictions = serde_json::from_value(valid.clone()).expect("v2 restriction");
|
||||
restriction.validate(100).expect("explicit output bucket");
|
||||
assert_eq!(restriction.output_bucket().expect("v2 output"), "artifacts");
|
||||
assert_eq!(serde_json::to_value(restriction).expect("v2 serialization"), valid);
|
||||
for key in ["version", "output_bucket"] {
|
||||
let mut bad = valid.clone();
|
||||
bad.as_object_mut().expect("object").remove(key);
|
||||
assert!(
|
||||
serde_json::from_value::<Restrictions>(bad).map_or(true, |r| r.validate(100).is_err()),
|
||||
"v2 required {key}"
|
||||
);
|
||||
}
|
||||
for (key, value) in [
|
||||
("version", serde_json::json!(1)),
|
||||
("version", serde_json::json!(3)),
|
||||
("version", serde_json::Value::Null),
|
||||
("output_bucket", serde_json::Value::Null),
|
||||
("output_bucket", serde_json::json!("")),
|
||||
("output_bucket", serde_json::json!("other/*")),
|
||||
("output_bucket", serde_json::json!("${aws:username}")),
|
||||
("output_bucket", serde_json::json!("other?")),
|
||||
("output_bucket", serde_json::json!("other\n")),
|
||||
("unknown", serde_json::json!(true)),
|
||||
] {
|
||||
let mut bad = valid.clone();
|
||||
bad[key] = value;
|
||||
assert!(
|
||||
serde_json::from_value::<Restrictions>(bad).map_or(true, |r| r.validate(100).is_err()),
|
||||
"invalid v2 {key}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restrictions_are_literal_bounded_and_expiring() {
|
||||
let mut restriction = Restrictions {
|
||||
version: None,
|
||||
output_bucket: None,
|
||||
deployment: "runner-a".into(),
|
||||
bucket: "photos".into(),
|
||||
input_prefix: "inputs/".into(),
|
||||
output_prefix: "outputs/".into(),
|
||||
expires_at_unix_seconds: 200,
|
||||
};
|
||||
assert!(restriction.validate(100).is_ok());
|
||||
restriction.output_prefix = "outputs/*".into();
|
||||
assert!(matches!(restriction.validate(100), Err(DelegationError::Invalid)));
|
||||
restriction.output_prefix = "outputs/".into();
|
||||
assert!(matches!(restriction.validate(200), Err(DelegationError::Invalid)));
|
||||
restriction.expires_at_unix_seconds = 200;
|
||||
restriction.input_prefix = "inputs/${aws:username}/".into();
|
||||
assert!(
|
||||
matches!(restriction.validate(100), Err(DelegationError::Invalid)),
|
||||
"C07_LITERAL_SCOPE_BOUND: policy variables are not literal restrictions"
|
||||
);
|
||||
assert!(record_path("../../users").is_err());
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ pub(crate) const EVENT_OIDC_DIAGNOSTICS: &str = "oidc_diagnostics";
|
||||
const EVENT_OIDC_STATE: &str = "oidc_state";
|
||||
|
||||
pub mod cache;
|
||||
pub mod delegation;
|
||||
pub mod error;
|
||||
pub mod federation;
|
||||
pub mod keyring;
|
||||
|
||||
@@ -91,6 +91,7 @@ pub(crate) mod crate_boundary {
|
||||
|
||||
pub(crate) mod object_store {
|
||||
pub(crate) use super::storage_contracts::{HTTPPreconditions, ListOperations, ObjectInfoOrErr, ObjectOperations};
|
||||
pub(crate) use rustfs_ecstore::api::config::com::read_config_preserve_empty_with_metadata_opts;
|
||||
}
|
||||
|
||||
pub(crate) mod runtime {
|
||||
|
||||
@@ -26,6 +26,11 @@ pub trait Store: Clone + Send + Sync + 'static {
|
||||
fn has_watcher(&self) -> bool;
|
||||
async fn save_iam_config<Item: Serialize + Send>(&self, item: Item, path: impl AsRef<str> + Send) -> Result<()>;
|
||||
async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item>;
|
||||
/// Only confirmed storage absence may return ConfigNotFound. Corrupt or
|
||||
/// undecryptable objects must remain errors, especially for revocations.
|
||||
async fn load_iam_config_strict<Item: DeserializeOwned>(&self, _path: impl AsRef<str> + Send) -> Result<Item> {
|
||||
Err(crate::error::Error::IamSysNotInitialized)
|
||||
}
|
||||
async fn delete_iam_config(&self, path: impl AsRef<str> + Send) -> Result<()>;
|
||||
|
||||
async fn save_user_identity(&self, name: &str, user_type: UserType, item: UserIdentity, ttl: Option<usize>) -> Result<()>;
|
||||
|
||||
@@ -42,6 +42,7 @@ use tracing::{debug, error, warn};
|
||||
|
||||
use crate::storage_api::object_store::{
|
||||
HTTPPreconditions, ListOperations, ObjectInfoOrErr as StorageObjectInfoOrErr, ObjectOperations,
|
||||
read_config_preserve_empty_with_metadata_opts,
|
||||
};
|
||||
|
||||
pub static IAM_CONFIG_PREFIX: LazyLock<String> = LazyLock::new(|| format!("{IAM_CONFIG_ROOT_PREFIX}/iam"));
|
||||
@@ -611,6 +612,11 @@ impl ObjectStore {
|
||||
Ok(serde_json::from_slice(&outcome.plain)?)
|
||||
}
|
||||
|
||||
fn decode_iam_config_strict<Item: DeserializeOwned>(data: &[u8]) -> Result<Item> {
|
||||
let outcome = Self::decrypt_data_with_source(data)?;
|
||||
Ok(serde_json::from_slice(&outcome.plain)?)
|
||||
}
|
||||
|
||||
/// Parameterized core of [`Store::load_user_identity`].
|
||||
async fn load_user_identity_with(&self, name: &str, user_type: UserType, mode: LoadMode) -> Result<UserIdentity> {
|
||||
let mut u: UserIdentity = self
|
||||
@@ -824,6 +830,12 @@ impl Store for ObjectStore {
|
||||
async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> {
|
||||
self.load_iam_config_with(path, LoadMode::Locked).await
|
||||
}
|
||||
async fn load_iam_config_strict<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> {
|
||||
let (data, _) =
|
||||
read_config_preserve_empty_with_metadata_opts(self.object_api.clone(), path.as_ref(), &LoadMode::Locked.read_opts())
|
||||
.await?;
|
||||
Self::decode_iam_config_strict(&data)
|
||||
}
|
||||
/// Saves IAM configuration with a retry mechanism on failure.
|
||||
///
|
||||
/// Attempts to save the IAM configuration up to 5 times if the storage layer is not ready,
|
||||
@@ -1458,6 +1470,40 @@ mod tests {
|
||||
assert!(result.is_err(), "corrupt stream_io data should fail decrypt");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn strict_revocation_load_keeps_real_crypto_failure_distinct_from_absence() {
|
||||
let cred = test_cred();
|
||||
let key = format!("{}:{}", cred.access_key, cred.secret_key);
|
||||
let plain = br#"{"version":1,"id":"revoked"}"#;
|
||||
let mut encrypted = rustfs_crypto::encrypt_stream_io(key.as_bytes(), plain).expect("real stored ciphertext");
|
||||
// Exercise the exact decoder used after the actual object read.
|
||||
assert!(ObjectStore::decode_iam_config_strict::<serde_json::Value>(&encrypted).is_ok());
|
||||
encrypted[50] ^= 0xFF;
|
||||
let error = ObjectStore::decode_iam_config_strict::<serde_json::Value>(&encrypted)
|
||||
.expect_err("corrupt tombstone must not become missing");
|
||||
assert!(
|
||||
matches!(error, Error::CryptoError(_)),
|
||||
"C07_REVOCATION_FAIL_CLOSED: preserve actual crypto error"
|
||||
);
|
||||
assert!(
|
||||
!crate::error::is_err_config_not_found(&error),
|
||||
"C07_REVOCATION_FAIL_CLOSED: corrupted revocation is not absence"
|
||||
);
|
||||
// The preserve-empty reader keeps an existing zero-byte object as
|
||||
// Ok(empty); ecstore's read_config_preserve_empty_distinguishes_empty_object
|
||||
// covers that storage boundary. Its payload must fail here, not become
|
||||
// ConfigNotFound and reactivate a revoked delegation.
|
||||
let empty =
|
||||
ObjectStore::decode_iam_config_strict::<serde_json::Value>(&[]).expect_err("an existing empty tombstone is corrupt");
|
||||
assert!(
|
||||
!crate::error::is_err_config_not_found(&empty),
|
||||
"C07_REVOCATION_FAIL_CLOSED: empty existing revocation is not absence"
|
||||
);
|
||||
let invalid_json =
|
||||
ObjectStore::decode_iam_config_strict::<serde_json::Value>(br#"{"version": }"#).expect_err("invalid stored JSON");
|
||||
assert!(!crate::error::is_err_config_not_found(&invalid_json));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_decrypt_data_short_data_fails() {
|
||||
let short = &[0x00u8; 40]; // less than 41-byte stream_io header, not valid JSON
|
||||
|
||||
+257
-7
@@ -75,6 +75,7 @@ pub const STATUS_ENABLED: &str = "enabled";
|
||||
pub const STATUS_DISABLED: &str = "disabled";
|
||||
|
||||
pub const POLICYNAME: &str = "policy";
|
||||
pub const PLUGIN_ATTEMPT_CREDENTIAL_CLAIM: &str = "rustfs:plugin-attempt-v1";
|
||||
pub const SESSION_POLICY_NAME: &str = "sessionPolicy";
|
||||
pub const SESSION_POLICY_NAME_EXTRACTED: &str = "sessionPolicy-extracted";
|
||||
const VERIFIED_FEDERATED_POLICY_CLAIM: &str = "x-rustfs-internal-federated-policy";
|
||||
@@ -168,6 +169,12 @@ enum PreparedServicePolicyMode {
|
||||
|
||||
#[derive(Clone)]
|
||||
enum PreparedIamMode {
|
||||
TerminalDeny,
|
||||
PluginAttemptSts {
|
||||
is_owner: bool,
|
||||
combined_policy: Policy,
|
||||
session_policy: PreparedSessionPolicy,
|
||||
},
|
||||
Opa,
|
||||
Owner,
|
||||
Deny,
|
||||
@@ -202,7 +209,7 @@ impl PreparedIamAuth {
|
||||
pub async fn needs_existing_object_tag_for_args(&self, args: &Args<'_>) -> bool {
|
||||
match &self.mode {
|
||||
PreparedIamMode::Opa => true,
|
||||
PreparedIamMode::Owner | PreparedIamMode::Deny => false,
|
||||
PreparedIamMode::Owner | PreparedIamMode::Deny | PreparedIamMode::TerminalDeny => false,
|
||||
PreparedIamMode::Regular { combined_policy } => {
|
||||
policy_needs_existing_object_tag_for_args(combined_policy, args).await
|
||||
}
|
||||
@@ -210,6 +217,11 @@ impl PreparedIamAuth {
|
||||
combined_policy,
|
||||
session_policy,
|
||||
..
|
||||
}
|
||||
| PreparedIamMode::PluginAttemptSts {
|
||||
combined_policy,
|
||||
session_policy,
|
||||
..
|
||||
} => {
|
||||
policy_needs_existing_object_tag_for_args(combined_policy, args).await
|
||||
|| prepared_session_policy_needs_existing_object_tag_for_args(session_policy, args).await
|
||||
@@ -239,14 +251,56 @@ impl PreparedIamAuth {
|
||||
pub fn combined_policy_for_view(&self) -> Option<&Policy> {
|
||||
match &self.mode {
|
||||
PreparedIamMode::Regular { combined_policy } => Some(combined_policy),
|
||||
PreparedIamMode::Sts { combined_policy, .. } => Some(combined_policy),
|
||||
PreparedIamMode::Sts { combined_policy, .. } | PreparedIamMode::PluginAttemptSts { combined_policy, .. } => {
|
||||
Some(combined_policy)
|
||||
}
|
||||
PreparedIamMode::ServiceAccount { combined_policy, .. } => Some(combined_policy),
|
||||
PreparedIamMode::Opa | PreparedIamMode::Owner | PreparedIamMode::Deny => None,
|
||||
PreparedIamMode::Opa | PreparedIamMode::Owner | PreparedIamMode::Deny | PreparedIamMode::TerminalDeny => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<T: Store> IamSys<T> {
|
||||
/// For the authenticated, explicitly configured broker handler only.
|
||||
pub async fn read_active_plugin_delegation_for_broker(
|
||||
&self,
|
||||
id: &str,
|
||||
now: u64,
|
||||
) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
|
||||
crate::delegation::read_active_for_broker(&self.store.api, id, now).await
|
||||
}
|
||||
pub async fn create_plugin_delegation(
|
||||
&self,
|
||||
owner: &str,
|
||||
restrictions: crate::delegation::Restrictions,
|
||||
now: u64,
|
||||
) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
|
||||
crate::delegation::create(&self.store.api, owner, restrictions, now).await
|
||||
}
|
||||
pub async fn revoke_plugin_delegation(
|
||||
&self,
|
||||
owner: &str,
|
||||
id: &str,
|
||||
) -> std::result::Result<(), crate::delegation::DelegationError> {
|
||||
crate::delegation::revoke(&self.store.api, owner, id).await
|
||||
}
|
||||
pub async fn read_active_plugin_delegation(
|
||||
&self,
|
||||
owner: &str,
|
||||
id: &str,
|
||||
now: u64,
|
||||
) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
|
||||
crate::delegation::read_active(&self.store.api, owner, id, now).await
|
||||
}
|
||||
/// Reads an owned delegation record for historical lookups, including expired or revoked records.
|
||||
pub async fn read_owned_plugin_delegation(
|
||||
&self,
|
||||
owner: &str,
|
||||
id: &str,
|
||||
) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
|
||||
crate::delegation::read_owned(&self.store.api, owner, id).await
|
||||
}
|
||||
|
||||
/// Create a new IamSys instance with the given IamCache store
|
||||
///
|
||||
/// # Arguments
|
||||
@@ -281,6 +335,16 @@ impl<T: Store> IamSys<T> {
|
||||
*guard = PolicyPluginState::Ready(client);
|
||||
}
|
||||
|
||||
/// Restricted credentials require the built-in parent/session intersection.
|
||||
/// External authorization bypasses that path, so only Disabled is supported.
|
||||
pub async fn ensure_restricted_credential_vending() -> Result<()> {
|
||||
if matches!(Self::policy_plugin_state().await, PolicyPluginState::Disabled) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(Error::IAMActionNotAllowed)
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn get_policy_plugin_client() -> Option<rustfs_policy::policy::opa::AuthZPlugin> {
|
||||
let policy_plugin_state = get_policy_plugin_state();
|
||||
let guard = policy_plugin_state.read().await;
|
||||
@@ -1383,17 +1447,50 @@ impl<T: Store> IamSys<T> {
|
||||
/// 1) know whether policy evaluation may need `s3:ExistingObjectTag`, and
|
||||
/// 2) evaluate with final conditions without re-merging identity policies.
|
||||
pub async fn prepare_auth(&self, args: &Args<'_>) -> PreparedIamAuth {
|
||||
if args.is_owner {
|
||||
let marked = args.claims.contains_key(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM);
|
||||
if args.is_owner && !marked {
|
||||
return PreparedIamAuth {
|
||||
needs_existing_object_tag: false,
|
||||
mode: PreparedIamMode::Owner,
|
||||
};
|
||||
}
|
||||
|
||||
if let Some(prepared) = Self::policy_plugin_state().await.prepared_iam_auth() {
|
||||
// One observed state decides this request's path. Marked credentials
|
||||
// never enter OPA, including after a restart or on another node.
|
||||
let state = Self::policy_plugin_state().await;
|
||||
if marked {
|
||||
if args.is_owner
|
||||
|| args.claims.get(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM) != Some(&Value::Bool(true))
|
||||
|| !matches!(state, PolicyPluginState::Disabled)
|
||||
{
|
||||
return PreparedIamAuth {
|
||||
needs_existing_object_tag: false,
|
||||
mode: PreparedIamMode::TerminalDeny,
|
||||
};
|
||||
}
|
||||
let prepared = self.prepare_builtin_auth(args).await;
|
||||
let mode = match prepared.mode {
|
||||
PreparedIamMode::Sts {
|
||||
is_owner,
|
||||
combined_policy,
|
||||
session_policy,
|
||||
} => PreparedIamMode::PluginAttemptSts {
|
||||
is_owner,
|
||||
combined_policy,
|
||||
session_policy,
|
||||
},
|
||||
_ => PreparedIamMode::TerminalDeny,
|
||||
};
|
||||
return PreparedIamAuth {
|
||||
needs_existing_object_tag: prepared.needs_existing_object_tag,
|
||||
mode,
|
||||
};
|
||||
}
|
||||
if let Some(prepared) = state.prepared_iam_auth() {
|
||||
return prepared;
|
||||
}
|
||||
|
||||
self.prepare_builtin_auth(args).await
|
||||
}
|
||||
async fn prepare_builtin_auth(&self, args: &Args<'_>) -> PreparedIamAuth {
|
||||
let Ok((is_svc, parent_user)) = self.is_service_account(args.account).await else {
|
||||
return PreparedIamAuth {
|
||||
needs_existing_object_tag: false,
|
||||
@@ -1425,6 +1522,20 @@ impl<T: Store> IamSys<T> {
|
||||
/// bucket policies on an IAM denial. An error must not become an implicit deny.
|
||||
pub async fn try_eval_prepared(&self, prepared: &PreparedIamAuth, args: &Args<'_>) -> Result<bool> {
|
||||
Ok(match &prepared.mode {
|
||||
PreparedIamMode::TerminalDeny => return Err(Error::IAMActionNotAllowed),
|
||||
PreparedIamMode::PluginAttemptSts {
|
||||
is_owner,
|
||||
combined_policy,
|
||||
session_policy,
|
||||
} => {
|
||||
// A false IAM result permits bucket-policy fallback. Restricted
|
||||
// plugin credentials must instead terminate authorization.
|
||||
let session_allowed = evaluate_prepared_session_policy(session_policy, args).await == Some(true);
|
||||
if !session_allowed || !(*is_owner || combined_policy.is_allowed(args).await) {
|
||||
return Err(Error::IAMActionNotAllowed);
|
||||
}
|
||||
true
|
||||
}
|
||||
PreparedIamMode::Opa => {
|
||||
let Some(opa_enable) = Self::get_policy_plugin_client().await else {
|
||||
tracing::warn!("eval_prepared: OPA mode requested but plugin is unavailable");
|
||||
@@ -2127,6 +2238,145 @@ mod tests {
|
||||
assert!(prepared.combined_policy_for_view().is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn marked_signed_sts_denial_is_terminal_before_public_bucket_fallback() {
|
||||
let iam = IamSys::new(IamCache::new(StsTestMockStore::new(false)).await.unwrap());
|
||||
let previous = IamSys::<StsTestMockStore>::policy_plugin_state().await;
|
||||
let signing_key = "c07-test-token-signing-key";
|
||||
let mut metadata=HashMap::from([
|
||||
("parent".into(),Value::String("sts-fallback-test-parent".into())),
|
||||
("exp".into(),serde_json::json!((OffsetDateTime::now_utc()+time::Duration::hours(1)).unix_timestamp())),
|
||||
(SESSION_POLICY_NAME.into(),Value::String(base64_simd::URL_SAFE_NO_PAD.encode_to_string(br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::mybucket/inside"]}]}"#))),
|
||||
]);
|
||||
metadata.insert(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM.into(), Value::Bool(true));
|
||||
let mut marked = get_new_credentials_with_metadata(&metadata, signing_key).unwrap();
|
||||
marked.parent_user = "sts-fallback-test-parent".into();
|
||||
let marked_claims: HashMap<String, Value> = rustfs_policy::utils::extract_claims(&marked.session_token, signing_key)
|
||||
.expect("signed test credentials must decode")
|
||||
.claims;
|
||||
let marked_user = UserIdentity::from(marked.clone());
|
||||
iam.store
|
||||
.cache
|
||||
.add_or_update_sts_account(&marked.access_key, &marked_user, OffsetDateTime::now_utc());
|
||||
metadata.remove(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM);
|
||||
let mut ordinary = get_new_credentials_with_metadata(&metadata, signing_key).unwrap();
|
||||
ordinary.parent_user = marked.parent_user.clone();
|
||||
let ordinary_claims: HashMap<String, Value> = rustfs_policy::utils::extract_claims(&ordinary.session_token, signing_key)
|
||||
.expect("signed test credentials must decode")
|
||||
.claims;
|
||||
iam.store.cache.add_or_update_sts_account(
|
||||
&ordinary.access_key,
|
||||
&UserIdentity::from(ordinary.clone()),
|
||||
OffsetDateTime::now_utc(),
|
||||
);
|
||||
let groups = None;
|
||||
let conditions = HashMap::new();
|
||||
let args = Args {
|
||||
account: &marked.access_key,
|
||||
groups: &groups,
|
||||
action: Action::S3Action(S3Action::GetObjectAction),
|
||||
bucket: "mybucket",
|
||||
object: "outside",
|
||||
conditions: &conditions,
|
||||
is_owner: false,
|
||||
claims: &marked_claims,
|
||||
deny_only: false,
|
||||
};
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Disabled;
|
||||
let denied = iam.prepare_auth(&args).await;
|
||||
let disabled_terminal = matches!(iam.try_eval_prepared(&denied, &args).await, Err(Error::IAMActionNotAllowed));
|
||||
let inside = Args {
|
||||
object: "inside",
|
||||
..args.clone()
|
||||
};
|
||||
let allowed = iam.prepare_auth(&inside).await;
|
||||
let inside_allowed = matches!(iam.try_eval_prepared(&allowed, &inside).await, Ok(true));
|
||||
let unmarked_args = Args {
|
||||
account: &ordinary.access_key,
|
||||
claims: &ordinary_claims,
|
||||
..args.clone()
|
||||
};
|
||||
let unmarked = iam.prepare_auth(&unmarked_args).await;
|
||||
let ordinary_false = matches!(iam.try_eval_prepared(&unmarked, &unmarked_args).await, Ok(false));
|
||||
let public:rustfs_policy::policy::BucketPolicy=serde_json::from_value(serde_json::json!({"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["*"]},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::mybucket/*"]}]})).unwrap();
|
||||
let public_allows = public
|
||||
.is_allowed(&rustfs_policy::policy::BucketPolicyArgs {
|
||||
account: &marked.access_key,
|
||||
groups: &groups,
|
||||
action: args.action,
|
||||
bucket: args.bucket,
|
||||
object: args.object,
|
||||
conditions: &conditions,
|
||||
is_owner: false,
|
||||
})
|
||||
.await;
|
||||
IamSys::<StsTestMockStore>::set_policy_plugin_client(opa::AuthZPlugin::new(opa::Args {
|
||||
url: "http://127.0.0.1:8181/allow".into(),
|
||||
auth_token: String::new(),
|
||||
}))
|
||||
.await;
|
||||
let opa_marked = iam.prepare_auth(&args).await;
|
||||
let ready_terminal = matches!(iam.try_eval_prepared(&opa_marked, &args).await, Err(Error::IAMActionNotAllowed));
|
||||
let opa_ordinary = iam.prepare_auth(&unmarked_args).await;
|
||||
let ordinary_keeps_opa = matches!(opa_ordinary.mode, PreparedIamMode::Opa);
|
||||
// The captured Disabled preparation stays on built-in intersection even
|
||||
// if an external policy client becomes ready before evaluation.
|
||||
let captured_still_terminal = matches!(iam.try_eval_prepared(&denied, &args).await, Err(Error::IAMActionNotAllowed));
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Initializing;
|
||||
let pending = iam.prepare_auth(&args).await;
|
||||
let initializing_terminal = matches!(iam.try_eval_prepared(&pending, &args).await, Err(Error::IAMActionNotAllowed));
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Failed;
|
||||
let failed = iam.prepare_auth(&args).await;
|
||||
let failed_terminal = matches!(iam.try_eval_prepared(&failed, &args).await, Err(Error::IAMActionNotAllowed));
|
||||
*get_policy_plugin_state().write().await = previous;
|
||||
assert!(public_allows, "C07_PUBLIC_BUCKET_CONTROL: real public policy permits outside object");
|
||||
assert!(
|
||||
disabled_terminal && captured_still_terminal,
|
||||
"C07_SCOPE_TERMINAL_BOUND: bucket policy must not rescue restricted scope denial"
|
||||
);
|
||||
assert!(
|
||||
ready_terminal && initializing_terminal && failed_terminal,
|
||||
"C07_EXTERNAL_POLICY_BOUND: marked persisted STS cannot enter OPA"
|
||||
);
|
||||
assert!(
|
||||
inside_allowed && ordinary_false && ordinary_keeps_opa,
|
||||
"C07_UNMARKED_COMPAT: preserve ordinary STS and valid marked scope"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn restricted_vending_rejects_actual_external_policy_plugin_state() {
|
||||
let previous = IamSys::<StsTestMockStore>::policy_plugin_state().await;
|
||||
IamSys::<StsTestMockStore>::set_policy_plugin_client(opa::AuthZPlugin::new(opa::Args {
|
||||
url: "http://127.0.0.1:8181/v1/data/rustfs/authz/allow".into(),
|
||||
auth_token: String::new(),
|
||||
}))
|
||||
.await;
|
||||
let active_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
|
||||
.await
|
||||
.is_err();
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Initializing;
|
||||
let initializing_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
|
||||
.await
|
||||
.is_err();
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Failed;
|
||||
let failed_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
|
||||
.await
|
||||
.is_err();
|
||||
*get_policy_plugin_state().write().await = PolicyPluginState::Disabled;
|
||||
let disabled_allowed = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
|
||||
.await
|
||||
.is_ok();
|
||||
*get_policy_plugin_state().write().await = previous;
|
||||
assert!(
|
||||
active_rejected && initializing_rejected && failed_rejected,
|
||||
"C07_EXTERNAL_POLICY_BOUND: external authorization cannot bypass minted session scope"
|
||||
);
|
||||
assert!(disabled_allowed);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_prepare_auth_requests_existing_object_tags_in_opa_mode() {
|
||||
|
||||
@@ -1,10 +1,67 @@
|
||||
use criterion::{Criterion, criterion_group, criterion_main};
|
||||
use metrics_util::debugging::DebuggingRecorder;
|
||||
use rustfs_io_metrics::{
|
||||
LocalBatchReadDecision, LocalBatchReadMetrics, LocalBatchReadStage, record_local_batch_read_decision,
|
||||
set_get_stage_metrics_enabled,
|
||||
};
|
||||
use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics, record_get_object_timeout_operation_start};
|
||||
use rustfs_io_metrics::{record_s3_op, s3_http_metrics::S3HttpRequestGuard};
|
||||
use rustfs_s3_ops::S3Operation;
|
||||
use std::hint::black_box;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
fn local_batch_attribution_sample() {
|
||||
let prepare = rustfs_io_metrics::get_stage_metrics_enabled().then(Instant::now);
|
||||
record_local_batch_read_decision(LocalBatchReadDecision::Selected, prepare.map(|started| started.elapsed()));
|
||||
if let Some(metrics) = LocalBatchReadMetrics::capture() {
|
||||
metrics.record_batch(4);
|
||||
let execution = Instant::now();
|
||||
metrics.record_stage(LocalBatchReadStage::QueueWait, true, execution.elapsed());
|
||||
for _ in 0..4 {
|
||||
for stage in [
|
||||
LocalBatchReadStage::Metadata,
|
||||
LocalBatchReadStage::Open,
|
||||
LocalBatchReadStage::Read,
|
||||
] {
|
||||
let started = Instant::now();
|
||||
metrics.record_stage(stage, true, started.elapsed());
|
||||
}
|
||||
metrics.record_shard(true, 32 * 1024, 32 * 1024);
|
||||
}
|
||||
metrics.record_stage(LocalBatchReadStage::Execution, true, execution.elapsed());
|
||||
}
|
||||
}
|
||||
|
||||
fn bench_local_batch_attribution(c: &mut Criterion) {
|
||||
let mut group = c.benchmark_group("local_batch_attribution_4_shards_debug_recorder");
|
||||
for (name, enabled) in [("disabled", false), ("enabled", true)] {
|
||||
set_get_stage_metrics_enabled(enabled);
|
||||
group.bench_function(name, |b| {
|
||||
b.iter_custom(|iterations| {
|
||||
let mut remaining = iterations;
|
||||
let mut elapsed = Duration::ZERO;
|
||||
while remaining > 0 {
|
||||
// Bound captured histogram storage independently of Criterion's
|
||||
// iteration count. Recorder setup/teardown is not measured.
|
||||
let chunk = remaining.min(128);
|
||||
let recorder = DebuggingRecorder::new();
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
let started = Instant::now();
|
||||
for _ in 0..chunk {
|
||||
local_batch_attribution_sample();
|
||||
}
|
||||
elapsed += started.elapsed();
|
||||
});
|
||||
remaining -= chunk;
|
||||
}
|
||||
elapsed
|
||||
});
|
||||
});
|
||||
}
|
||||
group.finish();
|
||||
set_get_stage_metrics_enabled(false);
|
||||
}
|
||||
|
||||
fn bench_record_get_object_timeout_operation_start(c: &mut Criterion) {
|
||||
c.bench_function("record_get_object_timeout_operation_start", |b| {
|
||||
@@ -54,6 +111,7 @@ criterion_group!(
|
||||
bench_record_get_object_timeout_operation_start,
|
||||
bench_s3_http_outcomes,
|
||||
bench_update_concurrent_requests,
|
||||
bench_metrics_collector_record_io_operation
|
||||
bench_metrics_collector_record_io_operation,
|
||||
bench_local_batch_attribution
|
||||
);
|
||||
criterion_main!(benches);
|
||||
|
||||
@@ -49,6 +49,11 @@
|
||||
#[macro_use]
|
||||
extern crate metrics;
|
||||
|
||||
mod local_batch_read;
|
||||
pub use local_batch_read::{
|
||||
LocalBatchReadDecision, LocalBatchReadMetrics, LocalBatchReadStage, record_local_batch_read_decision,
|
||||
};
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{
|
||||
LazyLock, Mutex,
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Opt-in attribution for the buffered local shard batch path.
|
||||
|
||||
use crate::get_stage_metrics_enabled;
|
||||
use metrics::{Counter, Histogram};
|
||||
use std::time::Duration;
|
||||
|
||||
/// One decision per reader-setup attempt, rather than per S3 request.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub enum LocalBatchReadDecision {
|
||||
Selected,
|
||||
MmapDisabled,
|
||||
CopySource,
|
||||
ReadCapExceeded,
|
||||
InlineData,
|
||||
NonLocalShard,
|
||||
InvalidPath,
|
||||
NoLocalShards,
|
||||
UnsupportedPlatform,
|
||||
}
|
||||
|
||||
impl LocalBatchReadDecision {
|
||||
const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Selected => "selected",
|
||||
Self::MmapDisabled => "mmap_disabled",
|
||||
Self::CopySource => "copy_source",
|
||||
Self::ReadCapExceeded => "read_cap_exceeded",
|
||||
Self::InlineData => "inline_data",
|
||||
Self::NonLocalShard => "non_local_shard",
|
||||
Self::InvalidPath => "invalid_path",
|
||||
Self::NoLocalShards => "no_local_shards",
|
||||
Self::UnsupportedPlatform => "unsupported_platform",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Record eligibility, including path preparation only when it actually ran.
|
||||
pub fn record_local_batch_read_decision(decision: LocalBatchReadDecision, preparation: Option<Duration>) {
|
||||
if !get_stage_metrics_enabled() {
|
||||
return;
|
||||
}
|
||||
let decision = decision.as_str();
|
||||
counter!("rustfs_io_get_object_local_batch_decisions_total", "decision" => decision).increment(1);
|
||||
if let Some(duration) = preparation {
|
||||
histogram!("rustfs_io_get_object_local_batch_prepare_seconds", "decision" => decision).record(duration.as_secs_f64());
|
||||
}
|
||||
}
|
||||
|
||||
/// Stages of the existing buffered positioned-read implementation.
|
||||
#[derive(Clone, Copy)]
|
||||
pub enum LocalBatchReadStage {
|
||||
QueueWait,
|
||||
Execution,
|
||||
Metadata,
|
||||
Open,
|
||||
Read,
|
||||
}
|
||||
|
||||
impl LocalBatchReadStage {
|
||||
const fn index(self) -> usize {
|
||||
match self {
|
||||
Self::QueueWait => 0,
|
||||
Self::Execution => 1,
|
||||
Self::Metadata => 2,
|
||||
Self::Open => 3,
|
||||
Self::Read => 4,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Handles captured before dispatch, kept alive by the actual blocking worker.
|
||||
///
|
||||
/// Capturing the recorder once avoids per-I/O registration and also preserves
|
||||
/// completion samples when the caller stops waiting. No timers or handles are
|
||||
/// created with GET-stage metrics disabled. The snapshot applies to the whole
|
||||
/// submitted batch, including work that finishes after the switch is disabled.
|
||||
pub struct LocalBatchReadMetrics {
|
||||
stages: [[Histogram; 2]; 5],
|
||||
shards: [Counter; 2],
|
||||
requested_bytes: [Counter; 2],
|
||||
read_bytes: [Counter; 2],
|
||||
batch_size: Histogram,
|
||||
}
|
||||
|
||||
fn outcome(success: bool) -> &'static str {
|
||||
if success { "success" } else { "error" }
|
||||
}
|
||||
|
||||
impl LocalBatchReadMetrics {
|
||||
/// Resolve bounded-label handles only for an explicitly instrumented batch.
|
||||
/// Boxing the enabled state keeps the blocking task small when disabled.
|
||||
pub fn capture() -> Option<Box<Self>> {
|
||||
if !get_stage_metrics_enabled() {
|
||||
return None;
|
||||
}
|
||||
let stages = ["queue_wait", "execution", "metadata", "open", "read"].map(|stage| {
|
||||
[false, true].map(|success| {
|
||||
histogram!("rustfs_io_get_object_local_batch_stage_seconds", "stage" => stage, "outcome" => outcome(success))
|
||||
})
|
||||
});
|
||||
Some(Box::new(Self {
|
||||
stages,
|
||||
shards: [false, true]
|
||||
.map(|success| counter!("rustfs_io_get_object_local_batch_shards_total", "outcome" => outcome(success))),
|
||||
requested_bytes: [false, true]
|
||||
.map(|success| counter!("rustfs_io_get_object_local_batch_requested_bytes_total", "outcome" => outcome(success))),
|
||||
read_bytes: [false, true]
|
||||
.map(|success| counter!("rustfs_io_get_object_local_batch_read_bytes_total", "outcome" => outcome(success))),
|
||||
batch_size: histogram!("rustfs_io_get_object_local_batch_shards"),
|
||||
}))
|
||||
}
|
||||
|
||||
/// Count shards actually submitted to this blocking batch.
|
||||
pub fn record_batch(&self, shards: usize) {
|
||||
self.batch_size.record(crate::shard_read_fanout_to_f64(shards));
|
||||
}
|
||||
|
||||
/// Record an executed stage; unattempted stages receive no sample.
|
||||
///
|
||||
/// Queue wait succeeds when the worker starts. Execution covers the whole
|
||||
/// batch, while metadata/open/read describe individual I/O operations.
|
||||
/// Read timing excludes buffer allocation; execution timing includes it.
|
||||
pub fn record_stage(&self, stage: LocalBatchReadStage, success: bool, duration: Duration) {
|
||||
self.stages[stage.index()][usize::from(success)].record(duration.as_secs_f64());
|
||||
}
|
||||
|
||||
/// Separate completed shard outcomes and bytes actually returned by read_at.
|
||||
pub fn record_shard(&self, success: bool, requested: usize, read: usize) {
|
||||
let index = usize::from(success);
|
||||
self.shards[index].increment(1);
|
||||
self.requested_bytes[index].increment(u64::try_from(requested).unwrap_or(u64::MAX));
|
||||
self.read_bytes[index].increment(u64::try_from(read).unwrap_or(u64::MAX));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::set_get_stage_metrics_enabled;
|
||||
use crate::tests::{METRICS_FLAG_LOCK, counter_total, histogram_samples};
|
||||
use metrics_util::debugging::DebuggingRecorder;
|
||||
use std::time::Duration;
|
||||
|
||||
#[test]
|
||||
fn local_batch_read_metrics_are_disabled_without_stage_opt_in() {
|
||||
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|error| error.into_inner());
|
||||
let recorder = DebuggingRecorder::new();
|
||||
let snapshotter = recorder.snapshotter();
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
set_get_stage_metrics_enabled(false);
|
||||
assert!(LocalBatchReadMetrics::capture().is_none());
|
||||
record_local_batch_read_decision(LocalBatchReadDecision::Selected, Some(Duration::from_millis(1)));
|
||||
});
|
||||
assert!(snapshotter.snapshot().into_vec().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_batch_read_handles_record_worker_costs_and_actual_bytes() {
|
||||
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|error| error.into_inner());
|
||||
let recorder = DebuggingRecorder::new();
|
||||
let snapshotter = recorder.snapshotter();
|
||||
let observation = metrics::with_local_recorder(&recorder, || {
|
||||
set_get_stage_metrics_enabled(true);
|
||||
record_local_batch_read_decision(LocalBatchReadDecision::Selected, Some(Duration::from_millis(2)));
|
||||
LocalBatchReadMetrics::capture().expect("enabled batch metrics must capture recorder handles")
|
||||
});
|
||||
set_get_stage_metrics_enabled(false);
|
||||
std::thread::spawn(move || {
|
||||
observation.record_stage(LocalBatchReadStage::QueueWait, true, Duration::from_millis(3));
|
||||
observation.record_stage(LocalBatchReadStage::Metadata, false, Duration::from_millis(4));
|
||||
observation.record_shard(false, 1024, 0);
|
||||
observation.record_shard(true, 16, 16);
|
||||
observation.record_batch(2);
|
||||
})
|
||||
.join()
|
||||
.expect("worker should record through the captured handles");
|
||||
set_get_stage_metrics_enabled(false);
|
||||
|
||||
let rows = snapshotter.snapshot().into_vec();
|
||||
assert_eq!(counter_total(&rows, "rustfs_io_get_object_local_batch_decisions_total"), Some(1));
|
||||
assert_eq!(counter_total(&rows, "rustfs_io_get_object_local_batch_shards_total"), Some(2));
|
||||
assert_eq!(counter_total(&rows, "rustfs_io_get_object_local_batch_requested_bytes_total"), Some(1040));
|
||||
assert_eq!(counter_total(&rows, "rustfs_io_get_object_local_batch_read_bytes_total"), Some(16));
|
||||
assert_eq!(histogram_samples(&rows, "rustfs_io_get_object_local_batch_prepare_seconds"), vec![0.002]);
|
||||
assert_eq!(
|
||||
histogram_samples(&rows, "rustfs_io_get_object_local_batch_stage_seconds"),
|
||||
vec![0.003, 0.004]
|
||||
);
|
||||
assert_eq!(histogram_samples(&rows, "rustfs_io_get_object_local_batch_shards"), vec![2.0]);
|
||||
for (key, _, _, _) in rows {
|
||||
for label in key.key().labels() {
|
||||
assert!(matches!(label.key(), "decision" | "stage" | "outcome"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -42,13 +42,14 @@ enum NotifyConfigStoreError {
|
||||
Read(String),
|
||||
Save(String),
|
||||
Converge { persisted: bool, error: String },
|
||||
Conflict,
|
||||
}
|
||||
|
||||
fn notification_convergence_error(persisted: bool, error: impl std::fmt::Display) -> NotificationError {
|
||||
let durable_state = if persisted { "persisted" } else { "unchanged" };
|
||||
NotificationError::Configuration(format!(
|
||||
"configuration was {durable_state} but notification runtime convergence failed: {error}"
|
||||
))
|
||||
NotificationError::ConfigurationConvergence {
|
||||
persisted,
|
||||
detail: error.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn supervise_config_update<T>(
|
||||
@@ -63,12 +64,62 @@ where
|
||||
})?
|
||||
}
|
||||
|
||||
fn configs_equal(first: Option<&KVS>, second: Option<&KVS>) -> bool {
|
||||
match (first, second) {
|
||||
(None, None) => true,
|
||||
(Some(first), Some(second)) => {
|
||||
if first == second {
|
||||
return true;
|
||||
}
|
||||
let mut first: Vec<_> = first.0.iter().collect();
|
||||
let mut second: Vec<_> = second.0.iter().collect();
|
||||
// Lookup uses the first occurrence of a key, so preserve duplicate-key order.
|
||||
first.sort_by(|a, b| a.key.cmp(&b.key));
|
||||
second.sort_by(|a, b| a.key.cmp(&b.key));
|
||||
first == second
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
fn replace_target_if_unchanged(
|
||||
config: &mut Config,
|
||||
target_type: &str,
|
||||
target_name: &str,
|
||||
expected: Option<&KVS>,
|
||||
expected_default: Option<&KVS>,
|
||||
kvs: &KVS,
|
||||
) -> Result<bool, NotifyConfigStoreError> {
|
||||
let targets = config.0.get(target_type);
|
||||
let mut matching = targets
|
||||
.into_iter()
|
||||
.flat_map(|targets| targets.iter())
|
||||
.filter(|(name, _)| name.eq_ignore_ascii_case(target_name));
|
||||
let current = matching.next();
|
||||
if matching.next().is_some() {
|
||||
return Err(NotifyConfigStoreError::Conflict);
|
||||
}
|
||||
let stored_name = current
|
||||
.map(|(name, _)| name.clone())
|
||||
.unwrap_or_else(|| target_name.to_owned());
|
||||
let current_default = targets.and_then(|targets| targets.get(rustfs_config::DEFAULT_DELIMITER));
|
||||
if !configs_equal(current.map(|(_, kvs)| kvs), expected) || !configs_equal(current_default, expected_default) {
|
||||
return Err(NotifyConfigStoreError::Conflict);
|
||||
}
|
||||
config
|
||||
.0
|
||||
.entry(target_type.to_owned())
|
||||
.or_default()
|
||||
.insert(stored_name, kvs.clone());
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
async fn update_server_config<F>(
|
||||
mut modifier: F,
|
||||
lifecycle: NotifyLifecycleCoordinator,
|
||||
) -> Result<Option<(crate::lifecycle::NotificationLifecycleTransition, bool)>, NotifyConfigStoreError>
|
||||
where
|
||||
F: FnMut(&mut Config) -> bool + Send + 'static,
|
||||
F: FnMut(&mut Config, Option<String>) -> Result<bool, NotifyConfigStoreError> + Send + 'static,
|
||||
{
|
||||
let Some(store) = resolve_notify_object_store_handle() else {
|
||||
return Err(NotifyConfigStoreError::StorageNotAvailable);
|
||||
@@ -79,7 +130,8 @@ where
|
||||
.await
|
||||
.map_err(NotifyConfigStoreError::Read)?;
|
||||
let mut config = snapshot.config.clone();
|
||||
if !modifier(&mut config) {
|
||||
let generation = snapshot.generation().map(|value| value.to_string());
|
||||
if !modifier(&mut config, generation)? {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
@@ -245,6 +297,30 @@ impl NotifyConfigManager {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Replace a target only when its persisted configuration is unchanged since validation.
|
||||
pub async fn set_target_config_if_unchanged(
|
||||
&self,
|
||||
target_type: &str,
|
||||
target_name: &str,
|
||||
expected: Option<KVS>,
|
||||
expected_default: Option<KVS>,
|
||||
expected_generation: Option<String>,
|
||||
kvs: KVS,
|
||||
) -> Result<(), NotificationError> {
|
||||
let ttype = target_type.to_lowercase();
|
||||
let tname = target_name.to_lowercase();
|
||||
self.update_config_and_reload_checked(move |config, generation| {
|
||||
if expected_generation
|
||||
.as_ref()
|
||||
.is_some_and(|expected| generation.as_ref() != Some(expected))
|
||||
{
|
||||
return Err(NotifyConfigStoreError::Conflict);
|
||||
}
|
||||
replace_target_if_unchanged(config, &ttype, &tname, expected.as_ref(), expected_default.as_ref(), &kvs)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn remove_target_config(&self, target_type: &str, target_name: &str) -> Result<(), NotificationError> {
|
||||
debug!(
|
||||
event = EVENT_NOTIFY_CONFIG_UPDATE,
|
||||
@@ -338,9 +414,17 @@ impl NotifyConfigManager {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn update_config_and_reload<F>(&self, modifier: F) -> Result<(), NotificationError>
|
||||
async fn update_config_and_reload<F>(&self, mut modifier: F) -> Result<(), NotificationError>
|
||||
where
|
||||
F: FnMut(&mut Config) -> bool + Send + 'static,
|
||||
{
|
||||
self.update_config_and_reload_checked(move |config, _generation| Ok(modifier(config)))
|
||||
.await
|
||||
}
|
||||
|
||||
async fn update_config_and_reload_checked<F>(&self, modifier: F) -> Result<(), NotificationError>
|
||||
where
|
||||
F: FnMut(&mut Config, Option<String>) -> Result<bool, NotifyConfigStoreError> + Send + 'static,
|
||||
{
|
||||
if self.lifecycle.state() == NotificationRuntimeState::Terminated {
|
||||
return Err(NotificationError::Initialization("Notification runtime has terminated".to_string()));
|
||||
@@ -356,6 +440,7 @@ impl NotifyConfigManager {
|
||||
NotifyConfigStoreError::Read(err) => NotificationError::ReadConfig(err),
|
||||
NotifyConfigStoreError::Save(err) => NotificationError::SaveConfig(err),
|
||||
NotifyConfigStoreError::Converge { persisted, error } => notification_convergence_error(persisted, error),
|
||||
NotifyConfigStoreError::Conflict => NotificationError::ConfigurationConflict,
|
||||
})?
|
||||
else {
|
||||
debug!(
|
||||
@@ -394,6 +479,126 @@ mod tests {
|
||||
use crate::rules::RulesMap;
|
||||
use crate::{NotificationError, NotificationRuntimeState};
|
||||
|
||||
#[test]
|
||||
fn config_comparison_preserves_duplicates_and_hidden_flags() {
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
let field = |key: &str, value: &str, hidden_if_empty: bool| KV {
|
||||
key: key.into(),
|
||||
value: value.into(),
|
||||
hidden_if_empty,
|
||||
};
|
||||
let original = KVS(vec![
|
||||
field("password", "secret", false),
|
||||
field("token", "", true),
|
||||
field("password", "secret", false),
|
||||
]);
|
||||
let reordered = KVS(vec![
|
||||
field("password", "secret", false),
|
||||
field("password", "secret", false),
|
||||
field("token", "", true),
|
||||
]);
|
||||
let missing_duplicate = KVS(vec![field("password", "secret", false), field("token", "", true)]);
|
||||
let changed_flag = KVS(vec![
|
||||
field("password", "secret", false),
|
||||
field("password", "secret", false),
|
||||
field("token", "", false),
|
||||
]);
|
||||
assert!(super::configs_equal(Some(&original), Some(&reordered)));
|
||||
assert!(!super::configs_equal(Some(&original), Some(&missing_duplicate)));
|
||||
assert!(!super::configs_equal(Some(&original), Some(&changed_flag)));
|
||||
assert!(!super::configs_equal(Some(&original), None));
|
||||
let first_value = KVS(vec![
|
||||
field("password", "old", false),
|
||||
field("token", "", true),
|
||||
field("password", "new", false),
|
||||
]);
|
||||
let reordered_keys = KVS(vec![
|
||||
field("token", "", true),
|
||||
field("password", "old", false),
|
||||
field("password", "new", false),
|
||||
]);
|
||||
let reordered_duplicates = KVS(vec![
|
||||
field("token", "", true),
|
||||
field("password", "new", false),
|
||||
field("password", "old", false),
|
||||
]);
|
||||
assert!(super::configs_equal(Some(&first_value), Some(&reordered_keys)));
|
||||
assert!(!super::configs_equal(Some(&first_value), Some(&reordered_duplicates)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn conditional_target_update_rejects_rotation_and_deletion() {
|
||||
use rustfs_config::server_config::{Config, KVS};
|
||||
let mut old = KVS::new();
|
||||
old.insert("password".into(), "old".into());
|
||||
let mut rotated = KVS::new();
|
||||
rotated.insert("password".into(), "new".into());
|
||||
let mut config = Config::new();
|
||||
config
|
||||
.0
|
||||
.entry("notify_nats".into())
|
||||
.or_default()
|
||||
.insert("primary".into(), rotated.clone());
|
||||
let result = super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &old);
|
||||
assert!(matches!(result, Err(super::NotifyConfigStoreError::Conflict)));
|
||||
assert_eq!(config.0["notify_nats"]["primary"], rotated);
|
||||
config.0.get_mut("notify_nats").expect("subsystem exists").remove("primary");
|
||||
assert!(matches!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &old),
|
||||
Err(super::NotifyConfigStoreError::Conflict)
|
||||
));
|
||||
assert!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", None, None, &old)
|
||||
.expect("create unchanged target")
|
||||
);
|
||||
assert!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &rotated)
|
||||
.expect("replace unchanged target")
|
||||
);
|
||||
assert_eq!(config.0["notify_nats"]["primary"], rotated);
|
||||
|
||||
let mut first = KVS::new();
|
||||
first.insert("url".into(), "nats://example.com".into());
|
||||
first.insert("token".into(), "secret".into());
|
||||
let mut reordered = KVS::new();
|
||||
reordered.insert("token".into(), "secret".into());
|
||||
reordered.insert("url".into(), "nats://example.com".into());
|
||||
config
|
||||
.0
|
||||
.get_mut("notify_nats")
|
||||
.expect("subsystem")
|
||||
.insert("primary".into(), first);
|
||||
assert!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&reordered), None, &rotated)
|
||||
.expect("key order does not signal a change")
|
||||
);
|
||||
let targets = config.0.get_mut("notify_nats").expect("subsystem");
|
||||
targets.remove("primary");
|
||||
targets.insert("Primary".into(), rotated.clone());
|
||||
assert!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&rotated), None, &old)
|
||||
.expect("case-insensitive match")
|
||||
);
|
||||
let targets = &config.0["notify_nats"];
|
||||
assert!(targets.contains_key("Primary"));
|
||||
assert!(!targets.contains_key("primary"));
|
||||
let mut old_default = KVS::new();
|
||||
old_default.insert("password".into(), "old-default".into());
|
||||
let mut new_default = KVS::new();
|
||||
new_default.insert("password".into(), "new-default".into());
|
||||
config
|
||||
.0
|
||||
.get_mut("notify_nats")
|
||||
.expect("subsystem")
|
||||
.insert("_".into(), new_default.clone());
|
||||
assert!(matches!(
|
||||
super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), Some(&old_default), &rotated),
|
||||
Err(super::NotifyConfigStoreError::Conflict)
|
||||
));
|
||||
assert_eq!(config.0["notify_nats"]["Primary"], old);
|
||||
assert_eq!(config.0["notify_nats"]["_"], new_default);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn supervised_config_update_survives_waiter_cancellation() {
|
||||
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
|
||||
@@ -550,13 +755,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn convergence_error_reports_durable_write_state() {
|
||||
for (persisted, expected) in [(true, "configuration was persisted"), (false, "configuration was unchanged")] {
|
||||
for persisted in [true, false] {
|
||||
let error = notification_convergence_error(persisted, "injected failure");
|
||||
let NotificationError::Configuration(message) = error else {
|
||||
panic!("convergence error should be a configuration error");
|
||||
};
|
||||
assert!(message.contains(expected), "unexpected convergence error: {message}");
|
||||
assert!(message.contains("injected failure"));
|
||||
assert!(matches!(error, NotificationError::ConfigurationConvergence { persisted: value, .. } if value == persisted));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,6 +38,12 @@ pub enum NotificationError {
|
||||
#[error("Configuration error: {0}")]
|
||||
Configuration(String),
|
||||
|
||||
#[error("Target configuration changed during validation")]
|
||||
ConfigurationConflict,
|
||||
|
||||
#[error("Notification configuration convergence failed (persisted: {persisted}): {detail}")]
|
||||
ConfigurationConvergence { persisted: bool, detail: String },
|
||||
|
||||
#[error("ARN not found: {0}")]
|
||||
ARNNotFound(String),
|
||||
|
||||
|
||||
@@ -298,6 +298,22 @@ impl NotificationSystem {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Replace a target only if its persisted configuration still matches the validated snapshot.
|
||||
pub async fn set_target_config_if_unchanged(
|
||||
&self,
|
||||
target_type: &str,
|
||||
target_name: &str,
|
||||
expected: Option<KVS>,
|
||||
expected_default: Option<KVS>,
|
||||
expected_generation: Option<String>,
|
||||
kvs: KVS,
|
||||
) -> Result<(), NotificationError> {
|
||||
self.services
|
||||
.config_manager
|
||||
.set_target_config_if_unchanged(target_type, target_name, expected, expected_default, expected_generation, kvs)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Removes all notification configurations for a bucket.
|
||||
/// If the configuration is successfully removed, the entire notification system will be automatically reloaded.
|
||||
///
|
||||
|
||||
@@ -83,6 +83,16 @@ const GET_OBJECT_DURATION_HISTOGRAM_BUCKETS: &[f64] = &[
|
||||
0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0,
|
||||
];
|
||||
|
||||
const LOCAL_BATCH_DURATION_HISTOGRAM_METRICS: &[&str] = &[
|
||||
"rustfs_io_get_object_local_batch_prepare_seconds",
|
||||
"rustfs_io_get_object_local_batch_stage_seconds",
|
||||
];
|
||||
|
||||
const LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS: &[f64] = &[
|
||||
0.0000001, 0.00000025, 0.0000005, 0.000001, 0.0000025, 0.000005, 0.00001, 0.000025, 0.00005, 0.0001, 0.00025, 0.0005, 0.001,
|
||||
0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0,
|
||||
];
|
||||
|
||||
const DEFAULT_OTLP_HTTP_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const OTEL_EXPORTER_OTLP_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_TIMEOUT";
|
||||
const OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT";
|
||||
@@ -481,21 +491,25 @@ fn build_meter_provider(
|
||||
}
|
||||
|
||||
fn get_object_duration_histogram_view(instrument: &Instrument) -> Option<Stream> {
|
||||
if !is_get_object_duration_histogram_metric(instrument.name()) {
|
||||
return None;
|
||||
}
|
||||
let boundaries = get_object_duration_histogram_buckets(instrument.name())?;
|
||||
|
||||
Stream::builder()
|
||||
.with_aggregation(Aggregation::ExplicitBucketHistogram {
|
||||
boundaries: GET_OBJECT_DURATION_HISTOGRAM_BUCKETS.to_vec(),
|
||||
boundaries: boundaries.to_vec(),
|
||||
record_min_max: true,
|
||||
})
|
||||
.build()
|
||||
.ok()
|
||||
}
|
||||
|
||||
fn is_get_object_duration_histogram_metric(name: &str) -> bool {
|
||||
GET_OBJECT_DURATION_HISTOGRAM_METRICS.contains(&name)
|
||||
fn get_object_duration_histogram_buckets(name: &str) -> Option<&'static [f64]> {
|
||||
if LOCAL_BATCH_DURATION_HISTOGRAM_METRICS.contains(&name) {
|
||||
Some(LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS)
|
||||
} else if GET_OBJECT_DURATION_HISTOGRAM_METRICS.contains(&name) {
|
||||
Some(GET_OBJECT_DURATION_HISTOGRAM_BUCKETS)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Build an optional [`SdkLoggerProvider`] for the given log endpoint.
|
||||
@@ -1098,10 +1112,22 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_get_object_duration_histogram_metric_match_is_scoped() {
|
||||
assert!(is_get_object_duration_histogram_metric("rustfs_io_get_object_stage_duration_seconds"));
|
||||
assert!(is_get_object_duration_histogram_metric("rustfs_io_get_object_request_duration_seconds"));
|
||||
assert!(!is_get_object_duration_histogram_metric("rustfs_io_put_object_request_duration_seconds"));
|
||||
assert!(!is_get_object_duration_histogram_metric("rustfs_io_get_object_response_size_bytes"));
|
||||
for name in [
|
||||
"rustfs_io_get_object_stage_duration_seconds",
|
||||
"rustfs_io_get_object_request_duration_seconds",
|
||||
] {
|
||||
assert_eq!(get_object_duration_histogram_buckets(name), Some(GET_OBJECT_DURATION_HISTOGRAM_BUCKETS));
|
||||
}
|
||||
for name in LOCAL_BATCH_DURATION_HISTOGRAM_METRICS {
|
||||
assert_eq!(get_object_duration_histogram_buckets(name), Some(LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS));
|
||||
}
|
||||
for name in [
|
||||
"rustfs_io_put_object_request_duration_seconds",
|
||||
"rustfs_io_get_object_response_size_bytes",
|
||||
"rustfs_io_get_object_local_batch_shards",
|
||||
] {
|
||||
assert!(get_object_duration_histogram_buckets(name).is_none());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1111,6 +1137,17 @@ mod tests {
|
||||
assert_eq!(GET_OBJECT_DURATION_HISTOGRAM_BUCKETS.last(), Some(&10.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_batch_duration_buckets_resolve_sub_microsecond_operations() {
|
||||
assert!(
|
||||
LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS
|
||||
.windows(2)
|
||||
.all(|pair| pair[0] < pair[1])
|
||||
);
|
||||
assert_eq!(LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS.first(), Some(&0.0000001));
|
||||
assert_eq!(LOCAL_BATCH_DURATION_HISTOGRAM_BUCKETS.last(), Some(&10.0));
|
||||
}
|
||||
|
||||
#[cfg(all(
|
||||
feature = "pyroscope",
|
||||
any(target_os = "macos", all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))
|
||||
|
||||
@@ -568,6 +568,24 @@ pub enum AdminAction {
|
||||
GetTableLifecycleAction,
|
||||
#[strum(serialize = "admin:SetTableLifecycle")]
|
||||
SetTableLifecycleAction,
|
||||
#[strum(serialize = "admin:CreatePluginDelegation")]
|
||||
CreatePluginDelegationAction,
|
||||
#[strum(serialize = "admin:IssuePluginAttempt")]
|
||||
IssuePluginAttemptAction,
|
||||
#[strum(serialize = "admin:SubmitPluginJob")]
|
||||
SubmitPluginJobAction,
|
||||
#[strum(serialize = "admin:GetPluginRun")]
|
||||
GetPluginRunAction,
|
||||
#[strum(serialize = "admin:CancelPluginRun")]
|
||||
CancelPluginRunAction,
|
||||
#[strum(serialize = "admin:ListPluginRuns")]
|
||||
ListPluginRunsAction,
|
||||
#[strum(serialize = "admin:ReplayPluginRun")]
|
||||
ReplayPluginRunAction,
|
||||
#[strum(serialize = "admin:GetPluginDelegation")]
|
||||
GetPluginDelegationAction,
|
||||
#[strum(serialize = "admin:RevokePluginDelegation")]
|
||||
RevokePluginDelegationAction,
|
||||
#[strum(serialize = "admin:GetTableCredentials")]
|
||||
GetTableCredentialsAction,
|
||||
#[strum(serialize = "admin:RunTableMaintenance")]
|
||||
@@ -713,6 +731,15 @@ impl AdminAction {
|
||||
| AdminAction::GetTableLifecycleAction
|
||||
| AdminAction::SetTableLifecycleAction
|
||||
| AdminAction::GetTableCredentialsAction
|
||||
| AdminAction::IssuePluginAttemptAction
|
||||
| AdminAction::SubmitPluginJobAction
|
||||
| AdminAction::GetPluginRunAction
|
||||
| AdminAction::CancelPluginRunAction
|
||||
| AdminAction::ListPluginRunsAction
|
||||
| AdminAction::ReplayPluginRunAction
|
||||
| AdminAction::GetPluginDelegationAction
|
||||
| AdminAction::CreatePluginDelegationAction
|
||||
| AdminAction::RevokePluginDelegationAction
|
||||
| AdminAction::RunTableMaintenanceAction
|
||||
| AdminAction::GetTableMetadataLocationAction
|
||||
| AdminAction::SetTableMetadataLocationAction
|
||||
@@ -913,6 +940,35 @@ mod tests {
|
||||
assert_eq!(arr[0].as_str().unwrap(), "s3:*");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plugin_delegation_actions_are_explicit_and_unscoped() {
|
||||
for (name, expected) in [
|
||||
("admin:GetPluginDelegation", AdminAction::GetPluginDelegationAction),
|
||||
("admin:CreatePluginDelegation", AdminAction::CreatePluginDelegationAction),
|
||||
("admin:RevokePluginDelegation", AdminAction::RevokePluginDelegationAction),
|
||||
("admin:SubmitPluginJob", AdminAction::SubmitPluginJobAction),
|
||||
("admin:GetPluginRun", AdminAction::GetPluginRunAction),
|
||||
("admin:CancelPluginRun", AdminAction::CancelPluginRunAction),
|
||||
("admin:ListPluginRuns", AdminAction::ListPluginRunsAction),
|
||||
("admin:ReplayPluginRun", AdminAction::ReplayPluginRunAction),
|
||||
] {
|
||||
assert_eq!(AdminAction::try_from(name).unwrap(), expected);
|
||||
assert!(expected.is_valid());
|
||||
assert!(!expected.is_table_resource_scoped());
|
||||
}
|
||||
assert_eq!(
|
||||
AdminAction::try_from("admin:IssuePluginAttempt").unwrap(),
|
||||
AdminAction::IssuePluginAttemptAction
|
||||
);
|
||||
assert!(AdminAction::IssuePluginAttemptAction.is_valid());
|
||||
assert!(AdminAction::SubmitPluginJobAction.is_valid());
|
||||
assert!(AdminAction::GetPluginRunAction.is_valid());
|
||||
assert!(AdminAction::CancelPluginRunAction.is_valid());
|
||||
assert!(AdminAction::ListPluginRunsAction.is_valid());
|
||||
assert!(AdminAction::ReplayPluginRunAction.is_valid());
|
||||
assert!(AdminAction::try_from("admin:IssuePluginCredentials").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_replication_metrics_admin_action_is_valid() {
|
||||
assert!(AdminAction::GetReplicationMetricsAction.is_valid());
|
||||
|
||||
@@ -53,6 +53,10 @@ pub enum ScannerError {
|
||||
#[error("Remote namespace scanner disk is already active")]
|
||||
RemoteDiskBusy,
|
||||
|
||||
/// Node resource pressure deferred work before its first storage poll.
|
||||
#[error("Scanner node resource admission deferred")]
|
||||
ResourceDeferred,
|
||||
|
||||
/// Partial data usage cache produced before the scanner stopped.
|
||||
#[error("Scanner stopped with partial data usage cache")]
|
||||
PartialCache(Box<DataUsageCache>),
|
||||
|
||||
+73
-14
@@ -50,9 +50,9 @@ use storage_api::owner::{
|
||||
};
|
||||
#[cfg(test)]
|
||||
use storage_api::owner::{
|
||||
EcstoreDiskOption, EcstoreDiskStore, EcstoreEndpoint, EcstoreEndpointServerPools, EcstoreEndpoints, EcstoreInstanceContext,
|
||||
EcstorePoolEndpoints, ecstore_config_init, ecstore_init_bucket_metadata_sys, ecstore_init_local_disks_with_instance_ctx,
|
||||
ecstore_new_disk,
|
||||
EcstoreDiskOption, EcstoreDiskStore, EcstoreEndpoint, EcstoreEndpointServerPools, EcstoreEndpoints, EcstoreExpiryState,
|
||||
EcstoreInstanceContext, EcstorePoolEndpoints, ecstore_config_init, ecstore_init_bucket_metadata_sys,
|
||||
ecstore_init_local_disks_with_instance_ctx, ecstore_new_disk,
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use uuid::Uuid;
|
||||
@@ -123,7 +123,7 @@ pub use storage_api::scan::{
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
|
||||
SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
|
||||
};
|
||||
pub use workload_admission::set_scanner_workload_admission_snapshot_provider;
|
||||
pub use workload_admission::{set_scanner_node_admission, set_scanner_workload_admission_snapshot_provider};
|
||||
|
||||
static SCANNER_ACTIVE_WORK_UNITS: AtomicU64 = AtomicU64::new(0);
|
||||
static SCANNER_RUNTIME_INSTANCES: AtomicU64 = AtomicU64::new(0);
|
||||
@@ -678,8 +678,37 @@ fn reset_tier_name_cache_for_test() {
|
||||
TIER_REGISTRY_GENERATION.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_runtime_free_version(oi: ScannerObjectInfo) {
|
||||
ecstore_expiry_state_handle().write().await.enqueue_free_version(oi);
|
||||
// Only a task running an isolated unit test can override these handles.
|
||||
#[cfg(test)]
|
||||
struct ExpiryLockTestContext {
|
||||
state: Arc<tokio::sync::RwLock<EcstoreExpiryState>>,
|
||||
store: Option<Arc<ECStore>>,
|
||||
completed_incarnation_reads: Arc<std::sync::atomic::AtomicUsize>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
tokio::task_local! {
|
||||
static EXPIRY_LOCK_TEST_CONTEXT: ExpiryLockTestContext;
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_runtime_free_version(oi: ScannerObjectInfo, cancellation: Option<&CancellationToken>) {
|
||||
#[cfg(test)]
|
||||
let state = EXPIRY_LOCK_TEST_CONTEXT
|
||||
.try_with(|context| context.state.clone())
|
||||
.unwrap_or_else(|_| ecstore_expiry_state_handle());
|
||||
#[cfg(not(test))]
|
||||
let state = ecstore_expiry_state_handle();
|
||||
let state = match cancellation {
|
||||
Some(cancel) => tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => return,
|
||||
state = state.write() => state,
|
||||
},
|
||||
None => state.write().await,
|
||||
};
|
||||
if !cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
state.enqueue_free_version(oi);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_runtime_newer_noncurrent(
|
||||
@@ -687,20 +716,50 @@ pub(crate) async fn enqueue_runtime_newer_noncurrent(
|
||||
to_delete_objs: Vec<ObjectToDelete>,
|
||||
event: Event,
|
||||
src: &LcEventSrc,
|
||||
cancellation: Option<&CancellationToken>,
|
||||
) -> bool {
|
||||
let Some(store) = ecstore_resolve_object_store_handle() else {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return false;
|
||||
}
|
||||
#[cfg(test)]
|
||||
let store = EXPIRY_LOCK_TEST_CONTEXT
|
||||
.try_with(|context| context.store.clone())
|
||||
.unwrap_or_else(|_| ecstore_resolve_object_store_handle());
|
||||
#[cfg(not(test))]
|
||||
let store = ecstore_resolve_object_store_handle();
|
||||
let Some(store) = store else {
|
||||
return false;
|
||||
};
|
||||
let Ok(bucket_incarnation_id) = store.bucket_incarnation_id(bucket).await else {
|
||||
return false;
|
||||
};
|
||||
ecstore_expiry_state_handle().write().await.enqueue_by_newer_noncurrent(
|
||||
bucket,
|
||||
to_delete_objs,
|
||||
event,
|
||||
src,
|
||||
bucket_incarnation_id,
|
||||
)
|
||||
#[cfg(test)]
|
||||
let _ = EXPIRY_LOCK_TEST_CONTEXT.try_with(|context| {
|
||||
context
|
||||
.completed_incarnation_reads
|
||||
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||
});
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return false;
|
||||
}
|
||||
#[cfg(test)]
|
||||
let state = EXPIRY_LOCK_TEST_CONTEXT
|
||||
.try_with(|context| context.state.clone())
|
||||
.unwrap_or_else(|_| ecstore_expiry_state_handle());
|
||||
#[cfg(not(test))]
|
||||
let state = ecstore_expiry_state_handle();
|
||||
let mut state = match cancellation {
|
||||
Some(cancel) => tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => return false,
|
||||
state = state.write() => state,
|
||||
},
|
||||
None => state.write().await,
|
||||
};
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return false;
|
||||
}
|
||||
state.enqueue_by_newer_noncurrent(bucket, to_delete_objs, event, src, bucket_incarnation_id)
|
||||
}
|
||||
|
||||
pub(crate) async fn queue_replication_heal(
|
||||
|
||||
@@ -705,7 +705,7 @@ async fn scan_and_persist_local_bucket(
|
||||
RemoteScannerServerError::worker("remote namespace scanner cache publication is blocked by data movement")
|
||||
})?;
|
||||
let cache_name = path_join_buf(&[&bucket, DATA_USAGE_CACHE_NAME]);
|
||||
let guard = acquire_scanner_cache_locks(set.as_ref(), &cache_name, source)
|
||||
let mut guard = acquire_scanner_cache_locks(set.as_ref(), &cache_name, source)
|
||||
.await
|
||||
.map_err(|err| {
|
||||
if err.is_contention() {
|
||||
@@ -791,19 +791,23 @@ async fn scan_and_persist_local_bucket(
|
||||
|
||||
let set_disks = scanner_set_disk_inventory(set.as_ref()).await;
|
||||
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel::<DataUsageCache>(1);
|
||||
let (node_tx, mut node_rx) = crate::workload_admission::scanner_node_request_channel();
|
||||
let scan_ctx = ctx.child_token();
|
||||
let scan = ScannerIODisk::nsscanner_disk(
|
||||
disk.clone(),
|
||||
scan_ctx.clone(),
|
||||
budget,
|
||||
set_disks,
|
||||
cache,
|
||||
None,
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: Some(checkpoint_tx),
|
||||
},
|
||||
let scan = crate::workload_admission::with_scanner_node_requests(
|
||||
node_tx,
|
||||
ScannerIODisk::nsscanner_disk(
|
||||
disk.clone(),
|
||||
scan_ctx.clone(),
|
||||
budget,
|
||||
set_disks,
|
||||
cache,
|
||||
None,
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: Some(checkpoint_tx),
|
||||
},
|
||||
),
|
||||
);
|
||||
tokio::pin!(scan);
|
||||
let fence_watch = watch_remote_scanner_request_fence(next_cycle, leader_epoch, store.clone(), NS_SCANNER_FENCE_POLL_INTERVAL);
|
||||
@@ -813,6 +817,27 @@ async fn scan_and_persist_local_bucket(
|
||||
let mut checkpoint_channel_closed = false;
|
||||
let outcome = loop {
|
||||
tokio::select! {
|
||||
request = async {
|
||||
match node_rx.as_mut() {
|
||||
Some(receiver) => receiver.recv().await,
|
||||
None => std::future::pending().await,
|
||||
}
|
||||
} => {
|
||||
let Some(request) = request else {
|
||||
node_rx = None;
|
||||
continue;
|
||||
};
|
||||
if let Err(error) = wait_for_remote_node_resumption(
|
||||
guard.resume_node_operation(set.clone(), &cache_name, source, &revisions, request),
|
||||
fence_watch.as_mut(),
|
||||
).await {
|
||||
scan_ctx.cancel();
|
||||
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
|
||||
return Err(RemoteScannerServerError::retry_bucket(format!(
|
||||
"remote namespace scanner could not resume node admission: {error}",
|
||||
)));
|
||||
}
|
||||
}
|
||||
result = &mut scan => {
|
||||
match result {
|
||||
Ok(outcome) => break outcome,
|
||||
@@ -1121,6 +1146,22 @@ pub(crate) async fn scan_remote_bucket(
|
||||
result
|
||||
}
|
||||
|
||||
/// Resource waits must keep checking the persisted leader fence.
|
||||
async fn wait_for_remote_node_resumption<F, G>(resume: F, fence_watch: Pin<&mut G>) -> Result<(), StorageError>
|
||||
where
|
||||
F: Future<Output = Result<(), StorageError>>,
|
||||
G: Future<Output = Result<(), ScannerError>>,
|
||||
{
|
||||
tokio::select! {
|
||||
biased;
|
||||
result = fence_watch => Err(match result {
|
||||
Ok(()) => StorageError::other("remote namespace scanner fence watcher stopped unexpectedly"),
|
||||
Err(error) => StorageError::other(error),
|
||||
}),
|
||||
result = resume => result,
|
||||
}
|
||||
}
|
||||
|
||||
fn finish_remote_scanner_stream(
|
||||
result: RemoteScannerStreamResult<RemoteScannerOutcome>,
|
||||
budget: &ScannerCycleBudget,
|
||||
|
||||
@@ -1006,6 +1006,58 @@ async fn fence_watcher_stops_work_after_persisted_leader_changes() {
|
||||
assert!(err.to_string().contains("requested_epoch=9"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn node_admission_wait_does_not_hide_replacement_leader() {
|
||||
use rustfs_concurrency::adaptive::{FeedbackController, FeedbackObservation, NodeAdmission, ResourcePressure};
|
||||
use std::num::NonZeroU16;
|
||||
|
||||
let store = Arc::new(MutableCycleStateStore {
|
||||
state: RwLock::new(crate::scanner::encode_scanner_cycle_fence_for_test(7, 9)),
|
||||
});
|
||||
let cache = Mutex::new(None);
|
||||
let refresh = AsyncMutex::new(());
|
||||
let watcher = watch_remote_scanner_request_fence_with_cache(
|
||||
7,
|
||||
9,
|
||||
store.clone(),
|
||||
Duration::from_millis(5),
|
||||
&cache,
|
||||
&refresh,
|
||||
Duration::from_millis(5),
|
||||
);
|
||||
tokio::pin!(watcher);
|
||||
let gate = NodeAdmission::new(NonZeroU16::new(1).expect("one shared node unit"));
|
||||
let storage_polled = AtomicBool::new(false);
|
||||
{
|
||||
let resume = async {
|
||||
let _permit = gate.acquire().await;
|
||||
storage_polled.store(true, Ordering::Relaxed);
|
||||
Ok(())
|
||||
};
|
||||
let waiting = wait_for_remote_node_resumption(resume, watcher.as_mut());
|
||||
tokio::pin!(waiting);
|
||||
assert!(futures::poll!(&mut waiting).is_pending(), "no resource sample grants a node unit");
|
||||
*store.state.write().expect("persist replacement leader") = crate::scanner::encode_scanner_cycle_fence_for_test(7, 10);
|
||||
let error = tokio::time::timeout(Duration::from_millis(100), waiting)
|
||||
.await
|
||||
.expect("resource pause must not hide a changed leader")
|
||||
.expect_err("old leader cannot resume storage work");
|
||||
assert!(error.to_string().contains("requested_epoch=9"));
|
||||
}
|
||||
assert!(!storage_polled.load(Ordering::Relaxed));
|
||||
assert_eq!(gate.snapshot().active, 0);
|
||||
gate.update(
|
||||
&mut FeedbackController::new(),
|
||||
Some(FeedbackObservation {
|
||||
generation: 1,
|
||||
max_concurrency: 1,
|
||||
pressure: ResourcePressure::Unknown,
|
||||
completed_units: None,
|
||||
}),
|
||||
);
|
||||
assert!(gate.try_acquire().is_some(), "failed fencing drops the old node queue ticket");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn concurrent_cycle_cache_misses_share_one_backend_read() {
|
||||
let request = RemoteScannerRequest(test_request(Uuid::new_v4()));
|
||||
|
||||
@@ -43,6 +43,7 @@ use crate::storage_api::owner::{
|
||||
};
|
||||
#[cfg(test)]
|
||||
use crate::storage_api::owner::{EcstoreExpirationStatus as ExpirationStatus, EcstoreLifecycleRule as LifecycleRule};
|
||||
use crate::workload_admission::run_scanner_node_operation;
|
||||
use metrics::{counter, describe_counter};
|
||||
use rustfs_common::trace_bus::{TraceEvent, TraceFunc, TraceKind, trace_emit, trace_subscriber_count};
|
||||
use rustfs_filemeta::{
|
||||
@@ -795,6 +796,7 @@ pub struct FolderScanner {
|
||||
resolved_failed_object_paths: HashSet<String>,
|
||||
|
||||
sleeper: DynamicSleeper,
|
||||
node_admission: Option<Arc<rustfs_concurrency::adaptive::NodeAdmission>>,
|
||||
// should_heal: Arc<dyn Fn() -> bool + Send + Sync>,
|
||||
disks: Vec<Arc<Disk>>,
|
||||
disks_quorum: usize,
|
||||
@@ -828,6 +830,8 @@ pub struct FolderScanner {
|
||||
pending_size_reconciliation_truncated: bool,
|
||||
#[cfg(test)]
|
||||
list_path_raw_options_observer: Option<mpsc::UnboundedSender<ListPathRawTimeoutSnapshot>>,
|
||||
#[cfg(test)]
|
||||
defer_abandoned_entries_until_finished: bool,
|
||||
}
|
||||
|
||||
struct RawEnumerationProgress {
|
||||
@@ -998,12 +1002,13 @@ async fn next_scanner_dir_entry(
|
||||
ctx: &CancellationToken,
|
||||
sleeper: &DynamicSleeper,
|
||||
entries: &mut ScannerDirectory,
|
||||
admission: Option<&Arc<rustfs_concurrency::adaptive::NodeAdmission>>,
|
||||
) -> Result<Option<tokio::fs::DirEntry>, ScannerError> {
|
||||
if entries.batch_entries == SCANNER_DIRECTORY_BATCH_ENTRIES {
|
||||
wait_for_scanner_io(ctx, tokio::task::yield_now()).await?;
|
||||
entries.batch_entries = 0;
|
||||
}
|
||||
let entry = wait_for_scanner_io(ctx, entries.next_entry()).await??;
|
||||
let entry = wait_for_scanner_io(ctx, run_scanner_node_operation(admission, ctx, entries.next_entry())).await???;
|
||||
if entry.is_some() {
|
||||
// Tokio prefetches bounded chunks. Pacing consumption also paces the
|
||||
// next refill, without charging an extra entry for EOF.
|
||||
@@ -1923,7 +1928,12 @@ impl FolderScanner {
|
||||
// directory that vanished during iteration instead of failing the
|
||||
// whole parent subtree. The bucket root (no parent) still propagates
|
||||
// NotFound so namespace removal is detected by the caller.
|
||||
let mut dir_reader = match wait_for_scanner_io(&ctx, ScannerDirectory::open(&dir_path)).await? {
|
||||
let mut dir_reader = match wait_for_scanner_io(
|
||||
&ctx,
|
||||
run_scanner_node_operation(self.node_admission.as_ref(), &ctx, ScannerDirectory::open(&dir_path)),
|
||||
)
|
||||
.await??
|
||||
{
|
||||
Ok(dir_reader) => Some(dir_reader),
|
||||
Err(e)
|
||||
if folder.parent.is_some()
|
||||
@@ -1961,7 +1971,7 @@ impl FolderScanner {
|
||||
// Hold at most one metadata entry until a candidate reaches
|
||||
// EOF: a later part must not leave object accounting or ILM work.
|
||||
let (entry, replayed_metadata) = if let Some(reader) = dir_reader.as_mut() {
|
||||
let entry = match next_scanner_dir_entry(&ctx, &self.sleeper, reader).await {
|
||||
let entry = match next_scanner_dir_entry(&ctx, &self.sleeper, reader, self.node_admission.as_ref()).await {
|
||||
Ok(Some(entry)) => entry,
|
||||
Ok(None) => {
|
||||
raw_enumeration_complete = true;
|
||||
@@ -2072,7 +2082,12 @@ impl FolderScanner {
|
||||
|
||||
// Ignore entries that disappeared during traversal or hit symlink
|
||||
// loops, but propagate other walk errors.
|
||||
let mut entry_type = match wait_for_scanner_io(&ctx, entry.file_type()).await? {
|
||||
let mut entry_type = match wait_for_scanner_io(
|
||||
&ctx,
|
||||
run_scanner_node_operation(self.node_admission.as_ref(), &ctx, entry.file_type()),
|
||||
)
|
||||
.await??
|
||||
{
|
||||
Ok(entry_type) => entry_type,
|
||||
Err(e) if e.kind() == ErrorKind::NotFound => {
|
||||
debug!(
|
||||
@@ -2123,7 +2138,12 @@ impl FolderScanner {
|
||||
let mut entry_metadata_len = None;
|
||||
if entry_type.is_symlink() {
|
||||
wait_for_scanner_io(&ctx, self.sleeper.admit_disk_io(1, 0)).await?;
|
||||
let metadata = match wait_for_scanner_io(&ctx, tokio::fs::metadata(&file_path)).await? {
|
||||
let metadata = match wait_for_scanner_io(
|
||||
&ctx,
|
||||
run_scanner_node_operation(self.node_admission.as_ref(), &ctx, tokio::fs::metadata(&file_path)),
|
||||
)
|
||||
.await??
|
||||
{
|
||||
Ok(metadata) => metadata,
|
||||
Err(e) if e.kind() == ErrorKind::NotFound => {
|
||||
debug!(
|
||||
@@ -2274,22 +2294,30 @@ impl FolderScanner {
|
||||
len
|
||||
} else {
|
||||
wait_for_scanner_io(&ctx, self.sleeper.admit_disk_io(1, 0)).await?;
|
||||
wait_for_scanner_io(&ctx, tokio::fs::metadata(&item.path))
|
||||
.await?
|
||||
.map(|metadata| metadata.len())
|
||||
.unwrap_or(0)
|
||||
wait_for_scanner_io(
|
||||
&ctx,
|
||||
run_scanner_node_operation(self.node_admission.as_ref(), &ctx, tokio::fs::metadata(&item.path)),
|
||||
)
|
||||
.await??
|
||||
.map(|metadata| metadata.len())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
} else {
|
||||
0
|
||||
};
|
||||
wait_for_scanner_io(&ctx, self.sleeper.admit_disk_io(1, budgeted_bytes)).await?;
|
||||
|
||||
self.object_metadata_visits.increment(1);
|
||||
let sz = match self
|
||||
.local_disk
|
||||
.get_size_with_tier_names(item.clone(), &self.tier_registry.names)
|
||||
.await
|
||||
{
|
||||
let size_result = run_scanner_node_operation(self.node_admission.as_ref(), &ctx, async {
|
||||
self.object_metadata_visits.increment(1);
|
||||
self.local_disk
|
||||
.get_size_with_tier_names(item.clone(), &self.tier_registry.names)
|
||||
.await
|
||||
})
|
||||
.await?;
|
||||
if self.node_admission.is_some() && ctx.is_cancelled() {
|
||||
return Err(ScannerError::Other("Operation cancelled".to_string()));
|
||||
}
|
||||
let sz = match size_result {
|
||||
Ok(sz) => sz,
|
||||
Err(e) => {
|
||||
let failure_action = classify_get_size_failure(&item, &e);
|
||||
@@ -2606,7 +2634,7 @@ impl FolderScanner {
|
||||
{
|
||||
Ok(outcome) => outcome,
|
||||
Err(e) => {
|
||||
if ctx.is_cancelled() {
|
||||
if ctx.is_cancelled() || matches!(&e, ScannerError::ResourceDeferred) {
|
||||
self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst)
|
||||
.await;
|
||||
return Err(e);
|
||||
@@ -2847,19 +2875,23 @@ impl FolderScanner {
|
||||
let mut seen_truncated_objects: HashSet<String> = HashSet::new();
|
||||
|
||||
loop {
|
||||
if agreed_closed && partial_closed && finished_closed {
|
||||
if child_ctx.is_cancelled() || (agreed_closed && partial_closed && finished_closed) {
|
||||
break;
|
||||
}
|
||||
#[cfg(test)]
|
||||
let defer_entries = self.defer_abandoned_entries_until_finished && !finished_closed;
|
||||
#[cfg(not(test))]
|
||||
let defer_entries = false;
|
||||
|
||||
select! {
|
||||
entry_name = agreed_rx.recv(), if !agreed_closed => {
|
||||
entry_name = agreed_rx.recv(), if !agreed_closed && !defer_entries => {
|
||||
let Some(entry_name) = entry_name else {
|
||||
agreed_closed = true;
|
||||
continue;
|
||||
};
|
||||
(self.update_current_path)(&entry_name).await;
|
||||
}
|
||||
entries = partial_rx.recv(), if !partial_closed => {
|
||||
entries = partial_rx.recv(), if !partial_closed && !defer_entries => {
|
||||
let Some(entries) = entries else {
|
||||
partial_closed = true;
|
||||
continue;
|
||||
@@ -2985,6 +3017,7 @@ impl FolderScanner {
|
||||
finished_closed = true;
|
||||
continue;
|
||||
};
|
||||
finished_closed = true;
|
||||
if disk_errors_are_only_missing_paths(&errs) {
|
||||
debug!(
|
||||
target: "rustfs::scanner::folder",
|
||||
@@ -3006,7 +3039,9 @@ impl FolderScanner {
|
||||
"Scanner list_path_raw finished with disk errors"
|
||||
);
|
||||
}
|
||||
child_ctx.cancel();
|
||||
// The producer awaited callback sends, not their
|
||||
// consumption. Drain the remaining candidate and
|
||||
// agreed frames before closing this child walk.
|
||||
}
|
||||
_ = child_ctx.cancelled() => {
|
||||
break;
|
||||
@@ -3024,7 +3059,10 @@ impl FolderScanner {
|
||||
if into.compacted {
|
||||
// In compacted mode child totals are accumulated directly into the parent entry.
|
||||
let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), into));
|
||||
fut.await.map_err(|e| ScannerError::Other(e.to_string()))?;
|
||||
fut.await.map_err(|error| match error {
|
||||
ScannerError::ResourceDeferred => ScannerError::ResourceDeferred,
|
||||
other => ScannerError::Other(other.to_string()),
|
||||
})?;
|
||||
self.send_update_for_entry(&this_hash, &folder.parent, into).await;
|
||||
tokio::task::yield_now().await;
|
||||
} else {
|
||||
@@ -3043,7 +3081,7 @@ impl FolderScanner {
|
||||
)
|
||||
.await
|
||||
{
|
||||
if ctx.is_cancelled() {
|
||||
if ctx.is_cancelled() || matches!(&e, ScannerError::ResourceDeferred) {
|
||||
self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst)
|
||||
.await;
|
||||
return Err(e);
|
||||
@@ -3217,6 +3255,7 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
failed_object_paths_seen: HashSet::new(),
|
||||
resolved_failed_object_paths: HashSet::new(),
|
||||
sleeper,
|
||||
node_admission: crate::workload_admission::scanner_node_admission(),
|
||||
disks,
|
||||
disks_quorum,
|
||||
usage_updates,
|
||||
@@ -3244,6 +3283,8 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
pending_size_reconciliation_truncated: false,
|
||||
#[cfg(test)]
|
||||
list_path_raw_options_observer: None,
|
||||
#[cfg(test)]
|
||||
defer_abandoned_entries_until_finished: false,
|
||||
};
|
||||
|
||||
let now = FolderScanner::now_secs();
|
||||
@@ -3333,7 +3374,7 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
if ctx.is_cancelled() {
|
||||
if ctx.is_cancelled() || matches!(&e, ScannerError::ResourceDeferred) {
|
||||
let root_hash = hash_path(&bucket_name);
|
||||
let root_has_progress = data_usage_root_has_progress(&root);
|
||||
let pending_heals_changed = scanner.pending_heals_changed;
|
||||
@@ -3408,6 +3449,19 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
close_disk_guard.close().await;
|
||||
return Err(ScannerError::NamespaceNotFoundCache(Box::new(partial_cache)));
|
||||
}
|
||||
if matches!(&e, ScannerError::ResourceDeferred) {
|
||||
// Admission pressure is not disk failure or an empty census.
|
||||
// Keep the previous tree when no new useful progress exists.
|
||||
let mut partial_cache = scanner.old_cache;
|
||||
if partial_cache.cache.is_empty() {
|
||||
partial_cache
|
||||
.cache
|
||||
.insert(hash_path(&bucket_name).0, DataUsageEntry::default());
|
||||
}
|
||||
partial_cache.info.snapshot_complete = false;
|
||||
close_disk_guard.close().await;
|
||||
return Err(ScannerError::PartialCache(Box::new(partial_cache)));
|
||||
}
|
||||
close_disk_guard.close().await;
|
||||
// No useful information, return original cache
|
||||
Err(e)
|
||||
@@ -3427,3 +3481,15 @@ pub(crate) use tests::checkpoint_fixture::run_checkpoint_fixture;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
#[cfg(test)]
|
||||
tokio::task_local! {
|
||||
pub(crate) static TEST_SIZE_DISPATCH_BOUNDARIES: tokio::sync::mpsc::UnboundedSender<&'static str>;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn observe_test_size_dispatch_boundary(stage: &'static str) -> bool {
|
||||
TEST_SIZE_DISPATCH_BOUNDARIES
|
||||
.try_with(|sender| sender.send(stage).is_ok())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
@@ -628,6 +628,22 @@ impl ScannerItem {
|
||||
tier_names: &[String],
|
||||
size_summary: &mut SizeSummary,
|
||||
) {
|
||||
self.apply_actions_cancellable(object_infos, lock_retention, versioning_config, tier_names, size_summary, None)
|
||||
.await;
|
||||
}
|
||||
|
||||
pub(crate) async fn apply_actions_cancellable(
|
||||
&mut self,
|
||||
object_infos: Vec<ObjectInfo>,
|
||||
lock_retention: Option<Arc<ObjectLockConfiguration>>,
|
||||
versioning_config: VersioningConfiguration,
|
||||
tier_names: &[String],
|
||||
size_summary: &mut SizeSummary,
|
||||
cancellation: Option<&CancellationToken>,
|
||||
) {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
let object_path = self.object_path();
|
||||
if object_infos.is_empty() {
|
||||
debug!(
|
||||
@@ -671,6 +687,9 @@ impl ScannerItem {
|
||||
let Some(lifecycle) = self.lifecycle.clone() else {
|
||||
let mut cumulative_size: i64 = 0;
|
||||
for (oi, resolved_size) in object_infos.iter().zip(resolved_sizes.iter()) {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
let accounting_size = match resolved_size {
|
||||
SizeResolution::Known { logical, .. } => *logical,
|
||||
// A valid compressed legacy sentinel has no logical size,
|
||||
@@ -678,7 +697,7 @@ impl ScannerItem {
|
||||
// physical size is only an input to those operations; it
|
||||
// is not folded into the logical total below.
|
||||
SizeResolution::Unknown { physical, .. } => {
|
||||
self.heal_actions(oi, *physical, size_summary).await;
|
||||
self.heal_actions(oi, *physical, size_summary, cancellation).await;
|
||||
size_summary.actions_accounting_unknown(oi);
|
||||
continue;
|
||||
}
|
||||
@@ -688,13 +707,16 @@ impl ScannerItem {
|
||||
}
|
||||
};
|
||||
|
||||
let size = self.heal_actions(oi, accounting_size, size_summary).await;
|
||||
let size = self.heal_actions(oi, accounting_size, size_summary, cancellation).await;
|
||||
|
||||
size_summary.actions_accounting(oi, size, accounting_size);
|
||||
|
||||
cumulative_size = cumulative_size.saturating_add(size);
|
||||
}
|
||||
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
self.alert_excessive_versions(object_infos.len(), cumulative_size);
|
||||
|
||||
debug!(
|
||||
@@ -736,6 +758,10 @@ impl ScannerItem {
|
||||
}
|
||||
};
|
||||
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Every version handed to the evaluator counts as an ILM-checked
|
||||
// version, whether or not a rule matched (NoneAction included). This is
|
||||
// reached only for buckets with lifecycle rules; it feeds
|
||||
@@ -754,6 +780,9 @@ impl ScannerItem {
|
||||
.all(|candidate| Self::tier_is_known(candidate, tier_names));
|
||||
'eventLoop: {
|
||||
for (i, event) in events.iter().enumerate() {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
let oi = &object_infos[i];
|
||||
let known_size = resolved_sizes[i].known_size();
|
||||
if has_corrupt_size
|
||||
@@ -772,7 +801,7 @@ impl ScannerItem {
|
||||
// accounting. Size-filtered or deferred events remain
|
||||
// pending, so retain the version-only physical counters.
|
||||
if let SizeResolution::Unknown { physical, .. } = &resolved_sizes[i] {
|
||||
self.heal_actions(oi, *physical, size_summary).await;
|
||||
self.heal_actions(oi, *physical, size_summary, cancellation).await;
|
||||
size_summary.actions_accounting_unknown(oi);
|
||||
}
|
||||
continue;
|
||||
@@ -829,7 +858,7 @@ impl ScannerItem {
|
||||
noncurrent_unknown.push(oi);
|
||||
} else {
|
||||
if let SizeResolution::Unknown { physical, .. } = &resolved_sizes[i] {
|
||||
self.heal_actions(oi, *physical, size_summary).await;
|
||||
self.heal_actions(oi, *physical, size_summary, cancellation).await;
|
||||
}
|
||||
size_summary.actions_accounting_unknown(oi);
|
||||
}
|
||||
@@ -846,7 +875,7 @@ impl ScannerItem {
|
||||
}
|
||||
IlmAction::NoneAction | IlmAction::ActionCount => {
|
||||
if let SizeResolution::Unknown { physical, .. } = &resolved_sizes[i] {
|
||||
self.heal_actions(oi, *physical, size_summary).await;
|
||||
self.heal_actions(oi, *physical, size_summary, cancellation).await;
|
||||
}
|
||||
size_summary.actions_accounting_unknown(oi);
|
||||
}
|
||||
@@ -864,7 +893,7 @@ impl ScannerItem {
|
||||
// side-effecting tier actions fail closed until the registry
|
||||
// recognizes the source again.
|
||||
if Self::action_blocked_by_unknown_tier(event.action, oi, all_versions_known, tier_names, &event.storage_class) {
|
||||
size = self.heal_actions(oi, actual_size, size_summary).await;
|
||||
size = self.heal_actions(oi, actual_size, size_summary, cancellation).await;
|
||||
size_summary.actions_accounting(oi, size, actual_size);
|
||||
cumulative_size += size;
|
||||
continue;
|
||||
@@ -973,7 +1002,7 @@ impl ScannerItem {
|
||||
}
|
||||
|
||||
IlmAction::NoneAction | IlmAction::ActionCount => {
|
||||
size = self.heal_actions(oi, actual_size, size_summary).await;
|
||||
size = self.heal_actions(oi, actual_size, size_summary, cancellation).await;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -984,6 +1013,9 @@ impl ScannerItem {
|
||||
}
|
||||
}
|
||||
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return;
|
||||
}
|
||||
if !to_delete_objs.is_empty()
|
||||
&& let Some(event) = noncurrent_events.first().cloned()
|
||||
{
|
||||
@@ -991,7 +1023,8 @@ impl ScannerItem {
|
||||
let count = u64::try_from(to_delete_objs.len()).unwrap_or(u64::MAX);
|
||||
let done_ilm = Metrics::time_ilm(action);
|
||||
let trace_started_at = trace_start_instant();
|
||||
let queued = enqueue_runtime_newer_noncurrent(&self.bucket, to_delete_objs, event, &LcEventSrc::Scanner).await;
|
||||
let queued =
|
||||
enqueue_runtime_newer_noncurrent(&self.bucket, to_delete_objs, event, &LcEventSrc::Scanner, cancellation).await;
|
||||
if let Some(trace_started_at) = trace_started_at {
|
||||
let state = if queued { "queued" } else { "not_queued" };
|
||||
trace_emit(|| {
|
||||
@@ -1022,15 +1055,28 @@ impl ScannerItem {
|
||||
}
|
||||
}
|
||||
}
|
||||
self.alert_excessive_versions(remaining_versions, cumulative_size);
|
||||
if !cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
self.alert_excessive_versions(remaining_versions, cumulative_size);
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) async fn heal_actions(&mut self, oi: &ObjectInfo, actual_size: i64, size_summary: &mut SizeSummary) -> i64 {
|
||||
pub(super) async fn heal_actions(
|
||||
&mut self,
|
||||
oi: &ObjectInfo,
|
||||
actual_size: i64,
|
||||
size_summary: &mut SizeSummary,
|
||||
cancellation: Option<&CancellationToken>,
|
||||
) -> i64 {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return actual_size;
|
||||
}
|
||||
if self.heal_enabled {
|
||||
self.enqueue_heal(oi).await;
|
||||
}
|
||||
|
||||
self.heal_replication(oi, size_summary).await;
|
||||
if !cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
self.heal_replication(oi, size_summary).await;
|
||||
}
|
||||
|
||||
actual_size
|
||||
}
|
||||
|
||||
@@ -30,7 +30,9 @@ use std::sync::Mutex;
|
||||
pub(super) mod checkpoint_fixture;
|
||||
pub(super) mod enumeration_restart;
|
||||
mod incremental_enumeration;
|
||||
mod listing_completion;
|
||||
mod metadata_visits;
|
||||
mod node_admission;
|
||||
mod usage_updates;
|
||||
|
||||
/// Reset the process-global alert cooldown map; test-only.
|
||||
@@ -364,6 +366,7 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
|
||||
failed_object_paths_seen: HashSet::new(),
|
||||
resolved_failed_object_paths: HashSet::new(),
|
||||
sleeper: SCANNER_SLEEPER.clone(),
|
||||
node_admission: None,
|
||||
disks: Vec::new(),
|
||||
disks_quorum: 0,
|
||||
usage_updates: None,
|
||||
@@ -394,6 +397,7 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
|
||||
pending_size_reconciliation_scopes: HashSet::new(),
|
||||
pending_size_reconciliation_truncated: false,
|
||||
list_path_raw_options_observer: None,
|
||||
defer_abandoned_entries_until_finished: false,
|
||||
};
|
||||
|
||||
(scanner, temp_dir)
|
||||
@@ -2262,7 +2266,7 @@ async fn test_heal_actions_returns_actual_size_without_inline_heal() {
|
||||
};
|
||||
let mut size_summary = SizeSummary::default();
|
||||
|
||||
let size = item.heal_actions(&object_info, 123, &mut size_summary).await;
|
||||
let size = item.heal_actions(&object_info, 123, &mut size_summary, None).await;
|
||||
assert_eq!(size, 123);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
// Licensed under the Apache License, Version 2.0.
|
||||
|
||||
use super::*;
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn abandoned_listing_drains_candidate_buffered_before_finished() {
|
||||
let (mut scanner, root) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
temp_dir: Some(root.clone()),
|
||||
};
|
||||
let bucket = "bucket";
|
||||
let object = "abandoned/object";
|
||||
tokio::fs::create_dir_all(root.join(bucket)).await.expect("local bucket");
|
||||
let mut disks = vec![scanner.local_disk.clone()];
|
||||
let mut original_metadata = Vec::new();
|
||||
for (index, version) in [(2, 0x1935), (3, 0x1935), (4, 0x1934)] {
|
||||
let disk_root = root.join(format!("disk-{index}"));
|
||||
let metadata = metadata_for_object_version(bucket, object, Some(Uuid::from_u128(version)));
|
||||
write_test_object_metadata_bytes(&disk_root, bucket, object, &metadata).await;
|
||||
original_metadata.push((disk_root.join(bucket).join(object).join(STORAGE_FORMAT_FILE), metadata));
|
||||
let endpoint = Endpoint::try_from(disk_root.to_string_lossy().as_ref()).expect("listing fixture endpoint");
|
||||
disks.push(
|
||||
new_disk(
|
||||
&endpoint,
|
||||
&DiskOption {
|
||||
cleanup: false,
|
||||
health_check: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("listing fixture disk"),
|
||||
);
|
||||
}
|
||||
for cache in [&mut scanner.old_cache, &mut scanner.new_cache] {
|
||||
cache.info.name = bucket.to_owned();
|
||||
}
|
||||
scanner.old_cache.replace(
|
||||
"bucket/abandoned/object",
|
||||
bucket,
|
||||
DataUsageEntry {
|
||||
objects: 1,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
scanner.heal_object_select = 1;
|
||||
scanner.disks = disks;
|
||||
scanner.disks_quorum = 2;
|
||||
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest);
|
||||
// Force the valid select ordering that receives terminal errors before the
|
||||
// final candidate. One candidate frame fits in the bounded partial channel.
|
||||
scanner.defer_abandoned_entries_until_finished = true;
|
||||
let (_commands, mut heal_rx) = rustfs_heal_contracts::heal_channel::init_heal_channels().expect("isolated heal channel");
|
||||
let (captured_tx, mut captured_rx) = mpsc::unbounded_channel();
|
||||
let responder = tokio::spawn(async move {
|
||||
while let Some(command) = heal_rx.recv().await {
|
||||
let rustfs_heal_contracts::heal_channel::HealReceiptCommand { request, response_tx } = command;
|
||||
captured_tx
|
||||
.send((request.object_prefix.clone(), request.object_version_id.clone()))
|
||||
.expect("candidate capture remains open");
|
||||
let _ = response_tx.send(Ok(rustfs_heal_contracts::heal_channel::HealAdmissionReceipt {
|
||||
result: HealAdmissionResult::Accepted,
|
||||
task_id: request.id,
|
||||
}));
|
||||
}
|
||||
});
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(2),
|
||||
scanner.scan_folder(
|
||||
CancellationToken::new(),
|
||||
CachedFolder {
|
||||
name: bucket.to_owned(),
|
||||
parent: None,
|
||||
object_heal_prob_div: 1,
|
||||
},
|
||||
&mut DataUsageEntry::default(),
|
||||
),
|
||||
)
|
||||
.await;
|
||||
responder.abort();
|
||||
result
|
||||
.expect("listing completion must close every callback channel")
|
||||
.expect("complete folder scan");
|
||||
let mut candidates = HashSet::new();
|
||||
while let Ok((object, version)) = captured_rx.try_recv() {
|
||||
if let (Some(object), Some(version)) = (object, version) {
|
||||
candidates.insert((object, version));
|
||||
}
|
||||
}
|
||||
let expected = [0x1934, 0x1935]
|
||||
.into_iter()
|
||||
// Raw base-object listings include the trailing separator. Preserve
|
||||
// the producer's candidate identity when draining its callback frame.
|
||||
.map(|version| (format!("{object}/"), Uuid::from_u128(version).to_string()))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
candidates, expected,
|
||||
"terminal disk errors must not discard buffered exact-version candidates"
|
||||
);
|
||||
for (path, metadata) in original_metadata {
|
||||
assert_eq!(tokio::fs::read(path).await.expect("retained metadata"), metadata);
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -172,12 +172,55 @@ fn record_checkpoint_save_failure(cache_name: &str, checkpoint: &DataUsageCache,
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct ScannerCacheLockGuards {
|
||||
scoped: NamespaceLockGuard,
|
||||
scoped: Option<NamespaceLockGuard>,
|
||||
}
|
||||
|
||||
impl ScannerCacheLockGuards {
|
||||
pub(crate) fn is_lock_lost(&self) -> bool {
|
||||
self.scoped.is_lock_lost()
|
||||
self.scoped.as_ref().is_none_or(NamespaceLockGuard::is_lock_lost)
|
||||
}
|
||||
|
||||
/// Suspend the same traversal while yielding its cache lease. Lock order on
|
||||
/// resumption is node admission -> cache lease -> cache revision reads.
|
||||
pub(crate) async fn resume_node_operation(
|
||||
&mut self,
|
||||
store: Arc<SetDisks>,
|
||||
cache_name: &str,
|
||||
source: DataUsageCacheSource,
|
||||
revisions: &DataUsageCacheRevisions,
|
||||
request: crate::workload_admission::ScannerNodeRequest,
|
||||
) -> Result<()> {
|
||||
if self.is_lock_lost() {
|
||||
return Err(Error::other("scanner cache lock was lost before node admission"));
|
||||
}
|
||||
if request.respond.is_closed() {
|
||||
return Ok(());
|
||||
}
|
||||
let crate::workload_admission::ScannerNodeRequest { owner, respond, cancel } = request;
|
||||
drop(self.scoped.take());
|
||||
let resume = async {
|
||||
let permit = owner.acquire().await;
|
||||
*self = acquire_scanner_cache_locks(store.as_ref(), cache_name, source)
|
||||
.await
|
||||
.map_err(Error::other)?;
|
||||
let current = crate::storage_api::ecstore_with_background_disk_io(
|
||||
cancel.clone(),
|
||||
DataUsageCache::read_revisions(store, cache_name),
|
||||
)
|
||||
.await?;
|
||||
if self.is_lock_lost() || ¤t != revisions {
|
||||
return Err(Error::other("scanner cache changed while waiting for node admission"));
|
||||
}
|
||||
// If the walk was dropped, send returns ownership and drops the
|
||||
// permit without recording completed work.
|
||||
let _ = respond.send(permit);
|
||||
Ok(())
|
||||
};
|
||||
tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => Err(Error::other("scanner node admission cancelled")),
|
||||
result = resume => result,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -215,6 +258,15 @@ impl std::fmt::Display for ScannerCacheLockError {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for ScannerCacheLockError {
|
||||
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
|
||||
match self {
|
||||
Self::Create { source, .. } => Some(source),
|
||||
Self::Acquire { source, .. } => Some(source),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn acquire_scanner_cache_locks(
|
||||
store: &SetDisks,
|
||||
cache_name: &str,
|
||||
@@ -237,7 +289,7 @@ pub(crate) async fn acquire_scanner_cache_locks(
|
||||
source,
|
||||
})?;
|
||||
|
||||
Ok(ScannerCacheLockGuards { scoped })
|
||||
Ok(ScannerCacheLockGuards { scoped: Some(scoped) })
|
||||
}
|
||||
|
||||
pub(super) async fn await_scanner_disk_shutdown<F>(scan: Pin<&mut F>)
|
||||
@@ -688,6 +740,15 @@ pub(super) fn completed_data_usage_info(
|
||||
.map(|candidate| (candidate.data, candidate.last_update))
|
||||
}
|
||||
|
||||
/// Certify an incarnation only when every contributing set reports the same non-nil identity.
|
||||
fn certified_bucket_incarnation(results: &[DataUsageCache], bucket: &str) -> Option<uuid::Uuid> {
|
||||
let mut identities = results
|
||||
.iter()
|
||||
.map(|result| result.info.scan_bucket_incarnations.get(bucket).copied());
|
||||
let first = identities.next().flatten().filter(|id| !id.is_nil())?;
|
||||
identities.all(|id| id == Some(first)).then_some(first)
|
||||
}
|
||||
|
||||
pub(super) fn completed_usage_candidate(
|
||||
results: &[DataUsageCache],
|
||||
scope: &ScannerSnapshotScope<'_>,
|
||||
@@ -740,7 +801,11 @@ pub(super) fn completed_usage_candidate(
|
||||
let merged_last_update = validated.last_update;
|
||||
let buckets_usage = bucket_entries
|
||||
.into_iter()
|
||||
.map(|(bucket, entry)| Ok((bucket, checked_bucket_usage_info(entry).ok_or(CandidateOutcome::UsageCounters)?)))
|
||||
.map(|(bucket, entry)| {
|
||||
let mut usage = checked_bucket_usage_info(entry).ok_or(CandidateOutcome::UsageCounters)?;
|
||||
usage.bucket_incarnation = certified_bucket_incarnation(results, &bucket).map(|id| id.to_string());
|
||||
Ok((bucket, usage))
|
||||
})
|
||||
.collect::<std::result::Result<HashMap<_, _>, CandidateOutcome>>()?;
|
||||
let bucket_sizes = buckets_usage
|
||||
.iter()
|
||||
|
||||
@@ -238,6 +238,71 @@ pub enum ScannerDurableDirtyUsageReplayScope {
|
||||
TopLevelEntries { entries: BTreeSet<String> },
|
||||
}
|
||||
|
||||
impl ScannerDurableDirtyUsageReplayRecord {
|
||||
/// Replays decoded producer hints with the canonical byte limit and semantic checks.
|
||||
///
|
||||
/// Callers must retain their framing and owner checks. This performs the same
|
||||
/// state update as [`replay_durable_dirty_usage_producer_record`]; it does not
|
||||
/// establish storage-verified source authority.
|
||||
pub fn replay(self) -> std::result::Result<ScannerDirtyUsageState, ScannerDurableDirtyUsageReplayError> {
|
||||
check_durable_dirty_usage_replay_bytes(&self)?;
|
||||
replay_decoded_dirty_usage_record(&self)
|
||||
}
|
||||
}
|
||||
|
||||
struct ReplayByteLimit {
|
||||
remaining: usize,
|
||||
}
|
||||
|
||||
impl std::io::Write for ReplayByteLimit {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.remaining = self
|
||||
.remaining
|
||||
.checked_sub(bytes.len())
|
||||
.ok_or_else(|| std::io::Error::other("dirty replay byte limit"))?;
|
||||
Ok(bytes.len())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn check_durable_dirty_usage_replay_bytes(
|
||||
record: &ScannerDurableDirtyUsageReplayRecord,
|
||||
) -> std::result::Result<(), ScannerDurableDirtyUsageReplayError> {
|
||||
// Bound structure traversal and string scanning before counting escaped JSON.
|
||||
// Each entry or prefix contributes at least one additional serialized byte.
|
||||
let mut remaining = SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES;
|
||||
let mut charge = |size: usize| -> std::result::Result<(), ScannerDurableDirtyUsageReplayError> {
|
||||
remaining = remaining
|
||||
.checked_sub(size)
|
||||
.ok_or(ScannerDurableDirtyUsageReplayError::ByteLimit)?;
|
||||
Ok(())
|
||||
};
|
||||
charge(record.writer_epoch.len())?;
|
||||
charge(record.entries.len())?;
|
||||
for entry in &record.entries {
|
||||
charge(entry.bucket.len())?;
|
||||
if let ScannerDurableDirtyUsageReplayScope::TopLevelEntries { entries } = &entry.scope {
|
||||
charge(entries.len())?;
|
||||
for entry in entries {
|
||||
charge(entry.len())?;
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut limit = ReplayByteLimit {
|
||||
remaining: SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES,
|
||||
};
|
||||
serde_json::to_writer(&mut limit, record).map_err(|error| {
|
||||
if error.is_io() {
|
||||
ScannerDurableDirtyUsageReplayError::ByteLimit
|
||||
} else {
|
||||
ScannerDurableDirtyUsageReplayError::InvalidJson
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
pub fn encode_durable_dirty_usage_producer_replay_record(
|
||||
entries: Vec<ScannerDurableDirtyUsageReplayEntry>,
|
||||
) -> std::result::Result<Vec<u8>, ScannerDurableDirtyUsageReplayError> {
|
||||
@@ -263,7 +328,13 @@ pub fn replay_durable_dirty_usage_producer_record(
|
||||
}
|
||||
let record: ScannerDurableDirtyUsageReplayRecord =
|
||||
serde_json::from_slice(bytes).map_err(|_| ScannerDurableDirtyUsageReplayError::InvalidJson)?;
|
||||
let replay = validate_durable_dirty_usage_replay_record(&record)?;
|
||||
replay_decoded_dirty_usage_record(&record)
|
||||
}
|
||||
|
||||
fn replay_decoded_dirty_usage_record(
|
||||
record: &ScannerDurableDirtyUsageReplayRecord,
|
||||
) -> std::result::Result<ScannerDirtyUsageState, ScannerDurableDirtyUsageReplayError> {
|
||||
let replay = validate_durable_dirty_usage_replay_record(record)?;
|
||||
let max_generation = replay.iter().map(|entry| entry.generation).max().unwrap_or(0);
|
||||
set_dirty_usage_generation_floor(max_generation)?;
|
||||
let pending = {
|
||||
@@ -772,6 +843,109 @@ mod scoped_dirty_usage_tests {
|
||||
);
|
||||
}
|
||||
|
||||
fn decoded_replay_fixture() -> ScannerDurableDirtyUsageReplayRecord {
|
||||
ScannerDurableDirtyUsageReplayRecord {
|
||||
schema: SCANNER_DURABLE_DIRTY_USAGE_REPLAY_SCHEMA,
|
||||
cache_key_format: crate::DATA_USAGE_CACHE_KEY_FORMAT,
|
||||
writer_epoch: "writer".to_string(),
|
||||
entries: vec![ScannerDurableDirtyUsageReplayEntry {
|
||||
bucket: "photos".to_string(),
|
||||
generation: 7,
|
||||
scope: ScannerDurableDirtyUsageReplayScope::TopLevelEntries {
|
||||
entries: BTreeSet::from(["2026".to_string(), "archive".to_string()]),
|
||||
},
|
||||
producers: SegmentInvalidationProducerIdentity::REQUIRED_PRODUCTION.into_iter().collect(),
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn durable_dirty_usage_decoded_replay_matches_byte_replay_at_size_boundary() {
|
||||
let record = decoded_replay_fixture();
|
||||
let fixed_bytes = serde_json::to_vec(&record).expect("fixture encoding").len() - record.writer_epoch.len();
|
||||
for size in [
|
||||
SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES - 1,
|
||||
SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES,
|
||||
SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES + 1,
|
||||
] {
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
let mut record = record.clone();
|
||||
record.writer_epoch = "x".repeat(size - fixed_bytes);
|
||||
let bytes = serde_json::to_vec(&record).expect("bounded fixture encoding");
|
||||
assert_eq!(bytes.len(), size);
|
||||
let expected = replay_durable_dirty_usage_producer_record(&bytes);
|
||||
let buckets = dirty_usage_buckets_for_tests();
|
||||
let scopes = dirty_usage_bucket_scopes_for_tests();
|
||||
let identities = dirty_usage_producer_identities().clone();
|
||||
let coverage = DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire);
|
||||
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
assert_eq!(record.replay(), expected, "canonical size {size}");
|
||||
assert_eq!(dirty_usage_buckets_for_tests(), buckets);
|
||||
assert_eq!(dirty_usage_bucket_scopes_for_tests(), scopes);
|
||||
assert_eq!(*dirty_usage_producer_identities(), identities);
|
||||
assert_eq!(DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire), coverage);
|
||||
}
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn durable_dirty_usage_decoded_size_counts_escaped_utf8() {
|
||||
for repeats in [6000, 8000] {
|
||||
let mut record = decoded_replay_fixture();
|
||||
record.writer_epoch = "\"\n雪\\".repeat(repeats);
|
||||
let bytes = serde_json::to_vec(&record).expect("escaped fixture encoding");
|
||||
let expected = if bytes.len() > SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES {
|
||||
Err(ScannerDurableDirtyUsageReplayError::ByteLimit)
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
assert_eq!(check_durable_dirty_usage_replay_bytes(&record), expected);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn durable_dirty_usage_decoded_replay_rejects_large_values_before_state_changes() {
|
||||
for field in ["epoch", "prefix", "entries"] {
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
let before = scanner_dirty_usage_state();
|
||||
let mut record = decoded_replay_fixture();
|
||||
match field {
|
||||
"epoch" => record.writer_epoch = "x".repeat(1024 * 1024),
|
||||
"prefix" => {
|
||||
record.entries[0].scope = ScannerDurableDirtyUsageReplayScope::TopLevelEntries {
|
||||
entries: BTreeSet::from(["x".repeat(1024 * 1024)]),
|
||||
};
|
||||
}
|
||||
_ => record.entries = vec![record.entries[0].clone(); SCANNER_DURABLE_DIRTY_USAGE_REPLAY_MAX_BYTES + 1],
|
||||
}
|
||||
assert_eq!(record.replay(), Err(ScannerDurableDirtyUsageReplayError::ByteLimit), "{field}");
|
||||
assert_eq!(scanner_dirty_usage_state(), before);
|
||||
assert!(dirty_usage_buckets_for_tests().is_empty());
|
||||
assert!(dirty_usage_bucket_scopes_for_tests().is_empty());
|
||||
assert!(dirty_usage_producer_identities().is_empty());
|
||||
assert_eq!(DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire), 0);
|
||||
}
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn durable_dirty_usage_decoded_replay_preserves_generation_exhaustion() {
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
let generation = DIRTY_USAGE_BUCKET_GENERATION.swap(u64::MAX, Ordering::AcqRel);
|
||||
let result = decoded_replay_fixture().replay();
|
||||
DIRTY_USAGE_BUCKET_GENERATION.store(generation, Ordering::Release);
|
||||
assert_eq!(result, Err(ScannerDurableDirtyUsageReplayError::InvalidGeneration));
|
||||
assert!(dirty_usage_buckets_for_tests().is_empty());
|
||||
assert!(dirty_usage_bucket_scopes_for_tests().is_empty());
|
||||
assert!(dirty_usage_producer_identities().is_empty());
|
||||
assert_eq!(DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire), 0);
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn durable_dirty_usage_replay_rejects_invalid_records_without_partial_state() {
|
||||
@@ -843,9 +1017,17 @@ mod scoped_dirty_usage_tests {
|
||||
|
||||
for (bytes, expected_error) in invalid_cases {
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
assert_eq!(replay_durable_dirty_usage_producer_record(&bytes), Err(expected_error));
|
||||
let before = scanner_dirty_usage_state();
|
||||
assert_eq!(replay_durable_dirty_usage_producer_record(&bytes), Err(expected_error.clone()));
|
||||
assert!(dirty_usage_buckets_for_tests().is_empty());
|
||||
assert!(dirty_usage_bucket_scopes_for_tests().is_empty());
|
||||
let record: ScannerDurableDirtyUsageReplayRecord = serde_json::from_slice(&bytes).expect("typed invalid record");
|
||||
assert_eq!(record.replay(), Err(expected_error));
|
||||
assert_eq!(scanner_dirty_usage_state(), before);
|
||||
assert!(dirty_usage_buckets_for_tests().is_empty());
|
||||
assert!(dirty_usage_bucket_scopes_for_tests().is_empty());
|
||||
assert!(dirty_usage_producer_identities().is_empty());
|
||||
assert_eq!(DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire), 0);
|
||||
}
|
||||
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
|
||||
@@ -1090,7 +1090,7 @@ impl ScannerIOCache for SetDisks {
|
||||
|
||||
// Lock order: scanner leader fence -> set-scoped per-bucket cache lock ->
|
||||
// cache object read/write.
|
||||
let cache_guard = match acquire_scanner_cache_locks(store_clone_clone.as_ref(), &cache_name, source).await {
|
||||
let mut cache_guard = match acquire_scanner_cache_locks(store_clone_clone.as_ref(), &cache_name, source).await {
|
||||
Ok(guard) => guard,
|
||||
Err(e) => {
|
||||
if e.is_contention() {
|
||||
@@ -1320,9 +1320,10 @@ impl ScannerIOCache for SetDisks {
|
||||
);
|
||||
|
||||
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel::<DataUsageCache>(1);
|
||||
let (node_tx, mut node_rx) = crate::workload_admission::scanner_node_request_channel();
|
||||
|
||||
let scan_ctx = ctx_clone.child_token();
|
||||
let scan = disk.clone().nsscanner_disk(
|
||||
let scan = crate::workload_admission::with_scanner_node_requests(node_tx, disk.clone().nsscanner_disk(
|
||||
scan_ctx.clone(),
|
||||
budget_clone.clone(),
|
||||
set_disk_inventory_clone.as_ref().clone(),
|
||||
@@ -1333,7 +1334,7 @@ impl ScannerIOCache for SetDisks {
|
||||
prefix_scan_scope,
|
||||
checkpoint_tx: Some(checkpoint_tx),
|
||||
},
|
||||
);
|
||||
));
|
||||
tokio::pin!(scan);
|
||||
let mut lock_watch = tokio::time::interval(SCANNER_CACHE_LOCK_POLL_INTERVAL);
|
||||
lock_watch.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
|
||||
@@ -1341,6 +1342,24 @@ impl ScannerIOCache for SetDisks {
|
||||
let scan_result = loop {
|
||||
tokio::select! {
|
||||
result = &mut scan => break result,
|
||||
request = async {
|
||||
match node_rx.as_mut() {
|
||||
Some(receiver) => receiver.recv().await,
|
||||
None => std::future::pending().await,
|
||||
}
|
||||
} => {
|
||||
let Some(request) = request else {
|
||||
node_rx = None;
|
||||
continue;
|
||||
};
|
||||
if let Err(error) = cache_guard.resume_node_operation(
|
||||
store_clone_clone.clone(), &cache_name, source, &revisions, request,
|
||||
).await {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(error);
|
||||
}
|
||||
}
|
||||
_ = lock_watch.tick() => {
|
||||
if cache_guard.is_lock_lost() {
|
||||
scan_ctx.cancel();
|
||||
|
||||
@@ -38,6 +38,13 @@ pub(super) fn tier_stats_template(tier_names: &[String]) -> HashMap<String, Tier
|
||||
tier_stats
|
||||
}
|
||||
|
||||
fn ensure_scanner_node_operation_active(cancellation: Option<&CancellationToken>) -> Result<()> {
|
||||
if cancellation.is_some_and(CancellationToken::is_cancelled) {
|
||||
return Err(StorageError::other("Scanner operation cancelled"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl ScannerIODisk for Disk {
|
||||
async fn get_size(&self, item: ScannerItem) -> Result<SizeSummary> {
|
||||
@@ -45,14 +52,18 @@ impl ScannerIODisk for Disk {
|
||||
}
|
||||
|
||||
async fn get_size_with_tier_names(&self, mut item: ScannerItem, tier_names: &[String]) -> Result<SizeSummary> {
|
||||
let cancellation = crate::workload_admission::scanner_node_operation_cancellation();
|
||||
let done_object = Metrics::time(Metric::ScanObject);
|
||||
|
||||
if !is_xl_meta_path(&item.path) {
|
||||
return Err(StorageError::other(SCANNER_SKIP_FILE_ERROR.to_string()));
|
||||
}
|
||||
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let metadata_object_path = item.object_path();
|
||||
let data = match self.read_metadata(&item.bucket, &metadata_object_path).await {
|
||||
let metadata_result = self.read_metadata(&item.bucket, &metadata_object_path).await;
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let data = match metadata_result {
|
||||
Ok(data) => data,
|
||||
Err(e) if DiskError::is_err_object_not_found(&e) || DiskError::is_err_version_not_found(&e) => {
|
||||
return Err(StorageError::other(SCANNER_SKIP_FILE_ERROR.to_string()));
|
||||
@@ -86,7 +97,10 @@ impl ScannerIODisk for Disk {
|
||||
// (which used to query it a second time). On failure keep the
|
||||
// historical fallback: default configuration (versioned = false) plus
|
||||
// the warn that `apply_actions` used to emit.
|
||||
let versioning_config = match BucketVersioningSys::get(&item.bucket).await {
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let versioning_result = BucketVersioningSys::get(&item.bucket).await;
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let versioning_config = match versioning_result {
|
||||
Ok(versioning_config) => versioning_config,
|
||||
Err(_) => {
|
||||
warn!(
|
||||
@@ -122,7 +136,9 @@ impl ScannerIODisk for Disk {
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let lock_config = object_lock_config_for_scanner_item(&item).await;
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
|
||||
// Count every version this object contributes to the scan, independent
|
||||
// of any lifecycle configuration, so scan-coverage metrics stay honest
|
||||
@@ -130,17 +146,33 @@ impl ScannerIODisk for Disk {
|
||||
// `object_infos`.
|
||||
global_metrics().record_scanner_versions_scanned(object_infos.len() as u64);
|
||||
|
||||
item.apply_actions(object_infos, lock_config, versioning_config, tier_names, &mut size_summary)
|
||||
.await;
|
||||
#[cfg(test)]
|
||||
let _ = crate::scanner_folder::observe_test_size_dispatch_boundary("apply_actions");
|
||||
item.apply_actions_cancellable(
|
||||
object_infos,
|
||||
lock_config,
|
||||
versioning_config,
|
||||
tier_names,
|
||||
&mut size_summary,
|
||||
cancellation.as_ref(),
|
||||
)
|
||||
.await;
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
|
||||
if !free_version_infos.is_empty() {
|
||||
for oi in free_version_infos {
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
if ScannerItem::tier_is_known(&oi, tier_names) {
|
||||
enqueue_runtime_free_version(oi).await;
|
||||
#[cfg(test)]
|
||||
if crate::scanner_folder::observe_test_size_dispatch_boundary("free_version_dispatch") {
|
||||
continue;
|
||||
}
|
||||
enqueue_runtime_free_version(oi, cancellation.as_ref()).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
done_object();
|
||||
|
||||
Ok(size_summary)
|
||||
|
||||
@@ -996,6 +996,73 @@ fn completed_data_usage_info_adds_same_bucket_across_unique_sets() {
|
||||
assert_eq!(bucket.delete_markers_count, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_usage_incarnation_is_optional_and_old_readers_ignore_it() {
|
||||
#[derive(serde::Deserialize)]
|
||||
struct LegacyBucketUsage {
|
||||
size: u64,
|
||||
}
|
||||
let legacy = BucketUsageInfo {
|
||||
size: 42,
|
||||
..Default::default()
|
||||
};
|
||||
let without = serde_json::to_value(&legacy).expect("encode legacy usage");
|
||||
assert!(without.get("bucket_incarnation").is_none());
|
||||
let restored: BucketUsageInfo = serde_json::from_value(without).expect("decode old usage");
|
||||
assert_eq!(restored.bucket_incarnation, None);
|
||||
let certified = BucketUsageInfo {
|
||||
size: 42,
|
||||
bucket_incarnation: Some(uuid::Uuid::from_u128(7).to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let encoded = serde_json::to_vec(&certified).expect("encode certified usage");
|
||||
let old: LegacyBucketUsage = serde_json::from_slice(&encoded).expect("old schema ignores additive field");
|
||||
assert_eq!(old.size, 42);
|
||||
let named = rmp_serde::to_vec_named(&certified).expect("encode certified named msgpack");
|
||||
let old: LegacyBucketUsage = rmp_serde::from_slice(&named).expect("old named reader ignores additive field");
|
||||
assert_eq!(old.size, 42);
|
||||
let old_bytes = rmp_serde::to_vec_named(&legacy).expect("encode old shape");
|
||||
let new: BucketUsageInfo = rmp_serde::from_slice(&old_bytes).expect("new reader accepts old shape");
|
||||
assert_eq!(new.bucket_incarnation, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn completed_usage_certifies_only_matching_non_nil_bucket_incarnations() {
|
||||
let buckets = vec!["bucket".to_string()];
|
||||
let id = uuid::Uuid::from_u128(7);
|
||||
let mut a = completed_root_cache("bucket", 2, 10, DataUsageCacheSource::new(0, 0));
|
||||
let mut b = completed_root_cache("bucket", 3, 20, DataUsageCacheSource::new(1, 0));
|
||||
a.info.scan_bucket_incarnations.insert("bucket".to_string(), id);
|
||||
b.info.scan_bucket_incarnations.insert("bucket".to_string(), id);
|
||||
let published = completed_data_usage_info_for_test(&[a.clone(), b.clone()], &buckets, false, false)
|
||||
.unwrap()
|
||||
.0;
|
||||
assert_eq!(
|
||||
published.buckets_usage["bucket"].bucket_incarnation.as_deref(),
|
||||
Some(id.to_string().as_str())
|
||||
);
|
||||
let encoded = serde_json::to_vec(&published).expect("serialize certified usage");
|
||||
let decoded: DataUsageInfo = serde_json::from_slice(&encoded).expect("read certified usage");
|
||||
assert_eq!(decoded.buckets_usage["bucket"].bucket_incarnation, Some(id.to_string()));
|
||||
b.info
|
||||
.scan_bucket_incarnations
|
||||
.insert("bucket".to_string(), uuid::Uuid::from_u128(8));
|
||||
let mismatch = completed_data_usage_info_for_test(&[a.clone(), b.clone()], &buckets, false, false)
|
||||
.unwrap()
|
||||
.0;
|
||||
assert_eq!(mismatch.buckets_usage["bucket"].bucket_incarnation, None);
|
||||
b.info.scan_bucket_incarnations.remove("bucket");
|
||||
let missing = completed_data_usage_info_for_test(&[a.clone(), b.clone()], &buckets, false, false)
|
||||
.unwrap()
|
||||
.0;
|
||||
assert_eq!(missing.buckets_usage["bucket"].bucket_incarnation, None);
|
||||
b.info
|
||||
.scan_bucket_incarnations
|
||||
.insert("bucket".to_string(), uuid::Uuid::nil());
|
||||
let nil = completed_data_usage_info_for_test(&[a, b], &buckets, false, false).unwrap().0;
|
||||
assert_eq!(nil.buckets_usage["bucket"].bucket_incarnation, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checked_bucket_usage_info_moves_replication_keys_and_preserves_the_snapshot() {
|
||||
let first_target = "arn:rustfs:replication:local::target-a".to_string();
|
||||
@@ -1057,6 +1124,7 @@ fn checked_bucket_usage_info_moves_replication_keys_and_preserves_the_snapshot()
|
||||
}
|
||||
let expected = BucketUsageInfo {
|
||||
size: 32768,
|
||||
bucket_incarnation: None,
|
||||
replication_pending_size_v1: 0,
|
||||
replication_failed_size_v1: 0,
|
||||
replicated_size_v1: 0,
|
||||
|
||||
@@ -46,7 +46,9 @@ use uuid::Uuid;
|
||||
|
||||
mod checkpoint_coverage_identity;
|
||||
mod collector_lifecycle;
|
||||
mod expiry_lock;
|
||||
mod listing_window_scope;
|
||||
mod node_admission;
|
||||
mod partial_bucket_carry;
|
||||
mod per_bucket_certification;
|
||||
mod scoped_entry_fallback;
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::*;
|
||||
use crate::storage_api::owner::EcstoreExpiryState;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::task::Poll;
|
||||
use std::time::Duration;
|
||||
|
||||
// Return Some(result) if the operation ended too early, None only if it was
|
||||
// actually polled pending after the prerequisite read had finished.
|
||||
async fn observe_queued<F: Future>(
|
||||
mut operation: Pin<&mut F>,
|
||||
read_complete: impl Fn() -> bool,
|
||||
observation: Duration,
|
||||
) -> std::result::Result<Option<F::Output>, tokio::time::error::Elapsed> {
|
||||
tokio::time::timeout(
|
||||
observation,
|
||||
std::future::poll_fn(|cx| match operation.as_mut().poll(cx) {
|
||||
Poll::Ready(value) => Poll::Ready(Some(value)),
|
||||
Poll::Pending if read_complete() => Poll::Ready(None),
|
||||
Poll::Pending => Poll::Pending,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn free_version_cancelled_while_waiting_for_expiry_write_lock_never_queues() {
|
||||
let state = EcstoreExpiryState::new_with_unconsumed_worker_channel(4);
|
||||
let observation = Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS);
|
||||
let context = crate::ExpiryLockTestContext {
|
||||
state: state.clone(),
|
||||
store: None,
|
||||
completed_incarnation_reads: Arc::new(AtomicUsize::new(0)),
|
||||
};
|
||||
crate::EXPIRY_LOCK_TEST_CONTEXT
|
||||
.scope(context, async {
|
||||
let object = crate::ScannerObjectInfo {
|
||||
bucket: "expiry-lock-bucket".to_string(),
|
||||
name: "free-object".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
let cancellation = CancellationToken::new();
|
||||
let guard = state.write().await;
|
||||
let mut pending = Box::pin(crate::enqueue_runtime_free_version(object.clone(), Some(&cancellation)));
|
||||
let first = observe_queued(pending.as_mut(), || true, observation).await;
|
||||
let observed_wait = matches!(&first, Ok(None));
|
||||
let already_finished = matches!(&first, Ok(Some(())));
|
||||
cancellation.cancel();
|
||||
let cancelled_while_locked = if already_finished {
|
||||
false
|
||||
} else {
|
||||
tokio::time::timeout(observation, pending.as_mut()).await.is_ok()
|
||||
};
|
||||
let pending_while_locked = guard.pending_tasks();
|
||||
drop(guard);
|
||||
if !already_finished && !cancelled_while_locked {
|
||||
pending.as_mut().await;
|
||||
}
|
||||
drop(pending);
|
||||
let pending_after_cancel = state.read().await.pending_tasks();
|
||||
|
||||
// Positive control queues a real free-version task while using the same
|
||||
// implementation and isolated, unconsumed worker channel.
|
||||
let mut positive = Box::pin(crate::enqueue_runtime_free_version(object, None));
|
||||
let positive_in_time = tokio::time::timeout(observation, positive.as_mut()).await.is_ok();
|
||||
if !positive_in_time {
|
||||
positive.as_mut().await;
|
||||
}
|
||||
drop(positive);
|
||||
let pending_after_control = state.read().await.pending_tasks();
|
||||
assert!(observed_wait, "must poll the real enqueue while the write lock is held");
|
||||
assert!(cancelled_while_locked, "cancellation must finish before the lock is released");
|
||||
assert_eq!(pending_while_locked, 0);
|
||||
assert_eq!(pending_after_cancel, 0);
|
||||
assert!(positive_in_time, "positive control must finish within the observation limit");
|
||||
assert_eq!(pending_after_control, 1, "positive control must queue a task");
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn newer_noncurrent_cancelled_after_real_read_while_waiting_for_expiry_write_lock_never_queues() {
|
||||
let (_temp_dir, store) = setup_local_scanner_store(1, 4).await;
|
||||
let bucket = "expiry-lock-bucket";
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create bucket");
|
||||
let expected_incarnation = store.bucket_incarnation_id(bucket).await.expect("real incarnation metadata");
|
||||
assert!(!expected_incarnation.is_nil());
|
||||
let state = EcstoreExpiryState::new_with_unconsumed_worker_channel(4);
|
||||
let reads = Arc::new(AtomicUsize::new(0));
|
||||
let context = crate::ExpiryLockTestContext {
|
||||
state: state.clone(),
|
||||
store: Some(store.clone()),
|
||||
completed_incarnation_reads: reads.clone(),
|
||||
};
|
||||
let observation = Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS);
|
||||
crate::EXPIRY_LOCK_TEST_CONTEXT
|
||||
.scope(context, async {
|
||||
let versions = vec![crate::ScannerObjectToDelete {
|
||||
object_name: "noncurrent-object".to_string(),
|
||||
version_id: Some(Uuid::new_v4()),
|
||||
..Default::default()
|
||||
}];
|
||||
let event = crate::Event::default();
|
||||
let src = crate::LcEventSrc::Scanner;
|
||||
let cancellation = CancellationToken::new();
|
||||
let guard = state.write().await;
|
||||
let mut pending = Box::pin(crate::enqueue_runtime_newer_noncurrent(
|
||||
bucket,
|
||||
versions.clone(),
|
||||
event.clone(),
|
||||
&src,
|
||||
Some(&cancellation),
|
||||
));
|
||||
let first = observe_queued(pending.as_mut(), || reads.load(Ordering::SeqCst) == 1, observation).await;
|
||||
let observed_wait_after_read = matches!(&first, Ok(None));
|
||||
let mut result = match first {
|
||||
Ok(Some(value)) => Some(value),
|
||||
_ => None,
|
||||
};
|
||||
cancellation.cancel();
|
||||
let cancelled_while_locked = if result.is_some() {
|
||||
false
|
||||
} else {
|
||||
match tokio::time::timeout(observation, pending.as_mut()).await {
|
||||
Ok(value) => {
|
||||
result = Some(value);
|
||||
true
|
||||
}
|
||||
Err(_) => false,
|
||||
}
|
||||
};
|
||||
let pending_while_locked = guard.pending_tasks();
|
||||
drop(guard);
|
||||
let result = match result {
|
||||
Some(value) => value,
|
||||
None => pending.as_mut().await,
|
||||
};
|
||||
drop(pending);
|
||||
let pending_after_cancel = state.read().await.pending_tasks();
|
||||
let mut positive = Box::pin(crate::enqueue_runtime_newer_noncurrent(bucket, versions, event, &src, None));
|
||||
let bounded_positive = tokio::time::timeout(observation, positive.as_mut()).await;
|
||||
let positive_in_time = bounded_positive.is_ok();
|
||||
let positive_result = match bounded_positive {
|
||||
Ok(value) => value,
|
||||
Err(_) => positive.as_mut().await,
|
||||
};
|
||||
drop(positive);
|
||||
let pending_after_control = state.read().await.pending_tasks();
|
||||
assert!(
|
||||
observed_wait_after_read,
|
||||
"real incarnation read must finish before waiting on the held lock"
|
||||
);
|
||||
assert!(cancelled_while_locked, "cancellation must finish before the lock is released");
|
||||
assert!(!result);
|
||||
assert_eq!(pending_while_locked, 0);
|
||||
assert_eq!(pending_after_cancel, 0);
|
||||
assert!(positive_in_time, "positive control must finish within the observation limit");
|
||||
assert!(positive_result, "positive control must enqueue the nonempty version batch");
|
||||
assert_eq!(pending_after_control, 1);
|
||||
assert_eq!(reads.load(Ordering::SeqCst), 2);
|
||||
})
|
||||
.await;
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
// Licensed under the Apache License, Version 2.0.
|
||||
|
||||
use super::*;
|
||||
use crate::workload_admission::ScannerNodeRequest;
|
||||
use rustfs_concurrency::adaptive::{FeedbackController, FeedbackObservation, NodeAdmission, ResourcePressure};
|
||||
use std::num::NonZeroU16;
|
||||
use tokio::sync::oneshot;
|
||||
|
||||
#[derive(Clone, Copy, PartialEq, Eq)]
|
||||
enum ResumeCase {
|
||||
Unchanged,
|
||||
Updated,
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
async fn check_cache_lease_resume(case: ResumeCase) {
|
||||
let (_directory, store) = setup_local_scanner_store(1, 4).await;
|
||||
let set = store.pools[0].disk_set[0].clone();
|
||||
let source = DataUsageCacheSource::new(0, 0);
|
||||
let cache_name = "node-resume/.usage-cache.bin";
|
||||
let cache = DataUsageCache::default();
|
||||
cache.save(set.clone(), cache_name).await.expect("seed cache");
|
||||
let revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("initial revisions");
|
||||
let gate = NodeAdmission::new(NonZeroU16::new(1).expect("one shared unit"));
|
||||
gate.update(
|
||||
&mut FeedbackController::new(),
|
||||
Some(FeedbackObservation {
|
||||
generation: 1,
|
||||
max_concurrency: 1,
|
||||
pressure: ResourcePressure::Unknown,
|
||||
completed_units: None,
|
||||
}),
|
||||
);
|
||||
let mut held = Some(gate.acquire().await);
|
||||
let mut guard = acquire_scanner_cache_locks(set.as_ref(), cache_name, source)
|
||||
.await
|
||||
.expect("initial cache lease");
|
||||
let ctx = CancellationToken::new();
|
||||
let (respond, response) = oneshot::channel();
|
||||
let request = ScannerNodeRequest {
|
||||
owner: gate.clone(),
|
||||
respond,
|
||||
cancel: ctx.clone(),
|
||||
};
|
||||
let mut expected = revisions.clone();
|
||||
let result = {
|
||||
let resume = guard.resume_node_operation(set.clone(), cache_name, source, &revisions, request);
|
||||
tokio::pin!(resume);
|
||||
assert!(futures::poll!(&mut resume).is_pending(), "shared node is occupied");
|
||||
let other = tokio::time::timeout(Duration::from_secs(1), acquire_scanner_cache_locks(set.as_ref(), cache_name, source))
|
||||
.await
|
||||
.expect("cache lock must be released before joining the node queue")
|
||||
.expect("another cache worker can acquire the lease");
|
||||
if case == ResumeCase::Updated {
|
||||
let mut newer = cache;
|
||||
newer.info.next_cycle = 1;
|
||||
newer
|
||||
.save(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("another worker updates the cache");
|
||||
expected = DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("new cache revisions");
|
||||
assert_ne!(expected, revisions);
|
||||
}
|
||||
drop(other);
|
||||
if case == ResumeCase::Cancelled {
|
||||
ctx.cancel();
|
||||
} else {
|
||||
held.take().expect("held node unit").complete();
|
||||
}
|
||||
tokio::time::timeout(Duration::from_secs(2), resume)
|
||||
.await
|
||||
.expect("node resumption must finish or observe cancellation")
|
||||
};
|
||||
if case == ResumeCase::Unchanged {
|
||||
result.expect("unchanged cache resumes the same walk");
|
||||
let permit = response.await.expect("coordinator returns the node permit");
|
||||
assert_eq!(gate.snapshot().active, 1);
|
||||
assert_eq!(gate.completed(), Some(1), "admitting scanner work must not count it as completed");
|
||||
assert!(!guard.is_lock_lost());
|
||||
let lock = set
|
||||
.new_ns_lock(RUSTFS_META_BUCKET, &scanner_cache_lock_resource(cache_name, source))
|
||||
.await
|
||||
.expect("competing cache lock");
|
||||
assert!(
|
||||
lock.get_write_lock_quiet(Duration::from_millis(20)).await.is_err(),
|
||||
"cache lease must be restored before the operation resumes"
|
||||
);
|
||||
permit.complete();
|
||||
} else {
|
||||
result.expect_err("cancelled or stale work cannot resume");
|
||||
assert!(response.await.is_err(), "failed resumption must not authorize storage work");
|
||||
if case == ResumeCase::Cancelled {
|
||||
assert!(guard.is_lock_lost());
|
||||
assert_eq!(gate.snapshot().active, 1, "the running heal keeps its unit");
|
||||
drop(held.take());
|
||||
assert_eq!(gate.completed(), Some(0));
|
||||
} else {
|
||||
assert_eq!(gate.completed(), Some(1));
|
||||
}
|
||||
}
|
||||
drop(guard);
|
||||
assert_eq!(gate.snapshot().active, 0);
|
||||
assert!(gate.try_acquire().is_some(), "dropped requests must leave no queue or capacity leak");
|
||||
assert_eq!(
|
||||
DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("final revisions"),
|
||||
expected
|
||||
);
|
||||
acquire_scanner_cache_locks(set.as_ref(), cache_name, source)
|
||||
.await
|
||||
.expect("final cache lease is available");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_node_wait_yields_cache_lease_before_fifo_admission() {
|
||||
check_cache_lease_resume(ResumeCase::Unchanged).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_node_wait_rejects_cache_changed_while_unlocked() {
|
||||
check_cache_lease_resume(ResumeCase::Updated).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_node_wait_cancellation_releases_queue_without_stealing_running_unit() {
|
||||
check_cache_lease_resume(ResumeCase::Cancelled).await;
|
||||
}
|
||||
@@ -22,6 +22,8 @@ use serde::{Deserialize, Serialize};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
pub(crate) use rustfs_ecstore::with_background_disk_io as ecstore_with_background_disk_io;
|
||||
#[cfg(all(test, unix))]
|
||||
pub(crate) use rustfs_ecstore::with_background_resource_unit as ecstore_with_background_resource_unit;
|
||||
|
||||
pub(crate) use s3s::dto::{
|
||||
BucketLifecycleConfiguration as EcstoreBucketLifecycleConfiguration, LifecycleRuleFilter as EcstoreLifecycleRuleFilter,
|
||||
@@ -39,6 +41,8 @@ pub(crate) use rustfs_ecstore::api::data_usage::{NativeScannerPauseBacklogWriteF
|
||||
|
||||
pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys as EcstoreBucketTargetSys;
|
||||
pub(crate) use rustfs_ecstore::api::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc as EcstoreLcEventSrc;
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::bucket::lifecycle::bucket_lifecycle_ops::ExpiryState as EcstoreExpiryState;
|
||||
pub(crate) use rustfs_ecstore::api::bucket::lifecycle::bucket_lifecycle_ops::{
|
||||
apply_expiry_rule as ecstore_apply_expiry_rule, apply_transition_rule as ecstore_apply_transition_rule,
|
||||
lifecycle_version_delete_target as ecstore_lifecycle_version_delete_target,
|
||||
@@ -99,6 +103,10 @@ pub(crate) use rustfs_ecstore::api::disk::{
|
||||
pub(crate) use rustfs_ecstore::api::disk::{
|
||||
DiskOption as EcstoreDiskOption, DiskStore as EcstoreDiskStore, new_disk as ecstore_new_disk,
|
||||
};
|
||||
#[cfg(all(test, unix))]
|
||||
pub(crate) use rustfs_ecstore::api::disk::{
|
||||
LocalPublicationPause as EcstoreLocalPublicationPause, LocalPublicationStage as EcstoreLocalPublicationStage,
|
||||
};
|
||||
pub(crate) use rustfs_ecstore::api::error::{
|
||||
Error as EcstoreErrorType, Result as EcstoreResultType, StorageError as EcstoreStorageError,
|
||||
};
|
||||
@@ -161,10 +169,17 @@ pub(crate) mod ecstore_diagnostic {
|
||||
pub(crate) type EcstoreHealResultItem = <EcstoreStore as storage_contracts::HealOperations>::HealResultItem;
|
||||
|
||||
pub(crate) mod owner {
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::EcstoreExpiryState;
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::ecstore_set_object_store_resolver;
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::storage_contracts::DiskSetSelector;
|
||||
#[cfg(all(test, unix))]
|
||||
pub(crate) use super::{
|
||||
EcstoreLocalPublicationPause, EcstoreLocalPublicationStage, ecstore_with_background_disk_io,
|
||||
ecstore_with_background_resource_unit,
|
||||
};
|
||||
pub(crate) use super::{
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan,
|
||||
ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner,
|
||||
|
||||
@@ -12,9 +12,143 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::sync::{Arc, LazyLock, RwLock};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, LazyLock, OnceLock, RwLock};
|
||||
|
||||
use rustfs_concurrency::adaptive::{NodeAdmission, NodePermit};
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
use tokio::sync::{mpsc, oneshot};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use crate::ScannerError;
|
||||
|
||||
static SCANNER_NODE_ADMISSION: OnceLock<Arc<NodeAdmission>> = OnceLock::new();
|
||||
|
||||
tokio::task_local! {
|
||||
static SCANNER_NODE_REQUESTS: ScannerNodeRequestSender;
|
||||
static SCANNER_NODE_OPERATION_CANCEL: CancellationToken;
|
||||
}
|
||||
|
||||
/// Cancellation for the currently admitted scanner operation, when present.
|
||||
pub(crate) fn scanner_node_operation_cancellation() -> Option<CancellationToken> {
|
||||
SCANNER_NODE_OPERATION_CANCEL.try_with(Clone::clone).ok()
|
||||
}
|
||||
|
||||
pub(crate) struct ScannerNodeRequest {
|
||||
pub(crate) owner: Arc<NodeAdmission>,
|
||||
pub(crate) respond: oneshot::Sender<NodePermit>,
|
||||
pub(crate) cancel: CancellationToken,
|
||||
}
|
||||
|
||||
pub(crate) type ScannerNodeRequestSender = mpsc::Sender<ScannerNodeRequest>;
|
||||
pub(crate) type ScannerNodeRequestReceiver = mpsc::Receiver<ScannerNodeRequest>;
|
||||
|
||||
/// One suspended operation per walk; disabled admission allocates no channel.
|
||||
pub(crate) fn scanner_node_request_channel() -> (Option<ScannerNodeRequestSender>, Option<ScannerNodeRequestReceiver>) {
|
||||
if scanner_node_admission().is_none() {
|
||||
return (None, None);
|
||||
}
|
||||
let (sender, receiver) = mpsc::channel(1);
|
||||
(Some(sender), Some(receiver))
|
||||
}
|
||||
|
||||
pub(crate) async fn with_scanner_node_requests<F: std::future::Future>(
|
||||
sender: Option<ScannerNodeRequestSender>,
|
||||
operation: F,
|
||||
) -> F::Output {
|
||||
match sender {
|
||||
Some(sender) => SCANNER_NODE_REQUESTS.scope(sender, operation).await,
|
||||
None => operation.await,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
tokio::task_local! {
|
||||
pub(crate) static TEST_SCANNER_NODE_ADMISSION: Arc<NodeAdmission>;
|
||||
pub(crate) static TEST_SCANNER_AFTER_NODE_ACQUIRE: Arc<dyn Fn() + Send + Sync>;
|
||||
}
|
||||
|
||||
/// Install the heal manager's node owner before starting scanner workers.
|
||||
/// Refuse replacement: existing scans must not outlive an independently renewed owner.
|
||||
pub fn set_scanner_node_admission(owner: Arc<NodeAdmission>) -> bool {
|
||||
SCANNER_NODE_ADMISSION.set(owner).is_ok()
|
||||
}
|
||||
|
||||
pub(crate) fn scanner_node_admission() -> Option<Arc<NodeAdmission>> {
|
||||
#[cfg(test)]
|
||||
if let Ok(owner) = TEST_SCANNER_NODE_ADMISSION.try_with(Arc::clone) {
|
||||
return Some(owner);
|
||||
}
|
||||
SCANNER_NODE_ADMISSION.get().cloned()
|
||||
}
|
||||
|
||||
/// Try the fast path first. A scoped coordinator releases the cache lease before
|
||||
/// joining the node queue and validates the cache before returning a permit.
|
||||
/// Uncoordinated callers defer instead of parking under an unknown lock.
|
||||
pub(crate) async fn run_scanner_node_operation<T, E>(
|
||||
owner: Option<&Arc<NodeAdmission>>,
|
||||
cancel: &CancellationToken,
|
||||
operation: impl std::future::Future<Output = Result<T, E>>,
|
||||
) -> Result<Result<T, E>, ScannerError> {
|
||||
let Some(owner) = owner else {
|
||||
return Ok(operation.await);
|
||||
};
|
||||
let operation = SCANNER_NODE_OPERATION_CANCEL.scope(cancel.clone(), operation);
|
||||
if cancel.is_cancelled() {
|
||||
return Err(ScannerError::Other("Operation cancelled".to_string()));
|
||||
}
|
||||
let permit = match owner.try_acquire() {
|
||||
Some(permit) => permit,
|
||||
None => {
|
||||
let sender = SCANNER_NODE_REQUESTS
|
||||
.try_with(Clone::clone)
|
||||
.map_err(|_| ScannerError::ResourceDeferred)?;
|
||||
let acquire = async {
|
||||
let (respond, response) = oneshot::channel();
|
||||
sender
|
||||
.send(ScannerNodeRequest {
|
||||
owner: owner.clone(),
|
||||
respond,
|
||||
cancel: cancel.clone(),
|
||||
})
|
||||
.await
|
||||
.map_err(|_| ScannerError::ResourceDeferred)?;
|
||||
response.await.map_err(|_| ScannerError::ResourceDeferred)
|
||||
};
|
||||
tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => return Err(ScannerError::Other("Operation cancelled".to_string())),
|
||||
permit = acquire => permit?,
|
||||
}
|
||||
}
|
||||
};
|
||||
#[cfg(test)]
|
||||
let _ = TEST_SCANNER_AFTER_NODE_ACQUIRE.try_with(|hook| hook());
|
||||
let operation_started = AtomicBool::new(false);
|
||||
tokio::pin!(operation);
|
||||
let observed = std::future::poll_fn(|context| {
|
||||
operation_started.store(true, Ordering::Relaxed);
|
||||
std::future::Future::poll(operation.as_mut(), context)
|
||||
});
|
||||
tokio::pin!(observed);
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => {
|
||||
// The node slot belongs to the waiter, not the native operation.
|
||||
drop(permit);
|
||||
if !operation_started.load(Ordering::Relaxed) {
|
||||
return Err(ScannerError::Other("Operation cancelled".to_string()));
|
||||
}
|
||||
// Keep polling the started operation under its existing locks.
|
||||
return Ok(observed.await);
|
||||
}
|
||||
result = &mut observed => result,
|
||||
};
|
||||
if result.is_ok() && !cancel.is_cancelled() {
|
||||
permit.complete();
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
type WorkloadSnapshotProviderRef = Arc<dyn WorkloadAdmissionSnapshotProvider + Send + Sync>;
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user