Compare commits

..

23 Commits

Author SHA1 Message Date
overtrue 3f994c59eb fix(select): classify function argument planner errors 2026-08-11 23:14:23 +08:00
GatewayJ 7c1d9dec8f Merge branch 'main' into fix/s3-select-error-semantics 2026-08-11 14:13:53 +08:00
Zhengchao An 31cb720471 fix: exclude e2e_test from s3s footprint ratchet baseline (#5949) 2026-08-11 05:48:27 +00:00
唐小鸭 603bdea516 fix(site-replication): route state RMW through one locked transaction (#5882)
* test(site-replication): pin retry-event lost-update against locked RMW (red)

P1-15 (rustfs/backlog#1675 B2): the site-replication retry-event writers
(enqueue/dequeue, which hang off every hook broadcast path) perform a
load -> mutate -> persist without taking SITE_REPLICATION_STATE_LOCK, so
a single process can lose a concurrent lock-holding writer's update; the
service-side reload path is equally unlocked, and no writer holds a
distributed lock across the read-modify-write, so multi-node RMW loses
updates even where the process lock is held.

Red evidence (current main): replaying enqueue's exact three steps around
a completed mark_pending_rotation_peer_acked commit wipes the rotation
ack — the final state holds the retry event but not the ack.

* fix(site-replication): route state RMW through one locked transaction

P1-15 PR1 (rustfs/backlog#1675 B2). The site-replication state object
(config/site-replication/state.json, which also carries the retry-event
queue) was mutated through read-modify-write sequences with inconsistent
locking: the retry-event writers on every hook broadcast path and the
RPC-driven service reload took no lock at all (single-process lost
updates, pinned by the red commit), and no writer held a distributed lock
across the whole RMW (cross-node lost updates everywhere).

- New admin/site_replication_state module: the state transaction boundary
  `with_site_replication_state_lock[_on]` — process mutex plus the
  distributed config-object write lock (the pattern proven by the repair
  state), with the shared path constant. The process mutex is transitional
  until PR2 migrates the remaining ~26 call sites.
- handlers: typed `update_site_replication_state` (no-lock load /
  persist-or-clear inside the boundary; normalizes the peer map exactly
  once, retiring the double-clone/double-normalize persist path, P2-22).
  Migrated: retry-event enqueue (always-write), dequeue (lock-free probe,
  transaction on hit), mark_pending_rotation/remove_peer_acked.
- service reload: the tolerant byte-level read->normalize->save now runs
  inside the same boundary via no-lock IO — a cluster-wide reload fan-out
  can no longer overwrite a concurrent state writer. Normalization
  semantics untouched (all six service-side tests unchanged and green).
- Add/PeerJoin/Edit handlers release the state guard before their peer
  fan-out: the transport helpers' retry-event bookkeeping now re-enters
  the state transaction and must not nest inside the guard (the
  adversarial review caught this as a re-entrancy deadlock; the fix
  mirrors the Remove/Rotate handlers' existing scope). The Edit non-
  refresh branch commits before fanning out — the old fanout-first order
  recorded retry events pointing at a state the local site had not saved.
- ecstore: delete_config_no_lock (+ facade/bridge exports) so the clear
  half of persist-or-clear works under the held object lock.

Red -> green: the red commit pinned the deterministic lost-update
interleaving (stale retry-event persist wiping a committed rotation ack);
the test now drives the real functions concurrently for 8 rounds and
asserts every retry event and every ack survives. Full
handlers/service site-replication unit suites green (171 + 6); dual-node
site-replication e2e (state edit fresh/stale, object replication) green;
fmt / clippy / logging guardrails clean.

Adversarial review: one blocking finding (the re-entrancy deadlock above)
fixed and re-verified by a full second pass over all 30 lock sites and
the Add/Join/Edit call graphs. Non-blocking notes recorded for PR2:
mark_* now persists on miss (persist-or-clear semantics; a miss-skip
return is a cheap follow-up), Add still holds the guard across the peer
join probe (pre-existing availability debt), and a timeout-guarded
unreachable-peer regression test for the fan-out paths.

* fix(site-replication): keep the state mutex behind an owner helper

CI's architecture migration guard lists SITE_REPLICATION_STATE_LOCK as an
owner-local static, so it may not be `pub(crate)`. Keep it private to the
new module and let the not-yet-migrated RMW call sites take it through
`site_replication_state_process_guard()` — the sanctioned owner-helper
pattern; the helper disappears with the mutex in PR2.

* fix(site-replication): keep peer-edit delivery under the state guard

Review follow-up (#5882).

Releasing the guard before the fan-out (my deadlock fix) traded the
ordering the guard used to provide: edit A could commit and stall while
edit B committed and reached a peer first, then A arrived last and won.
The peer edit handler applies whatever arrives — it has no generation or
updated-at fence — and a successful stale delivery is not repaired by the
retry queue, so the sites diverge silently.

The fan-out is back under the guard. What actually could not run there is
the retry-event bookkeeping, which re-enters the state transaction, so the
edit branch now delivers with the plain transport and settles the retry
queue after the guard is released: successes dequeue, the first failure
enqueues and is returned. Ordering and bookkeeping both preserved. The add
handler keeps its peer-edit finalize fan-out under the guard for the same
reason and releases only before bootstrap/back-fill, which send bucket-ops
(not peer edits) through retry-event transports.

The concurrency test could not tell the two guards apart — both writers
took both locks, so it passed with either removed. Replaced by two tests
that isolate one guard each, both verified by mutation:

- a process-only legacy writer (the shape the not-yet-migrated call sites
  still use) racing the transaction: fails when the transaction stops
  taking the process mutex;
- two writers that bypass the process mutex, as separate nodes do, driving
  the production object-lock path (`with_site_replication_state_object_lock`
  factored out for exactly this): fails when the distributed lock is
  removed.

Verification: handlers 173 + service 6 unit tests green; site-replication
dual-node and three-node edit e2e green; arch/layer/logging guardrails,
fmt and clippy clean.

* fix(site-replication): fence peer-edit delivery by generation

Review follow-up on the two remaining holes in the edit path.

Ordering was only process-local. `SITE_REPLICATION_STATE_LOCK` is per
node, so holding it across the fan-out orders the edits ONE node accepts
and nothing else: two nodes of the same site can both commit and reach a
peer in the opposite order, and the peer edit handler applied whatever
arrived last. Each edit now takes a generation from
`SiteReplicationState::edit_generation`, allocated in the same commit as
the edit itself — i.e. under the distributed state-object lock, so two
nodes can never share one. The generation rides the peer-edit request as
query parameters and the receiver rejects (acks without applying) a
delivery at or below the mark it already applied for that origin site,
recording the mark in the same commit as the edit it fences. Peers that
predate the fence send no parameters and are applied as before.

Retry settlement could discard a newer failure. After the guard is
released, a success for edit A removed every retry event for
(peer, peer-edit): if edit B committed, failed its own delivery and
enqueued while A was in flight, A erased it — local state B, peer on A,
nothing queued to converge them. Settlement now only removes events whose
recorded generation is not newer than the one being settled, and a later
failure never lowers the fence. Broadcast paths carry no generation and
settle unconditionally as before; their events live under their own
paths and cannot collide with a peer-edit delivery.

A departed peer's mark is dropped on load: a site that leaves drops below
two peers, which clears its state object and restarts its counter at
zero, so a leftover mark would reject every edit it sends after it
rejoins.

Tests: two-node generation uniqueness (drop the object lock and the two
nodes collide), the receiver's staleness predicate and its wiring, the
settlement interleaving (drop the fence and B's retry is erased), and the
rejoin reset.

Refs: rustfs/backlog#1675 (P1-15)
2026-08-11 13:41:28 +08:00
唐小鸭 a076ae4045 test(replication): pin the scanner existing-object compensation matrix (#5877)
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects
BEFORE the replication rule arrived, leaving the scanner's existing-object
resync pass — the only channel for such objects — without end-to-end
coverage, and the enqueue truth table partially unpinned at unit level.

e2e (both negative cells are contracts, asserted over multiple fast-scanner
cycles next to a replicated control key that proves the scanner and the
live path are running):
- test_scanner_compensates_existing_objects_across_write_paths: plain PUT,
  CopyObject and Snowball auto-extract products written pre-rule all
  converge via scanner compensation; a null-version object (PUT before the
  bucket became versioned) is pinned as never compensated (the scanner heal
  gate skips nil-version objects).
- test_scanner_never_compensates_when_existing_object_replication_disabled:
  ExistingObjectReplication=Disabled is a contract, not a delay — existing
  keys stay absent while post-rule writes replicate normally.

Unit truth-table pins (crates/replication):
- queue.rs: an empty replicate decision (Disabled existing-object, inbound
  REPLICA) skips heal queueing for every status; Completed without a resync
  decision skips.
- operation.rs: existing-object resync without a reset replicates exactly
  the never-replicated (Empty) objects.

Helper: put_bucket_replication_with_statuses parameterizes the previously
hardcoded ExistingObjectReplication status; the nextest count comments are
refreshed to the post-rebase totals.
2026-08-11 03:58:25 +00:00
houseme 8a8be12f0b perf(ecstore): raise replay cache auto capacity (#5946)
Increase the replay cache resource model so 16 CPU / 31-32 GiB field nodes auto-size to the 32M cap without an env override.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-11 11:37:58 +08:00
唐小鸭 2ecf6b4575 fix(replication): probe the version-identity contract in replication-check (#5881)
* test(replication): pin the version-fidelity probe contract (red)

P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.

Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.

test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.

Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).

* fix(replication): probe the version-identity contract in replication-check

P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.

- The probe PUT now carries the source version as `?versionId=` (the exact
  shape live replication uses since P0-5, and the only shape MinIO
  consumes; the internal source-version-id header alone would let the
  probe pass against targets the real data path drifts on). Reuses
  ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
  equal the sent source id. On mismatch the phase fails with the
  machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
  (new optional Code field on phase statuses; Go decoders ignore unknown
  keys), the overall target fails, the later version-addressed mutation
  phases are skipped, and cleanup still removes the probe via the id the
  target actually assigned (with the existing list-based sweep as backstop
  when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
  id (mirroring remove_object), and the replication PUT path audits it —
  every drifting PUT increments
  rustfs_replication_version_identity_drift_total and the first drift per
  target ARN logs a structured warning pointing at ?replication-check.
  The drift judgment is a pure function with an exemption-matrix test
  (empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.

Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.

* fix(e2e): stop the fake target self-deadlocking on version-id minting

The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.

The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.

* chore(test): refresh the nextest replication count invariant

The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
2026-08-11 03:04:05 +00:00
cxymds 2aa0148454 fix: report stalled object traffic as unready (#5936)
* fix: report stalled object traffic as unready

* fix: track fully received PUT storage progress
2026-08-11 10:55:22 +08:00
Xiaoyang Han 3289d40ce9 fix(ecstore): publish multipart parts on Windows (#5937)
* fix(ecstore): publish multipart parts on Windows

* test(ecstore): pin Windows multipart durability

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-11 10:01:03 +08:00
cxymds 849837e262 fix(rpc): negotiate replay-safe mutation authentication (#5928)
* fix(rpc): negotiate replay-safe mutation auth

* fix(rpc): preserve strict legacy replay scope
2026-08-11 01:03:25 +00:00
Henry Guo 727a10e111 fix(scanner): skip disk inventory in scan spans (#5933)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-11 09:01:51 +08:00
GatewayJ 4a8759239d fix(select): enforce typed S3 Select error semantics 2026-08-11 02:26:58 +08:00
houseme 3747d19ce5 perf(ecstore): hedge bounded GET metadata fanout (#5935)
Keep opt-in bounded GET data-read fanout from waiting on a single pending ReadVersion response when an unscheduled spare disk can satisfy quorum. Add a deterministic 2+2 regression that pauses the third scheduled metadata read and verifies the spare is started before returning.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-08-10 17:11:04 +00:00
houseme 1148e76279 test(ecstore): update prepared GET fanout default (#5932)
Assert the prepared GET metadata path keeps the default full data-read fanout after PR #5929 made bounded data-read fanout opt-in.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 15:38:37 +00:00
唐小鸭 320b788a50 test(admin): relax object-lambda SNI test timeout under full-suite load (#5923)
The SNI preservation test is the only object-lambda test doing a real
TLS handshake; the shared helper's 2s whole-request timeout turns
concurrent fsync-heavy TestECStoreEnv neighbors into a deterministic
TimedOut when the per-build nextest schedule overlaps them. The test
verifies SNI, not latency, so widen its budget to a still-bounded 30s.
2026-08-10 22:20:34 +08:00
唐小鸭 3c31eaf06f fix(replication): retry, persist and replay failed delete-marker purges (#5864)
* test(replication): pin delayed delete-marker purge failure handling (red)

P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:

- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
  four scripted 503s outlast every existing channel (version-purge
  replication + its in-process MRF fast retries + the watcher's single
  attempt = 3 target DELETEs, all faulted in the recorded run); the
  replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
  exhausted purge intents never reach the durable MRF journal, so a restart
  replays nothing (recorded run: 3 faulted attempts, zero post-restart).

Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
  attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
  restart, marker still present.

Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.

* fix(replication): retry, persist and replay failed delete-marker purges

P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.

Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
  results (warn + metrics on failure, including `target_client_missing`),
  supports retrying only the failed targets, and treats a target-side
  NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
  retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
  failed targets across its 5x1s watch window; on exhaustion it persists
  the purge intent to the durable MRF journal via the new
  `ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
  re-dispatch would loop unboundedly against a down target). Intent entries
  are shaped as marker-creation deletes so replay funnels into the stale-
  marker branch.
- The stale-marker branch (source marker already gone) now purges the
  targets instead of silently returning success — closing a latent leak —
  and reports the purge result as the replay outcome. Heal callers retry
  for the full window (the startup MRF processor runs before target
  clients initialize); live callers attempt once and fall back to a fresh
  durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
  by a unit test) no longer includes the delayed purge, so successfully
  replayed delete-marker entries are acknowledged instead of retained
  forever.

Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).

Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).

* chore(test): refresh the nextest replication count invariant

The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.

* fix(replication): purge the marker version the target actually assigned

Review follow-up (#5864), two real defects:

- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
  per-target marker version ids this round recorded were invisible to it.
  Against a target that mints its own ids the purge fell back to a
  source-derived id, the target answered the versioned DELETE with an
  idempotent 204, and that "success" cleared the retry set while the real
  marker stayed behind. The watcher now receives the merged replication
  state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
  without entering `failed_arns`, so an empty result made both the watcher
  and the MRF replay treat a purge that issued no DELETE as successful and
  drop the intent. The refusal is now a per-target failure (own metric
  label): the leak stays visible and the intent is retained instead of
  being acknowledged. The version decision also moved ahead of the client
  lookup, so the refusal is decided from metadata alone.

Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
2026-08-10 22:16:21 +08:00
houseme fe2516ee86 perf(ecstore): keep bounded GET fanout opt-in (#5929)
Keep GET data-read metadata early-stop and bounded fanout behind explicit environment switches so the default path preserves full fanout read-failure tolerance.

Retain the focused opt-in A/B coverage and the invalid parity full-fanout guard for heterogeneous set layouts.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 22:15:34 +08:00
cxymds 7ca69eb39c fix: correct SNSD cluster diagnostics (#5930) 2026-08-10 20:32:01 +08:00
hector 95627cb601 fix: create config file before fpm RPM packaging (#5924)
The RPM build step fails because fpm's --config-files flag requires
/etc/default/rustfs to exist in the staging area, but unlike the DEB
build (which creates it in its package directory structure), the fpm
command has no prior step creating this file.

Create the config file in a temporary directory and pass it to fpm
via a source=dest mapping, matching the DEB build's behavior.
2026-08-10 08:15:15 +00:00
houseme d97e059c3c fix(iam): merge OIDC extra root CAs (#5915)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 15:24:52 +08:00
houseme d900e11a09 perf(ecstore): expose replay cache RPC sources (#5926)
Track accepted replay cache records by gRPC operation and split Lock/Unlock and ReadVersion methods out of grpc_other so hotpath validation can attribute nonce pressure without changing replay protection semantics.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 15:13:27 +08:00
houseme f1ff9a36bc test(heal): cover replacement terminal recovery (#5920)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 14:51:26 +08:00
houseme 276eea1fba test(heal): cover replacement target evidence failures (#5919)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-10 14:50:55 +08:00
75 changed files with 8045 additions and 1018 deletions
+2 -2
View File
@@ -218,7 +218,7 @@ test-group = 'ecstore-serial-flaky'
# the nightly profile derives its set as "the replication module MINUS this
# allowlist", so any new replication test lands in nightly by default (never
# silently unrun) until it is explicitly blessed as fast here. Keep the two
# regexes byte-identical. Count invariant: 20 here + 36 nightly = 56 total
# regexes byte-identical. Count invariant: 20 here + 49 nightly = 69 total
# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
# (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -344,7 +344,7 @@ path = "junit.xml"
# object_lambda) — too heavy for the merge budget; they run in ci-7's
# nightly 4-node lane.
# * replication_extension_test — repl-1 already splits it into the PR
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (49 slow) lanes and reserves
# it for those, so e2e-full does not double-run it.
# * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
# manual-localhost:9000 reliant/policy tests are ci-13's migration.
+12
View File
@@ -322,6 +322,17 @@ jobs:
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
sudo gem install fpm
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$VERSION" \
@@ -362,6 +373,7 @@ jobs:
) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
Generated
+3 -1
View File
@@ -9457,7 +9457,6 @@ dependencies = [
"futures",
"hotpath",
"http 1.5.0",
"libc",
"metrics",
"rustfs-common",
"rustfs-concurrency",
@@ -9494,6 +9493,7 @@ dependencies = [
"moka",
"openidconnect",
"pollster",
"rcgen",
"reqwest",
"rustfs-config",
"rustfs-credentials",
@@ -9505,6 +9505,8 @@ dependencies = [
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-utils",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"serial_test",
+8
View File
@@ -97,6 +97,14 @@ Current guidance:
- enables minimal payload mode for GET health responses (`status`, `ready` only).
- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
- TTL for readiness cache evaluation.
- `RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE`
- withdraws readiness when bounded object read/write stages stop completing while requests remain active.
- default is `true`.
- `RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS`
- maximum time without completion in a bounded object stage before readiness is withdrawn.
- default is `30000`; `0` uses the default.
- the effective value is at least 5 seconds longer than `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`.
- this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
- enables busy protection behavior for health probes.
- default is `false`.
+13
View File
@@ -22,6 +22,19 @@ pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
/// Enable readiness withdrawal when bounded object read/write stages stop
/// completing while requests remain active.
pub const ENV_HEALTH_OBJECT_PROGRESS_ENABLE: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE";
pub const DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE: bool = true;
/// Requested time without completion in a bounded object stage before local
/// readiness is withdrawn (milliseconds). A value of `0` uses the default;
/// runtime adds a safety floor based on the object-lock acquisition timeout.
pub const ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS";
pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
/// Additional time beyond the configured object-lock acquisition deadline.
pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
/// Timeout for cluster health readiness collectors (milliseconds).
/// This bounds expensive storage and lock quorum checks used by cluster probes.
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
+5
View File
@@ -36,6 +36,11 @@ pub const ENV_TRUST_SYSTEM_CA: &str = "RUSTFS_TRUST_SYSTEM_CA";
/// To change this behavior, set the environment variable RUSTFS_TRUST_SYSTEM_CA=1
pub const DEFAULT_TRUST_SYSTEM_CA: bool = false;
/// Environment variable for an extra outbound root CA certificate bundle.
/// Use this to trust an internal CA for outbound HTTPS clients without replacing
/// the default operating-system/web PKI roots via SSL_CERT_FILE.
pub const ENV_RUSTFS_EXTRA_CA_CERT: &str = "RUSTFS_EXTRA_CA_CERT";
/// Environment variable to trust leaf certificates as CA
/// When set to "1", RustFS will treat leaf certificates as CA certificates for trust validation.
/// By default, this is disabled.
+129 -11
View File
@@ -30,10 +30,10 @@ use s3s::access::{S3Access, S3AccessContext};
use s3s::auth::SimpleAuth;
use s3s::dto::{
AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteObjectInput, DeleteObjectOutput, ETag,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
HeadObjectInput, HeadObjectOutput, PutObjectInput, PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat,
UploadPartInput, UploadPartOutput,
HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
};
use s3s::service::{S3Service, S3ServiceBuilder};
use s3s::validation::{AwsNameValidation, NameValidation};
@@ -91,6 +91,7 @@ pub enum Operation {
GetObject,
HeadObject,
DeleteObject,
ListObjectVersions,
CreateMultipartUpload,
UploadPart,
CompleteMultipartUpload,
@@ -109,6 +110,8 @@ pub enum FaultAction {
/// already have buffered the rest of the current frame; the journal reports
/// the threshold, and the backend never receives or stores the request.
DisconnectAfterBytes(usize),
/// Apply the request, then close the connection before returning its response.
DisconnectAfterResponse,
/// Drain a request body in fixed-size slices, sleeping after every slice.
SlowDrain { chunk_bytes: usize, delay: Duration },
/// Store the request normally but replace the response ETag.
@@ -141,6 +144,8 @@ struct ControlState {
#[derive(Default)]
struct StoreState {
assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
total_bytes: usize,
@@ -383,6 +388,22 @@ impl FakeS3Target {
.is_some_and(|version| !version.delete_marker)
}
/// Make the target mint its own version ids instead of mirroring the
/// forwarded source version id — models a generic S3 service.
pub fn assign_own_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_version_ids = enabled;
}
/// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones.
pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
}
pub fn active_multipart_upload_count(&self) -> usize {
lock(&self.backend.store).uploads.len()
}
/// Queue `times` copies of a fault for one operation.
pub fn inject(&self, operation: Operation, action: FaultAction, times: usize) {
if times == 0 {
@@ -433,6 +454,25 @@ impl FakeS3Target {
lock(&self.control).requests.drain(..).collect()
}
/// Stored versions for one key as `(version_id, is_delete_marker)`, oldest
/// first. Empty when the bucket or key does not exist. Lets purge tests
/// assert on the target's actual state instead of inferring it from the
/// request journal (a versioned DELETE is a silent no-op for missing ids).
pub fn stored_versions(&self, bucket: &str, key: &str) -> Vec<(String, bool)> {
let state = lock(&self.backend.store);
state
.buckets
.get(bucket)
.and_then(|bucket_state| bucket_state.objects.get(key))
.map(|versions| {
versions
.iter()
.map(|version| (version.version_id.clone(), version.delete_marker))
.collect()
})
.unwrap_or_default()
}
pub async fn shutdown(mut self) {
let _ = self.shutdown.send(true);
if let Some(task) = self.task.take() {
@@ -633,6 +673,7 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
let operation = match (method, key.is_some()) {
(&Method::HEAD, false) => Operation::HeadBucket,
(&Method::GET, false) if query.contains_key("versioning") => Operation::GetBucketVersioning,
(&Method::GET, false) if query.contains_key("versions") => Operation::ListObjectVersions,
(&Method::PUT, true) if upload_id.is_some() && part_number.is_some() => Operation::UploadPart,
(&Method::PUT, true) if upload_id.is_some() || query.contains_key("partNumber") => Operation::Unknown,
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
@@ -689,10 +730,17 @@ fn validate_retained_identifier(value: String, field: &str) -> S3Result<String>
}
}
fn new_version_id(headers: &HeaderMap) -> S3Result<String> {
/// `assign_own` models a target that mints its own version ids (a generic S3
/// service): the forwarded source-version-id header is validated but NOT
/// mirrored into the stored version.
fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
let Some(value) = header_value(headers, &SOURCE_VERSION_ID_HEADERS) else {
return Ok(Uuid::new_v4().to_string());
};
if assign_own {
validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
return Ok(Uuid::new_v4().to_string());
}
let value = validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
let version_id = Uuid::parse_str(&value).map_err(|_| s3s::s3_error!(InvalidArgument, "source version ID must be a UUID"))?;
Ok(version_id.to_string())
@@ -777,7 +825,10 @@ async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<Cont
update_consumed(control, fault.expect("matched fault").sequence, 0);
Err(scripted_disconnect_error())
}
Some(FaultAction::SlowDrain { .. }) | Some(FaultAction::WrongEtag) | None => Ok(()),
Some(FaultAction::SlowDrain { .. })
| Some(FaultAction::WrongEtag)
| Some(FaultAction::DisconnectAfterResponse)
| None => Ok(()),
}
}
@@ -818,7 +869,7 @@ async fn collect_stream(
Some(FaultAction::SlowDrain { chunk_bytes, delay }) => {
return collect_stream_slow(body, capacity, *chunk_bytes, *delay).await;
}
Some(FaultAction::WrongEtag) | None => {}
Some(FaultAction::WrongEtag) | Some(FaultAction::DisconnectAfterResponse) | None => {}
}
let mut output = BytesMut::with_capacity(capacity);
@@ -880,6 +931,9 @@ fn apply_response_fault<T>(mut response: S3Response<T>, fault: Option<&RequestFa
if fault.is_some_and(|fault| fault.action == FaultAction::WrongEtag) {
response.headers.insert(ETAG, HeaderValue::from_static(WRONG_ETAG));
}
if fault.is_some_and(|fault| fault.action == FaultAction::DisconnectAfterResponse) {
response.headers.insert(DISCONNECT_HEADER, HeaderValue::from_static("true"));
}
response
}
@@ -1068,6 +1122,63 @@ impl S3 for FakeBackend {
))
}
/// Prefix + max-keys subset only — enough for the replication-check probe
/// key allocation. No pagination markers or delimiter folding.
async fn list_object_versions(
&self,
req: S3Request<ListObjectVersionsInput>,
) -> S3Result<S3Response<ListObjectVersionsOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let state = lock(&self.store);
let Some(bucket_state) = state.buckets.get(&req.input.bucket) else {
return Err(s3s::s3_error!(NoSuchBucket, "bucket does not exist"));
};
let prefix = req.input.prefix.as_deref().unwrap_or_default();
let max_keys = req.input.max_keys.unwrap_or(1000).max(0) as usize;
let mut keys: Vec<&String> = bucket_state.objects.keys().filter(|key| key.starts_with(prefix)).collect();
keys.sort();
let mut versions = Vec::new();
let mut delete_markers = Vec::new();
'keys: for key in keys {
for version in bucket_state.objects[key].iter().rev() {
if versions.len() + delete_markers.len() >= max_keys {
break 'keys;
}
if version.delete_marker {
delete_markers.push(DeleteMarkerEntry {
key: Some(key.clone()),
version_id: Some(ObjectVersionId::from(version.version_id.clone())),
last_modified: Some(version.last_modified.clone()),
..Default::default()
});
} else {
versions.push(s3s::dto::ObjectVersion {
key: Some(key.clone()),
version_id: Some(ObjectVersionId::from(version.version_id.clone())),
last_modified: Some(version.last_modified.clone()),
e_tag: Some(ETag::Strong(version.e_tag.clone())),
size: Some(version.body.len() as i64),
..Default::default()
});
}
}
}
drop(state);
Ok(apply_response_fault(
S3Response::new(ListObjectVersionsOutput {
name: Some(req.input.bucket),
versions: Some(versions),
delete_markers: Some(delete_markers),
..Default::default()
}),
fault.as_ref(),
))
}
async fn put_object(&self, req: S3Request<PutObjectInput>) -> S3Result<S3Response<PutObjectOutput>> {
let fault = request_fault(&req);
let _body_permit = timeout(MAX_FAULT_DURATION, Arc::clone(&self.body_limit).acquire_owned())
@@ -1078,7 +1189,8 @@ impl S3 for FakeBackend {
let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let version_id = new_version_id(&headers)?;
let assign_own = lock(&self.store).assign_own_version_ids;
let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? {
Some(value) => value,
None => {
@@ -1121,7 +1233,7 @@ impl S3 for FakeBackend {
content_type: version.content_type,
metadata: version.metadata,
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
..Default::default()
}),
@@ -1143,7 +1255,7 @@ impl S3 for FakeBackend {
content_type: version.content_type,
metadata: version.metadata,
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
..Default::default()
}),
@@ -1212,7 +1324,9 @@ impl S3 for FakeBackend {
));
}
let version_id = new_version_id(&headers)?;
// `state` is the live store guard: read the flag from it. Re-locking
// would self-deadlock (the store mutex is not reentrant).
let version_id = new_version_id(&headers, state.assign_own_version_ids)?;
upsert_version(
&mut state,
&input.bucket,
@@ -1252,12 +1366,16 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert(
upload_id.clone(),
MultipartState {
bucket: input.bucket.clone(),
key: input.key.clone(),
version_id: new_version_id(&headers)?,
version_id,
content_type: input.content_type,
metadata: input.metadata,
parts: BTreeMap::new(),
@@ -18,6 +18,7 @@ use crate::common::{
};
use crate::fake_s3_target::{
FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
RequestRecord,
};
use crate::kms::common::{create_key_with_specific_id, sse_customer_key_md5_base64};
use crate::storage_api::replication_extension::BucketTargetSys;
@@ -628,6 +629,17 @@ async fn put_bucket_replication_with_delete_statuses(
target_arn: &str,
delete_marker_status: &str,
version_delete_status: Option<&str>,
) -> Result<(), Box<dyn Error + Send + Sync>> {
put_bucket_replication_with_statuses(env, bucket, target_arn, delete_marker_status, version_delete_status, "Enabled").await
}
async fn put_bucket_replication_with_statuses(
env: &RustFSTestEnvironment,
bucket: &str,
target_arn: &str,
delete_marker_status: &str,
version_delete_status: Option<&str>,
existing_object_status: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let delete_replication = version_delete_status
.map(|status| format!("<DeleteReplication><Status>{status}</Status></DeleteReplication>"))
@@ -644,7 +656,7 @@ async fn put_bucket_replication_with_delete_statuses(
</DeleteMarkerReplication>
{delete_replication}
<ExistingObjectReplication>
<Status>Enabled</Status>
<Status>{existing_object_status}</Status>
</ExistingObjectReplication>
<Destination>
<Bucket>{target_arn}</Bucket>
@@ -2594,6 +2606,9 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
assert_eq!(payload["Targets"][0]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
// A RustFS target adopts the source version id, so the P1-19
// version-identity probe passes.
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
@@ -7487,3 +7502,897 @@ async fn test_scanner_never_cascades_inbound_replicas() -> TestResult {
Ok(())
}
/// P1-19 review follow-up: multipart fixes the target version at initiate
/// and only reports it on completion, so a target can adopt PutObject
/// version ids and still mint its own there — the check must not report OK
/// while multipart deletes and heals would silently miss.
#[tokio::test]
#[serial]
async fn test_replication_check_flags_multipart_only_version_minting_target() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "multipart-fidelity-dst";
target.create_bucket(target_bucket);
// PutObject mirrors the source version id; CreateMultipartUpload does not.
target.assign_own_multipart_version_ids(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "multipart-fidelity-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(payload["Status"], "FAILED", "multipart drift must fail the check: {payload}");
let target_report = &payload["Targets"][0];
let fidelity = &target_report["Phases"]["VersionFidelity"];
assert_eq!(fidelity["Status"], "FAILED", "{payload}");
assert_eq!(fidelity["Code"], "BucketRemoteTargetVersionMismatch", "{payload}");
assert!(
fidelity["Error"]
.as_str()
.is_some_and(|error| error.contains("CreateMultipartUpload")),
"the failure must name the multipart path: {payload}"
);
// The PutObject leg mirrored, so it is the multipart probe that failed.
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "SKIPPED", "{payload}");
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
let probe_key = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::PutObject)
.and_then(|record| record.key)
.ok_or("the probe PUT never reached the fake target")?;
assert!(
target.stored_versions(target_bucket, &probe_key).is_empty(),
"both probe versions must be cleaned up on the mismatching target"
);
target.shutdown().await;
Ok(())
}
#[tokio::test]
#[serial]
async fn test_replication_check_aborts_failed_multipart_probes() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "multipart-cleanup-dst";
target.create_bucket(target_bucket);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "multipart-cleanup-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
for failed_operation in [FakeTargetOperation::UploadPart, FakeTargetOperation::CompleteMultipartUpload] {
target.clear_faults();
target.take_requests();
target.inject(failed_operation, FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE), 16);
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(
payload["Status"], "FAILED",
"the injected multipart failure must fail the check: {payload}"
);
let requests = target.requests();
assert!(
requests.iter().any(|request| {
request.operation == failed_operation
&& request.fault == Some(FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE))
}),
"the check must reach the injected {failed_operation:?} failure: {requests:?}"
);
assert!(
requests
.iter()
.any(|request| request.operation == FakeTargetOperation::AbortMultipartUpload),
"the failed {failed_operation:?} probe must be aborted: {requests:?}"
);
assert_eq!(
target.active_multipart_upload_count(),
0,
"the failed {failed_operation:?} probe must not leave multipart state"
);
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
}
target.clear_faults();
target.take_requests();
target.inject(FakeTargetOperation::CompleteMultipartUpload, FakeTargetFault::DisconnectAfterResponse, 16);
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "FAILED", "{payload}");
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "FAILED", "{payload}");
assert_eq!(
target_report["Phases"]["Cleanup"]["Status"], "OK",
"NoSuchUpload after an ambiguous complete means the multipart artifact is gone: {payload}"
);
let requests = target.requests();
let completed_key = requests
.iter()
.find(|request| {
request.operation == FakeTargetOperation::CompleteMultipartUpload
&& request.fault == Some(FakeTargetFault::DisconnectAfterResponse)
})
.and_then(|request| request.key.as_deref())
.expect("the scripted complete response disconnect must be observed");
assert!(
requests
.iter()
.any(|request| request.operation == FakeTargetOperation::AbortMultipartUpload),
"the ambiguous complete must still attempt abort: {requests:?}"
);
assert_eq!(target.active_multipart_upload_count(), 0);
assert!(
target.stored_versions(target_bucket, completed_key).is_empty(),
"outer cleanup must remove the object committed before the response disconnect"
);
target.clear_faults();
target.take_requests();
target.inject(FakeTargetOperation::UploadPart, FakeTargetFault::Status(StatusCode::FORBIDDEN), 16);
target.inject(
FakeTargetOperation::AbortMultipartUpload,
FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE),
16,
);
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "FAILED", "{payload}");
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "FAILED", "{payload}");
assert_eq!(
target_report["Phases"]["Cleanup"]["Status"], "FAILED",
"an unremoved multipart probe must be reported as a cleanup failure: {payload}"
);
assert_eq!(
target_report["Error"], "s3:ReplicateObject permissions missing for replication user",
"the primary multipart error must remain the target error: {payload}"
);
assert_eq!(
target_report["Phases"]["VersionFidelity"]["Error"], "s3:ReplicateObject permissions missing for replication user",
"{payload}"
);
assert_eq!(
target_report["Phases"]["Cleanup"]["Error"], "failed to abort multipart replication probe",
"{payload}"
);
assert!(
target.requests().iter().any(|request| {
request.operation == FakeTargetOperation::AbortMultipartUpload
&& request.fault == Some(FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE))
}),
"the abort failure must be observed"
);
assert_eq!(
target.active_multipart_upload_count(),
1,
"the report must match the retained multipart state"
);
target.shutdown().await;
Ok(())
}
/// P1-19 (backlog#1675): the supported replication contract is targets that
/// adopt the source version id (RustFS/MinIO semantics). A target that mints
/// its own version ids silently breaks every version-addressed operation that
/// follows — version deletes and heal re-drives never match, diverging the
/// two sides. replication-check must surface this explicitly: a
/// VersionFidelity phase that compares the probe PUT's response version id
/// against the sent source version id and fails with
/// BucketRemoteTargetVersionMismatch — while still cleaning up the probe
/// object via the version id the target actually assigned.
#[tokio::test]
#[serial]
async fn test_replication_check_flags_version_minting_target() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "version-fidelity-dst";
target.create_bucket(target_bucket);
target.assign_own_version_ids(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "version-fidelity-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(
payload["Status"], "FAILED",
"a version-minting target must fail the replication check: {payload}"
);
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "FAILED", "target must be FAILED: {payload}");
let fidelity = &target_report["Phases"]["VersionFidelity"];
assert_eq!(fidelity["Status"], "FAILED", "VersionFidelity phase must fail: {payload}");
assert_eq!(
fidelity["Code"], "BucketRemoteTargetVersionMismatch",
"the failure must carry a machine-readable code: {payload}"
);
// The probe PUT itself succeeded (fidelity is judged from its response);
// the later mutation phases are pointless against a drifting target and
// must be skipped, but cleanup still runs.
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "SKIPPED", "{payload}");
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
// The probe PUT must carry the source version as `?versionId=` — the
// exact shape live replication uses (P0-5), and the only shape MinIO
// consumes. The journal records the query value.
let probe_put = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::PutObject)
.ok_or("the probe PUT never reached the fake target")?;
let probe_query_version = probe_put
.version_id
.as_deref()
.ok_or("the probe PUT must carry a versionId query")?;
assert!(
uuid::Uuid::parse_str(probe_query_version).is_ok(),
"the probe versionId query must be the source uuid, got {probe_query_version}"
);
// No probe residue: cleanup must address the version id the target
// actually assigned, not the source id (which never matched anything).
let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
assert!(
target.stored_versions(target_bucket, &probe_key).is_empty(),
"the probe object must be cleaned up on the mismatching target"
);
Ok(())
}
// --- P1-21 (backlog#1675): delayed delete-marker purge failure handling ---
//
// The fixtures below wire a versioned source bucket to a FakeS3Target with the
// default replication shape: DeleteMarkerReplication=Enabled and
// DeleteReplication omitted. With version-delete replication unconfigured,
// purging the source marker version emits no replication event, and the data
// scanner cannot see a source version that is gone — the delayed purge watcher
// spawned by the marker replication is the ONLY channel that can remove the
// replicated marker from the target.
const DELAYED_PURGE_KEY: &str = "doc.txt";
fn delayed_purge_process_env() -> Vec<(&'static str, &'static str)> {
let mut env = replication_fast_env();
env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
env
}
async fn start_delayed_purge_fixture(
source_bucket: &str,
target_bucket: &str,
) -> Result<(FakeS3Target, RustFSTestEnvironment, Client), Box<dyn Error + Send + Sync>> {
let target = FakeS3Target::start().await?;
target.create_bucket(target_bucket);
let mut source_env = RustFSTestEnvironment::new().await?;
source_env
.start_rustfs_server_with_env(vec![], &delayed_purge_process_env())
.await?;
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
Ok((target, source_env, source_client))
}
/// PUT an object, stack a delete marker on it, and wait until the fake target
/// stores the marker replica. Returns the source marker version id — the fake
/// target mirrors it because delete replication forwards
/// `x-*-source-version-id`.
///
/// Timing budget for callers: the delayed purge watcher only observes the
/// source for ~4s after the marker replication completes, so the source-side
/// marker-version DELETE must be issued promptly after this returns (the
/// 100ms journal poll below keeps the detection latency small).
async fn replicate_delete_marker(
target: &FakeS3Target,
target_bucket: &str,
source_client: &Client,
source_bucket: &str,
) -> Result<String, Box<dyn Error + Send + Sync>> {
source_client
.put_object()
.bucket(source_bucket)
.key(DELAYED_PURGE_KEY)
.body(ByteStream::from_static(b"delayed purge payload"))
.send()
.await?;
let delete = source_client
.delete_object()
.bucket(source_bucket)
.key(DELAYED_PURGE_KEY)
.send()
.await?;
assert_eq!(delete.delete_marker(), Some(true), "unversioned DELETE must create a marker");
let marker_version = delete
.version_id()
.ok_or("source DELETE omitted the marker version ID")?
.to_string();
// Wait for ANY delete marker: a target that mints its own version ids
// does not mirror the source one.
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let replicated = target
.stored_versions(target_bucket, DELAYED_PURGE_KEY)
.iter()
.any(|(_, delete_marker)| *delete_marker);
if replicated {
return Ok(marker_version);
}
if tokio::time::Instant::now() >= deadline {
return Err(
format!("fake target never stored the replicated delete marker; journal: {:?}", target.requests()).into(),
);
}
sleep(Duration::from_millis(100)).await;
}
}
/// Journal records of purge attempts: target DELETE calls addressing the marker
/// version explicitly. The marker-creation replica DELETE carries no
/// `versionId` query, so the version id is an exact discriminator.
fn delayed_purge_attempts(target: &FakeS3Target, marker_version: &str) -> Vec<RequestRecord> {
target
.requests()
.into_iter()
.filter(|record| {
record.operation == FakeTargetOperation::DeleteObject
&& record.key.as_deref() == Some(DELAYED_PURGE_KEY)
&& record.version_id.as_deref() == Some(marker_version)
})
.collect()
}
async fn wait_for_target_marker_purged(
target: &FakeS3Target,
target_bucket: &str,
max_wait: Duration,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let deadline = tokio::time::Instant::now() + max_wait;
loop {
let marker_present = target
.stored_versions(target_bucket, DELAYED_PURGE_KEY)
.iter()
.any(|(_, delete_marker)| *delete_marker);
if !marker_present {
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!(
"target delete marker was never purged; target state: {:?}",
target.stored_versions(target_bucket, DELAYED_PURGE_KEY)
)
.into());
}
sleep(Duration::from_millis(200)).await;
}
}
/// P1-21: the delayed purge's single target DELETE currently swallows failures
/// (`let _ =`), so one transient target error strands the replicated marker on
/// the target forever. Contract under test: a failed purge attempt is retried
/// within the watch window and converges once the fault clears.
#[tokio::test]
#[serial]
async fn test_delayed_delete_marker_purge_retries_after_transient_target_failure() -> TestResult {
init_logging();
let source_bucket = "delayed-purge-retry-src";
let target_bucket = "delayed-purge-retry-dst";
let (target, _source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
// Four scripted failures. Fault budget accounting (each journal record
// consumes one fault, including the SDK's own per-request retries):
// deleting the marker version fans out over the version-purge replication
// channel (initial attempt + its fast in-memory MRF retries) plus the
// delayed purge watcher's single pre-fix attempt — three target DELETE calls
// in total today, empirically (see the exhaustion test's journal). Four
// faults outlast all of them, so only a delayed-purge retry in a later
// watch round can converge. If the SDK retry configuration ever changes,
// re-derive this budget from a fresh journal capture.
target.inject(
FakeTargetOperation::DeleteObject,
FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE),
4,
);
// Purge the marker at the source. The watcher spawned when the marker
// replication completed moments ago observes the source marker vanish
// within its watch window and drives the target purge.
source_client
.delete_object()
.bucket(source_bucket)
.key(DELAYED_PURGE_KEY)
.version_id(&marker_version)
.send()
.await?;
// Tight window on purpose: a fixed delayed purge retries on 1s rounds and
// converges within ~5s, while any straggling backoff retry from the other
// channels would land later and must not be what turns this test green.
wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(15)).await?;
let attempts = delayed_purge_attempts(&target, &marker_version);
assert!(
attempts.len() >= 2,
"expected the faulted purge attempt plus at least one retry, got: {attempts:?}"
);
assert!(
attempts.iter().any(|record| record.fault.is_none()),
"expected a clean purge attempt after the fault script drained, got: {attempts:?}"
);
target.shutdown().await;
Ok(())
}
/// P1-21 review follow-up: the watcher must purge the version the TARGET
/// assigned to the replicated marker, not one derived from the source uuid.
/// A target that mints its own version ids answers a source-derived purge
/// with an idempotent 204, which used to look like success and strand the
/// real marker on the target forever.
#[tokio::test]
#[serial]
async fn test_delayed_delete_marker_purge_uses_target_assigned_version() -> TestResult {
init_logging();
let source_bucket = "delayed-purge-mint-src";
let target_bucket = "delayed-purge-mint-dst";
let (target, _source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
// The target ignores the forwarded source-version-id header and mints its
// own ids for both the object and the replicated delete marker.
target.assign_own_version_ids(true);
let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
source_client
.delete_object()
.bucket(source_bucket)
.key(DELAYED_PURGE_KEY)
.version_id(&marker_version)
.send()
.await?;
// The replicated marker carries a target-minted version id, so nothing
// but the recorded mapping can address it.
wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(25)).await?;
target.shutdown().await;
Ok(())
}
/// P1-21: when every watch-window purge attempt fails, the purge intent must
/// survive as a durable MRF entry and replay on the next startup; once the
/// replayed purge succeeds, the entry must be acknowledged instead of being
/// retained as Missed forever.
#[tokio::test]
#[serial]
async fn test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart() -> TestResult {
init_logging();
let source_bucket = "delayed-purge-mrf-src";
let target_bucket = "delayed-purge-mrf-dst";
let (target, mut source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
// Outlast the whole watch window: every in-process purge attempt fails.
target.inject(
FakeTargetOperation::DeleteObject,
FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE),
64,
);
source_client
.delete_object()
.bucket(source_bucket)
.key(DELAYED_PURGE_KEY)
.version_id(&marker_version)
.send()
.await?;
// Let the watch window drain before restarting. The wall-clock length is
// not 5x1s: every faulted attempt embeds the SDK's own per-request 503
// retries (a few seconds each), so instead of a fixed sleep, wait until
// the faulted attempts stop arriving (the watcher exhausted its rounds and
// persisted the purge intent), then give the MRF persister its 100ms
// flush interval.
let deadline = tokio::time::Instant::now() + Duration::from_secs(60);
let mut last_seen = delayed_purge_attempts(&target, &marker_version).len();
let mut quiet_since = tokio::time::Instant::now();
loop {
sleep(Duration::from_millis(500)).await;
let seen = delayed_purge_attempts(&target, &marker_version).len();
if seen != last_seen {
last_seen = seen;
quiet_since = tokio::time::Instant::now();
}
if last_seen > 0 && quiet_since.elapsed() >= Duration::from_secs(5) {
break;
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("purge attempts never quiesced (saw {last_seen}); journal: {:?}", target.requests()).into());
}
}
sleep(Duration::from_secs(1)).await;
let marker_survives_faults = target
.stored_versions(target_bucket, DELAYED_PURGE_KEY)
.iter()
.any(|(_, delete_marker)| *delete_marker);
assert!(marker_survives_faults, "scripted faults must have blocked every in-process purge attempt");
target.clear_faults();
let attempts_before_restart = delayed_purge_attempts(&target, &marker_version).len();
// Startup MRF replay must re-drive the purge and clean the target.
source_env
.restart_server_preserving_data(vec![], &delayed_purge_process_env())
.await?;
wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(30)).await?;
let attempts_after_replay = delayed_purge_attempts(&target, &marker_version).len();
assert!(
attempts_after_replay > attempts_before_restart,
"the restart replay must have issued the purge DELETE"
);
// The successful replay must acknowledge the MRF entry: another restart may
// not re-drive the purge again.
source_env
.restart_server_preserving_data(vec![], &delayed_purge_process_env())
.await?;
sleep(Duration::from_secs(5)).await;
assert_eq!(
delayed_purge_attempts(&target, &marker_version).len(),
attempts_after_replay,
"acknowledged purge-intent MRF entries must not replay again"
);
target.shutdown().await;
Ok(())
}
// --- P1-20 (backlog#1675): scanner existing-object compensation matrix ---
//
// Every case below inverts the order used by the rest of this file: objects
// are written FIRST and the replication rule arrives afterwards, so the only
// channel that can move the pre-existing objects is the data scanner's
// existing-object resync pass. Negative cells ("never compensated") are
// contracts and are asserted over multiple scanner cycles, always next to a
// replicated control key that proves the scanner and the live path are
// running — an absent key on a dead scanner proves nothing.
/// Envs + buckets only: versioning, the remote target, and the rule variant
/// are wired by each test (the null-version case must PUT before the source
/// bucket becomes versioned). The source runs with FAST_SCANNER_ENV so
/// existing keys are rescanned within seconds instead of 16 dir cycles.
async fn build_scanner_compensation_pair(
source_bucket: &str,
target_bucket: &str,
) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), Box<dyn Error + Send + Sync>> {
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_process_env = replication_fast_env();
source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_process_env.extend_from_slice(FAST_SCANNER_ENV);
source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env.start_rustfs_server_without_cleanup(vec![]).await?;
source_env
.create_s3_client()
.create_bucket()
.bucket(source_bucket)
.send()
.await?;
target_env
.create_s3_client()
.create_bucket()
.bucket(target_bucket)
.send()
.await?;
Ok((source_env, target_env))
}
/// P1-20: objects that already exist when a rule with
/// ExistingObjectReplication=Enabled arrives are compensated by the scanner's
/// existing-object resync pass, whatever wrote them — plain PUT, CopyObject,
/// or Snowball auto-extract. The pinned exception is a null-version object
/// (written before the bucket became versioned): the scanner heal gate skips
/// nil-version objects entirely (`scanner_folder.rs` heal_replication), so it
/// must NEVER be compensated.
#[tokio::test]
#[serial]
async fn test_scanner_compensates_existing_objects_across_write_paths() -> TestResult {
init_logging();
let source_bucket = "scanner-comp-src";
let target_bucket = "scanner-comp-dst";
let (source_env, target_env) = build_scanner_compensation_pair(source_bucket, target_bucket).await?;
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
// Null-version cell: PUT before versioning; the object keeps the nil
// version id forever.
let null_key = "pre-versioning-null.txt";
source_client
.put_object()
.bucket(source_bucket)
.key(null_key)
.body(ByteStream::from_static(b"null version payload"))
.send()
.await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
// Pre-existing objects from three write paths, all before any replication
// config exists (their replication status stays Empty).
let plain_key = "existing-plain.txt";
let plain_payload = "existing plain payload";
source_client
.put_object()
.bucket(source_bucket)
.key(plain_key)
.body(ByteStream::from_static(plain_payload.as_bytes()))
.send()
.await?;
let copy_key = "existing-copy.txt";
source_client
.copy_object()
.bucket(source_bucket)
.key(copy_key)
.copy_source(format!("{source_bucket}/{plain_key}"))
.send()
.await?;
let member_key = "snowball/existing-member.txt";
let member_payload: &[u8] = b"existing snowball member payload";
let mut builder = tokio_tar::Builder::new(std::io::Cursor::new(Vec::new()));
let mut header = tokio_tar::Header::new_gnu();
header.set_size(member_payload.len() as u64);
header.set_mode(0o644);
header.set_cksum();
builder
.append_data(&mut header, member_key, std::io::Cursor::new(member_payload))
.await?;
let archive = builder.into_inner().await?.into_inner();
source_client
.put_object()
.bucket(source_bucket)
.key("existing-members.tar")
.metadata("Snowball-Auto-Extract", "true")
.body(ByteStream::from(archive))
.send()
.await?;
// The extracted member must exist locally before the rule arrives, or it
// would replicate through the live path instead of the scanner.
let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
if source_client
.head_object()
.bucket(source_bucket)
.key(member_key)
.send()
.await
.is_ok()
{
break;
}
if tokio::time::Instant::now() >= deadline {
return Err("snowball member was never extracted on the source".into());
}
sleep(Duration::from_millis(200)).await;
}
// Only now wire the remote target and the Enabled rule.
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
// Control key written after the rule replicates through the live path.
let control_key = "control-live.txt";
let control_payload = "control live payload";
source_client
.put_object()
.bucket(source_bucket)
.key(control_key)
.body(ByteStream::from_static(control_payload.as_bytes()))
.send()
.await?;
wait_for_replicated_object(&target_client, target_bucket, control_key, control_payload).await?;
// Scanner compensation for each pre-existing write path.
wait_for_replicated_object(&target_client, target_bucket, plain_key, plain_payload).await?;
wait_for_replicated_object(&target_client, target_bucket, copy_key, plain_payload).await?;
wait_for_replicated_object(&target_client, target_bucket, member_key, std::str::from_utf8(member_payload)?).await?;
// Null-version contract: with every sibling compensated (scanner proven
// live), the nil-version object must stay absent across further cycles.
assert_replication_key_absent(&target_client, target_bucket, null_key, Duration::from_secs(6)).await?;
Ok(())
}
/// P1-20: ExistingObjectReplication=Disabled is a contract, not a delay — the
/// scanner must NEVER compensate objects that predate the rule, while objects
/// written after the rule replicate normally (the setting only gates the
/// existing-object resync path).
#[tokio::test]
#[serial]
async fn test_scanner_never_compensates_when_existing_object_replication_disabled() -> TestResult {
init_logging();
let source_bucket = "scanner-disabled-src";
let target_bucket = "scanner-disabled-dst";
let (source_env, mut target_env) = build_scanner_compensation_pair(source_bucket, target_bucket).await?;
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
let existing_key = "existing-disabled.txt";
source_client
.put_object()
.bucket(source_bucket)
.key(existing_key)
.body(ByteStream::from_static(b"existing disabled payload"))
.send()
.await?;
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication_with_statuses(&source_env, source_bucket, &target_arn, "Enabled", None, "Disabled").await?;
// The live path is unaffected by the Disabled existing-object setting.
let control_key = "control-live.txt";
let control_payload = "control live payload";
source_client
.put_object()
.bucket(source_bucket)
.key(control_key)
.body(ByteStream::from_static(control_payload.as_bytes()))
.send()
.await?;
wait_for_replicated_object(&target_client, target_bucket, control_key, control_payload).await?;
// Scanner-only witness. A live-path control key alone would let this test
// pass while the existing-object scanner is disabled or wedged, so make
// the scanner itself observable: an object whose replication FAILED while
// the target was down can only be re-driven by the data scanner's
// replication heal pass (see FAST_SCANNER_ENV), and that pass is NOT
// gated by ExistingObjectReplication. The witness lives in the same
// bucket and prefix as the pre-existing key, so a heal pass that reached
// it necessarily walked the pre-existing key in the same scan.
let witness_key = "scanner-witness.txt";
let witness_payload = "scanner witness payload";
target_env.stop_server();
source_client
.put_object()
.bucket(source_bucket)
.key(witness_key)
.body(ByteStream::from_static(witness_payload.as_bytes()))
.send()
.await?;
wait_for_source_replication_status(&source_client, source_bucket, witness_key, "FAILED", false).await?;
target_env.restart_server_preserving_data(vec![], &[]).await?;
let target_client = target_env.create_s3_client();
wait_for_replicated_object(&target_client, target_bucket, witness_key, witness_payload).await?;
// The scanner demonstrably swept this bucket; the pre-existing key must
// still be absent, and stay absent over further cycles.
assert_replication_key_absent(&target_client, target_bucket, existing_key, Duration::from_secs(6)).await?;
Ok(())
}
+2 -2
View File
@@ -32,7 +32,7 @@ pub mod bucket {
pub mod bucket_target_sys {
pub use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
TargetClient,
TargetClient, append_version_id_query,
};
}
@@ -281,7 +281,7 @@ pub mod config {
pub mod com {
pub use crate::config::com::{
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
@@ -1450,7 +1450,7 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
/// member, so the query is spliced in via `map_request`, which runs at
/// `modify_before_signing`: the parameter becomes part of the SigV4 canonical
/// request.
fn append_version_id_query(uri: &str, version_id: &str) -> String {
pub fn append_version_id_query(uri: &str, version_id: &str) -> String {
let separator = if uri.contains('?') { '&' } else { '?' };
format!("{uri}{separator}versionId={}", urlencoding::encode(version_id))
}
@@ -1861,6 +1861,9 @@ impl TargetClient {
}
}
/// On success returns the version id the target assigned (from
/// `x-amz-version-id`), letting callers audit the version-identity
/// contract — a target that adopts the source version echoes it back.
pub async fn put_object(
&self,
bucket: &str,
@@ -1868,7 +1871,7 @@ impl TargetClient {
size: i64,
body: ByteStream,
opts: &PutObjectOptions,
) -> Result<(), S3ClientError> {
) -> Result<Option<String>, S3ClientError> {
let mut headers = opts.header();
let builder = self.client.put_object();
@@ -1903,7 +1906,7 @@ impl TargetClient {
.send()
.await
{
Ok(_) => Ok(()),
Ok(output) => Ok(output.version_id().map(ToOwned::to_owned)),
Err(e) => match e {
SdkError::ServiceError(service_err) => {
let err = service_err.into_err();
@@ -2567,6 +2567,12 @@ pub trait ReplicationPoolTrait: std::fmt::Debug {
async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission;
async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission;
async fn queue_replica_delete_batch(&self, deletes: &[DeletedObjectReplicationInfo]) -> ReplicationBatchAdmission;
/// Persist one entry straight to the durable MRF journal, bypassing the
/// live worker queues. For failures whose source state is already gone —
/// e.g. exhausted delete-marker purges — where only a startup replay can
/// retry, and live re-dispatch would loop unboundedly against a down
/// target.
async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission;
async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize);
async fn get_bucket_resync_status(&self, bucket: &str) -> Result<BucketReplicationResyncStatus, EcstoreError>;
async fn cancel_bucket_resync(&self, opts: ResyncOpts) -> Result<(), EcstoreError>;
@@ -2607,6 +2613,10 @@ impl<S: ReplicationStorage> ReplicationPoolTrait for ReplicationPool<S> {
self.queue_replica_delete_batch(deletes).await
}
async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission {
self.queue_mrf_save_admission(entry, "delete_marker_purge").await
}
async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize) {
self.resize(priority, max_workers, max_l_workers).await;
}
@@ -18,9 +18,10 @@ use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Result, is_err_object_not_found, is_err_version_not_found};
use super::replication_event_sink::{EventArgs, send_event, send_local_event};
use super::replication_filemeta_boundary::{
NULL_VERSION_ID, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicatedInfos,
ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType, VersionPurgeStatusType,
get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
MrfReplicateEntry, NULL_VERSION_ID, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType,
ReplicationWorkerOperation, VersionPurgeStatusType, get_replication_state, parse_replicate_decision,
replication_statuses_map, target_reset_header, version_purge_statuses_map,
};
use super::replication_lock_boundary::ReplicationLockTiming;
use super::replication_logging::{EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REPLICATION_RESYNC};
@@ -33,7 +34,7 @@ use super::replication_object_decision_boundary::{
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, should_retry_delete_marker_purge,
};
use super::replication_queue_boundary::DeletedObjectReplicationInfo;
use super::replication_queue_boundary::{DeletedObjectReplicationInfo, ReplicationQueueAdmission};
use super::replication_resync_boundary::ResyncStatusType;
use super::replication_resync_boundary::{
BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus, encode_resync_file, is_version_id_mismatch,
@@ -63,6 +64,7 @@ use futures::stream::StreamExt;
use http::HeaderMap;
use http_body::Frame;
use http_body_util::StreamBody;
use metrics::counter;
#[cfg(test)]
use rmp_serde;
use rustfs_s3_types::EventName;
@@ -72,10 +74,10 @@ use rustfs_utils::http::{
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
#[cfg(test)]
use s3s::dto::ReplicationConfiguration;
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, LazyLock, Mutex as StdMutex};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use tokio::io::AsyncRead;
@@ -100,6 +102,10 @@ const EVENT_REPLICATION_FORCE_DELETE_SKIPPED: &str = "replication_force_delete_s
const EVENT_RESYNC_TASK_FAILED: &str = "replication_resync_task_failed";
const EVENT_RESYNC_TARGET_OPERATION_FAILED: &str = "replication_resync_target_operation_failed";
const EVENT_RESYNC_RUNTIME_CHANNEL_FAILED: &str = "replication_resync_runtime_channel_failed";
const EVENT_DELETE_MARKER_PURGE_FAILED: &str = "replication_delete_marker_purge_failed";
const EVENT_DELETE_MARKER_PURGE_MRF: &str = "replication_delete_marker_purge_mrf";
const METRIC_DELETE_MARKER_PURGE_TOTAL: &str = "rustfs_replication_delete_marker_purge_total";
const EVENT_REPLICATION_VERSION_IDENTITY_DRIFT: &str = "replication_version_identity_drift";
const REPLICATION_TARGET_OFFLINE_ERROR_MARKERS: &[&str] = &[
"dispatch failure",
"timeouterror",
@@ -181,6 +187,57 @@ fn is_head_proxy_failure(err: &SdkError<HeadObjectError>) -> bool {
should_count_head_proxy_failure(is_not_found, code, raw_status)
}
const METRIC_VERSION_IDENTITY_DRIFT_TOTAL: &str = "rustfs_replication_version_identity_drift_total";
/// Targets that already produced a version-identity-drift warning this
/// process lifetime, by ARN. Deduping is advisory only (the metric still
/// counts every drifting PUT), so a reconfigured target re-warning only
/// after a restart is acceptable.
static VERSION_IDENTITY_WARNED_ARNS: LazyLock<StdMutex<HashSet<String>>> = LazyLock::new(|| StdMutex::new(HashSet::new()));
/// Runtime half of the P1-19 version-identity contract (the explicit probe
/// lives in replication-check's VersionFidelity phase): every replication PUT
/// response reveals whether the target adopted the source version id. A
/// target minting its own ids silently breaks version-addressed deletes and
/// heal, so surface it — once per target — instead of letting the divergence
/// accumulate unseen.
/// Pure drift judgment: the contract only applies when the source addressed a
/// real (non-nil) version uuid, and drift means the target answered with
/// anything else — including nothing at all.
fn version_identity_drifted(source_version_id: &str, assigned_version_id: Option<&str>) -> bool {
if source_version_id.is_empty() {
return false;
}
// A nil source uuid travels as the literal "null" (unversioned-source
// semantics); no identity contract applies to it.
if Uuid::parse_str(source_version_id).map(|uuid| uuid.is_nil()).unwrap_or(true) {
return false;
}
assigned_version_id != Some(source_version_id)
}
fn audit_target_version_identity(tgt_client: &TargetClient, source_version_id: &str, assigned_version_id: Option<&str>) {
if !version_identity_drifted(source_version_id, assigned_version_id) {
return;
}
counter!(METRIC_VERSION_IDENTITY_DRIFT_TOTAL).increment(1);
let mut warned = VERSION_IDENTITY_WARNED_ARNS
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
if warned.insert(tgt_client.arn.clone()) {
warn!(
event = EVENT_REPLICATION_VERSION_IDENTITY_DRIFT,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
arn = %tgt_client.arn,
endpoint = %tgt_client.endpoint,
sent_version_id = %source_version_id,
assigned_version_id = assigned_version_id.unwrap_or("<none>"),
"Replication target does not adopt source version ids; version-addressed replication cannot converge (run ?replication-check for details)"
);
}
}
async fn record_proxy_request(bucket: &str, api: &str, is_err: bool) {
if let Some(stats) = runtime_sources::replication_stats() {
stats.inc_proxy(bucket, api, is_err).await;
@@ -1271,7 +1328,12 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
reason = "source_version_missing",
"Skipping stale delete-marker replication"
);
return true;
// The marker is gone at the source, but a replica of it may
// already exist on the targets (a live race, or an MRF
// purge-intent replay landing here on purpose). Purge instead
// of just skipping; the result decides whether an MRF replay
// may acknowledge the entry.
return purge_stale_delete_marker_targets(&bucket, &dobj).await;
}
Err(err) => {
source_state_verified = false;
@@ -1485,29 +1547,6 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
let is_version_purge = is_version_delete_replication(&dobj.delete_object);
let requires_delayed_purge = should_retry_delete_marker_purge(&dobj.delete_object);
if requires_delayed_purge {
let bucket_clone = bucket.clone();
let dobj_clone = dobj.clone();
let dsc_clone = dsc.clone();
let storage_clone = storage.clone();
tokio::spawn(async move {
for _ in 0..5 {
if let Some(delete_marker_version_id) = dobj_clone.delete_object.delete_marker_version_id
&& source_delete_marker_missing(
&*storage_clone,
&bucket_clone,
&dobj_clone.delete_object.object_name,
delete_marker_version_id,
)
.await
{
replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await;
break;
}
tokio::time::sleep(TokioDuration::from_secs(1)).await;
}
});
}
let (replication_status, prev_status) = if !is_version_purge {
(
@@ -1550,6 +1589,24 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
drs.replication_timestamp = Some(OffsetDateTime::now_utc());
}
if requires_delayed_purge {
// Hand the watcher the MERGED replication state: `drs` folds this
// round's per-target results into the previous state, including the
// version ids the targets assigned to the markers they just created.
// Spawning with the pre-merge `dobj` made the purge fall back to a
// source-derived id, which a target that mints its own ids answers
// with an idempotent 204 — the intent was then dropped while the
// real marker stayed behind.
let bucket_clone = bucket.clone();
let mut dobj_clone = dobj.clone();
dobj_clone.delete_object.replication_state = Some(drs.clone());
let dsc_clone = dsc.clone();
let storage_clone = storage.clone();
tokio::spawn(async move {
watch_and_purge_source_delete_marker(bucket_clone, dobj_clone, dsc_clone, storage_clone).await;
});
}
let event_name = if replication_status == ReplicationStatusType::Completed {
EventName::ObjectReplicationComplete.to_string()
} else {
@@ -1608,12 +1665,36 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
}
};
replicate_delete_outcome(
expected_targets,
rinfos.targets.len(),
state_persisted,
source_state_verified,
&replication_status,
)
}
/// Whether a delete replication fully succeeded — the MRF replay acknowledges
/// (drops) an entry exactly when this returns true.
///
/// The delayed purge is deliberately NOT an input: holding the outcome hostage
/// to it (`&& !requires_delayed_purge`) forced `false` for every delete-marker
/// entry and retained them all in the durable MRF journal forever. Purge
/// failures persist their own purge-intent entry instead
/// (`watch_and_purge_source_delete_marker`), and replays of those entries
/// report purge success through `purge_stale_delete_marker_targets`.
fn replicate_delete_outcome(
expected_targets: usize,
replicated_targets: usize,
state_persisted: bool,
source_state_verified: bool,
replication_status: &ReplicationStatusType,
) -> bool {
expected_targets > 0
&& rinfos.targets.len() == expected_targets
&& replicated_targets == expected_targets
&& state_persisted
&& source_state_verified
&& !requires_delayed_purge
&& replication_status == ReplicationStatusType::Completed
&& *replication_status == ReplicationStatusType::Completed
}
async fn source_delete_marker_missing<S: EcstoreObjectOperations>(
@@ -1663,48 +1744,286 @@ fn delete_marker_purge_version_id(
})
}
async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo, dsc: &ReplicateDecision) {
/// One purge pass over the eligible targets. Returns the ARNs that must be
/// retried: the remote DELETE failed, or the target client was unavailable
/// (e.g. a runtime cache miss). Inconsistent recorded version mappings are a
/// deliberate refusal — retrying cannot make guessing a version id safe — so
/// they are logged and excluded from the retry set.
async fn replicate_delete_marker_purge_to_targets(
bucket: &str,
dobj: &DeletedObjectReplicationInfo,
dsc: &ReplicateDecision,
retry_arns: Option<&[String]>,
) -> Vec<String> {
let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
return;
return Vec::new();
};
let target_arns = dobj.admitted_target_arns();
let mut failed_arns = Vec::new();
for tgt_entry in dsc.targets_map.values() {
if !tgt_entry.replicate {
continue;
}
let target_arns = dobj.admitted_target_arns();
if !target_arns.is_empty() && !target_arns.iter().any(|arn| arn == &tgt_entry.arn) {
continue;
}
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
if let Some(retry_arns) = retry_arns
&& !retry_arns.iter().any(|arn| arn == &tgt_entry.arn)
{
continue;
};
}
// Decide the version first: refusing to guess is a per-target
// FAILURE, not a silent skip. Reporting it as success would let the
// watcher and the MRF replay drop the purge intent while the marker
// is still on the target — the leak stays visible instead (the
// entry is retained and keeps warning) until an operator repairs
// the metadata.
let Some(purge_version_id) = delete_marker_purge_version_id(
dobj.delete_object.replication_state.as_ref(),
&tgt_entry.arn,
delete_marker_version_id,
) else {
warn!(
event = EVENT_DELETE_MARKER_PURGE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket,
object = dobj.delete_object.object_name,
arn = tgt_entry.arn,
"Skipping delete-marker purge: recorded target version metadata is inconsistent"
reason = "recorded_target_version_inconsistent",
"Delete-marker purge refused: recorded target version metadata is inconsistent"
);
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "refused").increment(1);
failed_arns.push(tgt_entry.arn.clone());
continue;
};
let _ = tgt_client
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
warn!(
event = EVENT_DELETE_MARKER_PURGE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket,
object = dobj.delete_object.object_name,
arn = tgt_entry.arn,
reason = "target_client_missing",
"Delete-marker purge attempt failed"
);
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "failed").increment(1);
failed_arns.push(tgt_entry.arn.clone());
continue;
};
match tgt_client
.remove_object(
&tgt_client.bucket,
&dobj.delete_object.object_name,
purge_version_id,
replication_delete_marker_purge_remove_options(dobj.delete_object.delete_marker_mtime),
)
.await;
.await
{
Ok(_) => {
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "purged").increment(1);
}
// The marker version is already gone on the target: the purge goal
// is met. Strict S3 targets 404 here (RustFS/MinIO answer 204);
// treating it as a failure would retain the intent entry forever.
Err(error) if matches!(error.code.as_deref(), Some("NoSuchKey" | "NoSuchVersion")) => {
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "purged").increment(1);
}
Err(error) => {
warn!(
event = EVENT_DELETE_MARKER_PURGE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket,
object = dobj.delete_object.object_name,
arn = tgt_entry.arn,
error = %error,
reason = "target_delete_failed",
"Delete-marker purge attempt failed"
);
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "failed").increment(1);
mark_replication_target_offline_if_needed(&tgt_client, &error).await;
failed_arns.push(tgt_entry.arn.clone());
}
}
}
failed_arns
}
const DELETE_MARKER_PURGE_WATCH_ROUNDS: usize = 5;
const DELETE_MARKER_PURGE_WATCH_INTERVAL: TokioDuration = TokioDuration::from_secs(1);
/// Watch the source delete marker for a short window after its replication.
///
/// KNOWN NON-DURABLE WINDOW: this task is detached, so a process exit inside
/// the watch window loses an intent that has not been persisted yet. The
/// window predates this code (the previous implementation had no durable
/// channel at all, and no replay half either), so nothing regresses — closing
/// it needs a write-ahead intent recorded before the parent delete is
/// acknowledged, which is tracked as follow-up rather than done here: every
/// delete-marker replication would pay a journal write for a purge that
/// almost never happens.
///
/// If the marker disappears (deleted before or while the replica landed),
/// purge the replicated marker from the targets, retrying failed targets on
/// later rounds. When the window drains with targets still dirty, persist the
/// purge intent as a durable MRF entry so the next startup replays it through
/// `purge_stale_delete_marker_targets`.
async fn watch_and_purge_source_delete_marker<S: ReplicationStorage>(
bucket: String,
dobj: DeletedObjectReplicationInfo,
dsc: ReplicateDecision,
storage: Arc<S>,
) {
let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
return;
};
// `pending` is None until the source marker is observed missing; after the
// first purge pass it holds the targets that still need a successful purge.
let mut pending: Option<Vec<String>> = None;
for round in 0..DELETE_MARKER_PURGE_WATCH_ROUNDS {
pending = match pending.take() {
None => {
if source_delete_marker_missing(&*storage, &bucket, &dobj.delete_object.object_name, delete_marker_version_id)
.await
{
Some(replicate_delete_marker_purge_to_targets(&bucket, &dobj, &dsc, None).await)
} else {
None
}
}
Some(failed_arns) => Some(replicate_delete_marker_purge_to_targets(&bucket, &dobj, &dsc, Some(&failed_arns)).await),
};
if matches!(pending.as_deref(), Some([])) {
return;
}
if round + 1 < DELETE_MARKER_PURGE_WATCH_ROUNDS {
tokio::time::sleep(DELETE_MARKER_PURGE_WATCH_INTERVAL).await;
}
}
if let Some(failed_arns) = pending.filter(|failed_arns| !failed_arns.is_empty()) {
enqueue_delete_marker_purge_mrf(&dobj, failed_arns).await;
}
}
/// Shape an exhausted purge intent as a marker-creation delete entry. Replay
/// reconstructs it with `delete_marker: true`, finds the source marker gone,
/// and funnels into the stale-marker branch of `replicate_delete_with_outcome`
/// — which re-runs the purge without touching source state and reports purge
/// success as the replay outcome.
fn delete_marker_purge_mrf_entry(dobj: &DeletedObjectReplicationInfo, failed_arns: Vec<String>) -> MrfReplicateEntry {
let mut entry = dobj.to_mrf_entry();
entry.delete_marker = true;
entry.version_id = None;
entry.retry_count = 0;
entry.target_arns = failed_arns;
entry
}
async fn enqueue_delete_marker_purge_mrf(dobj: &DeletedObjectReplicationInfo, failed_arns: Vec<String>) {
let arns = failed_arns.join(",");
let miss_reason = match runtime_sources::replication_pool() {
None => Some("replication_pool_unavailable"),
Some(pool) => match pool.persist_mrf_entry(delete_marker_purge_mrf_entry(dobj, failed_arns)).await {
ReplicationQueueAdmission::Queued => None,
_ => Some("mrf_save_unavailable"),
},
};
match miss_reason {
None => {
warn!(
event = EVENT_DELETE_MARKER_PURGE_MRF,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = dobj.bucket,
object = dobj.delete_object.object_name,
arns,
state = "queued",
"Delete-marker purge exhausted its watch window; intent persisted to the MRF journal"
);
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "mrf_queued").increment(1);
}
Some(reason) => {
warn!(
event = EVENT_DELETE_MARKER_PURGE_MRF,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = dobj.bucket,
object = dobj.delete_object.object_name,
arns,
state = "missed",
reason,
"Delete-marker purge intent could not be persisted for retry"
);
counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "mrf_missed").increment(1);
}
}
}
/// The marker vanished at the source while its replication was still pending
/// (a live race), or this is an MRF purge-intent replay. Any marker already
/// replicated to a target must still be purged; run bounded retry passes and
/// report the result so an MRF replay only acknowledges the entry once every
/// target is clean. Live callers persist a fresh purge intent on failure;
/// replay callers (`ReplicationType::Heal`) rely on Missed retention instead,
/// so the journal does not accumulate duplicate entries.
///
/// Heal callers retry for the full watch window because the startup MRF
/// processor runs before bucket metadata (and thus target clients) finishes
/// initializing — the first pass can see `target_client_missing` and a later
/// round resolves the client; the replay loop is serial and startup-only, so
/// blocking it for up to the window per dirty entry is acceptable. Live
/// callers run on replication workers where a down target would pin a worker
/// for the whole window, so they attempt once and lean on the durable intent
/// entry instead.
async fn purge_stale_delete_marker_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo) -> bool {
let decision_str = dobj
.delete_object
.replication_state
.as_ref()
.map(|state| state.replicate_decision_str.clone())
.unwrap_or_default();
let dsc = match parse_replicate_decision(bucket, &decision_str) {
Ok(dsc) => dsc,
Err(error) => {
warn!(
event = EVENT_DELETE_MARKER_PURGE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket,
object = dobj.delete_object.object_name,
error = %error,
reason = "replicate_decision_parse_failed",
"Delete-marker purge attempt failed"
);
return false;
}
};
let rounds = if dobj.op_type == ReplicationType::Heal {
DELETE_MARKER_PURGE_WATCH_ROUNDS
} else {
1
};
let mut failed_arns = replicate_delete_marker_purge_to_targets(bucket, dobj, &dsc, None).await;
for _ in 1..rounds {
if failed_arns.is_empty() {
break;
}
tokio::time::sleep(DELETE_MARKER_PURGE_WATCH_INTERVAL).await;
failed_arns = replicate_delete_marker_purge_to_targets(bucket, dobj, &dsc, Some(&failed_arns)).await;
}
if failed_arns.is_empty() {
return true;
}
if dobj.op_type != ReplicationType::Heal {
enqueue_delete_marker_purge_mrf(dobj, failed_arns).await;
}
false
}
async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &DeletedObjectReplicationInfo, storage: Arc<S>) -> bool {
@@ -2605,6 +2924,13 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
let result = tgt_client
.put_object(&tgt_client.bucket, &object, transfer_size, byte_stream, &put_opts)
.await
.map(|assigned_version_id| {
audit_target_version_identity(
&tgt_client,
&put_opts.internal.source_version_id,
assigned_version_id.as_deref(),
)
})
.map_err(|e| std::io::Error::other(e.to_string()));
record_proxy_request(&bucket, "PutObject", result.is_err()).await;
if has_tagging_replication {
@@ -3012,6 +3338,13 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
let result = tgt_client
.put_object(&tgt_client.bucket, &object, transfer_size, byte_stream, &put_opts)
.await
.map(|assigned_version_id| {
audit_target_version_identity(
&tgt_client,
&put_opts.internal.source_version_id,
assigned_version_id.as_deref(),
)
})
.map_err(|e| std::io::Error::other(e.to_string()));
record_proxy_request(&bucket, "PutObject", result.is_err()).await;
if has_tagging_replication {
@@ -3203,21 +3536,34 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
let actual_size = replication_multipart_complete_actual_size(&object_info.user_defined);
cli.complete_multipart_upload(
dst_bucket,
object,
&upload_id,
uploaded_parts,
&replication_complete_multipart_options(actual_size, object_info.etag.clone().unwrap_or_default(), object_info.mod_time),
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
let completed = cli
.complete_multipart_upload(
dst_bucket,
object,
&upload_id,
uploaded_parts,
&replication_complete_multipart_options(
actual_size,
object_info.etag.clone().unwrap_or_default(),
object_info.mod_time,
),
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
// Multipart decides the target version at initiate time and only reveals
// it on completion, so this is where the identity contract is observable
// for this path. A target can mirror PutObject version ids and still mint
// its own here, which would leave multipart deletes and heals addressing
// a version that never existed.
audit_target_version_identity(&cli, &put_opts.internal.source_version_id, completed.version_id());
Ok(())
}
#[cfg(test)]
mod tests {
use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
use super::*;
use s3s::dto::{
@@ -3257,6 +3603,27 @@ mod tests {
ReplicationTargetStore::register_test_target(target).await;
}
/// P1-19 runtime spot-check exemption matrix: drift only applies when the
/// source addressed a real version uuid.
#[test]
fn test_version_identity_drift_judgment() {
let source = "6fa459ea-ee8a-3ca4-894e-db77e160355e";
for (sent, got, expected) in [
(source, Some(source), false),
(source, Some("0e304ce5-33e9-4b8a-9b12-9e40a53e6ded"), true),
(source, None, true),
("", None, false),
("null", Some("anything"), false),
("00000000-0000-0000-0000-000000000000", Some("anything"), false),
] {
assert_eq!(
version_identity_drifted(sent, got),
expected,
"sent {sent:?} got {got:?} must judge drift = {expected}"
);
}
}
#[test]
fn resync_admission_configuration_is_bounded() {
assert_eq!(ENV_REPL_RESYNC_MAX_JOBS, "RUSTFS_REPL_RESYNC_MAX_JOBS");
@@ -3581,6 +3948,101 @@ mod tests {
);
}
/// P1-21 regression guard for the outcome formula. A fully successful
/// delete-marker replication must acknowledge its MRF entry: the formula
/// once carried `&& !requires_delayed_purge`, which pinned every
/// delete-marker entry to Missed and retained the whole backlog forever.
/// (Deterministically staging a marker-creation entry in the durable
/// journal from e2e would require saturating the worker queues, so the
/// formula is pinned here instead; the purge-intent replay half is pinned
/// by the delayed-purge e2e pair.)
#[test]
fn test_replicate_delete_outcome_is_not_held_hostage_by_the_delayed_purge() {
assert!(
replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Completed),
"a completed delete-marker replication must be acknowledgeable even though a delayed purge watch is pending"
);
assert!(!replicate_delete_outcome(0, 0, true, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(2, 1, true, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, false, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, true, false, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Failed));
}
/// P1-21 review follow-up: a target whose recorded marker version is
/// inconsistent must be reported as a per-target FAILURE. Treating the
/// refusal as success let the watcher and the MRF replay drop the purge
/// intent while the marker was still on the target.
#[tokio::test]
async fn test_delete_marker_purge_reports_corrupt_recorded_version_as_failure() {
let arn = format!("arn:rustfs:replication:us-east-1:corrupt:{}", Uuid::new_v4());
let mut dsc = ReplicateDecision::new();
dsc.set(ReplicateTargetDecision::new(arn.clone(), true, false));
let mut state = ReplicationState {
target_delete_marker_version_ids_corrupt: true,
..Default::default()
};
state.targets.insert(arn.clone(), ReplicationStatusType::Completed);
let dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "doc.txt".to_string(),
delete_marker: true,
delete_marker_version_id: Some(Uuid::new_v4()),
replication_state: Some(state),
..Default::default()
},
bucket: "bucket-a".to_string(),
..Default::default()
};
// No target client is registered: the refusal must be decided from
// the recorded metadata alone, before any remote call is attempted.
let failed = replicate_delete_marker_purge_to_targets("bucket-a", &dobj, &dsc, None).await;
assert_eq!(
failed,
vec![arn],
"a refused purge must stay in the failed set so the intent is never acknowledged"
);
}
#[test]
fn test_delete_marker_purge_mrf_entry_replays_through_the_stale_marker_branch() {
let delete_marker_version_id = Uuid::new_v4();
let dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "doc.txt".to_string(),
// A version-purge flavored source event: the entry must still
// be reshaped as a marker-creation delete so replay funnels
// into the stale-marker branch instead of re-running the full
// delete replication (whose source-state stamping would fail
// against the already-purged version).
delete_marker: false,
version_id: Some(Uuid::new_v4()),
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
bucket: "bucket-a".to_string(),
..Default::default()
};
let entry = delete_marker_purge_mrf_entry(&dobj, vec!["arn:a".to_string()]);
assert!(entry.delete_marker, "purge intents must replay as marker-creation deletes");
assert_eq!(entry.version_id, None, "the purged data version must not leak into the replay");
assert_eq!(entry.delete_marker_version_id, Some(delete_marker_version_id));
assert_eq!(
entry.target_arns,
vec!["arn:a".to_string()],
"only the targets whose purge failed may be retried"
);
assert_eq!(entry.retry_count, 0);
assert_eq!(entry.bucket, "bucket-a");
assert_eq!(entry.object, "doc.txt");
}
#[test]
fn test_is_retryable_delete_replication_head_error_allows_delete_marker_head_responses() {
assert!(
+631 -47
View File
@@ -15,12 +15,12 @@
#[cfg(test)]
use crate::cluster::rpc::http_auth::RPC_REPLAY_SCOPE_VERSION_HEADER;
use crate::cluster::rpc::http_auth::{
RPC_AUTH_VERSION_HEADER, RPC_AUTH_VERSION_V2, RPC_BOOT_EPOCH_CHALLENGE_HEADER, RPC_BOOT_EPOCH_HEADER,
RPC_BOOT_EPOCH_PROOF_HEADER, RPC_CONTENT_SHA256_HEADER, TIMESTAMP_HEADER,
};
use crate::cluster::rpc::{
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience, verify_tonic_boot_epoch_response,
AuthenticatedPeerReplayCapabilities, RPC_AUTH_VERSION_HEADER, RPC_AUTH_VERSION_V2, RPC_BOOT_EPOCH_CHALLENGE_HEADER,
RPC_CONTENT_SHA256_HEADER, RPC_REPLAY_CACHE_CAPABILITY_HEADER, RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER,
RollingMutationBodyDigest, TIMESTAMP_HEADER, internode_rpc_body_digest_strict,
verify_tonic_peer_replay_capabilities_response,
};
use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience};
#[cfg(test)]
use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
@@ -233,7 +233,22 @@ pub struct ReplayScopeChannel<S> {
/// The channel type used by internode clients after v2 authentication and replay-scope handling.
pub type AuthenticatedChannel = ReplayScopeChannel<Channel>;
static PEER_BOOT_EPOCHS: LazyLock<Mutex<HashMap<String, Uuid>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum PeerReplayCapability {
Capable { boot_epoch: Uuid },
Revoked,
}
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
struct PeerReplayState {
boot_epoch: Option<Uuid>,
cache_capability: Option<PeerReplayCapability>,
}
#[derive(Clone, Copy, Debug)]
struct PeerReplayStateSnapshot(PeerReplayState);
static PEER_REPLAY_STATES: LazyLock<Mutex<HashMap<String, PeerReplayState>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
impl<S> ReplayScopeChannel<S> {
fn new(inner: S, audience: Option<String>) -> Self {
@@ -241,13 +256,67 @@ impl<S> ReplayScopeChannel<S> {
}
}
fn cached_peer_boot_epoch(audience: &str) -> Option<Uuid> {
PEER_BOOT_EPOCHS.lock().ok().and_then(|epochs| epochs.get(audience).copied())
fn peer_replay_state(audience: &str) -> PeerReplayState {
PEER_REPLAY_STATES
.lock()
.ok()
.and_then(|states| states.get(audience).copied())
.unwrap_or_default()
}
fn remember_peer_boot_epoch(audience: String, epoch: Uuid) {
if let Ok(mut epochs) = PEER_BOOT_EPOCHS.lock() {
epochs.insert(audience, epoch);
fn apply_peer_replay_response(
audience: String,
sent_state: PeerReplayState,
response: std::io::Result<AuthenticatedPeerReplayCapabilities>,
) {
if let Ok(mut states) = PEER_REPLAY_STATES.lock() {
let current_state = states.get(&audience).copied().unwrap_or_default();
let mut next_state = current_state;
if let Ok(response) = &response
&& sent_state.boot_epoch == current_state.boot_epoch
{
next_state.boot_epoch = Some(response.boot_epoch);
}
if sent_state.boot_epoch == current_state.boot_epoch {
let response_capability = response
.as_ref()
.ok()
.filter(|response| response.dynamic_replay_cache)
.map(|response| response.boot_epoch);
match (sent_state.cache_capability, current_state.cache_capability, response_capability) {
(None, None, Some(boot_epoch))
| (Some(PeerReplayCapability::Revoked), Some(PeerReplayCapability::Revoked), Some(boot_epoch)) => {
next_state.cache_capability = Some(PeerReplayCapability::Capable { boot_epoch });
}
(
Some(PeerReplayCapability::Capable {
boot_epoch: sent_boot_epoch,
}),
Some(PeerReplayCapability::Capable {
boot_epoch: current_boot_epoch,
}),
Some(response_boot_epoch),
) if sent_boot_epoch == current_boot_epoch => {
next_state.cache_capability = Some(PeerReplayCapability::Capable {
boot_epoch: response_boot_epoch,
});
}
(
Some(PeerReplayCapability::Capable {
boot_epoch: sent_boot_epoch,
}),
Some(PeerReplayCapability::Capable {
boot_epoch: current_boot_epoch,
}),
None,
) if sent_boot_epoch == current_boot_epoch => {
next_state.cache_capability = Some(PeerReplayCapability::Revoked);
}
_ => {}
}
}
states.insert(audience, next_state);
}
}
@@ -276,6 +345,11 @@ where
== Some(RPC_AUTH_VERSION_V2)
});
let challenge = authenticated.then(Uuid::new_v4);
let sent_state = request
.extensions()
.get::<PeerReplayStateSnapshot>()
.map(|snapshot| snapshot.0)
.unwrap_or_default();
if let (Some(audience), Some(challenge)) = (self.audience.as_deref(), challenge) {
// The challenge is independently HMAC-authenticated by the response proof. It is not
// part of v2 so old peers ignore it, while a new peer can safely advertise its epoch.
@@ -284,7 +358,7 @@ where
challenge.to_string().parse().expect("UUID must be a valid header value"),
);
if let (Some(boot_epoch), Some(timestamp), Some(content_sha256)) = (
cached_peer_boot_epoch(audience),
sent_state.boot_epoch,
request.headers().get(TIMESTAMP_HEADER).and_then(|value| value.to_str().ok()),
request
.headers()
@@ -303,16 +377,21 @@ where
Box::pin(async move {
let response = future.await?;
if let (Some(audience), Some(challenge)) = (audience, challenge) {
match verify_tonic_boot_epoch_response(&audience, challenge, response.headers()) {
Ok(epoch) => remember_peer_boot_epoch(audience, epoch),
Err(error)
if response.headers().contains_key(RPC_BOOT_EPOCH_HEADER)
|| response.headers().contains_key(RPC_BOOT_EPOCH_PROOF_HEADER) =>
{
debug!(error = %error, "peer boot epoch response proof was rejected")
}
Err(_) => {}
let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
if let Err(error) = &response_state
&& (response.headers().contains_key(RPC_REPLAY_CACHE_CAPABILITY_HEADER)
|| response.headers().contains_key(RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER))
{
debug!(
event = "internode_rpc_capability_proof_rejected",
component = "ecstore",
subsystem = "rpc_client",
result = "rejected",
error = %error,
"internode RPC capability proof rejected"
)
}
apply_peer_replay_response(audience, sent_state, response_state);
}
Ok(response)
})
@@ -321,6 +400,7 @@ where
pub struct TonicSignatureInterceptor {
audience: Option<String>,
body_digest_strict: bool,
}
impl tonic::service::Interceptor for TonicSignatureInterceptor {
@@ -337,9 +417,31 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
.metadata()
.get(RPC_CONTENT_SHA256_HEADER)
.and_then(|value| value.to_str().ok());
// RUSTFS_COMPAT_TODO(disk-mutation-body-digest): use cache-free v2 for peers without an authenticated boot epoch. Remove after every supported peer advertises the authenticated dynamic replay-cache capability and body-digest strict mode is the default.
// beta.11 verifies v2 body digests but stores their nonces in a fixed-size cache.
let rolling_mutation = req.extensions().get::<RollingMutationBodyDigest>().is_some();
let peer_state = PEER_REPLAY_STATES
.lock()
.map_err(|_| tonic::Status::unauthenticated("RPC peer capability state unavailable"))?
.get(audience)
.copied()
.unwrap_or_default();
let content_sha256 = if content_sha256.is_some() {
if peer_state.cache_capability == Some(PeerReplayCapability::Revoked) {
return Err(tonic::Status::unauthenticated("RPC peer replay capability changed"));
}
if rolling_mutation && !self.body_digest_strict && peer_state.boot_epoch.is_none() {
None
} else {
content_sha256
}
} else {
content_sha256
};
let headers = gen_tonic_signature_headers(audience, method.service(), method.method(), content_sha256)
.map_err(|_| tonic::Status::unauthenticated("No valid auth token"))?;
req.metadata_mut().as_mut().extend(headers);
req.extensions_mut().insert(PeerReplayStateSnapshot(peer_state));
inject_trace_context_into_metadata(req.metadata_mut());
inject_request_id_into_metadata(req.metadata_mut());
Ok(req)
@@ -347,7 +449,10 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
}
pub fn gen_tonic_signature_interceptor() -> TonicSignatureInterceptor {
TonicSignatureInterceptor { audience: None }
TonicSignatureInterceptor {
audience: None,
body_digest_strict: internode_rpc_body_digest_strict(),
}
}
pub struct NoOpInterceptor;
@@ -409,6 +514,7 @@ mod tests {
#[derive(Clone)]
struct EpochProofService {
audience: String,
include_capability: bool,
seen_headers: std::sync::Arc<Mutex<Vec<http::HeaderMap>>>,
}
@@ -430,29 +536,97 @@ mod tests {
.expect("client challenge must be syntactically valid")
.expect("authenticated client request must carry a boot epoch challenge");
let mut response = HttpResponse::new(());
response.headers_mut().extend(
tonic_boot_epoch_response_headers(&self.audience, challenge)
.expect("test server must be able to sign an epoch proof"),
);
let mut headers = tonic_boot_epoch_response_headers(&self.audience, challenge)
.expect("test server must be able to sign an epoch proof");
if !self.include_capability {
headers.remove(RPC_REPLAY_CACHE_CAPABILITY_HEADER);
headers.remove(RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER);
}
response.headers_mut().extend(headers);
std::future::ready(Ok(response))
}
}
#[derive(Clone)]
struct MissingProofService;
impl Service<HttpRequest<()>> for MissingProofService {
type Response = HttpResponse<()>;
type Error = std::convert::Infallible;
type Future = std::future::Ready<Result<Self::Response, Self::Error>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, _request: HttpRequest<()>) -> Self::Future {
std::future::ready(Ok(HttpResponse::new(())))
}
}
fn ensure_test_rpc_secret() {
runtime_sources::ensure_test_rpc_secret();
}
fn test_request() -> tonic::Request<()> {
test_request_for("Ping")
}
fn test_request_for(method: &'static str) -> tonic::Request<()> {
let mut request = tonic::Request::new(());
request
.extensions_mut()
.insert(tonic::GrpcMethod::new("node_service.NodeService", "Ping"));
.insert(tonic::GrpcMethod::new("node_service.NodeService", method));
request
}
fn test_interceptor() -> TonicSignatureInterceptor {
test_interceptor_for("node-a:9000", false)
}
fn test_interceptor_for(audience: &str, body_digest_strict: bool) -> TonicSignatureInterceptor {
TonicSignatureInterceptor {
audience: Some("node-a:9000".to_string()),
audience: Some(audience.to_string()),
body_digest_strict,
}
}
fn clear_peer_capability(audience: &str) {
PEER_REPLAY_STATES
.lock()
.expect("peer capability cache lock must not be poisoned")
.remove(audience);
}
fn rolling_mutation_request(method: &'static str) -> tonic::Request<()> {
let mut request = tonic::Request::new(rustfs_protos::proto_gen::node_service::GenerallyLockRequest {
args: "canonical mutation request".to_string(),
});
request
.extensions_mut()
.insert(tonic::GrpcMethod::new("node_service.NodeService", method));
crate::cluster::rpc::set_tonic_rolling_mutation_body_digest(&mut request).expect("test mutation digest must be attached");
request.map(|_| ())
}
fn replay_scope_request(audience: &str, method: &'static str) -> HttpRequest<()> {
let mut request = HttpRequest::builder()
.uri(format!("/node_service.NodeService/{method}"))
.body(())
.expect("test RPC request must build");
request.headers_mut().extend(
gen_tonic_signature_headers(audience, "node_service.NodeService", method, None).expect("v2 test headers must mint"),
);
request
.extensions_mut()
.insert(PeerReplayStateSnapshot(peer_replay_state(audience)));
request
}
fn authenticated_peer_response(boot_epoch: Uuid, dynamic_replay_cache: bool) -> AuthenticatedPeerReplayCapabilities {
AuthenticatedPeerReplayCapabilities {
boot_epoch,
dynamic_replay_cache,
}
}
@@ -567,6 +741,431 @@ mod tests {
);
}
#[test]
fn unknown_peer_mutations_use_cache_free_unsigned_v2() {
ensure_test_rpc_secret();
let audience = "legacy-body-digest-client-test:9000";
clear_peer_capability(audience);
let mut interceptor = test_interceptor_for(audience, false);
for method in ["Lock", "WriteAll"] {
let request = interceptor
.call(rolling_mutation_request(method))
.expect("interceptor call should succeed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some("UNSIGNED-PAYLOAD")
);
assert_eq!(
request
.metadata()
.get("x-rustfs-rpc-nonce")
.and_then(|value| value.to_str().ok()),
Some("unsigned")
);
assert!(
crate::cluster::rpc::verify_tonic_rpc_signature(
audience,
&format!("/node_service.NodeService/{method}"),
request.metadata().as_ref(),
)
.is_ok(),
"the cache-free request must retain valid audience- and method-bound v2 authentication"
);
}
}
#[test]
fn unknown_peer_exact_body_contract_remains_body_bound() {
ensure_test_rpc_secret();
let audience = "exact-body-contract-client-test:9000";
clear_peer_capability(audience);
let mut interceptor = test_interceptor_for(audience, false);
let mut request = test_request_for("ScannerActivity");
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, b"exact scanner activity body")
.expect("test exact body digest must be attached");
let expected_digest = request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok())
.expect("test request must carry its digest")
.to_string();
let request = interceptor.call(request).expect("interceptor call should succeed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some(expected_digest.as_str())
);
assert!(
crate::cluster::rpc::verify_tonic_rpc_signature(
audience,
"/node_service.NodeService/ScannerActivity",
request.metadata().as_ref(),
)
.is_ok()
);
}
#[test]
fn unknown_peer_iam_mutation_helper_remains_body_bound() {
ensure_test_rpc_secret();
let audience = "exact-iam-mutation-client-test:9000";
clear_peer_capability(audience);
let mut interceptor = test_interceptor_for(audience, false);
let mut request = tonic::Request::new(rustfs_protos::proto_gen::node_service::DeleteUserRequest {
access_key: "target-access-key".to_string(),
});
request
.extensions_mut()
.insert(tonic::GrpcMethod::new("node_service.NodeService", "DeleteUser"));
crate::cluster::rpc::set_tonic_mutation_body_digest(&mut request).expect("test IAM mutation digest must be attached");
let request = request.map(|_| ());
let expected_digest = request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok())
.expect("test IAM mutation must carry its digest")
.to_string();
let request = interceptor.call(request).expect("interceptor call should succeed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some(expected_digest.as_str())
);
assert!(
crate::cluster::rpc::verify_tonic_rpc_signature(
audience,
"/node_service.NodeService/DeleteUser",
request.metadata().as_ref(),
)
.is_ok(),
"IAM mutations must remain body-bound before capability discovery"
);
}
#[test]
fn authenticated_replay_cache_capability_enables_body_binding() {
ensure_test_rpc_secret();
let audience = "body-digest-capable-client-test:9000";
clear_peer_capability(audience);
let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
let service = EpochProofService {
audience: audience.to_string(),
include_capability: true,
seen_headers,
};
let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
.expect("authenticated capability probe must complete");
let mut interceptor = test_interceptor_for(audience, false);
let request = rolling_mutation_request("Lock");
let expected_digest = request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok())
.expect("test mutation must carry its digest")
.to_string();
let request = interceptor.call(request).expect("interceptor call should succeed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some(expected_digest.as_str())
);
let nonce = request
.metadata()
.get("x-rustfs-rpc-nonce")
.and_then(|value| value.to_str().ok())
.and_then(|value| Uuid::parse_str(value).ok())
.expect("capable peer body-bound mutation must carry a UUID nonce");
assert!(!nonce.is_nil());
assert!(
crate::cluster::rpc::verify_tonic_rpc_signature(
audience,
"/node_service.NodeService/Lock",
request.metadata().as_ref(),
)
.is_ok(),
"the body-bound request must retain valid audience- and method-bound v2 authentication"
);
clear_peer_capability(audience);
}
#[test]
fn invalid_capability_proof_does_not_enable_body_binding() {
ensure_test_rpc_secret();
let audience = "invalid-capability-client-test:9000";
clear_peer_capability(audience);
let service = EpochProofService {
audience: "wrong-capability-audience:9000".to_string(),
include_capability: true,
seen_headers: std::sync::Arc::new(Mutex::new(Vec::new())),
};
let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
.expect("invalid capability response must still complete");
let mut interceptor = test_interceptor_for(audience, false);
let request = interceptor
.call(rolling_mutation_request("Lock"))
.expect("legacy-compatible mutation must still be signed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some("UNSIGNED-PAYLOAD")
);
}
#[test]
fn legacy_boot_proof_keeps_mutations_body_bound_and_enables_non_ping_v3() {
ensure_test_rpc_secret();
let audience = "legacy-boot-proof-client-test:9000";
clear_peer_capability(audience);
let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
let service = EpochProofService {
audience: audience.to_string(),
include_capability: false,
seen_headers: seen_headers.clone(),
};
let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
.expect("legacy boot proof response must complete");
let state = peer_replay_state(audience);
assert!(state.boot_epoch.is_some(), "authenticated legacy proof must enable replay-scoped v3");
assert_eq!(state.cache_capability, None);
let mut interceptor = test_interceptor_for(audience, false);
let request = rolling_mutation_request("Lock");
let expected_digest = request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok())
.expect("test mutation must carry its digest")
.to_string();
let request = interceptor.call(request).expect("legacy-compatible mutation must be signed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some(expected_digest.as_str())
);
let (metadata, extensions, body) = request.into_parts();
let mut request = HttpRequest::new(body);
*request.uri_mut() = "/node_service.NodeService/Lock".parse().expect("test RPC URI must parse");
*request.headers_mut() = metadata.into_headers();
*request.extensions_mut() = extensions;
futures::executor::block_on(channel.call(request)).expect("legacy strict-compatible lock request must complete");
let headers = seen_headers.lock().expect("test header capture lock must not be poisoned");
assert!(
headers[1].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
"authenticated legacy boot proof must enable v3 on a non-Ping request"
);
}
#[test]
fn reordered_capability_responses_cannot_undo_newer_state() {
let audience = "reordered-capability-client-test:9000";
let epoch_one = Uuid::new_v4();
let epoch_two = Uuid::new_v4();
clear_peer_capability(audience);
let unknown = PeerReplayState::default();
apply_peer_replay_response(audience.to_string(), unknown, Ok(authenticated_peer_response(epoch_one, true)));
apply_peer_replay_response(audience.to_string(), unknown, Err(std::io::Error::other("delayed legacy response")));
let epoch_one_state = PeerReplayState {
boot_epoch: Some(epoch_one),
cache_capability: Some(PeerReplayCapability::Capable { boot_epoch: epoch_one }),
};
assert_eq!(peer_replay_state(audience), epoch_one_state);
apply_peer_replay_response(audience.to_string(), epoch_one_state, Err(std::io::Error::other("rollback response")));
apply_peer_replay_response(audience.to_string(), epoch_one_state, Ok(authenticated_peer_response(epoch_one, true)));
assert_eq!(
peer_replay_state(audience),
PeerReplayState {
boot_epoch: Some(epoch_one),
cache_capability: Some(PeerReplayCapability::Revoked),
}
);
let revoked = peer_replay_state(audience);
apply_peer_replay_response(audience.to_string(), revoked, Ok(authenticated_peer_response(epoch_two, true)));
apply_peer_replay_response(audience.to_string(), epoch_one_state, Ok(authenticated_peer_response(epoch_one, true)));
assert_eq!(
peer_replay_state(audience),
PeerReplayState {
boot_epoch: Some(epoch_two),
cache_capability: Some(PeerReplayCapability::Capable { boot_epoch: epoch_two }),
}
);
clear_peer_capability(audience);
}
#[test]
fn stale_capability_response_cannot_cross_a_new_boot_epoch() {
let audience = "cross-epoch-capability-client-test:9000";
let epoch_one = Uuid::new_v4();
let epoch_two = Uuid::new_v4();
let epoch_three = Uuid::new_v4();
clear_peer_capability(audience);
let revoked_epoch_one = PeerReplayState {
boot_epoch: Some(epoch_one),
cache_capability: Some(PeerReplayCapability::Revoked),
};
PEER_REPLAY_STATES
.lock()
.expect("peer replay state lock must not be poisoned")
.insert(audience.to_string(), revoked_epoch_one);
apply_peer_replay_response(
audience.to_string(),
revoked_epoch_one,
Ok(authenticated_peer_response(epoch_three, false)),
);
apply_peer_replay_response(audience.to_string(), revoked_epoch_one, Ok(authenticated_peer_response(epoch_two, true)));
assert_eq!(
peer_replay_state(audience),
PeerReplayState {
boot_epoch: Some(epoch_three),
cache_capability: Some(PeerReplayCapability::Revoked),
},
"a stale dynamic-cache proof must not cross a newer authenticated boot epoch"
);
clear_peer_capability(audience);
}
#[test]
fn interceptor_snapshot_prevents_delayed_legacy_response_from_revoking_capability() {
ensure_test_rpc_secret();
let audience = "capability-snapshot-client-test:9000";
clear_peer_capability(audience);
let boot_epoch = Uuid::new_v4();
let mut interceptor = test_interceptor_for(audience, false);
let request = interceptor
.call(rolling_mutation_request("Lock"))
.expect("legacy-compatible request must pass the interceptor");
assert_eq!(
request
.extensions()
.get::<PeerReplayStateSnapshot>()
.map(|snapshot| snapshot.0),
Some(PeerReplayState::default()),
"interceptor must preserve its unknown-state admission snapshot"
);
let capable_state = PeerReplayState {
boot_epoch: Some(boot_epoch),
cache_capability: Some(PeerReplayCapability::Capable { boot_epoch }),
};
PEER_REPLAY_STATES
.lock()
.expect("peer capability cache lock must not be poisoned")
.insert(audience.to_string(), capable_state);
let (metadata, extensions, body) = request.into_parts();
let mut request = HttpRequest::new(body);
*request.uri_mut() = "/node_service.NodeService/Lock".parse().expect("test RPC URI must parse");
*request.headers_mut() = metadata.into_headers();
*request.extensions_mut() = extensions;
let mut channel = ReplayScopeChannel::new(MissingProofService, Some(audience.to_string()));
futures::executor::block_on(channel.call(request)).expect("in-flight request response must complete");
assert_eq!(peer_replay_state(audience), capable_state);
clear_peer_capability(audience);
}
#[test]
fn strict_mode_keeps_unknown_peer_mutations_body_bound() {
ensure_test_rpc_secret();
let audience = "strict-body-digest-client-test:9000";
clear_peer_capability(audience);
let mut interceptor = test_interceptor_for(audience, true);
let request = rolling_mutation_request("WriteAll");
let expected_digest = request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok())
.expect("test mutation must carry its digest")
.to_string();
let request = interceptor.call(request).expect("interceptor call should succeed");
assert_eq!(
request
.metadata()
.get("x-rustfs-content-sha256")
.and_then(|value| value.to_str().ok()),
Some(expected_digest.as_str())
);
let nonce = request
.metadata()
.get("x-rustfs-rpc-nonce")
.and_then(|value| value.to_str().ok())
.and_then(|value| Uuid::parse_str(value).ok())
.expect("strict body-bound mutation must carry a UUID nonce");
assert!(!nonce.is_nil());
assert!(
crate::cluster::rpc::verify_tonic_rpc_signature(
audience,
"/node_service.NodeService/WriteAll",
request.metadata().as_ref(),
)
.is_ok()
);
}
#[test]
fn missing_capability_after_pin_fails_closed() {
ensure_test_rpc_secret();
let audience = "revoked-capability-client-test:9000";
let boot_epoch = Uuid::new_v4();
PEER_REPLAY_STATES
.lock()
.expect("peer capability cache lock must not be poisoned")
.insert(
audience.to_string(),
PeerReplayState {
boot_epoch: Some(boot_epoch),
cache_capability: Some(PeerReplayCapability::Capable { boot_epoch }),
},
);
let mut channel = ReplayScopeChannel::new(MissingProofService, Some(audience.to_string()));
futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
.expect("legacy response must complete before capability rejection");
let mut interceptor = test_interceptor_for(audience, false);
let error = interceptor
.call(rolling_mutation_request("Lock"))
.expect_err("a peer that loses its pinned capability must fail closed");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.message(), "RPC peer replay capability changed");
clear_peer_capability(audience);
}
#[test]
fn test_signature_interceptor_binds_audience_from_peer_uri() {
let interceptor = TonicInterceptor::Signature(gen_tonic_signature_interceptor())
@@ -583,27 +1182,15 @@ mod tests {
fn replay_scope_channel_uses_epoch_proof_before_sending_v3() {
ensure_test_rpc_secret();
let audience = "replay-scope-client-test:9000";
PEER_BOOT_EPOCHS
.lock()
.expect("peer epoch cache lock must not be poisoned")
.remove(audience);
clear_peer_capability(audience);
let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
let service = EpochProofService {
audience: audience.to_string(),
include_capability: true,
seen_headers: seen_headers.clone(),
};
let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
let make_request = || {
let mut request = HttpRequest::builder()
.uri("/node_service.NodeService/Ping")
.body(())
.expect("test RPC request must build");
request.headers_mut().extend(
gen_tonic_signature_headers(audience, "node_service.NodeService", "Ping", None)
.expect("v2 test headers must mint"),
);
request
};
let make_request = || replay_scope_request(audience, "Ping");
futures::executor::block_on(channel.call(make_request())).expect("first request must complete");
futures::executor::block_on(channel.call(make_request())).expect("second request must complete");
@@ -619,10 +1206,7 @@ mod tests {
headers[1].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
"the second request must carry the replay-scoped v3 signature"
);
PEER_BOOT_EPOCHS
.lock()
.expect("peer epoch cache lock must not be poisoned")
.remove(audience);
clear_peer_capability(audience);
}
#[test]
+266 -15
View File
@@ -40,8 +40,11 @@ use http::{HeaderMap, HeaderValue, Method, Uri};
use rustfs_credentials::{DEFAULT_SECRET_KEY, RPC_SECRET_REQUIRED_MESSAGE};
use rustfs_credentials::{RPC_SECRET_REQUIRED_OPERATOR_MESSAGE, try_get_rpc_token};
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_GRPC_OTHER, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC, global_internode_metrics,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_FORCE_UNLOCK, INTERNODE_OPERATION_GRPC_LOCK,
INTERNODE_OPERATION_GRPC_LOCK_BATCH, INTERNODE_OPERATION_GRPC_OTHER, INTERNODE_OPERATION_GRPC_READ_ALL,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_OPERATION_GRPC_REFRESH,
INTERNODE_OPERATION_GRPC_UNLOCK, INTERNODE_OPERATION_GRPC_UNLOCK_BATCH, INTERNODE_OPERATION_GRPC_WRITE_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC, global_internode_metrics,
};
use rustfs_object_data_cache::{MemoryBasis, resolve_effective_memory};
use rustfs_utils::get_env_bool;
@@ -70,10 +73,14 @@ pub const RPC_REPLAY_SCOPE_NONCE_HEADER: &str = "x-rustfs-rpc-replay-nonce";
pub const RPC_BOOT_EPOCH_HEADER: &str = "x-rustfs-rpc-boot-epoch";
pub const RPC_BOOT_EPOCH_CHALLENGE_HEADER: &str = "x-rustfs-rpc-boot-epoch-challenge";
pub const RPC_BOOT_EPOCH_PROOF_HEADER: &str = "x-rustfs-rpc-boot-epoch-proof";
pub(crate) const RPC_REPLAY_CACHE_CAPABILITY_HEADER: &str = "x-rustfs-rpc-replay-cache-capability";
pub(crate) const RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER: &str = "x-rustfs-rpc-replay-cache-capability-proof";
const RPC_REPLAY_SCOPE_VERSION_V3: &str = "3";
const RPC_RESPONSE_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-response-proof-v1\0";
const RPC_REPLAY_SCOPE_DOMAIN: &[u8] = b"rustfs-rpc-replay-scope-v3\0";
const RPC_BOOT_EPOCH_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-boot-epoch-proof-v1\0";
const RPC_REPLAY_CACHE_CAPABILITY_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-replay-cache-capability-proof-v1\0";
const RPC_REPLAY_CACHE_CAPABILITY_V1: &str = "dynamic-replay-cache-v1";
const HTTP_PUT_FILE_AUTH_DOMAIN: &[u8] = b"rustfs-http-put-file-auth-v1\0";
const HTTP_PUT_FILE_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-http-put-file-capability-v1\0";
const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
@@ -82,8 +89,9 @@ const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(601);
const REPLAY_CACHE_RETENTION_SECS: usize = 601;
const REPLAY_CACHE_ENTRY_BYTES_ESTIMATE: u64 = 128;
const REPLAY_CACHE_AUTO_MEMORY_PERCENT: u64 = 8;
const REPLAY_CACHE_AUTO_RPC_RPS_PER_CPU: usize = 2048;
// Keep 16 CPU / 32 GiB field nodes at the 32M cap without requiring an env override.
const REPLAY_CACHE_AUTO_MEMORY_PERCENT: u64 = 13;
const REPLAY_CACHE_AUTO_RPC_RPS_PER_CPU: usize = 4096;
const REPLAY_CACHE_AUTO_MAX_CAPACITY: usize = 33_554_432;
const NS_SCANNER_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-capability-v3";
pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
@@ -99,6 +107,10 @@ static INTERNODE_RPC_BODY_DIGEST_STRICT: LazyLock<bool> = LazyLock::new(|| {
rustfs_config::DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT,
)
});
pub(crate) fn internode_rpc_body_digest_strict() -> bool {
*INTERNODE_RPC_BODY_DIGEST_STRICT
}
static INTERNODE_RPC_REPLAY_SCOPE_STRICT: LazyLock<bool> = LazyLock::new(|| {
get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT,
@@ -340,6 +352,7 @@ struct RpcNonceCacheMetrics<'a> {
expired: usize,
entries: usize,
capacity: usize,
record_scope: Option<RpcReplayCacheMetricScope<'a>>,
overflow_scope: Option<RpcReplayCacheMetricScope<'a>>,
}
@@ -350,6 +363,13 @@ fn publish_nonce_cache_metrics(metrics: Option<RpcNonceCacheMetrics<'_>>) {
let internode_metrics = global_internode_metrics();
internode_metrics.record_replay_cache_evictions("expired", metrics.expired);
internode_metrics.record_replay_cache_state(metrics.entries, metrics.capacity);
if let Some(scope) = metrics.record_scope {
internode_metrics.record_replay_cache_record_for_operation_and_backend_path(
scope.operation,
scope.backend,
scope.rpc_path,
);
}
if let Some(scope) = metrics.overflow_scope {
internode_metrics.record_replay_cache_overflow_for_operation_and_backend_path(
scope.operation,
@@ -385,6 +405,7 @@ impl RpcNonceCache {
expired,
entries: self.nonces.len(),
capacity: record.capacity,
record_scope: None,
overflow_scope: None,
};
if self.nonces.contains(&record.nonce) {
@@ -409,6 +430,7 @@ impl RpcNonceCache {
Ok(()),
Some(RpcNonceCacheMetrics {
entries: self.nonces.len(),
record_scope: Some(record.metric_scope),
..metrics
}),
)
@@ -776,6 +798,50 @@ fn verify_boot_epoch_proof(secret: &str, audience: &str, challenge: Uuid, boot_e
.map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid RPC boot epoch proof"))
}
fn update_replay_cache_capability_proof(mac: &mut HmacSha256, audience: &str, challenge: Uuid, boot_epoch: Uuid) {
mac.update(RPC_REPLAY_CACHE_CAPABILITY_PROOF_DOMAIN);
for part in [
audience.as_bytes(),
b"|",
challenge.as_bytes(),
b"|",
boot_epoch.as_bytes(),
b"|",
RPC_REPLAY_CACHE_CAPABILITY_V1.as_bytes(),
] {
mac.update(part);
}
}
fn generate_replay_cache_capability_proof(
secret: &str,
audience: &str,
challenge: Uuid,
boot_epoch: Uuid,
) -> std::io::Result<String> {
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_replay_cache_capability_proof(&mut mac, audience, challenge, boot_epoch);
Ok(general_purpose::STANDARD.encode(mac.finalize().into_bytes()))
}
fn verify_replay_cache_capability_proof(
secret: &str,
audience: &str,
challenge: Uuid,
boot_epoch: Uuid,
proof: &str,
) -> std::io::Result<()> {
let proof = general_purpose::STANDARD
.decode(proof)
.map_err(|_| std::io::Error::other("Invalid RPC replay cache capability proof"))?;
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
update_replay_cache_capability_proof(&mut mac, audience, challenge, boot_epoch);
mac.verify_slice(&proof)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid RPC replay cache capability proof"))
}
fn non_nil_uuid(value: &str, name: &str) -> std::io::Result<Uuid> {
let value = Uuid::parse_str(value).map_err(|_| std::io::Error::other(format!("Invalid {name}")))?;
(!value.is_nil())
@@ -858,15 +924,34 @@ pub fn tonic_boot_epoch_challenge(headers: &HeaderMap) -> std::io::Result<Option
/// Build the authenticated response headers for a client boot-epoch challenge.
pub fn tonic_boot_epoch_response_headers(audience: &str, challenge: Uuid) -> std::io::Result<HeaderMap> {
let boot_epoch = tonic_rpc_boot_epoch();
let proof = generate_boot_epoch_proof(&get_shared_secret()?, audience, challenge, boot_epoch)?;
let secret = get_shared_secret()?;
let proof = generate_boot_epoch_proof(&secret, audience, challenge, boot_epoch)?;
let capability_proof = generate_replay_cache_capability_proof(&secret, audience, challenge, boot_epoch)?;
let mut headers = HeaderMap::new();
headers.insert(RPC_BOOT_EPOCH_HEADER, header_value(&boot_epoch.to_string(), RPC_BOOT_EPOCH_HEADER)?);
headers.insert(RPC_BOOT_EPOCH_PROOF_HEADER, header_value(&proof, RPC_BOOT_EPOCH_PROOF_HEADER)?);
headers.insert(
RPC_REPLAY_CACHE_CAPABILITY_HEADER,
HeaderValue::from_static(RPC_REPLAY_CACHE_CAPABILITY_V1),
);
headers.insert(
RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER,
header_value(&capability_proof, RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER)?,
);
Ok(headers)
}
/// Verify the server boot-epoch response for a challenge generated by this client.
pub fn verify_tonic_boot_epoch_response(audience: &str, challenge: Uuid, headers: &HeaderMap) -> std::io::Result<Uuid> {
verify_tonic_boot_epoch_response_with_secret(&get_shared_secret()?, audience, challenge, headers)
}
fn verify_tonic_boot_epoch_response_with_secret(
secret: &str,
audience: &str,
challenge: Uuid,
headers: &HeaderMap,
) -> std::io::Result<Uuid> {
let boot_epoch = headers
.get(RPC_BOOT_EPOCH_HEADER)
.and_then(|value| value.to_str().ok())
@@ -876,10 +961,47 @@ pub fn verify_tonic_boot_epoch_response(audience: &str, challenge: Uuid, headers
.get(RPC_BOOT_EPOCH_PROOF_HEADER)
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC boot epoch proof"))?;
verify_boot_epoch_proof(&get_shared_secret()?, audience, challenge, boot_epoch, proof)?;
verify_boot_epoch_proof(secret, audience, challenge, boot_epoch, proof)?;
Ok(boot_epoch)
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) struct AuthenticatedPeerReplayCapabilities {
pub(crate) boot_epoch: Uuid,
pub(crate) dynamic_replay_cache: bool,
}
pub(crate) fn verify_tonic_peer_replay_capabilities_response(
audience: &str,
challenge: Uuid,
headers: &HeaderMap,
) -> std::io::Result<AuthenticatedPeerReplayCapabilities> {
let secret = get_shared_secret()?;
let boot_epoch = verify_tonic_boot_epoch_response_with_secret(&secret, audience, challenge, headers)?;
let capability = headers.get(RPC_REPLAY_CACHE_CAPABILITY_HEADER);
let proof = headers.get(RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER);
if capability.is_none() && proof.is_none() {
return Ok(AuthenticatedPeerReplayCapabilities {
boot_epoch,
dynamic_replay_cache: false,
});
}
let capability = capability
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC replay cache capability"))?;
if capability != RPC_REPLAY_CACHE_CAPABILITY_V1 {
return Err(std::io::Error::other("Unsupported RPC replay cache capability"));
}
let proof = proof
.and_then(|value| value.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing RPC replay cache capability proof"))?;
verify_replay_cache_capability_proof(&secret, audience, challenge, boot_epoch, proof)?;
Ok(AuthenticatedPeerReplayCapabilities {
boot_epoch,
dynamic_replay_cache: true,
})
}
fn valid_content_sha256(value: &str) -> bool {
value == UNSIGNED_PAYLOAD
|| (value.len() == 64
@@ -913,7 +1035,15 @@ fn tonic_rpc_metric_operation(path: &str) -> &'static str {
match parse_tonic_rpc_path(path).ok().map(|(_, rpc_method)| rpc_method) {
Some("ReadAll") => INTERNODE_OPERATION_GRPC_READ_ALL,
Some("ReadMultiple") => INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
Some("ReadVersion") => INTERNODE_OPERATION_GRPC_READ_VERSION,
Some("BatchReadVersion") => INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
Some("WriteAll") => INTERNODE_OPERATION_GRPC_WRITE_ALL,
Some("Lock") => INTERNODE_OPERATION_GRPC_LOCK,
Some("UnLock") => INTERNODE_OPERATION_GRPC_UNLOCK,
Some("LockBatch") => INTERNODE_OPERATION_GRPC_LOCK_BATCH,
Some("UnLockBatch") => INTERNODE_OPERATION_GRPC_UNLOCK_BATCH,
Some("Refresh") => INTERNODE_OPERATION_GRPC_REFRESH,
Some("ForceUnLock") => INTERNODE_OPERATION_GRPC_FORCE_UNLOCK,
_ => INTERNODE_OPERATION_GRPC_OTHER,
}
}
@@ -1082,6 +1212,23 @@ pub fn set_tonic_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(
set_tonic_canonical_body_digest(request, &canonical_body)
}
pub fn set_tonic_rolling_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(
request: &mut tonic::Request<T>,
) -> std::io::Result<()> {
set_tonic_mutation_body_digest(request)?;
request.extensions_mut().insert(RollingMutationBodyDigest);
Ok(())
}
pub fn set_tonic_rolling_canonical_body_digest<T>(request: &mut tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
set_tonic_canonical_body_digest(request, canonical_body)?;
request.extensions_mut().insert(RollingMutationBodyDigest);
Ok(())
}
#[derive(Clone, Copy, Debug)]
pub(crate) struct RollingMutationBodyDigest;
pub fn verify_tonic_canonical_body_digest<T>(request: &tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
let version = request
.metadata()
@@ -1118,7 +1265,7 @@ pub fn verify_tonic_canonical_body_digest<T>(request: &tonic::Request<T>, canoni
/// including v1-downgraded ones. It converges independently of the signature-strict switch
/// (<https://github.com/rustfs/backlog/issues/1327>).
pub fn verify_tonic_mutation_body_digest<T>(request: &tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
verify_tonic_mutation_body_digest_with_strictness(request, canonical_body, *INTERNODE_RPC_BODY_DIGEST_STRICT)
verify_tonic_mutation_body_digest_with_strictness(request, canonical_body, internode_rpc_body_digest_strict())
}
/// [`verify_tonic_mutation_body_digest`] with the strict gate injected as a parameter, so both
@@ -2171,6 +2318,23 @@ mod tests {
assert!(verify_tonic_boot_epoch_response("node-a:9000", Uuid::new_v4(), &headers).is_err());
}
#[test]
fn replay_cache_capability_proof_binds_audience_challenge_epoch_and_value() {
ensure_test_rpc_secret();
let challenge = Uuid::new_v4();
let headers = tonic_boot_epoch_response_headers("node-a:9000", challenge).expect("capability headers should build");
let capabilities = verify_tonic_peer_replay_capabilities_response("node-a:9000", challenge, &headers)
.expect("matching capability proof should verify");
assert_eq!(capabilities.boot_epoch, tonic_rpc_boot_epoch());
assert!(capabilities.dynamic_replay_cache);
assert!(verify_tonic_peer_replay_capabilities_response("node-b:9000", challenge, &headers).is_err());
assert!(verify_tonic_peer_replay_capabilities_response("node-a:9000", Uuid::new_v4(), &headers).is_err());
let mut changed_capability = headers;
changed_capability.insert(RPC_REPLAY_CACHE_CAPABILITY_HEADER, HeaderValue::from_static("dynamic-replay-cache-v2"));
assert!(verify_tonic_peer_replay_capabilities_response("node-a:9000", challenge, &changed_capability).is_err());
}
#[test]
fn tonic_rpc_auth_failure_reason_maps_security_relevant_errors() {
for (message, reason) in [
@@ -2457,10 +2621,42 @@ mod tests {
tonic_rpc_metric_operation("/node_service.NodeService/ReadMultiple"),
INTERNODE_OPERATION_GRPC_READ_MULTIPLE
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/ReadVersion"),
INTERNODE_OPERATION_GRPC_READ_VERSION
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/BatchReadVersion"),
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/WriteAll"),
INTERNODE_OPERATION_GRPC_WRITE_ALL
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/Lock"),
INTERNODE_OPERATION_GRPC_LOCK
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/UnLock"),
INTERNODE_OPERATION_GRPC_UNLOCK
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/LockBatch"),
INTERNODE_OPERATION_GRPC_LOCK_BATCH
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/UnLockBatch"),
INTERNODE_OPERATION_GRPC_UNLOCK_BATCH
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/Refresh"),
INTERNODE_OPERATION_GRPC_REFRESH
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/ForceUnLock"),
INTERNODE_OPERATION_GRPC_FORCE_UNLOCK
);
assert_eq!(
tonic_rpc_metric_operation("/node_service.NodeService/SignalService"),
INTERNODE_OPERATION_GRPC_OTHER
@@ -2499,21 +2695,27 @@ mod tests {
assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
assert_eq!(decision.memory_basis, Some(MemoryBasis::Host));
assert_eq!(decision.memory_based_capacity, 10_737_418);
assert_eq!(decision.cpu_based_capacity, 9_846_784);
assert_eq!(decision.capacity, 9_846_784);
assert_eq!(decision.memory_based_capacity, 17_448_304);
assert_eq!(decision.cpu_based_capacity, 19_693_568);
assert_eq!(decision.capacity, 17_448_304);
}
#[test]
fn replay_cache_capacity_auto_uses_resource_model_on_larger_nodes() {
fn replay_cache_capacity_auto_uses_32m_on_field_sized_nodes() {
let gib = 1024_u64 * 1024 * 1024;
let decision =
replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 16, Some(32 * gib), Some(MemoryBasis::Host));
assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
assert_eq!(decision.memory_based_capacity, 21_474_836);
assert_eq!(decision.cpu_based_capacity, 19_693_568);
assert_eq!(decision.capacity, 19_693_568);
assert_eq!(decision.memory_based_capacity, 34_896_609);
assert_eq!(decision.cpu_based_capacity, 39_387_136);
assert_eq!(decision.capacity, REPLAY_CACHE_AUTO_MAX_CAPACITY);
let observed_field_node =
replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 16, Some(31 * gib), Some(MemoryBasis::Host));
assert_eq!(observed_field_node.memory_based_capacity, 33_806_090);
assert_eq!(observed_field_node.cpu_based_capacity, 39_387_136);
assert_eq!(observed_field_node.capacity, REPLAY_CACHE_AUTO_MAX_CAPACITY);
}
#[test]
@@ -2545,7 +2747,7 @@ mod tests {
let decision = replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Invalid, 8, None, None);
assert_eq!(decision.source, ReplayCacheCapacitySource::AutoInvalidEnv);
assert_eq!(decision.capacity, 9_846_784);
assert_eq!(decision.capacity, 19_693_568);
}
fn check_test_nonce_record(cache: &mut RpcNonceCache, record: RpcNonceRecord<'_>) -> std::io::Result<()> {
@@ -2554,6 +2756,13 @@ mod tests {
result
}
fn check_test_nonce_record_with_metrics<'a>(
cache: &mut RpcNonceCache,
record: RpcNonceRecord<'a>,
) -> (std::io::Result<()>, Option<RpcNonceCacheMetrics<'a>>) {
cache.check_and_record(record)
}
fn test_nonce_record(
nonce: Uuid,
signed_at: i64,
@@ -2597,6 +2806,48 @@ mod tests {
assert!(cache.nonces.contains(&nonce_b));
}
#[test]
fn nonce_cache_metrics_mark_successful_records_only() {
let now = Instant::now();
let expiry = now.checked_add(REPLAY_CACHE_RETENTION).expect("test expiry should fit");
let nonce_a = Uuid::new_v4();
let nonce_b = Uuid::new_v4();
let mut cache = RpcNonceCache::default();
let (recorded, metrics) =
check_test_nonce_record_with_metrics(&mut cache, test_nonce_record(nonce_a, 100, now, 100, expiry, 1));
recorded.expect("first nonce should be recorded");
let metrics = metrics.expect("successful nonce should publish metrics");
let record_scope = metrics.record_scope.expect("successful nonce should carry record scope");
assert_eq!(record_scope.operation, INTERNODE_OPERATION_GRPC_READ_ALL);
assert_eq!(record_scope.backend, INTERNODE_TRANSPORT_BACKEND_GRPC);
assert_eq!(record_scope.rpc_path, "/node_service.NodeService/ReadAll");
assert!(metrics.overflow_scope.is_none());
let (replay, metrics) =
check_test_nonce_record_with_metrics(&mut cache, test_nonce_record(nonce_a, 100, now, 100, expiry, 1));
assert_eq!(
replay.expect_err("duplicate nonce must fail closed").to_string(),
"RPC request replay detected"
);
let metrics = metrics.expect("replay rejection should still publish cache state");
assert!(metrics.record_scope.is_none());
assert!(metrics.overflow_scope.is_none());
let (overflow, metrics) =
check_test_nonce_record_with_metrics(&mut cache, test_nonce_record(nonce_b, 100, now, 100, expiry, 1));
assert_eq!(
overflow.expect_err("full cache must fail closed").to_string(),
"RPC replay cache capacity exceeded"
);
let metrics = metrics.expect("overflow should publish cache state");
assert!(metrics.record_scope.is_none());
let overflow_scope = metrics.overflow_scope.expect("overflow should keep diagnostic scope");
assert_eq!(overflow_scope.operation, INTERNODE_OPERATION_GRPC_READ_ALL);
assert_eq!(overflow_scope.backend, INTERNODE_TRANSPORT_BACKEND_GRPC);
assert_eq!(overflow_scope.rpc_path, "/node_service.NodeService/ReadAll");
}
// The `rpc_body_digest_fallback_counter` serial group covers every test that drives (or
// asserts on) the process-global body-digest fallback counter, so exact-delta assertions
// cannot race with each other.
+6 -5
View File
@@ -34,11 +34,12 @@ pub use client::{
pub use http_auth::{
TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers,
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
set_tonic_mutation_body_digest, sign_ns_scanner_capability, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest, set_tonic_rolling_mutation_body_digest,
sign_ns_scanner_capability, sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge,
tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_put_file_auth_trailer,
verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
+17 -2
View File
@@ -16,7 +16,6 @@ use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
};
use crate::cluster::rpc::http_auth::set_tonic_canonical_body_digest;
use crate::cluster::rpc::internode_data_transport::{
InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
WriteStreamRequest,
@@ -123,7 +122,7 @@ fn attach_mutation_body_digest<T>(
op: &'static str,
) -> Result<()> {
let canonical_body = canonical_body.map_err(|_| Error::other(format!("{op} request length cannot be represented")))?;
set_tonic_canonical_body_digest(request, &canonical_body).map_err(Error::other)
crate::cluster::rpc::set_tonic_rolling_canonical_body_digest(request, &canonical_body).map_err(Error::other)
}
fn decode_volume_infos(volume_infos: Vec<String>) -> Result<Vec<VolumeInfo>> {
@@ -3029,6 +3028,22 @@ mod tests {
static INIT: Once = Once::new();
#[test]
fn disk_mutation_digest_marks_rolling_compatibility() {
let mut request = Request::new(());
attach_mutation_body_digest(&mut request, Ok(b"canonical disk mutation".to_vec()), "WriteAll")
.expect("disk mutation digest must be attached");
assert!(
request
.extensions()
.get::<crate::cluster::rpc::http_auth::RollingMutationBodyDigest>()
.is_some(),
"remote-disk mutations must reach the cache-free compatibility gate"
);
}
// `#[serial(internode_metrics)]` marks every test that observes
// `global_internode_metrics()`. Those counters are a process-wide singleton:
// some of these tests snapshot a counter, run one decode, and assert on the
@@ -15,7 +15,7 @@
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
};
use crate::cluster::rpc::set_tonic_mutation_body_digest;
use crate::cluster::rpc::set_tonic_rolling_mutation_body_digest;
use async_trait::async_trait;
use bytes::Bytes;
use rustfs_lock::{
@@ -33,6 +33,10 @@ use tonic::Request;
use tonic::service::interceptor::InterceptedService;
use tracing::{debug, info, warn};
fn attach_lock_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(request: &mut Request<T>) -> std::io::Result<()> {
set_tonic_rolling_mutation_body_digest(request)
}
/// Remote lock client implementation
#[derive(Debug, Clone)]
pub struct RemoteClient {
@@ -319,7 +323,7 @@ impl LockClient for RemoteClient {
args: serde_json::to_string(&request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = match self.execute_rpc("lock", &resource_summary, client.lock(req)).await {
Ok(resp) => resp.into_inner(),
@@ -358,7 +362,7 @@ impl LockClient for RemoteClient {
})
.collect::<Result<Vec<_>>>()?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = match self
.execute_rpc("lock_batch", &resource_summary, client.lock_batch(req))
@@ -400,7 +404,7 @@ impl LockClient for RemoteClient {
let mut client = self.get_client().await?;
let resource_summary = unlock_request.resource.to_string();
let mut req = Request::new(GenerallyLockRequest { args: request_string });
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = self
.execute_rpc("release", &resource_summary, client.un_lock(req))
.await?
@@ -427,7 +431,7 @@ impl LockClient for RemoteClient {
})
.collect::<Result<Vec<_>>>()?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = self
.execute_rpc("release_batch", &resource_summary, client.un_lock_batch(req))
@@ -450,7 +454,7 @@ impl LockClient for RemoteClient {
args: serde_json::to_string(&refresh_request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = self
.execute_rpc("refresh", &resource_summary, client.refresh(req))
.await?
@@ -470,7 +474,7 @@ impl LockClient for RemoteClient {
args: serde_json::to_string(&force_request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
let resp = self
.execute_rpc("force_release", &resource_summary, client.force_un_lock(req))
.await?
@@ -495,7 +499,7 @@ impl LockClient for RemoteClient {
args: serde_json::to_string(&status_request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
});
set_tonic_mutation_body_digest(&mut req)?;
attach_lock_mutation_body_digest(&mut req)?;
// Try exclusive lock first with very short timeout
let resp = match self.execute_rpc("check_status", &resource_summary, client.lock(req)).await {
@@ -510,7 +514,7 @@ impl LockClient for RemoteClient {
args: serde_json::to_string(&status_request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
});
set_tonic_mutation_body_digest(&mut release_req)?;
attach_lock_mutation_body_digest(&mut release_req)?;
let _ = self
.execute_rpc("check_status_release", &resource_summary, client.un_lock(release_req))
.await;
@@ -626,6 +630,31 @@ mod tests {
.with_priority(LockPriority::Normal)
}
#[test]
fn lock_mutation_helper_marks_single_and_batch_requests_for_rolling_auth() {
let mut single = Request::new(GenerallyLockRequest {
args: "single-lock".to_string(),
});
attach_lock_mutation_body_digest(&mut single).expect("single lock digest must be attached");
assert!(
single
.extensions()
.get::<crate::cluster::rpc::http_auth::RollingMutationBodyDigest>()
.is_some()
);
let mut batch = Request::new(BatchGenerallyLockRequest {
args: vec!["batch-lock".to_string()],
});
attach_lock_mutation_body_digest(&mut batch).expect("batch lock digest must be attached");
assert!(
batch
.extensions()
.get::<crate::cluster::rpc::http_auth::RollingMutationBodyDigest>()
.is_some()
);
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
+38
View File
@@ -584,6 +584,44 @@ where
.await
}
/// `delete_config` with `no_lock` set — for callers already holding the
/// config object's namespace lock (e.g. inside `with_config_object_write_lock`),
/// where the locked variant would self-deadlock.
pub async fn delete_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<()>
where
S: ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
match api
.delete_object(
RUSTFS_META_BUCKET,
file,
ObjectOptions {
delete_prefix: true,
delete_prefix_object: true,
no_lock: true,
..Default::default()
},
)
.await
{
Ok(_) => Ok(()),
Err(err) => {
if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
Err(Error::ConfigNotFound)
} else {
Err(err)
}
}
}
}
#[instrument(skip(api))]
pub async fn delete_config<S>(api: Arc<S>, file: &str) -> Result<()>
where
+116 -1
View File
@@ -7890,6 +7890,11 @@ impl DiskAPI for LocalDisk {
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
}
// Windows rejects renaming a directory while one of its children is
// still open, even when the child handle shares delete access.
drop(new_meta);
if durability.syncs_commit_metadata() {
os::fsync_dir_std(&staging_path)?;
}
std::fs::rename(&staging_path, &transaction_path)?;
@@ -8096,7 +8101,7 @@ impl DiskAPI for LocalDisk {
let durability = effective_durability(dst_volume);
if durability.syncs_data_shards() && !src_is_dir {
let src = src_file_path.clone();
tokio::task::spawn_blocking(move || std::fs::File::open(&src)?.sync_data())
tokio::task::spawn_blocking(move || os::sync_file(&src))
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
@@ -11566,6 +11571,116 @@ mod test {
);
}
#[cfg(windows)]
#[tokio::test]
#[allow(clippy::await_holding_lock)]
async fn test_rename_part_commits_realistic_windows_multipart_path() {
use crate::disk::RUSTFS_META_MULTIPART_BUCKET;
use tempfile::tempdir;
let _mode = durability_mode_override::set(DurabilityMode::Strict);
assert_eq!(effective_durability(RUSTFS_META_MULTIPART_BUCKET), DurabilityMode::Strict);
let dir = tempdir().expect("temp dir should be created");
let root = dir.path().join("realistic-windows-multipart-root");
fs::create_dir_all(&root).await.expect("disk root should be created");
let endpoint = Endpoint::try_from(root.to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
ensure_test_volume(&disk, RUSTFS_META_MULTIPART_BUCKET).await;
let src_path = "upload/part.1";
let dst_path = concat!(
"6f897928dfe04a87a269ccd9f5a5897d9cbbdf6b55e4d903ef3cbc1125c0cb8f/",
"8f897819-2604-4f3d-b843-c32a45d198b2x1786372838834745500/",
"58ba822c-06e4-4332-81cc-be2c9d921900/part.1"
);
let transaction_path = disk
.io_get_object_path(RUSTFS_META_MULTIPART_BUCKET, &crate::disk::part_transaction_path(dst_path))
.expect("transaction path should resolve");
let deepest_marker = transaction_path
.parent()
.expect("transaction path should have a parent")
.join(".part-txn-00000000-0000-0000-0000-000000000000")
.join(PART_TRANSACTION_OLD_DATA_ABSENT);
assert!(
deepest_marker.as_os_str().len() > 260,
"regression path must cross the traditional Windows MAX_PATH boundary: {deepest_marker:?}"
);
let payload = Bytes::from_static(b"part payload");
let meta = Bytes::from_static(b"part metadata");
disk.write_all(RUSTFS_META_TMP_BUCKET, src_path, payload.clone())
.await
.expect("source part should be written");
disk.prepare_part_transaction(RUSTFS_META_TMP_BUCKET, src_path, RUSTFS_META_MULTIPART_BUCKET, dst_path, meta.clone())
.await
.expect("realistic Windows part transaction should be prepared");
disk.rename_part(RUSTFS_META_TMP_BUCKET, src_path, RUSTFS_META_MULTIPART_BUCKET, dst_path, meta.clone())
.await
.expect("realistic Windows part should be committed");
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, dst_path, PartTransactionAction::Commit)
.await
.expect("realistic Windows part transaction should be settled");
assert_eq!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, dst_path)
.await
.expect("destination part should be readable"),
payload
);
assert_eq!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &format!("{dst_path}.meta"))
.await
.expect("destination metadata should be readable"),
meta
);
let replacement_payload = Bytes::from_static(b"replacement part payload");
let replacement_meta = Bytes::from_static(b"replacement part metadata");
disk.write_all(RUSTFS_META_TMP_BUCKET, src_path, replacement_payload.clone())
.await
.expect("replacement source part should be written");
disk.prepare_part_transaction(
RUSTFS_META_TMP_BUCKET,
src_path,
RUSTFS_META_MULTIPART_BUCKET,
dst_path,
replacement_meta.clone(),
)
.await
.expect("replacement Windows part transaction should be prepared");
disk.rename_part(
RUSTFS_META_TMP_BUCKET,
src_path,
RUSTFS_META_MULTIPART_BUCKET,
dst_path,
replacement_meta.clone(),
)
.await
.expect("replacement Windows part should be committed");
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, dst_path, PartTransactionAction::Commit)
.await
.expect("replacement Windows part transaction should be settled");
assert_eq!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, dst_path)
.await
.expect("replacement destination part should be readable"),
replacement_payload
);
assert_eq!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &format!("{dst_path}.meta"))
.await
.expect("replacement destination metadata should be readable"),
replacement_meta
);
assert!(
matches!(disk.read_all(RUSTFS_META_TMP_BUCKET, src_path).await, Err(DiskError::FileNotFound)),
"successful replacement must remove its source part"
);
assert!(!transaction_path.exists(), "settled replacement must remove its transaction directory");
}
#[tokio::test]
async fn test_part_transaction_rolls_back_data_published_before_metadata() {
use tempfile::tempdir;
+1 -1
View File
@@ -497,7 +497,7 @@ pub(crate) mod file_sync_probe {
}
}
fn sync_file(path: &Path) -> io::Result<()> {
pub(crate) fn sync_file(path: &Path) -> io::Result<()> {
#[cfg(test)]
let _probe = file_sync_probe::enter(path);
#[cfg(test)]
+132 -10
View File
@@ -538,7 +538,7 @@ impl MetadataQuorumAccumulator {
}
pub(in crate::set_disk) fn default_write_quorum(&self) -> usize {
if self.default_parity_count == 0 {
if self.default_parity_count == 0 || self.default_parity_count >= self.total_disks {
return self.total_disks;
}
let data_blocks = self.total_disks.saturating_sub(self.default_parity_count);
@@ -550,7 +550,7 @@ impl MetadataQuorumAccumulator {
}
pub(in crate::set_disk) fn missing_response_quorum(&self) -> usize {
if self.default_parity_count == 0 {
if self.default_parity_count == 0 || self.default_parity_count >= self.total_disks {
self.total_disks
} else {
self.total_disks / 2
@@ -2330,6 +2330,8 @@ impl SetDisks {
let response_start = Instant::now();
let result = if let Some(disk) = disk {
Self::record_read_version_call(&object, index);
#[cfg(test)]
Self::read_version_fanout_barrier(&object, index).await;
disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
} else {
Err(DiskError::DiskNotFound)
@@ -2397,9 +2399,13 @@ impl SetDisks {
return Ok((ress, errors, diagnostics));
}
let pending_responses = join_set.len();
let should_hedge_single_pending_data_read =
read_data && pending_responses == 1 && accumulator.can_still_reach_early_stop_with_pending(pending_responses);
if bounded_fanout
&& next_disk_index < disks.len()
&& !accumulator.can_still_reach_early_stop_with_pending(join_set.len())
&& (!accumulator.can_still_reach_early_stop_with_pending(pending_responses)
|| should_hedge_single_pending_data_read)
{
if let Some(disk) = disks.get(next_disk_index).cloned() {
spawn_read_version(&mut join_set, next_disk_index, disk);
@@ -3317,6 +3323,12 @@ impl SetDisks {
#[inline(always)]
fn record_read_version_call(_object: &str, _disk_index: usize) {}
#[cfg(test)]
#[inline]
async fn read_version_fanout_barrier(object: &str, disk_index: usize) {
rename_fanout_barrier::checkpoint(object, disk_index, rename_fanout_barrier::PHASE_READ_VERSION).await;
}
/// Test-only awaitable pause point for the rename/commit fan-out (backlog#1325,
/// serving the barrier-style acceptances of #1312 / #1319 / #1313). `phase` is
/// [`rename_fanout_barrier::PHASE_RENAME`] or `PHASE_CLEANUP`. When a test has
@@ -4803,6 +4815,8 @@ pub(in crate::set_disk) mod rename_fanout_barrier_phase {
pub const RENAME: &str = "rename";
/// The per-disk old-data-dir cleanup phase of the commit fan-out.
pub const CLEANUP: &str = "cleanup";
/// The per-disk `read_version` phase of metadata read fan-out.
pub const READ_VERSION: &str = "read_version";
}
/// Test-only awaitable pause barrier + background-task introspection for the
@@ -4846,7 +4860,9 @@ pub(in crate::set_disk) mod rename_fanout_barrier {
use std::sync::{Arc, Mutex, OnceLock};
use tokio::sync::Notify;
pub use super::rename_fanout_barrier_phase::{CLEANUP as PHASE_CLEANUP, RENAME as PHASE_RENAME};
pub use super::rename_fanout_barrier_phase::{
CLEANUP as PHASE_CLEANUP, READ_VERSION as PHASE_READ_VERSION, RENAME as PHASE_RENAME,
};
/// One armed barrier: the fan-out task matching `(disk_index, phase)` pauses.
struct Armed {
@@ -5364,7 +5380,7 @@ mod tests {
}
#[tokio::test]
async fn bounded_metadata_early_stop_ab_limits_data_get_read_version_fanout() {
async fn bounded_metadata_early_stop_ab_hedges_data_get_read_version_fanout() {
const DISKS: usize = 4;
let bucket = "bounded-data-get-fanout-bucket";
let control_object = "bounded-data-get-control-object";
@@ -5376,7 +5392,7 @@ mod tests {
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", None),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("false")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
],
async {
@@ -5389,7 +5405,7 @@ mod tests {
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
DISKS as u64,
"control path should keep the default data-read full fanout"
"control path should keep full fanout when data-read early stop is explicitly disabled"
);
assert_eq!(diagnostics.total_responses(), DISKS);
},
@@ -5419,13 +5435,109 @@ mod tests {
.await
.expect("healthy object metadata should reach early-stop quorum");
assert!(
(3..=DISKS as u64).contains(&calls.total(disk_call_counters::KIND_READ_VERSION)),
"healthy 2+2 bounded data-read fanout may finish at quorum before a spare hedge is needed"
);
assert!(
(3..=DISKS).contains(&diagnostics.total_responses()),
"treatment path should return after reaching quorum, with at most the spare hedge response observed"
);
assert!(parts_metadata.iter().filter(|fi| fi.name == treatment_object).count() >= 3);
assert!(errs.iter().all(Option::is_none));
},
)
.await;
drop(dirs);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn bounded_data_get_hedges_single_pending_read_version() {
const DISKS: usize = 4;
let bucket = "bounded-data-get-hedge-bucket";
let object = "bounded-data-get-hedge-object";
let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
install_metadata_fanout_fileinfo(&disks, bucket, object, None).await;
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
],
async {
let barrier = rename_fanout_barrier::arm(object, 2, rename_fanout_barrier::PHASE_READ_VERSION);
let calls = disk_call_counters::observe(object);
let disks_for_read = disks.clone();
let mut read = tokio::spawn(async move {
SetDisks::read_all_fileinfo_observed(&disks_for_read, bucket, bucket, object, "", true, false, false, true, 2)
.await
});
tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
.await
.expect("third scheduled read_version should pause at the deterministic barrier");
tokio::time::timeout(BARRIER_PAUSE_GUARD, async {
while calls.for_disk(disk_call_counters::KIND_READ_VERSION, 3) == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("bounded data-read fanout should hedge by starting the spare disk");
let completed = tokio::time::timeout(BARRIER_PAUSE_GUARD, &mut read).await;
if completed.is_err() {
barrier.release();
}
let (parts_metadata, errs, diagnostics) = completed
.expect("spare metadata should allow early-stop without waiting for the paused disk")
.expect("metadata read task should not panic")
.expect("healthy spare metadata should resolve");
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
3,
"treatment path should stop after the 2+2 read/write quorum instead of issuing every disk read"
DISKS as u64,
"bounded data-read fanout should issue the paused disk plus one spare hedge"
);
assert_eq!(diagnostics.total_responses(), 3);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == treatment_object).count(), 3);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == object).count(), 3);
assert!(errs.iter().all(Option::is_none));
},
)
.await;
drop(dirs);
}
#[tokio::test]
async fn bounded_metadata_early_stop_defaults_keep_data_get_full_fanout() {
const DISKS: usize = 4;
let bucket = "bounded-data-get-default-bucket";
let object = "bounded-data-get-default-object";
let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
install_metadata_fanout_fileinfo(&disks, bucket, object, None).await;
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", None::<&str>),
],
async {
let calls = disk_call_counters::observe(object);
let (parts_metadata, errs, diagnostics) =
SetDisks::read_all_fileinfo_observed(&disks, bucket, bucket, object, "", true, false, false, true, 2)
.await
.expect("default data-read metadata should resolve");
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
DISKS as u64,
"default GET data-read metadata must keep full fanout for read-failure tolerance"
);
assert_eq!(diagnostics.total_responses(), DISKS);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == object).count(), DISKS);
assert!(errs.iter().all(Option::is_none));
},
)
@@ -6009,6 +6121,16 @@ mod tests {
assert_eq!(accumulator.candidate_latest_quorum(&impossible_parity), None);
}
#[test]
fn metadata_quorum_accumulator_treats_invalid_default_parity_as_full_fanout() {
let accumulator = MetadataQuorumAccumulator::new(2, 2, true);
assert_eq!(accumulator.default_write_quorum(), 2);
assert_eq!(accumulator.missing_response_quorum(), 2);
assert!(accumulator.can_still_reach_early_stop_with_pending(2));
assert!(!accumulator.can_still_reach_early_stop_with_pending(1));
}
#[test]
fn confirmed_missing_part_error_recognizes_legacy_and_s3_markers() {
assert!(!is_confirmed_missing_part_error(None));
+6 -9
View File
@@ -672,10 +672,10 @@ const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: usize = 128 * 102
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ENABLE";
// Enabled by default (backlog#872): the early-stop path only engages for
// requests `should_allow_metadata_early_stop` classifies as safe (metadata-only
// reads by default, without version_id / healing / free-version needs) and
// still requires a full read-quorum agreement before stopping. Set the env var
// to `false` to fall back to full-wait metadata fanout.
// requests `should_allow_metadata_early_stop` classifies as safe (latest-version
// metadata-only reads by default, without version_id / healing / free-version
// needs) and still requires a full read-quorum agreement before stopping. Set
// the env var to `false` to fall back to full-wait metadata fanout.
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: bool = true;
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT";
@@ -836,11 +836,8 @@ mod prepared_get_object_metadata_tests {
.prepare_get_object_metadata(bucket, object, &opts)
.await
.expect("prepared metadata should resolve");
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
4,
"preparation should fan out to each online disk exactly once"
);
let prepared_calls = calls.total(disk_call_counters::KIND_READ_VERSION);
assert_eq!(prepared_calls, 4, "default prepared GET metadata should keep full data-read fanout");
let mut reader = set_disks
.get_object_reader_with_prepared_metadata(bucket, object, None, HeaderMap::new(), &opts, metadata)
+90
View File
@@ -2558,6 +2558,96 @@ mod heal_result_report_tests {
);
}
#[tokio::test]
async fn replacement_target_readback_checks_the_requested_historical_version() {
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = "replacement-target-readback-versioned";
let object = "object.bin";
set.make_bucket(
bucket,
&MakeBucketOptions {
versioning_enabled: true,
..Default::default()
},
)
.await
.expect("versioned bucket should be created");
let mut old_reader = PutObjReader::from_vec(vec![0x5a; 1024 * 1024]);
let old_info = set
.put_object(
bucket,
object,
&mut old_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("old object version should be written");
let old_version = old_info
.version_id
.expect("versioned put should return the old version id")
.to_string();
let mut latest_reader = PutObjReader::from_vec(vec![0x33; 1024 * 1024]);
let latest_info = set
.put_object(
bucket,
object,
&mut latest_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("latest object version should be written");
let latest_version = latest_info
.version_id
.expect("versioned put should return the latest version id")
.to_string();
let old_source = disks[2]
.read_version("", bucket, object, &old_version, &ReadOptions::default())
.await
.expect("old version metadata should be readable");
let old_data_dir = old_source.data_dir.expect("old version should have a data directory");
let targets = vec![set.set_endpoints[0].to_string(), set.set_endpoints[1].to_string()];
assert!(
set.replacement_targets_have_version(bucket, object, &old_version, &targets)
.await
.expect("healthy historical target shards should be readable")
);
assert!(
set.replacement_targets_have_version(bucket, object, &latest_version, &targets)
.await
.expect("healthy latest target shards should be readable")
);
tokio::fs::remove_file(
temp_dirs[1]
.path()
.join(bucket)
.join(object)
.join(old_data_dir.to_string())
.join("part.1"),
)
.await
.expect("old target shard should be removed after the initial commit");
assert!(
!set.replacement_targets_have_version(bucket, object, &old_version, &targets)
.await
.expect("missing old target shard should be observable")
);
assert!(
set.replacement_targets_have_version(bucket, object, &latest_version, &targets)
.await
.expect("latest target evidence should remain independent")
);
}
#[tokio::test]
async fn format_heal_cached_layout_rejects_a_disk_from_another_slot() {
let mut _temp_dirs = Vec::new();
+16 -4
View File
@@ -3834,25 +3834,24 @@ mod tests {
assert!(metadata_early_stop_permitted(true, true, false, "", false, false));
// observe=false (non-observed fanout) also disables early-stop.
assert!(!metadata_early_stop_permitted(true, false, false, "", false, false));
// Data reads require their own explicit rollout gate.
assert!(!metadata_early_stop_permitted(true, true, true, "", false, false));
},
);
}
#[test]
fn metadata_early_stop_requires_explicit_data_read_opt_in() {
fn metadata_early_stop_keeps_data_reads_opt_in_by_default() {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
(ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, Some("true")),
(ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, None),
(ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, None),
],
|| {
assert!(!should_allow_metadata_early_stop(true, "", false, false));
assert!(!should_allow_metadata_early_stop(true, "version-id", false, false));
assert!(should_allow_metadata_early_stop(false, "", false, false));
assert!(should_allow_metadata_early_stop(false, "version-id", false, false));
assert!(!should_allow_metadata_early_stop(false, "version-id", false, false));
},
);
temp_env::with_vars(
@@ -3866,6 +3865,19 @@ mod tests {
assert!(should_allow_metadata_early_stop(true, "version-id", false, false));
},
);
temp_env::with_vars(
[
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
(ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, Some("true")),
(ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, Some("false")),
],
|| {
assert!(!should_allow_metadata_early_stop(true, "", false, false));
assert!(!should_allow_metadata_early_stop(true, "version-id", false, false));
assert!(should_allow_metadata_early_stop(false, "", false, false));
assert!(should_allow_metadata_early_stop(false, "version-id", false, false));
},
);
}
#[test]
-1
View File
@@ -91,7 +91,6 @@ metrics = { workspace = true }
base64 = { workspace = true }
[dev-dependencies]
libc = { workspace = true }
serde_json = { workspace = true, features = ["raw_value"] }
rustfs-test-utils = { workspace = true }
serial_test = { workspace = true }
+73 -4
View File
@@ -1229,6 +1229,12 @@ mod resume_loop_tests {
Timeout,
}
#[derive(Clone)]
enum ReplacementCommitEvidence {
Confirmed(bool),
Error(String),
}
#[derive(Default)]
struct FakeStorage {
/// page keyed by the *incoming* continuation token
@@ -1239,7 +1245,7 @@ mod resume_loop_tests {
results: Mutex<HashMap<String, HealResultItem>>,
/// Target-specific physical readback evidence per `compose_key`; the
/// fake models a healthy backend unless a test explicitly revokes it.
replacement_commit_evidence: Mutex<HashMap<String, bool>>,
replacement_commit_evidence: Mutex<HashMap<String, ReplacementCommitEvidence>>,
/// every heal_object call recorded as (name, version_id)
heal_calls: Mutex<Vec<(String, Option<String>)>>,
replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
@@ -1260,7 +1266,13 @@ mod resume_loop_tests {
self.replacement_commit_evidence
.lock()
.unwrap()
.insert(compose_key(name, version), committed);
.insert(compose_key(name, version), ReplacementCommitEvidence::Confirmed(committed));
}
fn set_replacement_commit_evidence_error(&self, name: &str, version: Option<&str>, message: &str) {
self.replacement_commit_evidence
.lock()
.unwrap()
.insert(compose_key(name, version), ReplacementCommitEvidence::Error(message.to_string()));
}
fn calls(&self) -> Vec<(String, Option<String>)> {
self.heal_calls.lock().unwrap().clone()
@@ -1354,12 +1366,17 @@ mod resume_loop_tests {
_opts: &HealOpts,
_targets: &[String],
) -> Result<bool> {
Ok(*self
match self
.replacement_commit_evidence
.lock()
.unwrap()
.get(&compose_key(object, version_id))
.unwrap_or(&true))
.cloned()
.unwrap_or(ReplacementCommitEvidence::Confirmed(true))
{
ReplacementCommitEvidence::Confirmed(committed) => Ok(committed),
ReplacementCommitEvidence::Error(message) => Err(Error::other(message)),
}
}
async fn list_objects_for_heal(&self, _b: &str, _p: &str) -> Result<Vec<HealListItem>> {
Ok(Vec::new())
@@ -2142,6 +2159,58 @@ mod resume_loop_tests {
assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
}
#[tokio::test]
async fn replacement_delete_marker_readback_error_keeps_auto_heal_resumable() {
let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
let healer = ErasureSetHealer::new(
env.storage.clone(),
Arc::new(RwLock::new(HealProgress::new())),
CancellationToken::new(),
env.healer.disk.clone(),
HealOpts::default(),
HealRequestSource::AutoHeal,
)
.with_replacement_targets(vec!["replacement-a".to_string()], Some("generation-a".to_string()));
env.storage.set_page(
None,
Page {
items: vec![item("object", Some("dm-v1"), true)],
next: None,
truncated: false,
},
);
env.storage.set_result(
"object",
Some("dm-v1"),
HealResultItem {
after: Infos {
drives: vec![HealDriveInfo {
endpoint: "replacement-a".to_string(),
state: "ok".to_string(),
..Default::default()
}],
},
..Default::default()
},
);
env.storage
.set_replacement_commit_evidence_error("object", Some("dm-v1"), "injected target readback failure");
let result = healer
.execute_heal_with_resume(&["b".to_string()], "pool_0_set_0", &env.resume, &env.checkpoint)
.await;
let error = result.expect_err("a target readback error must not complete automatic replacement");
let error = error.to_string();
assert!(error.contains("Transient heal skip"), "unexpected error: {error}");
assert!(error.contains("retry scheduled"), "unexpected error: {error}");
let state = env.resume.get_state().await;
assert!(!state.completed, "readback errors must leave the replacement task incomplete");
assert_eq!(state.retry_count, 1, "readback errors must arm the bounded retry path");
assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
assert_eq!(env.storage.calls(), vec![("object".to_string(), Some("dm-v1".to_string()))]);
}
#[tokio::test]
async fn manual_targeted_heal_keeps_existing_best_effort_result_semantics() {
let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+272 -5
View File
@@ -65,6 +65,18 @@ fn durable_replacement_recovery_is_due(state: &ResumeState, task_id: &str) -> bo
&& matches!(state.replacement_phase, ReplacementPhase::Verified | ReplacementPhase::CleanupPending)))
}
fn unblock_replacement_recovery_sets_after_validation(
blocked_sets: &mut HashSet<String>,
retry_succeeded: HashSet<String>,
retry_failed: &HashSet<String>,
) {
for set_disk_id in retry_succeeded {
if !retry_failed.contains(&set_disk_id) {
blocked_sets.remove(&set_disk_id);
}
}
}
// Admission/scheduler outcomes for per-object requests (Object/Metadata/MRF/
// ECDecode) log via demote_to_debug_when! — MRF, autoheal, and scanner
// recovery loops submit those per object, so a full queue or a retry storm
@@ -2532,11 +2544,7 @@ impl HealManager {
let mut blocked = replacement_recovery_blocked_sets
.lock()
.expect("replacement recovery blocked set lock poisoned");
for set_disk_id in retry_succeeded {
if !retry_failed.contains(&set_disk_id) {
blocked.remove(&set_disk_id);
}
}
unblock_replacement_recovery_sets_after_validation(&mut blocked, retry_succeeded, &retry_failed);
}
for disk in &local_disks {
let endpoint = disk.endpoint();
@@ -3491,11 +3499,13 @@ fn can_schedule_request(request: &HealRequest, running_per_set: &HashMap<String,
mod tests {
use super::*;
use crate::heal::EcstoreError;
use crate::heal::resume::{CheckpointManager, ReplacementTargetIdentity};
use crate::heal::storage::{HealObjectInfo, HealStorageAPI};
use crate::heal::task::{BatchHealFailure, HealOptions, HealPriority, HealRequest, HealTask, HealType};
use rustfs_common::heal_channel::{HealOpts, HealRequestSource};
use rustfs_concurrency::{WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot};
use rustfs_madmin::heal_commands::HealResultItem;
use std::sync::Mutex as StdMutex;
use tempfile::TempDir;
use super::super::{DiskOption, DiskStore, Endpoint, new_disk, storage_api::status::BucketInfo};
@@ -3616,6 +3626,9 @@ mod tests {
}
async fn list_buckets(&self) -> Result<Vec<BucketInfo>> {
if let Some(hook) = manager_recovery_test_hook() {
*hook.listed.lock().expect("manager recovery listed lock should not poison") = true;
}
Ok(Vec::new())
}
@@ -3638,6 +3651,12 @@ mod tests {
_version_id: Option<&str>,
_opts: &HealOpts,
) -> Result<(HealResultItem, Option<Error>)> {
if let Some(hook) = manager_recovery_test_hook() {
*hook
.heal_object_calls
.lock()
.expect("manager recovery object call lock should not poison") += 1;
}
if bucket == "retry-transition" {
return Ok((
HealResultItem::default(),
@@ -3651,10 +3670,38 @@ mod tests {
}
async fn heal_bucket(&self, _bucket: &str, _opts: &HealOpts) -> Result<HealResultItem> {
if let Some(hook) = manager_recovery_test_hook() {
*hook
.bucket_heal_calls
.lock()
.expect("manager recovery bucket call lock should not poison") += 1;
}
Ok(HealResultItem::default())
}
async fn heal_format(&self, _dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
if let Some(hook) = manager_recovery_test_hook() {
*hook
.global_format_calls
.lock()
.expect("manager recovery global format call lock should not poison") += 1;
}
Ok((HealResultItem::default(), None))
}
async fn heal_replacement_format(
&self,
_dry_run: bool,
_pool_index: usize,
_set_index: usize,
_targets: &[String],
) -> Result<(HealResultItem, Option<Error>)> {
if let Some(hook) = manager_recovery_test_hook() {
*hook
.replacement_format_calls
.lock()
.expect("manager recovery replacement format call lock should not poison") += 1;
}
Ok((HealResultItem::default(), None))
}
@@ -3674,6 +3721,89 @@ mod tests {
async fn get_disk_for_resume(&self, _set_disk_id: &str) -> Result<DiskStore> {
Err(Error::other("not implemented in tests"))
}
async fn get_replacement_resume_disk(
&self,
_set_disk_id: &str,
_task_id: &str,
_excluded_targets: &[String],
) -> Result<crate::heal::storage::ReplacementResumeDisk> {
let Some(hook) = manager_recovery_test_hook() else {
return Err(Error::other("not implemented in tests"));
};
Ok(crate::heal::storage::ReplacementResumeDisk::Existing(
hook.replacement_resume_disk.clone(),
))
}
}
struct ManagerRecoveryTestHook {
replacement_resume_disk: DiskStore,
listed: StdMutex<bool>,
global_format_calls: StdMutex<u32>,
replacement_format_calls: StdMutex<u32>,
bucket_heal_calls: StdMutex<u32>,
heal_object_calls: StdMutex<u32>,
}
static MANAGER_RECOVERY_TEST_HOOK: LazyLock<StdMutex<Option<Arc<ManagerRecoveryTestHook>>>> =
LazyLock::new(|| StdMutex::new(None));
struct ManagerRecoveryTestHookGuard;
impl ManagerRecoveryTestHook {
fn install(replacement_resume_disk: DiskStore) -> (Arc<Self>, ManagerRecoveryTestHookGuard) {
let hook = Arc::new(Self {
replacement_resume_disk,
listed: StdMutex::new(false),
global_format_calls: StdMutex::new(0),
replacement_format_calls: StdMutex::new(0),
bucket_heal_calls: StdMutex::new(0),
heal_object_calls: StdMutex::new(0),
});
let previous = MANAGER_RECOVERY_TEST_HOOK
.lock()
.expect("manager recovery hook lock should not poison")
.replace(hook.clone());
assert!(previous.is_none(), "manager recovery hook already installed");
(hook, ManagerRecoveryTestHookGuard)
}
}
impl Drop for ManagerRecoveryTestHookGuard {
fn drop(&mut self) {
*MANAGER_RECOVERY_TEST_HOOK
.lock()
.expect("manager recovery hook lock should not poison") = None;
}
}
fn manager_recovery_test_hook() -> Option<Arc<ManagerRecoveryTestHook>> {
MANAGER_RECOVERY_TEST_HOOK
.lock()
.expect("manager recovery hook lock should not poison")
.clone()
}
async fn make_manager_resume_disk(temp: &TempDir, name: &str) -> DiskStore {
let disk_path = temp.path().join(name);
std::fs::create_dir_all(&disk_path).expect("manager recovery disk directory should be created");
let endpoint = Endpoint::try_from(disk_path.to_string_lossy().as_ref()).expect("manager recovery endpoint should parse");
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("manager recovery disk should initialize");
let metadata_volume = disk.make_volume(super::super::RUSTFS_META_BUCKET).await;
assert!(
matches!(metadata_volume, Ok(()) | Err(DiskError::VolumeExists)),
"manager recovery metadata volume should exist: {metadata_volume:?}"
);
disk
}
fn bucket_request(bucket: &str, priority: HealPriority, source: HealRequestSource) -> HealRequest {
@@ -4525,6 +4655,143 @@ mod tests {
));
}
#[test]
fn replacement_recovery_retry_barrier_requires_all_set_records_to_validate() {
let mut blocked = HashSet::from(["pool_0_set_0".to_string(), "pool_0_set_1".to_string()]);
let retry_succeeded = HashSet::from(["pool_0_set_0".to_string(), "pool_0_set_1".to_string()]);
let retry_failed = HashSet::from(["pool_0_set_0".to_string()]);
unblock_replacement_recovery_sets_after_validation(&mut blocked, retry_succeeded, &retry_failed);
assert!(
blocked.contains("pool_0_set_0"),
"one failed disk record must keep the whole replacement set blocked"
);
assert!(
!blocked.contains("pool_0_set_1"),
"a blocked set may resume only after every retried record validates"
);
}
#[tokio::test]
async fn scheduler_completes_cleanup_pending_recovery_from_manager_anchor() {
let temp = TempDir::new().expect("temporary manager recovery directory should be created");
let anchor = make_manager_resume_disk(&temp, "anchor").await;
let task_id = ResumeUtils::generate_task_id();
let target = "replacement-a".to_string();
let identity = ReplacementTargetIdentity {
endpoint: target.clone(),
canonical_path: "/replacement/replacement-a".to_string(),
physical_device_ids: vec!["replacement-a".to_string()],
filesystem_identity: "identity-replacement-a".to_string(),
};
let resume_manager = ResumeManager::new_replacement_intent(
anchor.clone(),
task_id.clone(),
"pool_0_set_0".to_string(),
vec!["bucket-a".to_string()],
vec![target.clone()],
vec![identity],
)
.await
.expect("cleanup-pending replacement state should persist on the survivor anchor");
resume_manager
.mark_replacement_completed_and_verified()
.await
.expect("completion proof should persist before cleanup");
resume_manager
.mark_replacement_cleanup_pending()
.await
.expect("cleanup-pending state should persist before restart");
CheckpointManager::new(anchor.clone(), task_id.clone())
.await
.expect("checkpoint fixture should persist");
let (hook, _hook_guard) = ManagerRecoveryTestHook::install(anchor.clone());
let storage = Arc::new(MockStorage);
let manager = HealManager::new(storage.clone(), None);
let mut request = HealRequest::new(
HealType::ErasureSet {
buckets: vec!["bucket-a".to_string()],
set_disk_id: "pool_0_set_0".to_string(),
},
HealOptions {
pool_index: Some(0),
set_index: Some(0),
..HealOptions::default()
},
HealPriority::Low,
);
request.id = task_id.clone();
request.source = HealRequestSource::AutoHeal;
request.heal_endpoints = vec![target];
assert_eq!(
manager
.submit_heal_request(request)
.await
.expect("durable recovery request should be admitted"),
HealAdmissionResult::Accepted
);
manager
.replacement_recovery_anchors
.lock()
.expect("replacement recovery anchor lock should not poison")
.insert(task_id.clone(), anchor.endpoint().to_string());
process_manager_queue_once(&manager).await;
tokio::time::timeout(Duration::from_secs(2), async {
loop {
let resume_removed = !ResumeManager::has_resume_state(&anchor, &task_id).await;
let checkpoint_removed = !CheckpointManager::has_checkpoint(&anchor, &task_id).await;
let anchor_removed = !manager
.replacement_recovery_anchors
.lock()
.expect("replacement recovery anchor lock should not poison")
.contains_key(&task_id);
if resume_removed && checkpoint_removed && anchor_removed {
break;
}
tokio::task::yield_now().await;
}
})
.await
.expect("cleanup-pending recovery should finish through the manager scheduler");
assert!(
!*hook.listed.lock().expect("manager recovery listed lock should not poison"),
"cleanup-pending recovery must not list buckets or restart object healing"
);
assert_eq!(
*hook
.global_format_calls
.lock()
.expect("manager recovery global format call lock should not poison"),
0
);
assert_eq!(
*hook
.replacement_format_calls
.lock()
.expect("manager recovery replacement format call lock should not poison"),
0,
"manager-resumed terminal cleanup must not format replacement targets"
);
assert_eq!(
*hook
.bucket_heal_calls
.lock()
.expect("manager recovery bucket call lock should not poison"),
0
);
assert_eq!(
*hook
.heal_object_calls
.lock()
.expect("manager recovery object call lock should not poison"),
0
);
}
#[test]
fn test_retry_request_for_scoped_slowdown_preserves_scope() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -157,225 +157,4 @@ mod tests {
)
.await;
}
#[cfg(target_os = "linux")]
mod linux_privileged_tests {
use super::*;
use std::error::Error;
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
use std::path::Path;
const ENABLE_ENV: &str = "RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS";
struct MountGuard {
mounts: Vec<std::path::PathBuf>,
}
impl MountGuard {
fn new() -> Result<Self, Box<dyn Error + Send + Sync>> {
let rc = unsafe { libc::unshare(libc::CLONE_NEWNS) };
if rc != 0 {
return Err(format!("unshare(CLONE_NEWNS) failed: {}", std::io::Error::last_os_error()).into());
}
make_mounts_private()?;
Ok(Self { mounts: Vec::new() })
}
fn mount_tmpfs(&mut self, target: &Path, label: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
mount_tmpfs(target, label)?;
self.mounts.push(target.to_path_buf());
Ok(())
}
fn mount_bind(&mut self, source: &Path, target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
mount_bind(source, target)?;
self.mounts.push(target.to_path_buf());
Ok(())
}
}
impl Drop for MountGuard {
fn drop(&mut self) {
for mount in self.mounts.iter().rev() {
if let Ok(target) = c_path(mount) {
let _ = unsafe { libc::umount2(target.as_ptr(), libc::MNT_DETACH) };
}
}
}
}
fn c_path(path: &Path) -> Result<CString, Box<dyn Error + Send + Sync>> {
Ok(CString::new(path.as_os_str().as_bytes())?)
}
fn make_mounts_private() -> Result<(), Box<dyn Error + Send + Sync>> {
let root = CString::new("/")?;
let rc = unsafe {
libc::mount(
std::ptr::null(),
root.as_ptr(),
std::ptr::null(),
(libc::MS_REC | libc::MS_PRIVATE) as libc::c_ulong,
std::ptr::null(),
)
};
if rc != 0 {
return Err(format!("making the mount namespace private failed: {}", std::io::Error::last_os_error()).into());
}
Ok(())
}
fn mount_tmpfs(target: &Path, label: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
let source = CString::new(label)?;
let target = c_path(target)?;
let fstype = CString::new("tmpfs")?;
let data = CString::new("size=32m,mode=0700")?;
let rc = unsafe {
libc::mount(
source.as_ptr(),
target.as_ptr(),
fstype.as_ptr(),
(libc::MS_NOSUID | libc::MS_NODEV) as libc::c_ulong,
data.as_ptr().cast(),
)
};
if rc != 0 {
return Err(format!("mount(tmpfs) failed: {}", std::io::Error::last_os_error()).into());
}
Ok(())
}
fn mount_bind(source: &Path, target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
let source = c_path(source)?;
let target = c_path(target)?;
let rc = unsafe {
libc::mount(
source.as_ptr(),
target.as_ptr(),
std::ptr::null(),
libc::MS_BIND as libc::c_ulong,
std::ptr::null(),
)
};
if rc != 0 {
return Err(format!("mount(MS_BIND) failed: {}", std::io::Error::last_os_error()).into());
}
Ok(())
}
fn privileged_enabled() -> Result<bool, Box<dyn Error + Send + Sync>> {
let enabled = std::env::var(ENABLE_ENV)
.ok()
.is_some_and(|value| matches!(value.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"));
if !enabled {
return Ok(false);
}
if unsafe { libc::geteuid() } != 0 {
return Err(format!("{ENABLE_ENV}=1 requires root or CAP_SYS_ADMIN").into());
}
Ok(true)
}
fn run_privileged_mount_test<F, Fut>(test: F) -> Result<(), Box<dyn Error + Send + Sync>>
where
F: FnOnce(MountGuard) -> Fut + Send + 'static,
Fut: std::future::Future<Output = Result<(), Box<dyn Error + Send + Sync>>> + 'static,
{
if !privileged_enabled()? {
return Ok(());
}
std::thread::spawn(move || {
let guard = MountGuard::new()?;
let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build()?;
runtime.block_on(test(guard))
})
.join()
.map_err(|_| "privileged mount readiness test thread panicked")?
}
#[test]
#[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS=1"]
fn auto_replacement_readiness_accepts_an_independent_mount() -> Result<(), Box<dyn Error + Send + Sync>> {
run_privileged_mount_test(|mut mounts| async move {
let temp = TempDir::new().expect("temporary replacement roots should be created");
let target = temp.path().join("target");
let sibling = temp.path().join("sibling");
std::fs::create_dir(&target).expect("target mountpoint should be created");
std::fs::create_dir(&sibling).expect("sibling mountpoint should be created");
mounts.mount_tmpfs(&target, "rustfs-readiness-target")?;
mounts.mount_tmpfs(&sibling, "rustfs-readiness-sibling")?;
let target_endpoint = Endpoint::try_from(target.to_string_lossy().as_ref())?;
let sibling_endpoint = Endpoint::try_from(sibling.to_string_lossy().as_ref())?;
let target_disk = new_disk(
&target_endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await?;
let sibling_disk = new_disk(
&sibling_endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await?;
let identity = auto_replacement_target_identity(&target_disk, &[target_disk.clone(), sibling_disk.clone()]).await;
assert!(
identity.is_some(),
"a separately mounted replacement target with no sibling device overlap must be admitted"
);
Ok(())
})
}
#[test]
#[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS=1"]
fn auto_replacement_readiness_rejects_a_same_device_sibling_bind_mount() -> Result<(), Box<dyn Error + Send + Sync>> {
run_privileged_mount_test(|mut mounts| async move {
let temp = TempDir::new().expect("temporary replacement roots should be created");
let source = temp.path().join("source");
let target = temp.path().join("target");
let sibling = temp.path().join("sibling");
std::fs::create_dir(&source).expect("source mountpoint should be created");
std::fs::create_dir(&target).expect("target mountpoint should be created");
std::fs::create_dir(&sibling).expect("sibling mountpoint should be created");
mounts.mount_tmpfs(&source, "rustfs-readiness-shared-source")?;
mounts.mount_bind(&source, &target)?;
mounts.mount_bind(&source, &sibling)?;
let target_endpoint = Endpoint::try_from(target.to_string_lossy().as_ref())?;
let sibling_endpoint = Endpoint::try_from(sibling.to_string_lossy().as_ref())?;
let target_disk = new_disk(
&target_endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await?;
let sibling_disk = new_disk(
&sibling_endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await?;
assert!(
auto_replacement_target_identity(&target_disk, &[target_disk.clone(), sibling_disk.clone()])
.await
.is_none(),
"replacement readiness must reject a target sharing its physical device with a sibling endpoint"
);
Ok(())
})
}
}
}
+141
View File
@@ -2997,6 +2997,147 @@ mod tests {
assert!(!*storage.listed.lock().unwrap());
}
#[tokio::test]
async fn cleanup_pending_recovery_removes_checkpoint_without_rebuild_work() {
let temp = TempDir::new().expect("temporary resume disk directory should be created");
let anchor = make_resume_disk(&temp).await;
let task_id = crate::heal::resume::ResumeUtils::generate_task_id();
let identity = replacement_identity("replacement-a", "device-a", "filesystem-a");
let resume_manager = ResumeManager::new_replacement_intent(
anchor.clone(),
task_id.clone(),
"pool_0_set_0".to_string(),
vec!["bucket-a".to_string()],
vec!["replacement-a".to_string()],
vec![identity],
)
.await
.expect("terminal replacement state should persist on the survivor anchor");
resume_manager
.mark_replacement_completed_and_verified()
.await
.expect("terminal replacement proof should persist before cleanup");
resume_manager
.mark_replacement_cleanup_pending()
.await
.expect("failed cleanup must retain a cleanup-pending state");
CheckpointManager::new(anchor.clone(), task_id.clone())
.await
.expect("checkpoint fixture should persist");
assert!(
CheckpointManager::has_checkpoint(&anchor, &task_id).await,
"checkpoint fixture must exist before restart cleanup"
);
let storage = Arc::new(MockStorage {
replacement_resume_disk: Mutex::new(Some(anchor.clone())),
..Default::default()
});
let mut request = HealRequest::new(
HealType::ErasureSet {
buckets: vec!["bucket-a".to_string()],
set_disk_id: "pool_0_set_0".to_string(),
},
HealOptions {
pool_index: Some(0),
set_index: Some(0),
..HealOptions::default()
},
HealPriority::Low,
);
request.id = task_id.clone();
request.source = HealRequestSource::AutoHeal;
request.heal_endpoints = vec!["replacement-a".to_string()];
HealTask::from_replacement_recovery_request(request, storage.clone(), Some(anchor.endpoint().to_string()))
.execute()
.await
.expect("cleanup-pending recovery must finish terminal cleanup");
assert!(
!CheckpointManager::has_checkpoint(&anchor, &task_id).await,
"terminal cleanup must remove the retained checkpoint"
);
assert!(
!ResumeManager::has_resume_state(&anchor, &task_id).await,
"terminal cleanup must remove the retained resume state"
);
assert_eq!(*storage.global_format_calls.lock().unwrap(), 0);
assert!(
storage.replacement_format_calls.lock().unwrap().is_empty(),
"terminal checkpoint cleanup must not format replacement targets"
);
assert!(storage.bucket_heal_calls.lock().unwrap().is_empty());
assert!(storage.heal_object_calls.lock().unwrap().is_empty());
assert!(!*storage.listed.lock().unwrap());
}
#[tokio::test]
async fn verified_recovery_keeps_state_when_marker_clear_fails() {
let temp = TempDir::new().expect("temporary resume disk directory should be created");
let anchor = make_resume_disk(&temp).await;
let task_id = crate::heal::resume::ResumeUtils::generate_task_id();
let target = format!("replacement-marker-missing-{task_id}");
let identity = replacement_identity(&target, &target, &format!("identity-{target}"));
let resume_manager = ResumeManager::new_replacement_intent(
anchor.clone(),
task_id.clone(),
"pool_0_set_0".to_string(),
vec!["bucket-a".to_string()],
vec![target.clone()],
vec![identity],
)
.await
.expect("verified replacement state should persist on the survivor anchor");
resume_manager
.mark_replacement_completed_and_verified()
.await
.expect("verified state must persist proof before marker cleanup");
let storage = Arc::new(MockStorage {
replacement_resume_disk: Mutex::new(Some(anchor.clone())),
replacement_targets_ready: Mutex::new(true),
..Default::default()
});
let mut request = HealRequest::new(
HealType::ErasureSet {
buckets: vec!["bucket-a".to_string()],
set_disk_id: "pool_0_set_0".to_string(),
},
HealOptions {
pool_index: Some(0),
set_index: Some(0),
..HealOptions::default()
},
HealPriority::Low,
);
request.id = task_id.clone();
request.source = HealRequestSource::AutoHeal;
request.heal_endpoints = vec![target];
let error = HealTask::from_replacement_recovery_request(request, storage.clone(), Some(anchor.endpoint().to_string()))
.execute()
.await
.expect_err("marker clear failure must keep the durable terminal state retryable");
assert!(error.to_string().contains("healing marker target is unavailable"));
let state = ResumeManager::load_replacement_intent(anchor.clone(), &task_id)
.await
.expect("verified state must remain for retry after marker clear failure")
.get_state()
.await;
assert!(state.completed);
assert_eq!(state.replacement_phase, ReplacementPhase::Verified);
assert_eq!(*storage.global_format_calls.lock().unwrap(), 0);
assert!(
storage.replacement_format_calls.lock().unwrap().is_empty(),
"marker cleanup retry must not format replacement targets again"
);
assert!(storage.bucket_heal_calls.lock().unwrap().is_empty());
assert!(storage.heal_object_calls.lock().unwrap().is_empty());
assert!(!*storage.listed.lock().unwrap());
}
#[derive(Default)]
struct MockStorage {
listed: Mutex<bool>,
+3
View File
@@ -107,7 +107,10 @@ url = { workspace = true }
[dev-dependencies]
pollster.workspace = true
rcgen.workspace = true
rustfs-test-utils = { workspace = true }
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls-pki-types.workspace = true
serial_test = { workspace = true }
temp-env = { workspace = true, features = ["async_closure"] }
tempfile = { workspace = true }
+23 -2
View File
@@ -14,7 +14,7 @@
use crate::error::{Error, Result};
use manager::IamCache;
use oidc::OidcSys;
use oidc::{OidcExtraRootCaProvider, OidcSys};
use std::sync::{Arc, OnceLock};
use store::object::ObjectStore;
use sys::IamSys;
@@ -284,6 +284,23 @@ pub fn get_global_iam_sys() -> Option<Arc<IamSys<ObjectStore>>> {
/// Initialize the global OIDC system. Non-fatal if no OIDC providers are configured.
pub async fn init_oidc_sys() -> Result<()> {
init_oidc_sys_with_extra_root_ca(None).await
}
/// Initialize the global OIDC system with an additional outbound root CA bundle.
pub async fn init_oidc_sys_with_extra_root_ca(root_ca_pem: Option<&[u8]>) -> Result<()> {
init_oidc_sys_with_extra_root_ca_provider_inner(None, root_ca_pem).await
}
/// Initialize the global OIDC system with a reload-aware outbound root CA provider.
pub async fn init_oidc_sys_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<()> {
init_oidc_sys_with_extra_root_ca_provider_inner(Some(extra_root_ca_provider), None).await
}
async fn init_oidc_sys_with_extra_root_ca_provider_inner(
extra_root_ca_provider: Option<OidcExtraRootCaProvider>,
root_ca_pem: Option<&[u8]>,
) -> Result<()> {
if OIDC_SYS.get().is_some() {
debug!(
event = EVENT_OIDC_STATE,
@@ -303,7 +320,11 @@ pub async fn init_oidc_sys() -> Result<()> {
"OIDC runtime starting"
);
let oidc_sys = match OidcSys::new().await {
let oidc_sys_result = match extra_root_ca_provider {
Some(provider) => OidcSys::new_with_extra_root_ca_provider(provider).await,
None => OidcSys::new_with_extra_root_ca(root_ca_pem).await,
};
let oidc_sys = match oidc_sys_result {
Ok(sys) => {
if sys.has_providers() {
debug!(
+420 -48
View File
@@ -25,7 +25,7 @@ use openidconnect::{
JsonWebKeySetUrl, LogoutRequest, Nonce, PkceCodeChallenge, PkceCodeVerifier, PostLogoutRedirectUrl,
ProviderMetadataWithLogout, RedirectUrl, RequestTokenError, Scope,
};
use reqwest::Client;
use reqwest::{Certificate, Client};
use rustfs_config::oidc::*;
use rustfs_config::server_config::{Config as ServerConfig, KVS};
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MAX_OIDC_RESPONSE_SIZE};
@@ -38,7 +38,6 @@ use std::fmt;
use std::future::Future;
use std::net::IpAddr;
use std::pin::Pin;
#[cfg(test)]
use std::sync::Arc;
use std::sync::{LazyLock, Mutex, MutexGuard, RwLock};
use std::time::{Duration as StdDuration, Instant};
@@ -258,6 +257,7 @@ fn oidc_http_error_diagnostics(error: &OidcHttpError) -> (&'static str, String)
}
OidcHttpError::Reqwest(_) => ("request", String::new()),
OidcHttpError::Http(_) => ("http_build", String::new()),
OidcHttpError::ExtraRootCa(_) => ("extra_root_ca", String::new()),
OidcHttpError::ForbiddenOutbound(_) => ("forbidden_outbound", String::new()),
OidcHttpError::ResponseTooLarge(limit) => ("response_too_large", limit.to_string()),
}
@@ -270,6 +270,7 @@ fn oidc_http_error_diagnostics(error: &OidcHttpError) -> (&'static str, String)
pub enum OidcHttpError {
Reqwest(reqwest::Error),
Http(http::Error),
ExtraRootCa(String),
/// The outbound destination was rejected by the shared egress policy before any
/// connection was attempted (invalid URL, loopback/link-local/metadata/private IP,
/// or a malformed allow-origins configuration).
@@ -284,6 +285,7 @@ impl std::fmt::Display for OidcHttpError {
match self {
Self::Reqwest(e) => write!(f, "{e}"),
Self::Http(e) => write!(f, "{e}"),
Self::ExtraRootCa(reason) => write!(f, "failed to load OIDC extra root CA bundle: {reason}"),
Self::ForbiddenOutbound(reason) => write!(f, "outbound request rejected: {reason}"),
Self::ResponseTooLarge(limit) => write!(f, "oidc response body exceeds {limit} bytes"),
}
@@ -295,11 +297,48 @@ impl std::error::Error for OidcHttpError {
match self {
Self::Reqwest(e) => Some(e),
Self::Http(e) => Some(e),
Self::ForbiddenOutbound(_) | Self::ResponseTooLarge(_) => None,
Self::ExtraRootCa(_) | Self::ForbiddenOutbound(_) | Self::ResponseTooLarge(_) => None,
}
}
}
#[derive(Clone, Debug, Default)]
pub struct OidcExtraRootCaMaterial {
pub generation: u64,
pub root_ca_pem: Option<Vec<u8>>,
}
type OidcExtraRootCaFuture = Pin<Box<dyn Future<Output = Result<OidcExtraRootCaMaterial, String>> + Send>>;
type OidcExtraRootCaLoader = dyn Fn() -> OidcExtraRootCaFuture + Send + Sync;
#[derive(Clone)]
pub struct OidcExtraRootCaProvider {
loader: Arc<OidcExtraRootCaLoader>,
}
impl OidcExtraRootCaProvider {
pub fn new<F, Fut>(loader: F) -> Self
where
F: Fn() -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<OidcExtraRootCaMaterial, String>> + Send + 'static,
{
Self {
loader: Arc::new(move || Box::pin(loader())),
}
}
async fn load(&self) -> Result<OidcExtraRootCaMaterial, String> {
(self.loader)().await
}
}
#[derive(Clone, Default)]
struct CachedOidcExtraRootCerts {
generation: u64,
initialized: bool,
certs: Vec<Certificate>,
}
/// HTTP client adapter bridging reqwest 0.13 to the `openidconnect` `AsyncHttpClient` trait.
///
/// A fresh client is built for every request so the destination is re-validated and the
@@ -311,10 +350,26 @@ pub(crate) struct ReqwestHttpClient {
/// `None` in production: the process-cached outbound policy from the environment is used.
/// `Some(..)` only in tests, to explicitly allow a loopback mock endpoint.
policy_override: Option<OutboundPolicy>,
extra_root_certs: Arc<RwLock<CachedOidcExtraRootCerts>>,
extra_root_ca_provider: Option<OidcExtraRootCaProvider>,
#[cfg(test)]
dns_resolver_override: Option<Arc<dyn reqwest::dns::Resolve>>,
}
fn parse_oidc_extra_root_certs(source: &str, pem: &[u8]) -> Result<Vec<Certificate>, String> {
if pem.iter().all(|byte| byte.is_ascii_whitespace()) {
return Ok(Vec::new());
}
Certificate::from_pem_bundle(pem).map_err(|err| format!("failed to parse OIDC extra root CA bundle from {source}: {err}"))
}
fn oidc_extra_root_certs(root_ca_pem: Option<&[u8]>) -> Result<Vec<Certificate>, String> {
match root_ca_pem {
Some(pem) => parse_oidc_extra_root_certs("RustFS outbound TLS material", pem),
None => Ok(Vec::new()),
}
}
/// Build a reqwest client pinned to the shared outbound egress policy for a single request.
///
/// [`OutboundPolicy::resolver_for`] validates the URL shape and rejects loopback,
@@ -326,6 +381,7 @@ pub(crate) struct ReqwestHttpClient {
fn build_oidc_http_client(
uri: &str,
policy_override: Option<&OutboundPolicy>,
extra_root_certs: &[Certificate],
#[cfg(test)] dns_resolver_override: Option<Arc<dyn reqwest::dns::Resolve>>,
) -> Result<(Client, Url), OidcHttpError> {
let url = Url::parse(uri).map_err(|_| OidcHttpError::ForbiddenOutbound("invalid outbound OIDC URL".to_string()))?;
@@ -356,6 +412,9 @@ fn build_oidc_http_client(
if bypass_proxy {
builder = builder.no_proxy();
}
if !extra_root_certs.is_empty() {
builder = builder.tls_certs_merge(extra_root_certs.iter().cloned());
}
builder.build().map(|client| (client, url)).map_err(OidcHttpError::Reqwest)
}
@@ -410,19 +469,93 @@ fn should_bypass_proxy_for_oidc_uri(uri: &str) -> bool {
impl ReqwestHttpClient {
fn new() -> Result<Self, String> {
Self::new_with_extra_root_certs(Vec::new())
}
fn extra_root_cert_cache(certs: Vec<Certificate>) -> Arc<RwLock<CachedOidcExtraRootCerts>> {
Arc::new(RwLock::new(CachedOidcExtraRootCerts {
generation: 0,
initialized: true,
certs,
}))
}
fn new_with_extra_root_certs(extra_root_certs: Vec<Certificate>) -> Result<Self, String> {
Ok(Self {
policy_override: None,
extra_root_certs: Self::extra_root_cert_cache(extra_root_certs),
extra_root_ca_provider: None,
#[cfg(test)]
dns_resolver_override: None,
})
}
fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<Self, String> {
Ok(Self {
policy_override: None,
extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())),
extra_root_ca_provider: Some(extra_root_ca_provider),
#[cfg(test)]
dns_resolver_override: None,
})
}
async fn current_extra_root_certs(&self) -> Result<Vec<Certificate>, OidcHttpError> {
let Some(provider) = self.extra_root_ca_provider.as_ref() else {
return self
.extra_root_certs
.read()
.map(|cache| cache.certs.clone())
.map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")));
};
let material = provider.load().await.map_err(OidcHttpError::ExtraRootCa)?;
if let Ok(cache) = self.extra_root_certs.read()
&& cache.initialized
&& cache.generation == material.generation
{
return Ok(cache.certs.clone());
}
let certs = oidc_extra_root_certs(material.root_ca_pem.as_deref()).map_err(OidcHttpError::ExtraRootCa)?;
let mut cache = self
.extra_root_certs
.write()
.map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")))?;
cache.generation = material.generation;
cache.initialized = true;
cache.certs = certs.clone();
Ok(certs)
}
/// Test-only constructor that pins outbound requests to an explicit policy, so a
/// loopback mock server can be reached without depending on process-wide environment.
#[cfg(test)]
fn with_policy(policy: OutboundPolicy) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(Vec::new()),
extra_root_ca_provider: None,
dns_resolver_override: None,
}
}
#[cfg(test)]
fn with_policy_and_extra_root_certs(policy: OutboundPolicy, extra_root_certs: Vec<Certificate>) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(extra_root_certs),
extra_root_ca_provider: None,
dns_resolver_override: None,
}
}
#[cfg(test)]
fn with_policy_and_extra_root_ca_provider(policy: OutboundPolicy, extra_root_ca_provider: OidcExtraRootCaProvider) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())),
extra_root_ca_provider: Some(extra_root_ca_provider),
dns_resolver_override: None,
}
}
@@ -431,6 +564,8 @@ impl ReqwestHttpClient {
fn with_policy_and_dns_resolver(policy: OutboundPolicy, resolver: Arc<dyn reqwest::dns::Resolve>) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(Vec::new()),
extra_root_ca_provider: None,
dns_resolver_override: Some(resolver),
}
}
@@ -461,9 +596,11 @@ impl<'c> AsyncHttpClient<'c> for ReqwestHttpClient {
);
}
let extra_root_certs = self.current_extra_root_certs().await?;
let (client, url) = build_oidc_http_client(
&uri,
self.policy_override.as_ref(),
&extra_root_certs,
#[cfg(test)]
self.dns_resolver_override.clone(),
)?;
@@ -678,7 +815,22 @@ fn trusted_aud(other_audiences: &[String], audience: &Audience) -> bool {
impl OidcSys {
/// Parse environment variables and discover all configured OIDC providers.
pub async fn new() -> Result<Self, String> {
let http_client = ReqwestHttpClient::new()?;
Self::new_with_extra_root_ca(None).await
}
/// Parse environment variables and discover providers with an additional outbound root CA bundle.
pub(crate) async fn new_with_extra_root_ca(root_ca_pem: Option<&[u8]>) -> Result<Self, String> {
let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?;
Self::new_with_http_client(http_client).await
}
pub(crate) async fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<Self, String> {
let http_client = ReqwestHttpClient::new_with_extra_root_ca_provider(extra_root_ca_provider)?;
http_client.current_extra_root_certs().await.map_err(|err| err.to_string())?;
Self::new_with_http_client(http_client).await
}
async fn new_with_http_client(http_client: ReqwestHttpClient) -> Result<Self, String> {
let server_config = crate::server_config::current_server_config();
let parsed_configs = load_effective_oidc_provider_configs(server_config.as_ref());
let mut configs = HashMap::new();
@@ -1874,7 +2026,14 @@ pub fn load_effective_oidc_provider_configs(server_config: Option<&ServerConfig>
}
pub async fn validate_oidc_provider_config(config: &OidcProviderConfig) -> Result<OidcProviderValidationResult, String> {
let http_client = ReqwestHttpClient::new()?;
validate_oidc_provider_config_with_extra_root_ca(config, None).await
}
pub async fn validate_oidc_provider_config_with_extra_root_ca(
config: &OidcProviderConfig,
root_ca_pem: Option<&[u8]>,
) -> Result<OidcProviderValidationResult, String> {
let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?;
let state = OidcSys::discover_provider(config, &http_client).await?;
Ok(OidcProviderValidationResult {
@@ -2438,6 +2597,50 @@ mod tests {
}
}
fn read_mock_oidc_request_path(stream: &mut impl std::io::Read) -> String {
let mut request_bytes = Vec::new();
let mut buffer = [0u8; 4096];
loop {
match stream.read(&mut buffer) {
Ok(0) => break,
Ok(n) => request_bytes.extend_from_slice(&buffer[..n]),
Err(e) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => break,
Err(_) => break,
}
if request_bytes.windows(4).any(|w| w == b"\r\n\r\n") {
break;
}
if request_bytes.len() >= 8192 {
break;
}
}
let request = String::from_utf8_lossy(&request_bytes);
request
.lines()
.next()
.unwrap_or("")
.split_whitespace()
.nth(1)
.unwrap_or("")
.to_string()
}
fn mock_oidc_response(path: &str, discovery_body: &str, expected_jwks_path: &str, jwks_body: &str) -> String {
let (status, body) = if path.contains("/.well-known/openid-configuration") {
(200, discovery_body)
} else if path == expected_jwks_path {
(200, jwks_body)
} else {
(404, r#"{"error":"not found"}"#)
};
format!(
"HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
if status == 200 { "OK" } else { "Not Found" },
body.len()
)
}
fn start_mock_oidc_discovery_server<F>(
build_discovery_issuer: F,
max_requests: usize,
@@ -2445,7 +2648,6 @@ mod tests {
where
F: Fn(&str) -> (String, String, String) + Send + 'static,
{
use std::io::Read;
use std::io::Write;
use std::net::{Shutdown, TcpListener};
use std::sync::mpsc;
@@ -2516,41 +2718,8 @@ mod tests {
.set_read_timeout(Some(Duration::from_secs(1)))
.expect("failed to set discovery mock read timeout");
let mut request_bytes = Vec::new();
let mut buffer = [0u8; 4096];
loop {
match stream.read(&mut buffer) {
Ok(0) => break,
Ok(n) => request_bytes.extend_from_slice(&buffer[..n]),
Err(e) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => {
break;
}
Err(_) => break,
}
if request_bytes.windows(4).any(|w| w == b"\r\n\r\n") {
break;
}
if request_bytes.len() >= 8192 {
break;
}
}
let request = String::from_utf8_lossy(&request_bytes);
let path = request.lines().next().unwrap_or("").split_whitespace().nth(1).unwrap_or("");
let (status, body) = if path.contains("/.well-known/openid-configuration") {
(200, discovery_body.as_str())
} else if path == expected_jwks_path {
(200, jwks_body)
} else {
(404, r#"{"error":"not found"}"#)
};
let response = format!(
"HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
if status == 200 { "OK" } else { "Not Found" },
body.len()
);
let path = read_mock_oidc_request_path(&mut stream);
let response = mock_oidc_response(&path, &discovery_body, &expected_jwks_path, jwks_body);
let _ = stream.write_all(response.as_bytes());
let _ = stream.flush();
let _ = stream.shutdown(Shutdown::Both);
@@ -2568,6 +2737,114 @@ mod tests {
Some((base, handle))
}
fn start_mock_oidc_tls_discovery_server<F>(
build_discovery_issuer: F,
max_requests: usize,
) -> Option<(String, String, std::thread::JoinHandle<()>)>
where
F: Fn(&str) -> (String, String, String) + Send + 'static,
{
use std::io::Write;
use std::net::{Shutdown, TcpListener};
use std::sync::mpsc;
use std::time::{Duration, Instant};
const IDLE_SHUTDOWN: Duration = Duration::from_secs(1);
const ABSOLUTE_CAP: Duration = Duration::from_secs(5);
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let certified =
rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).expect("generate OIDC TLS test certificate");
let cert_pem = certified.cert.pem();
let server_config = rustls::ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(
vec![certified.cert.der().clone()],
rustls_pki_types::PrivateKeyDer::try_from(certified.signing_key.serialize_der())
.expect("convert OIDC TLS test private key"),
)
.expect("build OIDC TLS mock server config");
let listener = match TcpListener::bind("127.0.0.1:0") {
Ok(listener) => listener,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
Err(err) => panic!("test TLS listener should bind: {err}"),
};
let base = format!("https://{}", listener.local_addr().expect("listener local address should be available"));
let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base);
let discovery_body = serde_json::json!({
"issuer": discovery_issuer,
"authorization_endpoint": format!("{base}/authorize"),
"token_endpoint": format!("{base}/token"),
"jwks_uri": discovery_jwks_uri,
"response_types_supported": ["code"],
"response_modes_supported": ["query"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["RS256"],
})
.to_string();
let jwks_body = r#"{"keys":[]}"#;
let (ready_tx, ready_rx) = mpsc::channel();
let handle = std::thread::spawn(move || {
let server_config = Arc::new(server_config);
listener
.set_nonblocking(true)
.expect("failed to set TLS discovery mock listener non-blocking");
let _ = ready_tx.send(());
let mut seen = 0usize;
let start = Instant::now();
let mut last_completed = Instant::now();
loop {
if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN {
break;
}
if start.elapsed() >= ABSOLUTE_CAP {
break;
}
let tcp_stream = match listener.accept() {
Ok((stream, _)) => stream,
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(5));
continue;
}
Err(_) => break,
};
tcp_stream
.set_nonblocking(false)
.expect("failed to set TLS discovery mock stream blocking");
tcp_stream
.set_read_timeout(Some(Duration::from_secs(1)))
.expect("failed to set TLS discovery mock read timeout");
seen += 1;
let connection = match rustls::ServerConnection::new(server_config.clone()) {
Ok(connection) => connection,
Err(_) => break,
};
let mut stream = rustls::StreamOwned::new(connection, tcp_stream);
let path = read_mock_oidc_request_path(&mut stream);
let response = mock_oidc_response(&path, &discovery_body, &expected_jwks_path, jwks_body);
let _ = stream.write_all(response.as_bytes());
let _ = stream.flush();
let _ = stream.sock.shutdown(Shutdown::Both);
last_completed = Instant::now();
if seen >= max_requests {
break;
}
}
});
ready_rx
.recv_timeout(Duration::from_millis(100))
.expect("mock TLS OIDC discovery server should become ready");
Some((base, cert_pem, handle))
}
fn discovery_error_contains_all_variants(err: &str, base: &str) -> bool {
err.contains(base) && err.contains(&format!("{base}/")) && err.contains("discovery failed for all issuer variants")
}
@@ -2590,6 +2867,100 @@ mod tests {
})
}
#[tokio::test]
async fn oidc_discovery_accepts_extra_root_ca_for_https_provider() {
let Some((base, ca_pem, handle)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
return;
};
let config_url = format!("{base}/application/o/rustfs");
let config = build_mocked_oidc_provider_config("default", &config_url);
let origin = Url::parse(&config.config_url)
.expect("mock config_url should parse")
.origin()
.ascii_serialization();
let policy = OutboundPolicy::from_allowed_origins(&origin).expect("loopback TLS origin should be allowed");
let extra_root_certs =
parse_oidc_extra_root_certs("test OIDC TLS CA", ca_pem.as_bytes()).expect("test CA bundle should parse");
let http_client = ReqwestHttpClient::with_policy_and_extra_root_certs(policy, extra_root_certs);
let state = OidcSys::discover_provider(&config, &http_client)
.await
.expect("OIDC discovery should trust the extra root CA");
assert_eq!(state.metadata.issuer().to_string(), format!("{base}/application/o/rustfs"));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn oidc_discovery_refreshes_extra_root_ca_when_generation_changes() {
let Some((base_a, ca_pem_a, handle_a)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs-a"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
return;
};
let Some((base_b, ca_pem_b, handle_b)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs-b"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
assert!(handle_a.join().is_ok());
return;
};
let origin_a = Url::parse(&base_a)
.expect("mock base A should parse")
.origin()
.ascii_serialization();
let origin_b = Url::parse(&base_b)
.expect("mock base B should parse")
.origin()
.ascii_serialization();
let allowed_origins = format!("{origin_a},{origin_b}");
let policy = OutboundPolicy::from_allowed_origins(&allowed_origins).expect("loopback TLS origins should be allowed");
let material = Arc::new(Mutex::new(OidcExtraRootCaMaterial {
generation: 1,
root_ca_pem: Some(ca_pem_a.into_bytes()),
}));
let provider = OidcExtraRootCaProvider::new({
let material = material.clone();
move || {
let material = material.clone();
async move {
material
.lock()
.map(|material| material.clone())
.map_err(|e| format!("test OIDC extra CA material lock poisoned: {e}"))
}
}
});
let http_client = ReqwestHttpClient::with_policy_and_extra_root_ca_provider(policy, provider);
let config_a = build_mocked_oidc_provider_config("a", &format!("{base_a}/application/o/rustfs-a"));
let state_a = OidcSys::discover_provider(&config_a, &http_client)
.await
.expect("OIDC discovery should trust initial extra root CA");
assert_eq!(state_a.metadata.issuer().to_string(), format!("{base_a}/application/o/rustfs-a"));
{
let mut material = material
.lock()
.expect("test OIDC extra CA material lock should not be poisoned");
material.generation = 2;
material.root_ca_pem = Some(ca_pem_b.into_bytes());
}
let config_b = build_mocked_oidc_provider_config("b", &format!("{base_b}/application/o/rustfs-b"));
let state_b = OidcSys::discover_provider(&config_b, &http_client)
.await
.expect("OIDC discovery should refresh extra root CA after generation change");
assert_eq!(state_b.metadata.issuer().to_string(), format!("{base_b}/application/o/rustfs-b"));
assert!(handle_a.join().is_ok());
assert!(handle_b.join().is_ok());
}
#[tokio::test]
async fn test_validate_oidc_provider_config_retries_with_issuer_candidates() {
// Discovery document must advertise the canonical issuer path. The first candidate has no
@@ -2945,7 +3316,7 @@ mod tests {
// Cloud metadata endpoint is never allowed.
assert!(
matches!(
build_oidc_http_client("http://169.254.169.254/latest/meta-data/", None, None),
build_oidc_http_client("http://169.254.169.254/latest/meta-data/", None, &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"metadata endpoint must be rejected"
@@ -2953,7 +3324,7 @@ mod tests {
// Loopback is rejected by default (no allow-origins configured).
assert!(
matches!(
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", None, None),
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", None, &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"loopback must be rejected by default"
@@ -2961,7 +3332,7 @@ mod tests {
// A public hostname passes the up-front shape/host check; the resolved IP is still
// re-classified at connection time by the pinned resolver.
assert!(
build_oidc_http_client("https://accounts.example.com/.well-known/openid-configuration", None, None).is_ok(),
build_oidc_http_client("https://accounts.example.com/.well-known/openid-configuration", None, &[], None).is_ok(),
"public https endpoint should build"
);
}
@@ -2981,13 +3352,13 @@ mod tests {
fn build_oidc_http_client_honors_explicit_allowlist_for_loopback() {
let policy = OutboundPolicy::from_allowed_origins("http://127.0.0.1:8080").expect("origin should parse");
assert!(
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", Some(&policy), None).is_ok(),
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", Some(&policy), &[], None).is_ok(),
"explicitly allow-listed loopback origin should build"
);
// A metadata endpoint stays forbidden even when a loopback origin is allow-listed.
assert!(
matches!(
build_oidc_http_client("http://169.254.169.254/", Some(&policy), None),
build_oidc_http_client("http://169.254.169.254/", Some(&policy), &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"metadata endpoint stays forbidden despite an unrelated allow-list entry"
@@ -3190,8 +3561,9 @@ mod tests {
#[test]
fn oidc_metadata_endpoint_rejection_does_not_offer_allowlist_bypass() {
let error = build_oidc_http_client("http://169.254.169.254/latest/meta-data/", Some(&OutboundPolicy::default()), None)
.expect_err("metadata endpoint must remain forbidden");
let error =
build_oidc_http_client("http://169.254.169.254/latest/meta-data/", Some(&OutboundPolicy::default()), &[], None)
.expect_err("metadata endpoint must remain forbidden");
let message = error.to_string();
assert!(message.contains("metadata endpoint"));
+86 -10
View File
@@ -28,6 +28,14 @@ pub const INTERNODE_OPERATION_NS_SCANNER: &str = "ns_scanner";
pub const INTERNODE_OPERATION_GRPC_READ_ALL: &str = "grpc_read_all";
pub const INTERNODE_OPERATION_GRPC_WRITE_ALL: &str = "grpc_write_all";
pub const INTERNODE_OPERATION_GRPC_READ_MULTIPLE: &str = "grpc_read_multiple";
pub const INTERNODE_OPERATION_GRPC_READ_VERSION: &str = "grpc_read_version";
pub const INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION: &str = "grpc_batch_read_version";
pub const INTERNODE_OPERATION_GRPC_LOCK: &str = "grpc_lock";
pub const INTERNODE_OPERATION_GRPC_UNLOCK: &str = "grpc_unlock";
pub const INTERNODE_OPERATION_GRPC_LOCK_BATCH: &str = "grpc_lock_batch";
pub const INTERNODE_OPERATION_GRPC_UNLOCK_BATCH: &str = "grpc_unlock_batch";
pub const INTERNODE_OPERATION_GRPC_REFRESH: &str = "grpc_refresh";
pub const INTERNODE_OPERATION_GRPC_FORCE_UNLOCK: &str = "grpc_force_unlock";
pub const INTERNODE_OPERATION_GRPC_OTHER: &str = "grpc_other";
pub const INTERNODE_TRANSPORT_BACKEND_TCP_HTTP: &str = "tcp-http";
pub const INTERNODE_TRANSPORT_BACKEND_GRPC: &str = "grpc";
@@ -78,6 +86,7 @@ const INTERNODE_REPLAY_SCOPE_FALLBACK_TOTAL: &str = "rustfs_system_network_inter
const INTERNODE_REPLAY_CACHE_OVERFLOW_TOTAL: &str = "rustfs_system_network_internode_replay_cache_overflow_total";
const INTERNODE_REPLAY_CACHE_OVERFLOW_BY_OPERATION_TOTAL: &str =
"rustfs_system_network_internode_replay_cache_overflow_by_operation_total";
const INTERNODE_REPLAY_CACHE_RECORDS_TOTAL: &str = "rustfs_system_network_internode_replay_cache_records_total";
const INTERNODE_REPLAY_CACHE_ENTRIES: &str = "rustfs_system_network_internode_replay_cache_entries";
const INTERNODE_REPLAY_CACHE_CAPACITY: &str = "rustfs_system_network_internode_replay_cache_capacity";
const INTERNODE_REPLAY_CACHE_EVICTIONS_TOTAL: &str = "rustfs_system_network_internode_replay_cache_evictions_total";
@@ -157,6 +166,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
name: INTERNODE_REPLAY_CACHE_OVERFLOW_BY_OPERATION_TOTAL,
labels: SERVER_OPERATION_BACKEND_RPC_PATH_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_REPLAY_CACHE_RECORDS_TOTAL,
labels: SERVER_OPERATION_BACKEND_RPC_PATH_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_REPLAY_CACHE_ENTRIES,
labels: SERVER_LABELS,
@@ -619,6 +632,22 @@ impl InternodeMetrics {
.increment(1);
}
pub fn record_replay_cache_record_for_operation_and_backend_path(
&self,
operation: &'static str,
backend: &'static str,
rpc_path: &str,
) {
counter!(
INTERNODE_REPLAY_CACHE_RECORDS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
RPC_PATH_LABEL => rpc_path.to_owned()
)
.increment(1);
}
pub fn record_replay_cache_state(&self, entries: usize, capacity: usize) {
let entries = usize_to_u64_saturating(entries);
let capacity = usize_to_u64_saturating(capacity);
@@ -964,7 +993,7 @@ mod tests {
#[test]
fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 20);
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 21);
for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
}
@@ -986,14 +1015,18 @@ mod tests {
INTERNODE_OPERATION_METRICS[13].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL]
);
for metric in &INTERNODE_OPERATION_METRICS[14..16] {
assert_eq!(
INTERNODE_OPERATION_METRICS[14].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL]
);
for metric in &INTERNODE_OPERATION_METRICS[15..17] {
assert_eq!(metric.labels, &[SERVER_LABEL]);
}
assert_eq!(INTERNODE_OPERATION_METRICS[16].labels, &[SERVER_LABEL, REASON_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[17].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[17].labels, &[SERVER_LABEL, REASON_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[18].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]);
// Payload histogram + large-payload counter carry operation+backend labels.
assert_eq!(INTERNODE_OPERATION_METRICS[18].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[19].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[20].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
}
#[test]
@@ -1004,6 +1037,14 @@ mod tests {
assert_eq!(INTERNODE_OPERATION_WALK_DIR, "walk_dir");
assert_eq!(INTERNODE_OPERATION_GRPC_READ_ALL, "grpc_read_all");
assert_eq!(INTERNODE_OPERATION_GRPC_WRITE_ALL, "grpc_write_all");
assert_eq!(INTERNODE_OPERATION_GRPC_READ_VERSION, "grpc_read_version");
assert_eq!(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, "grpc_batch_read_version");
assert_eq!(INTERNODE_OPERATION_GRPC_LOCK, "grpc_lock");
assert_eq!(INTERNODE_OPERATION_GRPC_UNLOCK, "grpc_unlock");
assert_eq!(INTERNODE_OPERATION_GRPC_LOCK_BATCH, "grpc_lock_batch");
assert_eq!(INTERNODE_OPERATION_GRPC_UNLOCK_BATCH, "grpc_unlock_batch");
assert_eq!(INTERNODE_OPERATION_GRPC_REFRESH, "grpc_refresh");
assert_eq!(INTERNODE_OPERATION_GRPC_FORCE_UNLOCK, "grpc_force_unlock");
assert_eq!(INTERNODE_OPERATION_GRPC_OTHER, "grpc_other");
assert_eq!(INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, "tcp-http");
@@ -1048,26 +1089,30 @@ mod tests {
);
assert_eq!(
INTERNODE_OPERATION_METRICS[14].name,
"rustfs_system_network_internode_replay_cache_entries"
"rustfs_system_network_internode_replay_cache_records_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[15].name,
"rustfs_system_network_internode_replay_cache_capacity"
"rustfs_system_network_internode_replay_cache_entries"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[16].name,
"rustfs_system_network_internode_replay_cache_evictions_total"
"rustfs_system_network_internode_replay_cache_capacity"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[17].name,
"rustfs_system_storage_erasure_write_quorum_failures_total"
"rustfs_system_network_internode_replay_cache_evictions_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[18].name,
"rustfs_system_network_internode_operation_payload_bytes"
"rustfs_system_storage_erasure_write_quorum_failures_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[19].name,
"rustfs_system_network_internode_operation_payload_bytes"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[20].name,
"rustfs_system_network_internode_operation_large_payloads_total"
);
assert_eq!(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, "grpc_read_multiple");
@@ -1091,6 +1136,10 @@ mod tests {
INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL,
"rustfs_system_network_internode_signature_v1_fallback_total"
);
assert_eq!(
INTERNODE_REPLAY_CACHE_RECORDS_TOTAL,
"rustfs_system_network_internode_replay_cache_records_total"
);
assert_eq!(FAILURE_REASON_LABEL, "failure_reason");
assert_eq!(RPC_PATH_LABEL, "rpc_path");
assert_eq!(REASON_LABEL, "reason");
@@ -1144,6 +1193,11 @@ mod tests {
INTERNODE_TRANSPORT_BACKEND_GRPC,
"/node_service.NodeService/ReadAll",
);
metrics.record_replay_cache_record_for_operation_and_backend_path(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
"/node_service.NodeService/ReadVersion",
);
});
let snapshot = metrics.snapshot();
@@ -1179,6 +1233,28 @@ mod tests {
assert_eq!(labels.get(BACKEND_LABEL).map(String::as_str), Some(INTERNODE_TRANSPORT_BACKEND_GRPC));
assert_eq!(labels.get(RPC_PATH_LABEL).map(String::as_str), Some("/node_service.NodeService/ReadAll"));
assert!(labels.get(SERVER_LABEL).is_some_and(|value| !value.is_empty()));
let records: Vec<_> = entries
.iter()
.filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_REPLAY_CACHE_RECORDS_TOTAL)
.collect();
assert_eq!(records.len(), 1);
let labels: HashMap<_, _> = records[0]
.0
.key()
.labels()
.map(|label| (label.key().to_string(), label.value().to_string()))
.collect();
assert_eq!(
labels.get(OPERATION_LABEL).map(String::as_str),
Some(INTERNODE_OPERATION_GRPC_READ_VERSION)
);
assert_eq!(labels.get(BACKEND_LABEL).map(String::as_str), Some(INTERNODE_TRANSPORT_BACKEND_GRPC));
assert_eq!(
labels.get(RPC_PATH_LABEL).map(String::as_str),
Some("/node_service.NodeService/ReadVersion")
);
assert!(labels.get(SERVER_LABEL).is_some_and(|value| !value.is_empty()));
}
#[test]
+32
View File
@@ -635,6 +635,38 @@ mod tests {
assert!(!should_use_existing_delete_replication_info(false, false));
}
/// P1-20 truth-table pin (rustfs/backlog#1675): without any reset in play
/// (no per-target reset header on the object, empty reset id on the
/// target) the existing-object resync decision compensates exactly the
/// never-replicated objects — Empty replicates, any recorded status does
/// not.
#[test]
fn resync_target_without_reset_replicates_only_empty_status() {
let user_defined = HashMap::new();
let object = ReplicationResyncTargetObject {
mod_time: Some(OffsetDateTime::UNIX_EPOCH + Duration::seconds(10)),
user_defined: &user_defined,
};
for (status, expected) in [
(ReplicationStatusType::Empty, true),
(ReplicationStatusType::Completed, false),
// "COMPLETE" on disk parses to this legacy variant, so objects
// written by older versions reach the decision through it.
(ReplicationStatusType::CompletedLegacy, false),
(ReplicationStatusType::Pending, false),
(ReplicationStatusType::Failed, false),
(ReplicationStatusType::Replica, false),
] {
let label = format!("{status:?}");
let decision = resync_target_for_object(&object, "arn:target", "", None, status);
assert_eq!(
decision.replicate, expected,
"existing-object resync without a reset must replicate only never-replicated objects (status {label})"
);
}
}
#[test]
fn resync_target_includes_object_at_reset_before_boundary() {
let reset_before = OffsetDateTime::UNIX_EPOCH + Duration::seconds(30);
+42
View File
@@ -360,6 +360,48 @@ mod tests {
);
}
/// P1-20 truth-table pin (rustfs/backlog#1675): when no target replicates
/// — the decision is empty because ExistingObjectReplication is Disabled
/// for a never-replicated object, or because the object is an inbound
/// REPLICA (must_replicate returns an empty decision for those) — the
/// heal pass must skip entirely, whatever the recorded status says. The
/// scanner never compensates these objects.
#[test]
fn heal_queue_action_skips_when_no_target_replicates() {
for status in [
ReplicationStatusType::Empty,
ReplicationStatusType::Failed,
ReplicationStatusType::Replica,
] {
let mut roi = ReplicateObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
replication_status: status,
dsc: ReplicateDecision::new(),
..Default::default()
};
let action = replication_heal_queue_action(&mut roi);
assert!(
matches!(action, ReplicationHealQueueAction::Skip),
"an empty replicate decision must skip heal queueing (status {:?})",
roi.replication_status
);
}
}
/// P1-20 truth-table pin: a Completed object with no resync decision has
/// nothing left to heal — the scanner must not requeue it.
#[test]
fn heal_queue_action_skips_completed_object_without_resync() {
let mut roi = replicate_object_info(ReplicationStatusType::Completed);
let action = replication_heal_queue_action(&mut roi);
assert!(matches!(action, ReplicationHealQueueAction::Skip));
}
#[test]
fn heal_queue_action_routes_failed_objects_to_heal_queue() {
let mut roi = replicate_object_info(ReplicationStatusType::Failed);
+308 -15
View File
@@ -14,7 +14,12 @@
#![recursion_limit = "256"]
use datafusion::{common::DataFusionError, sql::sqlparser::parser::ParserError};
use datafusion::{
arrow::error::ArrowError,
common::{DataFusionError, SchemaError},
parquet::errors::ParquetError,
sql::sqlparser::parser::ParserError,
};
use std::{error::Error as StdError, fmt::Display};
use thiserror::Error;
@@ -67,23 +72,88 @@ pub enum QueryError {
StoreError { e: String },
}
#[derive(Debug, Error)]
#[non_exhaustive]
pub enum S3SelectPolicyError {
#[derive(Clone, Debug, Error, PartialEq, Eq)]
pub enum SelectError {
#[error("The file is not in a supported compression format. Only GZIP and BZIP2 are supported.")]
InvalidCompressionFormat,
#[error("The data source type is not valid. Only CSV, JSON, and Parquet are supported.")]
InvalidDataSource,
#[error(
"Object decompression failed. Check that the object is properly compressed using the format specified in the request."
)]
TruncatedInput,
#[error("An error occurred while parsing the CSV file. Check the file and try again.")]
CsvParsingError,
#[error("An error occurred while parsing the JSON file. Check the file and try again.")]
JsonParsingError,
#[error("An error occurred while parsing the Parquet file. Check the file and try again.")]
ParquetParsingError,
#[error("{message}")]
ParseSelectFailure { message: String },
#[error("The SQL expression is invalid.")]
InvalidQuery,
#[error("The SQL expression contains a data type that is not valid.")]
InvalidDataType,
#[error("An incorrect argument type was specified in a function call in the SQL expression.")]
IncorrectSqlFunctionArgumentType,
#[error("The data source path in the SQL expression is not supported.")]
DataSourcePathUnsupported,
#[error("Unsupported S3 Select SQL structure: {message}")]
UnsupportedSqlStructure { message: String },
#[error("We encountered an unsupported SQL operation.")]
UnsupportedSqlOperation,
#[error("A column name or a path provided does not exist in the SQL expression.")]
EvaluatorBindingDoesNotExist,
#[error("The field name matches to multiple fields in the file. Check the SQL expression and the file, and try again.")]
AmbiguousFieldName,
#[error("The value of a parameter in ScanRange element is invalid. Check the service API documentation and try again.")]
InvalidScanRange,
#[error("S3 Select query concurrency limit reached")]
QueryConcurrencyLimit,
#[error("S3 Select query exceeded the {seconds}-second execution limit")]
QueryTimeout { seconds: u64 },
#[error("S3 Select query resource limit exceeded")]
ResourceExhausted,
#[error("The specified bucket does not exist.")]
BucketNotFound,
#[error("The specified key does not exist.")]
ObjectNotFound,
#[error("The query was canceled")]
Canceled,
#[error("An internal error occurred.")]
InternalError,
}
pub type S3SelectPolicyError = SelectError;
const MAX_ERROR_SOURCE_DEPTH: usize = 16;
impl QueryError {
fn source_error<T: StdError + 'static>(&self) -> Option<&T> {
let mut err: &(dyn StdError + 'static) = self;
for _ in 0..16 {
for _ in 0..MAX_ERROR_SOURCE_DEPTH {
if let Some(source) = err.downcast_ref::<T>() {
return Some(source);
}
@@ -99,10 +169,113 @@ impl QueryError {
pub fn s3_select_policy_error(&self) -> Option<&S3SelectPolicyError> {
self.source_error()
}
pub fn select_error(&self) -> SelectError {
let mut err: &(dyn StdError + 'static) = match self {
Self::Datafusion { source } => source.as_ref(),
_ => self,
};
for _ in 0..MAX_ERROR_SOURCE_DEPTH {
if let Some(select_error) = classify_select_error_source(err) {
return select_error;
}
let Some(source) = err.source() else {
break;
};
err = source;
}
match self {
QueryError::NotImplemented { .. } => SelectError::UnsupportedSqlOperation,
QueryError::MultiStatement { .. } => SelectError::UnsupportedSqlStructure {
message: "multiple SQL statements are not supported".to_string(),
},
QueryError::BuildQueryDispatcher { .. } | QueryError::FunctionExists { .. } | QueryError::StoreError { .. } => {
SelectError::InternalError
}
QueryError::Cancel => SelectError::Canceled,
QueryError::FunctionNotExists { .. } => SelectError::InvalidQuery,
QueryError::Datafusion { .. } | QueryError::Parser { .. } => SelectError::InternalError,
}
}
}
impl From<S3SelectPolicyError> for QueryError {
fn from(value: S3SelectPolicyError) -> Self {
fn classify_select_error_source(err: &(dyn StdError + 'static)) -> Option<SelectError> {
if let Some(error) = err.downcast_ref::<SelectError>() {
return Some(error.clone());
}
if let Some(error) = err.downcast_ref::<object_store::SelectObjectStoreError>() {
return Some(error.select_error());
}
if let Some(error) = err.downcast_ref::<datafusion::object_store::Error>() {
return match error {
datafusion::object_store::Error::NotFound { source, .. } => Some(
source
.downcast_ref::<object_store::SelectObjectStoreError>()
.map_or(SelectError::ObjectNotFound, object_store::SelectObjectStoreError::select_error),
),
_ => None,
};
}
if let Some(error) = err.downcast_ref::<ParserError>() {
return Some(SelectError::ParseSelectFailure {
message: error.to_string(),
});
}
if let Some(error) = err.downcast_ref::<ArrowError>() {
return match error {
ArrowError::CsvError(_) => Some(SelectError::CsvParsingError),
ArrowError::JsonError(_) => Some(SelectError::JsonParsingError),
ArrowError::ParquetError(_) => Some(SelectError::ParquetParsingError),
ArrowError::CastError(_) | ArrowError::ParseError(_) => Some(SelectError::InvalidDataType),
ArrowError::MemoryError(_) => Some(SelectError::ResourceExhausted),
ArrowError::ExternalError(_) | ArrowError::IoError(_, _) => None,
_ => Some(SelectError::InternalError),
};
}
if let Some(error) = err.downcast_ref::<ParquetError>() {
return match error {
ParquetError::External(_) => None,
_ => Some(SelectError::ParquetParsingError),
};
}
if let Some(error) = err.downcast_ref::<SchemaError>() {
return Some(match error {
SchemaError::FieldNotFound { .. } => SelectError::EvaluatorBindingDoesNotExist,
SchemaError::AmbiguousReference { .. }
| SchemaError::DuplicateQualifiedField { .. }
| SchemaError::DuplicateUnqualifiedField { .. } => SelectError::AmbiguousFieldName,
});
}
if let Some(error) = err.downcast_ref::<DataFusionError>() {
return match error {
DataFusionError::NotImplemented(_) => Some(SelectError::UnsupportedSqlOperation),
DataFusionError::Plan(_) => Some(SelectError::InvalidQuery),
DataFusionError::ResourcesExhausted(_) => Some(SelectError::ResourceExhausted),
DataFusionError::Internal(_)
| DataFusionError::Execution(_)
| DataFusionError::Configuration(_)
| DataFusionError::Substrait(_)
| DataFusionError::Ffi(_) => Some(SelectError::InternalError),
DataFusionError::ArrowError(_, _)
| DataFusionError::ParquetError(_)
| DataFusionError::ObjectStore(_)
| DataFusionError::IoError(_)
| DataFusionError::SQL(_, _)
| DataFusionError::SchemaError(_, _)
| DataFusionError::ExecutionJoin(_)
| DataFusionError::External(_)
| DataFusionError::Context(_, _)
| DataFusionError::Diagnostic(_, _)
| DataFusionError::Collection(_)
| DataFusionError::Shared(_) => None,
};
}
None
}
impl From<SelectError> for QueryError {
fn from(value: SelectError) -> Self {
Self::Datafusion {
source: Box::new(DataFusionError::External(Box::new(value))),
}
@@ -161,7 +334,7 @@ mod tests {
};
assert_eq!(err.to_string(), "Multi-statement not allow, found num:2, sql:SELECT 1; SELECT 2;");
let err = S3SelectPolicyError::UnsupportedSqlStructure {
let err = SelectError::UnsupportedSqlStructure {
message: "JOIN is not supported".to_string(),
};
assert_eq!(err.to_string(), "Unsupported S3 Select SQL structure: JOIN is not supported");
@@ -170,11 +343,11 @@ mod tests {
assert_eq!(err.to_string(), "The query has been canceled");
assert_eq!(
S3SelectPolicyError::QueryConcurrencyLimit.to_string(),
SelectError::QueryConcurrencyLimit.to_string(),
"S3 Select query concurrency limit reached"
);
assert_eq!(
S3SelectPolicyError::QueryTimeout { seconds: 300 }.to_string(),
SelectError::QueryTimeout { seconds: 300 }.to_string(),
"S3 Select query exceeded the 300-second execution limit"
);
@@ -223,12 +396,132 @@ mod tests {
#[test]
fn policy_error_is_recoverable_from_query_error() {
let err: QueryError = S3SelectPolicyError::QueryTimeout { seconds: 300 }.into();
let err: QueryError = SelectError::QueryTimeout { seconds: 300 }.into();
assert!(matches!(
err.s3_select_policy_error(),
Some(S3SelectPolicyError::QueryTimeout { seconds: 300 })
));
assert!(matches!(err.s3_select_policy_error(), Some(SelectError::QueryTimeout { seconds: 300 })));
}
#[test]
fn query_error_classifies_data_errors_without_display_matching() {
let cases = [
(
DataFusionError::ArrowError(Box::new(ArrowError::CsvError("private csv detail".to_string())), None),
SelectError::CsvParsingError,
),
(
DataFusionError::ArrowError(Box::new(ArrowError::JsonError("private json detail".to_string())), None),
SelectError::JsonParsingError,
),
(
DataFusionError::ParquetError(Box::new(ParquetError::General("private parquet detail".to_string()))),
SelectError::ParquetParsingError,
),
(
DataFusionError::External(Box::new(SelectError::TruncatedInput)),
SelectError::TruncatedInput,
),
(
DataFusionError::ArrowError(
Box::new(ArrowError::InvalidArgumentError("private implementation detail".to_string())),
None,
),
SelectError::InternalError,
),
(
DataFusionError::ArrowError(Box::new(ArrowError::CastError("invalid cast".to_string())), None),
SelectError::InvalidDataType,
),
(
DataFusionError::ArrowError(Box::new(ArrowError::MemoryError("query memory limit".to_string())), None),
SelectError::ResourceExhausted,
),
(
DataFusionError::Execution("private execution detail".to_string()),
SelectError::InternalError,
),
(DataFusionError::Plan("invalid expression".to_string()), SelectError::InvalidQuery),
(
DataFusionError::NotImplemented("unsupported expression".to_string()),
SelectError::UnsupportedSqlOperation,
),
(
DataFusionError::SchemaError(
Box::new(SchemaError::FieldNotFound {
field: Box::new(datafusion::common::Column::from_name("missing")),
valid_fields: Vec::new(),
}),
Box::new(None),
),
SelectError::EvaluatorBindingDoesNotExist,
),
(
DataFusionError::SchemaError(
Box::new(SchemaError::AmbiguousReference {
field: Box::new(datafusion::common::Column::from_name("duplicate")),
}),
Box::new(None),
),
SelectError::AmbiguousFieldName,
),
];
for (source, expected) in cases {
let error = QueryError::from(source);
assert_eq!(error.select_error(), expected, "wrong classification for {error:?}");
}
}
#[test]
fn query_error_preserves_typed_object_store_classification() {
let bucket_error = QueryError::from(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::NotFound {
path: "private-bucket/private-object".to_string(),
source: Box::new(object_store::SelectObjectStoreError::BucketNotFound {
source: SelectStorageError::BucketNotFound("private-bucket".to_string()),
}),
})));
let object_error = QueryError::from(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::NotFound {
path: "private-bucket/private-object".to_string(),
source: Box::new(object_store::SelectObjectStoreError::ObjectNotFound {
source: SelectStorageError::ObjectNotFound("private-bucket".to_string(), "private-object".to_string()),
}),
})));
let scan_range_error =
QueryError::from(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::Generic {
store: "test",
source: Box::new(object_store::SelectObjectStoreError::InvalidScanRange),
})));
let storage_error = QueryError::from(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::Generic {
store: "test",
source: Box::new(object_store::SelectObjectStoreError::Storage {
source: SelectStorageError::LessData,
}),
})));
assert_eq!(bucket_error.select_error(), SelectError::BucketNotFound);
assert_eq!(object_error.select_error(), SelectError::ObjectNotFound);
assert_eq!(scan_range_error.select_error(), SelectError::InvalidScanRange);
assert_eq!(storage_error.select_error(), SelectError::InternalError);
}
#[test]
fn select_error_source_traversal_stops_at_the_depth_bound() {
#[derive(Debug)]
struct CyclicError;
impl std::fmt::Display for CyclicError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("cyclic error")
}
}
impl StdError for CyclicError {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
Some(self)
}
}
let error = QueryError::from(DataFusionError::External(Box::new(CyclicError)));
assert_eq!(error.select_error(), SelectError::InternalError);
}
#[test]
+115 -10
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::{
PrepareSelectObjectSnapshotError, SELECT_DEFAULT_READ_BUFFER_SIZE, SelectGetObjectReader, SelectObjectOptions,
PrepareSelectObjectSnapshotError, SELECT_DEFAULT_READ_BUFFER_SIZE, SelectError, SelectGetObjectReader, SelectObjectOptions,
SelectObjectSnapshot, SelectObjectSnapshotReadError, SelectStorageError, SelectStore, SnapshotConsistencyError,
query::{
parser::RustFsDialect,
@@ -115,6 +115,38 @@ pub(crate) enum EcObjectStoreBuildError {
Snapshot(#[source] SnapshotConsistencyError),
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum SelectObjectStoreError {
#[error("SelectObjectContent bucket does not exist")]
BucketNotFound {
#[source]
source: SelectStorageError,
},
#[error("SelectObjectContent object does not exist")]
ObjectNotFound {
#[source]
source: SelectStorageError,
},
#[error("SelectObjectContent storage failure")]
Storage {
#[source]
source: SelectStorageError,
},
#[error("SelectObjectContent ScanRange is invalid")]
InvalidScanRange,
}
impl SelectObjectStoreError {
pub(crate) fn select_error(&self) -> SelectError {
match self {
Self::BucketNotFound { .. } => SelectError::BucketNotFound,
Self::ObjectNotFound { .. } => SelectError::ObjectNotFound,
Self::InvalidScanRange => SelectError::InvalidScanRange,
Self::Storage { .. } => SelectError::InternalError,
}
}
}
#[derive(Clone, Copy, Debug)]
pub struct SelectScanRange {
start: u64,
@@ -502,8 +534,7 @@ fn map_prepare_snapshot_error(bucket: &str, object: &str, err: PrepareSelectObje
}
fn map_build_error_to_s3(error: EcObjectStoreBuildError) -> S3Error {
let message = error.to_string();
let mut s3_error = S3Error::with_message(S3ErrorCode::InternalError, message);
let mut s3_error = S3Error::with_message(S3ErrorCode::InternalError, SelectError::InternalError.to_string());
s3_error.set_source(Box::new(error));
s3_error
}
@@ -519,15 +550,21 @@ fn snapshot_read_error(bucket: &str, object: &str, err: SelectObjectSnapshotRead
}
fn map_storage_error(bucket: &str, object: &str, err: SelectStorageError) -> o_Error {
if select_is_err_bucket_not_found(&err) || select_is_err_object_not_found(&err) || select_is_err_version_not_found(&err) {
if select_is_err_bucket_not_found(&err) {
return o_Error::NotFound {
path: format!("{bucket}/{object}"),
source: Box::new(err),
source: Box::new(SelectObjectStoreError::BucketNotFound { source: err }),
};
}
if select_is_err_object_not_found(&err) || select_is_err_version_not_found(&err) {
return o_Error::NotFound {
path: format!("{bucket}/{object}"),
source: Box::new(SelectObjectStoreError::ObjectNotFound { source: err }),
};
}
o_Error::Generic {
store: "EcObjectStore",
source: Box::new(err),
source: Box::new(SelectObjectStoreError::Storage { source: err }),
}
}
@@ -602,7 +639,7 @@ fn parse_scan_range_from_bounds(
fn invalid_scan_range_store_error() -> o_Error {
o_Error::Generic {
store: "EcObjectStore",
source: format!("ScanRange: {INVALID_SCAN_RANGE_MESSAGE}").into(),
source: Box::new(SelectObjectStoreError::InvalidScanRange),
}
}
@@ -1150,7 +1187,11 @@ where
})?
.map_err(|e| o_Error::Generic {
store: "EcObjectStore",
source: Box::new(e),
source: if e.kind() == std::io::ErrorKind::InvalidData {
Box::new(SelectError::JsonParsingError)
} else {
Box::new(e)
},
})?;
// ── 3. Yield phase (one Bytes per NDJSON line) ───────────────────
@@ -1341,12 +1382,13 @@ mod test {
SELECT_DEFAULT_READ_BUFFER_SIZE, SelectObjectOptions, SelectObjectSnapshot, SelectScanRange, SnapshotConsistencyError,
bytes_stream, convert_csv_delimiter_stream, convert_field_delimiter_stream, convert_record_delimiter_stream,
extract_json_sub_path_from_expression, find_delimiter, flatten_json_document_to_ndjson, http_range_spec_from_get_range,
json_document_ndjson_stream, json_document_ndjson_stream_with_parser, scan_range_from_bounds, scan_range_stream,
select_read_headers, snapshot_last_modified, validate_json_document_size,
json_document_ndjson_stream, json_document_ndjson_stream_with_parser, map_storage_error, scan_range_from_bounds,
scan_range_stream, select_read_headers, snapshot_last_modified, validate_json_document_size,
};
use crate::query::session::{QueryExecutionGuard, QueryExecutionOwner, QueryExecutionTracker};
use crate::storage_api::SelectPutObjReader;
use crate::storage_api::object_store::ObjectIO as _;
use crate::{QueryError, SelectError, SelectStorageError};
use bytes::Bytes;
use datafusion::{
common::DataFusionError,
@@ -2688,6 +2730,69 @@ mod test {
assert!(output.next().await.is_none());
}
#[tokio::test]
async fn malformed_json_document_stream_has_typed_select_error() {
let input = b"{bad".to_vec();
let memory_pool: Arc<dyn MemoryPool> =
Arc::new(GreedyMemoryPool::new(input.len() * JSON_DOCUMENT_MEMORY_RESERVATION_MULTIPLIER));
let mut output = json_document_ndjson_stream(
Box::new(std::io::Cursor::new(input.clone())),
input.len() as u64,
None,
memory_pool,
None,
);
let source = output
.next()
.await
.expect("malformed JSON should produce one stream error")
.expect_err("malformed JSON DOCUMENT must fail");
let error = QueryError::from(DataFusionError::ObjectStore(Box::new(source)));
assert_eq!(error.select_error(), SelectError::JsonParsingError);
assert!(output.next().await.is_none());
}
#[test]
fn storage_error_mapper_preserves_protocol_classification() {
let classify = |source| QueryError::from(DataFusionError::ObjectStore(Box::new(source))).select_error();
assert_eq!(
classify(map_storage_error(
"private-bucket",
"private-object",
SelectStorageError::BucketNotFound("private-bucket".to_string()),
)),
SelectError::BucketNotFound
);
assert_eq!(
classify(map_storage_error(
"private-bucket",
"private-object",
SelectStorageError::ObjectNotFound("private-bucket".to_string(), "private-object".to_string()),
)),
SelectError::ObjectNotFound
);
assert_eq!(
classify(map_storage_error("private-bucket", "private-object", SelectStorageError::LessData)),
SelectError::InternalError
);
assert_eq!(
classify(scan_range_from_bounds(Some(10), None, 10).expect_err("out-of-bounds range must fail")),
SelectError::InvalidScanRange
);
let parquet_source = map_storage_error(
"private-bucket",
"private-object",
SelectStorageError::ObjectNotFound("private-bucket".to_string(), "private-object".to_string()),
);
let parquet_error = QueryError::from(DataFusionError::ParquetError(Box::new(
datafusion::parquet::errors::ParquetError::External(Box::new(parquet_source)),
)));
assert_eq!(parquet_error.select_error(), SelectError::ObjectNotFound);
}
#[test]
fn test_json_document_size_error_is_resource_exhausted() {
assert!(validate_json_document_size(super::MAX_JSON_DOCUMENT_BYTES).is_ok());
+8 -12
View File
@@ -433,7 +433,7 @@ impl SessionCtxFactory {
let path = Path::from(context.input.key.clone());
store.put(&path, data_bytes.into()).await.map_err(|e| {
error!("put data into memory failed: {}", e.to_string());
QueryError::StoreError { e: e.to_string() }
QueryError::from(DataFusionError::from(e))
})?;
df_session_state.with_object_store(&store_url, store).build()
@@ -477,16 +477,11 @@ fn test_parquet_bytes() -> QueryResult<Vec<u8>> {
let mut bytes = Vec::new();
{
let mut writer =
ArrowWriter::try_new(&mut bytes, schema, None).map_err(|e| QueryError::StoreError { e: e.to_string() })?;
writer
.write(&first_batch)
.map_err(|e| QueryError::StoreError { e: e.to_string() })?;
writer.flush().map_err(|e| QueryError::StoreError { e: e.to_string() })?;
writer
.write(&second_batch)
.map_err(|e| QueryError::StoreError { e: e.to_string() })?;
writer.close().map_err(|e| QueryError::StoreError { e: e.to_string() })?;
let mut writer = ArrowWriter::try_new(&mut bytes, schema, None).map_err(DataFusionError::from)?;
writer.write(&first_batch).map_err(DataFusionError::from)?;
writer.flush().map_err(DataFusionError::from)?;
writer.write(&second_batch).map_err(DataFusionError::from)?;
writer.close().map_err(DataFusionError::from)?;
}
Ok(bytes)
}
@@ -509,7 +504,8 @@ fn test_parquet_batch(
Arc::new(Int32Array::from(salaries.to_vec())),
],
)
.map_err(|e| QueryError::StoreError { e: e.to_string() })
.map_err(DataFusionError::from)
.map_err(QueryError::from)
}
#[cfg(test)]
+64 -29
View File
@@ -38,7 +38,7 @@ use datafusion::{
use futures::Stream;
use parking_lot::Mutex;
use rustfs_s3select_api::{
QueryError, QueryResult, S3SelectPolicyError,
QueryError, QueryResult, SelectError,
query::{
Query,
ast::ExtStatement,
@@ -128,7 +128,7 @@ impl QueryDispatcher for SimpleQueryDispatcher {
.query_admission
.clone()
.try_acquire_owned()
.map_err(|_| QueryError::from(S3SelectPolicyError::QueryConcurrencyLimit))?;
.map_err(|_| QueryError::from(SelectError::QueryConcurrencyLimit))?;
Ok(QueryAdmission::new(Arc::new(permit)))
}
@@ -245,7 +245,7 @@ impl SimpleQueryDispatcher {
.query_admission
.clone()
.try_acquire_owned()
.map_err(|_| QueryError::from(S3SelectPolicyError::QueryConcurrencyLimit))?;
.map_err(|_| QueryError::from(SelectError::QueryConcurrencyLimit))?;
Arc::new(permit)
}
};
@@ -293,7 +293,7 @@ impl SimpleQueryDispatcher {
) -> QueryResult<T> {
let deadline = query_tracker.deadline();
let timeout_error = || {
S3SelectPolicyError::QueryTimeout {
SelectError::QueryTimeout {
seconds: query_tracker.timeout_seconds(),
}
.into()
@@ -343,11 +343,11 @@ impl SimpleQueryDispatcher {
return QueryError::Cancel;
}
match query_tracker.status() {
QueryExecutionStatus::TimedOut => S3SelectPolicyError::QueryTimeout {
QueryExecutionStatus::TimedOut => SelectError::QueryTimeout {
seconds: query_tracker.timeout_seconds(),
}
.into(),
QueryExecutionStatus::Active if Instant::now() >= query_tracker.deadline() => S3SelectPolicyError::QueryTimeout {
QueryExecutionStatus::Active if Instant::now() >= query_tracker.deadline() => SelectError::QueryTimeout {
seconds: query_tracker.timeout_seconds(),
}
.into(),
@@ -430,15 +430,11 @@ impl SimpleQueryDispatcher {
} else if *info == *USE {
file_format = file_format.with_has_header(true);
} else {
return Err(QueryError::NotImplemented {
err: "unsupported FileHeaderInfo".to_string(),
});
return Err(SelectError::InvalidDataSource.into());
}
}
_ => {
return Err(QueryError::NotImplemented {
err: "unsupported FileHeaderInfo".to_string(),
});
return Err(SelectError::InvalidDataSource.into());
}
}
if let Some(quote) = csv.quote_character.as_ref() {
@@ -462,9 +458,7 @@ impl SimpleQueryDispatcher {
.unwrap_or_else(|| ".json".to_string());
(ListingOptions::new(Arc::new(file_format)).with_file_extension(file_ext), false, false)
} else {
return Err(QueryError::NotImplemented {
err: "not support this file type".to_string(),
});
return Err(SelectError::InvalidDataSource.into());
};
let resolve_schema = listing_options.infer_schema(session.inner(), &table_path).await?;
@@ -642,7 +636,7 @@ impl Stream for TrackedRecordBatchStream {
}
fn query_timeout_error(timeout_seconds: u64) -> datafusion::common::DataFusionError {
datafusion::common::DataFusionError::External(Box::new(S3SelectPolicyError::QueryTimeout {
datafusion::common::DataFusionError::External(Box::new(SelectError::QueryTimeout {
seconds: timeout_seconds,
}))
}
@@ -791,7 +785,7 @@ mod tests {
};
use futures::{StreamExt, TryStreamExt, stream};
use rustfs_s3select_api::{
QueryError, QueryResult, S3SelectPolicyError,
QueryError, QueryResult, SelectError,
query::{
Context as QueryContext, Query,
dispatcher::QueryDispatcher,
@@ -1338,6 +1332,47 @@ mod tests {
assert_eq!(dispatcher.memory_limit_bytes, DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES);
}
#[tokio::test]
async fn invalid_csv_header_info_is_typed_invalid_data_source() {
let mut input = test_input();
input
.request
.input_serialization
.csv
.as_mut()
.expect("test input should use CSV")
.file_header_info = Some(FileHeaderInfo::from_static("INVALID"));
let input = Arc::new(input);
let optimizer = Arc::new(CascadeOptimizerBuilder::default().build());
let scheduler = Arc::new(LocalScheduler {});
let dispatcher = SimpleQueryDispatcherBuilder::default()
.with_input(Arc::clone(&input))
.with_default_table_provider(Arc::new(BaseTableProvider::default()))
.with_session_factory(Arc::new(SessionCtxFactory::new(true)))
.with_parser(Arc::new(DefaultParser::default()))
.with_query_execution_factory(Arc::new(SqlQueryExecutionFactory::new(optimizer, scheduler)))
.with_func_manager(Arc::new(SimpleFunctionMetadataManager::default()))
.build()
.expect("query dispatcher should build");
let query = Query::new(
QueryContext {
input: Arc::clone(&input),
},
input.request.expression.clone(),
);
let query_state_machine = dispatcher
.build_query_state_machine(query)
.await
.expect("query should acquire admission");
let error = match dispatcher.build_logical_plan(query_state_machine).await {
Err(error) => error,
Ok(_) => panic!("invalid FileHeaderInfo must fail while building the provider"),
};
assert_eq!(error.select_error(), SelectError::InvalidDataSource);
}
#[tokio::test]
async fn csv_query_uses_custom_record_delimiter_across_file_partitions() {
const ROW_COUNT: usize = 200_000;
@@ -1420,7 +1455,7 @@ mod tests {
assert!(matches!(
result,
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryConcurrencyLimit))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryConcurrencyLimit))
));
}
@@ -1474,7 +1509,7 @@ mod tests {
assert!(matches!(
result,
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryConcurrencyLimit))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryConcurrencyLimit))
));
}
@@ -1571,7 +1606,7 @@ mod tests {
assert!(matches!(
result,
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryTimeout { seconds: 0 }))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryTimeout { seconds: 0 }))
));
assert_eq!(admission.available_permits(), 1);
}
@@ -1601,7 +1636,7 @@ mod tests {
assert!(matches!(
dispatcher.execute_logical_plan(logical_plan, query_state_machine).await,
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryTimeout { seconds: 300 }))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryTimeout { seconds: 300 }))
));
assert_eq!(admission.available_permits(), 1);
}
@@ -1742,7 +1777,7 @@ mod tests {
assert_eq!(admission.available_permits(), 1);
assert!(matches!(
dispatcher.build_logical_plan(query_state_machine).await,
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryTimeout { seconds: 1 }))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryTimeout { seconds: 1 }))
));
}
@@ -1843,7 +1878,7 @@ mod tests {
release_drop_tx.send(()).expect("release result drop");
assert!(matches!(
task.await.expect("deadline task should finish"),
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::QueryTimeout { seconds: 1 }))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::QueryTimeout { seconds: 1 }))
));
assert_eq!(admission.available_permits(), 1);
}
@@ -1985,8 +2020,8 @@ mod tests {
panic!("expected external query error");
};
assert!(matches!(
source.downcast_ref::<S3SelectPolicyError>(),
Some(S3SelectPolicyError::QueryTimeout { seconds: 300 })
source.downcast_ref::<SelectError>(),
Some(SelectError::QueryTimeout { seconds: 300 })
));
assert!(inner_dropped.load(Ordering::SeqCst));
assert_eq!(admission.available_permits(), 1);
@@ -2124,8 +2159,8 @@ mod tests {
panic!("expected external query error");
};
assert!(matches!(
source.downcast_ref::<S3SelectPolicyError>(),
Some(S3SelectPolicyError::QueryTimeout { seconds: 300 })
source.downcast_ref::<SelectError>(),
Some(SelectError::QueryTimeout { seconds: 300 })
));
assert_eq!(admission.available_permits(), 1);
assert!(output.next().await.is_none());
@@ -2174,8 +2209,8 @@ mod tests {
panic!("expected external query error");
};
assert!(matches!(
source.downcast_ref::<S3SelectPolicyError>(),
Some(S3SelectPolicyError::QueryTimeout { seconds: 1 })
source.downcast_ref::<SelectError>(),
Some(SelectError::QueryTimeout { seconds: 1 })
));
assert_eq!(admission.available_permits(), 1);
assert!(output.next().await.is_none());
@@ -38,7 +38,7 @@ use datafusion::{
};
use futures::{FutureExt, TryFutureExt, future::BoxFuture};
use rustfs_s3select_api::{
QueryError, QueryResult,
QueryResult,
object_store::{SelectScanRange, scan_range_from_bounds},
};
use s3s::dto::SelectObjectContentInput;
@@ -106,7 +106,10 @@ impl ParquetSelectTable {
let object_store_url = table_path.object_store();
let object_location = Path::from(input.key.clone());
let store = state.runtime_env().object_store(&object_store_url)?;
let object_meta = store.head(&object_location).await.map_err(query_store_error)?;
let object_meta = store
.head(&object_location)
.await
.map_err(datafusion::common::DataFusionError::from)?;
let reader = ObjectStoreParquetReader {
store: Arc::clone(&store),
@@ -115,7 +118,7 @@ impl ParquetSelectTable {
};
let builder = ParquetRecordBatchStreamBuilder::new(reader)
.await
.map_err(query_store_error)?;
.map_err(datafusion::common::DataFusionError::from)?;
let schema = Arc::clone(builder.schema());
let metadata = Arc::clone(builder.metadata());
let access_plan = parquet_access_plan(input, object_meta.size, metadata.as_ref())?;
@@ -180,7 +183,8 @@ fn parquet_access_plan(
let Some(scan_range) = input.request.scan_range.as_ref() else {
return Ok(None);
};
let scan_range = scan_range_from_bounds(scan_range.start, scan_range.end, object_size).map_err(query_store_error)?;
let scan_range = scan_range_from_bounds(scan_range.start, scan_range.end, object_size)
.map_err(datafusion::common::DataFusionError::from)?;
Ok(scan_range.map(|range| Arc::new(access_plan_for_scan_range(range, metadata))))
}
@@ -214,10 +218,6 @@ fn parquet_store_error(err: ObjectStoreError) -> ParquetError {
ParquetError::External(Box::new(err))
}
fn query_store_error(err: impl fmt::Display) -> QueryError {
QueryError::StoreError { e: err.to_string() }
}
#[cfg(test)]
mod tests {
use super::*;
@@ -227,7 +227,13 @@ mod tests {
datatypes::{DataType, Field, Schema, SchemaRef},
record_batch::RecordBatch,
},
object_store::memory::InMemory,
parquet::arrow::{ArrowWriter, arrow_reader::ParquetRecordBatchReaderBuilder},
prelude::SessionContext,
};
use rustfs_s3select_api::SelectError;
use s3s::dto::{
CSVOutput, ExpressionType, InputSerialization, OutputSerialization, ParquetInput, ScanRange, SelectObjectContentRequest,
};
use std::{
fs::File,
@@ -275,6 +281,85 @@ mod tests {
assert!(!plan.should_scan(1));
}
#[test]
fn parquet_access_plan_has_typed_invalid_scan_range_error() {
let metadata = two_row_group_metadata();
let mut input = parquet_input("test.parquet");
input.request.scan_range = Some(ScanRange {
start: Some(10),
end: None,
});
let error = parquet_access_plan(&input, 10, metadata.as_ref()).expect_err("out-of-bounds range must fail");
assert_eq!(error.select_error(), SelectError::InvalidScanRange);
}
#[tokio::test]
async fn try_new_preserves_missing_object_error() {
let store = Arc::new(InMemory::new());
let context = parquet_session(store);
let state = context.state();
let error = match ParquetSelectTable::try_new(&state, &parquet_input("missing.parquet")).await {
Ok(_) => panic!("missing parquet object must fail"),
Err(error) => error,
};
assert_eq!(error.select_error(), SelectError::ObjectNotFound);
}
#[tokio::test]
async fn try_new_preserves_parquet_metadata_error() {
let store = Arc::new(InMemory::new());
let object = Path::from("corrupt.parquet");
store
.put(&object, Bytes::from_static(b"not a parquet file").into())
.await
.expect("put corrupt parquet object");
let context = parquet_session(store);
let state = context.state();
let error = match ParquetSelectTable::try_new(&state, &parquet_input(object.as_ref())).await {
Ok(_) => panic!("corrupt parquet metadata must fail"),
Err(error) => error,
};
assert_eq!(error.select_error(), SelectError::ParquetParsingError);
}
fn parquet_session(store: Arc<dyn ObjectStore>) -> SessionContext {
let context = SessionContext::new();
let store_url = ObjectStoreUrl::parse("s3://test-bucket").expect("valid test object store URL");
context.register_object_store(store_url.as_ref(), store);
context
}
fn parquet_input(key: &str) -> SelectObjectContentInput {
SelectObjectContentInput {
bucket: "test-bucket".to_string(),
expected_bucket_owner: None,
key: key.to_string(),
sse_customer_algorithm: None,
sse_customer_key: None,
sse_customer_key_md5: None,
request: SelectObjectContentRequest {
expression: "SELECT * FROM S3Object".to_string(),
expression_type: ExpressionType::from_static(ExpressionType::SQL),
input_serialization: InputSerialization {
parquet: Some(ParquetInput::default()),
..Default::default()
},
output_serialization: OutputSerialization {
csv: Some(CSVOutput::default()),
..Default::default()
},
request_progress: None,
scan_range: None,
},
}
}
fn two_row_group_metadata() -> Arc<ParquetMetaData> {
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
+26 -12
View File
@@ -23,7 +23,7 @@ use datafusion::sql::{
},
};
use rustfs_s3select_api::{
QueryError, QueryResult, S3SelectPolicyError,
QueryError, QueryResult, SelectError,
query::{
ast::ExtStatement,
logical_planner::{LogicalPlanner, Plan, QueryPlan},
@@ -68,7 +68,7 @@ impl<'a, S: ContextProviderExtension + Send + Sync + 'a> SqlPlanner<'a, S> {
match stmt {
Statement::Query(_) => {
validate_s3_select_statement(&stmt)?;
let df_plan = self.df_planner.sql_statement_to_plan(stmt)?;
let df_plan = self.df_planner.sql_statement_to_plan(stmt).map_err(classify_planner_error)?;
let plan = Plan::Query(QueryPlan {
df_plan,
is_tag_scan: false,
@@ -76,11 +76,25 @@ impl<'a, S: ContextProviderExtension + Send + Sync + 'a> SqlPlanner<'a, S> {
Ok(plan)
}
_ => Err(QueryError::NotImplemented { err: stmt.to_string() }),
_ => Err(unsupported_structure("only SELECT queries are supported")),
}
}
}
fn classify_planner_error(error: datafusion::common::DataFusionError) -> QueryError {
if matches!(
&error,
datafusion::common::DataFusionError::Plan(message)
if message.starts_with("Failed to coerce arguments to satisfy a call to")
|| (message.starts_with("Internal error: Function '")
&& message.contains("' failed to match any signature, errors:"))
) {
return SelectError::IncorrectSqlFunctionArgumentType.into();
}
error.into()
}
fn validate_s3_select_statement(statement: &Statement) -> QueryResult<()> {
let Statement::Query(query) = statement else {
return Err(unsupported_structure("only SELECT queries are supported"));
@@ -191,7 +205,7 @@ fn validate_select(select: &Select) -> QueryResult<()> {
let ([ObjectNamePart::Identifier(table_name)] | [ObjectNamePart::Identifier(table_name), ObjectNamePart::Identifier(_)]) =
name.0.as_slice()
else {
return Err(unsupported_structure("the source must be S3Object"));
return Err(SelectError::DataSourcePathUnsupported.into());
};
let is_s3_object = if table_name.quote_style.is_some() {
table_name.value == "S3Object"
@@ -199,14 +213,14 @@ fn validate_select(select: &Select) -> QueryResult<()> {
table_name.value.eq_ignore_ascii_case("S3Object")
};
if !is_s3_object {
return Err(unsupported_structure("the source must be S3Object"));
return Err(SelectError::DataSourcePathUnsupported.into());
}
Ok(())
}
fn unsupported_structure(message: &str) -> QueryError {
S3SelectPolicyError::UnsupportedSqlStructure {
SelectError::UnsupportedSqlStructure {
message: message.to_string(),
}
.into()
@@ -234,7 +248,7 @@ mod tests {
use super::validate_s3_select_statement;
use crate::sql::parser::ExtParser;
use datafusion::sql::sqlparser::ast::Statement;
use rustfs_s3select_api::{S3SelectPolicyError, query::ast::ExtStatement};
use rustfs_s3select_api::{SelectError, query::ast::ExtStatement};
fn parse_statement(sql: &str) -> Statement {
let mut statements = ExtParser::parse_sql(sql).expect("SQL should parse");
@@ -271,7 +285,7 @@ mod tests {
validate_s3_select_statement(&statement),
Err(ref err) if matches!(
err.s3_select_policy_error(),
Some(S3SelectPolicyError::UnsupportedSqlStructure { message }) if message == "JOIN is not supported"
Some(SelectError::UnsupportedSqlStructure { message }) if message == "JOIN is not supported"
)
));
}
@@ -284,7 +298,7 @@ mod tests {
validate_s3_select_statement(&statement),
Err(ref err) if matches!(
err.s3_select_policy_error(),
Some(S3SelectPolicyError::UnsupportedSqlStructure { message }) if message == "subqueries are not supported"
Some(SelectError::UnsupportedSqlStructure { message }) if message == "subqueries are not supported"
)
));
}
@@ -297,7 +311,7 @@ mod tests {
validate_s3_select_statement(&statement),
Err(ref err) if matches!(
err.s3_select_policy_error(),
Some(S3SelectPolicyError::UnsupportedSqlStructure { message }) if message == "subqueries are not supported"
Some(SelectError::UnsupportedSqlStructure { message }) if message == "subqueries are not supported"
)
));
}
@@ -310,7 +324,7 @@ mod tests {
validate_s3_select_statement(&statement),
Err(ref err) if matches!(
err.s3_select_policy_error(),
Some(S3SelectPolicyError::UnsupportedSqlStructure { message }) if message == "the source must be S3Object"
Some(SelectError::DataSourcePathUnsupported)
)
));
}
@@ -326,7 +340,7 @@ mod tests {
assert!(
matches!(
validate_s3_select_statement(&statement),
Err(ref err) if matches!(err.s3_select_policy_error(), Some(S3SelectPolicyError::UnsupportedSqlStructure { .. }))
Err(ref err) if matches!(err.s3_select_policy_error(), Some(SelectError::UnsupportedSqlStructure { .. }))
),
"query should be rejected: {sql}"
);
@@ -16,7 +16,7 @@
mod error_handling_tests {
use crate::get_global_db;
use rustfs_s3select_api::{
QueryError,
QueryError, SelectError,
query::{Context, Query},
};
use s3s::dto::{
@@ -98,7 +98,6 @@ mod error_handling_tests {
"INSERT INTO S3Object VALUES (1, 'test')",
"UPDATE S3Object SET name = 'test'",
"DELETE FROM S3Object",
"CREATE TABLE test (id INT)",
"DROP TABLE S3Object",
];
@@ -113,6 +112,68 @@ mod error_handling_tests {
}
}
#[tokio::test]
async fn test_non_select_statement_is_typed_unsupported_structure() {
let sql = "CREATE TABLE test (id INT)";
let input = create_test_input_with_sql(sql);
let db = get_global_db(input.clone(), true).await.unwrap();
let query = Query::new(Context { input: Arc::new(input) }, sql.to_string());
let error = match db.execute(&query).await {
Err(error) => error,
Ok(_) => panic!("non-SELECT statement must fail"),
};
assert!(matches!(error.select_error(), SelectError::UnsupportedSqlStructure { .. }));
}
#[tokio::test]
async fn test_function_argument_coercion_failure_is_typed() {
for sql in ["SELECT ROUND(3.14, 1.1) FROM S3Object", "SELECT SQRT(1, 2) FROM S3Object"] {
let input = create_test_input_with_sql(sql);
let db = get_global_db(input.clone(), true)
.await
.expect("test database should initialize");
let query = Query::new(Context { input: Arc::new(input) }, sql.to_string());
let error = match db.execute(&query).await {
Err(error) => error,
Ok(_) => panic!("invalid function arguments must fail during planning: {sql}"),
};
assert_eq!(
error.select_error(),
SelectError::IncorrectSqlFunctionArgumentType,
"unexpected planner error for {sql}: {error:?}"
);
}
}
#[tokio::test]
async fn test_other_planner_failures_remain_invalid_query() {
for sql in [
"SELECT DEFINITELY_UNKNOWN_FUNCTION(1) FROM S3Object",
"SELECT 1 + 'text' FROM S3Object",
] {
let input = create_test_input_with_sql(sql);
let db = get_global_db(input.clone(), true)
.await
.expect("test database should initialize");
let query = Query::new(Context { input: Arc::new(input) }, sql.to_string());
let error = match db.execute(&query).await {
Err(error) => error,
Ok(_) => panic!("invalid query must fail during planning: {sql}"),
};
assert_eq!(
error.select_error(),
SelectError::InvalidQuery,
"unexpected planner error for {sql}: {error:?}"
);
}
}
#[tokio::test]
async fn test_invalid_column_references() {
let invalid_column_sqls = vec![
+39
View File
@@ -82,6 +82,7 @@ pub use storage_api::ScannerReplicationConfig as ReplicationConfig;
pub use storage_api::scan::SCANNER_ACTIVITY_PROTOCOL_VERSION;
static SCANNER_ACTIVE_WORK_UNITS: AtomicU64 = AtomicU64::new(0);
static SCANNER_RUNTIME_INSTANCES: AtomicU64 = AtomicU64::new(0);
static SCANNER_FOREGROUND_READ_ACTIVITY: AtomicU64 = AtomicU64::new(0);
static SCANNER_FOREGROUND_STREAM_READS: AtomicU64 = AtomicU64::new(0);
@@ -89,6 +90,10 @@ pub fn current_scanner_activity() -> u64 {
SCANNER_ACTIVE_WORK_UNITS.load(Ordering::Relaxed)
}
pub fn scanner_runtime_initialized() -> bool {
SCANNER_RUNTIME_INSTANCES.load(Ordering::Relaxed) > 0
}
pub fn set_foreground_read_activity(active: usize) {
let active = u64::try_from(active).unwrap_or(u64::MAX);
SCANNER_FOREGROUND_READ_ACTIVITY.store(active, Ordering::Relaxed);
@@ -138,6 +143,26 @@ impl ScannerActivityGuard {
}
}
pub(crate) struct ScannerRuntimeGuard;
impl ScannerRuntimeGuard {
pub(crate) fn new() -> Self {
SCANNER_RUNTIME_INSTANCES.fetch_add(1, Ordering::Relaxed);
Self
}
}
impl Drop for ScannerRuntimeGuard {
fn drop(&mut self) {
let _ = SCANNER_RUNTIME_INSTANCES.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| current.checked_sub(1));
}
}
#[cfg(test)]
fn reset_scanner_runtime_instances_for_test() {
SCANNER_RUNTIME_INSTANCES.store(0, Ordering::Relaxed);
}
impl Drop for ScannerActivityGuard {
fn drop(&mut self) {
let _ = SCANNER_ACTIVE_WORK_UNITS
@@ -559,4 +584,18 @@ mod tests {
set_foreground_read_activity(0);
assert_eq!(current_foreground_read_activity(), 1);
}
#[test]
#[serial]
fn scanner_runtime_guard_tracks_runtime_lifetime() {
reset_scanner_runtime_instances_for_test();
assert!(!scanner_runtime_initialized());
{
let _guard = ScannerRuntimeGuard::new();
assert!(scanner_runtime_initialized());
}
assert!(!scanner_runtime_initialized());
}
}
+3 -1
View File
@@ -34,7 +34,7 @@ use crate::scanner_io::{
scanner_dirty_usage_state, scanner_maintenance_changed, scanner_maintenance_generation,
};
use crate::sleeper::{SCANNER_SLEEPER, set_scanner_default_speed};
use crate::{DataUsageInfo, ScannerActivityGuard, ScannerError};
use crate::{DataUsageInfo, ScannerActivityGuard, ScannerError, ScannerRuntimeGuard};
use crate::{ScannerConfigObjectDelete, ScannerObjectIO, ScannerObjectOptions};
use bytes::Bytes;
use chrono::{DateTime, Utc};
@@ -1312,7 +1312,9 @@ pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc<ECStore>) {
let replication_active = startup_features.replication;
let ctx_clone = ctx;
let storeapi_clone = storeapi;
let runtime_guard = ScannerRuntimeGuard::new();
tokio::spawn(async move {
let _runtime_guard = runtime_guard;
let (usage_cache_is_cold, has_buckets) = initial_scanner_startup_usage_state(&storeapi_clone).await;
let sleep_time = initial_scanner_delay_for_startup(
scanner_start_delay().map(|duration| duration.as_secs()),
+1 -1
View File
@@ -3881,7 +3881,7 @@ impl ScannerIODisk for Disk {
Ok(size_summary)
}
#[tracing::instrument(skip(self, budget, updates, cache))]
#[tracing::instrument(skip(self, budget, updates, cache, set_disks))]
async fn nsscanner_disk(
self: Arc<Self>,
ctx: CancellationToken,
+1 -1
View File
@@ -23,7 +23,7 @@ for later deletion.
- `ns-scanner-rpc-v3` namespace scanner capability and activity handshake: old peers and legacy internode transports lack the authenticated startup-epoch handshake. The oldest peers send an empty activity request and receive a field-empty protocol-0 response. Protocol v4 binds the challenge and response topology but cannot authenticate distributed dirty-usage state. Protocol v5 binds the request version, acknowledgement target and generation, and the response dirty-usage state, but predates set-scoped scanner cache locks. Current protocol v6 additionally fences scanner cache lock-domain changes, so distributed scanner cycles publish usage only after every peer reports protocol v6 state. Servers retain protocol-0 and protocol-v4 codecs for rolling upgrades, while protocol-v5 peers are treated as previous-version peers that cannot safely participate in the new cache lock domain. Scanner selection treats HTTP 404/405/426 and the legacy MethodNotAllowed default as an explicit lack of remote scanner v3 support and assigns those disks to coordinator-driven workers; transient capability failures remain incomplete and do not activate the fallback. Remove the coordinator fallback after the minimum supported RustFS peer version implements namespace scanner protocol v3, remove protocol-0 activity requests and responses after every supported peer implements authenticated scanner activity protocol v4, remove the protocol-v4 activity codec after every supported peer implements protocol v5, and remove protocol-v5 previous-version rejection after every supported peer implements protocol v6; future protocol revisions must keep the same dual-version server/codec window before changing the advertised version.
- `#4648` walk-dir stream completion capability: old clients can append fallback output to an already-used metacache writer after a terminal body error, so servers emit terminal walk errors only to clients that sign the `walk_dir_stream_completion=error-v1` query capability and its request-body digest. Remove the legacy clean-EOF path after the minimum supported RustFS peer version always advertises this capability.
- `heal-rpc-auth-v2` internode gRPC authentication: servers temporarily accept legacy prefix signatures so old peers remain available during rolling upgrades. Remove the legacy fallback after the minimum supported RustFS peer version sends v2 authentication on every internode gRPC request.
- `disk-mutation-body-digest` internode mutating disk RPCs: servers temporarily accept mutating disk RPCs (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes) that carry no signature-bound canonical body digest, so peers from releases that predate body-digest signing remain available during rolling upgrades. Accepted digestless mutations increment the internode body-digest fallback counter; that counter must read zero fleet-wide across a release window before RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT is enabled. Because body-bound requests now consume replay-cache nonces on the receiver, deploy the raised RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY default fleet-wide before enabling strict mode, and watch the internode replay-cache overflow counter for undersized capacity during the rollout. Remove the digestless fallback after the minimum supported RustFS peer version body-binds every mutating disk RPC.
- `disk-mutation-body-digest` rolling-compatible internode mutations: servers temporarily accept high-frequency lock and disk mutations that carry no signature-bound canonical body digest, so older peers remain available during rolling upgrades. Clients use the authenticated but cache-free UNSIGNED-PAYLOAD v2 lane only for those explicitly marked mutations when a peer has no authenticated boot-epoch proof, as on beta.11. A peer with an authenticated boot epoch but no dynamic-cache capability, as on beta.12 or an unpatched RC1, receives body-bound v3 requests; a patched peer additionally pins the separately HMAC-bound dynamic-replay-cache-v1 capability. After that capability is pinned, a missing or invalid capability proof fails closed instead of silently permitting a rollback; an intentional rollback requires restarting the client process to clear the in-memory pin. IAM, service-control, heal-control, tier-mutation, and scanner-activity contracts remain body-bound. Accepted digestless mutations increment the internode body-digest fallback counter; that counter must read zero fleet-wide across a release window before RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT is enabled. Because body-bound requests consume replay-cache nonces on the receiver, deploy the raised RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY default fleet-wide before enabling strict mode, and watch the internode replay-cache overflow counter for undersized capacity during the rollout. Remove both client and server fallbacks after every supported peer advertises the authenticated replay-cache capability, body-binds every rolling-compatible mutation, and body-digest strict mode is the default.
- `heal-status-rpc-v1` node heal status capability: new peers treat an unimplemented BackgroundHealStatus RPC as an explicitly incomplete rolling-upgrade response. Remove the fallback after the minimum supported RustFS peer version implements BackgroundHealStatus.
- `replacement-recovery-status-v1` replacement recovery status capability: new peers treat an unimplemented ReplacementRecoveryStatus RPC as an explicitly non-definitive rolling-upgrade response, so Admin v4 cannot claim distributed replacement completion from old peers. Remove the fallback after the minimum supported RustFS peer version implements ReplacementRecoveryStatus.
- `backlog-1316` legacy encrypted multipart range seek: the feature remains opt-in until every server that can initiate, write, or complete multipart uploads supports the candidate-to-final marker protocol and uploadId commit lock, and pre-upgrade multipart uploads have drained. Remove the RUSTFS_ENCRYPTED_RANGE_SEEK switch after the minimum supported release does so; keep the quorum marker and malformed-layout full-read guards permanently.
+17
View File
@@ -42,6 +42,7 @@ target results remain present when another target fails.
"Versioning": { "Status": "OK" },
"ObjectLock": { "Status": "OK" },
"Put": { "Status": "OK" },
"VersionFidelity": { "Status": "OK" },
"DeleteMarker": { "Status": "OK" },
"VersionDelete": { "Status": "OK" },
"Cleanup": {
@@ -58,3 +59,19 @@ Phase states are `OK`, `FAILED`, or `SKIPPED`. Errors are single-line, bounded
to 512 bytes, and omit remote messages, endpoints, credentials, signatures, and
authorization material. A cleanup failure is always explicit; it is never
reported as a successful check.
`VersionFidelity` pins the version-identity contract on **both** write paths:
the probe PUT carries a source version id (header plus `?versionId=` query,
the exact shape live replication uses) and the target must answer with the
same id, and a second probe repeats it through CreateMultipartUpload ->
UploadPart -> CompleteMultipartUpload, where the target fixes the version at
initiate and only reports it on completion. A target can adopt PutObject ids
and still mint its own for multipart, which would leave multipart deletes and
heals addressing a version that never existed; the failure message names the
path that drifted. Targets that
mint their own version ids break every version-addressed operation that
follows (version deletes, heal re-drives), so the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`,
the later mutation phases are skipped, and cleanup still removes the probe via
the version id the target actually assigned. `Code` only appears on failures
that callers are expected to branch on; Go decoders ignore the unknown key.
+50 -3
View File
@@ -28,7 +28,7 @@ use crate::server::{
};
use crate::version::build;
use axum::{
Json, Router,
Extension, Json, Router,
body::Body,
extract::Request,
middleware,
@@ -632,13 +632,22 @@ fn setup_console_middleware_stack(
/// # Returns:
/// - A `Response` containing the health check result.
#[instrument]
async fn health_check(method: Method, uri: Uri) -> Response {
async fn health_check(
method: Method,
uri: Uri,
server_ctx: Option<Extension<Arc<crate::runtime_sources::ServerContextSlot>>>,
) -> Response {
let probe = if uri.path().strip_prefix(CONSOLE_PREFIX) == Some(HEALTH_READY_PATH) {
HealthProbe::Readiness
} else {
HealthProbe::Liveness
};
let readiness_report = collect_probe_readiness(probe).await;
let app_context = match server_ctx {
Some(Extension(server_ctx)) => server_ctx.installed_app_context(),
None => crate::runtime_sources::current_app_context(),
};
let object_traffic_health = app_context.map(|context| context.object_traffic_health());
let readiness_report = collect_probe_readiness(probe, object_traffic_health.as_deref()).await;
let uptime = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
@@ -919,6 +928,44 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn console_readiness_uses_the_request_server_object_progress() {
temp_env::async_with_vars([(rustfs_config::ENV_HEALTH_MINIMAL_RESPONSE_ENABLE, Some("false"))], async {
let object_traffic_health =
Arc::new(crate::app::object_traffic_health::ObjectTrafficHealth::enabled_for_test(Duration::ZERO));
let stalled = object_traffic_health
.track_write_storage()
.expect("write tracking must be enabled");
let app_context =
crate::app::gating_test_env::app_context_with_object_traffic_health(Arc::clone(&object_traffic_health)).await;
let server_ctx = crate::runtime_sources::ServerContextSlot::new();
assert!(server_ctx.install(app_context));
let response = health_check(
Method::GET,
format!("{CONSOLE_PREFIX}{HEALTH_READY_PATH}")
.parse()
.expect("console readiness URI"),
Some(Extension(server_ctx)),
)
.await;
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = response
.into_body()
.collect()
.await
.expect("console readiness body")
.to_bytes();
let payload: serde_json::Value = serde_json::from_slice(&body).expect("console readiness JSON");
assert_eq!(payload["ready"], false);
assert_eq!(payload["degradedReasons"], serde_json::json!(["object_write_stalled"]));
drop(stalled);
})
.await;
}
// setup_console_middleware_stack reads ENV_HEALTH_ENDPOINT_ENABLE (see above).
#[tokio::test]
#[serial]
+59 -12
View File
@@ -105,10 +105,9 @@ pub struct GetClusterSnapshotHandler {}
impl Operation for GetClusterSnapshotHandler {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
authorize_cluster_snapshot_request(&req).await?;
let snapshot = default_admin_usecase()
.execute_collect_cluster_read_only_snapshot()
.await
.map(ClusterSnapshotView::from);
let snapshot = default_admin_usecase().execute_collect_cluster_read_only_snapshot().await;
let server_info_endpoint = crate::runtime_sources::current_local_node_name().await;
let snapshot = snapshot.map(|snapshot| ClusterSnapshotView::from_snapshot(snapshot, server_info_endpoint));
build_json_response(StatusCode::OK, &ClusterSnapshotResponse { snapshot }, req.headers.get("x-request-id"))
}
}
@@ -166,6 +165,12 @@ pub(crate) struct ClusterSnapshotView {
impl From<ClusterReadOnlySnapshot> for ClusterSnapshotView {
fn from(snapshot: ClusterReadOnlySnapshot) -> Self {
Self::from_snapshot(snapshot, None)
}
}
impl ClusterSnapshotView {
fn from_snapshot(snapshot: ClusterReadOnlySnapshot, server_info_endpoint: Option<String>) -> Self {
let components = ClusterComponentStatusView::from_snapshot(&snapshot);
let summary = ClusterSnapshotSummary::from_snapshot_and_components(&snapshot, &components);
let actionable_pressure = cluster_has_actionable_pressure(&snapshot);
@@ -175,7 +180,7 @@ impl From<ClusterReadOnlySnapshot> for ClusterSnapshotView {
extensions_catalog_path: format!("{}{}", ADMIN_PREFIX, "/v4/extensions/catalog"),
components,
topology: snapshot.topology,
membership: ClusterMembershipView::from(snapshot.membership),
membership: ClusterMembershipView::from_snapshot(snapshot.membership, server_info_endpoint),
pool_state: ClusterPoolStateView::from(snapshot.pool_state),
local_storage: ClusterLocalStorageView::from(snapshot.local_storage),
peer_health: ClusterPeerHealthView::from(snapshot.peer_health),
@@ -324,8 +329,25 @@ pub(crate) struct ClusterMembershipView {
impl From<ClusterMembershipSnapshot> for ClusterMembershipView {
fn from(snapshot: ClusterMembershipSnapshot) -> Self {
Self::from_snapshot(snapshot, None)
}
}
impl ClusterMembershipView {
fn from_snapshot(snapshot: ClusterMembershipSnapshot, server_info_endpoint: Option<String>) -> Self {
Self {
nodes: snapshot.nodes.into_iter().map(ClusterNodeMembershipView::from).collect(),
nodes: snapshot
.nodes
.into_iter()
.map(|node| {
let endpoint = if node.is_local && node.node_id == "local" {
server_info_endpoint.clone()
} else {
None
};
ClusterNodeMembershipView::from_node(node, endpoint)
})
.collect(),
drives: snapshot.drives.into_iter().map(ClusterDriveMembershipView::from).collect(),
}
}
@@ -335,15 +357,18 @@ impl From<ClusterMembershipSnapshot> for ClusterMembershipView {
pub(crate) struct ClusterNodeMembershipView {
pub node_id: String,
pub grid_host: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub server_info_endpoint: Option<String>,
pub is_local: bool,
pub pools: Vec<usize>,
}
impl From<ClusterNodeMembership> for ClusterNodeMembershipView {
fn from(node: ClusterNodeMembership) -> Self {
impl ClusterNodeMembershipView {
fn from_node(node: ClusterNodeMembership, server_info_endpoint: Option<String>) -> Self {
Self {
node_id: node.node_id,
grid_host: node.grid_host,
server_info_endpoint,
is_local: node.is_local,
pools: node.pools,
}
@@ -891,7 +916,7 @@ fn summarize_named_capability_statuses<const N: usize>(
#[cfg(test)]
mod tests {
use super::{ClusterSnapshotResponse, ClusterSnapshotSummary, ClusterSnapshotView};
use super::{ClusterMembershipView, ClusterSnapshotResponse, ClusterSnapshotSummary, ClusterSnapshotView};
use crate::admin::storage_api::cluster::CapabilityState;
use crate::admin::storage_api::cluster::{CapabilityStatus, ObservabilitySnapshot, TopologySnapshot};
use crate::admin::storage_api::cluster::{
@@ -917,6 +942,11 @@ mod tests {
handler_block.contains("authorize_cluster_snapshot_request(&req).await?;"),
"cluster snapshot handler should require admin authorization"
);
assert!(
handler_block.contains("current_local_node_name().await")
&& handler_block.contains("ClusterSnapshotView::from_snapshot"),
"cluster snapshot handler should attach the v3 server-info identity"
);
assert!(
auth_block.contains("AdminAction::ServerInfoAdminAction"),
"cluster snapshot should require server info admin permission"
@@ -949,8 +979,8 @@ mod tests {
topology: TopologySnapshot::default(),
membership: ClusterMembershipSnapshot {
nodes: vec![ClusterNodeMembership {
node_id: "node-a".to_string(),
grid_host: "node-a:9000".to_string(),
node_id: "local".to_string(),
grid_host: String::new(),
is_local: true,
pools: vec![0],
}],
@@ -1020,7 +1050,8 @@ mod tests {
},
};
let value = serde_json::to_value(ClusterSnapshotView::from(snapshot)).expect("serialize view");
let value = serde_json::to_value(ClusterSnapshotView::from_snapshot(snapshot, Some(":::9000".to_string())))
.expect("serialize view");
assert_eq!(value["runtime_capabilities_path"], "/rustfs/admin/v4/runtime/capabilities");
assert_eq!(value["extensions_catalog_path"], "/rustfs/admin/v4/extensions/catalog");
assert_eq!(value["components"]["storage"]["source"], "runtime_readiness");
@@ -1031,6 +1062,7 @@ mod tests {
assert_eq!(value["components"]["listing"]["internode_stall_timeouts_total"], 2);
assert_eq!(value["components"]["usage"]["source"], "scanner_metrics");
assert_eq!(value["components"]["usage"]["condition"], "stale");
assert_eq!(value["membership"]["nodes"][0]["server_info_endpoint"], ":::9000");
assert_eq!(value["membership"]["drives"][0]["endpoint_type"], "url");
assert_eq!(value["workload_admission"][0]["class"], "repair");
assert_eq!(value["workload_admission"][0]["state"], "unknown");
@@ -1042,6 +1074,21 @@ mod tests {
assert_eq!(value["summary"]["rpc_boundary"]["state"], "supported");
assert_eq!(value["runtime_status"]["degraded_reasons"][0], "storage_and_lock_unavailable");
assert_eq!(value["actionable_pressure"], true);
let remote_membership = ClusterMembershipView::from_snapshot(
ClusterMembershipSnapshot {
nodes: vec![ClusterNodeMembership {
node_id: "node-b:9000".to_string(),
grid_host: "http://node-b:9000".to_string(),
is_local: false,
pools: vec![0],
}],
drives: Vec::new(),
},
Some(":::9000".to_string()),
);
let remote_value = serde_json::to_value(remote_membership).expect("serialize remote membership");
assert!(remote_value["nodes"][0].get("server_info_endpoint").is_none());
}
#[test]
+3 -1
View File
@@ -14,6 +14,7 @@
use super::profile::{TriggerProfileCPU, TriggerProfileMemory};
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::runtime_sources::app_context_from_req;
use crate::server::{
HEALTH_PREFIX, HEALTH_READY_PATH, PROFILE_CPU_PATH, PROFILE_MEMORY_PATH, build_health_response_parts,
collect_probe_readiness, probe_from_path,
@@ -51,6 +52,7 @@ pub struct HealthCheckHandler {}
#[async_trait::async_trait]
impl Operation for HealthCheckHandler {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let object_traffic_health = app_context_from_req(&req).map(|context| context.object_traffic_health());
// Extract the original HTTP Method (encapsulated by s3s into S3Request)
let method = req.method;
@@ -66,7 +68,7 @@ impl Operation for HealthCheckHandler {
}
let probe = probe_from_path(req.uri.path());
let readiness_report = collect_probe_readiness(probe).await;
let readiness_report = collect_probe_readiness(probe, object_traffic_health.as_deref()).await;
let response_parts =
build_health_response_parts(method.clone(), probe, readiness_report.as_ref(), "rustfs-endpoint", None, None);
+7 -1
View File
@@ -449,9 +449,15 @@ impl Operation for ValidateOidcConfigHandler {
request.provider_id.trim().to_string()
};
let provider_config = build_provider_config_from_validate(request, &provider_id)?;
let validation = rustfs_iam::oidc::validate_oidc_provider_config(&provider_config)
let oidc_extra_root_ca = crate::startup_auth::current_oidc_extra_root_ca_material()
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?;
let validation = rustfs_iam::oidc::validate_oidc_provider_config_with_extra_root_ca(
&provider_config,
oidc_extra_root_ca.root_ca_pem.as_deref(),
)
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?;
json_response(
StatusCode::OK,
File diff suppressed because it is too large Load Diff
+1
View File
@@ -24,6 +24,7 @@ pub mod router;
pub(crate) mod runtime_sources;
pub mod service;
pub mod site_replication_identity;
pub(crate) mod site_replication_state;
pub(crate) mod storage_api;
pub mod utils;
+379 -35
View File
@@ -17,7 +17,7 @@ use super::storage_api::bucket::metadata_sys;
use super::storage_api::bucket::replication::{self, BucketReplicationResyncStatus, BucketStats, ReplicationStatusType};
use super::storage_api::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
use super::storage_api::bucket::target_sys::{
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient,
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient, append_version_id_query,
};
use super::storage_api::bucket::versioning_sys::BucketVersioningSys;
use super::storage_api::bucket::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _};
@@ -42,6 +42,7 @@ use crate::server::{
};
use crate::storage::storage_api::lock_bucket_targets_metadata;
use aws_sdk_s3::primitives::ByteStream as AwsByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use bytes::Bytes;
use futures::{Stream, StreamExt};
use http::HeaderValue;
@@ -206,6 +207,9 @@ struct ReplicationResetStatusTarget {
const REPLICATION_CHECK_PROBE_PREFIX: &str = ".rustfs.sys/replication-check/";
const REPLICATION_CHECK_ERROR_MAX_BYTES: usize = 512;
/// RustFS extension code (no madmin analogue): the target does not adopt the
/// source version id, breaking the version-identity replication contract.
const REPLICATION_CHECK_CODE_VERSION_MISMATCH: &str = "BucketRemoteTargetVersionMismatch";
#[derive(Debug, Clone, serde::Serialize)]
struct ReplicationCheckResponse {
@@ -245,6 +249,8 @@ struct ReplicationCheckPhases {
object_lock: ReplicationCheckPhaseStatus,
#[serde(rename = "Put")]
put: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionFidelity")]
version_fidelity: ReplicationCheckPhaseStatus,
#[serde(rename = "DeleteMarker")]
delete_marker: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionDelete")]
@@ -259,6 +265,11 @@ struct ReplicationCheckPhaseStatus {
status: &'static str,
#[serde(rename = "Error", skip_serializing_if = "Option::is_none")]
error: Option<String>,
/// Machine-readable failure code (RustFS extension key; Go decoders
/// ignore unknown keys). Only set for failures that a caller is expected
/// to branch on, e.g. `BucketRemoteTargetVersionMismatch`.
#[serde(rename = "Code", skip_serializing_if = "Option::is_none")]
code: Option<&'static str>,
}
impl Default for ReplicationCheckPhaseStatus {
@@ -266,6 +277,7 @@ impl Default for ReplicationCheckPhaseStatus {
Self {
status: "SKIPPED",
error: None,
code: None,
}
}
}
@@ -275,6 +287,7 @@ impl ReplicationCheckPhaseStatus {
Self {
status: "OK",
error: None,
code: None,
}
}
@@ -282,6 +295,14 @@ impl ReplicationCheckPhaseStatus {
Self {
status: "FAILED",
error: Some(bound_replication_check_error(error.into())),
code: None,
}
}
fn failed_with_code(error: impl Into<String>, code: &'static str) -> Self {
Self {
code: Some(code),
..Self::failed(error)
}
}
}
@@ -2046,12 +2067,39 @@ fn fail_replication_check_target(result: &mut ReplicationCheckTargetStatus, erro
}
}
/// The probe PUT reports both sides of the version-identity contract: the
/// source version id it sent (header + `?versionId=` query, the exact shape
/// live replication uses) and the version id the target answered with.
struct ReplicationProbePutOutcome {
sent_version_id: String,
response_version_id: Option<String>,
}
struct ReplicationProbeMultipartError {
primary: S3ClientError,
cleanup_error: Option<String>,
}
impl From<S3ClientError> for ReplicationProbeMultipartError {
fn from(primary: S3ClientError) -> Self {
Self {
primary,
cleanup_error: None,
}
}
}
#[async_trait::async_trait]
trait ReplicationProbeOperations {
async fn put(&mut self) -> Result<Option<String>, S3ClientError>;
async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError>;
/// Multipart decides the target version at initiate time and only reports
/// it on completion, so the identity contract has to be probed separately
/// there: a target can adopt PutObject version ids and still mint its own
/// for CreateMultipartUpload.
async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError>;
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError>;
async fn delete_version(&mut self, version_id: Option<&str>) -> Result<(), S3ClientError>;
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String>;
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String>;
}
struct RemoteReplicationProbeOperations<'a> {
@@ -2063,10 +2111,14 @@ struct RemoteReplicationProbeOperations<'a> {
#[async_trait::async_trait]
impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
async fn put(&mut self) -> Result<Option<String>, S3ClientError> {
async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError> {
put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
}
async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
multipart_put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
}
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
delete_replication_probe_object(
self.client,
@@ -2090,20 +2142,49 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
.map(|_| ())
}
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String> {
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
cleanup_replication_probe(self.client, self.bucket, self.key, known_version_ids).await
}
}
/// `None` when the target adopted the source version id on this path.
fn version_fidelity_error(api: &str, outcome: &ReplicationProbePutOutcome) -> Option<String> {
if outcome.response_version_id.as_deref() == Some(outcome.sent_version_id.as_str()) {
return None;
}
Some(format!(
"target assigned version id {} instead of adopting the source version id {} on {api}; \
version-addressed replication (version deletes, heal) cannot converge on this target",
outcome.response_version_id.as_deref().unwrap_or("<none>"),
outcome.sent_version_id,
))
}
async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, operations: &mut impl ReplicationProbeOperations) {
let mut probe_version_id = None;
let mut multipart_probe_version_id = None;
let mut delete_marker_version_id = None;
let mut cleanup_required = true;
let mut multipart_cleanup_error = None;
match operations.put().await {
Ok(version_id) => {
probe_version_id = version_id;
Ok(outcome) => {
result.phases.put = ReplicationCheckPhaseStatus::passed();
// P1-19 version-identity contract: replication only converges on
// targets that adopt the source version id — version-addressed
// deletes and heal re-drives never match a minted id. Judge it
// from the probe PUT's own response; on mismatch the later
// mutation phases are pointless (they address by version id), but
// cleanup still runs against whatever id the target assigned.
match version_fidelity_error("PutObject", &outcome) {
None => result.phases.version_fidelity = ReplicationCheckPhaseStatus::passed(),
Some(error) => {
result.phases.version_fidelity =
ReplicationCheckPhaseStatus::failed_with_code(&error, REPLICATION_CHECK_CODE_VERSION_MISMATCH);
fail_replication_check_target(result, error);
}
}
probe_version_id = outcome.response_version_id;
}
Err(err) => {
let error = format_replication_check_client_error(&err, ReplicationCheckFailureContext::ReplicateObject);
@@ -2115,7 +2196,29 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
}
}
if result.phases.put.status == "OK" {
// The multipart path fixes the target version at initiate and only
// reports it on completion, so a target can adopt PutObject ids and still
// mint its own here — probe it before declaring the contract met.
if result.phases.version_fidelity.status == "OK" {
match operations.multipart_put().await {
Ok(outcome) => {
multipart_probe_version_id = outcome.response_version_id.clone();
if let Some(error) = version_fidelity_error("CreateMultipartUpload", &outcome) {
result.phases.version_fidelity =
ReplicationCheckPhaseStatus::failed_with_code(&error, REPLICATION_CHECK_CODE_VERSION_MISMATCH);
fail_replication_check_target(result, error);
}
}
Err(err) => {
let error = format_replication_check_client_error(&err.primary, ReplicationCheckFailureContext::ReplicateObject);
result.phases.version_fidelity = ReplicationCheckPhaseStatus::failed(&error);
fail_replication_check_target(result, error);
multipart_cleanup_error = err.cleanup_error;
}
}
}
if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
match operations.create_delete_marker(probe_version_id.as_deref()).await {
Ok(version_id) => {
delete_marker_version_id = version_id;
@@ -2138,19 +2241,31 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
}
}
if cleanup_required {
match operations
.cleanup([probe_version_id.as_deref(), delete_marker_version_id.as_deref()])
let cleanup_result = if cleanup_required {
operations
.cleanup([
probe_version_id.as_deref(),
multipart_probe_version_id.as_deref(),
delete_marker_version_id.as_deref(),
])
.await
{
Ok(()) => result.phases.cleanup = ReplicationCheckPhaseStatus::passed(),
Err(error) => {
result.phases.cleanup = ReplicationCheckPhaseStatus::failed(&error);
fail_replication_check_target(result, format!("probe cleanup failed: {error}"));
}
}
} else {
Ok(())
};
let mut cleanup_errors = Vec::new();
if let Some(error) = multipart_cleanup_error {
cleanup_errors.push(error);
}
if let Err(error) = cleanup_result {
cleanup_errors.push(error);
}
if cleanup_errors.is_empty() {
result.phases.cleanup = ReplicationCheckPhaseStatus::passed();
} else {
let error = cleanup_errors.join("; ");
result.phases.cleanup = ReplicationCheckPhaseStatus::failed(&error);
fail_replication_check_target(result, format!("probe cleanup failed: {error}"));
}
}
@@ -2225,13 +2340,7 @@ fn build_replication_probe_remove_options(now: OffsetDateTime, replication_delet
}
}
async fn put_replication_probe_object(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
now: OffsetDateTime,
) -> Result<Option<String>, S3ClientError> {
let options = build_replication_probe_put_options(now);
fn build_replication_probe_headers(options: &PutObjectOptions) -> HeaderMap {
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &options.internal.source_version_id);
insert_header(
@@ -2245,8 +2354,166 @@ async fn put_replication_probe_object(
HeaderName::from_static("x-amz-replication-status"),
HeaderValue::from_static(ReplicationStatusType::Replica.as_str()),
);
headers
}
target_client
/// Probe the identity contract on the multipart path: initiate carrying the
/// source version as `?versionId=` (where the target fixes the version),
/// upload one small part, and read the version the completion reports.
async fn multipart_put_replication_probe_object(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
now: OffsetDateTime,
) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
let options = build_replication_probe_put_options(now);
let sent_version_id = options.internal.source_version_id.clone();
let headers = build_replication_probe_headers(&options);
let initiate_headers = headers.clone();
let initiate_version_id = sent_version_id.clone();
let created = target_client
.client
.create_multipart_upload()
.bucket(target_bucket)
.key(probe_key)
.customize()
.map_request(move |mut req| {
for (key, value) in initiate_headers.clone() {
req.headers_mut().insert(key.expect("operation should succeed"), value);
}
let uri = append_version_id_query(req.uri(), &initiate_version_id);
req.set_uri(uri).map_err(std::io::Error::other)?;
Result::<_, std::io::Error>::Ok(req)
})
.send()
.await
.map_err(S3ClientError::from)
.map_err(ReplicationProbeMultipartError::from)?;
let upload_id = created
.upload_id()
.ok_or_else(|| S3ClientError::new("target multipart initiate returned no upload id"))
.map_err(ReplicationProbeMultipartError::from)?
.to_string();
let uploaded = match target_client
.client
.upload_part()
.bucket(target_bucket)
.key(probe_key)
.upload_id(&upload_id)
.part_number(1)
.content_length(8)
.body(AwsByteStream::from_static(b"aaaaaaaa"))
.send()
.await
{
Ok(uploaded) => uploaded,
Err(error) => {
return Err(abort_failed_replication_probe_multipart(
target_client,
target_bucket,
probe_key,
&upload_id,
S3ClientError::from(error),
)
.await);
}
};
let completed_part = CompletedPart::builder()
.part_number(1)
.set_e_tag(uploaded.e_tag().map(ToOwned::to_owned))
.build();
let complete_headers = headers.clone();
let completed = match target_client
.client
.complete_multipart_upload()
.bucket(target_bucket)
.key(probe_key)
.upload_id(&upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.set_parts(Some(vec![completed_part]))
.build(),
)
.customize()
.map_request(move |mut req| {
for (key, value) in complete_headers.clone() {
req.headers_mut().insert(key.expect("operation should succeed"), value);
}
Result::<_, std::io::Error>::Ok(req)
})
.send()
.await
{
Ok(completed) => completed,
Err(error) => {
return Err(abort_failed_replication_probe_multipart(
target_client,
target_bucket,
probe_key,
&upload_id,
S3ClientError::from(error),
)
.await);
}
};
Ok(ReplicationProbePutOutcome {
sent_version_id,
response_version_id: completed.version_id().map(ToOwned::to_owned),
})
}
async fn abort_failed_replication_probe_multipart(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
upload_id: &str,
primary_error: S3ClientError,
) -> ReplicationProbeMultipartError {
match target_client
.client
.abort_multipart_upload()
.bucket(target_bucket)
.key(probe_key)
.upload_id(upload_id)
.send()
.await
{
Ok(_) => ReplicationProbeMultipartError::from(primary_error),
Err(error) => {
let abort_error = S3ClientError::from(error);
if abort_error.code.as_deref() == Some("NoSuchUpload") {
ReplicationProbeMultipartError::from(primary_error)
} else {
ReplicationProbeMultipartError {
primary: primary_error,
cleanup_error: Some("failed to abort multipart replication probe".to_string()),
}
}
}
}
}
async fn put_replication_probe_object(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
now: OffsetDateTime,
) -> Result<ReplicationProbePutOutcome, S3ClientError> {
let options = build_replication_probe_put_options(now);
let sent_version_id = options.internal.source_version_id.clone();
let headers = build_replication_probe_headers(&options);
// Carry the source version as `?versionId=` exactly like a live
// replication PUT (P0-5 shape): the probe must exercise the query the
// real data path relies on, and the response tells us whether the target
// adopts the id. The probe id is always a fresh non-nil UUID, so the
// null-version mapping in the live path does not apply here.
let query_version_id = sent_version_id.clone();
let response = target_client
.client
.put_object()
.bucket(target_bucket)
@@ -2259,12 +2526,18 @@ async fn put_replication_probe_object(
for (key, value) in headers.clone() {
req.headers_mut().insert(key.expect("operation should succeed"), value);
}
let uri = append_version_id_query(req.uri(), &query_version_id);
req.set_uri(uri).map_err(std::io::Error::other)?;
Result::<_, std::io::Error>::Ok(req)
})
.send()
.await
.map(|output| output.version_id().map(ToOwned::to_owned))
.map_err(S3ClientError::from)
.map_err(S3ClientError::from)?;
Ok(ReplicationProbePutOutcome {
sent_version_id,
response_version_id: response.version_id().map(ToOwned::to_owned),
})
}
async fn delete_replication_probe_object(
@@ -3431,6 +3704,12 @@ mod tests {
#[derive(Default)]
struct ScriptedReplicationProbe {
put_error: Option<&'static str>,
/// Version id the scripted target answers with on PUT; None models a
/// mirroring target that echoes the sent source version id.
minted_version_id: Option<&'static str>,
/// Same, for the multipart leg: a target may mirror PutObject ids and
/// still mint its own at CreateMultipartUpload.
minted_multipart_version_id: Option<&'static str>,
delete_marker_error: Option<&'static str>,
version_delete_error: Option<&'static str>,
cleanup_error: Option<&'static str>,
@@ -3449,14 +3728,25 @@ mod tests {
#[async_trait::async_trait]
impl ReplicationProbeOperations for ScriptedReplicationProbe {
async fn put(&mut self) -> Result<Option<String>, S3ClientError> {
async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError> {
self.calls.push("put");
match self.put_error {
Some(code) => Err(scripted_probe_error(code)),
None => Ok(Some("object-version".to_string())),
None => Ok(ReplicationProbePutOutcome {
sent_version_id: "object-version".to_string(),
response_version_id: Some(self.minted_version_id.unwrap_or("object-version").to_string()),
}),
}
}
async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
self.calls.push("multipart-put");
Ok(ReplicationProbePutOutcome {
sent_version_id: "multipart-version".to_string(),
response_version_id: Some(self.minted_multipart_version_id.unwrap_or("multipart-version").to_string()),
})
}
async fn create_delete_marker(&mut self, _version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
self.calls.push("delete-marker");
match self.delete_marker_error {
@@ -3473,7 +3763,7 @@ mod tests {
}
}
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String> {
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
self.calls.push("cleanup");
self.cleanup_ids = known_version_ids
.into_iter()
@@ -3486,6 +3776,44 @@ mod tests {
}
}
/// P1-19: a target that mints its own version ids must fail the
/// VersionFidelity phase with the machine-readable mismatch code, skip
/// the version-addressed mutation phases (they cannot mean anything on a
/// drifting target), and still clean up using the id the target actually
/// assigned — the source-derived id would never match.
#[tokio::test]
async fn replication_probe_flags_version_minting_target() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe {
minted_version_id: Some("target-minted-version"),
..Default::default()
};
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "cleanup"]);
assert_eq!(result.status, "FAILED");
assert_eq!(result.phases.put.status, "OK");
assert_eq!(result.phases.version_fidelity.status, "FAILED");
assert_eq!(result.phases.version_fidelity.code, Some(REPLICATION_CHECK_CODE_VERSION_MISMATCH));
assert_eq!(result.phases.delete_marker.status, "SKIPPED");
assert_eq!(result.phases.version_delete.status, "SKIPPED");
assert_eq!(result.phases.cleanup.status, "OK");
assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None]);
}
#[tokio::test]
async fn replication_probe_passes_version_fidelity_for_mirroring_target() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe::default();
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(result.status, "OK");
assert_eq!(result.phases.version_fidelity.status, "OK");
assert_eq!(result.phases.version_fidelity.code, None);
}
#[tokio::test]
async fn replication_probe_attempts_cleanup_after_ambiguous_put_failure() {
let mut result = replication_check_target("arn:a", "OK", None);
@@ -3514,8 +3842,15 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(operations.cleanup_ids, [Some("object-version".to_string()), None]);
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(
operations.cleanup_ids,
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
None
]
);
assert_eq!(result.phases.delete_marker.status, "FAILED");
assert_eq!(result.phases.version_delete.status, "OK");
assert_eq!(result.phases.cleanup.status, "OK");
@@ -3532,10 +3867,14 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(
operations.cleanup_ids,
[Some("object-version".to_string()), Some("marker-version".to_string())]
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
Some("marker-version".to_string())
]
);
assert_eq!(result.phases.version_delete.status, "FAILED");
assert_eq!(result.phases.cleanup.status, "FAILED");
@@ -4671,6 +5010,11 @@ mod tests {
Url::parse(&format!("https://object-lambda.test:{}/transform", address.port())).expect("object lambda TLS endpoint");
let mut config = object_lambda_test_config(endpoint.clone());
config.client_ca = ca_path.to_string_lossy().into_owned();
// This is the only object-lambda test that performs a real TLS
// handshake; under a full-suite nextest run the CPU contention from
// neighboring tests pushes it past the helper's tight 2s request
// deadline. SNI preservation, not latency, is under test here.
config.response_header_timeout = Some(Duration::from_secs(30));
let response = build_object_lambda_http_client_with_resolver(&config, StaticResolver(address.ip()))
.expect("object lambda TLS client should build")
+30 -19
View File
@@ -16,14 +16,14 @@ use crate::admin::runtime_sources::{AppContext, current_app_context, current_obj
use crate::admin::site_replication_identity::{
deployment_id_for_endpoint, mark_unknown_peer_sync_enabled, normalize_peer_map_by_identity_with,
};
use crate::admin::storage_api::config::{read_admin_config, save_admin_config};
use crate::admin::site_replication_state::{SITE_REPLICATION_STATE_PATH, with_site_replication_state_lock_on};
use crate::admin::storage_api::error::Error as StorageError;
use crate::storage::storage_api::{read_config_no_lock, save_config_no_lock};
use rustfs_madmin::PeerInfo;
use s3s::{S3Error, S3ErrorCode, S3Result};
use serde_json::{Map, Value};
use tracing::info;
const SITE_REPLICATION_STATE_PATH: &str = "config/site-replication/state.json";
const SYNC_STATE_INITIALIZED_FIELD: &str = "sync_state_initialized";
fn normalize_peers_map(peers: &Map<String, Value>, initialize_sync_state: bool) -> Map<String, Value> {
@@ -113,25 +113,36 @@ pub async fn reload_site_replication_runtime_state_for_context(context: Option<&
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
match read_admin_config(store.clone(), SITE_REPLICATION_STATE_PATH).await {
Ok(data) => {
if let Some(normalized) =
normalize_site_replication_state_json(&data).map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, e))?
{
save_admin_config(store, SITE_REPLICATION_STATE_PATH, normalized)
.await
.map_err(|e| {
S3Error::with_message(S3ErrorCode::InternalError, format!("normalize site replication state failed: {e}"))
})?;
// The whole read -> normalize -> save is one RMW: run it inside the
// shared state transaction boundary (P1-15) so a cluster-wide reload
// fan-out cannot overwrite a concurrent state writer. IO must be the
// no-lock variants — the boundary already holds the object lock.
let lock_store = store.clone();
with_site_replication_state_lock_on(lock_store, move || async move {
match read_config_no_lock(store.clone(), SITE_REPLICATION_STATE_PATH).await {
Ok(data) => {
if let Some(normalized) = normalize_site_replication_state_json(&data)
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, e))?
{
save_config_no_lock(store, SITE_REPLICATION_STATE_PATH, normalized)
.await
.map_err(|e| {
S3Error::with_message(
S3ErrorCode::InternalError,
format!("normalize site replication state failed: {e}"),
)
})?;
}
Ok(())
}
Ok(())
Err(StorageError::ConfigNotFound) => Ok(()),
Err(err) => Err(S3Error::with_message(
S3ErrorCode::InternalError,
format!("failed to load site replication state: {err}"),
)),
}
Err(StorageError::ConfigNotFound) => Ok(()),
Err(err) => Err(S3Error::with_message(
S3ErrorCode::InternalError,
format!("failed to load site replication state: {err}"),
)),
}
})
.await
}
pub async fn reload_site_replication_runtime_state() -> S3Result<()> {
+110
View File
@@ -0,0 +1,110 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Locking primitive for the site-replication state object (P1-15,
//! rustfs/backlog#1675 B2).
//!
//! `config/site-replication/state.json` is mutated by read-modify-write
//! sequences spread over many call sites: admin handlers, the retry-event
//! writers on every hook broadcast path, and the service-side reload driven
//! over node RPC. Historically only some of them held the process-local
//! mutex and none held a distributed lock across the whole RMW, so
//! concurrent writers overwrote each other (single-process for the unlocked
//! writers, cross-node for everyone).
//!
//! `with_site_replication_state_lock` is the single transaction boundary:
//! it holds the process-local mutex AND the distributed config-object write
//! lock (the pattern proven by the repair state,
//! `update_site_replication_repair_state`) for the duration of the caller's
//! closure. All IO inside the closure must use the `*_no_lock` config
//! helpers — the locked variants would self-deadlock on the same object
//! lock. Do not perform peer network calls or take other config locks
//! inside the closure.
//!
//! The process-local mutex is transitional: call sites still outside this
//! primitive serialize against migrated ones through it. Once every RMW
//! call site goes through here (P1-15 PR2) it will be removed, leaving the
//! object lock as the only mechanism.
//!
//! Lock order (unchanged from the historical comment next to the mutex):
//! lifecycle -> bucket operation -> repair admission -> state (process
//! mutex, then state object lock) -> per-bucket metadata.
use crate::admin::storage_api::runtime::ECStore;
use crate::storage::storage_api::with_config_object_write_lock;
use s3s::{S3Error, S3ErrorCode, S3Result};
use std::sync::Arc;
use super::runtime_sources::current_object_store_handle;
/// Config object holding the whole site-replication state, including the
/// retry-event queue. Shared by the typed handler-side accessors and the
/// byte-level tolerant reload on the service side.
pub(crate) const SITE_REPLICATION_STATE_PATH: &str = "config/site-replication/state.json";
/// Transitional process-local mutex — see the module docs. Stays private to
/// this module (owner-local static, enforced by
/// `scripts/check_architecture_migration_rules.sh`); callers go through
/// [`site_replication_state_process_guard`].
static SITE_REPLICATION_STATE_LOCK: std::sync::LazyLock<tokio::sync::Mutex<()>> =
std::sync::LazyLock::new(|| tokio::sync::Mutex::new(()));
/// Owner helper for the transitional process mutex: the RMW call sites in
/// `handlers::site_replication` that PR2 has not migrated to
/// [`with_site_replication_state_lock`] yet hold this guard so they stay
/// mutually exclusive with the migrated ones. Removed together with the
/// mutex once every call site runs inside the transaction boundary.
pub(crate) async fn site_replication_state_process_guard() -> tokio::sync::MutexGuard<'static, ()> {
SITE_REPLICATION_STATE_LOCK.lock().await
}
/// Run `operation` under the site-replication state transaction boundary:
/// process mutex first, then the distributed state-object write lock.
pub(crate) async fn with_site_replication_state_lock<T, F, Fut>(operation: F) -> S3Result<T>
where
T: Send + 'static,
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = S3Result<T>> + Send + 'static,
{
let store =
current_object_store_handle().ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()))?;
with_site_replication_state_lock_on(store, operation).await
}
/// Context-store variant for callers that resolve their store from an
/// explicit [`AppContext`] (the service-side reload driven over node RPC).
pub(crate) async fn with_site_replication_state_lock_on<T, F, Fut>(store: Arc<ECStore>, operation: F) -> S3Result<T>
where
T: Send + 'static,
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = S3Result<T>> + Send + 'static,
{
let _process_guard = SITE_REPLICATION_STATE_LOCK.lock().await;
with_site_replication_state_object_lock(store, operation).await
}
/// The distributed half of the boundary on its own: the state-object write
/// lock, without the process mutex. This is the only thing that serializes
/// writers in *different* processes (the mutex cannot), so it is also what
/// the separate-nodes regression test drives.
pub(crate) async fn with_site_replication_state_object_lock<T, F, Fut>(store: Arc<ECStore>, operation: F) -> S3Result<T>
where
T: Send + 'static,
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = S3Result<T>> + Send + 'static,
{
with_config_object_write_lock(store, SITE_REPLICATION_STATE_PATH.to_string(), operation)
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("lock site replication state failed: {e}")))?
}
+1
View File
@@ -183,6 +183,7 @@ pub(crate) mod bandwidth {
}
pub(crate) mod bucket_target_sys {
pub(crate) use super::ecstore_bucket::bucket_target_sys::append_version_id_query;
pub(crate) type AdvancedPutOptions = super::ecstore_bucket::bucket_target_sys::AdvancedPutOptions;
pub(crate) type BucketTargetError = super::ecstore_bucket::bucket_target_sys::BucketTargetError;
pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys;
+13
View File
@@ -34,6 +34,7 @@ use super::interfaces::{
ScannerMetricsInterface, ServerConfigInterface, StorageClassInterface, TierConfigInterface, TransitionStateInterface,
};
use crate::app::object_data_cache::ObjectDataCacheAdapter;
use crate::app::object_traffic_health::ObjectTrafficHealth;
use rustfs_iam::{federation::FederatedIdentityService, store::object::ObjectStore, sys::IamSys};
use rustfs_kms::KmsServiceManager;
use std::sync::{Arc, OnceLock};
@@ -74,6 +75,7 @@ pub struct AppContext {
storage_class: Arc<dyn StorageClassInterface>,
buffer_config: Arc<dyn BufferConfigInterface>,
object_data_cache: Arc<ObjectDataCacheAdapter>,
object_traffic_health: Arc<ObjectTrafficHealth>,
}
impl AppContext {
@@ -122,6 +124,7 @@ impl AppContext {
storage_class: default_storage_class_interface(),
buffer_config: default_buffer_config_interface(),
object_data_cache,
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
}
}
@@ -137,6 +140,10 @@ impl AppContext {
self.object_store.clone()
}
pub(crate) fn object_traffic_health(&self) -> Arc<ObjectTrafficHealth> {
Arc::clone(&self.object_traffic_health)
}
pub fn iam(&self) -> Arc<dyn IamInterface> {
self.iam.clone()
}
@@ -342,9 +349,15 @@ impl AppContext {
storage_class: interfaces.storage_class,
buffer_config: interfaces.buffer_config,
object_data_cache: ObjectDataCacheAdapter::disabled_arc(),
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
}
}
pub(crate) fn with_test_object_traffic_health(mut self, object_traffic_health: Arc<ObjectTrafficHealth>) -> Self {
self.object_traffic_health = object_traffic_health;
self
}
pub(crate) fn with_test_runtime_config_interfaces(
mut self,
server_config: Arc<dyn ServerConfigInterface>,
+28
View File
@@ -25,6 +25,7 @@
use super::storage_api::test::bucket::metadata_sys;
use super::storage_api::test::contract::bucket::{BucketOperations, BucketOptions};
use super::storage_api::test::{ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints};
use super::{context::AppContext, object_traffic_health::ObjectTrafficHealth};
use std::path::PathBuf;
use std::sync::{Arc, OnceLock};
use tempfile::TempDir;
@@ -32,6 +33,7 @@ use tokio::fs;
use tokio_util::sync::CancellationToken;
static SHARED_GATING_ENV: OnceLock<(Vec<PathBuf>, Arc<ECStore>, TempDir)> = OnceLock::new();
static SHARED_GATING_INIT: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
/// Return a shared 4-disk `ECStore` with bucket metadata initialized.
///
@@ -42,6 +44,10 @@ pub(crate) async fn shared_gating_ecstore() -> Arc<ECStore> {
if let Some((_paths, store, _)) = SHARED_GATING_ENV.get() {
return store.clone();
}
let _init_guard = SHARED_GATING_INIT.lock().await;
if let Some((_paths, store, _)) = SHARED_GATING_ENV.get() {
return store.clone();
}
let temp_dir = TempDir::new().expect("create temp dir for gating test env");
let temp_path = temp_dir.path().to_path_buf();
@@ -101,6 +107,28 @@ pub(crate) async fn shared_gating_ecstore() -> Arc<ECStore> {
ecstore
}
pub(crate) async fn shared_gating_ambient() -> Arc<AppContext> {
let store = shared_gating_ecstore().await;
if let Some(ambient) = crate::runtime_sources::current_app_context() {
return ambient;
}
let _init_guard = SHARED_GATING_INIT.lock().await;
if crate::runtime_sources::current_app_context().is_none() {
super::runtime_sources::install_test_app_context(Arc::clone(&store)).await;
}
crate::runtime_sources::current_app_context().expect("object traffic test context must be installed")
}
pub(crate) fn app_context_from_current_environment(ambient: &AppContext) -> AppContext {
AppContext::new(ambient.object_store(), ambient.iam(), ambient.kms())
}
pub(crate) async fn app_context_with_object_traffic_health(object_traffic_health: Arc<ObjectTrafficHealth>) -> Arc<AppContext> {
let ambient = shared_gating_ambient().await;
Arc::new(app_context_from_current_environment(&ambient).with_test_object_traffic_health(object_traffic_health))
}
/// Like [`shared_gating_ecstore`], but also returns the backing disk paths so
/// tests can remove on-disk shards and simulate the object data vanishing
/// mid-stream.
+1
View File
@@ -21,6 +21,7 @@ pub mod context;
pub(crate) mod metadata_route;
pub mod multipart_usecase;
pub(crate) mod object_data_cache;
pub(crate) mod object_traffic_health;
pub mod object_usecase;
pub(crate) mod runtime_sources;
mod select_object;
+335
View File
@@ -0,0 +1,335 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub(crate) struct ObjectTrafficSnapshot {
pub(crate) read_stalled: bool,
pub(crate) write_stalled: bool,
}
/// Detects bounded object stages that stop returning from foreground requests.
/// Both success and error returns are progress: dependency correctness remains
/// the responsibility of the existing readiness checks.
#[derive(Debug)]
pub(crate) struct ObjectTrafficHealth {
started_at: Instant,
stall_after_ms: u64,
enabled: bool,
read_metadata: OperationProgress,
read_storage: OperationProgress,
write_storage: OperationProgress,
}
impl ObjectTrafficHealth {
pub(crate) fn from_env() -> Self {
let enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_ENABLE,
rustfs_config::DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE,
);
let configured_timeout_ms = rustfs_utils::get_env_u64(
rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS,
rustfs_config::DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS,
);
let requested_timeout_ms = if configured_timeout_ms == 0 {
rustfs_config::DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS
} else {
configured_timeout_ms
};
let minimum_timeout_ms = duration_ms_saturating(crate::storage::get_lock_acquire_timeout())
.saturating_add(rustfs_config::HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS);
let stall_after_ms = requested_timeout_ms.max(minimum_timeout_ms);
Self::new(enabled, stall_after_ms)
}
fn new(enabled: bool, stall_after_ms: u64) -> Self {
Self {
started_at: Instant::now(),
stall_after_ms,
enabled,
read_metadata: OperationProgress::default(),
read_storage: OperationProgress::default(),
write_storage: OperationProgress::default(),
}
}
pub(crate) fn track_read_metadata(&self) -> Option<ObjectTrafficProgressGuard<'_>> {
self.track(&self.read_metadata)
}
pub(crate) fn track_read_storage(&self) -> Option<ObjectTrafficProgressGuard<'_>> {
self.track(&self.read_storage)
}
pub(crate) fn track_write_storage(&self) -> Option<ObjectTrafficProgressGuard<'_>> {
self.track(&self.write_storage)
}
pub(crate) fn snapshot(&self) -> ObjectTrafficSnapshot {
if !self.enabled {
return ObjectTrafficSnapshot::default();
}
let now_ms = self.now_ms();
ObjectTrafficSnapshot {
read_stalled: self.read_metadata.is_stalled_at(now_ms, self.stall_after_ms)
|| self.read_storage.is_stalled_at(now_ms, self.stall_after_ms),
write_stalled: self.write_storage.is_stalled_at(now_ms, self.stall_after_ms),
}
}
fn track<'a>(&'a self, progress: &'a OperationProgress) -> Option<ObjectTrafficProgressGuard<'a>> {
if !self.enabled || !progress.begin_at(self.now_ms()) {
return None;
}
Some(ObjectTrafficProgressGuard { health: self, progress })
}
fn now_ms(&self) -> u64 {
duration_ms_saturating(self.started_at.elapsed())
}
#[cfg(test)]
pub(crate) fn enabled_for_test(stall_after: Duration) -> Self {
Self::new(true, duration_ms_saturating(stall_after))
}
#[cfg(test)]
pub(crate) fn read_storage_stalled_for_test(&self) -> bool {
self.read_storage.is_stalled_at(self.now_ms(), self.stall_after_ms)
}
}
#[derive(Debug, Default)]
struct OperationProgress {
active: AtomicU64,
last_progress_ms: AtomicU64,
}
impl OperationProgress {
fn begin_at(&self, now_ms: u64) -> bool {
let mut active = self.active.load(Ordering::Relaxed);
loop {
let Some(next) = active.checked_add(1) else {
return false;
};
if active == 0 {
self.last_progress_ms.fetch_max(now_ms, Ordering::Relaxed);
}
match self
.active
.compare_exchange_weak(active, next, Ordering::Release, Ordering::Relaxed)
{
Ok(_) => return true,
Err(observed) => active = observed,
}
}
}
fn complete_at(&self, now_ms: u64) {
self.last_progress_ms.fetch_max(now_ms, Ordering::Relaxed);
let previous = self.active.fetch_sub(1, Ordering::Release);
debug_assert!(previous > 0, "object traffic progress guard underflow");
}
fn is_stalled_at(&self, now_ms: u64, stall_after_ms: u64) -> bool {
self.active.load(Ordering::Acquire) > 0
&& now_ms.saturating_sub(self.last_progress_ms.load(Ordering::Relaxed)) >= stall_after_ms
}
}
#[must_use = "dropping the guard records operation completion"]
pub(crate) struct ObjectTrafficProgressGuard<'a> {
health: &'a ObjectTrafficHealth,
progress: &'a OperationProgress,
}
impl Drop for ObjectTrafficProgressGuard<'_> {
fn drop(&mut self) {
self.progress.complete_at(self.health.now_ms());
}
}
fn duration_ms_saturating(duration: Duration) -> u64 {
duration
.as_secs()
.saturating_mul(1_000)
.saturating_add(u64::from(duration.subsec_millis()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_active_operation_stalls_at_the_exact_boundary() {
let progress = OperationProgress::default();
assert!(progress.begin_at(10));
assert!(!progress.is_stalled_at(39, 30));
assert!(progress.is_stalled_at(40, 30));
}
#[test]
fn later_arrivals_do_not_hide_an_existing_stall() {
let progress = OperationProgress::default();
assert!(progress.begin_at(10));
assert!(progress.begin_at(35));
assert!(progress.is_stalled_at(40, 30));
}
#[test]
fn a_completion_resets_progress_until_the_remaining_operation_stalls() {
let progress = OperationProgress::default();
assert!(progress.begin_at(10));
assert!(progress.begin_at(20));
progress.complete_at(35);
assert!(!progress.is_stalled_at(64, 30));
assert!(progress.is_stalled_at(65, 30));
progress.complete_at(65);
assert!(!progress.is_stalled_at(u64::MAX, 30));
}
#[test]
fn a_stale_begin_timestamp_cannot_overwrite_newer_progress() {
let progress = OperationProgress::default();
assert!(progress.begin_at(10));
progress.complete_at(100);
assert!(progress.begin_at(10));
assert!(!progress.is_stalled_at(129, 30));
assert!(progress.is_stalled_at(130, 30));
}
#[test]
#[serial_test::serial]
fn environment_configuration_is_sanitized() {
temp_env::with_vars(
[
(rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_ENABLE, Some("false")),
(rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS, Some("1")),
],
|| {
let minimum_timeout_ms = duration_ms_saturating(crate::storage::get_lock_acquire_timeout())
.saturating_add(rustfs_config::HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS);
let health = ObjectTrafficHealth::from_env();
assert!(!health.enabled);
assert_eq!(health.stall_after_ms, minimum_timeout_ms);
},
);
temp_env::with_vars([(rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS, Some("0"))], || {
let minimum_timeout_ms = duration_ms_saturating(crate::storage::get_lock_acquire_timeout())
.saturating_add(rustfs_config::HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS);
let health = ObjectTrafficHealth::from_env();
assert_eq!(
health.stall_after_ms,
rustfs_config::DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS.max(minimum_timeout_ms)
);
});
}
#[test]
fn read_and_write_progress_are_independent() {
let health = ObjectTrafficHealth::enabled_for_test(Duration::ZERO);
let read = health.track_read_storage().expect("read tracking must be enabled");
assert_eq!(
health.snapshot(),
ObjectTrafficSnapshot {
read_stalled: true,
write_stalled: false,
}
);
drop(read);
let write = health.track_write_storage().expect("write tracking must be enabled");
assert_eq!(
health.snapshot(),
ObjectTrafficSnapshot {
read_stalled: false,
write_stalled: true,
}
);
drop(write);
assert_eq!(health.snapshot(), ObjectTrafficSnapshot::default());
}
#[test]
fn disabled_tracking_never_withdraws_readiness() {
let health = ObjectTrafficHealth::new(false, 0);
assert!(health.track_read_metadata().is_none());
assert!(health.track_read_storage().is_none());
assert!(health.track_write_storage().is_none());
assert_eq!(health.snapshot(), ObjectTrafficSnapshot::default());
}
#[test]
fn metadata_completions_do_not_hide_a_storage_stall() {
let health = ObjectTrafficHealth::enabled_for_test(Duration::ZERO);
let storage = health.track_read_storage().expect("read storage tracking must be enabled");
let metadata = health.track_read_metadata().expect("read metadata tracking must be enabled");
drop(metadata);
assert!(health.snapshot().read_stalled);
drop(storage);
}
#[tokio::test]
async fn aborting_a_tracked_future_clears_the_active_operation() {
let health = std::sync::Arc::new(ObjectTrafficHealth::enabled_for_test(Duration::ZERO));
let task_health = std::sync::Arc::clone(&health);
let task = tokio::spawn(async move {
let _progress = task_health.track_read_storage().expect("read tracking must be enabled");
std::future::pending::<()>().await;
});
if tokio::time::timeout(Duration::from_secs(2), async {
while !health.snapshot().read_stalled {
tokio::task::yield_now().await;
}
})
.await
.is_err()
{
task.abort();
let _ = task.await;
panic!("tracked future did not publish an active operation");
}
task.abort();
assert!(task.await.expect_err("tracked task must be cancelled").is_cancelled());
assert!(!health.snapshot().read_stalled);
}
#[tokio::test]
#[serial_test::serial]
async fn app_context_honors_the_disabled_progress_environment() {
let ambient = crate::app::gating_test_env::shared_gating_ambient().await;
temp_env::async_with_vars([(rustfs_config::ENV_HEALTH_OBJECT_PROGRESS_ENABLE, Some("false"))], async {
let context = crate::app::gating_test_env::app_context_from_current_environment(&ambient);
assert!(context.object_traffic_health().track_read_storage().is_none());
})
.await;
let installed = crate::app::runtime_sources::current_app_context().expect("test AppContext must remain installed");
assert!(std::sync::Arc::ptr_eq(&ambient, &installed));
}
}
+385 -31
View File
@@ -222,6 +222,7 @@ use crate::app::object_data_cache::{
};
#[cfg(test)]
use crate::app::object_data_cache::{ColdFillRole, ColdFillWaitOutcome, scope_cold_fill_disk_permit_owner_for_test};
use crate::app::object_traffic_health::ObjectTrafficHealth;
type S3StdError = Box<dyn std::error::Error + Send + Sync + 'static>;
@@ -2951,6 +2952,8 @@ fn normalize_delete_objects_version_id(
#[cfg(test)]
type DeleteSnapshotTestHook = (String, Arc<tokio::sync::Barrier>, Arc<tokio::sync::Barrier>);
#[cfg(test)]
type PutPostStoreTestHook = (String, Arc<tokio::sync::Barrier>, Arc<tokio::sync::Barrier>);
#[cfg(test)]
static DELETE_SNAPSHOT_TEST_HOOK: OnceLock<Mutex<Option<DeleteSnapshotTestHook>>> = OnceLock::new();
@@ -2958,6 +2961,8 @@ static DELETE_SNAPSHOT_TEST_HOOK: OnceLock<Mutex<Option<DeleteSnapshotTestHook>>
static DELETE_SOURCE_TEST_HOOK: OnceLock<Mutex<Option<DeleteSnapshotTestHook>>> = OnceLock::new();
#[cfg(test)]
static DELETE_OBJECTS_AUTH_TEST_HOOK: OnceLock<Mutex<Option<DeleteSnapshotTestHook>>> = OnceLock::new();
#[cfg(test)]
static PUT_POST_STORE_TEST_HOOK: OnceLock<Mutex<Option<PutPostStoreTestHook>>> = OnceLock::new();
#[cfg(test)]
pub(crate) fn install_delete_snapshot_test_hook(
@@ -3052,6 +3057,33 @@ async fn wait_for_delete_objects_auth_test_hook(bucket: &str) {
}
}
#[cfg(test)]
fn install_put_post_store_test_hook(bucket: String, entered: Arc<tokio::sync::Barrier>, resume: Arc<tokio::sync::Barrier>) {
*PUT_POST_STORE_TEST_HOOK
.get_or_init(|| Mutex::new(None))
.lock()
.expect("PUT post-store test hook lock should not be poisoned") = Some((bucket, entered, resume));
}
#[cfg(test)]
async fn wait_for_put_post_store_test_hook(bucket: &str) {
let hook = {
let mut slot = PUT_POST_STORE_TEST_HOOK
.get_or_init(|| Mutex::new(None))
.lock()
.expect("PUT post-store test hook lock should not be poisoned");
if slot.as_ref().is_some_and(|(expected_bucket, _, _)| expected_bucket == bucket) {
slot.take()
} else {
None
}
};
if let Some((_bucket, entered, resume)) = hook {
entered.wait().await;
resume.wait().await;
}
}
fn build_put_object_expiration_header(event: &lifecycle::Event) -> Option<String> {
if !event.action.delete() {
return None;
@@ -3895,6 +3927,14 @@ pub struct DefaultObjectUsecase {
get_object_timeout_policy: Option<GetObjectTimeoutPolicy>,
}
async fn track_object_read_setup<F>(health: Option<&ObjectTrafficHealth>, future: F) -> F::Output
where
F: std::future::Future,
{
let _progress = health.and_then(ObjectTrafficHealth::track_read_storage);
future.await
}
impl DefaultObjectUsecase {
fn should_use_large_put_concurrency_tuning(size: i64) -> bool {
size >= DEFAULT_PUT_LARGE_CONCURRENCY_TUNING_MIN_SIZE_BYTES
@@ -3951,6 +3991,13 @@ impl DefaultObjectUsecase {
current_object_data_cache_for_context(self.context.as_deref())
}
fn object_traffic_health(&self) -> Option<Arc<ObjectTrafficHealth>> {
self.context
.as_ref()
.map(|context| context.object_traffic_health())
.or_else(|| current_app_context().map(|context| context.object_traffic_health()))
}
fn base_buffer_size(&self) -> usize {
self.context
.clone()
@@ -4351,6 +4398,7 @@ impl DefaultObjectUsecase {
rs: Option<HTTPRangeSpec>,
opts: &ObjectOptions,
part_number: Option<usize>,
object_traffic_health: Option<Arc<ObjectTrafficHealth>>,
) -> S3Result<GetObjectPreparedRead> {
let read_start = std::time::Instant::now();
let read_stage_start = rustfs_io_metrics::get_stage_metrics_enabled().then_some(read_start);
@@ -4366,10 +4414,12 @@ impl DefaultObjectUsecase {
key,
)
.await?;
let reader = store
.get_object_reader(bucket, key, rs.clone(), req.headers.clone(), opts)
.await
.map_err(map_get_object_reader_error)?;
let reader = track_object_read_setup(
object_traffic_health.as_deref(),
store.get_object_reader(bucket, key, rs.clone(), req.headers.clone(), opts),
)
.await
.map_err(map_get_object_reader_error)?;
let read_setup =
Self::finish_get_object_read(req, manager, bucket, key, rs, part_number, read_start, reader, true).await?;
return Ok(GetObjectPreparedRead { io_planning, read_setup });
@@ -4390,10 +4440,12 @@ impl DefaultObjectUsecase {
.await?,
);
let mut prepared = Some(
store
.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts)
.await
.map_err(map_get_object_reader_error)?,
track_object_read_setup(
object_traffic_health.as_deref(),
store.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts),
)
.await
.map_err(map_get_object_reader_error)?,
);
let mut cache_fill_allowed = true;
let mut legacy_hook_missed = false;
@@ -4494,6 +4546,7 @@ impl DefaultObjectUsecase {
let headers = &req.headers;
let store = &store;
let range = &rs;
let object_traffic_health = &object_traffic_health;
move |producer| {
let adapter = Arc::clone(adapter);
let engine_plan = engine_plan.clone();
@@ -4503,6 +4556,7 @@ impl DefaultObjectUsecase {
let bucket = bucket.to_owned();
let key = key.to_owned();
let opts = opts.clone();
let object_traffic_health = object_traffic_health.as_ref().map(Arc::clone);
async move {
let producer_deadline = producer.deadline();
let cancellation = producer.cancellation_token();
@@ -4548,7 +4602,10 @@ impl DefaultObjectUsecase {
}
};
let prepare = store.prepare_get_object_reader(&bucket, &key, range.clone(), HeaderMap::new(), &opts);
let prepare = track_object_read_setup(
object_traffic_health.as_deref(),
store.prepare_get_object_reader(&bucket, &key, range.clone(), HeaderMap::new(), &opts),
);
let prepared = match match await_cold_fill_startup(prepare, &cancellation, producer_deadline).await {
Ok(result) => result,
Err(ColdFillStartupWaitError::Cancelled) => {
@@ -4602,7 +4659,8 @@ impl DefaultObjectUsecase {
|| {
#[cfg(test)]
record_cold_fill_reader_open_for_test(&reader_open_plan);
prepared.with_headers(h).into_reader()
let open_reader = prepared.with_headers(h).into_reader();
async move { track_object_read_setup(object_traffic_health.as_deref(), open_reader).await }
},
ColdFillProducerExecution {
expected,
@@ -4647,11 +4705,12 @@ impl DefaultObjectUsecase {
let io_planning = metadata_admission
.take()
.ok_or_else(|| s3_error!(InternalError, "prepared metadata admission is unavailable"))?;
let reader = prepared
.with_headers(req.headers.clone())
.into_reader()
.await
.map_err(map_get_object_reader_error)?;
let reader = track_object_read_setup(
object_traffic_health.as_deref(),
prepared.with_headers(req.headers.clone()).into_reader(),
)
.await
.map_err(map_get_object_reader_error)?;
(io_planning, reader)
} else {
let io_planning = Self::acquire_get_object_io_planning(
@@ -4665,19 +4724,25 @@ impl DefaultObjectUsecase {
)
.await?;
let reader = if legacy_hook_missed {
store
.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts)
.await
.map_err(map_get_object_reader_error)?
.with_headers(req.headers.clone())
.into_reader()
.await
.map_err(map_get_object_reader_error)?
let prepared = track_object_read_setup(
object_traffic_health.as_deref(),
store.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts),
)
.await
.map_err(map_get_object_reader_error)?;
track_object_read_setup(
object_traffic_health.as_deref(),
prepared.with_headers(req.headers.clone()).into_reader(),
)
.await
.map_err(map_get_object_reader_error)?
} else {
store
.get_object_reader(bucket, key, rs.clone(), req.headers.clone(), opts)
.await
.map_err(map_get_object_reader_error)?
track_object_read_setup(
object_traffic_health.as_deref(),
store.get_object_reader(bucket, key, rs.clone(), req.headers.clone(), opts),
)
.await
.map_err(map_get_object_reader_error)?
};
(io_planning, reader)
};
@@ -5485,8 +5550,8 @@ impl DefaultObjectUsecase {
debug!("Zero-copy write enabled for {} byte object (bucket={}, key={})", size, bucket, key);
}
let use_small_eager_put_path =
should_use_small_eager_put_path(size, &req.headers, server_side_encryption_requested, should_compress, false);
let use_empty_or_small_eager_put_path = size == 0
|| should_use_small_eager_put_path(size, &req.headers, server_side_encryption_requested, should_compress, false);
let zero_copy_eager_put_path_status =
zero_copy_eager_put_path_status(size, &req.headers, server_side_encryption_requested, should_compress, false);
let use_zero_copy_eager_put_path = zero_copy_eager_put_path_status == PUT_EAGER_STATUS_ELIGIBLE;
@@ -5498,7 +5563,7 @@ impl DefaultObjectUsecase {
"stream_compressed"
} else if use_zero_copy_eager_put_path {
"zero_copy_eager"
} else if use_small_eager_put_path {
} else if use_empty_or_small_eager_put_path {
"small_eager"
} else {
"streaming"
@@ -5712,7 +5777,7 @@ impl DefaultObjectUsecase {
let eager_body = read_zero_copy_put_body_exact(body, actual_size as usize).await?;
rustfs_io_metrics::record_zero_copy_write(actual_size as usize, zero_copy_start.elapsed().as_secs_f64() * 1000.0);
HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?
} else if use_small_eager_put_path {
} else if use_empty_or_small_eager_put_path {
if (actual_size as usize) <= POOL_BYPASS_MAX_SIZE {
// Bypass BytesPool for very small objects to avoid Small-tier
// Mutex contention under high concurrency. Direct allocation
@@ -5866,6 +5931,14 @@ impl DefaultObjectUsecase {
}
});
let object_traffic_health = if use_zero_copy_eager_put_path || use_empty_or_small_eager_put_path {
self.object_traffic_health()
} else {
None
};
let object_traffic_progress = object_traffic_health
.as_deref()
.and_then(ObjectTrafficHealth::track_write_storage);
let (obj_info, backfilled_old_current_size) = match store
.put_object_with_old_current_size(&bucket, &key, &mut reader, &opts)
.await
@@ -5912,6 +5985,9 @@ impl DefaultObjectUsecase {
return result;
}
};
drop(object_traffic_progress);
#[cfg(test)]
wait_for_put_post_store_test_hook(&bucket).await;
maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await;
let _ = invalidate_object_data_cache_after_put_success(&cache_adapter, &bucket, &key).await;
@@ -6352,6 +6428,10 @@ impl DefaultObjectUsecase {
// naming nonexistent buckets fail before the versioning lookup in
// get_opts. The store comes from the request-bound server context
// (backlog#1052 S6), not the process-global handle.
let object_traffic_health = self.object_traffic_health();
let object_metadata_progress = object_traffic_health
.as_deref()
.and_then(ObjectTrafficHealth::track_read_metadata);
let store_lookup_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
let Some(store) = self.object_store() else {
lifecycle.finish_err();
@@ -6392,6 +6472,7 @@ impl DefaultObjectUsecase {
rs,
opts,
} = request_context;
drop(object_metadata_progress);
let manager = get_concurrency_manager();
@@ -6407,6 +6488,7 @@ impl DefaultObjectUsecase {
rs,
&opts,
part_number,
object_traffic_health,
)
.await
{
@@ -11142,6 +11224,278 @@ mod tests {
(store, context)
}
#[tokio::test]
#[serial_test::serial(body_cache_hook)]
async fn object_progress_tracks_real_get_and_small_put_lock_waits() {
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
let object_traffic_health = Arc::new(ObjectTrafficHealth::enabled_for_test(Duration::ZERO));
let context = temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_DATA_CACHE_ENABLE, Some("false"))], async {
crate::app::gating_test_env::app_context_with_object_traffic_health(Arc::clone(&object_traffic_health)).await
})
.await;
let store = context.object_store();
let bucket = format!("object-progress-{}", Uuid::new_v4());
let object = "object.bin";
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("object progress bucket must be created");
put_real_cold_fill_object(&store, &bucket, object, b"initial").await;
let metadata_entered = Arc::new(tokio::sync::Barrier::new(2));
let metadata_resume = Arc::new(tokio::sync::Barrier::new(2));
crate::storage::options::install_versioning_config_test_hook(
bucket.clone(),
Arc::clone(&metadata_entered),
Arc::clone(&metadata_resume),
);
let metadata_input = GetObjectInput::builder()
.bucket(bucket.clone())
.key(object.to_string())
.build()
.expect("metadata GET input must build");
let metadata_usecase = DefaultObjectUsecase::with_context(Some(Arc::clone(&context)));
let metadata_get = tokio::spawn(async move {
metadata_usecase
.execute_get_object(build_request(metadata_input, Method::GET))
.await
});
tokio::time::timeout(Duration::from_secs(2), metadata_entered.wait())
.await
.expect("GET must enter the bucket metadata stage");
assert!(object_traffic_health.snapshot().read_stalled);
assert!(!metadata_get.is_finished(), "GET must still be waiting in bucket metadata");
metadata_resume.wait().await;
let metadata_response = tokio::time::timeout(Duration::from_secs(10), metadata_get)
.await
.expect("metadata GET must finish after release")
.expect("metadata GET task must join")
.expect("metadata GET must succeed after release");
assert!(!object_traffic_health.snapshot().read_stalled);
drop(metadata_response);
let read_lock = store
.new_ns_lock(&bucket, object)
.await
.expect("read test namespace lock must be created")
.get_write_lock(Duration::from_secs(5))
.await
.expect("read test namespace lock must be held");
let get_input = GetObjectInput::builder()
.bucket(bucket.clone())
.key(object.to_string())
.build()
.expect("GET input must build");
let get_usecase = DefaultObjectUsecase::with_context(Some(Arc::clone(&context)));
let get = tokio::spawn(async move { get_usecase.execute_get_object(build_request(get_input, Method::GET)).await });
tokio::time::timeout(Duration::from_secs(2), async {
while !object_traffic_health.read_storage_stalled_for_test() {
tokio::task::yield_now().await;
}
})
.await
.expect("blocked GET must publish a storage stall");
assert!(!get.is_finished(), "GET must still be waiting for the held namespace lock");
drop(read_lock);
let get_response = tokio::time::timeout(Duration::from_secs(10), get)
.await
.expect("GET must finish after releasing the lock")
.expect("GET task must join")
.expect("GET must succeed after releasing the lock");
assert!(!object_traffic_health.snapshot().read_stalled);
drop(get_response);
let write_lock = store
.new_ns_lock(&bucket, object)
.await
.expect("write test namespace lock must be created")
.get_write_lock(Duration::from_secs(5))
.await
.expect("write test namespace lock must be held");
let post_store_entered = Arc::new(tokio::sync::Barrier::new(2));
let post_store_resume = Arc::new(tokio::sync::Barrier::new(2));
install_put_post_store_test_hook(bucket.clone(), Arc::clone(&post_store_entered), Arc::clone(&post_store_resume));
let payload = Bytes::from_static(b"replacement");
let put_input = PutObjectInput::builder()
.bucket(bucket)
.key(object.to_string())
.body(Some(StreamingBlob::from(s3s::Body::from(payload.clone()))))
.content_length(Some(i64::try_from(payload.len()).expect("test payload length must fit i64")))
.build()
.expect("PUT input must build");
let put_usecase = DefaultObjectUsecase::with_context(Some(context));
let put = tokio::spawn(async move {
put_usecase
.execute_put_object(&FS::new(), build_request(put_input, Method::PUT))
.await
});
tokio::time::timeout(Duration::from_secs(2), async {
while !object_traffic_health.snapshot().write_stalled {
tokio::task::yield_now().await;
}
})
.await
.expect("blocked small PUT must publish a storage stall");
assert!(!put.is_finished(), "PUT must still be waiting for the held namespace lock");
drop(write_lock);
tokio::time::timeout(Duration::from_secs(10), post_store_entered.wait())
.await
.expect("PUT must reach the first post-store hook");
assert!(!object_traffic_health.snapshot().write_stalled);
assert!(!put.is_finished(), "PUT must remain blocked after the store guard has ended");
post_store_resume.wait().await;
tokio::time::timeout(Duration::from_secs(10), put)
.await
.expect("PUT must finish after releasing the lock")
.expect("PUT task must join")
.expect("PUT must succeed after releasing the lock");
let recovered = object_traffic_health.snapshot();
assert!(!recovered.read_stalled);
assert!(!recovered.write_stalled);
}
#[tokio::test]
async fn object_progress_tracks_zero_byte_and_zero_copy_put_lock_waits() {
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
let object_traffic_health = Arc::new(ObjectTrafficHealth::enabled_for_test(Duration::ZERO));
let context =
crate::app::gating_test_env::app_context_with_object_traffic_health(Arc::clone(&object_traffic_health)).await;
let store = context.object_store();
let bucket = format!("progress-buffered-{}", Uuid::new_v4());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("buffered PUT progress bucket must be created");
let extra_body_object = "zero-byte-extra.bin";
let extra_body_input = PutObjectInput::builder()
.bucket(bucket.clone())
.key(extra_body_object.to_string())
.body(Some(StreamingBlob::from(s3s::Body::from(Bytes::from_static(b"x")))))
.content_length(Some(88))
.build()
.expect("zero-byte extra-body PUT input must build");
let extra_body_usecase = DefaultObjectUsecase::with_context(Some(Arc::clone(&context)));
let mut extra_body_request = build_request(extra_body_input, Method::PUT);
extra_body_request.headers = streaming_headers(Some("0"));
let extra_body_err = extra_body_usecase
.execute_put_object(&FS::new(), extra_body_request)
.await
.expect_err("decoded zero-byte PUT with body data must fail");
assert_eq!(extra_body_err.code(), &S3ErrorCode::UnexpectedContent);
assert!(!object_traffic_health.snapshot().write_stalled);
let lookup_err = store
.get_object_info(&bucket, extra_body_object, &ObjectOptions::default())
.await
.expect_err("rejected zero-byte PUT must not create an object");
assert!(is_err_object_not_found(&lookup_err));
let zero_object = "zero-byte.bin";
let zero_write_lock = store
.new_ns_lock(&bucket, zero_object)
.await
.expect("zero-byte PUT namespace lock must be created")
.get_write_lock(Duration::from_secs(30))
.await
.expect("zero-byte PUT namespace lock must be held");
let (body_polled_tx, body_polled_rx) = tokio::sync::oneshot::channel();
let (body_release_tx, body_release_rx) = tokio::sync::oneshot::channel();
let pending_zero_body = StreamingBlob::wrap(futures::stream::once(async move {
body_polled_tx.send(()).expect("zero-byte body poll signal must be received");
body_release_rx.await.expect("zero-byte body EOF must be released");
Ok::<Bytes, std::io::Error>(Bytes::new())
}));
let zero_input = PutObjectInput::builder()
.bucket(bucket.clone())
.key(zero_object.to_string())
.body(Some(pending_zero_body))
.content_length(Some(87))
.build()
.expect("zero-byte PUT input must build");
let zero_usecase = DefaultObjectUsecase::with_context(Some(Arc::clone(&context)));
let mut zero_request = build_request(zero_input, Method::PUT);
zero_request.headers = streaming_headers(Some("0"));
let zero_put = tokio::spawn(async move { zero_usecase.execute_put_object(&FS::new(), zero_request).await });
tokio::time::timeout(Duration::from_secs(30), body_polled_rx)
.await
.expect("zero-byte PUT body must be polled for EOF")
.expect("zero-byte PUT body poll signal must be sent");
assert!(!object_traffic_health.snapshot().write_stalled);
assert!(!zero_put.is_finished(), "zero-byte PUT must still be waiting for request EOF");
body_release_tx.send(()).expect("zero-byte PUT body EOF must be released");
tokio::time::timeout(Duration::from_secs(30), async {
while !object_traffic_health.snapshot().write_stalled {
tokio::task::yield_now().await;
}
})
.await
.expect("fully received zero-byte PUT must publish a storage stall");
assert!(!zero_put.is_finished(), "zero-byte PUT must still be waiting for the held namespace lock");
drop(zero_write_lock);
tokio::time::timeout(Duration::from_secs(30), zero_put)
.await
.expect("zero-byte PUT must finish after releasing the lock")
.expect("zero-byte PUT task must join")
.expect("zero-byte PUT must succeed after releasing the lock");
assert!(!object_traffic_health.snapshot().write_stalled);
let zero_copy_object = "zero-copy-eager.jpg";
let zero_copy_payload = Bytes::from(vec![b'z'; 1024 * 1024 + 1]);
let zero_copy_size = i64::try_from(zero_copy_payload.len()).expect("zero-copy payload length must fit i64");
let zero_copy_headers = HeaderMap::new();
assert!(!is_disk_compressible(&zero_copy_headers, zero_copy_object));
assert_eq!(
zero_copy_eager_put_path_status(zero_copy_size, &zero_copy_headers, false, false, false),
PUT_EAGER_STATUS_ELIGIBLE,
"test payload must exercise the production zero-copy eager path",
);
let zero_copy_write_lock = store
.new_ns_lock(&bucket, zero_copy_object)
.await
.expect("zero-copy PUT namespace lock must be created")
.get_write_lock(Duration::from_secs(30))
.await
.expect("zero-copy PUT namespace lock must be held");
let zero_copy_input = PutObjectInput::builder()
.bucket(bucket)
.key(zero_copy_object.to_string())
.body(Some(StreamingBlob::from(s3s::Body::from(zero_copy_payload))))
.content_length(Some(zero_copy_size))
.build()
.expect("zero-copy PUT input must build");
let zero_copy_usecase = DefaultObjectUsecase::with_context(Some(context));
let zero_copy_put = tokio::spawn(async move {
zero_copy_usecase
.execute_put_object(&FS::new(), build_request(zero_copy_input, Method::PUT))
.await
});
tokio::time::timeout(Duration::from_secs(30), async {
while !object_traffic_health.snapshot().write_stalled {
tokio::task::yield_now().await;
}
})
.await
.expect("blocked zero-copy eager PUT must publish a storage stall");
assert!(
!zero_copy_put.is_finished(),
"zero-copy PUT must still be waiting for the held namespace lock"
);
drop(zero_copy_write_lock);
tokio::time::timeout(Duration::from_secs(30), zero_copy_put)
.await
.expect("zero-copy PUT must finish after releasing the lock")
.expect("zero-copy PUT task must join")
.expect("zero-copy PUT must succeed after releasing the lock");
assert!(!object_traffic_health.snapshot().write_stalled);
}
async fn put_real_cold_fill_object(store: &Arc<ECStore>, bucket: &str, object: &str, body: &[u8]) -> ObjectInfo {
let mut reader = PutObjReader::from_vec(body.to_vec());
store
+482 -211
View File
@@ -15,12 +15,13 @@ use datafusion::arrow::{
json::{WriterBuilder as JsonWriterBuilder, writer::LineDelimited},
record_batch::RecordBatch,
};
#[cfg(test)]
use datafusion::common::DataFusionError;
use datafusion::physical_plan::SendableRecordBatchStream;
use futures::StreamExt;
use http::{HeaderMap, HeaderName, HeaderValue, StatusCode, header::RANGE};
use rustfs_s3select_api::{
QueryError, S3SelectPolicyError,
QueryError, SelectError,
object_store::{INVALID_SCAN_RANGE_MESSAGE, validate_scan_range_bounds},
query::{Context, Query},
};
@@ -49,8 +50,13 @@ use tracing::info;
const MAX_SELECT_EXPRESSION_BYTES: usize = 256 * 1024;
const RECORDS_CHUNK_TARGET: usize = 128 * 1024;
const DATA_SOURCE_PATH_UNSUPPORTED_CODE: &str = "DataSourcePathUnsupported";
const INVALID_QUERY_CODE: &str = "InvalidQuery";
const PARSE_SELECT_FAILURE_CODE: &str = "ParseSelectFailure";
const BUSY_MESSAGE: &str = "The service is unavailable. Try again later.";
const EMPTY_SELECT_EXPRESSION_MESSAGE: &str = "empty SQL expression";
const SLOW_DOWN_MESSAGE: &str = "Reduce your request rate.";
const UNSUPPORTED_SQL_STRUCTURE_MESSAGE: &str = "We encountered an unsupported SQL structure. Check the SQL Reference.";
const SELECT_MINIO_SSEC_SEALED_KEY: &str = "X-Minio-Internal-Server-Side-Encryption-Sealed-Key";
const SELECT_MINIO_S3_SEALED_KEY: &str = "X-Minio-Internal-Server-Side-Encryption-S3-Sealed-Key";
const SELECT_MINIO_KMS_SEALED_KEY: &str = "X-Minio-Internal-Server-Side-Encryption-Kms-Sealed-Key";
@@ -82,12 +88,7 @@ trait SelectSnapshotFence {
impl SelectSnapshotFence for Arc<StorageSelectObjectSnapshot> {
fn ensure_snapshot_valid(&self) -> S3Result<()> {
self.ensure_valid().map_err(|error| {
let message = error.to_string();
let mut s3_error = S3Error::with_message(S3ErrorCode::InternalError, message);
s3_error.set_source(Box::new(error));
s3_error
})
self.ensure_valid().map_err(internal_select_error)
}
}
@@ -128,7 +129,7 @@ pub async fn execute_select_object_content(
let terminal_permit = tx
.clone()
.try_reserve_owned()
.map_err(|_| s3_error!(InternalError, "can't reserve Select terminal event capacity"))?;
.map_err(|_| map_select_error_to_s3(&SelectError::InternalError))?;
let response = select_object_response(rx, &snapshot.object_info().user_defined, &req.headers)?;
spawn_traced(async move {
send_select_events_until_deadline(
@@ -338,7 +339,7 @@ async fn send_select_events_until_deadline<L: SelectSnapshotFence>(
let outcome = match timeout_at(deadline, send_select_events(output, &tx, validation, &snapshot_lease)).await {
Ok(outcome) => outcome,
Err(_) => SelectProducerOutcome::Terminal(Err(map_query_error_to_s3(
S3SelectPolicyError::QueryTimeout {
SelectError::QueryTimeout {
seconds: timeout_seconds,
}
.into(),
@@ -416,12 +417,14 @@ async fn send_select_events(
let stats = SelectObjectContentEvent::Stats(StatsEvent {
details: Some(progress.to_stats()),
});
if tx.send(Ok(stats)).await.is_err() {
return SelectProducerOutcome::ReceiverClosed;
}
let stats_permit = match tx.reserve().await {
Ok(permit) => permit,
Err(_) => return SelectProducerOutcome::ReceiverClosed,
};
if let Err(error) = snapshot_fence.ensure_snapshot_valid() {
return SelectProducerOutcome::Terminal(Err(error));
}
stats_permit.send(Ok(stats));
SelectProducerOutcome::Terminal(Ok(SelectObjectContentEvent::End(EndEvent::default())))
}
@@ -441,7 +444,9 @@ fn validate_select_request(headers: &http::HeaderMap, input: &mut SelectObjectCo
let output_format = normalize_output_serialization(&mut input.request.output_serialization)?;
if input.request.expression.trim().is_empty() {
return Err(parse_select_failure(EMPTY_SELECT_EXPRESSION_MESSAGE));
return Err(map_select_error_to_s3(&SelectError::ParseSelectFailure {
message: EMPTY_SELECT_EXPRESSION_MESSAGE.to_string(),
}));
}
let progress_enabled = input
.request
@@ -466,13 +471,17 @@ fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()>
return Err(S3Error::new(S3ErrorCode::ObjectSerializationConflict));
}
if let Some(compression) = input.compression_type.as_ref()
&& compression.as_str() != CompressionType::NONE
{
return Err(s3_error!(
NotImplemented,
"SelectObjectContent currently supports only uncompressed input"
));
if let Some(compression) = input.compression_type.as_ref() {
match compression.as_str() {
CompressionType::NONE => {}
CompressionType::GZIP | CompressionType::BZIP2 => {
return Err(s3_error!(
NotImplemented,
"SelectObjectContent currently supports only uncompressed input"
));
}
_ => return Err(map_select_error_to_s3(&SelectError::InvalidCompressionFormat)),
}
}
input.compression_type = Some(CompressionType::from_static(CompressionType::NONE));
@@ -483,8 +492,18 @@ fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()>
"CSV AllowQuotedRecordDelimiter is not supported by SelectObjectContent"
));
}
csv.file_header_info
let file_header_info = csv
.file_header_info
.get_or_insert_with(|| FileHeaderInfo::from_static(FileHeaderInfo::NONE));
if !matches!(
file_header_info.as_str(),
FileHeaderInfo::NONE | FileHeaderInfo::USE | FileHeaderInfo::IGNORE
) {
return Err(S3Error::with_message(
S3ErrorCode::InvalidFileHeaderInfo,
"The FileHeaderInfo value is not valid. Only NONE, USE, and IGNORE are supported.",
));
}
validate_single_byte(csv.comments.as_deref(), S3ErrorCode::InvalidRequestParameter)?;
validate_single_byte(csv.quote_character.as_deref(), S3ErrorCode::InvalidRequestParameter)?;
validate_single_byte(csv.quote_escape_character.as_deref(), S3ErrorCode::InvalidRequestParameter)?;
@@ -575,12 +594,6 @@ fn invalid_scan_range_error() -> S3Error {
S3Error::with_message(S3ErrorCode::InvalidRequestParameter, INVALID_SCAN_RANGE_MESSAGE.to_string())
}
fn parse_select_failure(message: impl Into<String>) -> S3Error {
let mut err = S3Error::with_message(S3ErrorCode::Custom(PARSE_SELECT_FAILURE_CODE.into()), message.into());
err.set_status_code(StatusCode::BAD_REQUEST);
err
}
fn validate_single_byte(value: Option<&str>, code: S3ErrorCode) -> S3Result<()> {
if let Some(value) = value
&& value.len() != 1
@@ -646,12 +659,14 @@ async fn prepare_select_object_snapshot(
fn map_prepare_snapshot_error(err: StoragePrepareSelectObjectSnapshotError) -> S3Error {
match err {
StoragePrepareSelectObjectSnapshotError::Storage(err) => ApiError::from(err).into(),
err => {
let mut s3_error = S3Error::with_message(S3ErrorCode::InternalError, err.to_string());
s3_error.set_source(Box::new(err));
StoragePrepareSelectObjectSnapshotError::Storage(err) => {
let mut s3_error: S3Error = ApiError::from(err).into();
if s3_error.code() == &S3ErrorCode::InternalError {
s3_error.set_message(SelectError::InternalError.to_string());
}
s3_error
}
err => internal_select_error(err),
}
}
@@ -719,9 +734,7 @@ fn encode_csv_batch(batch: &RecordBatch, config: &CSVOutput) -> S3Result<Vec<u8>
}
let mut writer = builder.build(&mut buffer);
writer
.write(batch)
.map_err(|err| s3_error!(InternalError, "can't encode Select output to CSV: {}", err))?;
writer.write(batch).map_err(internal_select_error)?;
drop(writer);
Ok(buffer)
}
@@ -739,12 +752,8 @@ fn encode_json_batch(batch: &RecordBatch, config: &JSONOutput) -> S3Result<Vec<u
let mut writer = JsonWriterBuilder::new()
.with_explicit_nulls(true)
.build::<_, LineDelimited>(&mut buffer);
writer
.write(batch)
.map_err(|err| s3_error!(InternalError, "can't encode Select output to JSON: {}", err))?;
writer
.finish()
.map_err(|err| s3_error!(InternalError, "can't finish Select JSON output: {}", err))?;
writer.write(batch).map_err(internal_select_error)?;
writer.finish().map_err(internal_select_error)?;
drop(writer);
if let Some(delimiter) = config.record_delimiter.as_deref()
@@ -813,122 +822,60 @@ fn clamp_i64(value: u64) -> i64 {
}
fn map_query_error_to_s3(err: QueryError) -> S3Error {
if err.is_snapshot_consistency_error() {
let message = err.to_string();
let mut s3_error = S3Error::with_message(S3ErrorCode::InternalError, message);
s3_error.set_source(Box::new(err));
return s3_error;
}
if let Some(policy_error) = err.s3_select_policy_error() {
let message = policy_error.to_string();
return match policy_error {
S3SelectPolicyError::UnsupportedSqlStructure { .. } => {
S3Error::with_message(S3ErrorCode::UnsupportedSqlStructure, message)
}
S3SelectPolicyError::QueryConcurrencyLimit => S3Error::with_message(S3ErrorCode::SlowDown, message),
S3SelectPolicyError::QueryTimeout { .. } => S3Error::with_message(S3ErrorCode::Busy, message),
_ => S3Error::with_message(S3ErrorCode::InternalError, message),
};
}
let message = err.to_string();
let select_error = err.select_error();
map_select_error_to_s3(&select_error)
}
fn map_select_error_to_s3(err: &SelectError) -> S3Error {
match err {
QueryError::Parser { .. } => parse_select_failure(message),
QueryError::MultiStatement { .. } => S3Error::with_message(S3ErrorCode::UnsupportedSqlStructure, message),
QueryError::NotImplemented { .. } => S3Error::with_message(S3ErrorCode::NotImplemented, message),
QueryError::Datafusion { source } if is_resource_exhausted(source.as_ref()) => {
S3Error::with_message(S3ErrorCode::Busy, message)
SelectError::InvalidCompressionFormat => S3Error::with_message(S3ErrorCode::InvalidCompressionFormat, err.to_string()),
SelectError::InvalidDataSource => S3Error::with_message(S3ErrorCode::InvalidDataSource, err.to_string()),
SelectError::TruncatedInput => S3Error::with_message(S3ErrorCode::TruncatedInput, err.to_string()),
SelectError::CsvParsingError => S3Error::with_message(S3ErrorCode::CSVParsingError, err.to_string()),
SelectError::JsonParsingError => S3Error::with_message(S3ErrorCode::JSONParsingError, err.to_string()),
SelectError::ParquetParsingError => S3Error::with_message(S3ErrorCode::ParquetParsingError, err.to_string()),
SelectError::ParseSelectFailure { message } => custom_bad_request(PARSE_SELECT_FAILURE_CODE, message.clone()),
SelectError::InvalidQuery => custom_bad_request(INVALID_QUERY_CODE, err.to_string()),
SelectError::InvalidDataType => S3Error::with_message(S3ErrorCode::InvalidDataType, err.to_string()),
SelectError::IncorrectSqlFunctionArgumentType => {
S3Error::with_message(S3ErrorCode::IncorrectSqlFunctionArgumentType, err.to_string())
}
QueryError::Datafusion { source } if is_unexpected_eof(source.as_ref()) => {
S3Error::with_message(S3ErrorCode::InternalError, message)
SelectError::DataSourcePathUnsupported => custom_bad_request(DATA_SOURCE_PATH_UNSUPPORTED_CODE, err.to_string()),
SelectError::UnsupportedSqlStructure { .. } => {
S3Error::with_message(S3ErrorCode::UnsupportedSqlStructure, UNSUPPORTED_SQL_STRUCTURE_MESSAGE)
}
QueryError::Datafusion { source } if is_invalid_object_size(source.as_ref()) => {
S3Error::with_message(S3ErrorCode::InternalError, message)
SelectError::UnsupportedSqlOperation => S3Error::with_message(S3ErrorCode::UnsupportedSqlOperation, err.to_string()),
SelectError::EvaluatorBindingDoesNotExist => {
S3Error::with_message(S3ErrorCode::EvaluatorBindingDoesNotExist, err.to_string())
}
QueryError::Datafusion { .. } if looks_like_invalid_scan_range(&message) => {
SelectError::AmbiguousFieldName => S3Error::with_message(S3ErrorCode::AmbiguousFieldName, err.to_string()),
SelectError::InvalidScanRange => {
S3Error::with_message(S3ErrorCode::InvalidRequestParameter, INVALID_SCAN_RANGE_MESSAGE.to_string())
}
QueryError::Datafusion { .. } if looks_like_missing_binding(&message) => {
S3Error::with_message(S3ErrorCode::EvaluatorBindingDoesNotExist, message)
SelectError::QueryConcurrencyLimit => S3Error::with_message(S3ErrorCode::SlowDown, SLOW_DOWN_MESSAGE),
SelectError::QueryTimeout { .. } | SelectError::ResourceExhausted => {
S3Error::with_message(S3ErrorCode::Busy, BUSY_MESSAGE)
}
QueryError::Datafusion { .. } => S3Error::with_message(S3ErrorCode::UnsupportedSqlOperation, message),
QueryError::StoreError { .. } if looks_like_invalid_scan_range(&message) => {
S3Error::with_message(S3ErrorCode::InvalidRequestParameter, INVALID_SCAN_RANGE_MESSAGE.to_string())
SelectError::BucketNotFound => S3Error::with_message(S3ErrorCode::NoSuchBucket, err.to_string()),
SelectError::ObjectNotFound => S3Error::with_message(S3ErrorCode::NoSuchKey, err.to_string()),
SelectError::Canceled | SelectError::InternalError => {
S3Error::with_message(S3ErrorCode::InternalError, SelectError::InternalError.to_string())
}
QueryError::StoreError { .. } if looks_like_bucket_not_found(&message) => {
S3Error::with_message(S3ErrorCode::NoSuchBucket, message)
}
QueryError::StoreError { .. } if looks_like_object_not_found(&message) => {
S3Error::with_message(S3ErrorCode::NoSuchKey, message)
}
QueryError::StoreError { .. } => S3Error::with_message(S3ErrorCode::InternalError, message),
QueryError::BuildQueryDispatcher { .. }
| QueryError::Cancel
| QueryError::FunctionNotExists { .. }
| QueryError::FunctionExists { .. } => S3Error::with_message(S3ErrorCode::InternalError, message),
}
}
fn internal_select_error(_error: impl std::error::Error + Send + Sync + 'static) -> S3Error {
map_select_error_to_s3(&SelectError::InternalError)
}
fn custom_bad_request(code: &'static str, message: String) -> S3Error {
let mut err = S3Error::with_message(S3ErrorCode::Custom(code.into()), message);
err.set_status_code(StatusCode::BAD_REQUEST);
err
}
fn select_query_timeout_error(seconds: u64) -> S3Error {
map_query_error_to_s3(S3SelectPolicyError::QueryTimeout { seconds }.into())
}
fn looks_like_bucket_not_found(message: &str) -> bool {
message.contains("NoSuchBucket") || message.contains("bucket not found") || message.contains("BucketNotFound")
}
const MAX_ERROR_SOURCE_DEPTH: usize = 16;
fn error_chain_any(
mut err: &(dyn std::error::Error + 'static),
predicate: impl Fn(&(dyn std::error::Error + 'static)) -> bool,
) -> bool {
for _ in 0..MAX_ERROR_SOURCE_DEPTH {
if predicate(err) {
return true;
}
let Some(source) = err.source() else {
return false;
};
err = source;
}
false
}
fn is_resource_exhausted(err: &(dyn std::error::Error + 'static)) -> bool {
error_chain_any(err, |err| {
err.downcast_ref::<DataFusionError>()
.is_some_and(|err| matches!(err, DataFusionError::ResourcesExhausted(_)))
})
}
fn is_unexpected_eof(err: &(dyn std::error::Error + 'static)) -> bool {
error_chain_any(err, |err| {
err.downcast_ref::<std::io::Error>()
.is_some_and(|err| err.kind() == std::io::ErrorKind::UnexpectedEof)
})
}
fn is_invalid_object_size(err: &(dyn std::error::Error + 'static)) -> bool {
error_chain_any(err, |err| err.downcast_ref::<std::num::TryFromIntError>().is_some())
}
fn looks_like_object_not_found(message: &str) -> bool {
message.contains("NoSuchKey")
|| message.contains("NoSuchVersion")
|| message.contains("ObjectNotFound")
|| message.contains("object not found")
|| message.contains("NotFound")
}
fn looks_like_missing_binding(message: &str) -> bool {
message.contains("No field named")
|| message.contains("field not found")
|| message.contains("Schema error")
|| message.contains("No such column")
}
fn looks_like_invalid_scan_range(message: &str) -> bool {
message.contains("ScanRange:") || message.contains(INVALID_SCAN_RANGE_MESSAGE)
map_query_error_to_s3(SelectError::QueryTimeout { seconds }.into())
}
fn is_json_document(json: &JSONInput) -> bool {
@@ -942,8 +889,9 @@ mod tests {
use super::*;
use datafusion::{
arrow::{
array::{Array, ListArray},
datatypes::{Field, Int32Type, Schema},
array::{Array, ListArray, StringArray},
datatypes::{DataType, Field, Int32Type, Schema},
error::ArrowError,
},
physical_plan::stream::RecordBatchStreamAdapter,
sql::sqlparser::parser::ParserError,
@@ -951,6 +899,53 @@ mod tests {
use rustfs_test_utils::TestECStoreEnv;
use s3s::dto::{CSVInput, ParquetInput, ScanRange};
fn event_stream_headers(mut bytes: &[u8]) -> Vec<Vec<(String, String)>> {
let mut messages = Vec::new();
while !bytes.is_empty() {
assert!(bytes.len() >= 16, "event-stream message is truncated");
let total_len = u32::from_be_bytes(bytes[0..4].try_into().expect("event-stream total length")) as usize;
let headers_len = u32::from_be_bytes(bytes[4..8].try_into().expect("event-stream headers length")) as usize;
assert!(total_len >= 16 && total_len <= bytes.len(), "invalid event-stream message length");
assert!(12 + headers_len <= total_len - 4, "invalid event-stream headers length");
let mut headers = &bytes[12..12 + headers_len];
let mut decoded = Vec::new();
while !headers.is_empty() {
let name_len = headers[0] as usize;
assert!(headers.len() >= name_len + 4, "event-stream header is truncated");
let name = std::str::from_utf8(&headers[1..1 + name_len])
.expect("event-stream header name should be UTF-8")
.to_string();
assert_eq!(headers[1 + name_len], 7, "expected an event-stream string header");
let value_len = u16::from_be_bytes(
headers[2 + name_len..4 + name_len]
.try_into()
.expect("event-stream header value length"),
) as usize;
assert!(headers.len() >= name_len + 4 + value_len, "event-stream header value is truncated");
let value = std::str::from_utf8(&headers[4 + name_len..4 + name_len + value_len])
.expect("event-stream header value should be UTF-8")
.to_string();
decoded.push((name, value));
headers = &headers[4 + name_len + value_len..];
}
messages.push(decoded);
bytes = &bytes[total_len..];
}
messages
}
async fn http_xml_error(error: S3Error) -> (StatusCode, String) {
let response = error.to_http_response().expect("S3 error should serialize to HTTP");
let status = response.status();
let body = http_body_util::BodyExt::collect(response.into_body())
.await
.expect("S3 error body should be readable")
.to_bytes();
let body = std::str::from_utf8(&body).expect("S3 error XML should be UTF-8").to_string();
(status, body)
}
struct LeaseDropSignal(Option<tokio::sync::oneshot::Sender<()>>);
impl Drop for LeaseDropSignal {
@@ -1010,22 +1005,8 @@ mod tests {
.expect_err("production fence adapter must reject a lost storage snapshot");
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert!(error.to_string().contains("namespace lock was lost"));
}
#[derive(Debug)]
struct CyclicError;
impl std::fmt::Display for CyclicError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("cyclic error")
}
}
impl std::error::Error for CyclicError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
Some(self)
}
assert_eq!(error.message(), Some("An internal error occurred."));
assert!(error.source().is_none());
}
fn base_input() -> SelectObjectContentInput {
@@ -1272,15 +1253,15 @@ mod tests {
#[test]
fn map_query_policy_errors_to_s3_errors() {
let unsupported = map_query_error_to_s3(
S3SelectPolicyError::UnsupportedSqlStructure {
SelectError::UnsupportedSqlStructure {
message: "JOIN is not supported".to_string(),
}
.into(),
);
let saturated = map_query_error_to_s3(S3SelectPolicyError::QueryConcurrencyLimit.into());
let timed_out = map_query_error_to_s3(S3SelectPolicyError::QueryTimeout { seconds: 300 }.into());
let saturated = map_query_error_to_s3(SelectError::QueryConcurrencyLimit.into());
let timed_out = map_query_error_to_s3(SelectError::QueryTimeout { seconds: 300 }.into());
let stream_timed_out = map_query_error_to_s3(QueryError::Datafusion {
source: Box::new(DataFusionError::External(Box::new(S3SelectPolicyError::QueryTimeout { seconds: 300 }))),
source: Box::new(DataFusionError::External(Box::new(SelectError::QueryTimeout { seconds: 300 }))),
});
let exhausted = map_query_error_to_s3(QueryError::Datafusion {
source: Box::new(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::Generic {
@@ -1289,6 +1270,9 @@ mod tests {
}))),
});
let truncated = map_query_error_to_s3(QueryError::Datafusion {
source: Box::new(DataFusionError::External(Box::new(SelectError::TruncatedInput))),
});
let raw_storage_short_read = map_query_error_to_s3(QueryError::Datafusion {
source: Box::new(DataFusionError::ObjectStore(Box::new(datafusion::object_store::Error::Generic {
store: "EcObjectStore",
source: Box::new(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, "truncated object stream")),
@@ -1302,19 +1286,114 @@ mod tests {
});
assert_eq!(unsupported.code(), &S3ErrorCode::UnsupportedSqlStructure);
assert_eq!(unsupported.message(), Some("Unsupported S3 Select SQL structure: JOIN is not supported"));
assert_eq!(unsupported.message(), Some(UNSUPPORTED_SQL_STRUCTURE_MESSAGE));
assert_eq!(saturated.code(), &S3ErrorCode::SlowDown);
assert_eq!(saturated.message(), Some("S3 Select query concurrency limit reached"));
assert_eq!(timed_out.code(), &S3ErrorCode::Busy);
assert_eq!(timed_out.message(), Some("S3 Select query exceeded the 300-second execution limit"));
assert_eq!(stream_timed_out.code(), &S3ErrorCode::Busy);
assert_eq!(
stream_timed_out.message(),
Some("S3 Select query exceeded the 300-second execution limit")
);
assert_eq!(exhausted.code(), &S3ErrorCode::Busy);
assert_eq!(truncated.code(), &S3ErrorCode::InternalError);
assert_eq!(truncated.code(), &S3ErrorCode::TruncatedInput);
assert_eq!(raw_storage_short_read.code(), &S3ErrorCode::InternalError);
assert_eq!(invalid_object_size.code(), &S3ErrorCode::InternalError);
assert_eq!(invalid_object_size.message(), Some("An internal error occurred."));
}
#[test]
fn every_select_error_has_an_explicit_protocol_mapping() {
let cases = vec![
(
SelectError::InvalidCompressionFormat,
S3ErrorCode::InvalidCompressionFormat,
StatusCode::BAD_REQUEST,
),
(SelectError::InvalidDataSource, S3ErrorCode::InvalidDataSource, StatusCode::BAD_REQUEST),
(SelectError::TruncatedInput, S3ErrorCode::TruncatedInput, StatusCode::BAD_REQUEST),
(SelectError::CsvParsingError, S3ErrorCode::CSVParsingError, StatusCode::BAD_REQUEST),
(SelectError::JsonParsingError, S3ErrorCode::JSONParsingError, StatusCode::BAD_REQUEST),
(
SelectError::ParquetParsingError,
S3ErrorCode::ParquetParsingError,
StatusCode::BAD_REQUEST,
),
(
SelectError::ParseSelectFailure {
message: "invalid SELECT expression".to_string(),
},
S3ErrorCode::Custom(PARSE_SELECT_FAILURE_CODE.into()),
StatusCode::BAD_REQUEST,
),
(
SelectError::InvalidQuery,
S3ErrorCode::Custom(INVALID_QUERY_CODE.into()),
StatusCode::BAD_REQUEST,
),
(SelectError::InvalidDataType, S3ErrorCode::InvalidDataType, StatusCode::BAD_REQUEST),
(
SelectError::IncorrectSqlFunctionArgumentType,
S3ErrorCode::IncorrectSqlFunctionArgumentType,
StatusCode::BAD_REQUEST,
),
(
SelectError::DataSourcePathUnsupported,
S3ErrorCode::Custom(DATA_SOURCE_PATH_UNSUPPORTED_CODE.into()),
StatusCode::BAD_REQUEST,
),
(
SelectError::UnsupportedSqlStructure {
message: "JOIN is not supported".to_string(),
},
S3ErrorCode::UnsupportedSqlStructure,
StatusCode::BAD_REQUEST,
),
(
SelectError::UnsupportedSqlOperation,
S3ErrorCode::UnsupportedSqlOperation,
StatusCode::BAD_REQUEST,
),
(
SelectError::EvaluatorBindingDoesNotExist,
S3ErrorCode::EvaluatorBindingDoesNotExist,
StatusCode::BAD_REQUEST,
),
(SelectError::AmbiguousFieldName, S3ErrorCode::AmbiguousFieldName, StatusCode::BAD_REQUEST),
(
SelectError::InvalidScanRange,
S3ErrorCode::InvalidRequestParameter,
StatusCode::BAD_REQUEST,
),
(SelectError::QueryConcurrencyLimit, S3ErrorCode::SlowDown, StatusCode::SERVICE_UNAVAILABLE),
(
SelectError::QueryTimeout { seconds: 300 },
S3ErrorCode::Busy,
StatusCode::SERVICE_UNAVAILABLE,
),
(SelectError::ResourceExhausted, S3ErrorCode::Busy, StatusCode::SERVICE_UNAVAILABLE),
(SelectError::BucketNotFound, S3ErrorCode::NoSuchBucket, StatusCode::NOT_FOUND),
(SelectError::ObjectNotFound, S3ErrorCode::NoSuchKey, StatusCode::NOT_FOUND),
(SelectError::Canceled, S3ErrorCode::InternalError, StatusCode::INTERNAL_SERVER_ERROR),
(SelectError::InternalError, S3ErrorCode::InternalError, StatusCode::INTERNAL_SERVER_ERROR),
];
for (select_error, expected_code, expected_status) in cases {
let error = map_select_error_to_s3(&select_error);
assert_eq!(error.code(), &expected_code, "wrong mapping for {select_error:?}");
assert_eq!(error.status_code(), Some(expected_status), "wrong status for {select_error:?}");
assert!(
error.message().is_some_and(|message| !message.is_empty()),
"missing protocol message for {select_error:?}"
);
}
}
#[test]
fn internal_query_details_are_not_exposed_to_clients() {
let private_detail = "node-1:/private/object/path physical_plan=secret";
let error = map_query_error_to_s3(QueryError::from(DataFusionError::Internal(private_detail.to_string())));
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(error.message(), Some("An internal error occurred."));
assert!(!error.message().is_some_and(|message| message.contains(private_detail)));
assert!(!format!("{error:?}").contains(private_detail));
assert!(error.source().is_none());
}
#[test]
@@ -1329,23 +1408,12 @@ mod tests {
}
#[test]
fn prepare_snapshot_invalid_logical_size_fails_with_internal_error_and_source() {
fn prepare_snapshot_invalid_logical_size_fails_with_redacted_internal_error() {
let err = map_prepare_snapshot_error(StoragePrepareSelectObjectSnapshotError::InvalidLogicalSize { size: -1 });
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert!(
err.source()
.is_some_and(|source| source.downcast_ref::<StoragePrepareSelectObjectSnapshotError>().is_some())
);
}
#[test]
fn error_source_matching_stops_at_the_depth_bound() {
let err = CyclicError;
assert!(!is_resource_exhausted(&err));
assert!(!is_unexpected_eof(&err));
assert!(!is_invalid_object_size(&err));
assert!(err.source().is_none());
assert_eq!(err.message(), Some("An internal error occurred."));
}
#[tokio::test(start_paused = true)]
@@ -1422,6 +1490,39 @@ mod tests {
assert!(lease_released.await.is_ok(), "End should release the snapshot lease");
}
#[tokio::test(start_paused = true)]
async fn successful_stream_serializes_records_stats_and_end_without_error() {
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(StringArray::from(vec!["row"]))])
.expect("test record batch should be valid");
let output = Box::pin(RecordBatchStreamAdapter::new(
schema,
futures::stream::once(async move { Ok::<_, DataFusionError>(batch) }),
));
let (producer, rx, lease_released) = spawn_test_producer(output, 4);
producer.await.expect("producer should finish successfully");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
}
let messages = event_stream_headers(&encoded);
let event_types = messages
.iter()
.filter_map(|headers| {
headers
.iter()
.find_map(|(name, value)| (name == ":event-type").then_some(value.as_str()))
})
.collect::<Vec<_>>();
assert_eq!(event_types, ["Cont", "Records", "Stats", "End"]);
assert!(!messages.iter().flatten().any(|(name, value)| {
(name == ":message-type" && value == "error") || name == ":error-code" || name == ":error-message"
}));
assert!(lease_released.await.is_ok(), "End should release the snapshot lease");
}
#[tokio::test(start_paused = true)]
async fn eof_at_deadline_uses_reserved_slot_for_stats_then_end() {
let output = Box::pin(RecordBatchStreamAdapter::new(
@@ -1455,7 +1556,7 @@ mod tests {
Arc::new(Schema::empty()),
futures::stream::once(async {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
Err(DataFusionError::External(Box::new(S3SelectPolicyError::QueryConcurrencyLimit)))
Err(DataFusionError::External(Box::new(SelectError::QueryConcurrencyLimit)))
}),
));
let (producer, mut rx, lease_released) = spawn_test_producer(output, 2);
@@ -1475,6 +1576,172 @@ mod tests {
assert!(lease_released.await.is_ok(), "stream error should release the snapshot lease");
}
#[tokio::test(start_paused = true)]
async fn select_errors_use_http_codes_before_stream_and_error_frames_after_stream() {
fn csv_error() -> DataFusionError {
DataFusionError::ArrowError(Box::new(ArrowError::CsvError("private CSV parser state".to_string())), None)
}
fn json_error() -> DataFusionError {
DataFusionError::ArrowError(Box::new(ArrowError::JsonError("private JSON parser state".to_string())), None)
}
fn parquet_error() -> DataFusionError {
DataFusionError::ParquetError(Box::new(datafusion::parquet::errors::ParquetError::General(
"private Parquet parser state".to_string(),
)))
}
fn truncated_error() -> DataFusionError {
DataFusionError::External(Box::new(SelectError::TruncatedInput))
}
fn timeout_error() -> DataFusionError {
DataFusionError::External(Box::new(SelectError::QueryTimeout { seconds: 300 }))
}
let cases = [
(
csv_error as fn() -> DataFusionError,
S3ErrorCode::CSVParsingError,
StatusCode::BAD_REQUEST,
b"CSVParsingError" as &[u8],
),
(json_error, S3ErrorCode::JSONParsingError, StatusCode::BAD_REQUEST, b"JSONParsingError"),
(
parquet_error,
S3ErrorCode::ParquetParsingError,
StatusCode::BAD_REQUEST,
b"ParquetParsingError",
),
(truncated_error, S3ErrorCode::TruncatedInput, StatusCode::BAD_REQUEST, b"TruncatedInput"),
(timeout_error, S3ErrorCode::Busy, StatusCode::SERVICE_UNAVAILABLE, b"Busy"),
];
for (source, expected_code, expected_status, encoded_code) in cases {
let pre_stream = map_query_error_to_s3(QueryError::from(source()));
assert_eq!(pre_stream.code(), &expected_code);
assert_eq!(pre_stream.status_code(), Some(expected_status));
let expected_code_text = expected_code.as_str().to_string();
let (status, body) = http_xml_error(pre_stream).await;
assert_eq!(status, expected_status);
assert!(body.contains(&format!("<Code>{expected_code_text}</Code>")));
assert!(body.contains("<Message>"));
let output = Box::pin(RecordBatchStreamAdapter::new(
Arc::new(Schema::empty()),
futures::stream::once(async move { Err(source()) }),
));
let (producer, rx, lease_released) = spawn_test_producer(output, 2);
producer.await.expect("producer should emit the terminal Select error");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
}
let messages = event_stream_headers(&encoded);
let terminal_headers = messages.last().expect("event stream should contain a terminal error");
let encoded_code = std::str::from_utf8(encoded_code).expect("test error code should be UTF-8");
assert!(
terminal_headers
.iter()
.any(|(name, value)| name == ":message-type" && value == "error")
);
assert!(
terminal_headers
.iter()
.any(|(name, value)| name == ":error-code" && value == encoded_code)
);
assert!(
terminal_headers
.iter()
.any(|(name, value)| name == ":error-message" && !value.is_empty() && !value.contains("private"))
);
assert!(
!messages
.iter()
.flatten()
.any(|(name, value)| { name == ":event-type" && matches!(value.as_str(), "Stats" | "End") })
);
assert!(lease_released.await.is_ok(), "error frame should release the snapshot lease");
}
}
#[tokio::test]
async fn sql_and_compression_errors_serialize_as_http_xml() {
let sql_error = map_query_error_to_s3(QueryError::Parser {
source: ParserError::ParserError("unexpected token".to_string()),
});
let (sql_status, sql_body) = http_xml_error(sql_error).await;
assert_eq!(sql_status, StatusCode::BAD_REQUEST);
assert!(sql_body.contains("<Code>ParseSelectFailure</Code>"));
assert!(sql_body.contains("<Message>"));
let mut input = base_input();
input.request.input_serialization.compression_type = Some(CompressionType::from_static("SNAPPY"));
let compression_error =
validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown compression must fail before streaming");
let (compression_status, compression_body) = http_xml_error(compression_error).await;
assert_eq!(compression_status, StatusCode::BAD_REQUEST);
assert!(compression_body.contains("<Code>InvalidCompressionFormat</Code>"));
assert!(compression_body.contains("<Message>"));
}
#[tokio::test(start_paused = true)]
async fn stream_error_after_records_omits_stats_and_end() {
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(StringArray::from(vec!["row"]))])
.expect("test record batch should be valid");
let output = Box::pin(RecordBatchStreamAdapter::new(
schema,
futures::stream::iter([
Ok(batch),
Err(DataFusionError::ArrowError(
Box::new(ArrowError::CsvError("private CSV parser state".to_string())),
None,
)),
]),
));
let (producer, rx, lease_released) = spawn_test_producer(output, 3);
producer
.await
.expect("producer should emit records followed by the terminal error");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
}
let messages = event_stream_headers(&encoded);
assert_eq!(
messages
.iter()
.filter_map(|headers| {
headers
.iter()
.find_map(|(name, value)| (name == ":event-type").then_some(value.as_str()))
})
.collect::<Vec<_>>(),
["Cont", "Records"]
);
let terminal_headers = messages.last().expect("stream should contain a terminal error");
assert!(
terminal_headers
.iter()
.any(|(name, value)| name == ":message-type" && value == "error")
);
assert!(
terminal_headers
.iter()
.any(|(name, value)| name == ":error-code" && value == "CSVParsingError")
);
assert!(
!messages
.iter()
.flatten()
.any(|(name, value)| { name == ":event-type" && matches!(value.as_str(), "Stats" | "End") })
);
assert!(lease_released.await.is_ok(), "error should release the snapshot lease");
}
#[tokio::test(start_paused = true)]
async fn encoder_error_uses_reserved_terminal_slot() {
let values = ListArray::from_iter_primitive::<Int32Type, _, _>([Some([Some(1)])]);
@@ -1497,6 +1764,8 @@ mod tests {
.expect("encoder failure should send one terminal error")
.expect_err("terminal event should be an error");
assert_eq!(encoder_error.code(), &S3ErrorCode::InternalError);
assert_eq!(encoder_error.message(), Some("An internal error occurred."));
assert!(encoder_error.source().is_none());
assert!(rx.recv().await.is_none());
assert!(lease_released.await.is_ok(), "encoder error should release the snapshot lease");
}
@@ -1628,8 +1897,7 @@ mod tests {
};
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(snapshot_fence.0.load(std::sync::atomic::Ordering::Relaxed), 2);
assert!(matches!(rx.recv().await, Some(Ok(SelectObjectContentEvent::Stats(_)))));
assert!(rx.try_recv().is_err(), "snapshot loss after Stats must not enqueue End");
assert!(rx.try_recv().is_err(), "snapshot loss must not enqueue Stats or End");
}
#[test]
@@ -1658,6 +1926,27 @@ mod tests {
);
}
#[test]
fn validate_rejects_unknown_csv_header_mode_before_streaming() {
let mut input = base_input();
input
.request
.input_serialization
.csv
.as_mut()
.expect("base input should use CSV")
.file_header_info = Some(FileHeaderInfo::from_static("INVALID"));
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown header mode must fail");
assert_eq!(error.code(), &S3ErrorCode::InvalidFileHeaderInfo);
assert_eq!(error.status_code(), Some(StatusCode::BAD_REQUEST));
assert_eq!(
error.message(),
Some("The FileHeaderInfo value is not valid. Only NONE, USE, and IGNORE are supported.")
);
}
#[test]
fn validate_accepts_two_byte_csv_input_record_delimiter() {
let mut input = base_input();
@@ -1980,26 +2269,8 @@ mod tests {
}
#[test]
fn map_store_error_not_found_to_no_such_key() {
let err = map_query_error_to_s3(QueryError::StoreError {
e: "ObjectStore NotFound: bucket/object.csv".to_string(),
});
assert_eq!(err.code(), &S3ErrorCode::NoSuchKey);
}
#[test]
fn map_store_error_bucket_not_found_to_no_such_bucket() {
let err = map_query_error_to_s3(QueryError::StoreError {
e: "bucket not found".to_string(),
});
assert_eq!(err.code(), &S3ErrorCode::NoSuchBucket);
}
#[test]
fn map_scan_range_store_error_to_invalid_request_parameter() {
let err = map_query_error_to_s3(QueryError::StoreError {
e: "ScanRange: Start after EOF".to_string(),
});
fn map_typed_scan_range_error_to_invalid_request_parameter() {
let err = map_query_error_to_s3(SelectError::InvalidScanRange.into());
assert_eq!(err.code(), &S3ErrorCode::InvalidRequestParameter);
assert_eq!(err.message(), Some(INVALID_SCAN_RANGE_MESSAGE));
}
+139 -8
View File
@@ -12,11 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::readiness::{DependencyReadinessReport, ReadinessDegradedReason};
use super::readiness::{DependencyReadinessReport, ReadinessDegradedReason, record_readiness_overlay_reason};
use super::{
HEALTH_READY_PATH, MINIO_HEALTH_CLUSTER_PATH, MINIO_HEALTH_CLUSTER_READ_PATH, MINIO_HEALTH_READY_PATH,
collect_cluster_read_health_report, collect_cluster_write_health_report, collect_node_readiness_report,
};
use crate::app::object_traffic_health::{ObjectTrafficHealth, ObjectTrafficSnapshot};
use http::{Method, StatusCode};
use rustfs_kms::ProbeStatus;
use rustfs_kms::probe::{DEFAULT_PROBE_INTERVAL, ENV_KMS_PROBE_INTERVAL_SECS, MIN_PROBE_INTERVAL};
@@ -70,11 +71,33 @@ pub(crate) struct HealthPayloadContext<'a> {
pub(crate) include_dependency_details: bool,
}
pub(crate) async fn collect_probe_readiness(probe: HealthProbe) -> Option<DependencyReadinessReport> {
match readiness_source_for_probe(probe)? {
HealthReadinessSource::Node => Some(collect_node_readiness_report().await),
HealthReadinessSource::ClusterWrite => Some(collect_cluster_write_health_report().await),
HealthReadinessSource::ClusterRead => Some(collect_cluster_read_health_report().await),
pub(crate) async fn collect_probe_readiness(
probe: HealthProbe,
object_traffic_health: Option<&ObjectTrafficHealth>,
) -> Option<DependencyReadinessReport> {
let mut report = match readiness_source_for_probe(probe)? {
HealthReadinessSource::Node => collect_node_readiness_report().await,
HealthReadinessSource::ClusterWrite => collect_cluster_write_health_report().await,
HealthReadinessSource::ClusterRead => collect_cluster_read_health_report().await,
};
if probe == HealthProbe::Readiness
&& let Some(object_traffic_health) = object_traffic_health
{
apply_object_traffic_snapshot(&mut report, object_traffic_health.snapshot());
}
Some(report)
}
fn apply_object_traffic_snapshot(report: &mut DependencyReadinessReport, snapshot: ObjectTrafficSnapshot) {
if snapshot.read_stalled {
let reason = ReadinessDegradedReason::ObjectReadStalled;
report.degraded_reasons.push(reason);
record_readiness_overlay_reason(reason);
}
if snapshot.write_stalled {
let reason = ReadinessDegradedReason::ObjectWriteStalled;
report.degraded_reasons.push(reason);
record_readiness_overlay_reason(reason);
}
}
@@ -300,13 +323,19 @@ pub(crate) fn build_health_response_parts(
),
};
if probe == HealthProbe::Readiness && matches!(kms_ready, Some(false)) {
let object_traffic_stalled = degraded_reasons.iter().any(|reason| {
matches!(
reason,
ReadinessDegradedReason::ObjectReadStalled | ReadinessDegradedReason::ObjectWriteStalled
)
});
if probe == HealthProbe::Readiness && (object_traffic_stalled || matches!(kms_ready, Some(false))) {
health = HealthCheckState {
status_code: StatusCode::SERVICE_UNAVAILABLE,
status: "degraded",
ready: false,
};
if !degraded_reasons.contains(&ReadinessDegradedReason::KmsNotReady) {
if matches!(kms_ready, Some(false)) && !degraded_reasons.contains(&ReadinessDegradedReason::KmsNotReady) {
degraded_reasons.push(ReadinessDegradedReason::KmsNotReady);
}
}
@@ -365,6 +394,8 @@ pub(crate) fn build_health_payload(ctx: HealthPayloadContext<'_>) -> Value {
mod tests {
use super::super::readiness::DependencyReadiness;
use super::*;
use metrics_util::MetricKind;
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use rustfs_kms::{ProbeFailureKind, ProbeResult};
use serial_test::serial;
use temp_env::with_var;
@@ -394,6 +425,106 @@ mod tests {
}
}
#[tokio::test]
async fn readiness_collects_object_stalls_and_recovers_on_completion() {
let object_traffic_health = ObjectTrafficHealth::enabled_for_test(Duration::ZERO);
let read = object_traffic_health
.track_read_storage()
.expect("read tracking must be enabled");
let write = object_traffic_health
.track_write_storage()
.expect("write tracking must be enabled");
let stalled = collect_probe_readiness(HealthProbe::Readiness, Some(&object_traffic_health))
.await
.expect("readiness must have a dependency report");
assert!(stalled.degraded_reasons.contains(&ReadinessDegradedReason::ObjectReadStalled));
assert!(
stalled
.degraded_reasons
.contains(&ReadinessDegradedReason::ObjectWriteStalled)
);
drop(read);
drop(write);
let recovered = collect_probe_readiness(HealthProbe::Readiness, Some(&object_traffic_health))
.await
.expect("readiness must have a dependency report");
assert!(
!recovered
.degraded_reasons
.contains(&ReadinessDegradedReason::ObjectReadStalled)
);
assert!(
!recovered
.degraded_reasons
.contains(&ReadinessDegradedReason::ObjectWriteStalled)
);
}
#[test]
#[serial]
fn an_object_stall_degrades_readiness_without_changing_dependency_details() {
with_var(rustfs_config::ENV_HEALTH_MINIMAL_RESPONSE_ENABLE, Some("false"), || {
let mut report = ready_report();
report.degraded_reasons.push(ReadinessDegradedReason::ObjectReadStalled);
let parts =
build_health_response_parts(Method::GET, HealthProbe::Readiness, Some(&report), "rustfs-endpoint", None, None);
assert_eq!(parts.status_code, StatusCode::SERVICE_UNAVAILABLE);
let payload = parts.payload.expect("GET should include payload");
assert_eq!(payload["ready"], false);
assert_eq!(payload["details"]["storage"]["ready"], true);
assert_eq!(payload["degradedReasons"], json!(["object_read_stalled"]));
});
}
#[test]
fn object_stalls_do_not_change_liveness() {
let mut report = ready_report();
report.degraded_reasons.push(ReadinessDegradedReason::ObjectWriteStalled);
let parts =
build_health_response_parts(Method::HEAD, HealthProbe::Liveness, Some(&report), "rustfs-endpoint", None, None);
assert_eq!(parts.status_code, StatusCode::OK);
assert!(parts.payload.is_none());
}
#[test]
fn object_stall_overlay_records_the_final_readiness_metrics() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
metrics::with_local_recorder(&recorder, || {
let mut report = ready_report();
apply_object_traffic_snapshot(
&mut report,
ObjectTrafficSnapshot {
read_stalled: true,
write_stalled: false,
},
);
});
let entries = snapshotter.snapshot().into_vec();
let ready = entries.iter().find_map(|(composite, _, _, value)| {
(composite.kind() == MetricKind::Gauge && composite.key().name() == "rustfs_runtime_readiness_ready").then_some(value)
});
assert!(matches!(ready, Some(DebugValue::Gauge(value)) if value.into_inner() == 0.0));
let degraded = entries.iter().find_map(|(composite, _, _, value)| {
(composite.kind() == MetricKind::Counter
&& composite.key().name() == "rustfs_runtime_readiness_degraded_total"
&& composite
.key()
.labels()
.any(|label| label.key() == "reason" && label.value() == "object_read_stalled"))
.then_some(value)
});
assert!(matches!(degraded, Some(DebugValue::Counter(1))));
}
#[tokio::test(start_paused = true)]
async fn a_fresh_successful_round_keeps_the_service_ready() {
let round_at = Instant::now();
+2 -2
View File
@@ -1604,7 +1604,7 @@ fn process_connection(
.option_layer(if is_console { Some(RedirectLayer) } else { None })
.layer(BodylessStatusFixLayer)
.layer(HeadRequestBodyFixLayer)
.layer(PublicHealthEndpointLayer)
.layer(PublicHealthEndpointLayer::new(Arc::clone(&server_ctx)))
.option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer))
.layer(DoubleSlashListBucketsCompatLayer)
.service(service)
@@ -1701,7 +1701,7 @@ fn process_connection(
.option_layer(if is_console { Some(RedirectLayer) } else { None })
.layer(BodylessStatusFixLayer)
.layer(HeadRequestBodyFixLayer)
.layer(PublicHealthEndpointLayer)
.layer(PublicHealthEndpointLayer::new(Arc::clone(&server_ctx)))
.option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer))
.layer(DoubleSlashListBucketsCompatLayer)
.service(service)
+138 -15
View File
@@ -14,6 +14,7 @@
use super::runtime_sources;
use crate::admin::console::is_console_path;
use crate::app::object_traffic_health::ObjectTrafficHealth;
use crate::error::ApiError;
use crate::server::RemoteAddr;
use crate::server::cors;
@@ -1238,19 +1239,31 @@ where
}
#[derive(Clone)]
pub struct PublicHealthEndpointLayer;
pub struct PublicHealthEndpointLayer {
server_ctx: Arc<crate::runtime_sources::ServerContextSlot>,
}
impl PublicHealthEndpointLayer {
pub fn new(server_ctx: Arc<crate::runtime_sources::ServerContextSlot>) -> Self {
Self { server_ctx }
}
}
impl<S> Layer<S> for PublicHealthEndpointLayer {
type Service = PublicHealthEndpointService<S>;
fn layer(&self, inner: S) -> Self::Service {
PublicHealthEndpointService { inner }
PublicHealthEndpointService {
inner,
server_ctx: Arc::clone(&self.server_ctx),
}
}
}
#[derive(Clone)]
pub struct PublicHealthEndpointService<S> {
inner: S,
server_ctx: Arc<crate::runtime_sources::ServerContextSlot>,
}
fn health_endpoint_enabled() -> bool {
@@ -1318,6 +1331,7 @@ async fn health_kms_ready() -> bool {
async fn build_public_health_http_response<RestBody, GrpcBody>(
method: Method,
path: String,
object_traffic_health: Option<Arc<ObjectTrafficHealth>>,
) -> Response<HybridBody<RestBody, GrpcBody>>
where
RestBody: From<Bytes>,
@@ -1342,7 +1356,7 @@ where
.expect("failed to build health busy response");
}
let readiness_report = collect_probe_readiness(probe).await;
let readiness_report = collect_probe_readiness(probe, object_traffic_health.as_deref()).await;
let kms_ready = if probe == HealthProbe::Readiness && health_compat_kms_ready_check_enabled() {
Some(health_kms_ready().await)
} else {
@@ -1388,7 +1402,11 @@ where
if is_public_health_endpoint_request(method, path) {
let method = method.clone();
let path = path.to_owned();
return Box::pin(async move { Ok(build_public_health_http_response(method, path).await) });
let object_traffic_health = self
.server_ctx
.installed_app_context()
.map(|context| context.object_traffic_health());
return Box::pin(async move { Ok(build_public_health_http_response(method, path, object_traffic_health).await) });
}
let mut inner = self.inner.clone();
@@ -2185,6 +2203,17 @@ mod tests {
use temp_env::{async_with_vars, with_var};
use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
fn public_health_layer() -> PublicHealthEndpointLayer {
PublicHealthEndpointLayer::new(crate::runtime_sources::ServerContextSlot::new())
}
async fn public_health_layer_with_tracker(object_traffic_health: Arc<ObjectTrafficHealth>) -> PublicHealthEndpointLayer {
let app_context = crate::app::gating_test_env::app_context_with_object_traffic_health(object_traffic_health).await;
let server_ctx = crate::runtime_sources::ServerContextSlot::new();
assert!(server_ctx.install(app_context));
PublicHealthEndpointLayer::new(server_ctx)
}
#[derive(Clone, Debug)]
struct CaptureService;
@@ -2651,7 +2680,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2846,7 +2875,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2874,7 +2903,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2899,7 +2928,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2924,7 +2953,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2946,13 +2975,107 @@ mod tests {
.await;
}
#[tokio::test]
#[serial]
async fn public_readiness_aliases_use_the_installed_object_progress() {
async_with_vars(
[
(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true")),
(rustfs_config::ENV_HEALTH_MINIMAL_RESPONSE_ENABLE, Some("false")),
],
async {
let object_traffic_health = Arc::new(ObjectTrafficHealth::enabled_for_test(Duration::ZERO));
let stalled = object_traffic_health
.track_read_storage()
.expect("read tracking must be enabled");
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = public_health_layer_with_tracker(Arc::clone(&object_traffic_health))
.await
.layer(inner);
let response = service
.call(
Request::builder()
.method(Method::GET)
.uri(HEALTH_READY_PATH)
.body(Full::<Bytes>::from(Bytes::new()))
.expect("canonical readiness request"),
)
.await
.expect("canonical readiness response");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = BodyExt::collect(response.into_body())
.await
.expect("readiness body")
.to_bytes();
let payload: serde_json::Value = serde_json::from_slice(&body).expect("readiness JSON");
assert_eq!(payload["ready"], false);
assert_eq!(payload["degradedReasons"], serde_json::json!(["object_read_stalled"]));
let response = service
.call(
Request::builder()
.method(Method::HEAD)
.uri(MINIO_HEALTH_READY_PATH)
.body(Full::<Bytes>::from(Bytes::new()))
.expect("MinIO readiness request"),
)
.await
.expect("MinIO readiness response");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert!(
BodyExt::collect(response.into_body())
.await
.expect("HEAD body")
.to_bytes()
.is_empty()
);
let response = service
.call(
Request::builder()
.method(Method::GET)
.uri(HEALTH_COMPAT_LIVE_PATH)
.body(Full::<Bytes>::from(Bytes::new()))
.expect("liveness request"),
)
.await
.expect("liveness response");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(calls.load(Ordering::SeqCst), 0);
drop(stalled);
let response = service
.call(
Request::builder()
.method(Method::HEAD)
.uri(HEALTH_READY_PATH)
.body(Full::<Bytes>::from(Bytes::new()))
.expect("recovered readiness request"),
)
.await
.expect("recovered readiness response");
assert_eq!(response.status(), StatusCode::OK);
assert!(
BodyExt::collect(response.into_body())
.await
.expect("HEAD body")
.to_bytes()
.is_empty()
);
},
)
.await;
}
#[tokio::test]
#[serial]
async fn public_health_endpoint_layer_handles_minio_health_cluster_before_inner_service() {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -2977,7 +3100,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("true"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -3002,7 +3125,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("false"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -3027,7 +3150,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("false"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -3069,7 +3192,7 @@ mod tests {
async_with_vars([(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, Some("false"))], async {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
@@ -3092,7 +3215,7 @@ mod tests {
async fn public_health_endpoint_layer_forwards_non_health_requests() {
let inner = CountingHybridService::default();
let calls = inner.calls();
let mut service = PublicHealthEndpointLayer.layer(inner);
let mut service = public_health_layer().layer(inner);
let response = service
.call(
+9
View File
@@ -88,6 +88,8 @@ pub enum ReadinessDegradedReason {
IamNotReady,
LockQuorumUnavailable,
KmsNotReady,
ObjectReadStalled,
ObjectWriteStalled,
ClusterHealthTimeout,
PeerHealthUnavailable,
StorageAndIamUnavailable,
@@ -103,6 +105,8 @@ impl ReadinessDegradedReason {
ReadinessDegradedReason::IamNotReady => "iam_not_ready",
ReadinessDegradedReason::LockQuorumUnavailable => "lock_quorum_unavailable",
ReadinessDegradedReason::KmsNotReady => "kms_not_ready",
ReadinessDegradedReason::ObjectReadStalled => "object_read_stalled",
ReadinessDegradedReason::ObjectWriteStalled => "object_write_stalled",
ReadinessDegradedReason::ClusterHealthTimeout => "cluster_health_timeout",
ReadinessDegradedReason::PeerHealthUnavailable => "peer_health_unavailable",
ReadinessDegradedReason::StorageAndIamUnavailable => "storage_and_iam_unavailable",
@@ -714,6 +718,11 @@ fn record_readiness_report(report: &DependencyReadinessReport) {
}
}
pub(crate) fn record_readiness_overlay_reason(reason: ReadinessDegradedReason) {
gauge!(METRIC_RUNTIME_READINESS_READY).set(0.0);
counter!(METRIC_RUNTIME_READINESS_DEGRADED_TOTAL, "reason" => reason.as_str()).increment(1);
}
fn dependency_readiness_report_from_readiness(readiness: DependencyReadiness) -> DependencyReadinessReport {
DependencyReadinessReport {
degraded_reasons: degraded_reasons(readiness),
+139 -3
View File
@@ -24,9 +24,10 @@ use crate::startup_runtime_sources;
use rustfs_common::MtlsIdentityPem;
use rustfs_config::{
DEFAULT_SERVER_MTLS_ENABLE, DEFAULT_TLS_KEYLOG, DEFAULT_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_INTERVAL,
DEFAULT_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_SYSTEM_CA, ENV_MTLS_CLIENT_CERT, ENV_MTLS_CLIENT_KEY, ENV_SERVER_MTLS_ENABLE,
ENV_TLS_KEYLOG, ENV_TLS_RELOAD_ENABLE, ENV_TLS_RELOAD_INTERVAL, ENV_TRUST_LEAF_CERT_AS_CA, ENV_TRUST_SYSTEM_CA,
RUSTFS_CA_CERT, RUSTFS_CLIENT_CA_CERT_FILENAME, RUSTFS_CLIENT_CERT_FILENAME, RUSTFS_CLIENT_KEY_FILENAME, RUSTFS_TLS_CERT,
DEFAULT_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_SYSTEM_CA, ENV_MTLS_CLIENT_CERT, ENV_MTLS_CLIENT_KEY, ENV_RUSTFS_EXTRA_CA_CERT,
ENV_SERVER_MTLS_ENABLE, ENV_TLS_KEYLOG, ENV_TLS_RELOAD_ENABLE, ENV_TLS_RELOAD_INTERVAL, ENV_TRUST_LEAF_CERT_AS_CA,
ENV_TRUST_SYSTEM_CA, RUSTFS_CA_CERT, RUSTFS_CLIENT_CA_CERT_FILENAME, RUSTFS_CLIENT_CERT_FILENAME, RUSTFS_CLIENT_KEY_FILENAME,
RUSTFS_TLS_CERT,
};
use rustfs_tls_runtime::{
ServerTlsMaterial as RuntimeServerTlsMaterial, TlsGeneration, TlsSource, WebPkiClientVerifierOptions,
@@ -34,6 +35,7 @@ use rustfs_tls_runtime::{
};
use rustfs_utils::{get_env_bool, get_env_opt_str};
use rustls::pki_types::{CertificateDer, PrivateKeyDer, pem::PemObject};
use std::io::Cursor;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::RwLock;
@@ -267,6 +269,60 @@ fn map_runtime_tls_error(err: rustfs_tls_runtime::TlsRuntimeError) -> TlsMateria
}
}
pub(crate) async fn validate_configured_oidc_extra_ca_cert() -> Result<(), TlsMaterialError> {
if let Some(path) = configured_oidc_extra_ca_cert_path() {
let _ = load_configured_oidc_extra_ca_cert().await?;
info!(
component = LOG_COMPONENT_TLS,
subsystem = LOG_SUBSYSTEM_TLS,
event = "oidc_extra_ca_validated",
source = "oidc_extra_ca_bundle",
env_var = ENV_RUSTFS_EXTRA_CA_CERT,
path = ?path,
"OIDC extra root CA bundle validated"
);
}
Ok(())
}
pub(crate) async fn load_configured_oidc_extra_ca_cert() -> Result<Option<Vec<u8>>, TlsMaterialError> {
let Some(path) = configured_oidc_extra_ca_cert_path() else {
return Ok(None);
};
let data = tokio::fs::read(&path)
.await
.map_err(|e| TlsMaterialError::Io(format!("read extra CA bundle {path:?}: {e}")))?;
validate_cert_bundle(&data, &path)?;
Ok(Some(data))
}
fn configured_oidc_extra_ca_cert_path() -> Option<PathBuf> {
let path = get_env_opt_str(ENV_RUSTFS_EXTRA_CA_CERT)?;
let path = path.trim();
if path.is_empty() {
return None;
}
Some(PathBuf::from(path))
}
fn validate_cert_bundle(data: &[u8], path: &Path) -> Result<(), TlsMaterialError> {
let mut reader = Cursor::new(data);
let mut found = false;
let mut store = rustls::RootCertStore::empty();
for cert in CertificateDer::pem_reader_iter(&mut reader) {
let cert = cert.map_err(|e| TlsMaterialError::Parse(format!("invalid extra CA bundle {path:?}: {e}")))?;
store
.add(cert)
.map_err(|e| TlsMaterialError::Parse(format!("invalid extra CA bundle {path:?}: {e}")))?;
found = true;
}
if !found {
return Err(TlsMaterialError::Parse(format!("no certificate found in extra CA bundle {path:?}")));
}
Ok(())
}
/// Load a single certificate file and append PEM data.
/// Returns true if the file was successfully loaded.
async fn load_cert_file(path: &Path, pem_data: &mut Vec<u8>, desc: &str) -> bool {
@@ -681,6 +737,86 @@ mod tests {
fs::write(dir.join(rustfs_config::RUSTFS_TLS_KEY), signing_key.serialize_pem()).unwrap();
}
#[tokio::test]
#[serial_test::serial]
async fn oidc_extra_ca_cert_loads_configured_bundle() {
let CertifiedKey { cert, .. } =
rcgen::generate_simple_self_signed(vec!["extra-ca.example".to_string()]).expect("generate extra CA cert");
let temp_file = tempfile::NamedTempFile::new().expect("create extra CA file");
fs::write(temp_file.path(), cert.pem()).expect("write extra CA file");
let path = temp_file.path().to_string_lossy().to_string();
temp_env::async_with_vars([(ENV_RUSTFS_EXTRA_CA_CERT, Some(path.as_str()))], async {
let extra_ca = load_configured_oidc_extra_ca_cert()
.await
.expect("OIDC extra CA should load")
.expect("configured OIDC extra CA should be present");
assert!(extra_ca.starts_with(cert.pem().as_bytes()));
})
.await;
}
#[tokio::test]
#[serial_test::serial]
async fn oidc_extra_ca_cert_rejects_invalid_pem() {
let temp_file = tempfile::NamedTempFile::new().expect("create invalid extra CA file");
fs::write(temp_file.path(), b"not a certificate").expect("write invalid extra CA file");
let path = temp_file.path().to_string_lossy().to_string();
temp_env::async_with_vars([(ENV_RUSTFS_EXTRA_CA_CERT, Some(path.as_str()))], async {
let err = load_configured_oidc_extra_ca_cert()
.await
.expect_err("invalid extra CA should fail");
assert!(err.to_string().contains("no certificate found in extra CA bundle"));
})
.await;
}
#[tokio::test]
#[serial_test::serial]
async fn oidc_extra_ca_cert_rejects_malformed_der_certificate() {
let temp_file = tempfile::NamedTempFile::new().expect("create malformed extra CA file");
fs::write(
temp_file.path(),
b"-----BEGIN CERTIFICATE-----\nbm90IGEgY2VydA==\n-----END CERTIFICATE-----\n",
)
.expect("write malformed extra CA file");
let path = temp_file.path().to_string_lossy().to_string();
temp_env::async_with_vars([(ENV_RUSTFS_EXTRA_CA_CERT, Some(path.as_str()))], async {
let err = load_configured_oidc_extra_ca_cert()
.await
.expect_err("malformed DER in PEM framing should fail");
assert!(err.to_string().contains("invalid extra CA bundle"));
})
.await;
}
#[tokio::test]
#[serial_test::serial]
async fn load_tls_material_does_not_append_oidc_extra_ca_cert() {
let temp_dir = TempDir::new().expect("create TLS material dir");
write_test_cert_pair(temp_dir.path(), "server.example");
let CertifiedKey { cert, .. } =
rcgen::generate_simple_self_signed(vec!["extra-ca.example".to_string()]).expect("generate extra CA cert");
let temp_file = tempfile::NamedTempFile::new().expect("create extra CA file");
fs::write(temp_file.path(), cert.pem()).expect("write extra CA file");
let path = temp_file.path().to_string_lossy().to_string();
temp_env::async_with_vars([(ENV_RUSTFS_EXTRA_CA_CERT, Some(path.as_str()))], async {
let snapshot = load_tls_material(temp_dir.path().to_str().expect("TLS material dir should be utf-8"))
.await
.expect("TLS material should load");
assert!(snapshot.outbound.root_ca_pem.is_empty());
assert!(snapshot.server.is_some());
})
.await;
}
#[tokio::test]
async fn build_acceptor_accepts_root_single_cert_with_trailing_slash() {
ensure_rustls_crypto_provider();
+42 -2
View File
@@ -14,9 +14,12 @@
use rustfs_iam::{
federation::{FederatedIdentityRegistry, FederatedIdentityService, oidc::StandardOidcAdapter},
get_oidc, init_oidc_sys,
get_oidc, init_oidc_sys_with_extra_root_ca_provider,
oidc::{OidcExtraRootCaMaterial, OidcExtraRootCaProvider},
};
use std::{
collections::hash_map::DefaultHasher,
hash::{Hash, Hasher},
io::{Error, Result},
sync::Arc,
};
@@ -50,7 +53,7 @@ pub(crate) async fn init_auth_integrations() -> Result<()> {
}
}
match init_oidc_sys().await {
match init_oidc_sys_with_extra_root_ca_provider(oidc_extra_root_ca_provider()).await {
Ok(()) => {
if let Some(oidc) = get_oidc() {
let adapter = Arc::new(StandardOidcAdapter::new(oidc));
@@ -72,3 +75,40 @@ pub(crate) async fn init_auth_integrations() -> Result<()> {
Ok(())
}
pub(crate) fn oidc_extra_root_ca_provider() -> OidcExtraRootCaProvider {
OidcExtraRootCaProvider::new(current_oidc_extra_root_ca_material)
}
pub(crate) async fn current_oidc_extra_root_ca_material() -> std::result::Result<OidcExtraRootCaMaterial, String> {
let outbound_tls = crate::runtime_sources::current_outbound_tls_state().await;
let outbound_generation = outbound_tls.as_ref().map(|state| state.generation.0).unwrap_or_default();
let mut root_ca_pem = outbound_tls.as_ref().and_then(|state| state.root_ca_pem.clone());
if let Some(extra_ca_pem) = crate::server::tls_material::load_configured_oidc_extra_ca_cert()
.await
.map_err(|err| err.to_string())?
{
match root_ca_pem.as_mut() {
Some(root_ca_pem) => {
if !root_ca_pem.is_empty() && !root_ca_pem.ends_with(b"\n") {
root_ca_pem.push(b'\n');
}
root_ca_pem.extend_from_slice(&extra_ca_pem);
}
None => root_ca_pem = Some(extra_ca_pem),
}
}
Ok(OidcExtraRootCaMaterial {
generation: oidc_extra_root_ca_generation(outbound_generation, root_ca_pem.as_deref()),
root_ca_pem,
})
}
fn oidc_extra_root_ca_generation(outbound_generation: u64, root_ca_pem: Option<&[u8]>) -> u64 {
let mut hasher = DefaultHasher::new();
outbound_generation.hash(&mut hasher);
root_ca_pem.hash(&mut hasher);
hasher.finish()
}
+23 -12
View File
@@ -24,21 +24,14 @@ const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initializat
const TLS_STARTUP_GENERATION_CONSUMER: &str = "rustfs_server_startup";
pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> {
crate::server::tls_material::validate_configured_oidc_extra_ca_cert()
.await
.map_err(|err| Error::other(err.to_string()))?;
if let Some(tls_path) = normalized_tls_path(config.tls_path.as_deref()) {
match crate::server::tls_material::load_tls_material(tls_path).await {
Ok(snapshot) => {
let generation = next_tls_generation(startup_runtime_sources::current_outbound_tls_generation());
startup_runtime_sources::publish_outbound_tls_state(generation, &snapshot.outbound).await;
startup_runtime_sources::record_tls_generation(TLS_STARTUP_GENERATION_CONSUMER, generation.0);
info!(
target: "rustfs::main",
event = EVENT_TLS_OUTBOUND_INITIALIZED,
component = LOG_COMPONENT_MAIN,
subsystem = LOG_SUBSYSTEM_STARTUP,
tls_path,
generation = generation.0,
"Initialized TLS outbound material"
);
publish_outbound_tls_material(&snapshot.outbound, Some(tls_path)).await;
}
Err(err) => {
error!(
@@ -61,6 +54,24 @@ pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> {
Ok(())
}
async fn publish_outbound_tls_material(outbound: &rustfs_tls_runtime::OutboundTlsMaterial, tls_path: Option<&str>) {
let generation = next_tls_generation(startup_runtime_sources::current_outbound_tls_generation());
startup_runtime_sources::publish_outbound_tls_state(generation, outbound).await;
startup_runtime_sources::record_tls_generation(TLS_STARTUP_GENERATION_CONSUMER, generation.0);
info!(
target: "rustfs::main",
event = EVENT_TLS_OUTBOUND_INITIALIZED,
component = LOG_COMPONENT_MAIN,
subsystem = LOG_SUBSYSTEM_STARTUP,
state = "initialized",
tls_path = tls_path.unwrap_or(""),
generation = generation.0,
has_root_ca = !outbound.root_ca_pem.is_empty(),
has_mtls_identity = outbound.mtls_identity.is_some(),
"Initialized TLS outbound material"
);
}
fn normalized_tls_path(path: Option<&str>) -> Option<&str> {
path.map(str::trim).filter(|value| !value.is_empty())
}
+40
View File
@@ -59,11 +59,51 @@ use s3s::dto::VersioningConfiguration;
#[cfg(test)]
pub(crate) static VERSIONING_CONFIG_LOOKUPS: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
#[cfg(test)]
type VersioningConfigTestHook = (String, std::sync::Arc<tokio::sync::Barrier>, std::sync::Arc<tokio::sync::Barrier>);
#[cfg(test)]
static VERSIONING_CONFIG_TEST_HOOK: std::sync::OnceLock<std::sync::Mutex<Option<VersioningConfigTestHook>>> =
std::sync::OnceLock::new();
#[cfg(test)]
pub(crate) fn install_versioning_config_test_hook(
bucket: String,
entered: std::sync::Arc<tokio::sync::Barrier>,
resume: std::sync::Arc<tokio::sync::Barrier>,
) {
*VERSIONING_CONFIG_TEST_HOOK
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("versioning config test hook lock should not be poisoned") = Some((bucket, entered, resume));
}
#[cfg(test)]
async fn wait_for_versioning_config_test_hook(bucket: &str) {
let hook = {
let mut slot = VERSIONING_CONFIG_TEST_HOOK
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("versioning config test hook lock should not be poisoned");
if slot.as_ref().is_some_and(|(expected_bucket, _, _)| expected_bucket == bucket) {
slot.take()
} else {
None
}
};
if let Some((_bucket, entered, resume)) = hook {
entered.wait().await;
resume.wait().await;
}
}
/// Fetch the bucket's versioning configuration once so callers can derive
/// enabled/suspended state without repeated metadata-sys lookups per request.
pub(crate) async fn bucket_versioning_config(bucket: &str) -> VersioningConfiguration {
#[cfg(test)]
VERSIONING_CONFIG_LOOKUPS.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
#[cfg(test)]
wait_for_versioning_config_test_hook(bucket).await;
match BucketVersioningSys::get(bucket).await {
Ok(cfg) => cfg,
Err(err) => {
+4
View File
@@ -1034,6 +1034,10 @@ pub(crate) async fn save_config_no_lock(api: Arc<ECStore>, file: &str, data: Vec
ecstore_config::com::save_config_no_lock(api, file, data).await
}
pub(crate) async fn delete_config_no_lock(api: Arc<ECStore>, file: &str) -> Result<()> {
ecstore_config::com::delete_config_no_lock(api, file).await
}
pub(crate) async fn with_config_object_write_lock<F, Fut, T>(api: Arc<ECStore>, object: String, operation: F) -> Result<T>
where
F: FnOnce() -> Fut + Send + 'static,
+33 -17
View File
@@ -28,7 +28,7 @@ const REPLICATION_RUNTIME_NOT_INITIALIZED: &str = "replication runtime not initi
const REPLICATION_QUEUE_BACKLOG_PRESENT: &str = "replication queue has pending work";
const REPLICATION_QUEUE_STATS_UNAVAILABLE: &str = "replication queue stats unavailable";
const SCANNER_ADMISSION_SATURATED: &str = "scanner active work reached configured set-scan limit";
const SCANNER_ACTIVITY_IDLE_OR_NOT_INITIALIZED: &str = "scanner activity idle or not initialized";
const SCANNER_RUNTIME_NOT_INITIALIZED: &str = "scanner runtime not initialized";
const STORAGE_CONCURRENCY_PROVIDER_MISSING_FOREGROUND_READ: &str =
"storage concurrency provider did not expose foreground read admission";
const STORAGE_CONCURRENCY_PROVIDER_MISSING_FOREGROUND_WRITE: &str =
@@ -107,19 +107,24 @@ fn metadata_workload_admission_snapshot_from_initialized(runtime_initialized: bo
pub fn scanner_workload_admission_snapshot() -> WorkloadAdmissionSnapshot {
let runtime_config = rustfs_scanner::scanner_runtime_config_status();
scanner_workload_admission_snapshot_from_activity(
rustfs_scanner::scanner_runtime_initialized(),
rustfs_scanner::current_scanner_activity(),
runtime_config.max_concurrent_set_scans.value,
)
}
fn scanner_workload_admission_snapshot_from_activity(active: u64, limit: usize) -> WorkloadAdmissionSnapshot {
fn scanner_workload_admission_snapshot_from_activity(
runtime_initialized: bool,
active: u64,
limit: usize,
) -> WorkloadAdmissionSnapshot {
let effective_limit = if limit == 0 { None } else { Some(limit) };
let state = if effective_limit.is_some_and(|limit| usize::try_from(active).ok().is_some_and(|active| active >= limit)) {
AdmissionState::Saturated
} else if active > 0 {
AdmissionState::Open
} else {
let state = if !runtime_initialized {
AdmissionState::Unknown
} else if effective_limit.is_some_and(|limit| usize::try_from(active).ok().is_some_and(|active| active >= limit)) {
AdmissionState::Saturated
} else {
AdmissionState::Open
};
let snapshot = WorkloadAdmissionSnapshot::new(WorkloadClass::Scanner, state).with_counts(
@@ -130,7 +135,7 @@ fn scanner_workload_admission_snapshot_from_activity(active: u64, limit: usize)
match state {
AdmissionState::Saturated => snapshot.with_reason(SCANNER_ADMISSION_SATURATED),
AdmissionState::Unknown => snapshot.with_reason(SCANNER_ACTIVITY_IDLE_OR_NOT_INITIALIZED),
AdmissionState::Unknown => snapshot.with_reason(SCANNER_RUNTIME_NOT_INITIALIZED),
_ => snapshot,
}
}
@@ -277,7 +282,7 @@ mod tests {
#[test]
fn scanner_snapshot_reports_active_work_units() {
let snapshot = scanner_workload_admission_snapshot_from_activity(5, 8);
let snapshot = scanner_workload_admission_snapshot_from_activity(true, 5, 8);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Open);
@@ -288,29 +293,40 @@ mod tests {
}
#[test]
fn scanner_snapshot_is_unknown_when_idle_or_uninitialized() {
let snapshot = scanner_workload_admission_snapshot_from_activity(0, 8);
fn scanner_snapshot_is_open_when_initialized_and_idle() {
let snapshot = scanner_workload_admission_snapshot_from_activity(true, 0, 8);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Open);
assert_eq!(snapshot.active, Some(0));
assert_eq!(snapshot.limit, Some(8));
assert_eq!(snapshot.reason, None);
}
#[test]
fn scanner_snapshot_is_unknown_before_runtime_initialization() {
let snapshot = scanner_workload_admission_snapshot_from_activity(false, 0, 8);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Unknown);
assert_eq!(snapshot.active, Some(0));
assert_eq!(snapshot.limit, Some(8));
assert_eq!(snapshot.reason.as_deref(), Some(SCANNER_ACTIVITY_IDLE_OR_NOT_INITIALIZED));
assert_eq!(snapshot.reason.as_deref(), Some(SCANNER_RUNTIME_NOT_INITIALIZED));
}
#[test]
fn scanner_snapshot_treats_zero_set_scan_limit_as_topology_derived() {
let snapshot = scanner_workload_admission_snapshot_from_activity(0, 0);
let snapshot = scanner_workload_admission_snapshot_from_activity(true, 0, 0);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Unknown);
assert_eq!(snapshot.state, AdmissionState::Open);
assert_eq!(snapshot.limit, None);
assert_eq!(snapshot.reason.as_deref(), Some(SCANNER_ACTIVITY_IDLE_OR_NOT_INITIALIZED));
assert_eq!(snapshot.reason, None);
}
#[test]
fn scanner_snapshot_with_topology_derived_limit_reports_active_work_open() {
let snapshot = scanner_workload_admission_snapshot_from_activity(4, 0);
let snapshot = scanner_workload_admission_snapshot_from_activity(true, 4, 0);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Open);
@@ -321,7 +337,7 @@ mod tests {
#[test]
fn scanner_snapshot_reports_saturation_when_active_work_reaches_limit() {
let snapshot = scanner_workload_admission_snapshot_from_activity(4, 4);
let snapshot = scanner_workload_admission_snapshot_from_activity(true, 4, 4);
assert_eq!(snapshot.class, WorkloadClass::Scanner);
assert_eq!(snapshot.state, AdmissionState::Saturated);
+28
View File
@@ -842,6 +842,34 @@ for file in "${disk_logging_files[@]}"; do
fi
done
# `set_disks` expands every Disk through Debug, including raw format bytes and
# the full per-operation metrics ring. Keep it out of the INFO scanner span.
scanner_disk_skip_pattern='#\[(tracing::)?instrument\([^]]*skip\([^)]*\bset_disks\b[^)]*\)[^]]*\)\][[:space:]]*async fn nsscanner_disk\b'
if ! rg -U "$scanner_disk_skip_pattern" crates/scanner/src/scanner_io.rs >/dev/null; then
echo "❌ logging guardrail violation: nsscanner_disk must skip set_disks in its tracing instrumentation" >&2
exit 1
fi
for fixture in \
$'#[tracing::instrument(skip(self, budget, updates, cache, set_disks))]\nasync fn nsscanner_disk('; do
if ! printf '%s\n' "$fixture" | rg -U "$scanner_disk_skip_pattern" >/dev/null; then
echo "❌ logging guardrail self-test failed: safe nsscanner_disk span was rejected" >&2
echo "$fixture" >&2
exit 1
fi
done
for fixture in \
$'#[tracing::instrument(skip(self, budget, updates, cache))]\nasync fn nsscanner_disk(' \
$'#[tracing::instrument(skip(self, budget, updates, cache), fields(set_disks = set_disks.len()))]\nasync fn nsscanner_disk(' \
$'#[tracing::instrument(skip(self, budget, updates, cache, set_disks_count))]\nasync fn nsscanner_disk('; do
if printf '%s\n' "$fixture" | rg -U "$scanner_disk_skip_pattern" >/dev/null; then
echo "❌ logging guardrail self-test failed: unsafe nsscanner_disk span was accepted" >&2
echo "$fixture" >&2
exit 1
fi
done
# `forbidden_patterns` above only retires log lines that already shipped, so a
# newly written sentence-style log passes every check in this script — which is
# how one reaches review in the first place (PR #5822 added
+10 -7
View File
@@ -22,10 +22,13 @@ set -euo pipefail
cd "$(dirname "$0")/.."
# Baselines verified on 2026-08-06. Lower-only; see header.
S3S_IMPORT_FILES_BASELINE=236
S3_ERROR_LINES_BASELINE=1678
# Baselines verified on 2026-08-11. Lower-only; see header.
# Excludes crates/e2e_test/ — test infrastructure legitimately uses s3s
# to verify S3 behavior and does not widen the production s3s surface.
S3S_IMPORT_FILES_BASELINE=213
S3_ERROR_LINES_BASELINE=1617
S3S_PATH_PATTERN='(^|[^"[:alnum:]_])s3s::'
E2E_TEST_GLOB='--glob=!crates/e2e_test/**'
TMP_DIR="$(mktemp -d)"
trap 'rm -rf "$TMP_DIR"' EXIT
@@ -42,8 +45,8 @@ run_rg_to() {
fi
}
run_rg_to "$TMP_DIR/import_files" -l "$S3S_PATH_PATTERN" --type rust
run_rg_to "$TMP_DIR/error_lines" -c 's3_error!' --type rust
run_rg_to "$TMP_DIR/import_files" -l "$S3S_PATH_PATTERN" --type rust $E2E_TEST_GLOB
run_rg_to "$TMP_DIR/error_lines" -c 's3_error!' --type rust $E2E_TEST_GLOB
s3s_import_files="$(grep -c . "$TMP_DIR/import_files" || true)"
s3_error_lines="$(awk -F: '{sum += $NF} END {print sum + 0}' "$TMP_DIR/error_lines")"
@@ -76,9 +79,9 @@ check_ratchet() {
}
check_ratchet "files importing s3s" "$s3s_import_files" "$S3S_IMPORT_FILES_BASELINE" \
"rg -l '$S3S_PATH_PATTERN' --type rust"
"rg -l '$S3S_PATH_PATTERN' --type rust $E2E_TEST_GLOB"
check_ratchet "s3_error! invocation lines" "$s3_error_lines" "$S3_ERROR_LINES_BASELINE" \
"rg -c 's3_error!' --type rust"
"rg -c 's3_error!' --type rust $E2E_TEST_GLOB"
if ((status != 0)); then
exit 1