Compare commits

..

4 Commits

Author SHA1 Message Date
唐小鸭 65035481f6 refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)

Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.

Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
  replicate_delete_outcome, target_delete_version_id,
  delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
  SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
  ssec_passthrough_evidence_present (param-demoted to the echoed
  customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
  filemeta-independence contract)

ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
  resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
  capability enum so existing consumer paths keep compiling

Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.

* chore(replication): retire the datatypes.rs relay early

README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.

* chore(arch): extend migration guards to the M2-moved decision contracts

The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:

- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
  helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
  replicate_delete_outcome, resync_existing_delete_replication_info,
  target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
  classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
  fns/enums (ssec_passthrough_evidence_present deliberately excluded:
  ECStore keeps a thin HeadObjectOutput adapter under that name).

Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.

Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
2026-08-18 14:50:08 +08:00
唐小鸭 b2ff43eb1a Merge remote-tracking branch 'origin/main' into p1-18-merged
# Conflicts:
#	rustfs/src/admin/router.rs
2026-08-18 13:47:28 +08:00
唐小鸭 daecb93139 feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets

Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.

Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
  into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
  ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
  value is never re-proxied. Outbound client proxy calls send the marker
  as "true"; replication worker convergence HEADs send it as "false" so
  a peer's proxy layer cannot answer a convergence check by proxying
  back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
  when the marker is set, versioning is suspended, or no replication
  config; otherwise filter_target_arns -> TargetClient lookup, skipping
  targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
  the three tagging calls. Proxy calls never send the replication-check
  SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
  the target performs real decryption. Conditional (If-*) headers are
  not forwarded (MinIO parity); Range and part_number are, with
  parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
  MinIO-aligned (one total per proxied request, one failed when no
  target served it). The previous misattributed counters — replication
  worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
  removed; ReplProxyMetric now maps the tagging counters instead of
  dropping them.

e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.

Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.

Refs rustfs/backlog#1675 (P1-5)

* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
2026-08-18 08:47:22 +08:00
唐小鸭 21c2fb42bb refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:

- resync_bucket (522 lines -> 61-line step sequence): leader lock,
  target resolution, walk/collector/worker spawning, and dispatch loop
  extracted into focused helpers; pure decision helpers (DTO builders,
  HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
  seeding, read/stat option builders, skip-path notes, target HEAD
  action resolution, and the multipart/single-put payload transport
  extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
  ledger load, per-entry replay (delete/object/metadata), and retained
  entry resolution extracted; retry bookkeeping semantics preserved
  exactly (inner continue-paths push inside helpers, outer Missed push
  stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
  IAM item type.

No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
2026-08-17 17:08:45 +08:00
76 changed files with 9157 additions and 5103 deletions
+2 -2
View File
@@ -66,8 +66,8 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
./scripts/check_s3s_footprint.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
@echo "📣 Checking FIPS wording guard..."
./scripts/check_fips_wording.sh
.PHONY: log-analyzer-rules-check
-3
View File
@@ -117,9 +117,6 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
-3
View File
@@ -152,9 +152,6 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
+4 -4
View File
@@ -31,7 +31,7 @@ HTTP request
→ storage/ecfs (erasure coding, encryption, checksums)
→ ecstore (disk pool selection, data distribution)
→ rio (reader pipeline: encrypt → compress → hash → write)
→ io-core (buffer pool, storage profiling, admission control)
→ io-core (zero-copy I/O, buffer pool, direct I/O)
→ local disk / remote disk via RPC
```
@@ -55,7 +55,7 @@ rustfs/ # Workspace root (virtual manifest)
├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
│ ├── ecstore/ # Erasure-coded storage engine
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Buffer pool, storage profiling, admission control
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
│ ├── io-metrics/ # I/O metrics collection
│ ├── common/ # Shared runtime state, globals, data usage types
│ ├── config/ # Configuration types and parsing
@@ -302,7 +302,7 @@ The binary (`main.rs`) boots in this order:
│ │ │
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ ecstore │ │ rio │ │ io-core │
│ (core) │ │ (readers) │ │ (buffers)
│ (core) │ │ (readers) │ │ (zero-copy)
└─────┬──────┘ └─────────────┘ └─────────────┘
┌─────┬──┼──┬─────┬──────┐
@@ -314,7 +314,7 @@ The binary (`main.rs`) boots in this order:
- **"Where does S3 PutObject go?"**
`server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
`ecstore` distributes → `rio` encrypts/compresses → `io-core` supplies buffers
`ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
- **"Where are bucket policies enforced?"**
`app/bucket_usecase` calls into `crates/policy/`
Generated
+1 -3
View File
@@ -9465,7 +9465,6 @@ dependencies = [
"tokio-stream",
"tokio-util",
"tonic",
"tonic-prost",
"tower",
"tracing",
"tracing-core",
@@ -9539,8 +9538,6 @@ version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"base64 0.23.1",
"bytes",
"crc-fast",
"futures",
"hotpath",
"http 1.5.0",
@@ -9613,6 +9610,7 @@ version = "1.0.0-rc.2"
dependencies = [
"bytes",
"hotpath",
"memmap2",
"rustfs-io-metrics",
"thiserror 2.0.20",
"tokio",
-23
View File
@@ -224,13 +224,6 @@ pub struct HealOpts {
pub enum HealAdmissionDropReason {
QueueFull,
PolicyDropped,
/// HS-06: an admin heal start overlaps (same bucket with mutually
/// containing prefixes, or the same erasure set) an already running or
/// queued task. Only produced when RUSTFS_HEAL_OVERLAP_POLICY=minio_error.
AlreadyRunning,
/// HS-06: same as [`Self::AlreadyRunning`] but for paths that merely
/// contain (or are contained by) the active task's path.
OverlappingPaths,
}
impl HealAdmissionDropReason {
@@ -238,8 +231,6 @@ impl HealAdmissionDropReason {
match self {
Self::QueueFull => "queue_full",
Self::PolicyDropped => "policy_dropped",
Self::AlreadyRunning => "already_running",
Self::OverlappingPaths => "overlapping_paths",
}
}
}
@@ -326,9 +317,6 @@ pub enum HealChannelCommand {
Query {
heal_path: String,
client_token: String,
/// Incremental result cursor (HS-06): only items with a sequence
/// greater than this are returned; `None` keeps the full snapshot.
since_seq: Option<u64>,
response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
},
/// Cancel heal task
@@ -534,21 +522,10 @@ async fn receive_heal_channel_response(
/// Send heal query request
pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
query_heal_status_since(heal_path, client_token, None).await
}
/// Incremental heal query (HS-06): pass the client's last seen sequence
/// number to receive only newer result items.
pub async fn query_heal_status_since(
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<HealChannelResponse, String> {
let (response_tx, response_rx) = oneshot::channel();
send_heal_command(HealChannelCommand::Query {
heal_path,
client_token,
since_seq,
response_tx,
})
.await?;
-9
View File
@@ -205,12 +205,3 @@ pub const DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES: usize = 8 * 1024 * 1024;
/// Default MRF replay batch size.
pub const DEFAULT_HEAL_MRF_REPLAY_BATCH: usize = 256;
/// Environment variable selecting how admin heal starts behave when the
/// requested path overlaps an already running or queued heal: `merge`
/// (default, keep today's dedup/merge semantics) or `minio_error` (return a
/// typed already-running / overlapping-paths rejection like madmin).
pub const ENV_HEAL_OVERLAP_POLICY: &str = "RUSTFS_HEAL_OVERLAP_POLICY";
/// Default overlap policy: merge duplicate/overlapping requests.
pub const DEFAULT_HEAL_OVERLAP_POLICY: &str = "merge";
+2 -2
View File
@@ -4,8 +4,8 @@ This module is the shared failure-injection boundary for replication end-to-end
`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions. Each record also journals a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
+284 -5
View File
@@ -30,10 +30,12 @@ use s3s::access::{S3Access, S3AccessContext};
use s3s::auth::SimpleAuth;
use s3s::dto::{
AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput,
DeleteObjectTaggingInput, DeleteObjectTaggingOutput, ETag, GetBucketVersioningInput, GetBucketVersioningOutput,
GetObjectInput, GetObjectOutput, GetObjectTaggingInput, GetObjectTaggingOutput, HeadBucketInput, HeadBucketOutput,
HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
PutObjectOutput, PutObjectTaggingInput, PutObjectTaggingOutput, StreamingBlob, Tag, TagSet, Timestamp, TimestampFormat,
UploadPartInput, UploadPartOutput,
};
use s3s::service::{S3Service, S3ServiceBuilder};
use s3s::validation::{AwsNameValidation, NameValidation};
@@ -88,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
/// Wire prefix of the SSE-C passthrough replication transport headers
/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
/// generic S3, which silently discard unknown x-* headers.
const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -103,6 +112,9 @@ pub enum Operation {
GetObject,
HeadObject,
DeleteObject,
GetObjectTagging,
PutObjectTagging,
DeleteObjectTagging,
ListObjectVersions,
CreateMultipartUpload,
UploadPart,
@@ -149,6 +161,42 @@ impl ReplicationTimestampHeaders {
}
}
/// Read-proxy related headers observed on a request, journaled so proxy
/// tests can assert the exact wire contract: the anti-loop marker present,
/// the replication-check exemption absent, and the client SSE-C key family
/// forwarded verbatim. The SSE-C key value itself is never retained — only
/// its presence.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ProxyHeaderSnapshot {
pub source_proxy_request: Option<String>,
pub replication_check: Option<String>,
pub ssec_algorithm: Option<String>,
pub ssec_key_present: bool,
pub ssec_key_md5: Option<String>,
/// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header, so fail-closed tests can assert the
/// sender really shipped the material a dropping target discarded.
pub ssec_transport_present: bool,
}
impl ProxyHeaderSnapshot {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
source_proxy_request: header_value(headers, &["x-rustfs-source-proxy-request", "x-minio-source-proxy-request"])
.map(bounded_journal_value),
replication_check: header_value(headers, &["x-rustfs-source-replication-check", "x-minio-source-replication-check"])
.map(bounded_journal_value),
ssec_algorithm: header_value(headers, &["x-amz-server-side-encryption-customer-algorithm"])
.map(bounded_journal_value),
ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
ssec_transport_present: headers
.keys()
.any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -163,6 +211,7 @@ pub struct RequestRecord {
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
pub fault: Option<FaultAction>,
}
@@ -178,6 +227,10 @@ struct ControlState {
struct StoreState {
assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool,
/// MinIO-like mode: silently discard non-whitelisted replication
/// transport headers instead of storing them (see
/// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
drop_unlisted_replication_headers: bool,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
total_bytes: usize,
@@ -199,6 +252,12 @@ struct ObjectVersion {
delete_marker: bool,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
/// Object tags as ordered key/value pairs (PutObjectTagging replaces the
/// whole set, DeleteObjectTagging clears it).
tags: Vec<(String, String)>,
/// SSE-C passthrough transport headers stored with the version (RustFS
/// target behavior); empty when the drop mode discarded them.
replication_sse_headers: Vec<(String, String)>,
}
#[derive(Clone)]
@@ -208,6 +267,7 @@ struct MultipartState {
version_id: String,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
replication_sse_headers: Vec<(String, String)>,
parts: BTreeMap<i32, MultipartPart>,
}
@@ -428,6 +488,15 @@ impl FakeS3Target {
/// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones.
/// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header instead of storing it. The default (off)
/// models a RustFS target, which preserves the headers and echoes SSE-C
/// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
/// HEAD/GET of the replica.
pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
}
pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
}
@@ -569,6 +638,7 @@ impl S3Access for FaultAccess {
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let proxy_headers = ProxyHeaderSnapshot::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
@@ -576,6 +646,7 @@ impl S3Access for FaultAccess {
parsed,
content_length,
replication_timestamps,
proxy_headers,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
@@ -615,6 +686,9 @@ fn operation_from_s3_name(name: &str) -> Operation {
"GetObject" => Operation::GetObject,
"HeadObject" => Operation::HeadObject,
"DeleteObject" => Operation::DeleteObject,
"GetObjectTagging" => Operation::GetObjectTagging,
"PutObjectTagging" => Operation::PutObjectTagging,
"DeleteObjectTagging" => Operation::DeleteObjectTagging,
"CreateMultipartUpload" => Operation::CreateMultipartUpload,
"UploadPart" => Operation::UploadPart,
"CompleteMultipartUpload" => Operation::CompleteMultipartUpload,
@@ -630,6 +704,7 @@ fn record_request(
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -655,6 +730,7 @@ fn record_request(
content_length,
consumed_bytes: None,
replication_timestamps,
proxy_headers,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -721,6 +797,15 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
(&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
(&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
(&Method::GET, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::GetObjectTagging
}
(&Method::PUT, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::PutObjectTagging
}
(&Method::DELETE, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::DeleteObjectTagging
}
// A replication PUT addresses the source version via `?versionId=`.
(&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
(&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
@@ -788,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
Ok(version_id.to_string())
}
/// Capture the SSE-C passthrough transport headers a replication PUT carried.
/// Returns an empty set in the MinIO-like drop mode.
fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
if drop_unlisted {
return Vec::new();
}
headers
.iter()
.filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
.filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
.collect()
}
/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
/// replica: the customer algorithm restored from the transport headers.
fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
version
.replication_sse_headers
.iter()
.find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
.map(|(_, value)| value.clone())
}
fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
header_value(headers, &SOURCE_ETAG_HEADERS)
.map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1135,6 +1243,33 @@ fn find_version(state: &StoreState, bucket: &str, key: &str, version_id: Option<
Ok(version.clone())
}
/// Replace (or clear, with an empty vec) the tag set of the addressed
/// version, returning its version id. Mirrors `find_version` addressing:
/// explicit version id or the latest version, delete markers rejected.
fn set_version_tags(
state: &mut StoreState,
bucket: &str,
key: &str,
version_id: Option<&str>,
tags: Vec<(String, String)>,
) -> S3Result<String> {
// Resolve first (immutable) so the error paths match find_version.
let resolved = find_version(state, bucket, key, version_id)?.version_id;
let versions = state
.buckets
.get_mut(bucket)
.expect("bucket existence checked by find_version")
.objects
.get_mut(key)
.expect("key existence checked by find_version");
let version = versions
.iter_mut()
.find(|version| version.version_id == resolved)
.expect("version existence checked by find_version");
version.tags = tags;
Ok(resolved)
}
#[async_trait]
impl S3 for FakeBackend {
async fn head_bucket(&self, req: S3Request<HeadBucketInput>) -> S3Result<S3Response<HeadBucketOutput>> {
@@ -1231,7 +1366,10 @@ impl S3 for FakeBackend {
let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let assign_own = lock(&self.store).assign_own_version_ids;
let (assign_own, drop_unlisted) = {
let state = lock(&self.store);
(state.assign_own_version_ids, state.drop_unlisted_replication_headers)
};
let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? {
Some(value) => value,
@@ -1248,6 +1386,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: input.content_type,
metadata: input.metadata,
tags: Vec::new(),
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
};
upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
Ok(apply_response_fault(
@@ -1268,6 +1408,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(GetObjectOutput {
body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1277,6 +1418,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
@@ -1291,6 +1433,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(HeadObjectOutput {
content_length: Some(version.body.len() as i64),
@@ -1299,12 +1442,79 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
))
}
async fn get_object_tagging(&self, req: S3Request<GetObjectTaggingInput>) -> S3Result<S3Response<GetObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version = {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let tag_set: TagSet = version
.tags
.into_iter()
.map(|(key, value)| Tag {
key: Some(key),
value: Some(value),
})
.collect();
Ok(apply_response_fault(
S3Response::new(GetObjectTaggingOutput {
tag_set,
version_id: Some(ObjectVersionId::from(version.version_id)),
}),
fault.as_ref(),
))
}
async fn put_object_tagging(&self, req: S3Request<PutObjectTaggingInput>) -> S3Result<S3Response<PutObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let tags = input
.tagging
.tag_set
.into_iter()
.map(|tag| (tag.key.unwrap_or_default(), tag.value.unwrap_or_default()))
.collect();
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), tags)?
};
Ok(apply_response_fault(
S3Response::new(PutObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object_tagging(
&self,
req: S3Request<DeleteObjectTaggingInput>,
) -> S3Result<S3Response<DeleteObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), Vec::new())?
};
Ok(apply_response_fault(
S3Response::new(DeleteObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object(&self, req: S3Request<DeleteObjectInput>) -> S3Result<S3Response<DeleteObjectOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
@@ -1381,6 +1591,8 @@ impl S3 for FakeBackend {
delete_marker: true,
content_type: None,
metadata: None,
tags: Vec::new(),
replication_sse_headers: Vec::new(),
},
)?;
Ok(apply_response_fault(
@@ -1408,9 +1620,10 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below
// Read the flags before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let drop_unlisted = state.drop_unlisted_replication_headers;
let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert(
upload_id.clone(),
@@ -1420,6 +1633,7 @@ impl S3 for FakeBackend {
version_id,
content_type: input.content_type,
metadata: input.metadata,
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
parts: BTreeMap::new(),
},
);
@@ -1557,6 +1771,7 @@ impl S3 for FakeBackend {
version_id: upload.version_id.clone(),
content_type: upload.content_type.clone(),
metadata: upload.metadata.clone(),
replication_sse_headers: upload.replication_sse_headers.clone(),
parts: BTreeMap::new(),
},
selected,
@@ -1583,6 +1798,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: upload.content_type,
metadata: upload.metadata,
tags: Vec::new(),
replication_sse_headers: upload.replication_sse_headers,
};
let mut state = lock(&self.store);
let current = state
@@ -1787,6 +2004,65 @@ mod tests {
Ok(())
}
/// Default mode is RustFS-like: SSE-C passthrough transport headers are
/// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
/// MinIO-like: the headers are silently discarded, so no evidence comes
/// back — the exact difference the N2 fail-closed audit keys on. Both
/// modes journal that the sender shipped the transport headers.
#[tokio::test]
async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
let put_with_transport_headers = |key: &'static str| {
client
.put_object()
.bucket("target-bucket")
.key(key)
.body(ByteStream::from_static(b"ciphertext"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
Ok::<_, std::convert::Infallible>(request)
})
.send()
};
put_with_transport_headers("kept").await?;
let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
target.drop_unlisted_replication_headers(true);
put_with_transport_headers("dropped").await?;
let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
let requests = target.requests();
for key in ["kept", "dropped"] {
let record = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
.expect("PUT must be journaled");
assert!(
record.proxy_headers.ssec_transport_present,
"the journal must prove the sender shipped the transport headers for {key}"
);
}
let plain_head = requests
.iter()
.find(|record| record.operation == Operation::HeadObject)
.expect("HEAD must be journaled");
assert!(!plain_head.proxy_headers.ssec_transport_present);
target.shutdown().await;
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -3052,6 +3328,7 @@ mod tests {
version_id: index.to_string(),
content_type: None,
metadata: None,
replication_sse_headers: Vec::new(),
parts: BTreeMap::new(),
},
);
@@ -3074,6 +3351,7 @@ mod tests {
},
Some(0),
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3096,6 +3374,7 @@ mod tests {
},
None,
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
{
let bounded_records = lock(&bounded_control);
@@ -30,6 +30,7 @@ use md5::{Digest as Md5Digest, Md5};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::collections::HashMap;
use std::error::Error;
use std::io::Cursor;
@@ -355,6 +356,7 @@ async fn run_post_object_policy_case(
/// smuggles one extra field the policy never declared, and the upload must be
/// rejected with 403 AccessDenied naming the offending field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -482,6 +484,7 @@ async fn test_anonymous_post_object_rejects_fields_missing_from_policy_condition
/// sends a different one, and the upload must be rejected with 400
/// InvalidPolicyDocument naming the field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_exact_condition_policy_mismatches()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -686,6 +689,7 @@ async fn test_anonymous_post_object_rejects_exact_condition_policy_mismatches()
/// one of them with a different value, and the upload must be rejected with
/// 400 InvalidPolicyDocument naming the mismatched field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_policy_mismatches() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -753,6 +757,7 @@ async fn test_anonymous_post_object_rejects_object_lock_policy_mismatches() -> R
/// exact values, the form sends a different parameter value, and the upload
/// must be rejected with 400 InvalidPolicyDocument naming the parameter.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms_policy_mismatches() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -834,6 +839,7 @@ async fn test_anonymous_post_object_rejects_sse_kms_policy_mismatches() -> Resul
/// NotImplemented (SSE-KMS POST uploads are not implemented), not with a
/// policy error.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -888,6 +894,7 @@ async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -961,6 +968,7 @@ async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_requires_auth() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -994,6 +1002,7 @@ async fn test_anonymous_post_object_requires_auth() -> Result<(), Box<dyn std::e
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_honors_success_action_status() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1057,6 +1066,7 @@ async fn test_anonymous_post_object_honors_success_action_status() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_honors_success_action_redirect() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1129,6 +1139,7 @@ async fn test_anonymous_post_object_honors_success_action_redirect() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_defaults_to_no_content() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1174,6 +1185,7 @@ async fn test_anonymous_post_object_defaults_to_no_content() -> Result<(), Box<d
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1220,6 +1232,7 @@ async fn test_anonymous_post_object_rejects_sse_kms() -> Result<(), Box<dyn std:
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1277,6 +1290,7 @@ async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1349,6 +1363,7 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), B
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1422,6 +1437,7 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1472,6 +1488,7 @@ async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1535,6 +1552,7 @@ async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_condition
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_storage_class_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1588,6 +1606,7 @@ async fn test_anonymous_post_object_accepts_storage_class_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_storage_class_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1638,6 +1657,7 @@ async fn test_anonymous_post_object_rejects_storage_class_missing_from_policy_co
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_storage_class_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -1689,6 +1709,7 @@ async fn test_anonymous_post_object_rejects_invalid_storage_class_value() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_algorithm_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1744,6 +1765,7 @@ async fn test_anonymous_post_object_rejects_checksum_algorithm_missing_from_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_algorithm_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1800,6 +1822,7 @@ async fn test_anonymous_post_object_rejects_checksum_algorithm_policy_mismatch()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_auxiliary_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1863,6 +1886,7 @@ async fn test_anonymous_post_object_rejects_checksum_auxiliary_fields_missing_fr
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_allows_sse_c_fields_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1939,6 +1963,7 @@ async fn test_anonymous_post_object_allows_sse_c_fields_outside_policy_condition
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_c_exact_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -1997,6 +2022,7 @@ async fn test_anonymous_post_object_rejects_sse_c_exact_policy_mismatch() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_duplicate_key_form_values() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2046,6 +2072,7 @@ async fn test_anonymous_post_object_rejects_duplicate_key_form_values() -> Resul
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_success_action_status() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -2093,6 +2120,7 @@ async fn test_anonymous_post_object_rejects_invalid_success_action_status() -> R
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_success_action_redirect()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2140,6 +2168,7 @@ async fn test_anonymous_post_object_rejects_invalid_success_action_redirect()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_form_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2194,6 +2223,7 @@ async fn test_anonymous_post_object_rejects_form_fields_missing_from_policy_cond
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_form_fields_covered_by_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2250,6 +2280,7 @@ async fn test_anonymous_post_object_accepts_form_fields_covered_by_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_starts_with_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -2304,6 +2335,7 @@ async fn test_anonymous_post_object_rejects_starts_with_policy_mismatch() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_content_length_range_violation()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2356,6 +2388,7 @@ async fn test_anonymous_post_object_rejects_content_length_range_violation()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_success_action_status_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2412,6 +2445,7 @@ async fn test_anonymous_post_object_accepts_success_action_status_exact_policy_m
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_success_action_redirect_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2468,6 +2502,7 @@ async fn test_anonymous_post_object_rejects_success_action_redirect_policy_misma
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_success_action_redirect_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2533,6 +2568,7 @@ async fn test_anonymous_post_object_accepts_success_action_redirect_exact_policy
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_success_action_redirect_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2585,6 +2621,7 @@ async fn test_anonymous_post_object_rejects_success_action_redirect_missing_from
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_metadata_field_covered_by_starts_with()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2639,6 +2676,7 @@ async fn test_anonymous_post_object_accepts_metadata_field_covered_by_starts_wit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_type_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2696,6 +2734,7 @@ async fn test_anonymous_post_object_accepts_content_type_field_exact_policy_matc
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_type_field_covered_by_starts_with()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2753,6 +2792,7 @@ async fn test_anonymous_post_object_accepts_content_type_field_covered_by_starts
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_disposition_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2810,6 +2850,7 @@ async fn test_anonymous_post_object_accepts_content_disposition_field_exact_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_cache_control_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2867,6 +2908,7 @@ async fn test_anonymous_post_object_accepts_cache_control_field_exact_policy_mat
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_language_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2924,6 +2966,7 @@ async fn test_anonymous_post_object_accepts_content_language_field_exact_policy_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_encoding_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2981,6 +3024,7 @@ async fn test_anonymous_post_object_accepts_content_encoding_field_exact_policy_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_website_redirect_location_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3038,6 +3082,7 @@ async fn test_anonymous_post_object_accepts_website_redirect_location_exact_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_expires_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3095,6 +3140,7 @@ async fn test_anonymous_post_object_accepts_expires_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_retention_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3150,6 +3196,7 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3209,6 +3256,7 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_p
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3261,6 +3309,7 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3319,6 +3368,7 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_policy_mismat
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3376,6 +3426,7 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_missing_from_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_tagging_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3441,6 +3492,7 @@ async fn test_anonymous_post_object_accepts_tagging_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_metadata_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3499,6 +3551,7 @@ async fn test_anonymous_post_object_accepts_metadata_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_allows_x_ignore_fields_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3551,6 +3604,7 @@ async fn test_anonymous_post_object_allows_x_ignore_fields_outside_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sigv4_date_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3603,6 +3657,7 @@ async fn test_anonymous_post_object_rejects_sigv4_date_policy_mismatch() -> Resu
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_mismatched_bucket_form_field() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -3657,6 +3712,7 @@ async fn test_anonymous_post_object_rejects_mismatched_bucket_form_field() -> Re
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_multiple_bucket_values() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3708,6 +3764,7 @@ async fn test_anonymous_post_object_rejects_multiple_bucket_values() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_extra_content_disposition_field()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3763,6 +3820,7 @@ async fn test_anonymous_post_object_rejects_extra_content_disposition_field()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tar_entries_with_prefix_headers()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3833,6 +3891,7 @@ async fn test_signed_put_object_extract_expands_tar_entries_with_prefix_headers(
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_request_metadata_on_extracted_objects()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3897,6 +3956,7 @@ async fn test_signed_put_object_extract_preserves_request_metadata_on_extracted_
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3944,6 +4004,7 @@ async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Resul
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3986,6 +4047,7 @@ async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_invalid_storage_class() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4021,6 +4083,7 @@ async fn test_signed_put_object_extract_rejects_invalid_storage_class() -> Resul
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_rejects_write_offset_bytes_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4074,6 +4137,7 @@ async fn test_signed_put_object_rejects_write_offset_bytes_header() -> Result<()
}
#[tokio::test]
#[serial]
async fn test_raw_signed_put_object_write_offset_bytes_returns_minio_compatible_error_body()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4112,6 +4176,7 @@ async fn test_raw_signed_put_object_write_offset_bytes_returns_minio_compatible_
}
#[tokio::test]
#[serial]
async fn test_anonymous_put_object_write_offset_bytes_returns_minio_compatible_error_body()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4170,6 +4235,7 @@ async fn test_anonymous_put_object_write_offset_bytes_returns_minio_compatible_e
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4234,6 +4300,7 @@ async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_bucket_default_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4289,6 +4356,7 @@ async fn test_signed_put_object_extract_rejects_bucket_default_sse_kms() -> Resu
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_sse_c() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4353,6 +4421,7 @@ async fn test_signed_put_object_extract_preserves_sse_c() -> Result<(), Box<dyn
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_object_lock_legal_hold() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -4407,6 +4476,7 @@ async fn test_signed_put_object_extract_preserves_object_lock_legal_hold() -> Re
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -4466,6 +4536,7 @@ async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Res
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_pax_retention_overrides_request_retention()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4529,6 +4600,7 @@ async fn test_signed_put_object_extract_pax_retention_overrides_request_retentio
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4562,6 +4634,7 @@ async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_entry_mtime() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4597,6 +4670,7 @@ async fn test_signed_put_object_extract_preserves_entry_mtime() -> Result<(), Bo
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4650,6 +4724,7 @@ async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retention_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4959,6 +5034,7 @@ async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retent
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5000,6 +5076,7 @@ async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Bo
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_directory_markers_by_default()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5060,6 +5137,7 @@ async fn test_signed_put_object_extract_preserves_directory_markers_by_default()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tar_gz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5111,6 +5189,7 @@ async fn test_signed_put_object_extract_expands_tar_gz_archive() -> Result<(), B
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tgz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5162,6 +5241,7 @@ async fn test_signed_put_object_extract_expands_tgz_archive() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tbz2_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5213,6 +5293,7 @@ async fn test_signed_put_object_extract_expands_tbz2_archive() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_txz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5264,6 +5345,7 @@ async fn test_signed_put_object_extract_expands_txz_archive() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5337,6 +5419,7 @@ async fn test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_e
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_normalizes_prefix_header_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5379,6 +5462,7 @@ async fn test_signed_put_object_extract_normalizes_prefix_header_value() -> Resu
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tzst_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5430,6 +5514,7 @@ async fn test_signed_put_object_extract_expands_tzst_archive() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_missing_archive_extension() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -5463,6 +5548,7 @@ async fn test_signed_put_object_extract_rejects_missing_archive_extension() -> R
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_invalid_tar_gz_payload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -33,6 +33,7 @@ use aws_sdk_s3::types::{
ObjectLockMode, ObjectLockRetentionMode,
};
use chrono::{DateTime, Duration, Utc};
use serial_test::serial;
use tracing::info;
/// Initialize test logging
@@ -106,6 +107,7 @@ fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc>
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_compliance_retention() {
init_logging();
info!("🧪 Test: DeleteObject blocked by COMPLIANCE retention");
@@ -143,6 +145,7 @@ async fn test_delete_object_blocked_by_compliance_retention() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_governance_without_bypass() {
init_logging();
info!("🧪 Test: DeleteObject blocked by GOVERNANCE retention without bypass");
@@ -172,6 +175,7 @@ async fn test_delete_object_blocked_by_governance_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_by_governance_with_bypass() {
init_logging();
info!("🧪 Test: DeleteObject allowed by GOVERNANCE retention with bypass");
@@ -211,6 +215,7 @@ async fn test_delete_object_allowed_by_governance_with_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for retained current version");
@@ -261,6 +266,7 @@ async fn test_delete_object_creates_delete_marker_for_retained_current_version()
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: DeleteObject blocked by Legal Hold");
@@ -293,6 +299,7 @@ async fn test_delete_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_with_legal_hold_off() {
init_logging();
info!("🧪 Test: DeleteObject allowed with Legal Hold OFF");
@@ -328,6 +335,7 @@ async fn test_delete_object_allowed_with_legal_hold_off() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_after_legal_hold_removed() {
init_logging();
info!("🧪 Test: DeleteObject succeeds after Legal Hold is removed");
@@ -361,6 +369,7 @@ async fn test_delete_object_after_legal_hold_removed() {
}
#[tokio::test]
#[serial]
async fn test_get_object_legal_hold_returns_updated_status() {
init_logging();
info!("🧪 Test: GetObjectLegalHold returns updated status");
@@ -416,6 +425,7 @@ async fn test_get_object_legal_hold_returns_updated_status() {
}
#[tokio::test]
#[serial]
async fn test_get_object_retention_returns_configured_values() {
init_logging();
info!("🧪 Test: GetObjectRetention returns configured values");
@@ -466,6 +476,7 @@ async fn test_get_object_retention_returns_configured_values() {
// creating a new current version. The lock protects the existing version
// from deletion; it never blocks new versions.
#[tokio::test]
#[serial]
async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: PutObject overwrite of a legal-hold version creates a new version");
@@ -550,6 +561,7 @@ async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject applies requested Legal Hold");
@@ -601,6 +613,7 @@ async fn test_copy_object_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_does_not_inherit_source_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject does not inherit source Legal Hold");
@@ -694,6 +707,7 @@ async fn test_copy_object_does_not_inherit_source_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject overwrite of a legal-hold destination creates a new version");
@@ -773,6 +787,7 @@ async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CreateMultipartUpload applies requested Legal Hold");
@@ -838,6 +853,7 @@ async fn test_create_multipart_upload_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CreateMultipartUpload over a COMPLIANCE-retained key creates a new version");
@@ -917,6 +933,7 @@ async fn test_create_multipart_upload_creates_new_version_under_compliance_reten
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by Legal Hold");
@@ -976,6 +993,7 @@ async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_retention() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by retention");
@@ -1037,6 +1055,7 @@ async fn test_delete_completed_multipart_object_blocked_by_retention() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under Legal Hold");
@@ -1116,6 +1135,7 @@ async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under COMPLIANCE retention");
@@ -1189,6 +1209,7 @@ async fn test_complete_multipart_upload_creates_new_version_under_compliance_ret
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_legal_hold_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectLegalHold permission");
@@ -1252,6 +1273,7 @@ async fn test_write_paths_require_put_object_legal_hold_permission() {
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_retention_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectRetention permission");
@@ -1323,6 +1345,7 @@ async fn test_write_paths_require_put_object_retention_permission() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_objects_mixed_locked_unlocked() {
init_logging();
info!("🧪 Test: DeleteObjects with mixed locked and unlocked objects");
@@ -1404,6 +1427,7 @@ async fn test_delete_objects_mixed_locked_unlocked() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_cannot_shorten() {
init_logging();
info!("🧪 Test: PutObjectRetention cannot shorten COMPLIANCE retention");
@@ -1444,6 +1468,7 @@ async fn test_put_retention_compliance_cannot_shorten() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_can_extend() {
init_logging();
info!("🧪 Test: PutObjectRetention can extend COMPLIANCE retention");
@@ -1484,6 +1509,7 @@ async fn test_put_retention_compliance_can_extend() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_extend_without_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE can extend without bypass");
@@ -1527,6 +1553,7 @@ async fn test_put_retention_governance_extend_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_shorten_requires_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE requires bypass to shorten");
@@ -1588,6 +1615,7 @@ async fn test_put_retention_governance_shorten_requires_bypass() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_default_retention_applied_to_new_objects() {
init_logging();
info!("🧪 Test: Default retention is applied to new objects");
@@ -1657,6 +1685,7 @@ async fn test_default_retention_applied_to_new_objects() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
@@ -1741,6 +1770,7 @@ async fn test_delete_object_creates_delete_marker_for_default_retained_current_v
}
#[tokio::test]
#[serial]
async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
init_logging();
info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
@@ -1839,6 +1869,7 @@ async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_retention_uses_destination_policy() {
init_logging();
info!("🧪 Test: CopyObject retention follows destination policy");
@@ -2020,6 +2051,7 @@ async fn test_copy_object_retention_uses_destination_policy() {
}
#[tokio::test]
#[serial]
async fn test_multipart_default_retention_fixed_at_create() {
init_logging();
info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
@@ -2090,6 +2122,7 @@ async fn test_multipart_default_retention_fixed_at_create() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
init_logging();
info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
@@ -2210,6 +2243,7 @@ async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
}
#[tokio::test]
#[serial]
async fn test_versioning_auto_enabled_with_object_lock() {
init_logging();
info!("🧪 Test: Versioning is auto-enabled when Object Lock is configured");
@@ -2268,6 +2302,7 @@ async fn test_versioning_auto_enabled_with_object_lock() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_error_message_distinguishes_legal_hold_from_retention() {
init_logging();
info!("🧪 Test: Error messages distinguish Legal Hold from Retention");
File diff suppressed because it is too large Load Diff
-1
View File
@@ -273,7 +273,6 @@ proptest = "1"
rcgen.workspace = true
insta = { workspace = true, features = ["yaml", "json"] }
rustfs-crypto = { workspace = true }
tonic-prost = { workspace = true }
[build-dependencies]
shadow-rs = { workspace = true, default-features = false, features = ["build", "metadata"] }
+8 -7
View File
@@ -32,7 +32,7 @@ pub mod bucket {
pub mod bucket_target_sys {
pub use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
TargetClient, append_version_id_query,
SsecPassthroughCapability, TargetClient, append_version_id_query,
};
}
@@ -198,12 +198,13 @@ pub mod bucket {
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, unsupported_replication_config_field,
validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
get_global_replication_stats, get_proxy_targets, init_background_replication,
invalid_replication_config_status_field, persist_force_delete_intent, read_durable_mrf_backlog,
replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map, replication_target_arns,
resync_start_conflict_id, should_remove_replication_target, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
};
}
+306 -4
View File
@@ -27,10 +27,15 @@ use aws_sdk_s3::config::SharedHttpClient;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::error::SdkError;
use aws_sdk_s3::operation::complete_multipart_upload::CompleteMultipartUploadOutput;
use aws_sdk_s3::operation::delete_object_tagging::{DeleteObjectTaggingError, DeleteObjectTaggingOutput};
use aws_sdk_s3::operation::get_object::{GetObjectError, GetObjectOutput};
use aws_sdk_s3::operation::get_object_tagging::{GetObjectTaggingError, GetObjectTaggingOutput};
use aws_sdk_s3::operation::head_bucket::HeadBucketError;
use aws_sdk_s3::operation::head_object::HeadObjectError;
use aws_sdk_s3::operation::put_object_tagging::{PutObjectTaggingError, PutObjectTaggingOutput};
use aws_sdk_s3::operation::upload_part::UploadPartOutput;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::Tagging as SdkTagging;
use aws_sdk_s3::types::{
ChecksumMode, CompletedMultipartUpload, CompletedPart, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
};
@@ -57,8 +62,8 @@ use rustfs_utils::http::{
is_rustfs_header, is_standard_header, is_storageclass_header,
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_PROXY_REQUEST,
SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
insert_header,
};
@@ -294,9 +299,41 @@ struct TargetClientBuildProbe {
release: Arc<tokio::sync::Semaphore>,
}
/// SSE-C passthrough capability verdicts (see the enum's own docs in
/// `rustfs-replication`) are cached here per target ARN: entries follow the
/// `arn_remotes_map` lifecycle (rebuilding or removing a target resets its
/// capability to `Unknown`) and additionally expire after
/// [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], after which the next attempt
/// re-audits. Re-exported so existing `bucket_target_sys` consumers keep
/// their import path while the verdict vocabulary lives with the
/// replication decision logic.
pub use crate::bucket::replication::SsecPassthroughCapability;
/// How long an audited SSE-C passthrough verdict stays authoritative.
///
/// Trade-off: without a TTL a verdict is sticky for the process lifetime —
/// an `Unsupported` target that gets upgraded (or re-probed only via
/// replication-check) would keep failing SSE-C replication forever, and the
/// fail-open twin: a `Supported` verdict would outlive a backend swapped
/// behind the same endpoint/ARN. With the TTL, a bad target costs at most
/// one wasted PUT+HEAD audit per TTL window, and a changed backend is
/// re-discovered within the same window.
pub const SSEC_PASSTHROUGH_CAPABILITY_TTL: Duration = Duration::from_secs(10 * 60);
/// A recorded SSE-C passthrough verdict plus when it was recorded, so reads
/// can report staleness against [`SSEC_PASSTHROUGH_CAPABILITY_TTL`].
#[derive(Debug, Clone, Copy)]
struct SsecPassthroughRecord {
capability: SsecPassthroughCapability,
recorded_at: Instant,
}
#[derive(Debug, Default)]
pub struct BucketTargetSys {
pub arn_remotes_map: Arc<RwLock<HashMap<String, ArnTarget>>>,
/// SSE-C passthrough capability verdicts keyed by target ARN. See
/// [`SsecPassthroughCapability`]; reset alongside `arn_remotes_map`.
ssec_passthrough_map: Arc<RwLock<HashMap<String, SsecPassthroughRecord>>>,
pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
pub h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
target_h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
@@ -317,6 +354,7 @@ impl BucketTargetSys {
fn new() -> Self {
Self {
arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
targets_map: Arc::new(RwLock::new(HashMap::new())),
h_mutex: Arc::new(RwLock::new(HashMap::new())),
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
@@ -580,19 +618,59 @@ impl BucketTargetSys {
let update_mutex = self.target_update_mutex(bucket).await;
let _update_guard = update_mutex.lock().await;
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await;
if let Some(targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in targets {
arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn);
ssec_map.remove(&target.arn);
}
}
}
/// Cached SSE-C passthrough capability for a target ARN, plus whether the
/// verdict is older than [`SSEC_PASSTHROUGH_CAPABILITY_TTL`]. `(Unknown,
/// false)` when no verdict has been recorded since the target was built.
/// Staleness is computed here so the gate policy stays a pure function.
pub async fn ssec_passthrough_capability(&self, arn: &str) -> (SsecPassthroughCapability, bool) {
match self.ssec_passthrough_map.read().await.get(arn) {
Some(record) => (record.capability, record.recorded_at.elapsed() >= SSEC_PASSTHROUGH_CAPABILITY_TTL),
None => (SsecPassthroughCapability::Unknown, false),
}
}
/// Record an audited SSE-C passthrough verdict for a target ARN. Written by
/// the replication worker's HEAD-back audit and by the replication-check
/// SsecPassthrough probe phase.
pub async fn record_ssec_passthrough_capability(&self, arn: &str, capability: SsecPassthroughCapability) {
self.ssec_passthrough_map.write().await.insert(
arn.to_string(),
SsecPassthroughRecord {
capability,
recorded_at: Instant::now(),
},
);
}
/// Test hook: age an existing verdict so TTL expiry is observable without
/// waiting out the real window.
#[cfg(test)]
pub(crate) async fn backdate_ssec_passthrough_capability(&self, arn: &str, age: Duration) {
let backdated = Instant::now()
.checked_sub(age)
.expect("system uptime must exceed the backdate age");
if let Some(record) = self.ssec_passthrough_map.write().await.get_mut(arn) {
record.recorded_at = backdated;
}
}
pub async fn set_target(
&self,
bucket: &str,
@@ -948,15 +1026,21 @@ impl BucketTargetSys {
}
}
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await;
// Remove existing targets
if let Some(existing_targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in existing_targets {
arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn);
// A rebuilt/edited target may point at a different service:
// the SSE-C passthrough verdict must be re-audited from Unknown.
ssec_map.remove(&target.arn);
self.update_bandwidth_limit(bucket, &target.arn, 0);
}
}
@@ -1446,6 +1530,43 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
}
}
/// Resolve the S3 `versionId` for a proxied read against a remote target.
/// RustFS represents the null version internally as the nil UUID while the S3
/// API addresses it as the literal "null" (same mapping as
/// [`resolve_put_api_version_id`]); empty means "no version requested".
pub(crate) fn resolve_read_api_version_id(version_id: Option<String>) -> Option<String> {
let version_id = version_id?;
let trimmed = version_id.trim();
if trimmed.is_empty() {
None
} else if Uuid::parse_str(trimmed).is_ok_and(|uuid| uuid.is_nil()) {
Some(rustfs_filemeta::NULL_VERSION_ID.to_string())
} else {
Some(trimmed.to_string())
}
}
/// Outbound header set for a proxied read: the caller-provided passthrough
/// headers (client SSE-C key family, conditional headers) plus the anti-loop
/// `source-proxy-request` marker in both the x-rustfs- and x-minio- prefixes
/// (a MinIO target only understands the latter). Never adds
/// `source-replication-check`: that exemption channel belongs exclusively to
/// the replication worker's HEAD.
fn proxy_outbound_headers(mut extra_headers: HeaderMap) -> HeaderMap {
insert_header(&mut extra_headers, SUFFIX_SOURCE_PROXY_REQUEST, "true");
extra_headers
}
/// Copy `headers` onto an SDK request inside `customize().map_request` (runs
/// before signing, so the headers join the SigV4 canonical request).
fn apply_extra_headers(mut req: HttpRequest, headers: &HeaderMap) -> Result<HttpRequest, std::convert::Infallible> {
for (k, v) in headers.iter() {
req.headers_mut()
.insert(k.as_str().to_string(), v.to_str().unwrap_or("").to_string());
}
Ok(req)
}
/// Append `versionId=<id>` to an already-built request URI. aws-sdk-s3's
/// `PutObjectInput` / `CreateMultipartUploadInput` expose no version id
/// member, so the query is spliced in via `map_request`, which runs at
@@ -1853,6 +1974,13 @@ impl TargetClient {
// worker cannot hold; otherwise SSE-C replicas never converge on HEAD.
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
// `source-proxy-request: false` (MinIO `ProxyHeaderSet` semantics):
// the header's mere presence tells the receiver to answer LOCALLY
// instead of proxying the miss back to us. Without it, a not-found on
// the target gets read-proxied back to this source, echoes the source
// object with an identical ETag, and the worker concludes the object
// already converged — so it never actually replicates it.
insert_header(&mut headers, SUFFIX_SOURCE_PROXY_REQUEST, "false");
match self
.client
.head_object()
@@ -1877,6 +2005,129 @@ impl TargetClient {
}
}
/// HEAD used by the read-proxy path (GET/HEAD of an object not yet
/// replicated locally, MinIO `proxyHeadToRepTarget`).
///
/// Deliberately different from [`TargetClient::head_object`]: it must NOT
/// send `source-replication-check` — that header is the replication
/// worker's SSE-C metadata exemption channel. A proxied client request
/// instead forwards the client's own SSE-C headers (`extra_headers`) so
/// the target performs the real SSE-C validation/decryption. The
/// `source-proxy-request` marker is always added so the target does not
/// proxy the request onward (anti-loop).
pub async fn head_object_for_proxy(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
range: Option<String>,
part_number: Option<i32>,
extra_headers: HeaderMap,
) -> Result<HeadObjectOutput, SdkError<HeadObjectError>> {
let headers = proxy_outbound_headers(extra_headers);
self.client
.head_object()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.set_range(range)
.set_part_number(part_number)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// GET used by the read-proxy path (MinIO `proxyGetToReplicationTarget`).
/// Returns the streaming SDK output; callers must forward the body without
/// buffering it. Same header contract as [`Self::head_object_for_proxy`]:
/// anti-loop marker on, replication-check never sent, client SSE-C /
/// conditional headers forwarded verbatim via `extra_headers`.
pub async fn get_object(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
range: Option<String>,
part_number: Option<i32>,
extra_headers: HeaderMap,
) -> Result<GetObjectOutput, SdkError<GetObjectError>> {
let headers = proxy_outbound_headers(extra_headers);
self.client
.get_object()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.set_range(range)
.set_part_number(part_number)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// GetObjectTagging for the tagging read-proxy path
/// (MinIO `proxyGetTaggingToRepTarget`). Anti-loop marker always added.
pub async fn get_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
) -> Result<GetObjectTaggingOutput, SdkError<GetObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.get_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// PutObjectTagging for the tagging proxy path
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
pub async fn put_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
tagging: SdkTagging,
) -> Result<PutObjectTaggingOutput, SdkError<PutObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.put_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.tagging(tagging)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// DeleteObjectTagging for the tagging proxy path
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
pub async fn delete_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
) -> Result<DeleteObjectTaggingOutput, SdkError<DeleteObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.delete_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// On success returns the version id the target assigned (from
/// `x-amz-version-id`), letting callers audit the version-identity
/// contract — a target that adopts the source version echoes it back.
@@ -2506,6 +2757,57 @@ mod tests {
assert_eq!(health.last_online, Some(now));
}
/// N2 TTL contract, both flip directions: a recorded verdict is fresh
/// until [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], then reads as expired; a
/// re-audit that records the OPPOSITE verdict replaces it as fresh. The
/// worker gate maps expired verdicts to ProceedWithAudit (pinned in
/// `replication_target_boundary`), so together this proves an Unsupported
/// target recovers to Supported through the audit once its verdict ages
/// out — and a stale Supported one is re-proven rather than trusted.
#[tokio::test]
async fn ssec_passthrough_capability_ttl_expires_and_reaudit_flips_verdict() {
let sys = BucketTargetSys::default();
let arn = "arn:rustfs:replication:us-east-1:bucket:ssec-ttl";
let expired_age = SSEC_PASSTHROUGH_CAPABILITY_TTL + Duration::from_secs(1);
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unknown, false),
"an unrecorded target must read Unknown and never expired"
);
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Unsupported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, false)
);
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, true),
"an aged-out Unsupported verdict must read expired so the gate re-audits"
);
// The re-audit against an upgraded target records Supported afresh.
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Supported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, false),
"a fresh Supported verdict replaces the expired Unsupported one"
);
// And the fail-open twin: Supported also ages out.
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, true),
"an aged-out Supported verdict must read expired so the gate re-proves it"
);
}
#[tokio::test]
async fn list_targets_applies_health_stats_by_arn_and_preserves_endpoint_port() {
let sys = BucketTargetSys::default();
@@ -11,9 +11,9 @@ paths.
| Module | Current role | Split blocker |
|---|---|---|
| `config.rs` | Replication config helpers, rule matching, and tag filtering. | Uses replication-local filemeta/tagging boundaries and S3 DTOs directly. |
| `datatypes.rs` | ECStore compatibility re-export for resync status enums. | Re-exports `rustfs-replication` contracts while downstream facade consumers migrate. |
| `replication_object_decision_boundary.rs` | Object replication option DTOs, resync target projection, delete replication decisions, and multipart planning helpers. | Keeps ECStore runtime modules from importing object decision contracts directly from `rustfs-replication`. |
| `replication_pool.rs` | Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys, metadata paths, queue contracts through the queue boundary, file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
| `replication_proxy.rs` | Proxy-target selection for GET/HEAD/Tagging reads of objects not yet replicated locally (MinIO `getProxyTargets` parity: anti-loop, version-suspended, and no-config empty branches). | Uses replication config lookup, rule matching, and target clients through local boundaries. |
| `replication_queue_boundary.rs` | Queue/admission DTOs, heal queue DTOs, worker sizing, and backpressure helpers. | Keeps ECStore runtime modules from importing queue/backpressure contracts directly from `rustfs-replication`. |
| `replication_resync_boundary.rs` | Resync DTOs, status classifiers, persisted resync/MRF codec wrappers, and ECStore error mapping. | Keeps ECStore runtime modules from importing resync contract helpers directly from `rustfs-replication`. |
| `replication_resyncer.rs` | Object replication, delete replication, resync execution, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, object decisions and multipart planning through the object decision boundary, resync contracts through the resync boundary, queue DTOs through the queue boundary, error contracts through the error boundary, versioning systems, storage contracts through the replication storage boundary, config-derived storage class labels through the config store, runtime sources, notification events and local event host selection through the event sink, bandwidth reader wrapping, and SetDisks lock timing. |
@@ -117,9 +117,12 @@ Target end state:
their file names — so batch-merging them beforehand is explicitly rejected:
it forces synchronized guard-script/mod/import churn with zero functional
gain;
- the only module that can retire early is `datatypes.rs`: delete it once its
facade consumers import the resync status enums through `rustfs-replication`
directly.
- `datatypes.rs` retired early (its sanctioned exception): it was a pure
relay (`boundary -> datatypes -> mod.rs`), so the facade now re-exports
`ResyncStatusType` from the resync boundary directly and the relay file is
deleted. Note the original retirement wording ("consumers import through
`rustfs-replication` directly") conflicted with Migration Rule #15
consumers stay behind the ECStore facade; only the relay hop dissolves.
## Milestones
@@ -127,9 +130,9 @@ Target end state:
|---|---|---|
| M0 | Record the completion criteria and end state (this section). | Done |
| M1 | Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts. | Done — see Required Contracts |
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical |
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Done — moved the pure decision helpers with their unit tests: `resync_status_duration` (resync), `resync_existing_delete_replication_info` / `replicate_delete_outcome` / `target_delete_version_id` / `delete_marker_purge_version_id` / `delete_marker_purge_mrf_entry` (delete), `version_identity_drifted` / `is_replication_target_offline_error` / the SSE-C passthrough gate family incl. `SsecPassthroughCapability` (object; `ssec_passthrough_evidence_present` was param-demoted to the echoed customer-algorithm string, ECStore keeps the `HeadObjectOutput` adapter). ECStore imports them through the resync/object-decision/target boundaries; `bucket_target_sys` keeps only the verdict cache + TTL and re-exports the capability enum. Not moved (signatures carry ECStore or aws-sdk types): `verify_resync_head_result`, `resync_target_error_detail`, the `SdkError` classifiers (`has_raw_status`, `is_version_id_format_mismatch`), the `replicate_all_*` option/info builders, and `bounded_resync_max_jobs` (itself a pure clamp, but it forms one local configuration unit with the env-reading `configured_resync_max_jobs` and its ECStore-local constants — moving the clamp alone has negative value). |
| M3 | Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last. | Pending |
| M4 | Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`. | Pending |
| M4 | Retire the boundary modules together with their guard-script entries. | Pending (`datatypes.rs` already retired early alongside M2) |
The original first code-bearing step (narrow `ReplicationEventSink` /
`ReplicationRuntime` contracts) has landed — `replication_event_sink.rs`
@@ -1,15 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub use super::replication_resync_boundary::ResyncStatusType;
+4 -2
View File
@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub mod datatypes;
mod replication_bandwidth_boundary;
mod replication_config_boundary;
mod replication_config_store;
@@ -29,6 +28,7 @@ mod replication_object_bridge;
mod replication_object_config;
mod replication_object_decision_boundary;
pub(crate) mod replication_pool;
mod replication_proxy;
mod replication_queue_boundary;
mod replication_resync_boundary;
mod replication_resyncer;
@@ -43,7 +43,6 @@ pub(crate) mod replication_timing;
mod replication_versioning_boundary;
mod runtime_boundary;
pub use datatypes::ResyncStatusType;
pub use replication_config_boundary::{
ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
@@ -74,13 +73,16 @@ pub use replication_pool::{
get_global_replication_pool, get_global_replication_stats, init_background_replication, persist_force_delete_intent,
read_durable_mrf_backlog, resync_start_conflict_id,
};
pub use replication_proxy::get_proxy_targets;
pub use replication_queue_boundary::{
DeletedObjectReplicationInfo, ReplicationBatchAdmission, ReplicationHealQueueResult, ReplicationOperation,
ReplicationPriority, ReplicationQueueAdmission,
};
pub use replication_resync_boundary::ResyncStatusType;
pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
pub use replication_scanner_bridge::ReplicationScannerBridge;
pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
pub use replication_target_boundary::SsecPassthroughCapability;
pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;
@@ -12,12 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_filemeta::NULL_VERSION_ID;
pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
pub(crate) use rustfs_replication::{
REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos,
ReplicatedTargetInfo, ReplicationAction, ReplicationWorkerOperation, ResyncDecision, get_replication_state,
parse_replicate_decision, replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
REPLICATE_EXISTING, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction,
ReplicationWorkerOperation, ResyncDecision, get_replication_state, parse_replicate_decision,
replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
};
pub use rustfs_replication::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationState, ReplicationStatusType, ReplicationType,
@@ -18,9 +18,10 @@ pub use rustfs_replication::{
should_use_existing_delete_replication_source,
};
pub(crate) use rustfs_replication::{
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject,
delete_replication_missing_source_decision, delete_replication_object_opts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, resync_target_for_object,
should_retry_delete_marker_purge,
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject, delete_marker_purge_mrf_entry,
delete_marker_purge_version_id, delete_replication_missing_source_decision, delete_replication_object_opts,
heal_uses_delete_replication_path, is_retryable_delete_replication_head_error, is_version_delete_replication,
replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
replication_multipart_part_plan, resync_existing_delete_replication_info, resync_target_for_object,
should_retry_delete_marker_purge, target_delete_version_id,
};
@@ -667,6 +667,368 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
Err(EcstoreError::PreconditionFailed)
}
/// Acquires the MRF recovery leader lock for the startup replay.
/// Returns `None` (after logging) when the lock cannot be created or another
/// node is already processing the backlog.
async fn acquire_mrf_recovery_guard<S: ReplicationStorage>(storage: &Arc<S>) -> Option<rustfs_lock::NamespaceLockGuard> {
let recovery_lock = match storage
.new_ns_lock(
ReplicationMetadataStore::rustfs_meta_bucket(),
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
)
.await
{
Ok(lock) => lock,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to create the MRF recovery leader lock"
);
return None;
}
};
match recovery_lock
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
.await
{
Ok(guard) => Some(guard),
Err(_) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
"Another node is already processing the MRF recovery backlog"
);
None
}
}
}
/// Reads and decodes the on-disk MRF recovery file.
/// Returns `None` when there is nothing to replay: missing file (publishes an
/// empty available summary), read failure, or corrupt data (quarantined).
async fn load_mrf_recovery_entries<S: ReplicationStorage>(storage: &Arc<S>) -> Option<Vec<MrfReplicateEntry>> {
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
Ok(d) => d,
Err(EcstoreError::ConfigNotFound) => {
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
available: true,
buckets: Vec::new(),
});
return None;
}
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to load MRF recovery file"
);
return None;
}
};
match decode_mrf_file(&data) {
Ok(v) => Some(v),
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to decode MRF recovery file — preserving corrupt data"
);
quarantine_mrf_file(storage, &data).await;
None
}
}
}
/// Replays one MRF recovery entry by operation kind.
/// Returns `None` when the entry is skipped entirely (no admission outcome);
/// entries that must be retried later are pushed onto `retry_entries`.
async fn replay_mrf_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
match entry.op {
MrfOpKind::Delete => replay_mrf_delete_entry(entry, storage, retry_entries).await,
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
replay_mrf_object_entry(entry, storage, retry_entries).await
}
MrfOpKind::Metadata => replay_mrf_metadata_entry(entry, storage, retry_entries).await,
}
}
/// Replays a delete-kind MRF entry: force-delete intents replay directly,
/// stale force-delete generations are skipped, and plain deletes are
/// reconstructed as heal deletes.
async fn replay_mrf_delete_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
if should_replay_force_delete_intent(entry) {
let operation_id = entry.force_delete_id?;
let delete = force_delete_heal_replication_info(entry, operation_id);
if replicate_delete_with_outcome(delete, storage.clone()).await {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
} else if entry.force_delete_id.is_some() {
Some(ReplicationQueueAdmission::Skipped)
} else {
replay_mrf_reconstructed_delete(entry, storage, retry_entries).await
}
}
/// Pure DTO construction: heal replication info for a replayed force-delete intent.
fn force_delete_heal_replication_info(entry: &MrfReplicateEntry, operation_id: uuid::Uuid) -> DeletedObjectReplicationInfo {
DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
force_delete: true,
force_delete_id: Some(operation_id),
force_delete_target_arns: entry.target_arns.clone(),
force_delete_generation: entry.force_delete_generation,
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
}
}
/// Reconstruct a heal delete and re-queue it. We do NOT call
/// get_object_info here because the delete-marker or version may
/// already be absent from the local store — that is expected.
async fn replay_mrf_reconstructed_delete<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
..Default::default()
};
let dsc = resolve_mrf_delete_replicate_decision(entry, &oi, versioned, retry_entries).await?;
let dv = reconstructed_heal_delete_info(entry, &oi, &dsc);
if replicate_delete_with_outcome(dv, storage.clone()).await {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
/// The MRF entry does not persist the replication decision and the
/// source object is gone, so re-derive the decision from the live
/// bucket config (mirroring get_heal_replicate_object_info) and set
/// it on the reconstructed delete. Without this the decision string
/// is empty and the delete replicates to zero targets — a silent
/// no-op that leaves replicas diverged (backlog#858 / #799 B9).
async fn resolve_mrf_delete_replicate_decision(
entry: &MrfReplicateEntry,
oi: &ObjectInfo,
versioned: bool,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicateDecision> {
if entry.target_arns.is_empty() {
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
Ok(None) => None,
Err(_) => {
retry_entries.push(entry.clone());
None
}
Ok(Some(_)) => match check_replicate_delete_strict(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
..Default::default()
},
oi,
&ObjectOptions {
versioned,
..Default::default()
},
None,
)
.await
{
Ok(dsc) => Some(dsc),
Err(_) => {
retry_entries.push(entry.clone());
None
}
},
}
} else {
Some(replicate_decision_for_admitted_targets(&entry.target_arns))
}
}
/// Pure DTO construction: reconstructed heal delete carrying the re-derived
/// replication decision.
fn reconstructed_heal_delete_info(
entry: &MrfReplicateEntry,
oi: &ObjectInfo,
dsc: &ReplicateDecision,
) -> DeletedObjectReplicationInfo {
let mut rstate = oi.replication_state();
rstate.replicate_decision_str = dsc.to_string();
let delete_marker_mtime = entry
.delete_marker_mtime
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
delete_marker_mtime,
force_delete: entry.force_delete,
replication_state: Some(rstate),
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
}
}
/// Replays an Object/Heal/ExistingObject MRF entry against the live source object.
async fn replay_mrf_object_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
return None;
}
};
if entry.target_arns.is_empty() {
// Legacy entries predate target admission persistence. They cannot
// be safely attributed, so retain the old live-config fallback.
Some(queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
} else {
let roi = admitted_mrf_replicate_object(oi, entry, entry.op.replication_type());
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
}
/// Replays a metadata-kind MRF entry against the live source object.
async fn replay_mrf_metadata_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF metadata recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
return None;
}
};
if entry.target_arns.is_empty() {
Some(queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
} else {
let roi = admitted_mrf_replicate_object(oi, entry, ReplicationType::Metadata);
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
}
/// Pure DTO construction: replicate-object info for an entry with persisted
/// admitted targets, carrying over the entry's retry count.
fn admitted_mrf_replicate_object(oi: ObjectInfo, entry: &MrfReplicateEntry, op_type: ReplicationType) -> ReplicateObjectInfo {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, op_type);
roi.retry_count = entry.retry_count.max(0) as u32;
roi
}
/// Acknowledges the replayed MRF prefix and returns the retained backlog.
/// On acknowledgement failure the backlog is preserved for the next startup and
/// re-read (falling back to the replayed snapshot) so the published summary stays accurate.
async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
storage: &Arc<S>,
recovery_guard: &rustfs_lock::NamespaceLockGuard,
entries: &[MrfReplicateEntry],
retry_entries: &[MrfReplicateEntry],
) -> Vec<MrfReplicateEntry> {
match acknowledge_mrf_recovery(storage.clone(), recovery_guard, entries, retry_entries).await {
Ok(retained) => retained,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
);
match read_mrf_entries(storage.clone()).await {
Ok(current) => current,
Err(read_error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %read_error,
"Failed to refresh the MRF backlog after acknowledgement failure"
);
entries.to_vec()
}
}
}
}
}
#[derive(Debug, thiserror::Error)]
#[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
struct ResyncActiveConflictError {
@@ -1221,71 +1583,12 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let storage = self.storage.clone();
let handle = tokio::spawn(async move {
let recovery_lock = match storage
.new_ns_lock(
ReplicationMetadataStore::rustfs_meta_bucket(),
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
)
.await
{
Ok(lock) => lock,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to create the MRF recovery leader lock"
);
return;
}
};
let recovery_guard = match recovery_lock
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
.await
{
Ok(guard) => guard,
Err(_) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
"Another node is already processing the MRF recovery backlog"
);
return;
}
let Some(recovery_guard) = acquire_mrf_recovery_guard(&storage).await else {
return;
};
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
Ok(d) => d,
Err(EcstoreError::ConfigNotFound) => {
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
available: true,
buckets: Vec::new(),
});
return;
}
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to load MRF recovery file"
);
return;
}
};
let entries = match decode_mrf_file(&data) {
Ok(v) => v,
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to decode MRF recovery file — preserving corrupt data"
);
quarantine_mrf_file(&storage, &data).await;
return;
}
let Some(entries) = load_mrf_recovery_entries(&storage).await else {
return;
};
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
@@ -1294,187 +1597,8 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let mut retry_entries = Vec::new();
for entry in entries.iter() {
let admission = match entry.op {
MrfOpKind::Delete => {
if should_replay_force_delete_intent(entry) {
let Some(operation_id) = entry.force_delete_id else {
continue;
};
let delete = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
force_delete: true,
force_delete_id: Some(operation_id),
force_delete_target_arns: entry.target_arns.clone(),
force_delete_generation: entry.force_delete_generation,
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
};
if replicate_delete_with_outcome(delete, storage.clone()).await {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
} else if entry.force_delete_id.is_some() {
ReplicationQueueAdmission::Skipped
} else {
// Reconstruct a heal delete and re-queue it. We do NOT call
// get_object_info here because the delete-marker or version may
// already be absent from the local store — that is expected.
//
// The MRF entry does not persist the replication decision and the
// source object is gone, so re-derive the decision from the live
// bucket config (mirroring get_heal_replicate_object_info) and set
// it on the reconstructed delete. Without this the decision string
// is empty and the delete replicates to zero targets — a silent
// no-op that leaves replicas diverged (backlog#858 / #799 B9).
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
..Default::default()
};
let dsc = if entry.target_arns.is_empty() {
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
Ok(None) => continue,
Err(_) => {
retry_entries.push(entry.clone());
continue;
}
Ok(Some(_)) => match check_replicate_delete_strict(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned,
..Default::default()
},
None,
)
.await
{
Ok(dsc) => dsc,
Err(_) => {
retry_entries.push(entry.clone());
continue;
}
},
}
} else {
replicate_decision_for_admitted_targets(&entry.target_arns)
};
let mut rstate = oi.replication_state();
rstate.replicate_decision_str = dsc.to_string();
let delete_marker_mtime = entry
.delete_marker_mtime
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
let dv = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
delete_marker_mtime,
force_delete: entry.force_delete,
replication_state: Some(rstate),
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
};
if replicate_delete_with_outcome(dv, storage.clone()).await {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
continue;
}
};
if entry.target_arns.is_empty() {
// Legacy entries predate target admission persistence. They cannot
// be safely attributed, so retain the old live-config fallback.
queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
} else {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, entry.op.replication_type());
roi.retry_count = entry.retry_count.max(0) as u32;
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
MrfOpKind::Metadata => {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF metadata recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
continue;
}
};
if entry.target_arns.is_empty() {
queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
} else {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, ReplicationType::Metadata);
roi.retry_count = entry.retry_count.max(0) as u32;
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
let Some(admission) = replay_mrf_entry(entry, &storage, &mut retry_entries).await else {
continue;
};
if admission == ReplicationQueueAdmission::Missed {
@@ -1484,29 +1608,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
}
}
let retained = match acknowledge_mrf_recovery(storage.clone(), &recovery_guard, &entries, &retry_entries).await {
Ok(retained) => retained,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
);
match read_mrf_entries(storage.clone()).await {
Ok(current) => current,
Err(read_error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %read_error,
"Failed to refresh the MRF backlog after acknowledgement failure"
);
entries.clone()
}
}
}
};
let retained = resolve_retained_mrf_entries(&storage, &recovery_guard, &entries, &retry_entries).await;
let retained_count = retained.len();
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
@@ -0,0 +1,150 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Proxy-target selection for reads of objects not yet replicated locally
//! (MinIO `getProxyTargets`, bucket-replication.go).
//!
//! During the active-active replication lag window a GET/HEAD/Tagging request
//! for an object the local site does not have yet may be served by proxying to
//! a replication target. This module only *selects* the candidate targets; the
//! request-path callers perform the remote calls and response translation.
use std::sync::Arc;
use tracing::debug;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
use super::replication_object_config::get_replication_config;
use super::replication_storage_boundary::ObjectOptions;
use super::replication_target_boundary::{ReplicationTargetStore, TargetClient};
/// Returns the replication-target clients eligible to serve a proxied read of
/// `bucket/object`, in rule order. Mirrors MinIO's `getProxyTargets`:
///
/// - the `source-proxy-request` header family was present at all
/// (`opts.proxy_request` / `opts.proxy_header_set`, MinIO `ProxyRequest` /
/// `ProxyHeaderSet`) -> empty. "true" is the anti-loop marker of an
/// already-proxied client read; "false" is what a peer's replication
/// worker sends on convergence HEADs so the receiver answers locally —
/// proxying that miss back would echo the source object and fake
/// convergence, permanently skipping replication;
/// - the bucket's versioning is suspended for the object -> empty;
/// - no replication configuration / no matching rule -> empty;
/// - otherwise every distinct target ARN whose rules match the object,
/// resolved through the bucket target system, skipping targets that opted
/// out of proxying (`disable_proxy`).
pub async fn get_proxy_targets(bucket: &str, object: &str, opts: &ObjectOptions) -> Vec<Arc<TargetClient>> {
if opts.proxy_request || opts.proxy_header_set {
return Vec::new();
}
if opts.version_suspended {
return Vec::new();
}
let cfg = match get_replication_config(bucket).await {
Ok(Some(cfg)) => cfg,
Ok(None) => return Vec::new(),
Err(err) => {
debug!(bucket, object, error = %err, "read proxy: failed to load replication config; not proxying");
return Vec::new();
}
};
let arns = cfg.filter_target_arns(&ObjectOpts {
name: object.to_string(),
..Default::default()
});
let mut targets = Vec::with_capacity(arns.len());
for arn in arns {
let Some(client) = ReplicationTargetStore::remote_target_client(bucket, &arn).await else {
debug!(bucket, object, arn, "read proxy: no client for replication target ARN");
continue;
};
if client.disable_proxy {
continue;
}
targets.push(client);
}
targets
}
#[cfg(test)]
mod tests {
use super::*;
fn opts() -> ObjectOptions {
ObjectOptions::default()
}
/// Anti-loop: a request that was already proxied by a peer must never be
/// proxied onward, regardless of replication configuration.
#[tokio::test]
async fn proxy_request_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
proxy_request: true,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// MinIO `ProxyHeaderSet` parity: the header family being present at all
/// disables proxying, even with the value "false" — that is what a
/// peer's replication worker sends on convergence HEADs.
#[tokio::test]
async fn proxy_header_set_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
proxy_header_set: true,
proxy_request: false,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// Suspended versioning disables proxying (MinIO parity): the local null
/// version is authoritative and a remote read could resurrect data.
#[tokio::test]
async fn version_suspended_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
version_suspended: true,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// A bucket without replication configuration has nothing to proxy to.
/// (No metadata system is running in unit tests, so the config lookup
/// resolves to "no configuration" — the same empty-result contract.)
#[tokio::test]
async fn missing_replication_config_yields_no_targets() {
let targets = get_proxy_targets("bucket-without-replication", "object", &opts()).await;
assert!(targets.is_empty());
}
}
@@ -15,10 +15,15 @@
use super::replication_error_boundary::{Error, Result};
use super::replication_filemeta_boundary::MrfReplicateEntry;
/// Kept test-only: the runtime consumer was the worker HEAD's fake proxy
/// counting (removed in backlog#1675 P1-5); the resyncer tests still pin the
/// classifier's semantics for the real client read-proxy failure accounting.
#[cfg(test)]
pub(crate) use rustfs_replication::should_count_head_proxy_failure;
pub use rustfs_replication::{BucketReplicationResyncStatus, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus};
pub(crate) use rustfs_replication::{
is_version_id_mismatch, resync_state_accepts_update, sanitize_resync_error_detail, should_auto_resume_resync,
should_count_head_proxy_failure,
is_version_id_mismatch, resync_state_accepts_update, resync_status_duration, sanitize_resync_error_detail,
should_auto_resume_resync,
};
#[allow(
File diff suppressed because it is too large Load Diff
@@ -1161,6 +1161,31 @@ mod tests {
assert!(all.contains_key("proxy-only-bucket"));
}
/// Pins the read-proxy metric contract (backlog#1675 P1-5): the API
/// strings the GET/HEAD/Tagging proxy paths record map onto the
/// get/head/tagging totals, and only unexpected failures raise the
/// failed counters.
#[tokio::test]
async fn test_proxy_stats_map_read_proxy_apis_to_totals() {
let stats = ReplicationStats::new();
stats.inc_proxy("proxy-bucket", "GetObject", false).await;
stats.inc_proxy("proxy-bucket", "GetObject", true).await;
stats.inc_proxy("proxy-bucket", "HeadObject", false).await;
stats.inc_proxy("proxy-bucket", "GetObjectTagging", false).await;
stats.inc_proxy("proxy-bucket", "PutObjectTagging", false).await;
stats.inc_proxy("proxy-bucket", "DeleteObjectTagging", true).await;
let metric = stats.get_proxy_stats("proxy-bucket").await;
assert_eq!(metric.get_total, 2);
assert_eq!(metric.get_failed, 1);
assert_eq!(metric.head_total, 1);
assert_eq!(metric.head_failed, 0);
assert_eq!(metric.get_tag_total, 1);
assert_eq!(metric.put_tag_total, 1);
assert_eq!(metric.delete_tag_total, 1);
assert_eq!(metric.delete_tag_failed, 1);
}
#[tokio::test]
async fn test_calculate_bucket_replication_stats_merges_resync_metrics() {
let stats = ReplicationStats::new();
@@ -36,11 +36,15 @@ use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
pub(crate) use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient, resolve_read_api_version_id,
};
#[cfg(test)]
pub(crate) use crate::bucket::target::BucketTarget;
pub(crate) use crate::bucket::target::BucketTargets;
pub use rustfs_replication::SsecPassthroughCapability;
pub(crate) use rustfs_replication::{
SsecPassthroughGate, is_replication_target_offline_error, ssec_passthrough_gate, version_identity_drifted,
};
use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Error, Result};
@@ -65,6 +69,8 @@ static STANDARD_HEADERS: &[&str] = &[
];
const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
pub(crate) const ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED: &str = "replication target does not support SSE-C passthrough: the replica would lose its decryption material \
(run ?replication-check to re-probe)";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ReplicationSourceEncryption {
@@ -146,6 +152,13 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
rustfs_replication::is_ssec_encrypted(user_defined)
}
/// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
/// judgment owned by `rustfs-replication`: extract the echoed
/// customer-algorithm header and let the crate-owned policy decide.
pub(crate) fn ssec_passthrough_evidence_present(head: &HeadObjectOutput) -> bool {
rustfs_replication::ssec_passthrough_evidence_present(head.sse_customer_algorithm.as_deref())
}
pub(crate) struct ReplicationTargetStore;
impl ReplicationTargetStore {
@@ -165,6 +178,17 @@ impl ReplicationTargetStore {
BucketTargetSys::get().mark_target_offline(target_client).await
}
/// Returns the cached verdict and whether it has outlived its TTL.
pub(crate) async fn ssec_passthrough_capability(arn: &str) -> (SsecPassthroughCapability, bool) {
BucketTargetSys::get().ssec_passthrough_capability(arn).await
}
pub(crate) async fn record_ssec_passthrough_capability(arn: &str, capability: SsecPassthroughCapability) {
BucketTargetSys::get()
.record_ssec_passthrough_capability(arn, capability)
.await
}
#[cfg(test)]
pub(crate) async fn register_test_target(target_client: &Arc<TargetClient>) {
BucketTargetSys::get().arn_remotes_map.write().await.insert(
@@ -898,6 +922,27 @@ mod tests {
}
}
/// Pins the HeadObjectOutput field extraction feeding the crate-owned
/// evidence judgment (the gate/evidence policy matrix itself is pinned in
/// `rustfs-replication`'s object tests).
#[test]
fn ssec_passthrough_evidence_requires_customer_algorithm_echo() {
let with_evidence = HeadObjectOutput::builder().sse_customer_algorithm("AES256").build();
assert!(ssec_passthrough_evidence_present(&with_evidence));
let empty_algorithm = HeadObjectOutput::builder().sse_customer_algorithm("").build();
assert!(
!ssec_passthrough_evidence_present(&empty_algorithm),
"an empty echo is not evidence of preserved SSE-C material"
);
let without_evidence = HeadObjectOutput::builder().e_tag("\"abc\"").content_length(8).build();
assert!(
!ssec_passthrough_evidence_present(&without_evidence),
"a plain HEAD response must classify the target as having dropped the material"
);
}
#[test]
fn replication_put_options_adds_ssec_checksum_metadata() {
let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
@@ -233,17 +233,11 @@ pub struct NsScannerCapabilityRequest {
#[async_trait]
pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
async fn open_read_fresh(&self, request: ReadStreamRequest) -> Result<FileReader> {
self.open_read(request).await
}
/// Opens an owned-chunk stream when this transport can retain receive-buffer
/// ownership. `None` preserves the established `open_read` fallback.
async fn open_read_chunks(&self, _request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
Ok(None)
}
async fn open_read_chunks_fresh(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
self.open_read_chunks(request).await
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
async fn open_ns_scanner(&self, _request: NsScannerStreamRequest) -> Result<FileReader> {
@@ -275,15 +269,6 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
))
}
async fn open_read_fresh(&self, request: ReadStreamRequest) -> Result<FileReader> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
Ok(Box::new(
HttpReader::new_fresh_connection_with_stall_timeout(url, Method::GET, headers, None, request.stall_timeout).await?,
))
}
async fn open_read_chunks(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
@@ -293,16 +278,6 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
)))
}
async fn open_read_chunks_fresh(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
Ok(Some(Box::new(
HttpChunkReader::new_fresh_connection_with_stall_timeout(url, Method::GET, headers, None, request.stall_timeout)
.await?,
)))
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
let server_epoch = self.put_file_auth_capability(&request.endpoint).await?;
let nonce = server_epoch.map(|_| Uuid::new_v4());
File diff suppressed because it is too large Load Diff
+26 -104
View File
@@ -418,17 +418,6 @@ pub struct DiskHealthTracker {
pub last_capacity_free: AtomicU64,
/// Last successful capacity probe timestamp
pub last_capacity_probe_unix_secs: AtomicI64,
/// Authoritative atomically published runtime/status pair.
state_snapshot: AtomicU64,
transition_lock: std::sync::Mutex<()>,
}
fn pack_health_state(runtime_state: RuntimeDriveHealthState, status: u32) -> u64 {
(u64::from(runtime_state as u32) << 32) | u64::from(status)
}
fn unpack_health_state(snapshot: u64) -> (RuntimeDriveHealthState, u32) {
(RuntimeDriveHealthState::from_u32((snapshot >> 32) as u32), snapshot as u32)
}
#[derive(Debug)]
@@ -750,8 +739,6 @@ impl DiskHealthTracker {
last_capacity_used: AtomicU64::new(0),
last_capacity_free: AtomicU64::new(0),
last_capacity_probe_unix_secs: AtomicI64::new(0),
state_snapshot: AtomicU64::new(pack_health_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK)),
transition_lock: std::sync::Mutex::new(()),
}
}
@@ -788,52 +775,39 @@ impl DiskHealthTracker {
/// Check if disk is faulty
pub fn is_faulty(&self) -> bool {
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1 == DISK_HEALTH_FAULTY
}
fn publish_state(&self, runtime_state: RuntimeDriveHealthState, status: u32) {
self.state_snapshot
.store(pack_health_state(runtime_state, status), Ordering::Release);
self.runtime_state.store(runtime_state as u32, Ordering::Release);
self.status.store(status, Ordering::Release);
self.status.load(Ordering::Acquire) == DISK_HEALTH_FAULTY
}
/// Set disk as faulty
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn set_faulty(&self) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
}
/// Set disk as OK
pub fn set_ok(&self) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
}
#[cfg(test)]
pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let status = if state == RuntimeDriveHealthState::Offline {
DISK_HEALTH_FAULTY
} else {
DISK_HEALTH_OK
};
self.publish_state(state, status);
self.runtime_state.store(state as u32, Ordering::Release);
match state {
RuntimeDriveHealthState::Offline => self.set_faulty(),
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect | RuntimeDriveHealthState::Returning => {
self.set_ok();
}
}
}
pub fn swap_ok_to_faulty(&self) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let (_, status) = unpack_health_state(self.state_snapshot.load(Ordering::Acquire));
if status != DISK_HEALTH_OK {
return false;
}
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
true
self.status
.compare_exchange(DISK_HEALTH_OK, DISK_HEALTH_FAULTY, Ordering::AcqRel, Ordering::Relaxed)
.is_ok()
}
pub fn runtime_state(&self) -> RuntimeDriveHealthState {
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).0
RuntimeDriveHealthState::from_u32(self.runtime_state.load(Ordering::Acquire))
}
pub fn offline_duration(&self) -> Option<Duration> {
@@ -849,7 +823,6 @@ impl DiskHealthTracker {
}
pub fn mark_failure(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let now = current_unix_secs();
let next = match current {
@@ -878,19 +851,24 @@ impl DiskHealthTracker {
};
let became_offline = next == RuntimeDriveHealthState::Offline && current != RuntimeDriveHealthState::Offline;
if next == RuntimeDriveHealthState::Offline {
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
} else {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
}
self.transition_state(endpoint, current, next, reason);
became_offline
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn mark_offline(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
if current == RuntimeDriveHealthState::Offline {
return false;
}
self.consecutive_successes.store(0, Ordering::Release);
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
self.transition_state(endpoint, current, RuntimeDriveHealthState::Offline, reason);
true
}
@@ -904,10 +882,11 @@ impl DiskHealthTracker {
}
fn reset_for_store_init_retry_at(&self, endpoint: &Endpoint, now: Duration) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let now_nanos = unix_nanos(now);
let now_secs = unix_secs_i64(now);
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.runtime_state
.store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
self.offline_since_unix_secs.store(0, Ordering::Release);
@@ -919,7 +898,6 @@ impl DiskHealthTracker {
}
pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let next = match current {
RuntimeDriveHealthState::Online => RuntimeDriveHealthState::Online,
@@ -940,6 +918,7 @@ impl DiskHealthTracker {
let became_online = next == RuntimeDriveHealthState::Online;
if became_online {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
}
@@ -969,13 +948,7 @@ impl DiskHealthTracker {
return;
}
let current_status = unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1;
let status = match next {
RuntimeDriveHealthState::Offline => DISK_HEALTH_FAULTY,
RuntimeDriveHealthState::Returning => current_status,
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect => DISK_HEALTH_OK,
};
self.publish_state(next, status);
self.runtime_state.store(next as u32, Ordering::Release);
self.last_transition_unix_secs
.store(current_unix_secs() as i64, Ordering::Release);
@@ -1244,7 +1217,7 @@ impl LocalDiskWrapper {
return;
}
if health.is_faulty() {
if health.status.load(Ordering::Relaxed) != DISK_HEALTH_OK {
continue;
}
@@ -2936,57 +2909,6 @@ mod tests {
});
}
#[test]
#[serial_test::serial]
fn concurrent_failure_and_recovery_publish_one_health_snapshot() {
temp_env::with_var(rustfs_config::ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD, Some("2"), || {
let endpoint = Endpoint::try_from("/tmp/concurrent-health-snapshot").expect("endpoint should parse");
let health = Arc::new(DiskHealthTracker::new());
let transition_guard = health
.transition_lock
.lock()
.expect("health transition lock should not be poisoned");
let start = Arc::new(std::sync::Barrier::new(3));
let (completed_tx, completed_rx) = std::sync::mpsc::channel();
let workers = (0..2)
.map(|_| {
let health = Arc::clone(&health);
let endpoint = endpoint.clone();
let start = Arc::clone(&start);
let completed_tx = completed_tx.clone();
std::thread::spawn(move || {
start.wait();
health.mark_failure(&endpoint, "concurrent_test");
completed_tx.send(()).expect("completion receiver should remain available");
})
})
.collect::<Vec<_>>();
start.wait();
assert!(
matches!(
completed_rx.recv_timeout(Duration::from_millis(250)),
Err(std::sync::mpsc::RecvTimeoutError::Timeout)
),
"concurrent transitions must wait for the serialization lock"
);
drop(transition_guard);
completed_rx
.recv_timeout(Duration::from_secs(1))
.expect("first failure transition should complete after lock release");
completed_rx
.recv_timeout(Duration::from_secs(1))
.expect("second failure transition should complete after lock release");
for worker in workers {
worker.join().expect("health transition worker should not panic");
}
assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Offline);
assert!(health.is_faulty());
assert_eq!(health.consecutive_failures.load(Ordering::Acquire), 2);
});
}
#[test]
fn operation_success_recovers_suspect_drive_without_faulting() {
let endpoint = Endpoint::try_from("/tmp/runtime-state-suspect-success").expect("endpoint should parse");
+14
View File
@@ -277,6 +277,20 @@ pub struct ObjectOptions {
/// fence avoids recursively acquiring the read lock behind a queued writer.
pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
pub replication_request: bool,
/// True when the inbound request carried the
/// `{x-rustfs-,x-minio-}source-proxy-request` header family with the
/// value "true": the request was already proxied by a replication peer,
/// so this server must not proxy a local miss onward (anti-loop,
/// MinIO-compatible). The header only disables proxying — it grants no
/// capability — so no authorization gate is required to honor it.
pub proxy_request: bool,
/// True when the `source-proxy-request` header family was present at
/// all, regardless of value (MinIO's `ProxyHeaderSet`). A replication
/// peer sends `source-proxy-request: false` on its worker convergence
/// HEADs precisely so the receiver answers locally instead of proxying
/// back — otherwise a proxied 404->200 echo makes the worker believe the
/// object already converged and it never replicates it.
pub proxy_header_set: bool,
/// Source-cluster LWW timestamps carried by an authorized replication
/// request; None when the source never modified the category. Only the
/// replication-authorized options builders may set these.
+3
View File
@@ -42,6 +42,9 @@ const FILEINFO_PART_BITMAP_WORD_BITS: usize = std::mem::size_of::<u64>() * 8;
const FILEINFO_PART_BITMAP_WORDS: usize = MAX_FILEINFO_PARTS.div_ceil(FILEINFO_PART_BITMAP_WORD_BITS);
// Additional constants from Go version
// Intentionally duplicated (S3 wire literal): rustfs-replication and
// rustfs-object-data-cache carry their own independent "null" constants so
// they stay free of a rustfs-filemeta dependency. Keep all three in sync.
pub const NULL_VERSION_ID: &str = "null";
// pub const RUSTFS_ERASURE_UPGRADED: &str = "x-rustfs-internal-erasure-upgraded";
+16 -97
View File
@@ -66,37 +66,21 @@ struct HealTaskStatusPayload<'a> {
summary: &'a str,
items: &'a [HealResultItem],
truncated: bool,
/// Cursor for incremental consumption (HS-06): sequence of the next item
/// to be produced. Absent on responses without sequencing (0).
#[serde(skip_serializing_if = "u64_is_zero")]
next_seq: u64,
/// Oldest sequence still retained; with `truncated`, tells a lagging
/// client where to restart its cursor.
#[serde(skip_serializing_if = "u64_is_zero")]
min_seq: u64,
#[serde(skip_serializing_if = "Option::is_none")]
progress: Option<&'a HealProgress>,
}
fn u64_is_zero(value: &u64) -> bool {
*value == 0
}
fn encode_heal_task_status_payload(
summary: &str,
mut items: Vec<HealResultItem>,
progress: Option<&HealProgress>,
mut truncated: bool,
next_seq: u64,
min_seq: u64,
) -> Result<(Vec<u8>, bool)> {
loop {
let data = serde_json::to_vec(&HealTaskStatusPayload {
summary,
items: &items,
truncated,
next_seq,
min_seq,
progress,
})
.map_err(|e| Error::Serialization(format!("failed to serialize heal task status: {e}")))?;
@@ -125,10 +109,8 @@ fn encode_heal_status_response(
progress: Option<&HealProgress>,
detail: Option<String>,
truncated: bool,
next_seq: u64,
min_seq: u64,
) -> Result<(Vec<u8>, Option<String>)> {
let (data, truncated) = encode_heal_task_status_payload(summary, items, progress, truncated, next_seq, min_seq)?;
let (data, truncated) = encode_heal_task_status_payload(summary, items, progress, truncated)?;
Ok((data, heal_status_detail(detail, truncated)))
}
@@ -156,19 +138,8 @@ impl HealChannelProcessor {
/// Execute a token query directly against the manager.
pub async fn execute_query_request(&self, heal_path: String, client_token: String) -> Result<HealChannelResponse> {
self.execute_query_request_since(heal_path, client_token, None).await
}
/// Incremental variant of [`Self::execute_query_request`] (HS-06).
pub async fn execute_query_request_since(
&self,
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<HealChannelResponse> {
let (response_tx, response_rx) = oneshot::channel();
self.process_query_request(heal_path, client_token, since_seq, response_tx)
.await?;
self.process_query_request(heal_path, client_token, response_tx).await?;
response_rx
.await
.map_err(|err| Error::other(format!("heal query channel closed: {err}")))?
@@ -291,12 +262,8 @@ impl HealChannelProcessor {
HealChannelCommand::Query {
heal_path,
client_token,
since_seq,
response_tx,
} => {
self.process_query_request(heal_path, client_token, since_seq, response_tx)
.await
}
} => self.process_query_request(heal_path, client_token, response_tx).await,
HealChannelCommand::Cancel {
heal_path,
client_token,
@@ -417,7 +384,6 @@ impl HealChannelProcessor {
&self,
heal_path: String,
client_token: String,
since_seq: Option<u64>,
response_tx: oneshot::Sender<std::result::Result<HealChannelResponse, String>>,
) -> Result<()> {
debug!(
@@ -432,118 +398,72 @@ impl HealChannelProcessor {
);
let report = if heal_path.trim_matches('/').is_empty() {
self.heal_manager.get_task_report_since(&client_token, since_seq).await
self.heal_manager.get_task_report(&client_token).await
} else {
self.heal_manager
.get_task_report_for_path_since(&heal_path, &client_token, since_seq)
.await
self.heal_manager.get_task_report_for_path(&heal_path, &client_token).await
};
let (summary, detail, items, truncated, progress, next_seq, min_seq) = match report {
let (summary, detail, items, truncated, progress) = match report {
Ok(HealTaskReport {
status: HealTaskStatus::Pending | HealTaskStatus::Running,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"running".to_string(),
None,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
}) => ("running".to_string(), None, result_items, result_items_truncated, progress),
Ok(HealTaskReport {
status: HealTaskStatus::Retrying { error, retry_attempt },
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"running".to_string(),
Some(format!("heal task retrying after recoverable failure, attempt {retry_attempt}: {error}")),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Completed,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"finished".to_string(),
None,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
}) => ("finished".to_string(), None, result_items, result_items_truncated, progress),
Ok(HealTaskReport {
status: HealTaskStatus::Cancelled,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some("heal task cancelled".to_string()),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Timeout,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some("heal task timed out".to_string()),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Failed { error },
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some(error),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
}) => ("stopped".to_string(), Some(error), result_items, result_items_truncated, progress),
Err(crate::Error::TaskNotFound { .. }) => (
"notFound".to_string(),
Some("heal task not found or expired".to_string()),
Vec::new(),
false,
None,
0,
0,
),
Err(crate::Error::InvalidClientToken) => {
let response = HealChannelResponse {
@@ -570,8 +490,7 @@ impl HealChannelProcessor {
}
};
let (data, detail) =
encode_heal_status_response(&summary, items, progress.as_ref(), detail, truncated, next_seq, min_seq)?;
let (data, detail) = encode_heal_status_response(&summary, items, progress.as_ref(), detail, truncated)?;
let response = HealChannelResponse {
request_id: client_token,
@@ -886,7 +805,7 @@ mod tests {
..Default::default()
}];
let (data, detail) = encode_heal_status_response("running", items, None, None, false, 0, 0).unwrap();
let (data, detail) = encode_heal_status_response("running", items, None, None, false).unwrap();
assert!(data.len() <= MAX_HEAL_STATUS_PAYLOAD_SIZE);
let payload: serde_json::Value = serde_json::from_slice(&data).unwrap();
@@ -1656,7 +1575,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), "completed-token".to_string(), None, tx)
.process_query_request("bucket".to_string(), "completed-token".to_string(), tx)
.await
.expect("query should process");
@@ -1691,7 +1610,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), task_id.clone(), None, tx)
.process_query_request("bucket".to_string(), task_id.clone(), tx)
.await
.expect("query should process");
@@ -1724,7 +1643,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), "wrong-token".to_string(), None, tx)
.process_query_request("bucket".to_string(), "wrong-token".to_string(), tx)
.await
.expect("query should process");
@@ -1749,7 +1668,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request(String::new(), "wrong-token".to_string(), None, tx)
.process_query_request(String::new(), "wrong-token".to_string(), tx)
.await
.expect("query should process");
@@ -1786,7 +1705,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request(String::new(), task_id.clone(), None, tx)
.process_query_request(String::new(), task_id.clone(), tx)
.await
.expect("query should process");
+62 -435
View File
@@ -220,11 +220,6 @@ struct CompletedHealStatus {
result_items: Vec<HealResultItem>,
result_items_truncated: bool,
completed_at: SystemTime,
/// Sequence-stamped retained window, archived with the completion so
/// incremental consumers keep their cursor across the transition (HS-06).
seqed_items: Vec<(u64, HealResultItem)>,
next_seq: u64,
min_seq: u64,
}
#[derive(Debug, Clone)]
@@ -245,65 +240,6 @@ pub struct HealTaskReport {
pub result_items: Vec<HealResultItem>,
pub result_items_truncated: bool,
pub progress: Option<HealProgress>,
/// Cursor for incremental consumption: sequence number of the next item
/// to be produced. `0` on reports from sources without sequencing.
pub next_seq: u64,
/// Oldest sequence still retained (`0` together with `next_seq` when
/// sequencing is unavailable).
pub min_seq: u64,
}
/// Report from a live task, honoring the client's incremental cursor.
async fn active_task_report(task: &HealTask, since: Option<u64>) -> HealTaskReport {
let window = task.get_result_items_since(since).await;
HealTaskReport {
status: task.get_status().await,
result_items: window.items,
// The legacy flag stays set once anything was evicted; a lagging
// incremental cursor additionally marks this response truncated so
// the client knows to restart from `min_seq`.
result_items_truncated: task.result_items_truncated() || window.lagged,
progress: Some(task.get_progress().await),
next_seq: window.next_seq,
min_seq: window.min_seq,
}
}
fn empty_task_report(status: HealTaskStatus) -> HealTaskReport {
HealTaskReport {
status,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
next_seq: 0,
min_seq: 0,
}
}
fn completed_task_report(completed: &CompletedHealStatus, since: Option<u64>) -> HealTaskReport {
let mut lagged = false;
let result_items = match since {
None => completed.result_items.clone(),
Some(cursor) => {
if cursor + 1 < completed.min_seq {
lagged = true;
}
completed
.seqed_items
.iter()
.filter(|(seq, _)| *seq > cursor)
.map(|(_, item)| item.clone())
.collect()
}
};
HealTaskReport {
status: completed.status.clone(),
result_items,
result_items_truncated: completed.result_items_truncated || lagged,
progress: None,
next_seq: completed.next_seq,
min_seq: completed.min_seq,
}
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
@@ -595,11 +531,6 @@ impl PriorityHealQueue {
self.dedup_keys.contains_key(&key)
}
/// Iterate queued requests (used by the admin overlap check).
fn requests(&self) -> impl Iterator<Item = &HealRequest> {
self.heap.iter().map(|item| &item.request)
}
fn contains_request_id(&self, request_id: &str) -> bool {
self.heap.iter().any(|item| item.request.id == request_id)
}
@@ -758,80 +689,6 @@ fn recoverable_heal_retry_delay(retry_attempt: u32) -> Duration {
}
/// Heal config
/// HS-06 admin overlap policy.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum HealOverlapPolicy {
/// Default: overlapping admin starts merge into the existing task
/// (today's dedup semantics).
#[default]
Merge,
/// Return a typed already-running / overlapping-paths rejection like
/// madmin's ErrHealAlreadyRunning / ErrHealOverlappingPaths.
MinioError,
}
/// Path view of a heal type for overlap comparison: a bucket plus a
/// prefix/object path inside it (`None` bucket = cluster-wide, overlaps
/// everything).
fn heal_type_path_view(heal_type: &HealType) -> (Option<&str>, &str) {
match heal_type {
HealType::Cluster => (None, ""),
HealType::Bucket { bucket } => (Some(bucket), ""),
HealType::Prefix { bucket, prefix } => (Some(bucket), prefix),
HealType::Object { bucket, object, .. }
| HealType::Metadata { bucket, object }
| HealType::ECDecode { bucket, object, .. } => (Some(bucket), object),
// MRF/MetaPath heal keys on a meta path; treat the whole set of
// buckets as one namespace so it only overlaps itself exactly.
HealType::MRF { meta_path } => (Some("\u{0}mrf"), meta_path),
// Erasure-set heal: the set id is the overlap dimension.
HealType::ErasureSet { set_disk_id, .. } => (Some("\u{0}set"), set_disk_id),
}
}
/// How two heal paths relate for the admin overlap check (HS-06).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum OverlapVerdict {
/// Distinct targets: no conflict.
Disjoint,
/// Same target: an identical heal is already in flight.
SameTarget,
/// One target contains the other.
Overlapping,
}
fn prefix_paths_overlap(a: &str, b: &str) -> OverlapVerdict {
if a == b {
return OverlapVerdict::SameTarget;
}
if a.is_empty() || b.is_empty() || a.starts_with(b) || b.starts_with(a) {
return OverlapVerdict::Overlapping;
}
OverlapVerdict::Disjoint
}
fn heal_types_overlap(left: &HealType, right: &HealType) -> OverlapVerdict {
let (left_bucket, left_path) = heal_type_path_view(left);
let (right_bucket, right_path) = heal_type_path_view(right);
match (left_bucket, right_bucket) {
// Cluster-wide overlaps everything (but an exact cluster match is
// SameTarget).
(None, _) | (_, None) => {
if matches!(left, HealType::Cluster) && matches!(right, HealType::Cluster) {
OverlapVerdict::SameTarget
} else {
OverlapVerdict::Overlapping
}
}
(Some(lb), Some(rb)) => {
if lb != rb {
return OverlapVerdict::Disjoint;
}
prefix_paths_overlap(left_path, right_path)
}
}
}
#[derive(Debug, Clone)]
pub struct HealConfig {
/// Whether to enable auto heal
@@ -852,9 +709,6 @@ pub struct HealConfig {
pub low_priority_drop_when_full: bool,
/// Whether notify-driven scheduler wakeups are enabled.
pub event_driven_scheduler_enable: bool,
/// How admin heal starts behave on path overlap (HS-06): merge into the
/// existing task (default) or return a typed already-running rejection.
pub overlap_policy: HealOverlapPolicy,
/// Whether per-set bulkhead scheduling is enabled.
pub set_bulkhead_enable: bool,
/// Whether erasure-set page parallelism is enabled.
@@ -903,14 +757,6 @@ impl Default for HealConfig {
rustfs_config::ENV_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE,
rustfs_config::DEFAULT_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE,
);
let overlap_policy =
match rustfs_utils::get_env_str(rustfs_config::ENV_HEAL_OVERLAP_POLICY, rustfs_config::DEFAULT_HEAL_OVERLAP_POLICY)
.to_lowercase()
.as_str()
{
"minio_error" => HealOverlapPolicy::MinioError,
_ => HealOverlapPolicy::Merge,
};
let set_bulkhead_enable = rustfs_utils::get_env_bool(
rustfs_config::ENV_HEAL_SET_BULKHEAD_ENABLE,
rustfs_config::DEFAULT_HEAL_SET_BULKHEAD_ENABLE,
@@ -947,7 +793,6 @@ impl Default for HealConfig {
low_priority_merge_enable,
low_priority_drop_when_full,
event_driven_scheduler_enable,
overlap_policy,
set_bulkhead_enable,
page_parallel_enable,
mainline_throttle_enable,
@@ -1914,50 +1759,6 @@ impl HealManager {
request: HealRequest,
preserve_alias: bool,
) -> Result<HealAdmissionReceipt> {
// HS-06 forceStart semantics (admin only): MinIO stops the old task
// first and then starts the new one. Cancel any active admin task
// overlapping this request's path before entering admission, so the
// fresh task is never merged into the one being replaced.
if request.source == HealRequestSource::Admin && request.force_start {
let overlapping: Vec<String> = {
let active_heals = self.active_heals.lock().await;
active_heals
.iter()
.filter(|(task_id, task)| {
task.source == HealRequestSource::Admin
&& heal_types_overlap(&request.heal_type, &task.heal_type) != OverlapVerdict::Disjoint
&& *task_id != &request.id
})
.map(|(task_id, _)| task_id.clone())
.collect()
};
for task_id in overlapping {
match self.cancel_task(&task_id).await {
Ok(_) => info!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
cancelled_task_id = %task_id,
result = "force_start_cancelled_overlap",
"Admin forceStart cancelled an overlapping heal task"
),
Err(err) => warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
cancelled_task_id = %task_id,
error = %err,
result = "force_start_cancel_failed",
"Admin forceStart failed to cancel an overlapping heal task"
),
}
}
}
let config = self.config.read().await;
let dedup_key = PriorityHealQueue::make_dedup_key(&request);
@@ -1980,15 +1781,7 @@ impl HealManager {
.or_else(|| retrying_heal_for_dedup_key(&retrying_heals, &dedup_key).map(|(task_id, _)| (task_id, "retrying")))
});
if let Some((merged_task_id, duplicate_state)) = duplicate.flatten() {
// HS-06: under the minio_error overlap policy an exact duplicate
// admin start reports the typed AlreadyRunning rejection instead
// of the silent merge (MinIO's ErrHealAlreadyRunning).
let admission =
if request.source == HealRequestSource::Admin && config.overlap_policy == HealOverlapPolicy::MinioError {
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
} else {
Self::duplicate_admission_for_request(&request, &config)
};
let admission = Self::duplicate_admission_for_request(&request, &config);
drop(retrying_heals);
drop(queue);
drop(active_heals);
@@ -2034,62 +1827,6 @@ impl HealManager {
});
}
// HS-06 typed overlap rejection (admin only, minio_error policy):
// paths containing or contained by an active/queued task reject with
// AlreadyRunning / OverlappingPaths instead of merging. Exact
// duplicates already merged above; scanner/autoheal/read-repair
// sources never take this path.
if request.source == HealRequestSource::Admin && config.overlap_policy == HealOverlapPolicy::MinioError {
let mut rejection = None;
for (task_id, task) in active_heals.iter() {
match heal_types_overlap(&request.heal_type, &task.heal_type) {
OverlapVerdict::SameTarget => {
rejection = Some((HealAdmissionDropReason::AlreadyRunning, task_id.clone()));
break;
}
OverlapVerdict::Overlapping => {
rejection = Some((HealAdmissionDropReason::OverlappingPaths, task_id.clone()));
}
OverlapVerdict::Disjoint => {}
}
}
if rejection.is_none() {
for queued in queue.requests() {
match heal_types_overlap(&request.heal_type, &queued.heal_type) {
OverlapVerdict::SameTarget => {
rejection = Some((HealAdmissionDropReason::AlreadyRunning, queued.id.clone()));
break;
}
OverlapVerdict::Overlapping => {
rejection = Some((HealAdmissionDropReason::OverlappingPaths, queued.id.clone()));
}
OverlapVerdict::Disjoint => {}
}
}
}
if let Some((reason, overlap_task_id)) = rejection {
drop(retrying_heals);
drop(queue);
drop(active_heals);
Self::record_admission_metric(request.source, HealAdmissionResult::Dropped(reason), "overlap_rejected");
warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
overlap_task_id = %overlap_task_id,
reason = reason.as_str(),
result = "overlap_rejected",
"Admin heal start rejected by overlap policy"
);
return Ok(HealAdmissionReceipt {
result: HealAdmissionResult::Dropped(reason),
task_id: overlap_task_id,
});
}
}
let mut task_id = request.id.clone();
let admission = Self::admit_request_to_queue(&mut queue, request, &config, "submit");
if admission == HealAdmissionResult::Merged
@@ -2162,25 +1899,28 @@ impl HealManager {
}
pub async fn get_task_report(&self, task_id: &str) -> Result<HealTaskReport> {
self.get_task_report_since(task_id, None).await
}
/// Incremental variant of [`Self::get_task_report`] (HS-06): `since` is
/// the client's last seen sequence number; `None` keeps the legacy
/// full-snapshot semantics.
pub async fn get_task_report_since(&self, task_id: &str, since: Option<u64>) -> Result<HealTaskReport> {
let canonical_task_id = self.canonical_task_id(task_id).await;
{
let active_heals = self.active_heals.lock().await;
if let Some(task) = active_heals.get(&canonical_task_id) {
return Ok(active_task_report(task, since).await);
return Ok(HealTaskReport {
status: task.get_status().await,
result_items: task.get_result_items().await,
result_items_truncated: task.result_items_truncated(),
progress: Some(task.get_progress().await),
});
}
}
{
let retrying_heals = self.retrying_heals.lock().await;
if let Some(retrying) = retrying_heals.get(&canonical_task_id) {
return Ok(empty_task_report(retrying.status()));
return Ok(HealTaskReport {
status: retrying.status(),
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
}
}
@@ -2190,21 +1930,36 @@ impl HealManager {
if let Some(completed) = completed_heals.get(&canonical_task_id)
&& completed_status_is_retrying(&completed.status)
{
return Ok(completed_task_report(completed, since));
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
}
}
{
let queue = self.heal_queue.lock().await;
if queue.contains_request_id(&canonical_task_id) {
return Ok(empty_task_report(HealTaskStatus::Pending));
return Ok(HealTaskReport {
status: HealTaskStatus::Pending,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
}
}
let mut completed_heals = self.completed_heals.lock().await;
prune_completed_heal_statuses(&mut completed_heals);
if let Some(completed) = completed_heals.get(&canonical_task_id) {
return Ok(completed_task_report(completed, since));
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
}
Err(Error::TaskNotFound {
@@ -2213,23 +1968,18 @@ impl HealManager {
}
pub async fn get_task_report_for_path(&self, heal_path: &str, task_id: &str) -> Result<HealTaskReport> {
self.get_task_report_for_path_since(heal_path, task_id, None).await
}
/// Incremental variant of [`Self::get_task_report_for_path`] (HS-06).
pub async fn get_task_report_for_path_since(
&self,
heal_path: &str,
task_id: &str,
since: Option<u64>,
) -> Result<HealTaskReport> {
let canonical_task_id = self.canonical_task_id(task_id).await;
{
let active_heals = self.active_heals.lock().await;
if let Some(task) = active_heals.get(&canonical_task_id)
&& heal_type_matches_path(&task.heal_type, heal_path)
{
return Ok(active_task_report(task, since).await);
return Ok(HealTaskReport {
status: task.get_status().await,
result_items: task.get_result_items().await,
result_items_truncated: task.result_items_truncated(),
progress: Some(task.get_progress().await),
});
}
}
@@ -2238,7 +1988,12 @@ impl HealManager {
if let Some(retrying) = retrying_heals.get(&canonical_task_id)
&& heal_type_matches_path(&retrying.request.heal_type, heal_path)
{
return Ok(empty_task_report(retrying.status()));
return Ok(HealTaskReport {
status: retrying.status(),
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
}
}
@@ -2249,14 +2004,24 @@ impl HealManager {
&& heal_type_matches_path(&completed.heal_type, heal_path)
&& completed_status_is_retrying(&completed.status)
{
return Ok(completed_task_report(completed, since));
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
}
}
{
let queue = self.heal_queue.lock().await;
if queue.contains_request_id_matching_path(&canonical_task_id, heal_path) {
return Ok(empty_task_report(HealTaskStatus::Pending));
return Ok(HealTaskReport {
status: HealTaskStatus::Pending,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
}
}
@@ -2266,7 +2031,12 @@ impl HealManager {
if let Some(completed) = completed_heals.get(&canonical_task_id)
&& heal_type_matches_path(&completed.heal_type, heal_path)
{
return Ok(completed_task_report(completed, since));
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
}
}
@@ -3461,16 +3231,12 @@ impl HealManager {
completed_task.get_status().await
};
let completed_progress = completed_task.get_progress().await;
let final_window = completed_task.get_result_items_since(None).await;
let completed_status_entry = CompletedHealStatus {
heal_type: completed_task.heal_type.clone(),
status: completed_status.clone(),
result_items: final_window.items.clone(),
result_items: completed_task.get_result_items().await,
result_items_truncated: completed_task.result_items_truncated(),
completed_at: SystemTime::now(),
seqed_items: completed_task.get_seqed_result_items().await,
next_seq: final_window.next_seq,
min_seq: final_window.min_seq,
};
let mut completed_heals_guard = completed_heals_clone.lock().await;
prune_completed_heal_statuses(&mut completed_heals_guard);
@@ -5242,9 +5008,6 @@ mod tests {
},
result_items: Vec::new(),
result_items_truncated: false,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
@@ -5526,136 +5289,6 @@ mod tests {
assert_eq!(snapshot.queued_by_source.internal, 0);
}
// HS-06 (backlog#1870): overlap policy + forceStart semantics.
fn manager_with_policy(policy: HealOverlapPolicy) -> HealManager {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
HealManager::new(
storage,
Some(HealConfig {
overlap_policy: policy,
..Default::default()
}),
)
}
fn admin_prefix_request(bucket: &str, prefix: &str) -> HealRequest {
let mut request = HealRequest::new(
HealType::Prefix {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
},
HealOptions::default(),
HealPriority::Normal,
);
request.source = HealRequestSource::Admin;
request
}
async fn insert_active_task(manager: &HealManager, request: HealRequest) -> String {
let task = Arc::new(HealTask::from_request(request, manager.storage.clone()));
let task_id = task.id.clone();
manager.active_heals.lock().await.insert(task_id.clone(), task);
task_id
}
#[tokio::test]
async fn overlap_policy_minio_error_rejects_same_and_containing_paths() {
let manager = manager_with_policy(HealOverlapPolicy::MinioError);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
// Same target: typed AlreadyRunning.
let same = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/"))
.await
.expect("admission must decide");
assert_eq!(
same,
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning),
"an identical target must reject with already-running"
);
// Contained path: typed OverlappingPaths.
let nested = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/app/"))
.await
.expect("admission must decide");
assert_eq!(
nested,
HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths),
"a path inside the active task's path must reject with overlapping-paths"
);
// Containing path (bucket-wide vs nested active): also overlapping.
let wide = manager
.submit_heal_request(admin_prefix_request("bucket-a", ""))
.await
.expect("admission must decide");
assert_eq!(
wide,
HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths),
"a bucket-wide start overlapping a nested active heal must reject"
);
// Disjoint bucket: unaffected.
let disjoint = manager
.submit_heal_request(admin_prefix_request("bucket-b", "logs/"))
.await
.expect("admission must decide");
assert_eq!(disjoint, HealAdmissionResult::Accepted);
}
#[tokio::test]
async fn overlap_policy_default_merge_keeps_today_semantics() {
let manager = manager_with_policy(HealOverlapPolicy::Merge);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
// Different-dedup-key overlap still merges under the default policy:
// the nested path dedups to its own key but nothing rejects it.
let nested = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/app/"))
.await
.expect("admission must decide");
assert_eq!(nested, HealAdmissionResult::Accepted, "default policy must not reject overlaps");
// Non-admin sources never get overlap rejections even under minio_error.
let manager = manager_with_policy(HealOverlapPolicy::MinioError);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
let mut scanner_request = admin_prefix_request("bucket-a", "logs/app/");
scanner_request.source = HealRequestSource::Scanner;
let admitted = manager
.submit_heal_request(scanner_request)
.await
.expect("admission must decide");
assert_eq!(admitted, HealAdmissionResult::Accepted, "scanner sources must never be overlap-rejected");
}
#[tokio::test]
async fn admin_force_start_cancels_overlapping_active_task_first() {
let manager = manager_with_policy(HealOverlapPolicy::Merge);
let old_id = insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
let mut replacement = admin_prefix_request("bucket-a", "logs/");
replacement.force_start = true;
let receipt = manager
.submit_heal_request_with_receipt(replacement)
.await
.expect("force-start submission must decide");
assert!(receipt.result.is_admitted(), "the new task must be admitted (Accepted or Merged)");
let old_task_gone = {
let active_heals = manager.active_heals.lock().await;
!active_heals.contains_key(&old_id)
};
assert!(
old_task_gone,
"the overlapping admin task must be cancelled (removed from the active table) before the new one starts"
);
assert!(
matches!(manager.get_task_status(&old_id).await, Err(Error::TaskNotFound { .. })),
"a cancelled task must no longer resolve as an active heal"
);
}
#[tokio::test]
async fn test_operations_snapshot_counts_active_by_source_and_priority() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -5958,9 +5591,6 @@ mod tests {
status: HealTaskStatus::Completed,
result_items: Vec::new(),
result_items_truncated: false,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
@@ -5995,9 +5625,6 @@ mod tests {
..Default::default()
}],
result_items_truncated: true,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
+4 -136
View File
@@ -32,7 +32,7 @@ use std::{
future::Future,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, Ordering},
atomic::{AtomicBool, Ordering},
},
time::{Duration, Instant, SystemTime},
};
@@ -351,20 +351,6 @@ impl HealRequest {
}
/// Heal task
/// Incremental view over a task's retained result items (HS-06).
///
/// `next_seq` is the cursor a client should pass on its next poll; `min_seq`
/// is the oldest sequence still retained; `lagged` means the client's cursor
/// fell behind `min_seq` and items were skipped — the client should restart
/// from `min_seq`.
#[derive(Debug, Clone)]
pub struct HealResultWindow {
pub items: Vec<HealResultItem>,
pub next_seq: u64,
pub min_seq: u64,
pub lagged: bool,
}
pub struct HealTask {
/// Task ID
pub id: String,
@@ -387,16 +373,8 @@ pub struct HealTask {
pub status: Arc<RwLock<HealTaskStatus>>,
/// Progress tracking
pub progress: Arc<RwLock<HealProgress>>,
/// Result items collected from storage heal calls, each stamped with a
/// monotonically increasing sequence number for incremental consumption
/// (the client passes the last seen seq back and receives only newer
/// items; see `get_result_items_since`).
pub result_items: Arc<RwLock<Vec<(u64, HealResultItem)>>>,
/// Next sequence number to assign; starts at 1.
next_item_seq: Arc<AtomicU64>,
/// Sequence number of the oldest item still inside the retention window;
/// equals `next_item_seq` while the window is empty.
min_available_seq: Arc<AtomicU64>,
/// Result items collected from storage heal calls.
pub result_items: Arc<RwLock<Vec<HealResultItem>>>,
result_items_truncated: Arc<AtomicBool>,
batch_failure: Arc<RwLock<Option<BatchHealFailure>>>,
batch_failure_recorded: Arc<AtomicBool>,
@@ -448,8 +426,6 @@ impl HealTask {
status: Arc::new(RwLock::new(HealTaskStatus::Pending)),
progress: Arc::new(RwLock::new(HealProgress::new())),
result_items: Arc::new(RwLock::new(Vec::new())),
next_item_seq: Arc::new(AtomicU64::new(1)),
min_available_seq: Arc::new(AtomicU64::new(1)),
result_items_truncated: Arc::new(AtomicBool::new(false)),
batch_failure: Arc::new(RwLock::new(None)),
batch_failure_recorded: Arc::new(AtomicBool::new(false)),
@@ -935,63 +911,18 @@ impl HealTask {
}
pub async fn get_result_items(&self) -> Vec<HealResultItem> {
self.result_items.read().await.iter().map(|(_, item)| item.clone()).collect()
}
/// Sequence-stamped retained window, used when archiving a completed
/// task so incremental cursors survive the transition (HS-06).
pub async fn get_seqed_result_items(&self) -> Vec<(u64, HealResultItem)> {
self.result_items.read().await.clone()
}
/// Incremental result window (HS-06): `since = None` returns the full
/// retained window (legacy snapshot semantics); `since = Some(seq)`
/// returns only items stamped with a sequence greater than `seq`.
/// `lagged` warns that the caller's cursor fell behind the window start
/// and items were skipped (the response carries `min_seq` as the catch-up
/// cursor).
pub async fn get_result_items_since(&self, since: Option<u64>) -> HealResultWindow {
let result_items = self.result_items.read().await;
let next_seq = self.next_item_seq.load(Ordering::Relaxed);
let min_seq = self.min_available_seq.load(Ordering::Relaxed);
let mut lagged = false;
let items = match since {
None => result_items.iter().map(|(_, item)| item.clone()).collect::<Vec<_>>(),
Some(cursor) => {
if cursor + 1 < min_seq {
lagged = true;
}
result_items
.iter()
.filter(|(seq, _)| *seq > cursor)
.map(|(_, item)| item.clone())
.collect::<Vec<_>>()
}
};
HealResultWindow {
items,
next_seq,
min_seq,
lagged,
}
}
pub fn result_items_truncated(&self) -> bool {
self.result_items_truncated.load(Ordering::Relaxed)
}
async fn record_result_item(&self, result: HealResultItem) {
let seq = self.next_item_seq.fetch_add(1, Ordering::Relaxed);
let mut result_items = self.result_items.write().await;
if result_items.len() < MAX_RETAINED_HEAL_RESULT_ITEMS {
result_items.push((seq, result));
result_items.push(result);
} else {
// Slide the window: the oldest item leaves and the cursor for the
// oldest still-available item moves forward with it.
result_items.remove(0);
self.min_available_seq
.store(result_items.first().map_or(seq, |(oldest, _)| *oldest), Ordering::Relaxed);
result_items.push((seq, result));
self.result_items_truncated.store(true, Ordering::Relaxed);
}
}
@@ -3949,69 +3880,6 @@ mod tests {
assert!(task.result_items_truncated());
}
// HS-06 (backlog#1870): incremental result windows.
#[tokio::test]
async fn result_items_seq_is_monotonic_and_incremental_slices_work() {
let storage = Arc::new(MockStorage::default());
let task = HealTask::from_request(HealRequest::bucket("bucket-a".to_string()), storage);
for round in 0..5u64 {
let item = HealResultItem {
object_size: round as usize,
..Default::default()
};
task.record_result_item(item).await;
}
let full = task.get_result_items_since(None).await;
assert_eq!(full.items.len(), 5, "None keeps the full-snapshot semantics");
assert_eq!(full.next_seq, 6, "next_seq is one past the last assigned");
assert_eq!(full.min_seq, 1, "nothing was evicted yet");
assert!(!full.lagged);
// Incremental: only items newer than the cursor.
let incremental = task.get_result_items_since(Some(3)).await;
assert_eq!(
incremental.items.iter().map(|item| item.object_size).collect::<Vec<_>>(),
vec![3, 4],
"only sequences greater than the cursor are returned"
);
assert_eq!(incremental.next_seq, 6);
// A cursor at the head is not lagging.
assert!(!task.get_result_items_since(Some(0)).await.lagged);
}
#[tokio::test]
async fn result_items_window_slide_moves_min_seq_and_flags_lagging_cursors() {
let storage = Arc::new(MockStorage::default());
let task = HealTask::from_request(HealRequest::bucket("bucket-a".to_string()), storage);
// Fill the window completely, then push two more items: seq 1 and 2
// are evicted by the slide.
for _ in 0..(MAX_RETAINED_HEAL_RESULT_ITEMS + 2) {
task.record_result_item(HealResultItem::default()).await;
}
let full = task.get_result_items_since(None).await;
assert_eq!(full.items.len(), MAX_RETAINED_HEAL_RESULT_ITEMS);
assert_eq!(full.min_seq, 3, "each evicted head item moved the oldest-available cursor");
assert!(task.result_items_truncated());
// A client still polling from before the eviction is lagging.
let lagging = task.get_result_items_since(Some(0)).await;
assert!(lagging.lagged, "a cursor behind min_seq must be flagged");
assert_eq!(lagging.min_seq, 3, "the response tells the client where to restart");
// A cursor inside the window is fine.
assert!(!task.get_result_items_since(Some(3)).await.lagged);
// The lagging client restarts from min_seq and gets the full window.
let catch_up = task.get_result_items_since(Some(3)).await;
assert_eq!(catch_up.items.len(), MAX_RETAINED_HEAL_RESULT_ITEMS - 1);
assert!(!catch_up.lagged);
}
#[tokio::test]
async fn test_recursive_bucket_heal_skips_object_dir_candidates() {
let storage = Arc::new(MockStorage {
-3
View File
@@ -9,9 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Removed
#### rustfs-io-core
- **Zero-consumer modules** (added in 0.0.5): `reader`, `writer`, `bufreader_optimizer`, `shared_memory`, `direct_io`, `timeout_wrapper`, `io_priority_queue`, and `scheduler` had no caller in the workspace and were removed (rustfs/backlog#1824). The scheduling algorithm and the request timeout wrapper that RustFS actually runs live in `rustfs/src/storage/`; this crate keeps the config shapes they project into. `OperationProgress` moved to the new `progress` module and is still exported as `rustfs_io_core::OperationProgress`.
#### rustfs-io-metrics
- **Unified configuration** (added in 0.0.5): the zero-consumer `IoConfig`, `CacheSettings`, `IoSchedulerSettings`, `BackpressureSettings`, `TimeoutSettings`, `DeadlockDetectionSettings` types and their `DEFAULT_*` constants were removed (rustfs/rustfs#6008); rustfs-io-core's `IoSchedulerConfig`/`BackpressureConfig` remain the canonical configuration types.
+3 -2
View File
@@ -20,8 +20,8 @@ license.workspace = true
repository.workspace = true
rust-version.workspace = true
homepage.workspace = true
description = "Shared I/O primitives for RustFS (buffer pool, storage profiling, backpressure, deadlock detection)"
keywords = ["io", "buffer", "pool", "rustfs", "backpressure"]
description = "Buffered I/O reader and writer implementations for RustFS (mmap-then-copy, aligned pread)"
keywords = ["io", "reader", "writer", "rustfs", "mmap"]
categories = ["development-tools", "filesystem"]
[lints]
@@ -38,6 +38,7 @@ hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["io-util", "fs", "sync", "rt-multi-thread"] }
memmap2 = { workspace = true }
rustfs-io-metrics = { workspace = true }
tracing = { workspace = true }
+120 -18
View File
@@ -23,20 +23,67 @@
## Overview
**rustfs-io-core** holds the shared I/O primitives for [RustFS](https://rustfs.com), a distributed object storage system. It provides:
**rustfs-io-core** is the core I/O scheduling module for [RustFS](https://rustfs.com), a distributed object storage system. It provides:
- **Buffer Pool**: Tiered `BytesPool` for buffer reuse
- **Storage Profiling**: Storage-media and access-pattern model (`io_profile`)
- **Scheduler Configuration**: The `IoSchedulerConfig` / `IoPriorityQueueConfig` shapes the storage layer projects into
- **I/O Scheduler**: Adaptive buffer size calculation and load management
- **Priority Queue**: Request priority scheduling with starvation prevention
- **Backpressure Control**: System overload protection with graceful degradation
- **Deadlock Detection**: Wait-for graph based deadlock detection algorithm
- **Lock Optimizer**: Adaptive spin lock optimization
- **Progress Tracking**: Byte progress and staleness for long-running operations
The scheduling algorithm itself lives in `rustfs/src/storage/concurrency/io_schedule.rs`; this crate carries the configuration shapes it projects into, not a second implementation.
- **Timeout Wrapper**: Dynamic timeout calculation and operation progress tracking
## Features
### I/O Scheduler
Adaptive I/O scheduling with dynamic buffer size calculation based on file size, access pattern, and system load:
```rust
use rustfs_io_core::{IoScheduler, IoSchedulerConfig, IoLoadLevel};
use rustfs_io_core::io_profile::{StorageMedia, AccessPattern};
// Create scheduler
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * 1024, // 64 KB
max_buffer_size: 1024 * 1024, // 1 MB
..Default::default()
};
let scheduler = IoScheduler::new(config);
// Calculate optimal buffer size
let buffer_size = calculate_optimal_buffer_size(
10 * 1024 * 1024, // 10 MB file
64 * 1024, // base buffer
true, // sequential access
4, // concurrent requests
StorageMedia::Ssd,
IoLoadLevel::Low,
);
```
### Priority Queue
Priority queue with starvation prevention:
```rust
use rustfs_io_core::{IoPriorityQueue, IoPriority, IoQueueStatus};
let queue = IoPriorityQueue::<()>::new(100);
// Enqueue request
let request_id = queue.enqueue(IoPriority::High, (), 1024);
// Dequeue request
if let Some((priority, data)) = queue.dequeue() {
println!("Processing priority {:?} request", priority);
}
// Check queue status
let status = queue.status();
println!("High priority waiting: {}", status.high_priority_waiting);
```
### Backpressure Control
System overload protection:
@@ -101,23 +148,71 @@ let stats = optimizer.stats();
println!("Locks acquired: {}", stats.total_acquired());
```
### Progress Tracking
### Timeout Wrapper
Byte progress and staleness for long-running operations:
Dynamic timeout calculation:
```rust
use rustfs_io_core::OperationProgress;
use rustfs_io_core::{RequestTimeoutWrapper, TimeoutConfig};
use std::time::Duration;
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
let config = TimeoutConfig {
base_timeout: Duration::from_secs(5),
timeout_per_mb: Duration::from_millis(100),
max_timeout: Duration::from_secs(300),
..Default::default()
};
let wrapper = RequestTimeoutWrapper::new(config);
progress.update(500);
assert_eq!(progress.progress_percent(), Some(50.0));
assert!(!progress.is_stale());
// Calculate operation timeout
let timeout = wrapper.calculate_timeout(10 * 1024 * 1024); // 10 MB
```
## Buffer Size Calculation
Multiple buffer size calculation functions are provided:
```rust
use rustfs_io_core::{
get_concurrency_aware_buffer_size,
get_advanced_buffer_size,
get_buffer_size_for_media,
calculate_optimal_buffer_size,
KI_B, MI_B,
};
use rustfs_io_core::io_profile::StorageMedia;
// Basic calculation
let size1 = get_concurrency_aware_buffer_size(1024 * 1024, 64 * 1024);
// Advanced calculation (considering access pattern)
let size2 = get_advanced_buffer_size(10 * 1024 * 1024, 64 * 1024, true);
// Media type optimization
let size3 = get_buffer_size_for_media(64 * 1024, StorageMedia::Ssd);
// Comprehensive calculation
let size4 = calculate_optimal_buffer_size(
100 * 1024 * 1024, // 100 MB file
64 * 1024, // base buffer
true, // sequential access
4, // concurrent requests
StorageMedia::Nvme,
IoLoadLevel::Low,
);
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `RUSTFS_MAX_CONCURRENT_READS` | Max concurrent reads | 64 |
| `RUSTFS_BASE_BUFFER_SIZE` | Base buffer size | 65536 |
| `RUSTFS_MAX_BUFFER_SIZE` | Max buffer size | 1048576 |
| `RUSTFS_IO_TIMEOUT_SECS` | I/O timeout seconds | 30 |
### Code Configuration
```rust
@@ -145,11 +240,12 @@ rustfs-io-core/
├── src/
│ ├── lib.rs # Module entry
│ ├── config.rs # Configuration types
│ ├── pool.rs # Tiered buffer pool
│ ├── scheduler.rs # I/O scheduler
│ ├── io_priority_queue.rs # Priority queue
│ ├── backpressure.rs # Backpressure control
│ ├── deadlock_detector.rs # Deadlock detection
│ ├── lock_optimizer.rs # Lock optimization
│ ├── progress.rs # Operation progress tracking
│ ├── timeout_wrapper.rs # Timeout wrapper
│ └── io_profile.rs # I/O profile
└── Cargo.toml
```
@@ -158,15 +254,21 @@ rustfs-io-core/
```bash
# Run all tests
cargo nextest run --package rustfs-io-core
cargo test --package rustfs-io-core
# Run specific tests
cargo nextest run --package rustfs-io-core -E 'test(backpressure)'
cargo test --package rustfs-io-core --lib scheduler
# Run benchmarks
cargo bench --package rustfs-io-core
```
## Documentation
- [API Documentation](https://docs.rs/rustfs-io-core)
- [I/O Scheduler Design](./docs/scheduler-design.md)
- [Backpressure Control Design](./docs/backpressure-design.md)
- [Deadlock Detection Algorithm](./docs/deadlock-detection.md)
## Related Modules
+131 -18
View File
@@ -23,20 +23,71 @@
## 📖 概述
**rustfs-io-core** 是 [RustFS](https://rustfs.com) 分布式对象存储系统的共享 I/O 基础组件。它提供了:
**rustfs-io-core** 是 [RustFS](https://rustfs.com) 分布式对象存储系统的核心 I/O 调度模块。它提供了:
- **缓冲池**:分级复用的 `BytesPool`
- **存储画像**存储介质与访问模式模型(`io_profile`
- **调度配置**:存储层投影使用的 `IoSchedulerConfig` / `IoPriorityQueueConfig`
- **I/O 调度器**:自适应缓冲区大小计算和负载管理
- **优先级队列**支持饥饿预防的请求优先级调度
- **背压控制**:系统过载保护和优雅降级
- **死锁检测**:基于等待图的死锁检测算法
- **锁优化**:自适应自旋锁优化
- **进度追踪**长耗时操作的字节进度与停滞判定
调度算法本身位于 `rustfs/src/storage/concurrency/io_schedule.rs`;本 crate 只承载它投影使用的配置形状,不是第二套实现。
- **超时包装器**动态超时计算和操作进度追踪
## ✨ 核心功能
### I/O 调度器 (IoScheduler)
自适应 I/O 调度,根据文件大小、访问模式和系统负载动态调整缓冲区大小:
```rust
use rustfs_io_core::{IoScheduler, IoSchedulerConfig, IoLoadLevel};
use rustfs_io_core::io_profile::{StorageMedia, AccessPattern};
// 创建调度器
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * 1024, // 64 KB
max_buffer_size: 1024 * 1024, // 1 MB
..Default::default()
};
let scheduler = IoScheduler::new(config);
// 计算最优缓冲区大小
let buffer_size = scheduler.calculate_buffer_size(
10 * 1024 * 1024, // 10 MB 文件
true, // 顺序访问
StorageMedia::Ssd,
IoLoadLevel::Low,
);
println!("缓冲区大小: {} bytes", buffer_size);
```
### 优先级队列 (IoPriorityQueue)
支持饥饿预防的优先级队列:
```rust
use rustfs_io_core::{IoPriorityQueue, IoPriority, IoQueueStatus};
let queue = IoPriorityQueue::<()>::new(100);
// 入队请求
let request_id = queue.enqueue(
IoPriority::High,
(), // 请求数据
1024, // 请求大小
);
// 出队请求
if let Some((priority, data)) = queue.dequeue() {
println!("处理优先级 {:?} 的请求", priority);
}
// 检查队列状态
let status = queue.status();
println!("高优先级等待: {}", status.high_priority_waiting);
println!("低优先级等待: {}", status.low_priority_waiting);
```
### 背压控制 (BackpressureMonitor)
系统过载保护:
@@ -114,23 +165,78 @@ let stats = optimizer.stats();
println!("获取锁次数: {}", stats.locks_acquired.load(std::sync::atomic::Ordering::Relaxed));
```
### 进度追踪 (OperationProgress)
### 超时包装器 (RequestTimeoutWrapper)
长耗时操作的字节进度与停滞判定
动态超时计算
```rust
use rustfs_io_core::OperationProgress;
use rustfs_io_core::{RequestTimeoutWrapper, TimeoutConfig};
use std::time::Duration;
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
let config = TimeoutConfig {
base_timeout: Duration::from_secs(5),
timeout_per_mb: Duration::from_millis(100),
max_timeout: Duration::from_secs(300),
..Default::default()
};
let wrapper = RequestTimeoutWrapper::new(config);
progress.update(500);
assert_eq!(progress.progress_percent(), Some(50.0));
assert!(!progress.is_stale());
// 计算操作超时
let timeout = wrapper.calculate_timeout(10 * 1024 * 1024); // 10 MB
println!("超时时间: {:?}", timeout);
// 执行带超时的操作
let result = wrapper.execute_with_timeout(async {
// 异步操作
Ok::<_, std::io::Error>(())
}, timeout).await;
```
## 📊 缓冲区大小计算
模块提供了多种缓冲区大小计算函数:
```rust
use rustfs_io_core::{
get_concurrency_aware_buffer_size,
get_advanced_buffer_size,
get_buffer_size_for_media,
calculate_optimal_buffer_size,
KI_B, MI_B,
};
use rustfs_io_core::io_profile::StorageMedia;
// 基础计算
let size1 = get_concurrency_aware_buffer_size(1024 * 1024, 64 * 1024);
// 高级计算(考虑访问模式)
let size2 = get_advanced_buffer_size(10 * 1024 * 1024, 64 * 1024, true);
// 媒体类型优化
let size3 = get_buffer_size_for_media(64 * 1024, StorageMedia::Ssd);
// 综合计算
let size4 = calculate_optimal_buffer_size(
100 * 1024 * 1024, // 100 MB 文件
64 * 1024, // 基础缓冲区
true, // 顺序访问
4, // 并发请求数
StorageMedia::Nvme,
IoLoadLevel::Low,
);
```
## 🔧 配置
### 环境变量
| 变量名 | 描述 | 默认值 |
|--------|------|--------|
| `RUSTFS_MAX_CONCURRENT_READS` | 最大并发读数 | 64 |
| `RUSTFS_BASE_BUFFER_SIZE` | 基础缓冲区大小 | 65536 |
| `RUSTFS_MAX_BUFFER_SIZE` | 最大缓冲区大小 | 1048576 |
| `RUSTFS_IO_TIMEOUT_SECS` | I/O 超时秒数 | 30 |
### 代码配置
```rust
@@ -158,11 +264,12 @@ rustfs-io-core/
├── src/
│ ├── lib.rs # 模块入口
│ ├── config.rs # 配置类型
│ ├── pool.rs # 分级缓冲池
│ ├── scheduler.rs # I/O 调度器
│ ├── io_priority_queue.rs # 优先级队列
│ ├── backpressure.rs # 背压控制
│ ├── deadlock_detector.rs # 死锁检测
│ ├── lock_optimizer.rs # 锁优化
│ ├── progress.rs # 操作进度追踪
│ ├── timeout_wrapper.rs # 超时包装器
│ └── io_profile.rs # I/O 配置文件
└── Cargo.toml
```
@@ -171,15 +278,21 @@ rustfs-io-core/
```bash
# 运行所有测试
cargo nextest run --package rustfs-io-core
cargo test --package rustfs-io-core
# 运行特定测试
cargo nextest run --package rustfs-io-core -E 'test(backpressure)'
cargo test --package rustfs-io-core --lib scheduler
# 运行基准测试
cargo bench --package rustfs-io-core
```
## 📚 文档
- [API 文档](https://docs.rs/rustfs-io-core)
- [I/O 调度器设计](./docs/scheduler-design.md)
- [背压控制原理](./docs/backpressure-design.md)
- [死锁检测算法](./docs/deadlock-detection.md)
## 🔗 相关模块
@@ -0,0 +1,190 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Example demonstrating I/O scheduler usage.
use rustfs_io_core::io_profile::StorageMedia;
use rustfs_io_core::{
BackpressureMonitor, BackpressureState, DeadlockDetector, IoLoadLevel, IoScheduler, IoSchedulerConfig, KI_B, LockOptimizer,
LockType, MI_B, calculate_optimal_buffer_size, get_buffer_size_for_media,
};
use std::time::Duration;
fn main() {
println!("=== rustfs-io-core Example ===\n");
// 1. I/O scheduler example
io_scheduler_example();
// 2. Buffer size calculation example
buffer_size_example();
// 3. Backpressure control example
backpressure_example();
// 4. Deadlock detection example
deadlock_detection_example();
// 5. Lock optimizer example
lock_optimizer_example();
}
fn io_scheduler_example() {
println!("--- I/O Scheduler ---");
// Create scheduler with configuration
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * KI_B,
max_buffer_size: MI_B,
..Default::default()
};
let scheduler = IoScheduler::new(config);
println!(" Max concurrent reads: {}", scheduler.config().max_concurrent_reads);
println!(" Base buffer size: {} KB", scheduler.config().base_buffer_size / KI_B);
println!(" Max buffer size: {} KB", scheduler.config().max_buffer_size / KI_B);
// Calculate buffer sizes for different scenarios
let scenarios = [
("Small file", 10 * KI_B as i64, true, StorageMedia::Ssd),
("Medium file", MI_B as i64, true, StorageMedia::Ssd),
("Large sequential", 100 * MI_B as i64, true, StorageMedia::Ssd),
("Large random", 100 * MI_B as i64, false, StorageMedia::Ssd),
("NVMe large", 100 * MI_B as i64, true, StorageMedia::Nvme),
("HDD large", 100 * MI_B as i64, true, StorageMedia::Hdd),
];
for (name, size, sequential, media) in scenarios {
let buffer = calculate_optimal_buffer_size(size, 64 * KI_B, sequential, 4, media, IoLoadLevel::Low);
println!(" {}: {} bytes ({} KB)", name, buffer, buffer / KI_B);
}
println!();
}
fn buffer_size_example() {
println!("--- Buffer Size Calculation ---");
// Comprehensive calculation
let size1 = calculate_optimal_buffer_size(10 * MI_B as i64, 64 * KI_B, true, 4, StorageMedia::Ssd, IoLoadLevel::Low);
println!(" Comprehensive (10MB, sequential, SSD): {} KB", size1 / KI_B);
// Media type optimization
let media_types = [
StorageMedia::Nvme,
StorageMedia::Ssd,
StorageMedia::Hdd,
StorageMedia::Unknown,
];
for media in media_types {
let size = get_buffer_size_for_media(64 * KI_B, media);
println!(" {} optimized: {} KB", media.as_str(), size / KI_B);
}
println!();
}
fn backpressure_example() {
println!("--- Backpressure Control ---");
let monitor = BackpressureMonitor::with_defaults();
// Check initial state
let state = monitor.state();
let state_str = match state {
BackpressureState::Normal => "Normal",
BackpressureState::Warning => "Warning",
BackpressureState::Critical => "Critical",
};
println!(" Initial state: {}", state_str);
// Check if active
let is_active = monitor.is_active();
println!(" Backpressure active: {}", is_active);
// Try to acquire permit
if monitor.try_acquire() {
println!(" Successfully acquired permit");
monitor.release();
println!(" Released permit");
}
// View statistics
println!(" Total processed: {}", monitor.total_processed());
println!(" Total rejected: {}", monitor.total_rejected());
println!();
}
fn deadlock_detection_example() {
println!("--- Deadlock Detection ---");
let detector = DeadlockDetector::with_defaults();
// Register locks
let mutex1 = detector.register_lock(LockType::Mutex);
let mutex2 = detector.register_lock(LockType::Mutex);
println!(" Registered locks: mutex1={}, mutex2={}", mutex1, mutex2);
// Simulate normal operation
detector.record_acquire(mutex1, 1); // Thread 1 acquires mutex1
detector.record_acquire(mutex2, 2); // Thread 2 acquires mutex2
println!(" Normal operation: no deadlock");
// Detect deadlock
if detector.detect_deadlock().is_none() {
println!(" Detection result: no deadlock");
}
// Simulate deadlock scenario
detector.record_wait(mutex2, 1); // Thread 1 waits for mutex2
detector.record_wait(mutex1, 2); // Thread 2 waits for mutex1
// Detect deadlock
if let Some(deadlock) = detector.detect_deadlock() {
println!(" Detection result: deadlock found {:?}", deadlock);
}
// Cleanup
detector.unregister_lock(mutex1);
detector.unregister_lock(mutex2);
println!();
}
fn lock_optimizer_example() {
println!("--- Lock Optimizer ---");
let optimizer = LockOptimizer::with_defaults();
// Simulate lock operations
for _i in 0..5 {
optimizer.on_acquire();
// Simulate work
std::thread::sleep(Duration::from_millis(10));
optimizer.on_release(Duration::from_millis(10));
}
// View statistics
let stats = optimizer.stats();
let acquired = stats.total_acquired();
let avg_hold = stats.avg_hold_time();
let contention = stats.contention_rate();
println!(" Locks acquired: {}", acquired);
println!(" Average hold time: {:?}", avg_hold);
println!(" Contention rate: {:.2}%", contention * 100.0);
println!();
}
+227
View File
@@ -0,0 +1,227 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! BufReader layer optimizer for minimizing redundant buffering layers.
//!
//! This module provides optimization for BufReader usage in data paths,
//! including layer count limiting and dynamic buffer size adjustment.
use std::sync::atomic::{AtomicU64, Ordering};
/// BufReader optimization configuration.
#[derive(Debug, Clone)]
pub struct BufReaderConfig {
/// Maximum number of nested BufReader layers (default: 2)
pub max_layers: u32,
/// Buffer size for small files (default: 8KB)
pub small_file_buffer: usize,
/// Buffer size for large files (default: 64KB)
pub large_file_buffer: usize,
/// Threshold for large file classification (default: 1MB)
pub large_file_threshold: usize,
}
impl Default for BufReaderConfig {
fn default() -> Self {
Self {
max_layers: 2,
small_file_buffer: 8 * 1024, // 8KB
large_file_buffer: 64 * 1024, // 64KB
large_file_threshold: 1024 * 1024, // 1MB
}
}
}
/// BufReader optimization statistics.
#[derive(Debug, Default)]
pub struct BufReaderStats {
/// Total number of readers created
pub total_readers: AtomicU64,
/// Number of redundant layers eliminated
pub eliminated_layers: AtomicU64,
/// Number of buffer size adjustments
pub buffer_size_adjustments: AtomicU64,
}
/// BufReader layer optimizer.
///
/// Analyzes and optimizes BufReader nesting in data paths,
/// dynamically adjusting buffer sizes based on data characteristics.
pub struct BufReaderOptimizer {
config: BufReaderConfig,
stats: BufReaderStats,
}
impl BufReaderOptimizer {
/// Create a new BufReader optimizer with the given configuration.
pub fn new(config: BufReaderConfig) -> Self {
Self {
config,
stats: BufReaderStats::default(),
}
}
/// Create a new BufReader optimizer with default configuration.
pub fn with_defaults() -> Self {
Self::new(BufReaderConfig::default())
}
/// Calculate the optimal buffer size based on data size.
///
/// Returns the appropriate buffer size based on whether the data
/// is classified as a small or large file.
pub fn optimal_buffer_size(&self, data_size: Option<usize>) -> usize {
match data_size {
Some(size) if size >= self.config.large_file_threshold => self.config.large_file_buffer,
Some(_) => self.config.small_file_buffer,
None => self.config.small_file_buffer,
}
}
/// Optimize a reader by wrapping it with an appropriately sized BufReader.
///
/// This method applies the optimal buffer size based on the expected
/// data size and tracks statistics.
pub fn optimize<R: tokio::io::AsyncRead + Unpin>(&self, reader: R, data_size: Option<usize>) -> tokio::io::BufReader<R> {
let buffer_size = self.optimal_buffer_size(data_size);
self.stats.total_readers.fetch_add(1, Ordering::Relaxed);
tokio::io::BufReader::with_capacity(buffer_size, reader)
}
/// Get the statistics for this optimizer.
pub fn stats(&self) -> &BufReaderStats {
&self.stats
}
/// Get the configuration for this optimizer.
pub fn config(&self) -> &BufReaderConfig {
&self.config
}
}
/// Marker trait for buffered sources.
///
/// Types implementing this trait are considered already buffered
/// and should not be wrapped with additional BufReader layers.
pub trait BufferedSource: tokio::io::AsyncRead {}
impl BufReaderOptimizer {
/// Check if a reader is already a buffered source.
///
/// Returns true if the reader implements `BufferedSource`,
/// indicating it should not be wrapped with BufReader.
pub fn is_buffered_source<R: BufferedSource + ?Sized>(&self, _reader: &R) -> bool {
true
}
/// Eliminate redundant BufReader layers if possible.
///
/// This method attempts to reduce the nesting depth of BufReader
/// layers to improve performance.
pub fn eliminate_redundant_layers<R: tokio::io::AsyncRead + Unpin>(&self, reader: R) -> R {
// For now, just return the reader as-is
// Future implementation could detect and unwrap nested BufReaders
self.stats.eliminated_layers.fetch_add(0, Ordering::Relaxed);
reader
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::AsyncReadExt;
#[test]
fn test_default_config() {
let config = BufReaderConfig::default();
assert_eq!(config.max_layers, 2);
assert_eq!(config.small_file_buffer, 8 * 1024);
assert_eq!(config.large_file_buffer, 64 * 1024);
assert_eq!(config.large_file_threshold, 1024 * 1024);
}
#[test]
fn test_optimal_buffer_size_small_file() {
let optimizer = BufReaderOptimizer::with_defaults();
// Small file (< 1MB)
assert_eq!(optimizer.optimal_buffer_size(Some(100)), 8 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(1024)), 8 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(512 * 1024)), 8 * 1024);
}
#[test]
fn test_optimal_buffer_size_large_file() {
let optimizer = BufReaderOptimizer::with_defaults();
// Large file (>= 1MB)
assert_eq!(optimizer.optimal_buffer_size(Some(1024 * 1024)), 64 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(10 * 1024 * 1024)), 64 * 1024);
}
#[test]
fn test_optimal_buffer_size_unknown() {
let optimizer = BufReaderOptimizer::with_defaults();
// Unknown size
assert_eq!(optimizer.optimal_buffer_size(None), 8 * 1024);
}
#[tokio::test]
async fn test_optimize_creates_bufreader() {
let optimizer = BufReaderOptimizer::with_defaults();
let data = vec![1u8, 2, 3, 4, 5];
let cursor = std::io::Cursor::new(data.clone());
let mut reader = optimizer.optimize(cursor, Some(5));
let mut buf = vec![0u8; 5];
let n = reader.read(&mut buf).await.unwrap();
assert_eq!(n, 5);
assert_eq!(buf, data);
}
#[test]
fn test_stats_tracking() {
let optimizer = BufReaderOptimizer::with_defaults();
assert_eq!(optimizer.stats().total_readers.load(Ordering::Relaxed), 0);
let cursor = std::io::Cursor::new(vec![1u8, 2, 3]);
let _reader = optimizer.optimize(cursor, Some(3));
assert_eq!(optimizer.stats().total_readers.load(Ordering::Relaxed), 1);
}
#[test]
fn test_custom_config() {
let config = BufReaderConfig {
max_layers: 3,
small_file_buffer: 4 * 1024,
large_file_buffer: 128 * 1024,
large_file_threshold: 2 * 1024 * 1024,
};
let optimizer = BufReaderOptimizer::new(config);
assert_eq!(optimizer.optimal_buffer_size(Some(1024 * 1024)), 4 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(3 * 1024 * 1024)), 128 * 1024);
}
}
+332
View File
@@ -0,0 +1,332 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Aligned pread-based file reader.
//!
//! This module provides an aligned, position-based file reader that uses
//! `pread`/`FileExt::read_at` for I/O operations. It performs reads at
//! 512-byte-aligned offsets and sizes, making it suitable as a foundation
//! for workloads where alignment matters.
//!
//! Note: This reader does **not** set the `O_DIRECT` flag and therefore does
//! not bypass the OS page cache. It is an aligned `pread`-based reader, not
//! true Direct I/O. To implement true O_DIRECT on Linux, the file must be
//! opened with `O_DIRECT` via `libc::open`.
//!
//! # Platform Support
//!
//! The `read_at` implementation is only available on Unix-like platforms.
//! On other platforms, this reader will return an error.
use std::io::{self};
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
/// Errors that can occur during aligned pread operations.
#[derive(Debug, Clone)]
pub enum AlignedPreadError {
/// Platform doesn't support `read_at`-based I/O
UnsupportedPlatform,
/// File descriptor doesn't support this reader
UnsupportedFile,
/// I/O error occurred
Io(String),
/// Invalid alignment (reads require 512-byte-aligned offset and size)
AlignmentError { offset: u64, size: usize },
}
impl std::fmt::Display for AlignedPreadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::UnsupportedPlatform => write!(f, "Aligned pread not supported on this platform"),
Self::UnsupportedFile => write!(f, "File doesn't support this reader"),
Self::Io(msg) => write!(f, "I/O error: {}", msg),
Self::AlignmentError { offset, size } => {
write!(f, "Alignment error: offset={}, size={}", offset, size)
}
}
}
}
impl std::error::Error for AlignedPreadError {}
impl From<io::Error> for AlignedPreadError {
fn from(err: io::Error) -> Self {
Self::Io(err.to_string())
}
}
/// Aligned pread-based file reader for Unix platforms.
///
/// This reader performs I/O using `pread`/`FileExt::read_at` at
/// 512-byte-aligned offsets and sizes, without modifying the file's
/// current position.
///
/// **Note:** This reader does **not** set the `O_DIRECT` flag and therefore
/// does **not** bypass the OS page cache. It is an aligned `pread`-based
/// reader. To implement true O_DIRECT, the file must be opened with
/// `O_DIRECT` via `libc::open`.
///
/// # Platform Support
///
/// Only available on Linux (uses `FileExt::read_at`). On other platforms,
/// use `BytesBufferedReader` instead.
///
/// # Alignment Requirements
///
/// Reads have strict alignment requirements:
/// - File offset must be aligned to 512 bytes
/// - Buffer size must be a multiple of 512 bytes
/// - Buffer address must be aligned (handled internally)
///
/// # Example
///
/// ```ignore
/// use rustfs_io_core::AlignedPreadReader;
///
/// // Linux only
/// #[cfg(target_os = "linux")]
/// let reader = AlignedPreadReader::new(file, offset, size)?;
/// ```
#[cfg(target_os = "linux")]
pub struct AlignedPreadReader {
/// Underlying file handle used for aligned pread I/O
file: std::fs::File,
/// Current read position
pos: u64,
/// Remaining bytes to read
remaining: usize,
/// Buffer for aligned reads
buffer: Vec<u8>,
/// Current position in the buffer
buffer_pos: usize,
/// Amount of data in the buffer
buffer_len: usize,
}
#[cfg(target_os = "linux")]
impl AlignedPreadReader {
/// Alignment requirement for reads (512 bytes for most systems)
pub const ALIGNMENT: usize = 512;
/// Create a new aligned pread-based reader.
///
/// # Arguments
///
/// * `file` - File to read from
/// * `offset` - Starting offset in the file (must be 512-byte aligned)
/// * `size` - Number of bytes to read (must be 512-byte aligned)
///
/// # Returns
///
/// An `AlignedPreadReader` that reads the file at the given offset.
///
/// # Errors
///
/// Returns an error if offset or size are not 512-byte aligned.
pub fn new(file: std::fs::File, offset: u64, size: usize) -> Result<Self, AlignedPreadError> {
// Check alignment
if !offset.is_multiple_of(Self::ALIGNMENT as u64) {
return Err(AlignedPreadError::AlignmentError { offset, size });
}
if !size.is_multiple_of(Self::ALIGNMENT) {
return Err(AlignedPreadError::AlignmentError { offset, size });
}
Ok(Self {
file,
pos: offset,
remaining: size,
buffer: Vec::new(),
buffer_pos: 0,
buffer_len: 0,
})
}
/// Read a chunk of data using aligned pread.
///
/// This method performs aligned reads and handles the buffering required
/// by this aligned pread implementation. It does not use `O_DIRECT`.
fn read_chunk(&mut self, buf: &mut [u8]) -> io::Result<usize> {
// If buffer is exhausted, read more data
if self.buffer_pos >= self.buffer_len {
if self.remaining == 0 {
return Ok(0);
}
// Allocate aligned buffer
let chunk_size = (self.remaining).min(64 * 1024); // 64KB chunks
let aligned_size = chunk_size.div_ceil(Self::ALIGNMENT) * Self::ALIGNMENT;
self.buffer = vec![0u8; aligned_size];
// Use pread for atomic read at position (no file offset modification)
use std::os::unix::fs::FileExt;
let n = self.file.read_at(&mut self.buffer, self.pos)?;
self.buffer_pos = 0;
self.buffer_len = n;
self.pos += n as u64;
self.remaining -= n;
if n == 0 {
return Ok(0);
}
}
// Copy from buffer to user buffer
let available = self.buffer_len - self.buffer_pos;
let to_copy = buf.len().min(available);
buf[..to_copy].copy_from_slice(&self.buffer[self.buffer_pos..self.buffer_pos + to_copy]);
self.buffer_pos += to_copy;
Ok(to_copy)
}
}
#[cfg(target_os = "linux")]
impl AsyncRead for AlignedPreadReader {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let filled = buf.filled().len();
let mut remaining = buf.initialize_unfilled();
while !remaining.is_empty() {
match self.read_chunk(remaining) {
Ok(0) => break,
Ok(n) => {
remaining = &mut remaining[n..];
}
Err(e) => return Poll::Ready(Err(e)),
}
}
let _n_read = buf.filled().len() - filled;
Poll::Ready(Ok(()))
}
}
/// Aligned pread reader stub for non-Linux platforms.
///
/// On non-Linux platforms, `read_at`-based I/O is not available through this
/// type. This stub exists to provide a consistent API across platforms.
#[cfg(not(target_os = "linux"))]
pub struct AlignedPreadReader {
_priv: (),
}
#[cfg(not(target_os = "linux"))]
impl AlignedPreadReader {
/// Create a new aligned pread reader (not supported on this platform).
///
/// Always returns an error on non-Linux platforms.
pub fn new(_file: std::fs::File, _offset: u64, _size: usize) -> Result<Self, AlignedPreadError> {
Err(AlignedPreadError::UnsupportedPlatform)
}
}
#[cfg(not(target_os = "linux"))]
impl AsyncRead for AlignedPreadReader {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::new(
io::ErrorKind::Unsupported,
"Aligned pread-based I/O not supported on this platform",
)))
}
}
impl std::fmt::Debug for AlignedPreadReader {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
#[cfg(target_os = "linux")]
{
f.debug_struct("AlignedPreadReader")
.field("pos", &self.pos)
.field("remaining", &self.remaining)
.field("buffer_len", &self.buffer_len)
.finish()
}
#[cfg(not(target_os = "linux"))]
{
f.debug_struct("AlignedPreadReader")
.field("platform", &"unsupported")
.finish()
}
}
}
/// Historical name for aligned pread errors.
#[deprecated(since = "1.0.0-beta.8", note = "use AlignedPreadError; this reader does not set O_DIRECT")]
pub type DirectIoError = AlignedPreadError;
/// Historical name for the aligned pread-based reader.
#[deprecated(since = "1.0.0-beta.8", note = "use AlignedPreadReader; this reader does not set O_DIRECT")]
pub type DirectIoReader = AlignedPreadReader;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_alignment_check() {
#[cfg(target_os = "linux")]
{
// Valid alignment
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(
AlignedPreadReader::new(file, 0, 512).is_ok(),
"Should succeed with aligned offset and size"
);
let file = std::fs::File::open("/dev/zero").expect("open /dev/zero for alias");
assert!(
AlignedPreadReader::new(file, 0, 512).is_ok(),
"Should succeed through aligned pread alias"
);
// Invalid offset
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(AlignedPreadReader::new(file, 1, 512).is_err(), "Should fail with unaligned offset");
// Invalid size
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(AlignedPreadReader::new(file, 0, 511).is_err(), "Should fail with unaligned size");
}
#[cfg(not(target_os = "linux"))]
{
// Non-Linux should return UnsupportedPlatform
let file = std::fs::File::open(std::env::current_exe().unwrap()).unwrap();
assert!(matches!(
AlignedPreadReader::new(file, 0, 512),
Err(AlignedPreadError::UnsupportedPlatform)
));
}
}
#[test]
#[allow(deprecated)]
fn test_legacy_direct_io_alias() {
#[cfg(target_os = "linux")]
{
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(DirectIoReader::new(file, 0, 512).is_ok());
}
#[cfg(not(target_os = "linux"))]
{
let file = std::fs::File::open(std::env::current_exe().unwrap()).unwrap();
assert!(matches!(DirectIoReader::new(file, 0, 512), Err(AlignedPreadError::UnsupportedPlatform)));
}
}
}
+381
View File
@@ -0,0 +1,381 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! I/O priority queue for scheduling I/O operations.
//!
//! This module provides a priority queue implementation for I/O operations
//! with support for starvation prevention and fair scheduling.
use crate::config::IoPriorityQueueConfig;
use crate::scheduler::IoPriority;
use std::collections::VecDeque;
use std::time::{Duration, Instant};
/// A queued I/O request.
#[derive(Debug, Clone)]
pub struct IoRequest {
/// Request ID.
pub id: u64,
/// Request priority.
pub priority: IoPriority,
/// Request size in bytes.
pub size: usize,
/// Queue time.
pub queued_at: Instant,
/// Whether this is a sequential read.
pub is_sequential: bool,
}
impl IoRequest {
/// Create a new I/O request.
pub fn new(id: u64, priority: IoPriority, size: usize, is_sequential: bool) -> Self {
Self {
id,
priority,
size,
queued_at: Instant::now(),
is_sequential,
}
}
/// Get the wait time in the queue.
pub fn wait_time(&self) -> Duration {
self.queued_at.elapsed()
}
}
/// Queue status for a priority level.
#[derive(Debug, Clone, Default)]
pub struct IoQueueStatus {
/// Number of requests in the queue.
pub count: usize,
/// Total size of all requests.
pub total_size: usize,
/// Oldest request wait time.
pub oldest_wait: Option<Duration>,
/// Number of requests processed.
pub processed: u64,
}
impl IoQueueStatus {
/// Create new queue status.
pub fn new() -> Self {
Self::default()
}
}
/// I/O priority queue.
pub struct IoPriorityQueue {
/// Queue configuration.
config: IoPriorityQueueConfig,
/// High priority queue.
high: VecDeque<IoRequest>,
/// Normal priority queue.
normal: VecDeque<IoRequest>,
/// Low priority queue.
low: VecDeque<IoRequest>,
/// Next request ID.
next_id: u64,
/// Last dequeue time for each priority (for starvation prevention).
last_dequeue: [Option<Instant>; 3],
/// Statistics for each queue.
stats: [IoQueueStatus; 3],
}
impl IoPriorityQueue {
/// Create a new priority queue with the given configuration.
pub fn new(config: IoPriorityQueueConfig) -> Self {
Self {
config,
high: VecDeque::with_capacity(100),
normal: VecDeque::with_capacity(500),
low: VecDeque::with_capacity(200),
next_id: 0,
last_dequeue: [None, None, None],
stats: [IoQueueStatus::new(), IoQueueStatus::new(), IoQueueStatus::new()],
}
}
/// Create with default configuration.
pub fn with_defaults() -> Self {
Self::new(IoPriorityQueueConfig::default())
}
/// Get the configuration.
pub fn config(&self) -> &IoPriorityQueueConfig {
&self.config
}
/// Enqueue a request.
pub fn enqueue(&mut self, priority: IoPriority, size: usize, is_sequential: bool) -> u64 {
let id = self.next_id;
self.next_id += 1;
let request = IoRequest::new(id, priority, size, is_sequential);
match priority {
IoPriority::High => {
if self.high.len() < self.config.high_capacity {
self.high.push_back(request);
}
}
IoPriority::Normal => {
if self.normal.len() < self.config.normal_capacity {
self.normal.push_back(request);
}
}
IoPriority::Low => {
if self.low.len() < self.config.low_capacity {
self.low.push_back(request);
}
}
}
id
}
/// Dequeue the next request.
///
/// Uses weighted fair queuing with starvation prevention.
pub fn dequeue(&mut self) -> Option<IoRequest> {
let now = Instant::now();
// Check for starvation: if a lower priority queue hasn't been served in a while,
// give it priority
let normal_starved = self.is_starved(IoPriority::Normal, now);
let low_starved = self.is_starved(IoPriority::Low, now);
// Priority order with starvation consideration
// Check conditions first, then dequeue
let dequeue_high = !self.high.is_empty() && !low_starved && !normal_starved;
let dequeue_normal = !self.normal.is_empty() && !low_starved;
let dequeue_low = !self.low.is_empty();
let dequeue_high_fallback = !self.high.is_empty();
let dequeue_normal_fallback = !self.normal.is_empty();
if dequeue_high {
let request = self.high.pop_front();
if request.is_some() {
self.last_dequeue[0] = Some(Instant::now());
self.stats[0].processed += 1;
}
request
} else if dequeue_normal {
let request = self.normal.pop_front();
if request.is_some() {
self.last_dequeue[1] = Some(Instant::now());
self.stats[1].processed += 1;
}
request
} else if dequeue_low {
let request = self.low.pop_front();
if request.is_some() {
self.last_dequeue[2] = Some(Instant::now());
self.stats[2].processed += 1;
}
request
} else if dequeue_high_fallback {
let request = self.high.pop_front();
if request.is_some() {
self.last_dequeue[0] = Some(Instant::now());
self.stats[0].processed += 1;
}
request
} else if dequeue_normal_fallback {
let request = self.normal.pop_front();
if request.is_some() {
self.last_dequeue[1] = Some(Instant::now());
self.stats[1].processed += 1;
}
request
} else {
None
}
}
/// Check if a priority level is starved.
fn is_starved(&self, priority: IoPriority, now: Instant) -> bool {
let idx = match priority {
IoPriority::High => 0,
IoPriority::Normal => 1,
IoPriority::Low => 2,
};
if let Some(last) = self.last_dequeue[idx] {
now.duration_since(last) > self.config.starvation_threshold
} else {
false
}
}
/// Get the total number of queued requests.
pub fn len(&self) -> usize {
self.high.len() + self.normal.len() + self.low.len()
}
/// Check if the queue is empty.
pub fn is_empty(&self) -> bool {
self.high.is_empty() && self.normal.is_empty() && self.low.is_empty()
}
/// Get queue status for a priority level.
pub fn status(&self, priority: IoPriority) -> IoQueueStatus {
let (queue, idx) = match priority {
IoPriority::High => (&self.high, 0),
IoPriority::Normal => (&self.normal, 1),
IoPriority::Low => (&self.low, 2),
};
let mut status = self.stats[idx].clone();
status.count = queue.len();
status.total_size = queue.iter().map(|r| r.size).sum();
status.oldest_wait = queue.front().map(|r| r.wait_time());
status
}
/// Get the total queue status.
pub fn total_status(&self) -> IoQueueStatus {
let mut total = IoQueueStatus::new();
total.count = self.len();
total.total_size = self
.high
.iter()
.chain(self.normal.iter())
.chain(self.low.iter())
.map(|r| r.size)
.sum();
total.processed = self.stats.iter().map(|s| s.processed).sum();
total.oldest_wait = self
.high
.front()
.map(|r| r.wait_time())
.or_else(|| self.normal.front().map(|r| r.wait_time()))
.or_else(|| self.low.front().map(|r| r.wait_time()));
total
}
/// Clear all queues.
pub fn clear(&mut self) {
self.high.clear();
self.normal.clear();
self.low.clear();
}
/// Peek at the next request without removing it.
pub fn peek(&self) -> Option<&IoRequest> {
if !self.high.is_empty() {
self.high.front()
} else if !self.normal.is_empty() {
self.normal.front()
} else {
self.low.front()
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_enqueue_dequeue() {
let mut queue = IoPriorityQueue::with_defaults();
let id1 = queue.enqueue(IoPriority::High, 1024, true);
let id2 = queue.enqueue(IoPriority::Normal, 2048, false);
let id3 = queue.enqueue(IoPriority::Low, 4096, true);
assert_eq!(queue.len(), 3);
// High priority should be dequeued first
let req1 = queue.dequeue().unwrap();
assert_eq!(req1.id, id1);
assert_eq!(req1.priority, IoPriority::High);
let req2 = queue.dequeue().unwrap();
assert_eq!(req2.id, id2);
assert_eq!(req2.priority, IoPriority::Normal);
let req3 = queue.dequeue().unwrap();
assert_eq!(req3.id, id3);
assert_eq!(req3.priority, IoPriority::Low);
assert!(queue.is_empty());
}
#[test]
fn test_queue_status() {
let mut queue = IoPriorityQueue::with_defaults();
queue.enqueue(IoPriority::High, 1024, true);
queue.enqueue(IoPriority::High, 2048, true);
queue.enqueue(IoPriority::Normal, 4096, false);
let high_status = queue.status(IoPriority::High);
assert_eq!(high_status.count, 2);
assert_eq!(high_status.total_size, 3072);
let normal_status = queue.status(IoPriority::Normal);
assert_eq!(normal_status.count, 1);
assert_eq!(normal_status.total_size, 4096);
let total = queue.total_status();
assert_eq!(total.count, 3);
assert_eq!(total.total_size, 7168);
}
#[test]
fn test_queue_capacity() {
let config = IoPriorityQueueConfig {
high_capacity: 2,
normal_capacity: 2,
low_capacity: 2,
..Default::default()
};
let mut queue = IoPriorityQueue::new(config);
queue.enqueue(IoPriority::High, 1024, true);
queue.enqueue(IoPriority::High, 1024, true);
queue.enqueue(IoPriority::High, 1024, true); // Should be dropped
assert_eq!(queue.status(IoPriority::High).count, 2);
}
#[test]
fn test_clear() {
let mut queue = IoPriorityQueue::with_defaults();
queue.enqueue(IoPriority::High, 1024, true);
queue.enqueue(IoPriority::Normal, 2048, false);
queue.enqueue(IoPriority::Low, 4096, true);
assert_eq!(queue.len(), 3);
queue.clear();
assert!(queue.is_empty());
}
#[test]
fn test_peek() {
let mut queue = IoPriorityQueue::with_defaults();
queue.enqueue(IoPriority::Normal, 2048, false);
queue.enqueue(IoPriority::High, 1024, true);
let peeked = queue.peek().unwrap();
assert_eq!(peeked.priority, IoPriority::High);
// Peek shouldn't remove the item
assert_eq!(queue.len(), 2);
}
}
+61 -12
View File
@@ -12,39 +12,85 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Shared I/O primitives for RustFS.
//! Buffered I/O reader and writer implementations for RustFS.
//!
//! This crate holds the buffer pool and the concurrency-control primitives
//! that the storage layer builds on:
//! This crate provides buffered readers and writers for I/O operations.
//! Prefer `BytesBufferedReader`, `BytesMutWriter`, and `AlignedPreadReader`
//! for new code. Historical `ZeroCopy*` and `DirectIo*` names remain exported
//! for backward compatibility.
//!
//! - Tiered `BytesPool` for buffer management
//! - Storage-media and access-pattern profiling (`io_profile`)
//! - Scheduler and priority-queue configuration shapes
//! - Backpressure admission, deadlock detection, lock optimization
//! - Progress tracking for long-running operations
//! # Features
//!
//! - Memory-mapped file reading (mmap-then-copy) on Unix platforms
//! - Bytes-based buffered wrapping
//! - AsyncRead trait implementations
//! - Tiered BytesPool for buffer management
//! - Aligned pread-based reader (NOT true Direct I/O / O_DIRECT)
//!
//! # Example
//!
//! ```ignore
//! use rustfs_io_core::BytesPool;
//! use rustfs_io_core::{BytesBufferedReader, BytesPool};
//! use bytes::Bytes;
//!
//! // Create from existing bytes (zero-copy)
//! let data = Bytes::from("hello world");
//! let reader = BytesBufferedReader::from_bytes(data);
//!
//! // Create from file using buffered reads
//! let reader = BytesBufferedReader::from_file_read(&file, 0, 1024).await?;
//!
//! // Use BytesPool
//! let pool = BytesPool::new_tiered();
//! let mut buffer = pool.acquire_buffer(8192).await;
//! ```
pub mod backpressure;
pub mod bufreader_optimizer;
pub mod config;
pub mod deadlock_detector;
pub mod direct_io;
pub mod io_priority_queue;
pub mod io_profile;
pub mod lock_optimizer;
pub mod pool;
pub mod progress;
pub mod reader;
pub mod scheduler;
pub mod shared_memory;
pub mod timeout_wrapper;
pub mod writer;
#[cfg(target_os = "linux")]
pub use direct_io::{AlignedPreadError, AlignedPreadReader};
#[cfg(target_os = "linux")]
#[allow(deprecated)]
pub use direct_io::{DirectIoError, DirectIoReader};
pub use pool::{BytesPool, BytesPoolConfig, BytesPoolMetrics, PooledBuffer};
#[allow(deprecated)]
pub use reader::ZeroCopyObjectReader;
pub use reader::{BytesBufferedReader, ZeroCopyReadError};
#[allow(deprecated)]
pub use writer::ZeroCopyObjectWriter;
pub use writer::{BytesMutWriter, ZeroCopyWriteError};
// BufReader optimizer exports
pub use bufreader_optimizer::{BufReaderConfig, BufReaderOptimizer, BufReaderStats, BufferedSource};
// Shared memory exports
pub use shared_memory::{ArcData, ArcMetadata, SharedMemoryConfig, SharedMemoryPool, SharedMemoryStats};
// Config exports
pub use config::{ConfigError, IoPriorityQueueConfig, IoSchedulerConfig};
// Scheduler exports
pub use scheduler::{
BandwidthTier, IoLoadLevel, IoLoadMetrics, IoPriority, IoScheduler, IoSchedulingContext, IoStrategy, KI_B, MI_B,
calculate_optimal_buffer_size, get_advanced_buffer_size, get_buffer_size_for_media, get_concurrency_aware_buffer_size,
};
// Priority queue exports
pub use io_priority_queue::{IoPriorityQueue, IoQueueStatus, IoRequest};
// Backpressure exports
pub use backpressure::{BackpressureConfig, BackpressureError, BackpressureMonitor, BackpressureState};
@@ -54,5 +100,8 @@ pub use deadlock_detector::{DeadlockDetector, DeadlockDetectorConfig, LockInfo,
// Lock optimizer exports
pub use lock_optimizer::{LockGuard, LockOptimizeConfig, LockOptimizer, LockStats};
// Progress tracking exports
pub use progress::OperationProgress;
// Timeout wrapper exports
pub use timeout_wrapper::{
OperationProgress, RequestTimeoutWrapper, TimeoutConfig, TimeoutError, TimeoutStats, calculate_adaptive_timeout,
estimate_bytes_per_second,
};
-138
View File
@@ -1,138 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Progress tracking for long-running I/O operations.
//!
//! Re-exported as `rustfs_concurrency::OperationProgress` for the storage
//! timeout implementation, which uses `is_stale` to tell a slow transfer
//! apart from a stalled one.
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
/// Operation progress tracker.
#[derive(Debug)]
pub struct OperationProgress {
/// Total size (if known).
pub total_size: Option<u64>,
/// Bytes processed.
bytes_processed: AtomicU64,
/// Last update time.
last_update: std::sync::Mutex<Instant>,
/// Stale timeout.
stale_timeout: Duration,
/// Start time for transfer rate calculation.
start_time: Instant,
}
impl OperationProgress {
/// Create new operation progress.
pub fn new(total_size: Option<u64>, stale_timeout: Duration) -> Self {
Self {
total_size,
bytes_processed: AtomicU64::new(0),
last_update: std::sync::Mutex::new(Instant::now()),
stale_timeout,
start_time: Instant::now(),
}
}
/// Update progress.
pub fn update(&self, bytes: u64) {
self.bytes_processed.store(bytes, Ordering::Relaxed);
if let Ok(mut last) = self.last_update.lock() {
*last = Instant::now();
}
}
/// Add to progress.
pub fn add(&self, bytes: u64) {
self.bytes_processed.fetch_add(bytes, Ordering::Relaxed);
if let Ok(mut last) = self.last_update.lock() {
*last = Instant::now();
}
}
/// Get current progress.
pub fn current(&self) -> u64 {
self.bytes_processed.load(Ordering::Relaxed)
}
/// Check if progress is stale.
pub fn is_stale(&self) -> bool {
if let Ok(last) = self.last_update.lock() {
last.elapsed() > self.stale_timeout
} else {
false
}
}
/// Get progress percentage.
pub fn progress_percent(&self) -> Option<f64> {
self.total_size.map(|total| {
if total == 0 {
100.0
} else {
let processed = self.bytes_processed.load(Ordering::Relaxed);
(processed as f64 / total as f64 * 100.0).min(100.0)
}
})
}
/// Get remaining bytes.
pub fn remaining(&self) -> Option<u64> {
self.total_size.map(|total| {
let processed = self.bytes_processed.load(Ordering::Relaxed);
total.saturating_sub(processed)
})
}
/// Calculate transfer rate in bytes per second.
///
/// Returns 0 if no time has elapsed or no data transferred.
pub fn transfer_rate(&self) -> u64 {
let processed = self.bytes_processed.load(Ordering::Relaxed);
if processed == 0 {
return 0;
}
let elapsed = self.start_time.elapsed().as_secs_f64();
if elapsed > 0.0 {
(processed as f64 / elapsed) as u64
} else {
0
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_operation_progress() {
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
assert_eq!(progress.current(), 0);
assert_eq!(progress.progress_percent(), Some(0.0));
progress.update(500);
assert_eq!(progress.current(), 500);
assert_eq!(progress.progress_percent(), Some(50.0));
progress.add(300);
assert_eq!(progress.current(), 800);
assert_eq!(progress.remaining(), Some(200));
}
}
+412
View File
@@ -0,0 +1,412 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bytes-backed object reader implementation.
use bytes::Bytes;
use std::io;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
/// Errors that can occur during Bytes-backed read operations.
#[derive(Debug, Clone)]
pub enum ZeroCopyReadError {
/// I/O error occurred.
Io(String),
/// Memory mapping error.
Mmap(String),
/// Invalid offset or size.
InvalidRange,
}
impl std::fmt::Display for ZeroCopyReadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Io(msg) => write!(f, "I/O error: {}", msg),
Self::Mmap(msg) => write!(f, "Mmap error: {}", msg),
Self::InvalidRange => write!(f, "Invalid offset or size"),
}
}
}
impl std::error::Error for ZeroCopyReadError {}
impl From<io::Error> for ZeroCopyReadError {
fn from(err: io::Error) -> Self {
Self::Io(err.to_string())
}
}
/// Bytes-backed object reader.
///
/// `from_bytes` wraps existing `Bytes` without copying, but file constructors
/// copy file data into owned `Bytes` after mmap or normal reads.
///
/// # Example
///
/// ```ignore
/// use bytes::Bytes;
/// use rustfs_io_core::BytesBufferedReader;
///
/// // Create from bytes without copying the `Bytes` buffer
/// let data = Bytes::from("hello world");
/// let reader = BytesBufferedReader::from_bytes(data);
///
/// // Read using AsyncRead trait
/// let mut buf = vec![0u8; 1024];
/// let n = reader.read(&mut buf[..]).await?;
/// ```
pub struct BytesBufferedReader {
/// Internal data source (could be mmap or owned bytes)
data: Bytes,
/// Current read position
pos: usize,
}
/// Historical name for the bytes-backed object reader.
#[deprecated(
since = "1.0.0-beta.8",
note = "use BytesBufferedReader; file constructors copy into owned Bytes"
)]
pub type ZeroCopyObjectReader = BytesBufferedReader;
impl BytesBufferedReader {
/// Create a reader from existing bytes.
///
/// This is a true zero-copy operation - the Bytes are wrapped
/// without any allocation or copying.
///
/// # Arguments
///
/// * `data` - Bytes to wrap
///
/// # Example
///
/// ```ignore
/// let data = Bytes::from("hello world");
/// let reader = BytesBufferedReader::from_bytes(data);
/// ```
pub fn from_bytes(data: Bytes) -> Self {
Self { data, pos: 0 }
}
/// Create a Bytes-backed reader from a file using mmap-then-copy.
///
/// This maps the requested file range and copies it into owned `Bytes`
/// before returning. It does not expose the mmap as a zero-copy buffer.
///
/// # Arguments
///
/// * `path` - Path to the file to memory map
/// * `offset` - Offset within the file to start reading
/// * `size` - Number of bytes to read
///
/// # Returns
///
/// A reader backed by copied file data.
///
/// # Errors
///
/// Returns an error if the file cannot be memory mapped.
///
/// # Example
///
/// ```ignore
/// let reader = BytesBufferedReader::from_file_mmap_path("large_file.bin", 0, 1024).await?;
/// ```
#[cfg(unix)]
// SAFETY: The mmap is created from a read-only file handle for the
// caller-provided range, then copied into owned `Bytes` before the file and
// mapping are dropped.
#[allow(unsafe_code)]
pub async fn from_file_mmap_path(path: &std::path::Path, offset: u64, size: usize) -> Result<Self, ZeroCopyReadError> {
use memmap2::MmapOptions;
let path = path.to_path_buf();
let (offset, size) = (offset, size);
tokio::task::spawn_blocking(move || {
// Open the file in sync context
let std_file = std::fs::File::open(&path).map_err(|e| ZeroCopyReadError::Io(e.to_string()))?;
// SAFETY: `std_file` remains open while the mapping is created and
// copied, and the mapped bytes are not exposed beyond this closure.
let mmap = unsafe { MmapOptions::new().offset(offset).len(size).map(&std_file) }
.map_err(|e| ZeroCopyReadError::Mmap(e.to_string()))?;
// Convert to Bytes (this is a copy, but only done once)
Ok(Self {
data: Bytes::copy_from_slice(&mmap),
pos: 0,
})
})
.await
.map_err(|e| ZeroCopyReadError::Io(e.to_string()))?
}
/// Create a Bytes-backed reader from a file using normal reads.
///
/// This path reads the requested range into an owned buffer and wraps it in
/// `Bytes`. It does not perform mmap or zero-copy file I/O.
///
/// # Arguments
///
/// * `file` - File to read from
/// * `offset` - Offset within the file to start reading
/// * `size` - Number of bytes to map
///
/// # Returns
///
/// A reader backed by copied file data.
///
/// # Errors
///
/// Returns an error if the file cannot be read.
///
/// # Example
///
/// ```ignore
/// let file = tokio::fs::File::open("large_file.bin").await?;
/// let reader = BytesBufferedReader::from_file_read(&file, 0, 1024).await?;
/// ```
#[cfg(unix)]
pub async fn from_file_read(file: &tokio::fs::File, offset: u64, size: usize) -> Result<Self, ZeroCopyReadError> {
use tokio::io::{AsyncReadExt, AsyncSeekExt, SeekFrom};
let mut cloned = file.try_clone().await?;
cloned.seek(SeekFrom::Start(offset)).await?;
let mut buffer = vec![0u8; size];
cloned.read_exact(&mut buffer).await?;
Ok(Self {
data: Bytes::from(buffer),
pos: 0,
})
}
/// Create a Bytes-backed reader from a file (non-Unix fallback).
///
/// On platforms that don't support mmap, this falls back to regular file I/O.
#[cfg(not(unix))]
pub async fn from_file_read(file: &tokio::fs::File, offset: u64, size: usize) -> Result<Self, ZeroCopyReadError> {
use tokio::io::{AsyncReadExt, AsyncSeekExt, SeekFrom};
let mut cloned = file.try_clone().await?;
cloned.seek(SeekFrom::Start(offset)).await?;
let mut buffer = vec![0u8; size];
cloned.read_exact(&mut buffer).await?;
Ok(Self {
data: Bytes::from(buffer),
pos: 0,
})
}
/// Historical name for `from_file_read`.
#[deprecated(
since = "1.0.0-beta.8",
note = "use from_file_read; this method performs normal reads into owned Bytes"
)]
pub async fn from_file_mmap(file: &tokio::fs::File, offset: u64, size: usize) -> Result<Self, ZeroCopyReadError> {
Self::from_file_read(file, offset, size).await
}
/// Get the remaining data as Bytes (zero-copy).
///
/// This returns a slice of the remaining data without copying.
/// The returned Bytes shares the underlying memory with this reader.
///
/// # Example
///
/// ```ignore
/// let remaining = reader.remaining_bytes();
/// println!("Remaining: {} bytes", remaining.len());
/// ```
pub fn remaining_bytes(&self) -> Bytes {
self.data.slice(self.pos..)
}
/// Get the total length of the data.
pub fn len(&self) -> usize {
self.data.len()
}
/// Check if the reader has reached the end.
pub fn is_empty(&self) -> bool {
self.pos >= self.data.len()
}
/// Get the current read position.
pub fn position(&self) -> usize {
self.pos
}
}
impl AsyncRead for BytesBufferedReader {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let remaining = self.data.len() - self.pos;
if remaining == 0 {
return Poll::Ready(Ok(()));
}
let to_read = std::cmp::min(remaining, buf.remaining());
let slice = &self.data[self.pos..self.pos + to_read];
buf.put_slice(slice);
self.pos += to_read;
Poll::Ready(Ok(()))
}
}
impl std::fmt::Debug for BytesBufferedReader {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("BytesBufferedReader")
.field("data_len", &self.data.len())
.field("pos", &self.pos)
.field("remaining", &(self.data.len() - self.pos))
.finish()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::PathBuf;
use tokio::io::AsyncReadExt;
fn temp_file_path(test_name: &str) -> PathBuf {
let nonce = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.expect("system time should be after unix epoch")
.as_nanos();
std::env::temp_dir().join(format!("rustfs-io-core-{test_name}-{}-{nonce}", std::process::id()))
}
#[tokio::test]
async fn test_from_bytes() {
let data = Bytes::from("hello world");
let mut reader = BytesBufferedReader::from_bytes(data.clone());
let mut buf = [0u8; 11];
let n = reader.read(&mut buf[..]).await.unwrap();
assert_eq!(n, 11);
assert_eq!(&buf[..n], b"hello world");
}
#[tokio::test]
async fn test_preferred_reader_alias() {
let data = Bytes::from("hello world");
let mut reader = BytesBufferedReader::from_bytes(data);
let mut buf = [0u8; 5];
let n = reader.read(&mut buf[..]).await.expect("read bytes from alias");
assert_eq!(n, 5);
assert_eq!(&buf[..n], b"hello");
}
#[tokio::test]
async fn test_from_file_read_reads_requested_range() {
let path = temp_file_path("from-file-read");
tokio::fs::write(&path, b"hello world")
.await
.expect("write temp file for reader test");
let file = tokio::fs::File::open(&path).await.expect("open temp file for reader test");
let mut reader = BytesBufferedReader::from_file_read(&file, 6, 5)
.await
.expect("read requested range into Bytes");
let mut output = Vec::new();
reader.read_to_end(&mut output).await.expect("drain reader output");
assert_eq!(output, b"world");
let _ = tokio::fs::remove_file(path).await;
}
#[tokio::test]
#[allow(deprecated)]
async fn test_from_file_mmap_legacy_alias_reads_requested_range() {
let path = temp_file_path("from-file-mmap");
tokio::fs::write(&path, b"hello world")
.await
.expect("write temp file for legacy reader test");
let file = tokio::fs::File::open(&path)
.await
.expect("open temp file for legacy reader test");
let mut reader = BytesBufferedReader::from_file_mmap(&file, 0, 5)
.await
.expect("read requested range through legacy alias");
let mut output = Vec::new();
reader.read_to_end(&mut output).await.expect("drain legacy reader output");
assert_eq!(output, b"hello");
let _ = tokio::fs::remove_file(path).await;
}
#[tokio::test]
async fn test_remaining_bytes() {
let data = Bytes::from("hello world");
let reader = BytesBufferedReader::from_bytes(data);
let remaining = reader.remaining_bytes();
assert_eq!(remaining.len(), 11);
assert_eq!(&remaining[..], b"hello world");
}
#[tokio::test]
async fn test_position() {
let data = Bytes::from("hello world");
let mut reader = BytesBufferedReader::from_bytes(data);
assert_eq!(reader.position(), 0);
let mut buf = [0u8; 5];
reader.read_exact(&mut buf[..]).await.unwrap();
assert_eq!(reader.position(), 5);
}
#[tokio::test]
async fn test_is_empty() {
let data = Bytes::from("");
let reader = BytesBufferedReader::from_bytes(data);
assert!(reader.is_empty());
let data = Bytes::from("hello");
let reader = BytesBufferedReader::from_bytes(data);
assert!(!reader.is_empty());
}
#[tokio::test]
#[allow(deprecated)]
async fn test_legacy_reader_alias() {
let data = Bytes::from("hello world");
let mut reader = ZeroCopyObjectReader::from_bytes(data);
let mut buf = [0u8; 5];
let n = reader.read(&mut buf[..]).await.expect("read bytes through legacy alias");
assert_eq!(n, 5);
assert_eq!(&buf[..n], b"hello");
}
}
+882
View File
@@ -0,0 +1,882 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! I/O scheduler for adaptive buffer sizing and load management.
//!
//! This module provides the core I/O scheduling logic that determines
//! optimal buffer sizes, I/O strategies, and load management decisions.
use crate::config::IoSchedulerConfig;
use crate::io_profile::{AccessPattern, StorageMedia, StorageProfile};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
/// I/O priority levels.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
pub enum IoPriority {
/// High priority for small, latency-sensitive operations.
High,
/// Normal priority for standard operations.
#[default]
Normal,
/// Low priority for large, throughput-oriented operations.
Low,
}
impl IoPriority {
/// Determine priority based on request size.
///
/// A negative `size` means the size is unknown (-1 by convention) and maps
/// to `Normal`; casting it to `usize` would wrap to a huge value and
/// misclassify the request as `Low`.
pub fn from_size(size: i64, high_threshold: usize, low_threshold: usize) -> Self {
if size < 0 {
return IoPriority::Normal;
}
let size = size as usize;
if size < high_threshold {
IoPriority::High
} else if size > low_threshold {
IoPriority::Low
} else {
IoPriority::Normal
}
}
/// Get the priority as a string for metrics labels.
pub fn as_str(&self) -> &'static str {
match self {
IoPriority::High => "high",
IoPriority::Normal => "normal",
IoPriority::Low => "low",
}
}
/// Check if this is high priority.
pub fn is_high(&self) -> bool {
matches!(self, IoPriority::High)
}
/// Check if this is normal priority.
pub fn is_normal(&self) -> bool {
matches!(self, IoPriority::Normal)
}
/// Check if this is low priority.
pub fn is_low(&self) -> bool {
matches!(self, IoPriority::Low)
}
}
impl std::fmt::Display for IoPriority {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.as_str())
}
}
/// I/O load level.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Default)]
pub enum IoLoadLevel {
/// Low load - system is underutilized.
Low,
/// Medium load - system is moderately utilized.
#[default]
Medium,
/// High load - system is heavily utilized.
High,
/// Critical load - system is overloaded.
Critical,
}
impl IoLoadLevel {
/// Get the load level as a string for metrics labels.
pub fn as_str(&self) -> &'static str {
match self {
IoLoadLevel::Low => "low",
IoLoadLevel::Medium => "medium",
IoLoadLevel::High => "high",
IoLoadLevel::Critical => "critical",
}
}
/// Determine load level from wait time.
pub fn from_wait_time(wait_time: Duration, low_threshold: Duration, high_threshold: Duration) -> Self {
if wait_time <= low_threshold {
IoLoadLevel::Low
} else if wait_time <= high_threshold {
IoLoadLevel::Medium
} else if wait_time <= high_threshold * 2 {
IoLoadLevel::High
} else {
IoLoadLevel::Critical
}
}
}
impl std::fmt::Display for IoLoadLevel {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.as_str())
}
}
/// Bandwidth tier classification.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
pub enum BandwidthTier {
/// Low bandwidth (< 100 MB/s).
Low,
/// Medium bandwidth (100-500 MB/s).
#[default]
Medium,
/// High bandwidth (> 500 MB/s).
High,
/// Unknown bandwidth.
Unknown,
}
impl BandwidthTier {
/// Determine bandwidth tier from bytes per second.
pub fn from_bps(bps: u64) -> Self {
const MB: u64 = 1024 * 1024;
if bps < 100 * MB {
BandwidthTier::Low
} else if bps < 500 * MB {
BandwidthTier::Medium
} else {
BandwidthTier::High
}
}
/// Get the tier as a string for metrics labels.
pub fn as_str(&self) -> &'static str {
match self {
BandwidthTier::Low => "low",
BandwidthTier::Medium => "medium",
BandwidthTier::High => "high",
BandwidthTier::Unknown => "unknown",
}
}
}
/// I/O strategy decision.
#[derive(Debug, Clone)]
pub struct IoStrategy {
/// Buffer size to use for I/O operations.
pub buffer_size: usize,
/// Buffer multiplier based on storage media.
pub buffer_multiplier: f64,
/// Whether to enable readahead.
pub enable_readahead: bool,
/// Whether to use buffered I/O.
pub use_buffered_io: bool,
// Performance state
/// Current number of concurrent requests.
pub concurrent_requests: usize,
/// Observed bandwidth in bytes per second.
pub observed_bandwidth_bps: Option<u64>,
/// Bandwidth tier classification.
pub bandwidth_tier: BandwidthTier,
/// Current load level.
pub load_level: IoLoadLevel,
// Priority
/// I/O priority for this operation.
pub priority: IoPriority,
// Decision flags
/// Whether to throttle random I/O.
pub should_throttle_random_io: bool,
/// Whether to expand buffer for sequential access.
pub should_expand_for_sequential: bool,
/// Whether to reduce buffer due to concurrency.
pub should_reduce_for_concurrency: bool,
/// Whether to reduce buffer due to low bandwidth.
pub should_reduce_for_bandwidth: bool,
}
impl Default for IoStrategy {
fn default() -> Self {
Self {
buffer_size: 128 * 1024,
buffer_multiplier: 1.0,
enable_readahead: true,
use_buffered_io: true,
concurrent_requests: 0,
observed_bandwidth_bps: None,
bandwidth_tier: BandwidthTier::Medium,
load_level: IoLoadLevel::Low,
priority: IoPriority::Normal,
should_throttle_random_io: false,
should_expand_for_sequential: false,
should_reduce_for_concurrency: false,
should_reduce_for_bandwidth: false,
}
}
}
impl IoStrategy {
/// Create a new strategy with default values.
pub fn new() -> Self {
Self::default()
}
/// Create a strategy for sequential access.
pub fn sequential(buffer_size: usize) -> Self {
Self {
buffer_size,
enable_readahead: true,
should_expand_for_sequential: true,
..Self::default()
}
}
/// Create a strategy for random access.
pub fn random(buffer_size: usize) -> Self {
Self {
buffer_size,
enable_readahead: false,
should_throttle_random_io: true,
..Self::default()
}
}
}
/// I/O load metrics.
#[derive(Debug, Clone, Default)]
pub struct IoLoadMetrics {
/// Number of samples in the current window.
pub sample_count: usize,
/// Total wait time in the window.
pub total_wait_time: Duration,
/// Maximum wait time in the window.
pub max_wait_time: Duration,
/// Average wait time.
pub avg_wait_time: Duration,
/// Current load level.
pub load_level: IoLoadLevel,
}
impl IoLoadMetrics {
/// Create new load metrics.
pub fn new() -> Self {
Self::default()
}
/// Add a wait time sample.
pub fn add_sample(&mut self, wait_time: Duration) {
self.sample_count += 1;
self.total_wait_time += wait_time;
if wait_time > self.max_wait_time {
self.max_wait_time = wait_time;
}
self.avg_wait_time = if self.sample_count > 0 {
self.total_wait_time / self.sample_count as u32
} else {
Duration::ZERO
};
}
/// Update load level based on thresholds.
pub fn update_load_level(&mut self, low_threshold: Duration, high_threshold: Duration) {
self.load_level = IoLoadLevel::from_wait_time(self.avg_wait_time, low_threshold, high_threshold);
}
/// Reset the metrics.
pub fn reset(&mut self) {
*self = Self::default();
}
}
/// I/O scheduler.
pub struct IoScheduler {
/// Scheduler configuration.
config: IoSchedulerConfig,
/// Active request counter.
active_requests: AtomicUsize,
/// Load metrics.
load_metrics: std::sync::Mutex<IoLoadMetrics>,
}
impl IoScheduler {
/// Create a new I/O scheduler with the given configuration.
pub fn new(config: IoSchedulerConfig) -> Self {
Self {
config,
active_requests: AtomicUsize::new(0),
load_metrics: std::sync::Mutex::new(IoLoadMetrics::new()),
}
}
/// Create a new I/O scheduler with default configuration.
pub fn with_defaults() -> Self {
Self::new(IoSchedulerConfig::default())
}
/// Get the scheduler configuration.
pub fn config(&self) -> &IoSchedulerConfig {
&self.config
}
/// Get the current number of active requests.
pub fn active_requests(&self) -> usize {
self.active_requests.load(Ordering::Relaxed)
}
/// Increment the active request count.
pub fn increment_requests(&self) {
self.active_requests.fetch_add(1, Ordering::Relaxed);
}
/// Decrement the active request count.
pub fn decrement_requests(&self) {
self.active_requests.fetch_sub(1, Ordering::Relaxed);
}
/// Calculate I/O strategy for a request.
pub fn calculate_strategy(&self, file_size: i64, permit_wait_time: Duration, is_sequential: bool) -> IoStrategy {
let concurrent_requests = self.active_requests.load(Ordering::Relaxed);
// Determine priority based on file size
let priority = IoPriority::from_size(
file_size,
self.config.high_priority_size_threshold,
self.config.low_priority_size_threshold,
);
// Determine load level
let load_level =
IoLoadLevel::from_wait_time(permit_wait_time, self.config.load_low_threshold(), self.config.load_high_threshold());
// Calculate base buffer size
let base_buffer = self.config.base_buffer_size;
// Adjust for concurrency
let concurrency_factor = match concurrent_requests {
0..=2 => 1.0,
3..=4 => 0.75,
5..=8 => 0.5,
_ => 0.4,
};
// Adjust for load level
let load_factor = match load_level {
IoLoadLevel::Low => 1.2,
IoLoadLevel::Medium => 1.0,
IoLoadLevel::High => 0.7,
IoLoadLevel::Critical => 0.5,
};
// Adjust for access pattern
let sequential_factor = if is_sequential { 1.5 } else { 1.0 };
// Calculate final buffer size
let buffer_size = (base_buffer as f64 * concurrency_factor * load_factor * sequential_factor) as usize;
let buffer_size = buffer_size.clamp(self.config.min_buffer_size, self.config.max_buffer_size);
IoStrategy {
buffer_size,
buffer_multiplier: concurrency_factor * load_factor * sequential_factor,
enable_readahead: is_sequential && load_level != IoLoadLevel::Critical,
use_buffered_io: true,
concurrent_requests,
observed_bandwidth_bps: None,
bandwidth_tier: BandwidthTier::Unknown,
load_level,
priority,
should_throttle_random_io: !is_sequential && load_level >= IoLoadLevel::High,
should_expand_for_sequential: is_sequential && load_level <= IoLoadLevel::Medium,
should_reduce_for_concurrency: concurrent_requests > 4,
should_reduce_for_bandwidth: false,
}
}
/// Calculate multi-factor I/O strategy.
pub fn calculate_multi_factor_strategy(
&self,
file_size: i64,
permit_wait_time: Duration,
is_sequential: bool,
storage_profile: Option<&StorageProfile>,
) -> IoStrategy {
let mut strategy = self.calculate_strategy(file_size, permit_wait_time, is_sequential);
// Apply storage profile adjustments
if let Some(profile) = storage_profile {
// Adjust buffer size based on storage media
let media_factor = match profile.media {
StorageMedia::Nvme => 1.5,
StorageMedia::Ssd => 1.2,
StorageMedia::Hdd => 0.8,
StorageMedia::Unknown => 1.0,
};
strategy.buffer_size = (strategy.buffer_size as f64 * media_factor).min(self.config.max_buffer_size as f64) as usize;
// Apply sequential boost if applicable
if is_sequential {
strategy.buffer_size = (strategy.buffer_size as f64 * profile.sequential_boost_multiplier)
.min(self.config.max_buffer_size as f64) as usize;
}
// Apply random penalty if applicable
if !is_sequential {
strategy.buffer_size = (strategy.buffer_size as f64 * profile.random_penalty_multiplier)
.max(self.config.min_buffer_size as f64) as usize;
}
// Update readahead preference
strategy.enable_readahead = strategy.enable_readahead && profile.prefers_readahead;
}
strategy
}
/// Record a wait time sample for load tracking.
pub fn record_wait_time(&self, wait_time: Duration) {
if let Ok(mut metrics) = self.load_metrics.lock() {
metrics.add_sample(wait_time);
metrics.update_load_level(self.config.load_low_threshold(), self.config.load_high_threshold());
}
}
/// Get current load metrics.
pub fn load_metrics(&self) -> IoLoadMetrics {
if let Ok(metrics) = self.load_metrics.lock() {
metrics.clone()
} else {
IoLoadMetrics::default()
}
}
}
impl Default for IoScheduler {
fn default() -> Self {
Self::with_defaults()
}
}
// ============================================================================
// Buffer Size Calculation Functions
// ============================================================================
/// Constants for buffer size calculations.
pub const KI_B: usize = 1024;
pub const MI_B: usize = 1024 * 1024;
/// Get concurrency-aware buffer size.
///
/// Adjusts buffer size based on the current level of concurrent requests.
/// Higher concurrency leads to smaller buffers to reduce memory pressure.
///
/// # Arguments
///
/// * `file_size` - Size of the file being read (-1 if unknown)
/// * `base_buffer_size` - Base buffer size from workload profile
///
/// # Returns
///
/// Adjusted buffer size in bytes
pub fn get_concurrency_aware_buffer_size(file_size: i64, base_buffer_size: usize) -> usize {
// Get current concurrency level from global counter
let concurrent_requests = 1; // Default to 1 if no global counter available
// Define concurrency thresholds
let medium_threshold = 4;
let high_threshold = 8;
// Calculate adaptive multiplier based on concurrency
let adaptive_multiplier = if concurrent_requests <= 2 {
// Low concurrency (1-2): use full buffer size
1.0
} else if concurrent_requests <= medium_threshold {
// Medium concurrency (3-4): slightly reduce buffer size (75% of base)
0.75
} else if concurrent_requests <= high_threshold {
// Higher concurrency (5-8): more aggressive reduction (50% of base)
0.5
} else {
// Very high concurrency (>8): minimize memory per request (40% of base)
0.4
};
// Calculate the adjusted buffer size
let adjusted_size = (base_buffer_size as f64 * adaptive_multiplier) as usize;
// Ensure we stay within reasonable bounds
let min_buffer = if file_size > 0 && file_size < 100 * KI_B as i64 {
32 * KI_B // For very small files, use minimum buffer
} else {
64 * KI_B // Standard minimum buffer size
};
let max_buffer = if concurrent_requests > high_threshold {
256 * KI_B // Cap at 256KB for high concurrency
} else {
MI_B // Cap at 1MB for lower concurrency
};
adjusted_size.clamp(min_buffer, max_buffer)
}
/// Advanced concurrency-aware buffer sizing with file size optimization.
///
/// This enhanced version considers both concurrency level and file size patterns
/// to provide even better performance characteristics.
///
/// # Arguments
///
/// * `file_size` - Size of the file being read (-1 if unknown)
/// * `base_buffer_size` - Baseline buffer size from workload profile
/// * `is_sequential` - Whether this is a sequential read (hint for optimization)
/// * `concurrent_requests` - Current number of concurrent requests
///
/// # Returns
///
/// Optimized buffer size in bytes
pub fn get_advanced_buffer_size(
file_size: i64,
base_buffer_size: usize,
is_sequential: bool,
concurrent_requests: usize,
) -> usize {
// For very small files, use smaller buffers regardless of concurrency
if file_size > 0 && file_size < 256 * KI_B as i64 {
return (file_size as usize / 4).clamp(16 * KI_B, 64 * KI_B);
}
// Base calculation from standard function
let standard_size = get_concurrency_aware_buffer_size(file_size, base_buffer_size);
let medium_threshold = 4;
let high_threshold = 8;
// For sequential reads, we can be more aggressive with buffer sizes
if is_sequential && concurrent_requests <= medium_threshold {
// Boost buffer size for sequential reads under low concurrency
let boosted = (standard_size as f64 * 1.5) as usize;
return boosted.min(MI_B);
}
// For random reads under high concurrency, reduce buffer size
if !is_sequential && concurrent_requests > high_threshold {
let reduced = (standard_size as f64 * 0.7) as usize;
return reduced.max(32 * KI_B);
}
standard_size
}
/// Get buffer size with storage media optimization.
///
/// Adjusts buffer size based on storage media characteristics.
///
/// # Arguments
///
/// * `base_size` - Base buffer size
/// * `media` - Storage media type
///
/// # Returns
///
/// Optimized buffer size for the storage media
pub fn get_buffer_size_for_media(base_size: usize, media: StorageMedia) -> usize {
let multiplier = match media {
StorageMedia::Nvme => 1.5, // NVMe can handle larger buffers
StorageMedia::Ssd => 1.2, // SSD benefits from moderate buffers
StorageMedia::Hdd => 0.8, // HDD prefers smaller buffers to reduce seek overhead
StorageMedia::Unknown => 1.0,
};
(base_size as f64 * multiplier).min(MI_B as f64) as usize
}
/// Calculate optimal buffer size using multi-factor analysis.
///
/// This is the main entry point for buffer size calculation, considering
/// all factors: concurrency, storage media, access pattern, and load.
///
/// # Arguments
///
/// * `file_size` - Size of the file being read
/// * `base_buffer_size` - Base buffer size
/// * `is_sequential` - Whether access is sequential
/// * `concurrent_requests` - Current concurrency level
/// * `media` - Storage media type
/// * `load_level` - Current I/O load level
///
/// # Returns
///
/// Optimally calculated buffer size
pub fn calculate_optimal_buffer_size(
file_size: i64,
base_buffer_size: usize,
is_sequential: bool,
concurrent_requests: usize,
media: StorageMedia,
load_level: IoLoadLevel,
) -> usize {
// Start with advanced buffer size calculation
let mut buffer_size = get_advanced_buffer_size(file_size, base_buffer_size, is_sequential, concurrent_requests);
// Apply storage media optimization
buffer_size = get_buffer_size_for_media(buffer_size, media);
// Apply load-based adjustment
let load_multiplier = match load_level {
IoLoadLevel::Low => 1.2,
IoLoadLevel::Medium => 1.0,
IoLoadLevel::High => 0.7,
IoLoadLevel::Critical => 0.5,
};
buffer_size = (buffer_size as f64 * load_multiplier) as usize;
// Final bounds check
buffer_size.clamp(32 * KI_B, MI_B)
}
/// I/O scheduling context for multi-factor strategy calculation.
#[derive(Debug, Clone)]
pub struct IoSchedulingContext {
/// File size in bytes (-1 if unknown).
pub file_size: i64,
/// Base buffer size from configuration.
pub base_buffer_size: usize,
/// Time spent waiting for permit.
pub permit_wait_duration: Duration,
/// Whether access is sequential.
pub is_sequential_hint: bool,
/// Detected access pattern.
pub access_pattern: AccessPattern,
/// Detected storage media.
pub storage_media: StorageMedia,
/// Observed bandwidth in bytes per second.
pub observed_bandwidth_bps: Option<u64>,
/// Current concurrent request count.
pub concurrent_requests: usize,
}
impl Default for IoSchedulingContext {
fn default() -> Self {
Self {
file_size: -1,
base_buffer_size: 128 * KI_B,
permit_wait_duration: Duration::ZERO,
is_sequential_hint: true,
access_pattern: AccessPattern::Unknown,
storage_media: StorageMedia::Unknown,
observed_bandwidth_bps: None,
concurrent_requests: 1,
}
}
}
impl IoSchedulingContext {
/// Create a new scheduling context.
pub fn new(file_size: i64, base_buffer_size: usize) -> Self {
Self {
file_size,
base_buffer_size,
..Self::default()
}
}
/// Builder pattern: set sequential hint.
pub fn with_sequential(mut self, is_sequential: bool) -> Self {
self.is_sequential_hint = is_sequential;
self.access_pattern = if is_sequential {
AccessPattern::Sequential
} else {
AccessPattern::Random
};
self
}
/// Builder pattern: set storage media.
pub fn with_media(mut self, media: StorageMedia) -> Self {
self.storage_media = media;
self
}
/// Builder pattern: set bandwidth.
pub fn with_bandwidth(mut self, bps: u64) -> Self {
self.observed_bandwidth_bps = Some(bps);
self
}
/// Builder pattern: set concurrency.
pub fn with_concurrency(mut self, count: usize) -> Self {
self.concurrent_requests = count;
self
}
/// Builder pattern: set wait duration.
pub fn with_wait_duration(mut self, duration: Duration) -> Self {
self.permit_wait_duration = duration;
self
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_io_priority() {
assert_eq!(IoPriority::from_size(1024, 64 * 1024, 4 * 1024 * 1024), IoPriority::High);
assert_eq!(IoPriority::from_size(1024 * 1024, 64 * 1024, 4 * 1024 * 1024), IoPriority::Normal);
assert_eq!(IoPriority::from_size(10 * 1024 * 1024, 64 * 1024, 4 * 1024 * 1024), IoPriority::Low);
}
#[test]
fn test_io_priority_unknown_size_is_normal() {
// -1 means "size unknown" and must not wrap to usize::MAX (=> Low).
assert_eq!(IoPriority::from_size(-1, 64 * 1024, 4 * 1024 * 1024), IoPriority::Normal);
assert_eq!(IoPriority::from_size(i64::MIN, 64 * 1024, 4 * 1024 * 1024), IoPriority::Normal);
}
#[test]
fn test_io_load_level() {
let low = Duration::from_millis(5);
let high = Duration::from_millis(50);
assert_eq!(IoLoadLevel::from_wait_time(Duration::from_millis(1), low, high), IoLoadLevel::Low);
assert_eq!(IoLoadLevel::from_wait_time(Duration::from_millis(20), low, high), IoLoadLevel::Medium);
assert_eq!(IoLoadLevel::from_wait_time(Duration::from_millis(60), low, high), IoLoadLevel::High);
assert_eq!(IoLoadLevel::from_wait_time(Duration::from_millis(150), low, high), IoLoadLevel::Critical);
}
#[test]
fn test_bandwidth_tier() {
assert_eq!(BandwidthTier::from_bps(50 * 1024 * 1024), BandwidthTier::Low);
assert_eq!(BandwidthTier::from_bps(200 * 1024 * 1024), BandwidthTier::Medium);
assert_eq!(BandwidthTier::from_bps(600 * 1024 * 1024), BandwidthTier::High);
}
#[test]
fn test_io_strategy_default() {
let strategy = IoStrategy::default();
assert!(strategy.buffer_size > 0);
assert!(strategy.enable_readahead);
}
#[test]
fn test_io_scheduler() {
let scheduler = IoScheduler::with_defaults();
let strategy = scheduler.calculate_strategy(1024 * 1024, Duration::from_millis(5), true);
assert!(strategy.buffer_size > 0);
assert!(strategy.enable_readahead);
assert_eq!(strategy.load_level, IoLoadLevel::Low);
}
#[test]
fn test_io_scheduler_with_concurrency() {
let scheduler = IoScheduler::with_defaults();
// Simulate concurrent requests
scheduler.increment_requests();
scheduler.increment_requests();
scheduler.increment_requests();
let strategy = scheduler.calculate_strategy(1024 * 1024, Duration::from_millis(5), true);
assert_eq!(strategy.concurrent_requests, 3);
}
#[test]
fn test_load_metrics() {
let mut metrics = IoLoadMetrics::new();
metrics.add_sample(Duration::from_millis(10));
metrics.add_sample(Duration::from_millis(20));
metrics.add_sample(Duration::from_millis(30));
assert_eq!(metrics.sample_count, 3);
assert_eq!(metrics.avg_wait_time, Duration::from_millis(20));
assert_eq!(metrics.max_wait_time, Duration::from_millis(30));
}
#[test]
fn test_get_concurrency_aware_buffer_size() {
// Test with default concurrency (1)
let size = get_concurrency_aware_buffer_size(1024 * 1024, 128 * KI_B);
assert!(size >= 64 * KI_B);
assert!(size <= MI_B);
// Test with small file
let size = get_concurrency_aware_buffer_size(50 * KI_B as i64, 128 * KI_B);
assert!(size >= 32 * KI_B);
}
#[test]
fn test_get_advanced_buffer_size() {
// Sequential read with low concurrency
let size = get_advanced_buffer_size(10 * MI_B as i64, 128 * KI_B, true, 2);
assert!(size >= 128 * KI_B);
// Random read with high concurrency
let size = get_advanced_buffer_size(10 * MI_B as i64, 128 * KI_B, false, 10);
assert!(size >= 32 * KI_B);
// Very small file
let size = get_advanced_buffer_size(100 * KI_B as i64, 128 * KI_B, true, 1);
assert!(size <= 64 * KI_B);
}
#[test]
fn test_get_buffer_size_for_media() {
let base = 128 * KI_B;
// NVMe should get larger buffers
let nvme_size = get_buffer_size_for_media(base, StorageMedia::Nvme);
assert!(nvme_size > base);
// SSD should get slightly larger buffers
let ssd_size = get_buffer_size_for_media(base, StorageMedia::Ssd);
assert!(ssd_size > base);
// HDD should get smaller buffers
let hdd_size = get_buffer_size_for_media(base, StorageMedia::Hdd);
assert!(hdd_size < base);
}
#[test]
fn test_calculate_optimal_buffer_size() {
// Low load, sequential, NVMe
let size = calculate_optimal_buffer_size(10 * MI_B as i64, 128 * KI_B, true, 2, StorageMedia::Nvme, IoLoadLevel::Low);
assert!(size >= 32 * KI_B);
assert!(size <= MI_B);
// Critical load, random, HDD
let size =
calculate_optimal_buffer_size(10 * MI_B as i64, 128 * KI_B, false, 10, StorageMedia::Hdd, IoLoadLevel::Critical);
assert!(size >= 32 * KI_B);
assert!(size <= MI_B);
}
#[test]
fn test_io_scheduling_context() {
let ctx = IoSchedulingContext::new(10 * MI_B as i64, 256 * KI_B)
.with_sequential(true)
.with_media(StorageMedia::Nvme)
.with_bandwidth(500 * MI_B as u64)
.with_concurrency(4);
assert_eq!(ctx.file_size, 10 * MI_B as i64);
assert_eq!(ctx.base_buffer_size, 256 * KI_B);
assert!(ctx.is_sequential_hint);
assert_eq!(ctx.storage_media, StorageMedia::Nvme);
assert_eq!(ctx.observed_bandwidth_bps, Some(500 * MI_B as u64));
assert_eq!(ctx.concurrent_requests, 4);
}
}
+320
View File
@@ -0,0 +1,320 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Shared memory pool for zero-copy data sharing.
//!
//! This module provides Arc-based shared memory management for
//! efficient cross-task data passing without serialization.
use std::convert::AsRef;
use std::ops::Deref;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Instant;
/// Shared memory pool configuration.
#[derive(Debug, Clone)]
pub struct SharedMemoryConfig {
/// Whether shared memory is enabled
pub enabled: bool,
/// Maximum pool size in bytes
pub max_pool_size: usize,
/// Maximum object size in bytes
pub max_object_size: usize,
}
impl Default for SharedMemoryConfig {
fn default() -> Self {
Self {
enabled: true,
max_pool_size: 100 * 1024 * 1024, // 100MB
max_object_size: 10 * 1024 * 1024, // 10MB
}
}
}
/// Shared memory pool statistics.
#[derive(Debug, Default)]
pub struct SharedMemoryStats {
/// Total number of objects created
pub total_objects: AtomicU64,
/// Total number of shared references
pub total_shared_refs: AtomicU64,
/// Current memory usage in bytes
pub current_memory: AtomicU64,
/// Peak memory usage in bytes
pub peak_memory: AtomicU64,
}
/// Arc data metadata.
#[derive(Clone, Debug)]
pub struct ArcMetadata {
/// Size of the data (if measurable)
pub size: Option<usize>,
/// Creation timestamp
pub created_at: Instant,
}
/// Arc-based data wrapper for zero-copy sharing.
///
/// This wrapper uses Arc to enable shared ownership of data
/// across multiple tasks without copying.
pub struct ArcData<T> {
/// The wrapped data
inner: Arc<T>,
/// Metadata about the data
metadata: ArcMetadata,
}
impl<T> Clone for ArcData<T> {
fn clone(&self) -> Self {
Self {
inner: Arc::clone(&self.inner),
metadata: self.metadata.clone(),
}
}
}
impl<T> ArcData<T> {
/// Create a new ArcData wrapper.
pub fn new(data: T) -> Self {
ArcData {
inner: Arc::new(data),
metadata: ArcMetadata {
size: None,
created_at: Instant::now(),
},
}
}
/// Create a new ArcData wrapper with known size.
pub fn with_size(data: T, size: usize) -> Self {
ArcData {
inner: Arc::new(data),
metadata: ArcMetadata {
size: Some(size),
created_at: Instant::now(),
},
}
}
/// Get the reference count.
pub fn ref_count(&self) -> usize {
Arc::strong_count(&self.inner)
}
/// Convert into the underlying Arc.
pub fn into_arc(self) -> Arc<T> {
self.inner
}
/// Get the metadata.
pub fn metadata(&self) -> &ArcMetadata {
&self.metadata
}
/// Get the size if known.
pub fn size(&self) -> Option<usize> {
self.metadata.size
}
}
impl<T> AsRef<T> for ArcData<T> {
fn as_ref(&self) -> &T {
&self.inner
}
}
impl<T> Deref for ArcData<T> {
type Target = T;
fn deref(&self) -> &Self::Target {
&self.inner
}
}
impl<T> std::fmt::Debug for ArcData<T> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ArcData")
.field("ref_count", &self.ref_count())
.field("metadata", &self.metadata)
.finish()
}
}
/// Shared memory pool for managing Arc-based shared data.
pub struct SharedMemoryPool {
config: SharedMemoryConfig,
stats: SharedMemoryStats,
}
impl SharedMemoryPool {
/// Create a new shared memory pool with the given configuration.
pub fn new(config: SharedMemoryConfig) -> Self {
Self {
config,
stats: SharedMemoryStats::default(),
}
}
/// Create a new shared memory pool with default configuration.
pub fn with_defaults() -> Self {
Self::new(SharedMemoryConfig::default())
}
/// Create shared data.
///
/// This method wraps the data in an ArcData for zero-copy sharing.
pub fn create<T>(&self, data: T) -> ArcData<T> {
self.stats.total_objects.fetch_add(1, Ordering::Relaxed);
ArcData::new(data)
}
/// Create shared data with known size.
///
/// This method tracks memory usage for statistics.
pub fn create_with_size<T>(&self, data: T, size: usize) -> ArcData<T> {
self.stats.total_objects.fetch_add(1, Ordering::Relaxed);
// Update memory statistics
self.stats.current_memory.fetch_add(size as u64, Ordering::Relaxed);
// Update peak memory
let current = self.stats.current_memory.load(Ordering::Relaxed);
let mut peak = self.stats.peak_memory.load(Ordering::Relaxed);
if current > peak {
peak = current;
self.stats.peak_memory.store(peak, Ordering::Relaxed);
}
ArcData::with_size(data, size)
}
/// Share data by increasing reference count.
///
/// This method creates a new ArcData that shares the underlying data
/// without copying.
pub fn share<T>(&self, data: &ArcData<T>) -> ArcData<T> {
self.stats.total_shared_refs.fetch_add(1, Ordering::Relaxed);
data.clone()
}
/// Get the statistics for this pool.
pub fn stats(&self) -> &SharedMemoryStats {
&self.stats
}
/// Get the configuration for this pool.
pub fn config(&self) -> &SharedMemoryConfig {
&self.config
}
/// Check if the pool is enabled.
pub fn is_enabled(&self) -> bool {
self.config.enabled
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_arc_data_new() {
let data = vec![1u8, 2, 3, 4, 5];
let arc_data = ArcData::new(data.clone());
assert_eq!(arc_data.as_ref(), &data);
assert_eq!(arc_data.ref_count(), 1);
}
#[test]
fn test_arc_data_clone() {
let data = vec![1u8, 2, 3, 4, 5];
let arc_data = ArcData::new(data);
assert_eq!(arc_data.ref_count(), 1);
let arc_data2 = arc_data.clone();
assert_eq!(arc_data.ref_count(), 2);
assert_eq!(arc_data2.ref_count(), 2);
let arc_data3 = arc_data.clone();
assert_eq!(arc_data.ref_count(), 3);
assert_eq!(arc_data2.ref_count(), 3);
assert_eq!(arc_data3.ref_count(), 3);
}
#[test]
fn test_arc_data_deref() {
let data = vec![1u8, 2, 3, 4, 5];
let arc_data = ArcData::new(data);
// Test Deref trait
assert_eq!(arc_data.len(), 5);
assert_eq!(arc_data[0], 1);
}
#[test]
fn test_shared_memory_pool_create() {
let pool = SharedMemoryPool::with_defaults();
let data = vec![1u8, 2, 3, 4, 5];
let arc_data = pool.create(data.clone());
assert_eq!(arc_data.as_ref(), &data);
assert_eq!(pool.stats().total_objects.load(Ordering::Relaxed), 1);
}
#[test]
fn test_shared_memory_pool_share() {
let pool = SharedMemoryPool::with_defaults();
let data = vec![1u8, 2, 3, 4, 5];
let arc_data = pool.create(data);
assert_eq!(arc_data.ref_count(), 1);
let shared = pool.share(&arc_data);
assert_eq!(arc_data.ref_count(), 2);
assert_eq!(shared.ref_count(), 2);
assert_eq!(pool.stats().total_shared_refs.load(Ordering::Relaxed), 1);
}
#[test]
fn test_shared_memory_pool_with_size() {
let pool = SharedMemoryPool::with_defaults();
let data = vec![1u8; 1024];
let arc_data = pool.create_with_size(data, 1024);
assert_eq!(arc_data.size(), Some(1024));
assert_eq!(pool.stats().current_memory.load(Ordering::Relaxed), 1024);
}
#[test]
fn test_default_config() {
let config = SharedMemoryConfig::default();
assert!(config.enabled);
assert_eq!(config.max_pool_size, 100 * 1024 * 1024);
assert_eq!(config.max_object_size, 10 * 1024 * 1024);
}
}
+501
View File
@@ -0,0 +1,501 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Timeout wrapper for I/O operations.
//!
//! This module provides timeout management for I/O operations with
//! dynamic timeout calculation based on operation size.
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
/// Timeout configuration.
#[derive(Debug, Clone)]
pub struct TimeoutConfig {
/// Base timeout for small operations.
pub base_timeout: Duration,
/// Timeout per MB of data.
pub timeout_per_mb: Duration,
/// Maximum timeout.
pub max_timeout: Duration,
/// Minimum timeout.
pub min_timeout: Duration,
/// GetObject operation timeout.
pub get_object_timeout: Duration,
/// PutObject operation timeout.
pub put_object_timeout: Duration,
/// ListObjects operation timeout.
pub list_objects_timeout: Duration,
/// Whether dynamic timeout is enabled.
pub enable_dynamic_timeout: bool,
}
impl Default for TimeoutConfig {
fn default() -> Self {
Self {
base_timeout: Duration::from_secs(5),
timeout_per_mb: Duration::from_millis(100),
max_timeout: Duration::from_secs(300),
min_timeout: Duration::from_secs(1),
get_object_timeout: Duration::from_secs(30),
put_object_timeout: Duration::from_secs(60),
list_objects_timeout: Duration::from_secs(10),
enable_dynamic_timeout: true,
}
}
}
impl TimeoutConfig {
/// Create new timeout configuration.
pub fn new() -> Self {
Self::default()
}
/// Calculate dynamic timeout based on size.
pub fn calculate_timeout(&self, size_bytes: u64) -> Duration {
if !self.enable_dynamic_timeout {
return self.base_timeout;
}
let mb = size_bytes as f64 / (1024.0 * 1024.0);
let timeout = self.base_timeout + self.timeout_per_mb.mul_f64(mb);
timeout.clamp(self.min_timeout, self.max_timeout)
}
/// Validate the configuration.
pub fn validate(&self) -> Result<(), TimeoutError> {
if self.min_timeout > self.max_timeout {
return Err(TimeoutError::InvalidConfig("min_timeout must be <= max_timeout".to_string()));
}
if self.base_timeout < self.min_timeout || self.base_timeout > self.max_timeout {
return Err(TimeoutError::InvalidConfig(
"base_timeout must be between min_timeout and max_timeout".to_string(),
));
}
Ok(())
}
}
/// Timeout error.
#[derive(Debug, Clone, thiserror::Error)]
pub enum TimeoutError {
/// Operation timed out.
#[error("Operation timed out after {0:?}")]
TimedOut(Duration),
/// Invalid configuration.
#[error("Invalid timeout config: {0}")]
InvalidConfig(String),
}
/// Operation progress tracker.
#[derive(Debug)]
pub struct OperationProgress {
/// Total size (if known).
pub total_size: Option<u64>,
/// Bytes processed.
bytes_processed: AtomicU64,
/// Last update time.
last_update: std::sync::Mutex<Instant>,
/// Stale timeout.
stale_timeout: Duration,
/// Start time for transfer rate calculation.
start_time: Instant,
}
impl OperationProgress {
/// Create new operation progress.
pub fn new(total_size: Option<u64>, stale_timeout: Duration) -> Self {
Self {
total_size,
bytes_processed: AtomicU64::new(0),
last_update: std::sync::Mutex::new(Instant::now()),
stale_timeout,
start_time: Instant::now(),
}
}
/// Update progress.
pub fn update(&self, bytes: u64) {
self.bytes_processed.store(bytes, Ordering::Relaxed);
if let Ok(mut last) = self.last_update.lock() {
*last = Instant::now();
}
}
/// Add to progress.
pub fn add(&self, bytes: u64) {
self.bytes_processed.fetch_add(bytes, Ordering::Relaxed);
if let Ok(mut last) = self.last_update.lock() {
*last = Instant::now();
}
}
/// Get current progress.
pub fn current(&self) -> u64 {
self.bytes_processed.load(Ordering::Relaxed)
}
/// Check if progress is stale.
pub fn is_stale(&self) -> bool {
if let Ok(last) = self.last_update.lock() {
last.elapsed() > self.stale_timeout
} else {
false
}
}
/// Get progress percentage.
pub fn progress_percent(&self) -> Option<f64> {
self.total_size.map(|total| {
if total == 0 {
100.0
} else {
let processed = self.bytes_processed.load(Ordering::Relaxed);
(processed as f64 / total as f64 * 100.0).min(100.0)
}
})
}
/// Get remaining bytes.
pub fn remaining(&self) -> Option<u64> {
self.total_size.map(|total| {
let processed = self.bytes_processed.load(Ordering::Relaxed);
total.saturating_sub(processed)
})
}
/// Calculate transfer rate in bytes per second.
///
/// Returns 0 if no time has elapsed or no data transferred.
pub fn transfer_rate(&self) -> u64 {
let processed = self.bytes_processed.load(Ordering::Relaxed);
if processed == 0 {
return 0;
}
let elapsed = self.start_time.elapsed().as_secs_f64();
if elapsed > 0.0 {
(processed as f64 / elapsed) as u64
} else {
0
}
}
}
/// Request timeout wrapper.
pub struct RequestTimeoutWrapper {
/// Configuration.
config: TimeoutConfig,
/// Start time.
start_time: Instant,
/// Operation progress.
progress: Option<OperationProgress>,
}
impl RequestTimeoutWrapper {
/// Create a new timeout wrapper.
pub fn new(config: TimeoutConfig) -> Self {
Self {
config,
start_time: Instant::now(),
progress: None,
}
}
/// Create with progress tracking.
pub fn with_progress(config: TimeoutConfig, total_size: Option<u64>, stale_timeout: Duration) -> Self {
Self {
config,
start_time: Instant::now(),
progress: Some(OperationProgress::new(total_size, stale_timeout)),
}
}
/// Get the configuration.
pub fn config(&self) -> &TimeoutConfig {
&self.config
}
/// Get elapsed time.
pub fn elapsed(&self) -> Duration {
self.start_time.elapsed()
}
/// Get remaining time.
pub fn remaining(&self, timeout: Duration) -> Option<Duration> {
let elapsed = self.elapsed();
if elapsed >= timeout { None } else { Some(timeout - elapsed) }
}
/// Check if timed out.
pub fn is_timed_out(&self, size: Option<u64>) -> bool {
let timeout = self.get_timeout(size);
self.elapsed() > timeout
}
/// Get the timeout for a given size.
pub fn get_timeout(&self, size: Option<u64>) -> Duration {
if self.config.enable_dynamic_timeout {
if let Some(s) = size {
self.config.calculate_timeout(s)
} else {
self.config.base_timeout
}
} else {
self.config.base_timeout
}
}
/// Check if timed out and return error if so.
pub fn check_timeout(&self, size: Option<u64>) -> Result<(), TimeoutError> {
if self.is_timed_out(size) {
Err(TimeoutError::TimedOut(self.get_timeout(size)))
} else {
Ok(())
}
}
/// Get progress.
pub fn progress(&self) -> Option<&OperationProgress> {
self.progress.as_ref()
}
/// Update progress.
pub fn update_progress(&self, bytes: u64) {
if let Some(ref progress) = self.progress {
progress.update(bytes);
}
}
/// Check if operation is stalled (no progress for a while).
pub fn is_stalled(&self) -> bool {
self.progress.as_ref().is_some_and(|p| p.is_stale())
}
/// Get progress percentage.
pub fn progress_percent(&self) -> Option<f64> {
self.progress.as_ref().and_then(|p| p.progress_percent())
}
}
/// Timeout statistics.
#[derive(Debug, Default)]
pub struct TimeoutStats {
/// Total operations.
pub total_operations: AtomicU64,
/// Timed out operations.
pub timed_out: AtomicU64,
/// Total wait time in nanoseconds.
pub total_wait_time_ns: AtomicU64,
/// Maximum wait time in nanoseconds.
pub max_wait_time_ns: AtomicU64,
}
impl TimeoutStats {
/// Create new timeout statistics.
pub fn new() -> Self {
Self::default()
}
/// Record an operation.
pub fn record_operation(&self, wait_time: Duration) {
self.total_operations.fetch_add(1, Ordering::Relaxed);
let ns = wait_time.as_nanos() as u64;
self.total_wait_time_ns.fetch_add(ns, Ordering::Relaxed);
let mut current = self.max_wait_time_ns.load(Ordering::Relaxed);
while ns > current {
match self
.max_wait_time_ns
.compare_exchange_weak(current, ns, Ordering::Relaxed, Ordering::Relaxed)
{
Ok(_) => break,
Err(actual) => current = actual,
}
}
}
/// Record a timeout.
pub fn record_timeout(&self) {
self.timed_out.fetch_add(1, Ordering::Relaxed);
}
/// Get timeout rate.
pub fn timeout_rate(&self) -> f64 {
let total = self.total_operations.load(Ordering::Relaxed);
let timed_out = self.timed_out.load(Ordering::Relaxed);
if total == 0 { 0.0 } else { timed_out as f64 / total as f64 }
}
/// Get average wait time.
pub fn avg_wait_time(&self) -> Duration {
let total = self.total_wait_time_ns.load(Ordering::Relaxed);
let count = self.total_operations.load(Ordering::Relaxed);
total.checked_div(count).map(Duration::from_nanos).unwrap_or(Duration::ZERO)
}
/// Reset statistics.
pub fn reset(&self) {
self.total_operations.store(0, Ordering::Relaxed);
self.timed_out.store(0, Ordering::Relaxed);
self.total_wait_time_ns.store(0, Ordering::Relaxed);
self.max_wait_time_ns.store(0, Ordering::Relaxed);
}
}
/// Calculate adaptive timeout based on historical data and current conditions.
///
/// This function adjusts the timeout based on:
/// - Historical transfer rate
/// - Recent timeout count
/// - Object size
pub fn calculate_adaptive_timeout(
base_timeout: Duration,
historical_rate_bps: Option<u64>,
recent_timeout_count: u32,
object_size: u64,
) -> Duration {
// If we have recent timeouts, increase timeout
let timeout_multiplier = if recent_timeout_count > 3 {
2.0 // Double timeout if many recent timeouts
} else if recent_timeout_count > 1 {
1.5 // 50% increase if some timeouts
} else {
1.0 // No adjustment
};
// Adaptive timeout bounds: 5 seconds minimum, 10 minutes maximum.
const MIN_SECS: f64 = 5.0;
const MAX_SECS: f64 = 600.0;
// If we have historical rate data, use it for estimation
let estimated_secs = match historical_rate_bps {
Some(rate) if rate > 0 => (object_size as f64 / rate as f64) * 1.2, // 20% buffer
_ => base_timeout.as_secs_f64(),
};
// Clamp BEFORE constructing the Duration: `from_secs_f64` panics when the
// estimate overflows Duration (huge object_size with a tiny historical rate).
Duration::from_secs_f64((estimated_secs * timeout_multiplier).clamp(MIN_SECS, MAX_SECS))
}
/// Estimate bytes per second transfer rate.
///
/// This is used for adaptive timeout calculation.
pub fn estimate_bytes_per_second(object_size: u64, expected_duration: Duration) -> u64 {
let secs = expected_duration.as_secs_f64();
if secs > 0.0 {
(object_size as f64 / secs) as u64
} else {
// Return a reasonable default (1 MB/s)
1024 * 1024
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_timeout_config() {
let config = TimeoutConfig::default();
assert!(config.validate().is_ok());
// Small file
let timeout = config.calculate_timeout(1024);
assert!(timeout >= config.min_timeout);
// Large file
let timeout = config.calculate_timeout(100 * 1024 * 1024);
assert!(timeout <= config.max_timeout);
}
#[test]
fn test_timeout_config_validation() {
let config = TimeoutConfig {
min_timeout: Duration::from_secs(10),
max_timeout: Duration::from_secs(5),
..Default::default()
};
assert!(config.validate().is_err());
}
#[test]
fn test_adaptive_timeout_extreme_estimate_does_not_panic() {
// A huge object with a tiny historical rate used to overflow
// Duration::from_secs_f64 and panic; it must clamp to the upper bound.
let timeout = calculate_adaptive_timeout(Duration::from_secs(30), Some(1), 0, u64::MAX);
assert_eq!(timeout, Duration::from_secs(600));
// Tiny estimates clamp to the lower bound.
let timeout = calculate_adaptive_timeout(Duration::from_secs(30), Some(u64::MAX), 0, 1);
assert_eq!(timeout, Duration::from_secs(5));
}
#[test]
fn test_operation_progress() {
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
assert_eq!(progress.current(), 0);
assert_eq!(progress.progress_percent(), Some(0.0));
progress.update(500);
assert_eq!(progress.current(), 500);
assert_eq!(progress.progress_percent(), Some(50.0));
progress.add(300);
assert_eq!(progress.current(), 800);
assert_eq!(progress.remaining(), Some(200));
}
#[test]
fn test_request_timeout_wrapper() {
let config = TimeoutConfig {
base_timeout: Duration::from_millis(100),
enable_dynamic_timeout: false,
..Default::default()
};
let wrapper = RequestTimeoutWrapper::new(config);
assert!(!wrapper.is_timed_out(None));
std::thread::sleep(Duration::from_millis(150));
assert!(wrapper.is_timed_out(None));
assert!(wrapper.check_timeout(None).is_err());
}
#[test]
fn test_timeout_stats() {
let stats = TimeoutStats::new();
stats.record_operation(Duration::from_millis(10));
stats.record_operation(Duration::from_millis(20));
stats.record_timeout();
assert_eq!(stats.total_operations.load(Ordering::Relaxed), 2);
assert_eq!(stats.timed_out.load(Ordering::Relaxed), 1);
assert!((stats.timeout_rate() - 0.5).abs() < 0.01);
}
#[test]
fn test_progress_tracking() {
let config = TimeoutConfig::default();
let wrapper = RequestTimeoutWrapper::with_progress(config, Some(1000), Duration::from_secs(1));
wrapper.update_progress(500);
assert_eq!(wrapper.progress_percent(), Some(50.0));
assert!(!wrapper.is_stalled());
}
}
+443
View File
@@ -0,0 +1,443 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! BytesMut-backed object writer for optimized write operations.
//!
//! It uses `BytesMut` for efficient buffering; writes into that buffer may
//! still copy input bytes. The historical `ZeroCopyObjectWriter` name remains
//! available as a deprecated compatibility alias.
use bytes::{BufMut, Bytes, BytesMut};
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::AsyncWrite;
/// BytesMut-backed object writer for optimized write operations.
///
/// This writer minimizes memory allocations by:
/// - Using BytesMut for efficient buffer growth
/// - Accepting `Bytes` inputs for efficient buffer handling
/// - Optional integration with BytesPool for buffer reuse
///
/// # Example
///
/// ```ignore
/// use rustfs_io_core::BytesMutWriter;
/// use bytes::Bytes;
///
/// #[tokio::main]
/// async fn main() -> Result<(), Box<dyn std::error::Error>> {
/// let mut writer = BytesMutWriter::new();
///
/// // Write into the internal BytesMut buffer
/// let data = Bytes::from("hello world");
/// writer.write_buffered(data).await?;
///
/// // Get the result as Bytes (zero-copy conversion)
/// let result = writer.into_bytes();
///
/// Ok(())
/// }
/// ```
pub struct BytesMutWriter {
/// Internal buffer using BytesMut for efficient growth
buffer: BytesMut,
/// Total bytes written
bytes_written: usize,
/// Whether the writer has been finalized
finalized: bool,
}
/// Historical name for the BytesMut-backed object writer.
#[deprecated(since = "1.0.0-beta.8", note = "use BytesMutWriter; writes append into a BytesMut buffer")]
pub type ZeroCopyObjectWriter = BytesMutWriter;
impl BytesMutWriter {
/// Create a new bytes-backed object writer with default capacity (8KB).
///
/// # Example
///
/// ```ignore
/// let writer = BytesMutWriter::new();
/// ```
pub fn new() -> Self {
Self::with_capacity(8 * 1024)
}
/// Create a new bytes-backed object writer with specified capacity.
///
/// # Arguments
///
/// * `capacity` - Initial buffer capacity in bytes
///
/// # Example
///
/// ```ignore
/// let writer = BytesMutWriter::with_capacity(64 * 1024);
/// ```
pub fn with_capacity(capacity: usize) -> Self {
Self {
buffer: BytesMut::with_capacity(capacity),
bytes_written: 0,
finalized: false,
}
}
/// Write data into the internal buffer.
///
/// This method accepts `Bytes` for API compatibility, then appends the
/// bytes into the internal `BytesMut` buffer.
///
/// # Arguments
///
/// * `data` - Data to append to the internal buffer
///
/// # Returns
///
/// * `Ok(usize)` - Number of bytes written
/// * `Err(ZeroCopyWriteError)` - Write error
///
/// # Example
///
/// ```ignore
/// let data = Bytes::from("hello world");
/// let written = writer.write_buffered(data).await?;
/// ```
pub async fn write_buffered(&mut self, data: Bytes) -> Result<usize, ZeroCopyWriteError> {
if self.finalized {
return Err(ZeroCopyWriteError::Finalized("Cannot write to finalized writer".to_string()));
}
let len = data.len();
self.buffer.put(data);
self.bytes_written += len;
Ok(len)
}
/// Historical name for `write_buffered`.
#[deprecated(
since = "1.0.0-beta.8",
note = "use write_buffered; this method appends bytes into an internal buffer"
)]
pub async fn write_zero_copy(&mut self, data: Bytes) -> Result<usize, ZeroCopyWriteError> {
self.write_buffered(data).await
}
/// Write a slice of data.
///
/// # Arguments
///
/// * `data` - Data slice to write
///
/// # Returns
///
/// * `Ok(usize)` - Number of bytes written
/// * `Err(ZeroCopyWriteError)` - Write error
pub async fn write_slice(&mut self, data: &[u8]) -> Result<usize, ZeroCopyWriteError> {
if self.finalized {
return Err(ZeroCopyWriteError::Finalized("Cannot write to finalized writer".to_string()));
}
let len = data.len();
self.buffer.put_slice(data);
self.bytes_written += len;
Ok(len)
}
/// Finalize the writer and consume it, returning the written data as Bytes.
///
/// This converts the internal BytesMut to Bytes, which is a zero-copy
/// operation that freezes the buffer.
///
/// # Returns
///
/// The written data as Bytes
///
/// # Example
///
/// ```ignore
/// let result = writer.into_bytes();
/// ```
pub fn into_bytes(mut self) -> Bytes {
self.finalized = true;
self.buffer.freeze()
}
/// Get the current buffer as a slice (without consuming).
///
/// # Returns
///
/// Slice of the current buffer content
pub fn as_slice(&self) -> &[u8] {
&self.buffer[..]
}
/// Get the total number of bytes written.
///
/// # Returns
///
/// Number of bytes written
pub fn bytes_written(&self) -> usize {
self.bytes_written
}
/// Get the current buffer capacity.
///
/// # Returns
///
/// Current buffer capacity in bytes
pub fn capacity(&self) -> usize {
self.buffer.capacity()
}
/// Get the current buffer length.
///
/// # Returns
///
/// Current buffer length in bytes
pub fn len(&self) -> usize {
self.buffer.len()
}
/// Check if the buffer is empty.
///
/// # Returns
///
/// `true` if buffer is empty, `false` otherwise
pub fn is_empty(&self) -> bool {
self.buffer.is_empty()
}
/// Clear the buffer, resetting it to empty.
///
/// This does not change the capacity, just resets the length to 0.
pub fn clear(&mut self) {
self.buffer.clear();
self.bytes_written = 0;
self.finalized = false;
}
/// Reserve additional capacity in the buffer.
///
/// # Arguments
///
/// * `additional` - Additional capacity to reserve
pub fn reserve(&mut self, additional: usize) {
self.buffer.reserve(additional);
}
}
impl Default for BytesMutWriter {
fn default() -> Self {
Self::new()
}
}
impl std::fmt::Debug for BytesMutWriter {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("BytesMutWriter")
.field("buffer_len", &self.buffer.len())
.field("buffer_capacity", &self.buffer.capacity())
.field("bytes_written", &self.bytes_written)
.field("finalized", &self.finalized)
.finish()
}
}
/// AsyncWrite implementation for BytesMutWriter.
///
/// This allows the writer to be used with tokio's async I/O utilities.
impl AsyncWrite for BytesMutWriter {
fn poll_write(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<Result<usize, tokio::io::Error>> {
if self.finalized {
return Poll::Ready(Err(tokio::io::Error::new(
tokio::io::ErrorKind::WriteZero,
"Cannot write to finalized writer",
)));
}
let len = buf.len();
self.buffer.put_slice(buf);
self.bytes_written += len;
Poll::Ready(Ok(len))
}
fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Result<(), tokio::io::Error>> {
// Nothing to flush for in-memory buffer
Poll::Ready(Ok(()))
}
fn poll_shutdown(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Result<(), tokio::io::Error>> {
self.finalized = true;
Poll::Ready(Ok(()))
}
}
/// Zero-copy write error types.
#[derive(Debug, thiserror::Error)]
pub enum ZeroCopyWriteError {
/// I/O error occurred
#[error("I/O error: {0}")]
Io(#[from] tokio::io::Error),
/// Writer has been finalized and cannot accept more writes
#[error("Writer finalized: {0}")]
Finalized(String),
/// Invalid input provided
#[error("Invalid input: {0}")]
InvalidInput(String),
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_new_writer() {
let writer = BytesMutWriter::new();
assert!(writer.is_empty());
assert_eq!(writer.bytes_written(), 0);
assert!(writer.capacity() >= 8 * 1024);
}
#[tokio::test]
async fn test_write_buffered() {
let mut writer = BytesMutWriter::new();
let data = Bytes::from("hello world");
let written = writer.write_buffered(data).await.unwrap();
assert_eq!(written, 11);
assert_eq!(writer.bytes_written(), 11);
assert_eq!(writer.as_slice(), b"hello world");
}
#[tokio::test]
async fn test_preferred_writer_alias() {
let mut writer = BytesMutWriter::new();
let written = writer
.write_buffered(Bytes::from("hello world"))
.await
.expect("write bytes through alias");
assert_eq!(written, 11);
assert_eq!(writer.as_slice(), b"hello world");
}
#[tokio::test]
async fn test_write_slice() {
let mut writer = BytesMutWriter::new();
let data = b"hello world";
let written = writer.write_slice(data).await.unwrap();
assert_eq!(written, 11);
assert_eq!(writer.bytes_written(), 11);
assert_eq!(writer.as_slice(), b"hello world");
}
#[tokio::test]
async fn test_into_bytes() {
let mut writer = BytesMutWriter::new();
let data = Bytes::from("hello world");
writer.write_buffered(data).await.unwrap();
let result = writer.into_bytes();
assert_eq!(result.as_ref(), b"hello world");
}
#[tokio::test]
async fn test_write_after_finalize() {
let mut writer = BytesMutWriter::new();
let data = Bytes::from("hello");
writer.write_buffered(data).await.unwrap();
let _result = writer.into_bytes();
// Create new writer and try to write after finalize
let mut writer2 = BytesMutWriter::new();
writer2.write_buffered(Bytes::from("test")).await.unwrap();
let _ = writer2.into_bytes();
// Writing to a consumed writer should work via new writer
let mut writer3 = BytesMutWriter::new();
let result = writer3.write_buffered(Bytes::from("final")).await;
assert!(result.is_ok());
}
#[tokio::test]
async fn test_clear() {
let mut writer = BytesMutWriter::new();
writer.write_slice(b"hello").await.unwrap();
writer.clear();
assert!(writer.is_empty());
assert_eq!(writer.bytes_written(), 0);
// Capacity should remain
assert!(writer.capacity() > 0);
}
#[tokio::test]
async fn test_reserve() {
let mut writer = BytesMutWriter::with_capacity(10);
let initial_capacity = writer.capacity();
writer.reserve(1000);
// Reserve ensures at least the additional capacity can be added
// but may allocate more than requested
assert!(writer.capacity() >= initial_capacity);
}
#[tokio::test]
async fn test_multiple_writes() {
let mut writer = BytesMutWriter::new();
writer.write_buffered(Bytes::from("hello ")).await.unwrap();
writer.write_slice(b"world").await.unwrap();
assert_eq!(writer.as_slice(), b"hello world");
assert_eq!(writer.bytes_written(), 11);
}
#[tokio::test]
async fn test_async_write() {
use tokio::io::AsyncWriteExt;
let mut writer = BytesMutWriter::new();
let data = b"hello world";
let written = writer.write(data).await.unwrap();
assert_eq!(written, 11);
assert_eq!(writer.as_slice(), b"hello world");
}
#[tokio::test]
async fn test_debug() {
let writer = BytesMutWriter::new();
let debug_str = format!("{:?}", writer);
assert!(debug_str.contains("BytesMutWriter"));
assert!(debug_str.contains("buffer_len"));
}
#[tokio::test]
#[allow(deprecated)]
async fn test_legacy_writer_alias() {
let mut writer = ZeroCopyObjectWriter::new();
let written = writer.write_zero_copy(Bytes::from("hello")).await.unwrap();
assert_eq!(written, 5);
assert_eq!(writer.as_slice(), b"hello");
}
}
+24 -4
View File
@@ -1729,10 +1729,30 @@ mod tests {
assert!(config.validate().is_ok(), "deprecated mount_path must not be required");
}
// The "VaultKv2 must not claim Transit wrapping" documentation-claim
// invariant is enforced by scripts/check_fips_wording.sh, which scans every
// file in crates/kms rather than a fixed include_str! list
// (rustfs/backlog#1884).
#[test]
fn test_vault_kv2_sources_do_not_claim_transit_wrapping() {
let sources = [
("config.rs", include_str!("config.rs")),
("api_types.rs", include_str!("api_types.rs")),
("backends/vault.rs", include_str!("backends/vault.rs")),
("lib.rs", include_str!("lib.rs")),
];
// Assemble the needles at runtime so this guard does not match its own source.
let needles = [
format!("wrapping via {}", "Transit"),
format!("KV v2 + {}", "Transit"),
format!("KV2+{}", "Transit"),
format!("you would use Vault's {} engine", "transit"),
];
for (name, source) in sources {
for needle in &needles {
assert!(
!source.contains(needle.as_str()),
"{name} still describes the Vault KV2 backend with `{needle}`"
);
}
}
}
#[test]
fn test_legacy_persisted_vault_transit_config_uses_metadata_defaults() {
+69 -282
View File
@@ -15,10 +15,8 @@
use std::collections::HashMap;
use std::hash::{Hash, Hasher};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
use tokio::sync::RwLock;
use tokio::time::Instant;
use crate::{
FastLockGuard, GlobalLockManager, LockClient, LockId, LockInfo, LockManager, LockMetadata, LockPriority, LockRequest,
@@ -28,51 +26,43 @@ use crate::{
/// Default shard count for guard storage (must be power of 2)
const DEFAULT_GUARD_SHARD_COUNT: usize = 64;
type GuardShard = Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>;
type GuardStorage = Arc<Vec<GuardShard>>;
/// Local lock client using FastLock with sharded guard storage for better concurrency
#[derive(Debug)]
pub struct LocalClient {
/// Sharded guard storage to reduce lock contention
guard_storage: GuardStorage,
guard_storage: Vec<Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>>,
/// Mask for fast shard index calculation (shard_count - 1)
shard_mask: usize,
/// Optional lock manager (if None, uses global singleton)
manager: Option<Arc<GlobalLockManager>>,
reaper_started: AtomicBool,
reaper_interval: Duration,
}
#[derive(Debug)]
struct LocalGuardEntry {
guard: FastLockGuard,
expires_at: SystemTime,
deadline: Instant,
ttl: Duration,
/// Owner recorded at acquire time; used only for reclaim diagnostics (#899).
owner: String,
}
impl LocalGuardEntry {
fn new(guard: FastLockGuard, ttl: Duration) -> Self {
fn new(guard: FastLockGuard, ttl: Duration, owner: String) -> Self {
let now = SystemTime::now();
let monotonic_now = Instant::now();
Self {
guard,
expires_at: now.checked_add(ttl).unwrap_or(now),
deadline: monotonic_now.checked_add(ttl).unwrap_or(monotonic_now),
expires_at: now + ttl,
ttl,
owner,
}
}
fn is_expired(&self) -> bool {
self.deadline <= Instant::now()
self.expires_at <= SystemTime::now()
}
fn refresh(&mut self) {
let now = SystemTime::now();
let monotonic_now = Instant::now();
self.expires_at = now.checked_add(self.ttl).unwrap_or(now);
self.deadline = monotonic_now.checked_add(self.ttl).unwrap_or(monotonic_now);
self.expires_at = SystemTime::now() + self.ttl;
}
}
@@ -87,38 +77,26 @@ impl LocalClient {
pub fn with_shard_count(shard_count: usize) -> Self {
assert!(shard_count.is_power_of_two(), "Shard count must be power of 2");
let guard_storage: Vec<GuardShard> = (0..shard_count).map(|_| Arc::new(RwLock::new(HashMap::new()))).collect();
let guard_storage: Vec<Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>> =
(0..shard_count).map(|_| Arc::new(RwLock::new(HashMap::new()))).collect();
Self::with_storage(Arc::new(guard_storage), None, crate::fast_lock::CLEANUP_INTERVAL)
}
fn with_storage(guard_storage: GuardStorage, manager: Option<Arc<GlobalLockManager>>, reaper_interval: Duration) -> Self {
let shard_count = guard_storage.len();
debug_assert!(shard_count.is_power_of_two());
Self {
guard_storage,
shard_mask: shard_count - 1,
manager,
reaper_started: AtomicBool::new(false),
reaper_interval,
manager: None,
}
}
/// Create new local client with a specific lock manager
/// This allows simulating multi-node environments where each node has its own lock backend
pub fn with_manager(manager: Arc<GlobalLockManager>) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), crate::fast_lock::CLEANUP_INTERVAL)
}
#[cfg(test)]
pub(crate) fn with_manager_and_reaper_interval(manager: Arc<GlobalLockManager>, reaper_interval: Duration) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), reaper_interval)
Self {
guard_storage: (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.collect(),
shard_mask: DEFAULT_GUARD_SHARD_COUNT - 1,
manager: Some(manager),
}
}
/// Get the lock manager (injected manager if available, otherwise global singleton)
@@ -140,63 +118,52 @@ impl LocalClient {
}
async fn reclaim_expired_guards_for_resource(&self, resource: &crate::ObjectKey) -> usize {
let expired_entries = Self::extract_expired_guards(&self.guard_storage, Some(resource)).await;
Self::release_reclaimed_guards(expired_entries, Some(resource))
}
let mut reclaimed = 0usize;
async fn extract_expired_guards(storage: &GuardStorage, resource: Option<&crate::ObjectKey>) -> Vec<LocalGuardEntry> {
let mut expired_entries = Vec::new();
for shard in storage.iter() {
let mut guards = shard.write().await;
expired_entries.extend(
guards
.extract_if(|lock_id, entry| {
resource.is_none_or(|resource| &lock_id.resource == resource) && entry.is_expired()
})
.map(|(_, entry)| entry),
);
}
expired_entries
}
for shard in &self.guard_storage {
let expired_entries = {
let mut guards = shard.write().await;
let mut retained = HashMap::with_capacity(guards.len());
let mut expired_entries = Vec::new();
fn release_reclaimed_guards(
entries: impl IntoIterator<Item = LocalGuardEntry>,
resource: Option<&crate::ObjectKey>,
) -> usize {
let mut reclaimed = 0;
for mut entry in entries {
let _ = entry.guard.release();
rustfs_io_metrics::record_lock_reclaimed();
reclaimed += 1;
}
if reclaimed > 0 {
if let Some(resource) = resource {
tracing::debug!(event = "lock_guard_reclaimed", resource = %resource, count = reclaimed, "expired lock guards reclaimed");
} else {
tracing::debug!(event = "lock_guard_reaper_sweep", count = reclaimed, "expired lock guards reclaimed");
for (lock_id, entry) in std::mem::take(&mut *guards) {
if &lock_id.resource == resource && entry.is_expired() {
expired_entries.push(entry);
} else {
retained.insert(lock_id, entry);
}
}
*guards = retained;
expired_entries
};
for mut entry in expired_entries {
// An expired entry whose owner never refreshed it (a dead coordinator, #698) is
// reclaimed so a live contender can re-form quorum. With guard heartbeats in place
// (#899) a live owner keeps its entry from expiring, so reaching here means the
// lease genuinely lapsed. Surface it for observability; the reclaim decision itself
// is unchanged.
let since_last_refresh = entry
.expires_at
.checked_sub(entry.ttl)
.and_then(|last_refresh| SystemTime::now().duration_since(last_refresh).ok())
.unwrap_or(entry.ttl);
tracing::warn!(
owner = %entry.owner,
resource = %resource,
ttl_ms = entry.ttl.as_millis() as u64,
since_last_refresh_ms = since_last_refresh.as_millis() as u64,
"reclaiming expired lock guard whose lease was not refreshed"
);
rustfs_io_metrics::record_lock_reclaimed();
let _ = entry.guard.release();
reclaimed = reclaimed.saturating_add(1);
}
}
reclaimed
}
fn ensure_reaper(&self) {
if self.reaper_started.swap(true, Ordering::AcqRel) {
return;
}
let storage = Arc::downgrade(&self.guard_storage);
let interval = self.reaper_interval;
tokio::spawn(async move {
let mut ticker = tokio::time::interval(interval);
loop {
ticker.tick().await;
let Some(storage) = storage.upgrade() else {
break;
};
let expired_entries = Self::extract_expired_guards(&storage, None).await;
Self::release_reclaimed_guards(expired_entries, None);
}
});
}
}
impl Default for LocalClient {
@@ -208,36 +175,28 @@ impl Default for LocalClient {
#[async_trait::async_trait]
impl LockClient for LocalClient {
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
self.ensure_reaper();
let lock_manager = self.get_lock_manager();
let reclaimed_before_acquire = self.reclaim_expired_guards_for_resource(&request.resource).await;
let acquire_deadline = Instant::now()
.checked_add(request.acquire_timeout)
.unwrap_or_else(Instant::now);
let build_lock_request = |acquire_timeout| match request.lock_type {
let build_lock_request = || match request.lock_type {
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
.with_acquire_timeout(acquire_timeout),
.with_acquire_timeout(request.acquire_timeout),
LockType::Shared => crate::ObjectLockRequest::new_read(request.resource.clone(), request.owner.clone())
.with_acquire_timeout(acquire_timeout),
.with_acquire_timeout(request.acquire_timeout),
};
let mut retried_after_reclaim = reclaimed_before_acquire > 0;
loop {
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
}
match lock_manager.acquire_lock(build_lock_request(remaining)).await {
match lock_manager.acquire_lock(build_lock_request()).await {
Ok(guard) => {
let lock_id = request.lock_id.clone();
let acquired_at = SystemTime::now();
let expires_at = acquired_at.checked_add(request.ttl).unwrap_or(acquired_at);
let expires_at = acquired_at + request.ttl;
{
let shard = self.get_shard(&lock_id);
let mut guards = shard.write().await;
guards.insert(lock_id.clone(), LocalGuardEntry::new(guard, request.ttl));
guards.insert(lock_id.clone(), LocalGuardEntry::new(guard, request.ttl, request.owner.clone()));
}
let lock_info = LockInfo {
@@ -297,24 +256,12 @@ impl LockClient for LocalClient {
async fn refresh(&self, lock_id: &LockId) -> Result<bool> {
let shard = self.get_shard(lock_id);
let expired_entry = {
let mut guards = shard.write().await;
let Some(entry) = guards.get_mut(lock_id) else {
return Ok(false);
};
if entry.is_expired() {
guards.remove(lock_id)
} else {
entry.refresh();
None
}
};
if let Some(entry) = expired_entry {
Self::release_reclaimed_guards([entry], Some(&lock_id.resource));
Ok(false)
} else {
let mut guards = shard.write().await;
if let Some(entry) = guards.get_mut(lock_id) {
entry.refresh();
Ok(true)
} else {
Ok(false)
}
}
@@ -370,163 +317,3 @@ impl LockClient for LocalClient {
true
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{GlobalLockManager, LockClient, LockRequest, LockType};
fn request(resource: crate::ObjectKey, owner: &str, ttl: Duration) -> LockRequest {
LockRequest::new(resource, LockType::Exclusive, owner)
.with_ttl(ttl)
.with_acquire_timeout(Duration::from_millis(80))
}
async fn wait_until_reaped(client: &LocalClient, lock_id: &LockId) {
for _ in 0..80 {
if client.check_status(lock_id).await.unwrap().is_none() {
return;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
panic!("lock guard was not reaped before test deadline");
}
#[tokio::test(flavor = "current_thread")]
async fn expired_guard_is_reaped_without_resource_reacquire() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(5));
let request = request(crate::ObjectKey::new("bucket", "unique-chunk"), "owner-a", Duration::from_millis(10));
let lock_id = request.lock_id.clone();
assert!(client.acquire_lock(&request).await.unwrap().success);
assert!(client.check_status(&lock_id).await.unwrap().is_some());
tokio::time::sleep(Duration::from_millis(15)).await;
wait_until_reaped(&client, &lock_id).await;
let direct = manager
.acquire_lock(crate::ObjectLockRequest::new_write(request.resource.clone(), "owner-b"))
.await;
assert!(direct.is_ok());
}
#[tokio::test(flavor = "current_thread")]
async fn sibling_client_cannot_reclaim_but_owner_reaper_releases_shared_lock() {
let manager = Arc::new(GlobalLockManager::new());
let owner = LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(5));
let contender = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_millis(5));
let request_a = request(crate::ObjectKey::new("bucket", "shared-resource"), "owner-a", Duration::from_millis(10));
assert!(owner.acquire_lock(&request_a).await.unwrap().success);
let request_b = request(request_a.resource.clone(), "owner-b", Duration::from_millis(20))
.with_acquire_timeout(Duration::from_millis(5));
assert!(!contender.acquire_lock(&request_b).await.unwrap().success);
tokio::time::sleep(Duration::from_millis(25)).await;
assert!(owner.check_status(&request_a.lock_id).await.unwrap().is_none());
assert!(contender.acquire_lock(&request_b).await.unwrap().success);
}
#[tokio::test(flavor = "current_thread")]
async fn refresh_wins_before_deadline_and_reaper_wins_after_deadline() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_millis(5));
let request = request(crate::ObjectKey::new("bucket", "refresh-race"), "owner-a", Duration::from_millis(25));
let lock_id = request.lock_id.clone();
assert!(client.acquire_lock(&request).await.unwrap().success);
tokio::time::sleep(Duration::from_millis(10)).await;
assert!(client.refresh(&lock_id).await.unwrap());
tokio::time::sleep(Duration::from_millis(15)).await;
assert!(client.check_status(&lock_id).await.unwrap().is_some());
wait_until_reaped(&client, &lock_id).await;
}
#[tokio::test(start_paused = true)]
async fn refresh_after_expiry_releases_guard_without_reviving_it() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_secs(60));
client.reaper_started.store(true, Ordering::Release);
let lock_request = request(
crate::ObjectKey::new("bucket", "refresh-after-expiry"),
"owner-a",
Duration::from_secs(10),
);
let lock_id = lock_request.lock_id.clone();
assert!(
client
.acquire_lock(&lock_request)
.await
.expect("initial owner should acquire the lock")
.success
);
tokio::time::advance(Duration::from_secs(11)).await;
assert!(
!client
.refresh(&lock_id)
.await
.expect("expired refresh should return a result"),
"an expired guard must not be refreshed"
);
assert!(
client
.check_status(&lock_id)
.await
.expect("expired guard status should be readable")
.is_none(),
"expired guard should be removed after refresh"
);
let contender = request(
crate::ObjectKey::new("bucket", "refresh-after-expiry"),
"owner-b",
Duration::from_secs(10),
);
assert!(
client
.acquire_lock(&contender)
.await
.expect("contender should receive an acquisition result")
.success,
"released guard must be acquirable by a new owner"
);
}
#[tokio::test(flavor = "current_thread")]
async fn zero_ttl_is_reaped_and_oversized_ttl_does_not_panic() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_millis(5));
let zero = request(crate::ObjectKey::new("bucket", "zero-ttl"), "owner-zero", Duration::ZERO);
let zero_id = zero.lock_id.clone();
assert!(client.acquire_lock(&zero).await.unwrap().success);
wait_until_reaped(&client, &zero_id).await;
let huge = request(crate::ObjectKey::new("bucket", "huge-ttl"), "owner-huge", Duration::MAX);
let huge_id = huge.lock_id.clone();
assert!(client.acquire_lock(&huge).await.unwrap().success);
wait_until_reaped(&client, &huge_id).await;
}
#[tokio::test(flavor = "current_thread")]
async fn acquire_retry_preserves_total_deadline() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_secs(60));
let first = request(crate::ObjectKey::new("bucket", "deadline-budget"), "owner-a", Duration::from_millis(10));
assert!(client.acquire_lock(&first).await.unwrap().success);
let second =
request(first.resource.clone(), "owner-b", Duration::from_millis(30)).with_acquire_timeout(Duration::from_millis(60));
let started = Instant::now();
let response = client.acquire_lock(&second).await.unwrap();
assert!(!response.success, "the first attempt consumed the caller's acquire budget");
assert!(
started.elapsed() < Duration::from_millis(100),
"reclaim retry must not double the acquire budget"
);
let recovered = client.acquire_lock(&second).await.unwrap();
assert!(recovered.success, "the reclaimed guard must be available to the next request");
}
}
-111
View File
@@ -840,117 +840,6 @@ async fn test_namespace_lock_distributed_reclaims_expired_same_resource_after_fa
);
}
#[tokio::test]
async fn four_node_failed_release_converges_without_replica_repair() {
let managers = (0..4).map(|_| Arc::new(GlobalLockManager::new())).collect::<Vec<_>>();
let flaky_clients = managers
.iter()
.map(|manager| {
Arc::new(FlakyReleaseClient {
inner: LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(5)),
failed_releases_remaining: AtomicUsize::new(usize::MAX),
release_attempts: AtomicUsize::new(0),
})
})
.collect::<Vec<_>>();
let clients = flaky_clients
.iter()
.map(|client| client.clone() as Arc<dyn LockClient>)
.collect::<Vec<_>>();
let lock = NamespaceLock::Distributed(DistributedLock::new("four-node-expired-lease".to_string(), clients, 3));
let resource = create_test_object_key("bucket", "object-four-node-expired");
let request = LockRequest::new(resource.clone(), LockType::Exclusive, "owner-a")
.with_acquire_timeout(Duration::from_millis(300))
.with_ttl(Duration::from_millis(40));
let mut guard = lock
.acquire_guard(&request)
.await
.expect("initial acquire should not error")
.expect("initial acquire should reach quorum");
assert!(guard.release(), "release should be acknowledged while RPC cleanup is pending");
for _ in 0..40 {
if flaky_clients.iter().all(|client| client.release_attempts() >= 3) {
break;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
let deadline = tokio::time::Instant::now() + Duration::from_secs(2);
loop {
let all_reaped =
futures::future::join_all(flaky_clients.iter().map(|client| client.inner.check_status(&request.lock_id)))
.await
.into_iter()
.all(|status| status.expect("status should not error").is_none());
if all_reaped {
break;
}
assert!(tokio::time::Instant::now() < deadline, "all four local lease entries must converge");
tokio::time::sleep(Duration::from_millis(10)).await;
}
for suffix in ["chunk-0", "chunk-1", ".rustfs.sys/multipart/upload-0"] {
for client in &flaky_clients {
let orphan = LockRequest::new(create_test_object_key("bucket", suffix), LockType::Exclusive, "orphan")
.with_ttl(Duration::from_millis(25));
assert!(client.inner.acquire_lock(&orphan).await.expect("orphan acquire").success);
}
}
tokio::time::sleep(Duration::from_millis(80)).await;
let recovered = lock
.acquire_guard(
&LockRequest::new(resource, LockType::Exclusive, "owner-b")
.with_acquire_timeout(Duration::from_millis(300))
.with_ttl(Duration::from_millis(40)),
)
.await
.expect("recovery acquire should not error")
.expect("four-node quorum should recover after local reapers run");
drop(recovered);
}
#[tokio::test]
async fn four_node_stale_quorum_contention_respects_acquire_deadline() {
let managers = (0..4).map(|_| Arc::new(GlobalLockManager::new())).collect::<Vec<_>>();
let node_clients = managers
.iter()
.map(|manager| Arc::new(LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(5))))
.collect::<Vec<_>>();
let resource = create_test_object_key("bucket", "stale-quorum");
let stale = LockRequest::new(resource.clone(), LockType::Exclusive, "stale-owner").with_ttl(Duration::from_millis(180));
for client in &node_clients {
assert!(client.acquire_lock(&stale).await.expect("stale acquire").success);
}
let clients = node_clients
.iter()
.map(|client| client.clone() as Arc<dyn LockClient>)
.collect::<Vec<_>>();
let lock = NamespaceLock::Distributed(DistributedLock::new("stale-quorum-deadline".to_string(), clients, 3));
let contender = LockRequest::new(resource.clone(), LockType::Exclusive, "new-owner")
.with_acquire_timeout(Duration::from_millis(150))
.with_ttl(Duration::from_millis(100));
let started = tokio::time::Instant::now();
let response = lock.acquire_guard(&contender).await.expect("contention should not error");
assert!(response.is_none(), "unexpired leases must not be force-reclaimed");
assert!(started.elapsed() < Duration::from_millis(350), "acquire must respect its deadline");
tokio::time::sleep(Duration::from_millis(80)).await;
let recovered = lock
.acquire_guard(
&LockRequest::new(resource, LockType::Exclusive, "new-owner")
.with_acquire_timeout(Duration::from_millis(300))
.with_ttl(Duration::from_millis(100)),
)
.await
.expect("post-expiry acquire should not error")
.expect("quorum should recover after local reapers clear stale leases");
drop(recovered);
}
#[tokio::test]
async fn test_namespace_lock_distributed_retries_transient_acquire_timeout() {
let managers = (0..3).map(|_| Arc::new(GlobalLockManager::new())).collect::<Vec<_>>();
+10 -66
View File
@@ -177,38 +177,16 @@ impl StartCommand {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)]
pub enum Command {
Start {
request: StartCommand,
},
Query {
heal_path: String,
client_token: String,
/// Incremental result cursor (HS-06): only items with a sequence
/// greater than this are returned. Absent = legacy full snapshot.
/// Optional + defaulted so older peers stay wire-compatible.
#[serde(default, skip_serializing_if = "Option::is_none")]
since_seq: Option<u64>,
},
Cancel {
heal_path: String,
client_token: String,
},
Start { request: StartCommand },
Query { heal_path: String, client_token: String },
Cancel { heal_path: String, client_token: String },
}
#[derive(Debug)]
pub enum ExecutableCommand {
Start {
request: HealChannelRequest,
},
Query {
heal_path: String,
client_token: String,
since_seq: Option<u64>,
},
Cancel {
heal_path: String,
client_token: String,
},
Start { request: HealChannelRequest },
Query { heal_path: String, client_token: String },
Cancel { heal_path: String, client_token: String },
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -249,22 +227,8 @@ impl Envelope {
)
}
pub fn query(
request_id: String,
metadata: RequestMetadata,
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<Self, String> {
Self::new(
request_id,
metadata,
Command::Query {
heal_path,
client_token,
since_seq,
},
)
pub fn query(request_id: String, metadata: RequestMetadata, heal_path: String, client_token: String) -> Result<Self, String> {
Self::new(request_id, metadata, Command::Query { heal_path, client_token })
}
pub fn cancel(
@@ -322,15 +286,7 @@ impl Envelope {
Command::Start { request } => ExecutableCommand::Start {
request: request.into_channel_request(self.request_id.clone())?,
},
Command::Query {
heal_path,
client_token,
since_seq,
} => ExecutableCommand::Query {
heal_path,
client_token,
since_seq,
},
Command::Query { heal_path, client_token } => ExecutableCommand::Query { heal_path, client_token },
Command::Cancel { heal_path, client_token } => ExecutableCommand::Cancel { heal_path, client_token },
};
Ok((self.request_id, self.coordinator_epoch, command))
@@ -349,11 +305,6 @@ pub enum Admission {
Full,
DroppedQueueFull,
DroppedPolicy,
/// HS-06: admin start rejected because the same target is already being
/// healed (RUSTFS_HEAL_OVERLAP_POLICY=minio_error only).
DroppedAlreadyRunning,
/// HS-06: admin start rejected because its path overlaps an active heal.
DroppedOverlappingPaths,
}
impl From<HealAdmissionResult> for Admission {
@@ -364,8 +315,6 @@ impl From<HealAdmissionResult> for Admission {
HealAdmissionResult::Full => Self::Full,
HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull) => Self::DroppedQueueFull,
HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped) => Self::DroppedPolicy,
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning) => Self::DroppedAlreadyRunning,
HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths) => Self::DroppedOverlappingPaths,
}
}
}
@@ -382,8 +331,6 @@ impl Admission {
Self::Full => HealAdmissionResult::Full,
Self::DroppedQueueFull => HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull),
Self::DroppedPolicy => HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped),
Self::DroppedAlreadyRunning => HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning),
Self::DroppedOverlappingPaths => HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths),
}
}
}
@@ -645,7 +592,6 @@ mod tests {
metadata(2, 7),
"bucket/prefix".to_string(),
"token".to_string(),
None,
)
.unwrap();
let cancel = Envelope::cancel(
@@ -721,7 +667,6 @@ mod tests {
RequestMetadata::new([0x11; 16], 1_700_000_000_000, 1_700_000_030_000, 9),
"bucket/prefix".to_string(),
"client-token".to_string(),
None,
)
.unwrap();
let cancel = Envelope::cancel(
@@ -804,7 +749,7 @@ mod tests {
assert!(Envelope::start(test_request(request_id.clone()), metadata(0, 7)).is_err());
assert!(Envelope::start(test_request(request_id.clone()), metadata(1, 0)).is_err());
assert!(Envelope::start(test_request(request_id.clone()), RequestMetadata::new([1; 16], 1_000, 31_001, 7),).is_err());
assert!(Envelope::query(request_id.clone(), metadata(1, 7), String::new(), String::new(), None).is_err());
assert!(Envelope::query(request_id.clone(), metadata(1, 7), String::new(), String::new()).is_err());
assert!(Envelope::cancel(request_id.clone(), metadata(1, 7), String::new(), String::new()).is_ok());
let mut noncanonical_request = test_request(request_id.to_uppercase());
@@ -837,7 +782,6 @@ mod tests {
metadata(1, 7),
"x".repeat(ENVELOPE_MAX_SIZE),
"token".to_string(),
None,
)
.unwrap();
let error = super::encode_envelope(&oversized).unwrap_err();
+1 -1
View File
@@ -2557,7 +2557,7 @@ mod tests {
fn production_source(source: &'static str, file_name: &str) -> &'static str {
source
.split("\n#[cfg(test)]\nmod tests")
.split("\n#[cfg(test)]")
.next()
.unwrap_or_else(|| panic!("{file_name} should contain production source before tests"))
}
+202 -4
View File
@@ -14,8 +14,13 @@
use std::any::Any;
use uuid::Uuid;
use crate::storage_api::DeletedObject;
use crate::{MrfOpKind, MrfReplicateEntry, ReplicationState, ReplicationType, ReplicationWorkerOperation};
use crate::{
MrfOpKind, MrfReplicateEntry, NULL_VERSION_ID, REPLICATE_EXISTING_DELETE, ReplicateObjectInfo, ReplicationState,
ReplicationStatusType, ReplicationType, ReplicationWorkerOperation,
};
#[derive(Debug, Clone, Default)]
pub struct DeletedObjectReplicationInfo {
@@ -117,17 +122,114 @@ pub fn is_retryable_delete_replication_head_error(is_not_found: bool, code: Opti
!(is_not_found || matches!(code, Some("MethodNotAllowed" | "405")))
}
/// Build the delete-replication work item for an existing delete marker or
/// version purge discovered during a resync scan.
pub fn resync_existing_delete_replication_info(roi: &ReplicateObjectInfo, target_arn: &str) -> DeletedObjectReplicationInfo {
let (version_id, dm_version_id) = if roi.version_purge_status.is_empty() {
(None, roi.version_id)
} else {
(roi.version_id, None)
};
DeletedObjectReplicationInfo {
delete_object: DeletedObject {
object_name: roi.name.clone(),
delete_marker_version_id: dm_version_id,
version_id,
replication_state: roi.replication_state.clone(),
delete_marker: roi.delete_marker,
delete_marker_mtime: roi.mod_time,
..Default::default()
},
bucket: roi.bucket.clone(),
event_type: REPLICATE_EXISTING_DELETE.to_string(),
op_type: ReplicationType::ExistingObject,
target_arn: target_arn.to_string(),
..Default::default()
}
}
/// Whether a delete replication fully succeeded — the MRF replay acknowledges
/// (drops) an entry exactly when this returns true.
///
/// The delayed purge is deliberately NOT an input: holding the outcome hostage
/// to it (`&& !requires_delayed_purge`) forced `false` for every delete-marker
/// entry and retained them all in the durable MRF journal forever. Purge
/// failures persist their own purge-intent entry instead
/// (`watch_and_purge_source_delete_marker`), and replays of those entries
/// report purge success through `purge_stale_delete_marker_targets`.
pub fn replicate_delete_outcome(
expected_targets: usize,
replicated_targets: usize,
state_persisted: bool,
source_state_verified: bool,
replication_status: &ReplicationStatusType,
) -> bool {
expected_targets > 0
&& replicated_targets == expected_targets
&& state_persisted
&& source_state_verified
&& *replication_status == ReplicationStatusType::Completed
}
pub fn target_delete_version_id(version_id: Uuid, version_purge: bool) -> Option<String> {
if version_id.is_nil() {
version_purge.then(|| NULL_VERSION_ID.to_string())
} else {
Some(version_id.to_string())
}
}
/// Which version a delete-marker purge should address on one target.
///
/// `None` means do not purge at all: the recorded mapping disagreed across the
/// dual internal prefixes, and guessing an id could destroy a live version on
/// the target. `Some(id)` is the exact version the target reported when it
/// accepted the marker; falling back to a source-derived id is only correct
/// when the target mirrors source version ids, which a generic S3 target does
/// not.
pub fn delete_marker_purge_version_id(
state: Option<&ReplicationState>,
arn: &str,
delete_marker_version_id: Uuid,
) -> Option<Option<String>> {
if state.is_some_and(|state| state.target_delete_marker_version_ids_corrupt) {
return None;
}
let recorded = state.and_then(|state| state.target_delete_marker_version_ids.get(arn).cloned());
Some(match recorded {
Some(version_id) => Some(version_id),
None => target_delete_version_id(delete_marker_version_id, true),
})
}
/// Shape an exhausted purge intent as a marker-creation delete entry. Replay
/// reconstructs it with `delete_marker: true`, finds the source marker gone,
/// and funnels into the stale-marker branch of `replicate_delete_with_outcome`
/// — which re-runs the purge without touching source state and reports purge
/// success as the replay outcome.
pub fn delete_marker_purge_mrf_entry(dobj: &DeletedObjectReplicationInfo, failed_arns: Vec<String>) -> MrfReplicateEntry {
let mut entry = dobj.to_mrf_entry();
entry.delete_marker = true;
entry.version_id = None;
entry.retry_count = 0;
entry.target_arns = failed_arns;
entry
}
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use super::{
DeletedObjectReplicationInfo, is_retryable_delete_replication_head_error, is_version_delete_replication,
should_retry_delete_marker_purge,
DeletedObjectReplicationInfo, delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
is_retryable_delete_replication_head_error, is_version_delete_replication, replicate_delete_outcome,
should_retry_delete_marker_purge, target_delete_version_id,
};
use crate::storage_api::DeletedObject;
use crate::{
MrfOpKind, ReplicationState, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, VersionPurgeStatusType,
MrfOpKind, NULL_VERSION_ID, ReplicationState, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation,
VersionPurgeStatusType,
};
use uuid::Uuid;
@@ -328,4 +430,100 @@ mod tests {
assert!(!is_retryable_delete_replication_head_error(true, Some("NoSuchKey")));
assert!(is_retryable_delete_replication_head_error(false, Some("AccessDenied")));
}
/// P1-21 regression guard for the outcome formula. A fully successful
/// delete-marker replication must acknowledge its MRF entry: the formula
/// once carried `&& !requires_delayed_purge`, which pinned every
/// delete-marker entry to Missed and retained the whole backlog forever.
/// (Deterministically staging a marker-creation entry in the durable
/// journal from e2e would require saturating the worker queues, so the
/// formula is pinned here instead; the purge-intent replay half is pinned
/// by the delayed-purge e2e pair.)
#[test]
fn test_replicate_delete_outcome_is_not_held_hostage_by_the_delayed_purge() {
assert!(
replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Completed),
"a completed delete-marker replication must be acknowledgeable even though a delayed purge watch is pending"
);
assert!(!replicate_delete_outcome(0, 0, true, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(2, 1, true, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, false, true, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, true, false, &ReplicationStatusType::Completed));
assert!(!replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Failed));
}
#[test]
fn test_delete_marker_purge_mrf_entry_replays_through_the_stale_marker_branch() {
let delete_marker_version_id = Uuid::new_v4();
let dobj = DeletedObjectReplicationInfo {
delete_object: DeletedObject {
object_name: "doc.txt".to_string(),
// A version-purge flavored source event: the entry must still
// be reshaped as a marker-creation delete so replay funnels
// into the stale-marker branch instead of re-running the full
// delete replication (whose source-state stamping would fail
// against the already-purged version).
delete_marker: false,
version_id: Some(Uuid::new_v4()),
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
bucket: "bucket-a".to_string(),
..Default::default()
};
let entry = delete_marker_purge_mrf_entry(&dobj, vec!["arn:a".to_string()]);
assert!(entry.delete_marker, "purge intents must replay as marker-creation deletes");
assert_eq!(entry.version_id, None, "the purged data version must not leak into the replay");
assert_eq!(entry.delete_marker_version_id, Some(delete_marker_version_id));
assert_eq!(
entry.target_arns,
vec!["arn:a".to_string()],
"only the targets whose purge failed may be retried"
);
assert_eq!(entry.retry_count, 0);
assert_eq!(entry.bucket, "bucket-a");
assert_eq!(entry.object, "doc.txt");
}
#[test]
fn target_delete_version_id_preserves_explicit_null_purges() {
let version_id = Uuid::new_v4();
assert_eq!(target_delete_version_id(version_id, true), Some(version_id.to_string()));
assert_eq!(target_delete_version_id(Uuid::nil(), true).as_deref(), Some(NULL_VERSION_ID));
assert_eq!(target_delete_version_id(Uuid::nil(), false), None);
}
#[test]
fn delete_marker_purge_prefers_the_recorded_target_version() {
let source = Uuid::new_v4();
let arn = "arn:rustfs:replication::target:bucket";
// No recorded mapping: fall back to deriving from the source uuid.
assert_eq!(delete_marker_purge_version_id(None, arn, source), Some(Some(source.to_string())));
// Recorded mapping wins — a generic S3 target assigns its own id, so the
// derived one would purge the wrong version or nothing at all.
let mut state = ReplicationState::default();
state
.target_delete_marker_version_ids
.insert(arn.to_string(), "target-assigned-id".to_string());
assert_eq!(
delete_marker_purge_version_id(Some(&state), arn, source),
Some(Some("target-assigned-id".to_string()))
);
// A mapping recorded for a different ARN must not be reused.
assert_eq!(
delete_marker_purge_version_id(Some(&state), "arn:rustfs:replication::other:bucket", source),
Some(Some(source.to_string()))
);
// Inconsistent persisted metadata: refuse to purge rather than guess.
let mut corrupt = state.clone();
corrupt.target_delete_marker_version_ids_corrupt = true;
assert_eq!(delete_marker_purge_version_id(Some(&corrupt), arn, source), None);
}
}
+5
View File
@@ -27,6 +27,11 @@ use uuid::Uuid;
pub const REPLICATION_RESET: &str = "replication-reset";
pub const REPLICATION_STATUS: &str = "replication-status";
/// The S3 wire spelling of the unversioned ("null") version id. Owned here as
/// part of the replication wire contracts; `rustfs-filemeta` keeps its own
/// copy of the same literal (the crates are intentionally independent).
pub const NULL_VERSION_ID: &str = "null";
// ReplicateQueued - replication being queued trail
pub const REPLICATE_QUEUED: &str = "replicate:queue";
+8 -6
View File
@@ -37,11 +37,12 @@ pub use config::{
validate_replication_config_target_arns,
};
pub use delete::{
DeletedObjectReplicationInfo, is_retryable_delete_replication_head_error, is_version_delete_replication,
should_retry_delete_marker_purge,
DeletedObjectReplicationInfo, delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
is_retryable_delete_replication_head_error, is_version_delete_replication, replicate_delete_outcome,
resync_existing_delete_replication_info, should_retry_delete_marker_purge, target_delete_version_id,
};
pub use filemeta::{
REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL, REPLICATE_HEAL_DELETE, REPLICATE_INCOMING,
NULL_VERSION_ID, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL, REPLICATE_HEAL_DELETE, REPLICATE_INCOMING,
REPLICATE_INCOMING_DELETE, REPLICATE_MRF, REPLICATE_QUEUED, REPLICATION_RESET, REPLICATION_STATUS, ReplicateDecision,
ReplicateObjectInfo, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState,
ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision,
@@ -58,8 +59,9 @@ pub use multipart::{
replication_multipart_complete_actual_size, replication_multipart_part_plan,
};
pub use object::{
ReplicationSourceObject, ReplicationTargetObject, content_matches_by_etag, replication_action_for_target,
replication_etags_match, target_is_newer_than_source_null_version,
ReplicationSourceObject, ReplicationTargetObject, SsecPassthroughCapability, SsecPassthroughGate, content_matches_by_etag,
is_replication_target_offline_error, replication_action_for_target, replication_etags_match,
ssec_passthrough_evidence_present, ssec_passthrough_gate, target_is_newer_than_source_null_version, version_identity_drifted,
};
pub use operation::{
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteSource, ReplicationDeleteStateSource,
@@ -76,7 +78,7 @@ pub use queue::{
pub use resync::{
BucketReplicationResyncStatus, Error, RESYNC_FILE_MAX_BYTES, Result, ResyncOpts, ResyncStatusType,
TargetReplicationResyncStatus, decode_resync_file, encode_resync_file, is_version_id_mismatch, resync_state_accepts_update,
sanitize_resync_error_detail, should_auto_resume_resync, should_count_head_proxy_failure,
resync_status_duration, sanitize_resync_error_detail, should_auto_resume_resync, should_count_head_proxy_failure,
};
pub use rule::ReplicationRuleExt;
pub use runtime::{
+211 -2
View File
@@ -157,11 +157,130 @@ fn comparable_metadata(metadata: Option<&HashMap<String, String>>) -> HashMap<St
comparable
}
/// Runtime half of the P1-19 version-identity contract (the explicit probe
/// lives in replication-check's VersionFidelity phase): every replication PUT
/// response reveals whether the target adopted the source version id. A
/// target minting its own ids silently breaks version-addressed deletes and
/// heal, so surface it — once per target — instead of letting the divergence
/// accumulate unseen.
/// Pure drift judgment: the contract only applies when the source addressed a
/// real (non-nil) version uuid, and drift means the target answered with
/// anything else — including nothing at all.
pub fn version_identity_drifted(source_version_id: &str, assigned_version_id: Option<&str>) -> bool {
if source_version_id.is_empty() {
return false;
}
// A nil source uuid travels as the literal "null" (unversioned-source
// semantics); no identity contract applies to it.
if uuid::Uuid::parse_str(source_version_id)
.map(|uuid| uuid.is_nil())
.unwrap_or(true)
{
return false;
}
assigned_version_id != Some(source_version_id)
}
const REPLICATION_TARGET_OFFLINE_ERROR_MARKERS: &[&str] = &[
"dispatch failure",
"timeouterror",
"timed out",
"connection refused",
"connection reset",
"connection closed",
"connection aborted",
"broken pipe",
"dns error",
"failed to lookup address",
"name or service not known",
"deadline has elapsed",
"tcp connect error",
];
/// True when a target operation error reads as a network/transport failure —
/// the only class of error that should mark a replication target offline.
pub fn is_replication_target_offline_error(err: &(impl std::fmt::Display + ?Sized)) -> bool {
let message = err.to_string().to_ascii_lowercase();
REPLICATION_TARGET_OFFLINE_ERROR_MARKERS
.iter()
.any(|marker| message.contains(marker))
}
/// Whether a replication target preserves the SSE-C passthrough transport
/// headers (`X-Rustfs-Replication-*`) end to end.
///
/// A target that silently drops those headers (MinIO, generic S3) stores the
/// forwarded ciphertext without its decryption material — an unreadable
/// replica that used to report COMPLETED. The replication worker audits the
/// first passthrough PUT per target (HEAD-back for SSE-C evidence) and caches
/// the verdict; a fresh `Unsupported` fails SSE-C replication closed before
/// any PUT is sent. The verdict cache (per-ARN map, lifecycle, and TTL) is
/// owned by the runtime's bucket target system; this crate owns only the
/// verdict vocabulary and the gate policy below.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum SsecPassthroughCapability {
#[default]
Unknown,
Supported,
Unsupported,
}
/// Fail-closed decision for an SSE-C passthrough replication attempt, derived
/// from the target's cached [`SsecPassthroughCapability`]. Pure so the policy
/// can migrate with the worker (M2) without dragging the cache along; the
/// caller computes `expired` from the cache record's age (see the runtime's
/// `SSEC_PASSTHROUGH_CAPABILITY_TTL`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SsecPassthroughGate {
/// Not an SSE-C object, or the target has a fresh proof that it preserves
/// the passthrough transport headers: replicate without a HEAD-back audit.
Proceed,
/// No usable verdict — first SSE-C attempt since the target was (re)built,
/// or the recorded verdict (in either direction) aged out: PUT, then HEAD
/// the replica back and require SSE-C evidence before reporting COMPLETED.
ProceedWithAudit,
/// The target was recently proven to drop the passthrough headers: do not
/// send the PUT, report FAILED (the object stays on the normal MRF retry
/// channel and re-audits once the verdict expires).
FailClosed,
}
pub fn ssec_passthrough_gate(ssec: bool, capability: SsecPassthroughCapability, expired: bool) -> SsecPassthroughGate {
if !ssec {
return SsecPassthroughGate::Proceed;
}
// An expired verdict — Supported or Unsupported — must be re-earned: a
// stale Unsupported would otherwise stick forever after a target upgrade,
// and a stale Supported would fail open after a backend swap behind the
// same endpoint.
if expired {
return SsecPassthroughGate::ProceedWithAudit;
}
match capability {
SsecPassthroughCapability::Supported => SsecPassthroughGate::Proceed,
SsecPassthroughCapability::Unknown => SsecPassthroughGate::ProceedWithAudit,
SsecPassthroughCapability::Unsupported => SsecPassthroughGate::FailClosed,
}
}
/// True when a replication-check HEAD of the replica proves the SSE-C
/// material survived passthrough: a RustFS target restores the transport
/// headers into the stored SSE-C keys and its HEAD echoes
/// `x-amz-server-side-encryption-customer-algorithm` (the replication-check
/// exemption skips key validation but not the metadata echo). A target that
/// dropped the headers stored a plain object and echoes nothing. The caller
/// extracts the echoed customer-algorithm value from its HEAD response type.
pub fn ssec_passthrough_evidence_present(sse_customer_algorithm: Option<&str>) -> bool {
sse_customer_algorithm.is_some_and(|algo| !algo.is_empty())
}
#[cfg(test)]
mod tests {
use super::{
ReplicationSourceObject, ReplicationTargetObject, content_matches_by_etag, replication_action_for_target,
replication_etags_match, target_is_newer_than_source_null_version,
ReplicationSourceObject, ReplicationTargetObject, SsecPassthroughCapability, SsecPassthroughGate,
content_matches_by_etag, is_replication_target_offline_error, replication_action_for_target, replication_etags_match,
ssec_passthrough_evidence_present, ssec_passthrough_gate, target_is_newer_than_source_null_version,
version_identity_drifted,
};
use crate::filemeta::{ReplicationAction, ReplicationType};
use crate::http::AMZ_OBJECT_LOCK_MODE;
@@ -269,6 +388,96 @@ mod tests {
);
}
/// P1-19 runtime spot-check exemption matrix: drift only applies when the
/// source addressed a real version uuid.
#[test]
fn test_version_identity_drift_judgment() {
let source = "6fa459ea-ee8a-3ca4-894e-db77e160355e";
for (sent, got, expected) in [
(source, Some(source), false),
(source, Some("0e304ce5-33e9-4b8a-9b12-9e40a53e6ded"), true),
(source, None, true),
("", None, false),
("null", Some("anything"), false),
("00000000-0000-0000-0000-000000000000", Some("anything"), false),
] {
assert_eq!(
version_identity_drifted(sent, got),
expected,
"sent {sent:?} got {got:?} must judge drift = {expected}"
);
}
}
#[test]
fn replication_target_offline_error_classifier_is_network_scoped() {
assert!(is_replication_target_offline_error("put_object dispatch failure: connector error"));
assert!(is_replication_target_offline_error("request TimeoutError after retry"));
assert!(is_replication_target_offline_error("tcp connect error: connection refused"));
assert!(!is_replication_target_offline_error("put_object failed: AccessDenied: denied"));
assert!(!is_replication_target_offline_error("put_object failed: NoSuchBucket"));
}
/// N2 fail-closed policy: SSE-C replication may only proceed silently
/// against a target with a FRESH proof that it preserves the passthrough
/// transport headers. Unknown targets must be audited; freshly-flagged
/// dropping targets must never receive the PUT; an expired verdict in
/// EITHER direction must be re-earned through the audit — a sticky
/// Unsupported would outlive a target upgrade, and a sticky Supported
/// would fail open after a backend swap behind the same endpoint.
#[test]
fn ssec_passthrough_gate_is_fail_closed_and_ttl_bounded() {
for capability in [
SsecPassthroughCapability::Unknown,
SsecPassthroughCapability::Supported,
SsecPassthroughCapability::Unsupported,
] {
for expired in [false, true] {
assert_eq!(
ssec_passthrough_gate(false, capability, expired),
SsecPassthroughGate::Proceed,
"non-SSE-C objects must never be gated on the passthrough capability"
);
}
}
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Supported, false),
SsecPassthroughGate::Proceed
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unknown, false),
SsecPassthroughGate::ProceedWithAudit
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unsupported, false),
SsecPassthroughGate::FailClosed
);
// Expiry flips both directions back to the audit.
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unsupported, true),
SsecPassthroughGate::ProceedWithAudit,
"an expired Unsupported verdict must allow a re-audit (upgraded target recovers without operator action)"
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Supported, true),
SsecPassthroughGate::ProceedWithAudit,
"an expired Supported verdict must be re-proven (backend swap behind the same endpoint must not fail open)"
);
}
#[test]
fn ssec_passthrough_evidence_requires_customer_algorithm_echo() {
assert!(ssec_passthrough_evidence_present(Some("AES256")));
assert!(
!ssec_passthrough_evidence_present(Some("")),
"an empty echo is not evidence of preserved SSE-C material"
);
assert!(
!ssec_passthrough_evidence_present(None),
"a plain HEAD response must classify the target as having dropped the material"
);
}
#[test]
fn replication_action_detects_tags_and_object_lock_metadata_differences() {
let mut source_metadata = HashMap::new();
+41
View File
@@ -309,6 +309,31 @@ pub fn is_version_id_mismatch(code: Option<&str>, raw_status: Option<u16>) -> bo
}
}
pub fn resync_status_duration(
status: ResyncStatusType,
start_time: Option<OffsetDateTime>,
now: OffsetDateTime,
) -> Option<std::time::Duration> {
if !matches!(
status,
ResyncStatusType::ResyncCompleted | ResyncStatusType::ResyncFailed | ResyncStatusType::ResyncCanceled
) {
return None;
}
let millis = (now - start_time?).whole_milliseconds();
if millis < 0 {
return None;
}
let millis = if millis > i128::from(u64::MAX) {
u64::MAX
} else {
u64::try_from(millis).ok()?
};
Some(std::time::Duration::from_millis(millis))
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct BucketReplicationResyncStatus {
pub version: u16,
@@ -712,6 +737,22 @@ mod tests {
assert!(!should_auto_resume_resync(ResyncStatusType::ResyncFailed));
}
#[test]
fn test_resync_status_duration_only_tracks_terminal_status() {
let start = match OffsetDateTime::from_unix_timestamp(1_700_000_000) {
Ok(start) => start,
Err(err) => panic!("valid test timestamp: {err}"),
};
let end = start + time::Duration::seconds(2);
assert_eq!(
resync_status_duration(ResyncStatusType::ResyncCompleted, Some(start), end),
Some(std::time::Duration::from_millis(2000))
);
assert_eq!(resync_status_duration(ResyncStatusType::ResyncStarted, Some(start), end), None);
assert_eq!(resync_status_duration(ResyncStatusType::ResyncFailed, None, end), None);
}
#[test]
fn resync_state_accepts_update_only_for_matching_run() {
let current = TargetReplicationResyncStatus {
+3 -59
View File
@@ -715,13 +715,6 @@ async fn get_http_client(url: &str) -> io::Result<Client> {
Ok(cached.client_for(disable_proxy))
}
async fn get_fresh_http_client(url: &str) -> io::Result<Client> {
let tuning = internode_http_client_tuning();
let disable_proxy = should_disable_proxy_for_url(url, tuning);
let outbound_tls = crate::http_runtime_sources::outbound_tls_state().await;
build_http_client(disable_proxy, tuning, &outbound_tls).await
}
fn internode_request_context(method: &Method, url: &str, operation: Option<&'static str>) -> InternodeHttpRequestContext {
let target = reqwest::Url::parse(url)
.ok()
@@ -969,28 +962,6 @@ impl HttpReader {
Self::with_capacity_and_stall_timeout(url, method, headers, body, 0, stall_timeout).await
}
pub async fn new_fresh_connection_with_stall_timeout(
url: String,
method: Method,
headers: HeaderMap,
body: Option<Vec<u8>>,
stall_timeout: Option<Duration>,
) -> io::Result<Self> {
let init = Self::open(&url, &method, &headers, body, stall_timeout, true).await?;
Ok(Self {
inner: StreamReader::new(init.stream),
url,
method,
headers,
track_internode_metrics: init.track_internode_metrics,
internode_operation: init.internode_operation,
stall_timer: None,
stall_timeout: init.stall_timeout,
request_started: init.request_started,
duration_recorded: false,
})
}
/// Create a new HttpReader from a URL. The request is performed immediately.
pub async fn with_capacity(
url: String,
@@ -1010,7 +981,7 @@ impl HttpReader {
_read_buf_size: usize,
stall_timeout: Option<Duration>,
) -> io::Result<Self> {
let init = Self::open(&url, &method, &headers, body, stall_timeout, false).await?;
let init = Self::open(&url, &method, &headers, body, stall_timeout).await?;
Ok(Self {
inner: StreamReader::new(init.stream),
url,
@@ -1031,16 +1002,10 @@ impl HttpReader {
headers: &HeaderMap,
body: Option<Vec<u8>>,
stall_timeout: Option<Duration>,
force_fresh_connection: bool,
) -> io::Result<HttpReaderInit> {
let track_internode_metrics = is_internode_rpc_url(url);
let internode_operation = internode_rpc_operation(url);
let client = if force_fresh_connection {
get_fresh_http_client(url).await
} else {
get_http_client(url).await
}
.inspect_err(|_| {
let client = get_http_client(url).await.inspect_err(|_| {
record_internode_error(track_internode_metrics, internode_operation);
})?;
let mut request: RequestBuilder = client.request(method.clone(), url).headers(headers.clone());
@@ -1156,28 +1121,7 @@ impl HttpChunkReader {
body: Option<Vec<u8>>,
stall_timeout: Option<Duration>,
) -> io::Result<Self> {
let init = HttpReader::open(&url, &method, &headers, body, stall_timeout, false).await?;
Ok(Self {
inner: init.stream,
current: None,
track_internode_metrics: init.track_internode_metrics,
internode_operation: init.internode_operation,
stall_timer: None,
stall_timeout: init.stall_timeout,
request_started: init.request_started,
duration_recorded: false,
consecutive_empty_chunks: 0,
})
}
pub async fn new_fresh_connection_with_stall_timeout(
url: String,
method: Method,
headers: HeaderMap,
body: Option<Vec<u8>>,
stall_timeout: Option<Duration>,
) -> io::Result<Self> {
let init = HttpReader::open(&url, &method, &headers, body, stall_timeout, true).await?;
let init = HttpReader::open(&url, &method, &headers, body, stall_timeout).await?;
Ok(Self {
inner: init.stream,
current: None,
-7
View File
@@ -1770,13 +1770,6 @@ impl FolderScanner {
HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped) => {
self.clear_pending_scanner_heal(kind, bucket, object, version_id);
}
// Admin-only overlap rejections (HS-06); the scanner never sees
// them, but if it ever does, treat them as terminal like any
// other policy drop rather than endlessly retrying.
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
| HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths) => {
self.clear_pending_scanner_heal(kind, bucket, object, version_id);
}
}
}
@@ -53,8 +53,6 @@ Suggested boilerplate when the topic cannot be avoided:
`README.md` and `CHANGELOG.md` currently contain no FIPS-related wording; `scripts/check_fips_wording.sh` is the grep guard for that public baseline. Any future occurrence of the banned strings in either file should be treated as a defect and either removed or brought under the qualifier rule above. This document intentionally contains the terminology needed to define the policy and is not part of that narrow outward-material scan.
The same script carries a second block for the adjacent over-claim: no file under `crates/kms` may describe the Vault KV2 backend as wrapping key material through Vault's Transit engine. `KmsBackend::VaultKv2` stores RustFS-wrapped key material in Vault's KV v2 engine and never calls Transit, so that wording would tell an operator their key material is cryptographically isolated inside Vault when it is not. Use the `VaultTransit` backend when that isolation is the requirement.
## The `rustfs-crypto` `fips` feature: what it actually does
`crates/crypto/Cargo.toml` declares `default = ["crypto", "fips"]`, so the feature is on in every normal build. Its entire effect is **which algorithm the write path selects**; the implementation is RustCrypto either way.
+1 -23
View File
@@ -66,9 +66,6 @@ struct HealInitParams {
client_token: String,
force_start: bool,
force_stop: bool,
/// Incremental result cursor (HS-06): only result items with a sequence
/// greater than this are returned; absent means full snapshot.
since_seq: Option<u64>,
}
fn extract_heal_init_params(body: &Bytes, uri: &Uri, params: Params<'_, '_>) -> S3Result<HealInitParams> {
@@ -101,16 +98,6 @@ fn extract_heal_init_params(body: &Bytes, uri: &Uri, params: Params<'_, '_>) ->
}
hip.force_stop = parse_heal_query_bool(value.as_ref())?;
}
"sinceSeq" => {
if !seen.insert("sinceSeq") {
return Err(s3_error!(InvalidArgument, "duplicate heal query parameter"));
}
hip.since_seq = Some(
value
.parse::<u64>()
.map_err(|_| s3_error!(InvalidArgument, "sinceSeq must be a non-negative integer"))?,
);
}
_ => return Err(s3_error!(InvalidArgument, "unknown heal query parameter")),
}
}
@@ -991,15 +978,7 @@ fn reject_heal_admission(result: rustfs_common::heal_channel::HealAdmissionResul
result.result_label(),
result.reason_label()
),
// Overlap rejections (HS-06) share this arm: the s3s footprint
// ratchet forbids new s3_error! sites, and the typed reason is
// preserved through reason_label() ("already_running" /
// "overlapping_paths") so madmin-style clients can distinguish.
HealAdmissionResult::Dropped(
HealAdmissionDropReason::PolicyDropped
| HealAdmissionDropReason::AlreadyRunning
| HealAdmissionDropReason::OverlappingPaths,
) => s3_error!(
HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped) => s3_error!(
OperationAborted,
"heal request not admitted: admission={}, reason={}",
result.result_label(),
@@ -1424,7 +1403,6 @@ impl Operation for HealHandler {
new_heal_control_metadata(&route)?,
heal_path,
client_token.clone(),
hip.since_seq,
)
.map_err(|err| s3_error!(InternalError, "encode heal control query failed: {err}"))?;
let response = submit_cluster_heal_channel_command(context, route, envelope, &request_id, client_token).await?;
+262 -242
View File
@@ -66,18 +66,20 @@ use rustfs_config::{
};
use rustfs_iam::error::is_err_no_such_service_account;
use rustfs_iam::federation::OIDC_VIRTUAL_PARENT_CLAIM;
use rustfs_iam::store::object::ObjectStore;
use rustfs_iam::store::{MappedPolicy, UserType, sr_wire_user_type, user_type_from_sr_wire};
use rustfs_iam::sys::{
NewServiceAccountOpts, SITE_REPLICATOR_SERVICE_ACCOUNT, UpdateServiceAccountOpts, get_claims_from_token_with_secret,
IamSys, NewServiceAccountOpts, SITE_REPLICATOR_SERVICE_ACCOUNT, UpdateServiceAccountOpts, get_claims_from_token_with_secret,
};
use rustfs_madmin::{
AddOrUpdateUserReq, BucketBandwidth, GroupAddRemove, GroupStatus, IDPSettings, InProgressMetric, InQueueMetric,
LDAPConfigSettings, LDAPSettings, OpenIDProviderSettings, PeerInfo, PeerSite, QStat, ReplProxyMetric, ReplicateAddStatus,
ReplicateEditStatus, ReplicateRemoveStatus, ResyncBucketStatus, SITE_REPL_API_VERSION, SR_IAM_ITEM_STS_ACC,
SR_IAM_ITEM_STS_ACC_LEGACY, SRBucketInfo, SRBucketMeta, SRBucketStatsSummary, SRGroupInfo, SRGroupStatsSummary, SRIAMItem,
SRIAMPolicy, SRILMExpiryStatsSummary, SRInfo, SRMetric, SRMetricsSummary, SRPeerError, SRPeerJoinReq, SRPendingOperation,
SRPolicyMapping, SRPolicyStatsSummary, SRRemoveReq, SRResyncOpStatus, SRRetryStats, SRSessionPolicy, SRSiteSummary,
SRStateEditReq, SRStateInfo, SRStatusInfo, SRSvcAccCreate, SRUserStatsSummary, SiteReplicationInfo, SyncStatus, WorkerStat,
SRIAMPolicy, SRIAMUser, SRILMExpiryStatsSummary, SRInfo, SRMetric, SRMetricsSummary, SRPeerError, SRPeerJoinReq,
SRPendingOperation, SRPolicyMapping, SRPolicyStatsSummary, SRRemoveReq, SRResyncOpStatus, SRRetryStats, SRSTSCredential,
SRSessionPolicy, SRSiteSummary, SRStateEditReq, SRStateInfo, SRStatusInfo, SRSvcAccChange, SRSvcAccCreate,
SRUserStatsSummary, SiteReplicationInfo, SyncStatus, WorkerStat,
};
use rustfs_policy::policy::{
Policy,
@@ -4584,7 +4586,10 @@ async fn build_metrics_summary(local_peer: &PeerInfo) -> SRMetricsSummary {
head_failed_total: non_negative_u64(node.proxy_head_failed),
put_tag_total: non_negative_u64(node.proxy_put_tag_total),
put_tag_failed_total: non_negative_u64(node.proxy_put_tag_failed),
..Default::default()
get_tag_total: non_negative_u64(node.proxy_get_tag_total),
get_tag_failed_total: non_negative_u64(node.proxy_get_tag_failed),
remove_tag_total: non_negative_u64(node.proxy_delete_tag_total),
remove_tag_failed_total: non_negative_u64(node.proxy_delete_tag_failed),
},
metrics,
uptime: node.uptime,
@@ -9358,247 +9363,16 @@ async fn apply_iam_item(item: SRIAMItem) -> S3Result<()> {
let incoming_updated_at = item.updated_at;
match item.r#type.as_str() {
"policy" => {
if let Some(policy) = item.policy {
let policy: Policy =
serde_json::from_value(policy).map_err(|e| s3_error!(InvalidRequest, "invalid policy body: {}", e))?;
iam_sys.set_policy(&item.name, policy).await.map_err(ApiError::from)?;
} else {
iam_sys.delete_policy(&item.name, true).await.map_err(ApiError::from)?;
}
Ok(())
}
"policy-mapping" => {
let Some(mapping) = item.policy_mapping else {
return Err(s3_error!(InvalidRequest, "policyMapping is required"));
};
let user_type =
user_type_from_sr_wire(mapping.user_type).ok_or_else(|| s3_error!(InvalidRequest, "invalid userType"))?;
iam_sys
.policy_db_set(&mapping.user_or_group, user_type, mapping.is_group, &mapping.policy)
.await
.map_err(ApiError::from)?;
Ok(())
}
"group-info" => {
let Some(group_info) = item.group_info else {
return Err(s3_error!(InvalidRequest, "groupInfo is required"));
};
let update = group_info.update_req;
if !group_info_requires_upsert(&update) {
iam_sys
.remove_users_from_group(&update.group, update.members)
.await
.map_err(ApiError::from)?;
return Ok(());
}
iam_sys
.add_users_to_group(&update.group, update.members)
.await
.map_err(ApiError::from)?;
iam_sys
.set_group_status(&update.group, matches!(update.status, GroupStatus::Enabled))
.await
.map_err(ApiError::from)?;
Ok(())
}
"policy" => apply_iam_policy_item(&iam_sys, &item.name, item.policy).await,
"policy-mapping" => apply_iam_policy_mapping_item(&iam_sys, item.policy_mapping).await,
"group-info" => apply_iam_group_info_item(&iam_sys, item.group_info).await,
// MinIO madmin-go sends `SRIAMItemSTSAcc = "sts-account"`. The legacy alias
// `sts-credential` (emitted by older RustFS releases) stays accepted permanently
// so mixed-version RustFS sites keep replicating STS credentials during rolling
// upgrades; it is a compatibility layer, not temporary code.
SR_IAM_ITEM_STS_ACC | SR_IAM_ITEM_STS_ACC_LEGACY => {
let Some(sts_credential) = item.sts_credential else {
return Err(s3_error!(InvalidRequest, "stsCredential is required"));
};
let Some(secret) = current_token_signing_key() else {
return Err(s3_error!(InvalidRequest, "token signing key not initialized"));
};
let claims = get_claims_from_token_with_secret(&sts_credential.session_token, &secret)
.map_err(|e| s3_error!(InvalidRequest, "invalid STS session token: {e}"))?;
let expiration = claims
.get("exp")
.and_then(claims_unix_timestamp)
.map(OffsetDateTime::from_unix_timestamp)
.transpose()
.map_err(|e| s3_error!(InvalidRequest, "invalid STS expiry: {e}"))?;
let groups = string_list_claim(&claims, "groups");
let compatibility_policy = sts_replication_compatibility_policy(&claims, &sts_credential.parent_policy_mapping);
let cred = rustfs_credentials::Credentials {
access_key: sts_credential.access_key.clone(),
secret_key: sts_credential.secret_key.clone(),
session_token: sts_credential.session_token.clone(),
expiration,
status: "on".to_string(),
parent_user: sts_credential.parent_user.clone(),
groups,
claims: Some(claims),
..Default::default()
};
iam_sys
.set_temp_user(&sts_credential.access_key, &cred, compatibility_policy)
.await
.map_err(ApiError::from)?;
Ok(())
}
"iam-user" => {
let Some(user) = item.iam_user else {
return Err(s3_error!(InvalidRequest, "iamUser is required"));
};
if let Some(local) = iam_sys.get_user(&user.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
if user.is_delete_req {
iam_sys.delete_user(&user.access_key, true).await.map_err(ApiError::from)?;
} else {
let Some(user_req) = user.user_req else {
return Err(s3_error!(InvalidRequest, "userReq is required"));
};
let is_status_only_update = user_req.secret_key.is_empty() && user_req.policy.is_none();
if is_status_only_update {
iam_sys
.set_user_status(&user.access_key, user_req.status)
.await
.map_err(ApiError::from)?;
} else {
iam_sys
.create_user(&user.access_key, &user_req)
.await
.map_err(ApiError::from)?;
}
}
Ok(())
}
"service-account" => {
let Some(change) = item.svc_acc_change else {
return Err(s3_error!(InvalidRequest, "serviceAccountChange is required"));
};
let envelope = change.oidc_service_account_envelope;
if let Some(create) = change.create {
let local_updated_at = iam_sys
.get_user(&create.access_key)
.await
.map(|local| local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH));
let replicated_policy = if create.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT {
if local_updated_at.is_some_and(|local_updated_at| is_stale_update(local_updated_at, incoming_updated_at)) {
return Ok(());
}
ReplicatedServiceAccountPolicy {
policy: Some(site_replicator_service_account_policy()?),
is_envelope: false,
}
} else {
let Some(replicated_policy) = decode_service_account_replication_policy(
&create,
envelope.as_ref(),
incoming_updated_at,
local_updated_at,
)?
else {
return Ok(());
};
replicated_policy
};
match iam_sys.get_service_account(&create.access_key).await {
Ok((existing, _)) => {
if existing.parent_user != create.parent {
return Err(s3_error!(
InvalidRequest,
"service account {} already exists with a different parent user",
create.access_key
));
}
iam_sys
.update_service_account(
&create.access_key,
UpdateServiceAccountOpts {
name: replicated_policy.metadata_for_existing_account(create.name),
description: replicated_policy.metadata_for_existing_account(create.description),
session_policy: replicated_policy.for_existing_account(),
secret_key: Some(create.secret_key),
expiration: create.expiration,
status: (!create.status.is_empty()).then_some(create.status),
parent_user: None,
allow_site_replicator_account: create.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT,
},
)
.await
.map_err(ApiError::from)?;
}
Err(err) if is_err_no_such_service_account(&err) => {
iam_sys
.new_service_account(
&create.parent,
Some(create.groups),
NewServiceAccountOpts {
session_policy: replicated_policy.policy,
access_key: create.access_key,
secret_key: create.secret_key,
name: (!create.name.is_empty()).then_some(create.name),
description: (!create.description.is_empty()).then_some(create.description),
expiration: create.expiration,
allow_site_replicator_account: true,
claims: Some(create.claims),
},
)
.await
.map_err(ApiError::from)?;
}
Err(err) => return Err(ApiError::from(err).into()),
}
return Ok(());
}
if let Some(update) = change.update {
if let Some(local) = iam_sys.get_user(&update.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
let allow_site_replicator_account = update.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT;
let session_policy = if allow_site_replicator_account {
Some(site_replicator_service_account_policy()?)
} else {
update.session_policy.as_str().and_then(|raw| serde_json::from_str(raw).ok())
};
iam_sys
.update_service_account(
&update.access_key,
UpdateServiceAccountOpts {
session_policy,
secret_key: (!update.secret_key.is_empty()).then_some(update.secret_key),
name: (!update.name.is_empty()).then_some(update.name),
description: (!update.description.is_empty()).then_some(update.description),
expiration: update.expiration,
status: (!update.status.is_empty()).then_some(update.status),
// Peers replicate credentials, never the local parent binding:
// each site resolves its own parent from its own IAM.
parent_user: None,
allow_site_replicator_account,
},
)
.await
.map_err(ApiError::from)?;
return Ok(());
}
if let Some(delete) = change.delete {
if let Some(local) = iam_sys.get_user(&delete.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
iam_sys
.delete_service_account(&delete.access_key, true)
.await
.map_err(ApiError::from)?;
return Ok(());
}
Err(s3_error!(InvalidRequest, "serviceAccountChange is empty"))
}
SR_IAM_ITEM_STS_ACC | SR_IAM_ITEM_STS_ACC_LEGACY => apply_iam_sts_account_item(&iam_sys, item.sts_credential).await,
"iam-user" => apply_iam_user_item(&iam_sys, item.iam_user, incoming_updated_at).await,
"service-account" => apply_iam_service_account_item(&iam_sys, item.svc_acc_change, incoming_updated_at).await,
_ => Err(s3_error!(
NotImplemented,
"site replication IAM item type `{}` is not supported",
@@ -9607,6 +9381,252 @@ async fn apply_iam_item(item: SRIAMItem) -> S3Result<()> {
}
}
async fn apply_iam_policy_item(iam_sys: &IamSys<ObjectStore>, name: &str, policy: Option<Value>) -> S3Result<()> {
if let Some(policy) = policy {
let policy: Policy =
serde_json::from_value(policy).map_err(|e| s3_error!(InvalidRequest, "invalid policy body: {}", e))?;
iam_sys.set_policy(name, policy).await.map_err(ApiError::from)?;
} else {
iam_sys.delete_policy(name, true).await.map_err(ApiError::from)?;
}
Ok(())
}
async fn apply_iam_policy_mapping_item(iam_sys: &IamSys<ObjectStore>, policy_mapping: Option<SRPolicyMapping>) -> S3Result<()> {
let Some(mapping) = policy_mapping else {
return Err(s3_error!(InvalidRequest, "policyMapping is required"));
};
let user_type = user_type_from_sr_wire(mapping.user_type).ok_or_else(|| s3_error!(InvalidRequest, "invalid userType"))?;
iam_sys
.policy_db_set(&mapping.user_or_group, user_type, mapping.is_group, &mapping.policy)
.await
.map_err(ApiError::from)?;
Ok(())
}
async fn apply_iam_group_info_item(iam_sys: &IamSys<ObjectStore>, group_info: Option<SRGroupInfo>) -> S3Result<()> {
let Some(group_info) = group_info else {
return Err(s3_error!(InvalidRequest, "groupInfo is required"));
};
let update = group_info.update_req;
if !group_info_requires_upsert(&update) {
iam_sys
.remove_users_from_group(&update.group, update.members)
.await
.map_err(ApiError::from)?;
return Ok(());
}
iam_sys
.add_users_to_group(&update.group, update.members)
.await
.map_err(ApiError::from)?;
iam_sys
.set_group_status(&update.group, matches!(update.status, GroupStatus::Enabled))
.await
.map_err(ApiError::from)?;
Ok(())
}
async fn apply_iam_sts_account_item(iam_sys: &IamSys<ObjectStore>, sts_credential: Option<SRSTSCredential>) -> S3Result<()> {
let Some(sts_credential) = sts_credential else {
return Err(s3_error!(InvalidRequest, "stsCredential is required"));
};
let Some(secret) = current_token_signing_key() else {
return Err(s3_error!(InvalidRequest, "token signing key not initialized"));
};
let claims = get_claims_from_token_with_secret(&sts_credential.session_token, &secret)
.map_err(|e| s3_error!(InvalidRequest, "invalid STS session token: {e}"))?;
let expiration = claims
.get("exp")
.and_then(claims_unix_timestamp)
.map(OffsetDateTime::from_unix_timestamp)
.transpose()
.map_err(|e| s3_error!(InvalidRequest, "invalid STS expiry: {e}"))?;
let groups = string_list_claim(&claims, "groups");
let compatibility_policy = sts_replication_compatibility_policy(&claims, &sts_credential.parent_policy_mapping);
let cred = rustfs_credentials::Credentials {
access_key: sts_credential.access_key.clone(),
secret_key: sts_credential.secret_key.clone(),
session_token: sts_credential.session_token.clone(),
expiration,
status: "on".to_string(),
parent_user: sts_credential.parent_user.clone(),
groups,
claims: Some(claims),
..Default::default()
};
iam_sys
.set_temp_user(&sts_credential.access_key, &cred, compatibility_policy)
.await
.map_err(ApiError::from)?;
Ok(())
}
async fn apply_iam_user_item(
iam_sys: &IamSys<ObjectStore>,
iam_user: Option<SRIAMUser>,
incoming_updated_at: Option<OffsetDateTime>,
) -> S3Result<()> {
let Some(user) = iam_user else {
return Err(s3_error!(InvalidRequest, "iamUser is required"));
};
if let Some(local) = iam_sys.get_user(&user.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
if user.is_delete_req {
iam_sys.delete_user(&user.access_key, true).await.map_err(ApiError::from)?;
} else {
let Some(user_req) = user.user_req else {
return Err(s3_error!(InvalidRequest, "userReq is required"));
};
let is_status_only_update = user_req.secret_key.is_empty() && user_req.policy.is_none();
if is_status_only_update {
iam_sys
.set_user_status(&user.access_key, user_req.status)
.await
.map_err(ApiError::from)?;
} else {
iam_sys
.create_user(&user.access_key, &user_req)
.await
.map_err(ApiError::from)?;
}
}
Ok(())
}
async fn apply_iam_service_account_item(
iam_sys: &IamSys<ObjectStore>,
svc_acc_change: Option<SRSvcAccChange>,
incoming_updated_at: Option<OffsetDateTime>,
) -> S3Result<()> {
let Some(change) = svc_acc_change else {
return Err(s3_error!(InvalidRequest, "serviceAccountChange is required"));
};
let envelope = change.oidc_service_account_envelope;
if let Some(create) = change.create {
let local_updated_at = iam_sys
.get_user(&create.access_key)
.await
.map(|local| local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH));
let replicated_policy = if create.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT {
if local_updated_at.is_some_and(|local_updated_at| is_stale_update(local_updated_at, incoming_updated_at)) {
return Ok(());
}
ReplicatedServiceAccountPolicy {
policy: Some(site_replicator_service_account_policy()?),
is_envelope: false,
}
} else {
let Some(replicated_policy) =
decode_service_account_replication_policy(&create, envelope.as_ref(), incoming_updated_at, local_updated_at)?
else {
return Ok(());
};
replicated_policy
};
match iam_sys.get_service_account(&create.access_key).await {
Ok((existing, _)) => {
if existing.parent_user != create.parent {
return Err(s3_error!(
InvalidRequest,
"service account {} already exists with a different parent user",
create.access_key
));
}
iam_sys
.update_service_account(
&create.access_key,
UpdateServiceAccountOpts {
name: replicated_policy.metadata_for_existing_account(create.name),
description: replicated_policy.metadata_for_existing_account(create.description),
session_policy: replicated_policy.for_existing_account(),
secret_key: Some(create.secret_key),
expiration: create.expiration,
status: (!create.status.is_empty()).then_some(create.status),
parent_user: None,
allow_site_replicator_account: create.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT,
},
)
.await
.map_err(ApiError::from)?;
}
Err(err) if is_err_no_such_service_account(&err) => {
iam_sys
.new_service_account(
&create.parent,
Some(create.groups),
NewServiceAccountOpts {
session_policy: replicated_policy.policy,
access_key: create.access_key,
secret_key: create.secret_key,
name: (!create.name.is_empty()).then_some(create.name),
description: (!create.description.is_empty()).then_some(create.description),
expiration: create.expiration,
allow_site_replicator_account: true,
claims: Some(create.claims),
},
)
.await
.map_err(ApiError::from)?;
}
Err(err) => return Err(ApiError::from(err).into()),
}
return Ok(());
}
if let Some(update) = change.update {
if let Some(local) = iam_sys.get_user(&update.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
let allow_site_replicator_account = update.access_key == SITE_REPLICATOR_SERVICE_ACCOUNT;
let session_policy = if allow_site_replicator_account {
Some(site_replicator_service_account_policy()?)
} else {
update.session_policy.as_str().and_then(|raw| serde_json::from_str(raw).ok())
};
iam_sys
.update_service_account(
&update.access_key,
UpdateServiceAccountOpts {
session_policy,
secret_key: (!update.secret_key.is_empty()).then_some(update.secret_key),
name: (!update.name.is_empty()).then_some(update.name),
description: (!update.description.is_empty()).then_some(update.description),
expiration: update.expiration,
status: (!update.status.is_empty()).then_some(update.status),
// Peers replicate credentials, never the local parent binding:
// each site resolves its own parent from its own IAM.
parent_user: None,
allow_site_replicator_account,
},
)
.await
.map_err(ApiError::from)?;
return Ok(());
}
if let Some(delete) = change.delete {
if let Some(local) = iam_sys.get_user(&delete.access_key).await
&& is_stale_update(local.update_at.unwrap_or(OffsetDateTime::UNIX_EPOCH), incoming_updated_at)
{
return Ok(());
}
iam_sys
.delete_service_account(&delete.access_key, true)
.await
.map_err(ApiError::from)?;
return Ok(());
}
Err(s3_error!(InvalidRequest, "serviceAccountChange is empty"))
}
fn claims_unix_timestamp(value: &Value) -> Option<i64> {
match value {
Value::Number(number) => number.as_i64(),
+262 -8
View File
@@ -17,7 +17,8 @@ use super::storage_api::bucket::metadata_sys;
use super::storage_api::bucket::replication::{self, BucketReplicationResyncStatus, BucketStats, ReplicationStatusType};
use super::storage_api::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
use super::storage_api::bucket::target_sys::{
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient, append_version_id_query,
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, SsecPassthroughCapability, TargetClient,
append_version_id_query,
};
use super::storage_api::bucket::versioning_sys::BucketVersioningSys;
use super::storage_api::bucket::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _};
@@ -70,6 +71,9 @@ use rustfs_utils::egress::{OutboundDnsResolver, OutboundPolicy};
use rustfs_utils::http::headers::{
AMZ_CHECKSUM_CRC32, AMZ_CHECKSUM_CRC32C, AMZ_CHECKSUM_CRC64NVME, AMZ_CHECKSUM_SHA1, AMZ_CHECKSUM_SHA256, AMZ_CHECKSUM_TYPE,
};
use rustfs_utils::http::object_encryption_keys::{
REPLICATION_SSEC_ALGORITHM_HEADER, REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_SSEC_ORIGINAL_SIZE_HEADER,
};
use rustfs_utils::http::{
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_VERSION_ID, get_source_scheme, insert_header,
@@ -213,6 +217,13 @@ const REPLICATION_CHECK_ERROR_MAX_BYTES: usize = 512;
/// RustFS extension code (no madmin analogue): the target does not adopt the
/// source version id, breaking the version-identity replication contract.
const REPLICATION_CHECK_CODE_VERSION_MISMATCH: &str = "BucketRemoteTargetVersionMismatch";
/// RustFS extension code (no madmin analogue): the target drops the
/// `X-Rustfs-Replication-*` SSE-C passthrough headers, so an SSE-C replica
/// would lose its decryption material (N2 fail-closed).
const REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH: &str = "BucketRemoteSsecPassthroughUnsupported";
/// Syntactically valid stand-in SSE-C key MD5 for the passthrough probe (the
/// probe object is never decrypted; it only has to round-trip the metadata).
const REPLICATION_CHECK_SSEC_PROBE_KEY_MD5: &str = "AAAAAAAAAAAAAAAAAAAAAA==";
#[derive(Debug, Clone, serde::Serialize)]
struct ReplicationCheckResponse {
@@ -254,6 +265,8 @@ struct ReplicationCheckPhases {
put: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionFidelity")]
version_fidelity: ReplicationCheckPhaseStatus,
#[serde(rename = "SsecPassthrough")]
ssec_passthrough: ReplicationCheckPhaseStatus,
#[serde(rename = "DeleteMarker")]
delete_marker: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionDelete")]
@@ -1852,7 +1865,7 @@ fn build_replication_check_response(mut targets: Vec<ReplicationCheckTargetStatu
let data = serde_json::to_vec(&ReplicationCheckResponse {
status: status.to_string(),
active_mutation: true,
mutation_description: "Writes a probe object, creates a delete marker, deletes the probe version, and cleans up all probe artifacts on each target.",
mutation_description: "Writes probe objects (including an SSE-C passthrough probe), creates a delete marker, deletes the probe versions, and cleans up all probe artifacts on each target.",
probe_namespace: REPLICATION_CHECK_PROBE_PREFIX,
targets,
})
@@ -2069,6 +2082,25 @@ async fn check_replication_target(
time: OffsetDateTime::now_utc(),
};
execute_replication_probe(&mut result, &mut operations).await;
// Sync the probe verdict into the runtime capability cache: the
// replication worker then fails SSE-C replication closed on a flagged
// target (or skips its own HEAD-back audit on a proven one) without
// re-learning what the probe just established.
match (result.phases.ssec_passthrough.status, result.phases.ssec_passthrough.code) {
("OK", _) => {
BucketTargetSys::get()
.record_ssec_passthrough_capability(&target.arn, SsecPassthroughCapability::Supported)
.await;
}
("FAILED", Some(REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH)) => {
BucketTargetSys::get()
.record_ssec_passthrough_capability(&target.arn, SsecPassthroughCapability::Unsupported)
.await;
}
_ => {}
}
result
}
@@ -2087,6 +2119,15 @@ struct ReplicationProbePutOutcome {
response_version_id: Option<String>,
}
/// Outcome of the SSE-C passthrough probe: whether the HEAD-back of the probe
/// replica echoed SSE-C evidence (the customer-algorithm header a RustFS
/// target restores from the passthrough transport headers), plus the version
/// the target assigned so cleanup can address it.
struct ReplicationSsecProbeOutcome {
evidence_present: bool,
version_id: Option<String>,
}
struct ReplicationProbeMultipartError {
primary: S3ClientError,
cleanup_error: Option<String>,
@@ -2109,9 +2150,14 @@ trait ReplicationProbeOperations {
/// there: a target can adopt PutObject version ids and still mint its own
/// for CreateMultipartUpload.
async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError>;
/// PUT a probe version carrying the SSE-C passthrough transport headers,
/// HEAD it back through the replication-check channel, and report whether
/// the SSE-C evidence survived. Cleanup of the created version is the
/// caller's job (the outcome carries its version id).
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError>;
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError>;
async fn delete_version(&mut self, version_id: Option<&str>) -> Result<(), S3ClientError>;
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String>;
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String>;
}
struct RemoteReplicationProbeOperations<'a> {
@@ -2131,6 +2177,10 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
multipart_put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
}
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
ssec_passthrough_probe_object(self.client, self.bucket, self.key, self.time).await
}
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
delete_replication_probe_object(
self.client,
@@ -2154,7 +2204,7 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
.map(|_| ())
}
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String> {
cleanup_replication_probe(self.client, self.bucket, self.key, known_version_ids).await
}
}
@@ -2175,6 +2225,7 @@ fn version_fidelity_error(api: &str, outcome: &ReplicationProbePutOutcome) -> Op
async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, operations: &mut impl ReplicationProbeOperations) {
let mut probe_version_id = None;
let mut multipart_probe_version_id = None;
let mut ssec_probe_version_id = None;
let mut delete_marker_version_id = None;
let mut cleanup_required = true;
let mut multipart_cleanup_error = None;
@@ -2230,6 +2281,38 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
}
}
// N2: probe SSE-C passthrough with the same transport headers live
// replication sends. A target that drops them (MinIO, generic S3) stores
// the probe as a plain object and echoes no SSE-C evidence on the
// HEAD-back; SSE-C replicas there would silently lose their decryption
// material, so the target must be flagged with a machine-readable code.
// Deliberately unlike VersionFidelity, a failed SsecPassthrough phase
// does NOT fail the target overall: version-identity drift breaks the
// replication contract for every object, while dropped SSE-C passthrough
// headers only limit a capability — a plaintext-only deployment against a
// MinIO target is perfectly healthy and must not turn red. The phase's
// own FAILED + machine-readable Code remains for madmin consumers (and
// the verdict still reaches the runtime capability cache).
if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
match operations.ssec_passthrough_probe().await {
Ok(outcome) => {
ssec_probe_version_id = outcome.version_id;
if outcome.evidence_present {
result.phases.ssec_passthrough = ReplicationCheckPhaseStatus::passed();
} else {
let error = "target drops SSE-C passthrough replication headers; \
SSE-C replicas would lose their decryption material on this target";
result.phases.ssec_passthrough =
ReplicationCheckPhaseStatus::failed_with_code(error, REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH);
}
}
Err(err) => {
let error = format_replication_check_client_error(&err, ReplicationCheckFailureContext::ReplicateObject);
result.phases.ssec_passthrough = ReplicationCheckPhaseStatus::failed(&error);
}
}
}
if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
match operations.create_delete_marker(probe_version_id.as_deref()).await {
Ok(version_id) => {
@@ -2258,6 +2341,7 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
.cleanup([
probe_version_id.as_deref(),
multipart_probe_version_id.as_deref(),
ssec_probe_version_id.as_deref(),
delete_marker_version_id.as_deref(),
])
.await
@@ -2552,6 +2636,72 @@ async fn put_replication_probe_object(
})
}
/// PUT a fresh probe version carrying the SSE-C passthrough transport headers
/// (the wire shape live SSE-C replication uses), then HEAD it back through the
/// worker channel (replication-check exemption + proxy suppression). A RustFS
/// target restores the transport headers into stored SSE-C metadata and its
/// HEAD echoes `x-amz-server-side-encryption-customer-algorithm`; a target
/// that dropped the headers echoes nothing. The probe body is never SSE-C
/// encrypted — only the metadata round-trip matters — and the version is
/// deleted by the shared probe cleanup.
async fn ssec_passthrough_probe_object(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
now: OffsetDateTime,
) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
let options = build_replication_probe_put_options(now);
let sent_version_id = options.internal.source_version_id.clone();
let mut headers = build_replication_probe_headers(&options);
// These are full wire names (not x-rustfs/x-minio suffixes), so they must
// be inserted verbatim — `insert_header` would mangle them.
for (name, value) in [
(REPLICATION_SSEC_ALGORITHM_HEADER, "AES256"),
(REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_CHECK_SSEC_PROBE_KEY_MD5),
(REPLICATION_SSEC_ORIGINAL_SIZE_HEADER, "8"),
] {
let name = name
.parse::<HeaderName>()
.map_err(|err| S3ClientError::new(format!("invalid ssec probe header name: {err}")))?;
let value =
HeaderValue::from_str(value).map_err(|err| S3ClientError::new(format!("invalid ssec probe header value: {err}")))?;
headers.insert(name, value);
}
let query_version_id = sent_version_id.clone();
let response = target_client
.client
.put_object()
.bucket(target_bucket)
.key(probe_key)
.content_length(8)
.body(AwsByteStream::from_static(b"aaaaaaaa"))
.customize()
.map_request(move |mut req| {
for (key, value) in headers.clone() {
req.headers_mut().insert(key.expect("operation should succeed"), value);
}
let uri = append_version_id_query(req.uri(), &query_version_id);
req.set_uri(uri).map_err(std::io::Error::other)?;
Result::<_, std::io::Error>::Ok(req)
})
.send()
.await
.map_err(S3ClientError::from)?;
let version_id = response.version_id().map(ToOwned::to_owned);
let head_version = version_id.clone().or_else(|| Some(sent_version_id.clone()));
let head = target_client
.head_object(target_bucket, probe_key, head_version)
.await
.map_err(S3ClientError::from)?;
Ok(ReplicationSsecProbeOutcome {
evidence_present: head.sse_customer_algorithm().is_some_and(|algorithm| !algorithm.is_empty()),
version_id,
})
}
async fn delete_replication_probe_object(
target_client: &TargetClient,
target_bucket: &str,
@@ -3722,6 +3872,12 @@ mod tests {
/// Same, for the multipart leg: a target may mirror PutObject ids and
/// still mint its own at CreateMultipartUpload.
minted_multipart_version_id: Option<&'static str>,
/// Transport failure of the SSE-C passthrough probe itself.
ssec_probe_error: Option<&'static str>,
/// Models a MinIO-like target that drops the SSE-C passthrough
/// headers: the probe HEAD-back echoes no SSE-C evidence. The default
/// (false) models a RustFS target that preserves them.
ssec_evidence_missing: bool,
delete_marker_error: Option<&'static str>,
version_delete_error: Option<&'static str>,
cleanup_error: Option<&'static str>,
@@ -3759,6 +3915,17 @@ mod tests {
})
}
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
self.calls.push("ssec-probe");
match self.ssec_probe_error {
Some(code) => Err(scripted_probe_error(code)),
None => Ok(ReplicationSsecProbeOutcome {
evidence_present: !self.ssec_evidence_missing,
version_id: Some("ssec-version".to_string()),
}),
}
}
async fn create_delete_marker(&mut self, _version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
self.calls.push("delete-marker");
match self.delete_marker_error {
@@ -3775,7 +3942,7 @@ mod tests {
}
}
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String> {
self.calls.push("cleanup");
self.cleanup_ids = known_version_ids
.into_iter()
@@ -3810,8 +3977,9 @@ mod tests {
assert_eq!(result.phases.version_fidelity.code, Some(REPLICATION_CHECK_CODE_VERSION_MISMATCH));
assert_eq!(result.phases.delete_marker.status, "SKIPPED");
assert_eq!(result.phases.version_delete.status, "SKIPPED");
assert_eq!(result.phases.ssec_passthrough.status, "SKIPPED");
assert_eq!(result.phases.cleanup.status, "OK");
assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None]);
assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None, None]);
}
#[tokio::test]
@@ -3824,6 +3992,70 @@ mod tests {
assert_eq!(result.status, "OK");
assert_eq!(result.phases.version_fidelity.status, "OK");
assert_eq!(result.phases.version_fidelity.code, None);
assert_eq!(result.phases.ssec_passthrough.status, "OK");
assert_eq!(result.phases.ssec_passthrough.code, None);
}
/// N2: a target that drops the SSE-C passthrough transport headers must
/// fail the SsecPassthrough phase with the machine-readable code while the
/// target overall stays OK — deliberately unlike VersionFidelity: this is
/// a capability limit, not a broken replication contract, and a
/// plaintext-only deployment against such a target must not turn red. The
/// other mutation phases keep running and the probe version is cleaned up.
#[tokio::test]
async fn replication_probe_flags_ssec_passthrough_dropping_target_without_failing_target() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe {
ssec_evidence_missing: true,
..Default::default()
};
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!(result.status, "OK", "a capability-only failure must not fail the target overall");
assert_eq!(result.error, None);
assert_eq!(result.phases.ssec_passthrough.status, "FAILED");
assert_eq!(result.phases.ssec_passthrough.code, Some(REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH));
assert_eq!(
operations.cleanup_ids,
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
Some("marker-version".to_string())
]
);
}
/// A transport failure of the SSE-C probe is not evidence of a dropping
/// target: the phase fails without the capability code (the runtime cache
/// stays Unknown and the worker keeps auditing), and the target overall
/// stays OK.
#[tokio::test]
async fn replication_probe_ssec_transport_failure_carries_no_capability_code() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe {
ssec_probe_error: Some("InternalError"),
..Default::default()
};
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(result.status, "OK");
assert_eq!(result.phases.ssec_passthrough.status, "FAILED");
assert_eq!(result.phases.ssec_passthrough.code, None);
assert_eq!(operations.cleanup_ids[2], None, "a failed ssec probe leaves no version to clean");
}
#[tokio::test]
@@ -3854,12 +4086,23 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!(
operations.cleanup_ids,
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
None
]
);
@@ -3879,12 +4122,23 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!(
operations.cleanup_ids,
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
Some("marker-version".to_string())
]
);
+1
View File
@@ -196,6 +196,7 @@ pub(crate) mod bucket_target_sys {
pub(crate) type PutObjectOptions = super::ecstore_bucket::bucket_target_sys::PutObjectOptions;
pub(crate) type RemoveObjectOptions = super::ecstore_bucket::bucket_target_sys::RemoveObjectOptions;
pub(crate) type S3ClientError = super::ecstore_bucket::bucket_target_sys::S3ClientError;
pub(crate) type SsecPassthroughCapability = super::ecstore_bucket::bucket_target_sys::SsecPassthroughCapability;
pub(crate) type TargetClient = super::ecstore_bucket::bucket_target_sys::TargetClient;
}
+248 -3
View File
@@ -46,9 +46,10 @@ use super::storage_api::object_usecase::bucket::{
replication::{
DeleteReplicationConfigSnapshot, REPLICATE_INCOMING_DELETE, ReplicationStatusType, commit_force_delete_intent,
delete_replication_state_from_config, delete_replication_version_id, deleted_object_has_pending_replication_delete,
force_delete_target_set, has_active_delete_rule, load_delete_config_snapshot, must_replicate_object,
persist_force_delete_intent, schedule_object_replication, schedule_replication_delete, schedule_replication_deletes,
set_deleted_object_replication_state, should_schedule_delete_replication, should_use_existing_delete_replication_info,
force_delete_target_set, get_read_proxy_targets, has_active_delete_rule, load_delete_config_snapshot,
must_replicate_object, persist_force_delete_intent, record_replication_proxy, schedule_object_replication,
schedule_replication_delete, schedule_replication_deletes, set_deleted_object_replication_state,
should_schedule_delete_replication, should_use_existing_delete_replication_info,
},
tagging::decode_tags,
validate_restore_request,
@@ -6620,6 +6621,226 @@ impl DefaultObjectUsecase {
})
}
/// Headers a proxied read forwards verbatim to the replication target:
/// only the client's SSE-C key family, so the target performs the real
/// SSE-C decryption (never the replication-check exemption). HTTP
/// conditional headers (If-Match & co.) are deliberately NOT forwarded —
/// MinIO does not forward them either, and a remote 304/412 would leak a
/// conditional evaluation against a replica the local site never saw.
/// Range and part-number travel as typed SDK parameters instead.
fn proxy_read_passthrough_headers(headers: &HeaderMap) -> HeaderMap {
const FORWARDED: &[&str] = &[
"x-amz-server-side-encryption-customer-algorithm",
"x-amz-server-side-encryption-customer-key",
"x-amz-server-side-encryption-customer-key-md5",
];
let mut forwarded = HeaderMap::new();
for name in FORWARDED {
if let Ok(header_name) = http::HeaderName::from_str(name)
&& let Some(value) = headers.get(&header_name)
{
forwarded.insert(header_name, value.clone());
}
}
forwarded
}
/// True when a proxied SDK call failed because the target does not have
/// the object either (service-level not-found or a raw 404, which also
/// covers NoSuchVersion): the caller tries the next target silently.
fn proxy_sdk_error_is_not_found<E>(err: &aws_sdk_s3::error::SdkError<E>) -> bool {
err.raw_response().is_some_and(|resp| resp.status().as_u16() == 404)
}
/// Serve a GET whose local read failed with not-found by proxying to the
/// bucket's replication targets (MinIO `proxyGetToReplicationTarget`,
/// backlog#1675 P1-5). Returns None when no target can serve the object;
/// the caller then returns the original local error.
async fn proxy_get_object_to_replication_targets(
req: &S3Request<GetObjectInput>,
bucket: &str,
key: &str,
opts: &ObjectOptions,
) -> Option<GetObjectOutput> {
let targets = get_read_proxy_targets(bucket, key, opts).await;
if targets.is_empty() {
return None;
}
let extra_headers = Self::proxy_read_passthrough_headers(&req.headers);
let range = req
.headers
.get(http::header::RANGE)
.and_then(|value| value.to_str().ok())
.map(str::to_owned);
let part_number = req.input.part_number;
for target in targets {
match target
.get_object(
&target.bucket,
key,
opts.version_id.clone(),
range.clone(),
part_number,
extra_headers.clone(),
)
.await
{
Ok(remote) => {
// MinIO-aligned accounting: one total per proxy attempt
// (targets were available), one failed when no target
// served it — never per target.
record_replication_proxy(bucket, "GetObject", false).await;
return Some(Self::proxy_sdk_get_output_to_s3s(remote));
}
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
debug!(bucket, key, arn = %target.arn, "read proxy: target does not have the object");
}
Err(err) => {
warn!(bucket, key, arn = %target.arn, error = %err, "read proxy: GET against replication target failed");
}
}
}
record_replication_proxy(bucket, "GetObject", true).await;
None
}
/// Serve a HEAD whose local lookup failed with not-found by proxying to
/// the bucket's replication targets (MinIO `proxyHeadToRepTarget`).
async fn proxy_head_object_to_replication_targets(
req: &S3Request<HeadObjectInput>,
bucket: &str,
key: &str,
opts: &ObjectOptions,
) -> Option<HeadObjectOutput> {
let targets = get_read_proxy_targets(bucket, key, opts).await;
if targets.is_empty() {
return None;
}
let extra_headers = Self::proxy_read_passthrough_headers(&req.headers);
let range = req
.headers
.get(http::header::RANGE)
.and_then(|value| value.to_str().ok())
.map(str::to_owned);
let part_number = req.input.part_number;
for target in targets {
match target
.head_object_for_proxy(
&target.bucket,
key,
opts.version_id.clone(),
range.clone(),
part_number,
extra_headers.clone(),
)
.await
{
Ok(remote) => {
// MinIO-aligned accounting: one total per proxy attempt,
// one failed when no target served it.
record_replication_proxy(bucket, "HeadObject", false).await;
return Some(Self::proxy_sdk_head_output_to_s3s(remote));
}
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
debug!(bucket, key, arn = %target.arn, "read proxy: target does not have the object");
}
Err(err) => {
warn!(bucket, key, arn = %target.arn, error = %err, "read proxy: HEAD against replication target failed");
}
}
}
record_replication_proxy(bucket, "HeadObject", true).await;
None
}
/// Translate a proxied SDK GET response into the s3s output, forwarding
/// the body as a stream (no buffering, no local persistence).
fn proxy_sdk_get_output_to_s3s(remote: aws_sdk_s3::operation::get_object::GetObjectOutput) -> GetObjectOutput {
let body = remote.body;
let body_stream = tokio_util::io::ReaderStream::with_capacity(body.into_async_read(), 64 * 1024);
GetObjectOutput {
body: Some(StreamingBlob::wrap(body_stream)),
content_length: remote.content_length,
content_range: remote.content_range,
content_type: remote.content_type.as_deref().and_then(|v| ContentType::from_str(v).ok()),
content_encoding: remote.content_encoding,
content_disposition: remote.content_disposition,
content_language: remote.content_language,
cache_control: remote.cache_control,
accept_ranges: Some(ACCEPT_RANGES_BYTES.to_string()),
e_tag: remote.e_tag.as_deref().and_then(|v| ETag::from_str(v).ok()),
last_modified: remote
.last_modified
.and_then(|dt| OffsetDateTime::from_unix_timestamp_nanos(dt.as_nanos()).ok())
.map(Timestamp::from),
metadata: remote.metadata,
version_id: remote.version_id,
server_side_encryption: remote
.server_side_encryption
.map(|sse| ServerSideEncryption::from(sse.as_str().to_string())),
sse_customer_algorithm: remote.sse_customer_algorithm,
sse_customer_key_md5: remote.sse_customer_key_md5,
ssekms_key_id: remote.ssekms_key_id,
parts_count: remote.parts_count,
tag_count: remote.tag_count,
storage_class: remote.storage_class.map(|sc| StorageClass::from(sc.as_str().to_string())),
expiration: remote.expiration,
restore: remote.restore,
checksum_crc32: remote.checksum_crc32,
checksum_crc32c: remote.checksum_crc32_c,
checksum_crc64nvme: remote.checksum_crc64_nvme,
checksum_sha1: remote.checksum_sha1,
checksum_sha256: remote.checksum_sha256,
checksum_type: remote.checksum_type.map(|ct| ChecksumType::from(ct.as_str().to_string())),
..Default::default()
}
}
/// Translate a proxied SDK HEAD response into the s3s output.
///
/// Known gaps: the SDK's HeadObjectOutput does not model 206/Content-Range
/// for a ranged HEAD (the SDK exposes no content_range member on HEAD),
/// and s3s' typed HeadObjectOutput has no tag_count field (the local path
/// injects x-amz-tagging-count as a raw header) — both are dropped for
/// proxied HEADs.
fn proxy_sdk_head_output_to_s3s(remote: aws_sdk_s3::operation::head_object::HeadObjectOutput) -> HeadObjectOutput {
HeadObjectOutput {
content_length: remote.content_length,
content_type: remote.content_type.as_deref().and_then(|v| ContentType::from_str(v).ok()),
content_encoding: remote.content_encoding,
content_disposition: remote.content_disposition,
content_language: remote.content_language,
cache_control: remote.cache_control,
accept_ranges: Some(ACCEPT_RANGES_BYTES.to_string()),
e_tag: remote.e_tag.as_deref().and_then(|v| ETag::from_str(v).ok()),
last_modified: remote
.last_modified
.and_then(|dt| OffsetDateTime::from_unix_timestamp_nanos(dt.as_nanos()).ok())
.map(Timestamp::from),
metadata: remote.metadata,
version_id: remote.version_id,
server_side_encryption: remote
.server_side_encryption
.map(|sse| ServerSideEncryption::from(sse.as_str().to_string())),
sse_customer_algorithm: remote.sse_customer_algorithm,
sse_customer_key_md5: remote.sse_customer_key_md5,
ssekms_key_id: remote.ssekms_key_id,
parts_count: remote.parts_count,
storage_class: remote.storage_class.map(|sc| StorageClass::from(sc.as_str().to_string())),
expiration: remote.expiration,
restore: remote.restore,
checksum_crc32: remote.checksum_crc32,
checksum_crc32c: remote.checksum_crc32_c,
checksum_crc64nvme: remote.checksum_crc64_nvme,
checksum_sha1: remote.checksum_sha1,
checksum_sha256: remote.checksum_sha256,
checksum_type: remote.checksum_type.map(|ct| ChecksumType::from(ct.as_str().to_string())),
..Default::default()
}
}
#[instrument(name = "execute_get_object", level = "trace", skip(self, req))]
pub async fn execute_get_object(&self, req: S3Request<GetObjectInput>) -> S3Result<S3Response<GetObjectOutput>> {
self.execute_get_object_boxed(req).await
@@ -6745,6 +6966,19 @@ impl DefaultObjectUsecase {
{
Ok(prepared_read) => prepared_read,
Err(err) => {
// Active-active replication lag window: an object missing
// locally (and only missing — other errors keep their
// semantics) may still be served by proxying the GET to a
// replication target (backlog#1675 P1-5).
if matches!(*err.code(), S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion)
&& let Some(output) = Self::proxy_get_object_to_replication_targets(&req, &bucket, &key, &opts).await
{
lifecycle.finish_ok();
let response = wrap_response_with_cors(&bucket, &req.method, &req.headers, output).await;
let result = Ok(response);
let _ = helper.version_id(version_id_for_event).complete(&result);
return result;
}
lifecycle.finish_err();
return Err(err);
}
@@ -8654,6 +8888,17 @@ impl DefaultObjectUsecase {
let msg = head_prefix_not_found_message(&bucket, &key, has_children);
return Err(S3Error::with_message(S3ErrorCode::NoSuchKey, msg));
}
// Active-active replication lag window: an object missing
// locally may still be served by proxying the HEAD to a
// replication target (backlog#1675 P1-5).
if let Some(output) = Self::proxy_head_object_to_replication_targets(&req, &bucket, &key, &opts).await {
let response = wrap_response_with_cors(&bucket, &req.method, &req.headers, output).await;
let result = Ok(response);
let _ = helper
.version_id(req.input.version_id.clone().unwrap_or_default())
.complete(&result);
return result;
}
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
}
// Other errors, such as insufficient permissions, still return the original error
+18
View File
@@ -627,6 +627,24 @@ pub(crate) mod bucket {
#[cfg(test)]
pub(crate) use replication_contracts::replication_statuses_map;
/// Remote replication-target client used by the read-proxy path.
pub(crate) type ProxyTargetClient = crate::storage::storage_api::ecstore_bucket::bucket_target_sys::TargetClient;
/// Proxy-request metric recorder (get/head/tagging totals + failures).
pub(crate) use crate::storage::storage_api::record_replication_proxy;
/// Replication targets eligible to serve a proxied GET/HEAD/Tagging of
/// an object not present locally (MinIO `getProxyTargets`; empty when
/// the request was itself proxied, versioning is suspended, or no
/// replication rule matches). backlog#1675 P1-5.
pub(crate) async fn get_read_proxy_targets(
bucket: &str,
object: &str,
opts: &crate::storage::storage_api::StorageObjectOptions,
) -> Vec<Arc<ProxyTargetClient>> {
replication_contracts::get_proxy_targets(bucket, object, opts).await
}
pub(crate) async fn persist_force_delete_intent(
store: Arc<crate::storage::storage_api::ECStore>,
bucket: String,
+229 -38
View File
@@ -12,15 +12,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::StorageVersioningConfigExt as _;
use super::{
BUCKET_ACCELERATE_CONFIG, BUCKET_LOGGING_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_VERSIONING_CONFIG,
BUCKET_WEBSITE_CONFIG, BucketVersioningSys, OBJECT_LOCK_CONFIG, StorageError, check_retention_for_modification, decode_tags,
decode_tags_to_map, delete_bucket_metadata_config_if_incarnation, encode_tags, get_bucket_accelerate_config,
get_bucket_logging_config, get_bucket_object_lock_config, get_bucket_replication_config, get_bucket_request_payment_config,
get_bucket_website_config, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
record_replication_proxy, serialize, update_bucket_metadata_config_if_incarnation,
get_bucket_logging_config, get_bucket_object_lock_config, get_bucket_request_payment_config, get_bucket_website_config,
is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, record_replication_proxy, serialize,
update_bucket_metadata_config_if_incarnation,
};
use super::{StorageReplicationConfigExt as _, StorageVersioningConfigExt as _};
use crate::admin::handlers::site_replication::site_replication_bucket_meta_hook;
use crate::error::ApiError;
use crate::storage::access::{apply_bucket_generation_guard, bucket_config_mutation_incarnation, has_bypass_governance_header};
@@ -59,7 +59,7 @@ const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
const LOG_SUBSYSTEM_TAGGING: &str = "tagging";
use crate::app::storage_api::object_usecase::bucket::replication::{
ReplicateDecision, must_replicate_metadata, schedule_metadata_replication,
ReplicateDecision, get_read_proxy_targets, must_replicate_metadata, schedule_metadata_replication,
};
use crate::storage::storage_api::ecfs_consumer::StorageObjectOptions as ObjectOptions;
@@ -105,18 +105,152 @@ impl FS {
&self.server_ctx
}
async fn replication_tagging_enabled(bucket: &str, object: &str) -> bool {
get_bucket_replication_config(bucket)
.await
.map(|(cfg, _)| cfg.has_active_rules(object, true))
.unwrap_or(false)
/// Not-found classifier for proxied SDK tagging calls: a raw 404 covers
/// NoSuchKey and NoSuchVersion alike; the caller silently tries the next
/// replication target.
fn proxy_sdk_error_is_not_found<E>(err: &aws_sdk_s3::error::SdkError<E>) -> bool {
err.raw_response().is_some_and(|resp| resp.status().as_u16() == 404)
}
async fn record_replication_tagging_metric(bucket: &str, object: &str, api: &str, is_err: bool) {
if !Self::replication_tagging_enabled(bucket, object).await {
return;
/// Selector options for a tagging proxy. Reuses `get_opts` so the
/// anti-loop `source-proxy-request` header family and the bucket's
/// version-suspension state gate proxying exactly like GET/HEAD.
async fn tagging_proxy_opts(
bucket: &str,
object: &str,
version_id: Option<String>,
headers: &http::HeaderMap,
) -> Option<ObjectOptions> {
get_opts(bucket, object, version_id, None, headers).await.ok()
}
/// Serve a GetObjectTagging for an object missing locally by proxying to
/// the bucket's replication targets (MinIO `proxyGetTaggingToRepTarget`,
/// backlog#1675 P1-5). None means no target had the object.
async fn proxy_get_object_tagging(
bucket: &str,
object: &str,
version_id: Option<String>,
headers: &http::HeaderMap,
) -> Option<TagSet> {
let opts = Self::tagging_proxy_opts(bucket, object, version_id, headers).await?;
let targets = get_read_proxy_targets(bucket, object, &opts).await;
if targets.is_empty() {
return None;
}
record_replication_proxy(bucket, api, is_err).await;
for target in targets {
match target
.get_object_tagging(&target.bucket, object, opts.version_id.clone())
.await
{
Ok(remote) => {
// MinIO-aligned accounting: one total per proxy attempt,
// one failed when no target served it.
record_replication_proxy(bucket, "GetObjectTagging", false).await;
return Some(
remote
.tag_set
.into_iter()
.map(|tag| Tag {
key: Some(tag.key),
value: Some(tag.value),
})
.collect(),
);
}
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
debug!(bucket, object, arn = %target.arn, "tagging proxy: target does not have the object");
}
Err(err) => {
warn!(bucket, object, arn = %target.arn, error = %err, "tagging proxy: GetObjectTagging against replication target failed");
}
}
}
record_replication_proxy(bucket, "GetObjectTagging", true).await;
None
}
/// Apply a PutObjectTagging for an object missing locally on a
/// replication target (MinIO `proxyTaggingToRepTarget`).
async fn proxy_put_object_tagging(
bucket: &str,
object: &str,
version_id: Option<String>,
headers: &http::HeaderMap,
tag_set: &TagSet,
) -> Option<()> {
let opts = Self::tagging_proxy_opts(bucket, object, version_id, headers).await?;
let mut tagging = aws_sdk_s3::types::Tagging::builder();
for tag in tag_set {
let sdk_tag = aws_sdk_s3::types::Tag::builder()
.key(tag.key.clone().unwrap_or_default())
.value(tag.value.clone().unwrap_or_default())
.build()
.ok()?;
tagging = tagging.tag_set(sdk_tag);
}
let tagging = tagging.build().ok()?;
let targets = get_read_proxy_targets(bucket, object, &opts).await;
if targets.is_empty() {
return None;
}
for target in targets {
match target
.put_object_tagging(&target.bucket, object, opts.version_id.clone(), tagging.clone())
.await
{
Ok(_) => {
// MinIO-aligned accounting: one total per proxy attempt,
// one failed when no target served it.
record_replication_proxy(bucket, "PutObjectTagging", false).await;
return Some(());
}
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
debug!(bucket, object, arn = %target.arn, "tagging proxy: target does not have the object");
}
Err(err) => {
warn!(bucket, object, arn = %target.arn, error = %err, "tagging proxy: PutObjectTagging against replication target failed");
}
}
}
record_replication_proxy(bucket, "PutObjectTagging", true).await;
None
}
/// Apply a DeleteObjectTagging for an object missing locally on a
/// replication target (MinIO `proxyTaggingToRepTarget`).
async fn proxy_delete_object_tagging(
bucket: &str,
object: &str,
version_id: Option<String>,
headers: &http::HeaderMap,
) -> Option<()> {
let opts = Self::tagging_proxy_opts(bucket, object, version_id, headers).await?;
let targets = get_read_proxy_targets(bucket, object, &opts).await;
if targets.is_empty() {
return None;
}
for target in targets {
match target
.delete_object_tagging(&target.bucket, object, opts.version_id.clone())
.await
{
Ok(_) => {
// MinIO-aligned accounting: one total per proxy attempt,
// one failed when no target served it.
record_replication_proxy(bucket, "DeleteObjectTagging", false).await;
return Some(());
}
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
debug!(bucket, object, arn = %target.arn, "tagging proxy: target does not have the object");
}
Err(err) => {
warn!(bucket, object, arn = %target.arn, error = %err, "tagging proxy: DeleteObjectTagging against replication target failed");
}
}
}
record_replication_proxy(bucket, "DeleteObjectTagging", true).await;
None
}
pub async fn get_object_tag_conditions_for_policy(
@@ -447,7 +581,27 @@ impl S3 for FS {
let mut opts = get_opts(&bucket, &object, version_id.clone(), None, &req.headers)
.await
.map_err(ApiError::from)?;
let existing_object_info = store.get_object_info(&bucket, &object, &opts).await.map_err(ApiError::from)?;
let existing_object_info = match store.get_object_info(&bucket, &object, &opts).await {
Ok(info) => info,
Err(e) => {
// Replication lag window: apply the tagging delete on a
// replication target that already has the object
// (backlog#1675 P1-5). No local object exists, so no bucket
// notification event is emitted for the proxied write.
if (is_err_object_not_found(&e) || is_err_version_not_found(&e))
&& Self::proxy_delete_object_tagging(&bucket, &object, version_id.clone(), &req.headers)
.await
.is_some()
{
counter!("rustfs_delete_object_tagging_success").increment(1);
let duration = start_time.elapsed();
histogram!("rustfs_object_tagging_operation_duration_seconds", "operation" => "delete")
.record(duration.as_secs_f64());
return Ok(S3Response::new(DeleteObjectTaggingOutput { version_id }));
}
return Err(ApiError::from(e).into());
}
};
let dsc = must_replicate_metadata(
&bucket,
&object,
@@ -470,7 +624,6 @@ impl S3 for FS {
}
let delete_tags_result = store.delete_object_tags(&bucket, &object, &opts).await;
Self::record_replication_tagging_metric(&bucket, &object, "DeleteObjectTagging", delete_tags_result.is_err()).await;
let object_info = delete_tags_result.map_err(|e| {
error!(
component = LOG_COMPONENT_STORAGE,
@@ -928,32 +1081,49 @@ impl S3 for FS {
..Default::default()
};
let tags_result = store.get_object_tags(bucket, object, &opts).await;
Self::record_replication_tagging_metric(bucket, object, "GetObjectTagging", tags_result.is_err()).await;
let tags = tags_result.map_err(|e| {
if is_err_object_not_found(&e) {
debug!(
let tags = match store.get_object_tags(bucket, object, &opts).await {
Ok(tags) => tags,
Err(e) => {
// Replication lag window: the object may exist on a
// replication target even though it is missing locally —
// proxy the tagging read there (backlog#1675 P1-5).
if (is_err_object_not_found(&e) || is_err_version_not_found(&e))
&& let Some(tag_set) =
Self::proxy_get_object_tagging(bucket, object, req.input.version_id.clone(), &req.headers).await
{
counter!("rustfs_get_object_tagging_success").increment(1);
let duration = start_time.elapsed();
histogram!("rustfs_object_tagging_operation_duration_seconds", "operation" => "get")
.record(duration.as_secs_f64());
return Ok(S3Response::new(GetObjectTaggingOutput {
tag_set,
version_id: req.input.version_id.clone(),
}));
}
if is_err_object_not_found(&e) {
debug!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_TAGGING,
event = "object_tagging_not_found",
bucket = %bucket,
object = %object,
error = %e,
"Object tags not found"
);
return Err(s3_error!(NoSuchKey));
}
error!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_TAGGING,
event = "object_tagging_not_found",
event = "object_tagging_get_failed",
bucket = %bucket,
object = %object,
error = %e,
"Object tags not found"
"Failed to load object tags"
);
return s3_error!(NoSuchKey);
return Err(ApiError::from(e).into());
}
error!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_TAGGING,
event = "object_tagging_get_failed",
bucket = %bucket,
object = %object,
error = %e,
"Failed to load object tags"
);
ApiError::from(e).into()
})?;
};
let tag_set = decode_tags(tags.as_str());
debug!(
@@ -1629,14 +1799,36 @@ impl S3 for FS {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let tags = encode_tags(tagging.tag_set);
let tags = encode_tags(tagging.tag_set.clone());
debug!("Encoded tags: {}", tags);
let version_id = req.input.version_id.clone();
let mut opts = get_opts(&bucket, &object, version_id.clone(), None, &req.headers)
.await
.map_err(ApiError::from)?;
let existing_object_info = store.get_object_info(&bucket, &object, &opts).await.map_err(ApiError::from)?;
let existing_object_info = match store.get_object_info(&bucket, &object, &opts).await {
Ok(info) => info,
Err(e) => {
// Replication lag window: apply the tagging update on a
// replication target that already has the object
// (backlog#1675 P1-5). No local object exists, so no bucket
// notification event is emitted for the proxied write.
if (is_err_object_not_found(&e) || is_err_version_not_found(&e))
&& Self::proxy_put_object_tagging(&bucket, &object, version_id.clone(), &req.headers, &tagging.tag_set)
.await
.is_some()
{
counter!("rustfs_put_object_tagging_success").increment(1);
let duration = start_time.elapsed();
histogram!("rustfs_object_tagging_operation_duration_seconds", "operation" => "put")
.record(duration.as_secs_f64());
return Ok(S3Response::new(PutObjectTaggingOutput {
version_id: req.input.version_id.clone(),
}));
}
return Err(ApiError::from(e).into());
}
};
let dsc = must_replicate_metadata(
&bucket,
&object,
@@ -1659,7 +1851,6 @@ impl S3 for FS {
}
let put_tags_result = store.put_object_tags(&bucket, &object, &tags, &opts).await;
Self::record_replication_tagging_metric(&bucket, &object, "PutObjectTagging", put_tags_result.is_err()).await;
let object_info = put_tags_result.map_err(|e| {
error!("Failed to put object tags: {}", e);
counter!("rustfs_put_object_tagging_failure").increment(1);
+18 -18
View File
@@ -55,24 +55,24 @@ pub(crate) use storage_api::{
QuotaError, RUSTFS_META_BUCKET, RawFileInfo, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
ReplicationStats, ReplicationStatusType, Result, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
StorageDeletedObject, StorageDiskRpcExt, StorageError, StorageGetObjectReader, StorageObjectInfo, StorageObjectOptions,
StorageObjectToDelete, StoragePeerS3ClientExt, StoragePutObjReader, StorageReplicationConfigExt, StorageVersioningConfigExt,
TONIC_RPC_PREFIX, TierConfigMgr, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, WorkloadAdmissionSnapshotProviderRef,
WriteEncryption, WritePlan, access_consumer, add_object_lock_years, all_local_disk, all_local_disk_path,
check_retention_for_modification, collect_local_metrics, compression_metadata_value, contract, decode_tags,
decode_tags_to_map, delete_bucket_metadata_config, delete_bucket_metadata_config_if_incarnation, disk_drive_path,
disk_endpoint, ecfs_consumer, ecfs_extend_consumer, ecstore_admin, ecstore_bucket, ecstore_capacity, ecstore_client,
ecstore_cluster, ecstore_compression, ecstore_config, ecstore_data_usage, ecstore_disk, ecstore_error, ecstore_event,
ecstore_layout, ecstore_metrics, ecstore_notification, ecstore_rebalance, ecstore_rio, ecstore_rpc, ecstore_set_disk,
ecstore_storage, ecstore_tier, encode_tags, find_local_disk_by_ref, get_bucket_accelerate_config, get_bucket_cors_config,
get_bucket_logging_config, get_bucket_metadata, get_bucket_notification_config, get_bucket_object_lock_config,
get_bucket_replication_config, get_bucket_request_payment_config, get_bucket_sse_config, get_bucket_website_config,
get_local_server_property, get_lock_acquire_timeout, head_prefix_consumer, helper_consumer, init_background_replication,
init_bucket_metadata_sys, init_ecstore_config, init_local_disks_with_instance_ctx, init_lock_clients,
is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, is_valid_storage_class, options_consumer,
prewarm_local_disk_id_map_with_instance_ctx, read_config, record_replication_proxy, rpc_consumer, runtime_sources_consumer,
s3_api_consumer, serialize, table_catalog_path_hash, to_s3s_etag, topology_snapshot_from_endpoint_pools_with_capabilities,
try_migrate_bucket_metadata, try_migrate_iam_config, try_migrate_server_config, update_bucket_metadata_config,
update_bucket_metadata_config_if_incarnation, verify_rpc_signature, wrap_reader,
StorageObjectToDelete, StoragePeerS3ClientExt, StoragePutObjReader, StorageVersioningConfigExt, TONIC_RPC_PREFIX,
TierConfigMgr, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, WorkloadAdmissionSnapshotProviderRef, WriteEncryption,
WritePlan, access_consumer, add_object_lock_years, all_local_disk, all_local_disk_path, check_retention_for_modification,
collect_local_metrics, compression_metadata_value, contract, decode_tags, decode_tags_to_map, delete_bucket_metadata_config,
delete_bucket_metadata_config_if_incarnation, disk_drive_path, disk_endpoint, ecfs_consumer, ecfs_extend_consumer,
ecstore_admin, ecstore_bucket, ecstore_capacity, ecstore_client, ecstore_cluster, ecstore_compression, ecstore_config,
ecstore_data_usage, ecstore_disk, ecstore_error, ecstore_event, ecstore_layout, ecstore_metrics, ecstore_notification,
ecstore_rebalance, ecstore_rio, ecstore_rpc, ecstore_set_disk, ecstore_storage, ecstore_tier, encode_tags,
find_local_disk_by_ref, get_bucket_accelerate_config, get_bucket_cors_config, get_bucket_logging_config, get_bucket_metadata,
get_bucket_notification_config, get_bucket_object_lock_config, get_bucket_request_payment_config, get_bucket_sse_config,
get_bucket_website_config, get_local_server_property, get_lock_acquire_timeout, head_prefix_consumer, helper_consumer,
init_background_replication, init_bucket_metadata_sys, init_ecstore_config, init_local_disks_with_instance_ctx,
init_lock_clients, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, is_valid_storage_class,
options_consumer, prewarm_local_disk_id_map_with_instance_ctx, read_config, record_replication_proxy, rpc_consumer,
runtime_sources_consumer, s3_api_consumer, serialize, table_catalog_path_hash, to_s3s_etag,
topology_snapshot_from_endpoint_pools_with_capabilities, try_migrate_bucket_metadata, try_migrate_iam_config,
try_migrate_server_config, update_bucket_metadata_config, update_bucket_metadata_config_if_incarnation, verify_rpc_signature,
wrap_reader,
};
#[cfg(test)]
+93 -3
View File
@@ -19,9 +19,10 @@ use http::{HeaderMap, HeaderValue};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_FORCE_DELETE, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID, SUFFIX_TAGGING_TIMESTAMP, get_header,
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_PROXY_REQUEST,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
SUFFIX_TAGGING_TIMESTAMP, get_header,
header_compat::{MINIO_ENCRYPTION_PREFIX, RUSTFS_ENCRYPTION_PREFIX},
insert_header_map, insert_str,
metadata_compat::{MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX},
@@ -276,6 +277,19 @@ pub async fn get_opts(
// Background scanner still performs full integrity checks asynchronously.
opts.skip_verify_bitrot = get_skip_verify_bitrot();
// Anti-loop markers for the replication read proxy
// (`{x-rustfs-,x-minio-}source-proxy-request` header family).
// MinIO semantics: the header being PRESENT at all (`ProxyHeaderSet`)
// disables proxying, whatever its value — a peer's replication worker
// sends "false" on its convergence HEADs so the receiver answers locally
// instead of proxying the miss back (a proxied echo would fake
// convergence and the object would never replicate). Deliberately not
// gated on replication authorization: the header only disables proxying
// (it grants nothing).
let proxy_header = get_header(headers, SUFFIX_SOURCE_PROXY_REQUEST);
opts.proxy_header_set = proxy_header.is_some();
opts.proxy_request = proxy_header.map(|v| v.as_ref() == "true").unwrap_or_default();
fill_conditional_writes_opts_from_header(headers, &mut opts)?;
Ok(opts)
@@ -2544,4 +2558,80 @@ mod tests {
}
}
}
/// The replication read-proxy anti-loop markers must be honored under
/// both interop prefixes (a MinIO peer sends x-minio-, a RustFS peer
/// sends both). `proxy_request` is set only for the literal value
/// "true", while `proxy_header_set` (MinIO `ProxyHeaderSet`) is set by
/// the header's mere presence — "false" (the replication worker's
/// convergence-HEAD marker) and arbitrary values included — so the
/// selector refuses to proxy either way.
#[tokio::test]
async fn test_get_opts_parses_source_proxy_request_under_both_prefixes() {
for header_name in ["x-rustfs-source-proxy-request", "x-minio-source-proxy-request"] {
let mut headers = HeaderMap::new();
headers.insert(header_name, HeaderValue::from_static("true"));
let opts = get_opts("test-bucket", "test-object", None, None, &headers)
.await
.expect("get_opts should succeed");
assert!(opts.proxy_request, "{header_name} must set opts.proxy_request");
assert!(opts.proxy_header_set, "{header_name} must set opts.proxy_header_set");
}
let opts = get_opts("test-bucket", "test-object", None, None, &HeaderMap::new())
.await
.expect("get_opts should succeed");
assert!(!opts.proxy_request, "absent header must leave proxy_request off");
assert!(!opts.proxy_header_set, "absent header must leave proxy_header_set off");
for (header_name, value) in [
("x-minio-source-proxy-request", "false"),
("x-rustfs-source-proxy-request", "false"),
("x-minio-source-proxy-request", "anything-else"),
] {
let mut headers = HeaderMap::new();
headers.insert(header_name, HeaderValue::from_static(value));
let opts = get_opts("test-bucket", "test-object", None, None, &headers)
.await
.expect("get_opts should succeed");
assert!(!opts.proxy_request, "{header_name}: non-'true' value must leave proxy_request off");
assert!(
opts.proxy_header_set,
"{header_name}: value {value:?} must still set proxy_header_set (presence disables proxying)"
);
}
}
/// Pin that the source-proxy-request transport family cannot be
/// materialized as bare stored metadata via an `x-*-meta-` disguise: the
/// reserved-key namespacing (`x-rustfs-source-` / `x-minio-source-`
/// prefixes in `is_reserved_user_metadata_key`) must keep covering it.
#[test]
fn test_source_proxy_request_family_is_reserved_user_metadata() {
let mut headers = HeaderMap::new();
headers.insert("x-amz-meta-x-minio-source-proxy-request", HeaderValue::from_static("true"));
headers.insert("x-rustfs-meta-x-rustfs-source-proxy-request", HeaderValue::from_static("true"));
// The bare transport header itself is not a user-metadata prefix and
// must never land in stored metadata at all.
headers.insert("x-minio-source-proxy-request", HeaderValue::from_static("true"));
let metadata = extract_metadata(&headers);
assert!(
!metadata.contains_key("x-minio-source-proxy-request"),
"bare source-proxy-request key must not be storable: {metadata:?}"
);
assert!(
!metadata.contains_key("x-rustfs-source-proxy-request"),
"bare source-proxy-request key must not be storable: {metadata:?}"
);
assert!(
metadata.contains_key("x-amz-meta-x-minio-source-proxy-request"),
"disguised key must be namespaced back under x-amz-meta-: {metadata:?}"
);
assert!(
metadata.contains_key("x-amz-meta-x-rustfs-source-proxy-request"),
"disguised key must be namespaced back under x-amz-meta-: {metadata:?}"
);
}
}
+2 -8
View File
@@ -571,12 +571,8 @@ async fn execute_heal_control_envelope_with_manager(
admission: receipt.result.into(),
}
}
rustfs_protos::heal_control::ExecutableCommand::Query {
heal_path,
client_token,
since_seq,
} => {
let response = timeout(remaining, processor.execute_query_request_since(heal_path, client_token, since_seq))
rustfs_protos::heal_control::ExecutableCommand::Query { heal_path, client_token } => {
let response = timeout(remaining, processor.execute_query_request(heal_path, client_token))
.await
.map_err(|_| Status::deadline_exceeded("heal control query expired before execution"))?
.map_err(|_| Status::internal("heal control query failed"))?;
@@ -2521,7 +2517,6 @@ mod tests {
metadata(),
"bucket/prefix".to_string(),
canonical_token.clone(),
None,
)
.unwrap();
let query_result = execute_heal_control_envelope_with_manager(query, coordinator_epoch, Some(Arc::clone(&manager)))
@@ -2560,7 +2555,6 @@ mod tests {
metadata(),
"bucket/prefix".to_string(),
canonical_token,
None,
)
.unwrap();
let stopped_result = execute_heal_control_envelope_with_manager(stopped_query, coordinator_epoch, Some(manager))
+8 -18
View File
@@ -809,6 +809,10 @@ impl StorageReplicationStatsHandle {
proxy_head_failed: metrics.proxied.head_failed,
proxy_put_tag_total: metrics.proxied.put_tag_total,
proxy_put_tag_failed: metrics.proxied.put_tag_failed,
proxy_get_tag_total: metrics.proxied.get_tag_total,
proxy_get_tag_failed: metrics.proxied.get_tag_failed,
proxy_delete_tag_total: metrics.proxied.delete_tag_total,
proxy_delete_tag_failed: metrics.proxied.delete_tag_failed,
replica_size: metrics.replica_size,
replica_count: metrics.replica_count,
}
@@ -845,6 +849,10 @@ pub(crate) struct ReplicationSiteMetricsSnapshot {
pub(crate) proxy_head_failed: i64,
pub(crate) proxy_put_tag_total: i64,
pub(crate) proxy_put_tag_failed: i64,
pub(crate) proxy_get_tag_total: i64,
pub(crate) proxy_get_tag_failed: i64,
pub(crate) proxy_delete_tag_total: i64,
pub(crate) proxy_delete_tag_failed: i64,
pub(crate) replica_size: i64,
pub(crate) replica_count: i64,
}
@@ -1491,12 +1499,6 @@ pub(crate) async fn get_bucket_object_lock_config(
ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await
}
pub(crate) async fn get_bucket_replication_config(
bucket: &str,
) -> Result<(s3s::dto::ReplicationConfiguration, time::OffsetDateTime)> {
ecstore_bucket::metadata_sys::get_replication_config(bucket).await
}
pub(crate) async fn persist_force_delete_intent(
api: Arc<ECStore>,
entry: ecstore_bucket::replication::MrfReplicateEntry,
@@ -1842,18 +1844,6 @@ pub(crate) async fn find_local_disk_by_ref(disk_ref: &str) -> Option<DiskStore>
ecstore_storage::find_local_disk_by_ref(disk_ref).await
}
pub(crate) trait StorageReplicationConfigExt {
fn has_active_rules(&self, prefix: &str, recursive: bool) -> bool;
}
impl StorageReplicationConfigExt for s3s::dto::ReplicationConfiguration {
fn has_active_rules(&self, prefix: &str, recursive: bool) -> bool {
<s3s::dto::ReplicationConfiguration as ecstore_bucket::replication::ReplicationConfigurationExt>::has_active_rules(
self, prefix, recursive,
)
}
}
pub(crate) trait StorageVersioningConfigExt {
fn enabled(&self) -> bool;
}
+31 -5
View File
@@ -242,6 +242,7 @@ REPLICATION_RESYNC_CONTRACT_BACKSLIDE_HITS_FILE="${TMP_DIR}/replication_resync_c
REPLICATION_RESYNC_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_resync_boundary_bypass_hits.txt"
REPLICATION_OBJECT_DECISION_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_object_decision_boundary_bypass_hits.txt"
REPLICATION_OBJECT_COMPARE_CONTRACT_BACKSLIDE_HITS_FILE="${TMP_DIR}/replication_object_compare_contract_backslide_hits.txt"
REPLICATION_M2_DECISION_CONTRACT_BACKSLIDE_HITS_FILE="${TMP_DIR}/replication_m2_decision_contract_backslide_hits.txt"
REPLICATION_MRF_WIRE_FORMAT_BACKSLIDE_HITS_FILE="${TMP_DIR}/replication_mrf_wire_format_backslide_hits.txt"
STORAGE_REPLICATION_HANDLE_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/storage_replication_handle_boundary_bypass_hits.txt"
STORAGE_REPLICATION_CRATE_BYPASS_HITS_FILE="${TMP_DIR}/storage_replication_crate_bypass_hits.txt"
@@ -2981,7 +2982,7 @@ fi
(
cd "$ROOT_DIR"
replication_resync_status=0
rg -n --with-filename '^\s*(?:pub(?:\([^)]*\))?\s+)?(?:(?:struct|enum)\s+(?:ResyncOpts|TargetReplicationResyncStatus|BucketReplicationResyncStatus|ResyncStatusType)|fn\s+(?:resync_state_accepts_update|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch))\b' \
rg -n --with-filename '^\s*(?:pub(?:\([^)]*\))?\s+)?(?:(?:struct|enum)\s+(?:ResyncOpts|TargetReplicationResyncStatus|BucketReplicationResyncStatus|ResyncStatusType)|fn\s+(?:resync_state_accepts_update|resync_status_duration|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch))\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' >"$REPLICATION_RESYNC_CONTRACT_BACKSLIDE_HITS_FILE" || replication_resync_status=$?
if [[ "$replication_resync_status" -ne 0 && "$replication_resync_status" -ne 1 ]]; then
@@ -2996,7 +2997,7 @@ fi
(
cd "$ROOT_DIR"
replication_resync_boundary_status=0
rg -n --with-filename 'rustfs_replication::(resync::(RESYNC_META_FORMAT|RESYNC_META_VERSION|WIRE_ZERO_TIME_UNIX)|mrf::(MRF_META_FORMAT|MRF_META_VERSION)|(encode_resync_file|decode_resync_file|encode_mrf_file|decode_mrf_file|BucketReplicationResyncStatus|ResyncOpts|TargetReplicationResyncStatus|resync_state_accepts_update|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch)\b)' \
rg -n --with-filename 'rustfs_replication::(resync::(RESYNC_META_FORMAT|RESYNC_META_VERSION|WIRE_ZERO_TIME_UNIX)|mrf::(MRF_META_FORMAT|MRF_META_VERSION)|(encode_resync_file|decode_resync_file|encode_mrf_file|decode_mrf_file|BucketReplicationResyncStatus|ResyncOpts|TargetReplicationResyncStatus|resync_state_accepts_update|resync_status_duration|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch)\b)' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' \
--glob '!replication_resync_boundary.rs' >"$REPLICATION_RESYNC_BOUNDARY_BYPASS_HITS_FILE" || replication_resync_boundary_status=$?
@@ -3005,7 +3006,7 @@ fi
fi
replication_resync_boundary_grouped_status=0
rg -n -U --with-filename 'use\s+rustfs_replication::\{[^}]*\b(encode_resync_file|decode_resync_file|encode_mrf_file|decode_mrf_file|BucketReplicationResyncStatus|ResyncOpts|TargetReplicationResyncStatus|resync_state_accepts_update|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch)\b' \
rg -n -U --with-filename 'use\s+rustfs_replication::\{[^}]*\b(encode_resync_file|decode_resync_file|encode_mrf_file|decode_mrf_file|BucketReplicationResyncStatus|ResyncOpts|TargetReplicationResyncStatus|resync_state_accepts_update|resync_status_duration|should_count_head_proxy_failure|should_auto_resume_resync|is_version_id_mismatch)\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' \
--glob '!replication_resync_boundary.rs' >>"$REPLICATION_RESYNC_BOUNDARY_BYPASS_HITS_FILE" || replication_resync_boundary_grouped_status=$?
@@ -3021,7 +3022,7 @@ fi
(
cd "$ROOT_DIR"
replication_object_decision_boundary_status=0
rg -n --with-filename 'rustfs_replication::(MustReplicateOptions|ReplicationDeleteSource|ReplicationMultipartPartInput|ReplicationResyncTargetObject|delete_replication_missing_source_decision|delete_replication_object_opts|heal_uses_delete_replication_path|is_retryable_delete_replication_head_error|is_version_delete_replication|replication_etags_match|replication_multipart_complete_actual_size|replication_multipart_part_plan|resync_target_for_object|should_retry_delete_marker_purge)\b' \
rg -n --with-filename 'rustfs_replication::(MustReplicateOptions|ReplicationDeleteSource|ReplicationMultipartPartInput|ReplicationResyncTargetObject|delete_marker_purge_mrf_entry|delete_marker_purge_version_id|delete_replication_missing_source_decision|delete_replication_object_opts|heal_uses_delete_replication_path|is_retryable_delete_replication_head_error|is_version_delete_replication|replicate_delete_outcome|replication_etags_match|replication_multipart_complete_actual_size|replication_multipart_part_plan|resync_existing_delete_replication_info|resync_target_for_object|should_retry_delete_marker_purge|target_delete_version_id)\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' \
--glob '!replication_object_decision_boundary.rs' >"$REPLICATION_OBJECT_DECISION_BOUNDARY_BYPASS_HITS_FILE" || replication_object_decision_boundary_status=$?
@@ -3030,7 +3031,7 @@ fi
fi
replication_object_decision_boundary_grouped_status=0
rg -n -U --with-filename 'use\s+rustfs_replication::\{[^}]*\b(MustReplicateOptions|ReplicationDeleteSource|ReplicationMultipartPartInput|ReplicationResyncTargetObject|delete_replication_missing_source_decision|delete_replication_object_opts|heal_uses_delete_replication_path|is_retryable_delete_replication_head_error|is_version_delete_replication|replication_etags_match|replication_multipart_complete_actual_size|replication_multipart_part_plan|resync_target_for_object|should_retry_delete_marker_purge)\b' \
rg -n -U --with-filename 'use\s+rustfs_replication::\{[^}]*\b(MustReplicateOptions|ReplicationDeleteSource|ReplicationMultipartPartInput|ReplicationResyncTargetObject|delete_marker_purge_mrf_entry|delete_marker_purge_version_id|delete_replication_missing_source_decision|delete_replication_object_opts|heal_uses_delete_replication_path|is_retryable_delete_replication_head_error|is_version_delete_replication|replicate_delete_outcome|replication_etags_match|replication_multipart_complete_actual_size|replication_multipart_part_plan|resync_existing_delete_replication_info|resync_target_for_object|should_retry_delete_marker_purge|target_delete_version_id)\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' \
--glob '!replication_object_decision_boundary.rs' >>"$REPLICATION_OBJECT_DECISION_BOUNDARY_BYPASS_HITS_FILE" || replication_object_decision_boundary_grouped_status=$?
@@ -3058,6 +3059,25 @@ if [[ -s "$REPLICATION_OBJECT_COMPARE_CONTRACT_BACKSLIDE_HITS_FILE" ]]; then
report_failure "replication object comparison contracts must stay in crates/replication: $(paste -sd '; ' "$REPLICATION_OBJECT_COMPARE_CONTRACT_BACKSLIDE_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
replication_m2_decision_status=0
# M2-moved pure decision contracts must not be redefined in ECStore. The
# ssec_passthrough_evidence_present name is deliberately absent: ECStore
# keeps a thin HeadObjectOutput adapter under that name in
# replication_target_boundary.rs which delegates to the crate-owned pure fn.
rg -n --with-filename '^\s*(?:pub(?:\([^)]*\))?\s+)?(?:enum\s+(?:SsecPassthroughCapability|SsecPassthroughGate)|fn\s+(?:replicate_delete_outcome|target_delete_version_id|delete_marker_purge_version_id|delete_marker_purge_mrf_entry|resync_existing_delete_replication_info|version_identity_drifted|is_replication_target_offline_error|ssec_passthrough_gate))\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' >"$REPLICATION_M2_DECISION_CONTRACT_BACKSLIDE_HITS_FILE" || replication_m2_decision_status=$?
if [[ "$replication_m2_decision_status" -ne 0 && "$replication_m2_decision_status" -ne 1 ]]; then
exit "$replication_m2_decision_status"
fi
)
if [[ -s "$REPLICATION_M2_DECISION_CONTRACT_BACKSLIDE_HITS_FILE" ]]; then
report_failure "M2-moved replication decision contracts must stay in crates/replication: $(paste -sd '; ' "$REPLICATION_M2_DECISION_CONTRACT_BACKSLIDE_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
rg -n --with-filename 'rmp_serde::(to_vec_named|from_slice)|LittleEndian::(write_u16|read_u16)|const\s+MRF_META_(FORMAT|VERSION):\s+u16\s*=\s*1\b' \
@@ -3728,6 +3748,12 @@ fi
crates/ecstore/src/bucket/replication \
--glob '*.rs' |
rg -v '^crates/ecstore/src/bucket/replication/replication_target_boundary\.rs:' || true
# M2-moved target decision contracts (pure gate family, offline classifier,
# version identity drift) route through the target boundary only.
rg -n -U --with-filename 'rustfs_replication::\{[^;]*\b(?:SsecPassthroughCapability|SsecPassthroughGate|is_replication_target_offline_error|ssec_passthrough_gate|ssec_passthrough_evidence_present|version_identity_drifted)\b|rustfs_replication::(?:SsecPassthroughCapability|SsecPassthroughGate|is_replication_target_offline_error|ssec_passthrough_gate|ssec_passthrough_evidence_present|version_identity_drifted)\b' \
crates/ecstore/src/bucket/replication \
--glob '*.rs' |
rg -v '^crates/ecstore/src/bucket/replication/replication_target_boundary\.rs:' || true
) >"$REPLICATION_TARGET_BOUNDARY_BYPASS_HITS_FILE"
if [[ -s "$REPLICATION_TARGET_BOUNDARY_BYPASS_HITS_FILE" ]]; then
+9 -78
View File
@@ -1,30 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
# Guard: cryptographic capability wording must not over-claim what RustFS
# actually does. Two independent blocks, both anchored to the policy in
# docs/operations/kms-cryptographic-compliance.md:
#
# 1. Outward README and CHANGELOG material must not make an unsupported
# FIPS validation or certification claim. This block intentionally scans
# only the two public project-facing documents; the permitted qualifiers
# live in the policy document.
#
# 2. Nothing in crates/kms may describe the Vault KV2 backend as wrapping
# key material through Vault's Transit engine. `KmsBackend::VaultKv2`
# stores RustFS-wrapped key material in Vault's KV v2 engine and never
# calls Transit (see crates/kms/src/config.rs and
# docs/operations/kms-backend-security.md), so such prose tells operators
# their key material is cryptographically isolated inside Vault when it is
# not.
#
# Block 2 replaces the unit test `test_vault_kv2_sources_do_not_claim_transit_wrapping`
# that used to live in crates/kms/src/config.rs (rustfs/backlog#1884). The
# invariant is a documentation-claim invariant, so it has no behavioral twin by
# construction and belongs in a wording guard rather than in a test. The test
# could only see four `include_str!`-pinned files and stopped compiling —
# rather than reporting a violation — the moment one of them was renamed; this
# block scans every file in the crate and reports a rename explicitly.
# Guard: outward README and CHANGELOG material must not make an unsupported
# FIPS validation or certification claim. The detailed policy and permitted
# qualifiers live in docs/operations/kms-cryptographic-compliance.md; this
# check intentionally scans only the two public project-facing documents.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT_DIR="${CHECK_FIPS_WORDING_ROOT:-$(cd "${SCRIPT_DIR}/.." && pwd)}"
@@ -41,34 +21,12 @@ FORBIDDEN_PATTERNS=(
'(meets|satisfies)[[:space:]]+FIPS'
)
KMS_CRATE_DIR="crates/kms"
# The four files the retired unit test pinned with include_str!. They stay
# listed so that moving one out of crates/kms is reported here instead of
# silently shrinking the scan; the scan itself is not limited to them.
KMS_PINNED_SOURCES=(
"crates/kms/src/config.rs"
"crates/kms/src/api_types.rs"
"crates/kms/src/backends/vault.rs"
"crates/kms/src/lib.rs"
)
# Literal, case-sensitive, and byte-for-byte the needles the retired test built
# at runtime via format!("wrapping via {}", "Transit") and friends.
KMS_VAULT_KV2_FORBIDDEN=(
'wrapping via Transit'
'KV v2 + Transit'
'KV2+Transit'
"you would use Vault's transit engine"
)
fips_status=0
kms_status=0
status=0
for target in "${TARGETS[@]}"; do
if [[ ! -f "$target" ]]; then
printf 'FIPS wording guard failed: %s is missing\n' "$target" >&2
fips_status=1
status=1
continue
fi
@@ -77,41 +35,14 @@ for target in "${TARGETS[@]}"; do
if [[ -n "$matches" ]]; then
printf 'FIPS wording guard failed: forbidden pattern /%s/ in %s:\n%s\n' \
"$pattern" "$target" "$matches" >&2
fips_status=1
status=1
fi
done
done
for source in "${KMS_PINNED_SOURCES[@]}"; do
if [[ ! -f "$source" ]]; then
printf 'KMS wording guard failed: %s is missing; update KMS_PINNED_SOURCES in scripts/check_fips_wording.sh after moving it\n' \
"$source" >&2
kms_status=1
fi
done
if [[ -d "$KMS_CRATE_DIR" ]]; then
for pattern in "${KMS_VAULT_KV2_FORBIDDEN[@]}"; do
matches="$(grep -r -F -n -- "$pattern" "$KMS_CRATE_DIR" || true)"
if [[ -n "$matches" ]]; then
printf 'KMS wording guard failed: forbidden Vault KV2 claim "%s" in %s:\n%s\n' \
"$pattern" "$KMS_CRATE_DIR" "$matches" >&2
kms_status=1
fi
done
fi
if [[ "$fips_status" -ne 0 ]]; then
if [[ "$status" -ne 0 ]]; then
printf 'Remove unsupported FIPS validation wording from README.md or CHANGELOG.md.\n' >&2
fi
if [[ "$kms_status" -ne 0 ]]; then
printf 'The Vault KV2 backend does not wrap key material through Vault Transit; fix the wording in crates/kms.\n' >&2
fi
if [[ "$fips_status" -ne 0 || "$kms_status" -ne 0 ]]; then
exit 1
exit "$status"
fi
printf 'FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).\n'
printf 'KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).\n'
+1 -1
View File
@@ -25,7 +25,7 @@ cd "$(dirname "$0")/.."
# Baselines verified on 2026-08-11. Lower-only; see header.
# Excludes crates/e2e_test/ — test infrastructure legitimately uses s3s
# to verify S3 behavior and does not widen the production s3s surface.
S3S_IMPORT_FILES_BASELINE=211
S3S_IMPORT_FILES_BASELINE=213
S3_ERROR_LINES_BASELINE=1620
S3S_PATH_PATTERN='(^|[^"[:alnum:]_])s3s::'
E2E_TEST_GLOB='--glob=!crates/e2e_test/**'