Compare commits

..

30 Commits

Author SHA1 Message Date
overtrue a8022e60ba chore(ecstore): drop the disk dead_code blanket
Removing the blanket exposes 36 items in the lowest storage layer: 7 deleted, 29 kept with reasoned item-level allows. That is the smallest deletion share of this burn-down, and the reason is a verification limit rather than a judgement call.

disk/local.rs carries 141 `#[cfg(target_os = "linux")]` sites — the densest platform gating in the tree, because O_DIRECT and io_uring only exist there. The direct-I/O cluster (six ENV_RUSTFS_OBJECT_DIRECT_IO_* constants plus is_direct_io_read_enabled, is_direct_io_write_enabled, get_direct_io_read_threshold, direct_write_staging_capacity, direct_write_tail_split and DIRECT_WRITE_STAGING_BYTES) reads as dead on macOS purely because its production callers at local.rs:1766, 3114 and 4605 sit inside Linux-gated blocks. direct_write_staging_capacity even documents itself as "Platform-independent (no O_DIRECT), so it is unit-tested on any host".

Deleting those would leave every local check green — 4096 tests pass, clippy is clean, make pre-commit exits 0 — and break the Linux build in CI, because all four local lanes compile for aarch64-apple-darwin. Cross-checking locally is not available either: cargo check --target x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a Linux C cross-compiler. Their allows name the platform reason so the next reader on a non-Linux host does not repeat the investigation.

Deleted, all in files with no target_os gating at all (os.rs, disk_store.rs):

- HealthDiskCtxKey and HealthDiskCtxValue with its private log_success. Note that DiskHealthTracker::log_success is a different method of the same name and is live from cluster/rpc/peer_s3_client.rs and remote_disk.rs — the two have to be told apart by type, not by name.
- LocalDiskWrapper::new_with_health and check_id.
- os.rs file_exists and lock_destination_directory_for_path_access.

Kept with allows: DiskHealthTracker's set_faulty, mark_offline, waiting_count and last_success have test callers in remote_disk.rs, so they only look dead in the lib target. to_disk_error, remove_all and sync_dir_files are asserted by their own files' tests. The reclaim, mmap and path-cache field groups are written but never read back.

Placement follows the same rule as the earlier roots: per-method allows inside impl DiskHealthTracker and impl LocalDisk, since both are mostly live and a block-level allow would be a smaller version of the blanket this issue removes. Struct-level allows are used only where the warning covers that struct's own fields. The three cached_read_env! functions take their allow inside the macro invocation, before the fn line, because the macro forwards $(#[$meta:meta])* onto the generated item.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. The Linux lane is not covered locally and is left to CI.

Ref rustfs/backlog#1823 (step 2).
2026-08-16 11:53:10 +08:00
唐小鸭 e26668e62c fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition (#6128)
* test(ecstore): pin madmin-compatible ARN partition contract

Red-light evidence for backlog#1675 P1-7: madmin-go's ParseARN
hard-rejects any ARN that does not start with 'arn:minio:', while RustFS
generates and only accepts 'arn:rustfs:'. mc/madmin tooling therefore
cannot decode RustFS remote-target listings, and MinIO-era replication
configs are rejected as StaleTarget when re-registered. The new tests
pin the target contract (generate arn:minio:, parse both partitions,
reject unknown partitions) and fail against the current single-partition
gate.

* fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition

madmin-go's ParseARN hard-rejects any partition other than 'arn:minio:',
so native mc/madmin tooling could not decode RustFS remote-target
listings, and re-registering a MinIO-era replication config failed its
StaleTarget check against freshly minted arn:rustfs: targets
(backlog#1675 P1-7, route A).

- ARN Display now emits 'arn:minio:'; FromStr accepts a {minio, rustfs}
  partition whitelist (the legacy partition stays readable forever for
  persisted bucket-targets.json / replication configs). The whitelist is
  the only structural gate — BucketTargetType::from_str never fails —
  so it deliberately rejects foreign partitions such as arn:aws:.
- No data migration: every runtime match between targets, rules and
  stats keys is full-string equality, so existing arn:rustfs: targets
  keep matching their persisted rules; site replication already
  preserves MinIO-era ARNs on reconcile (pinned by existing tests).
- Rolling upgrade note: upgrade all cluster nodes before creating new
  remote targets — a not-yet-upgraded node rejects remove-remote-target
  for a freshly minted arn:minio: ARN with BucketRemoteArnInvalid.
- Out of scope: notification/SQS ARNs (crates/targets) keep the
  arn:rustfs:sqs: partition; they have their own compatibility story.
2026-08-16 10:28:45 +08:00
Henry Guo 8d3511c1b3 fix(heal): preserve timeout budget across retries (#6101)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-16 10:28:11 +08:00
Henry Guo d172d05e86 fix(ecstore): overlap metacache reader deadlines (#6098)
* fix(ecstore): overlap metacache reader deadlines

* fix(admin): avoid span guards across awaits

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-16 10:27:59 +08:00
Zhengchao An 0d86c50760 fix(ecstore): classify remote inline early-stop misses (#6136) 2026-08-16 09:50:16 +08:00
houseme 526d6f667e perf(ecstore): defer pending inline data shards (#6137) 2026-08-16 09:44:00 +08:00
唐小鸭 dcf3e4b9e8 fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold (#6129)
* test(replication): pin missing LWW timestamp header transport

Red-light tests for the replication timestamp three-header contract:

- put_object_headers_carry_replication_timestamp_headers pins that
  PutObjectOptions::header() must emit the
  x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
  headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
  and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
  pin that an authorized replication PUT / multipart complete must persist
  the inbound timestamps into the internal metadata keys while unauthorized
  requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
  (ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
  assertions can observe what a real target receives; self-test included.

* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold

Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.

- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
  timestamp from PutObjectOptions::header(); names and RFC3339 values
  interoperate with MinIO (minio-go constants.go, object-api-options.go),
  pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
  epoch values in header(), so "never modified" is not sent as a
  modification made now.
- Parse the headers only on authorized replication PUTs and multipart
  completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
  persist them into the dual-prefix internal metadata keys so the
  outbound pass (replication_target_boundary) reads the source's
  timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
  DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
  without it the sender only ever had the mod_time fallback to offer.

Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.

* fix(replication): load the stored tagging timestamp independently of remaining tags

Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.

* fix(storage): reserve replication transport names at metadata ingest

Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.

The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.

* fix(replication): harden timestamp replay

* fix(app): route retention helper through facade

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:56:04 +08:00
唐小鸭 04b9c8fd36 fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf (#6126)
* test(admin): pin madmin ReplicationMRF stream contract for /v3/replication/mrf

Red-light evidence for backlog#1675 P1-13 (mrf half): madmin's
BucketReplicationMRF decodes the response one ReplicationMRF document at
a time, so the current aggregate envelope decodes as a single phantom
row with an empty object in 'mc replicate backlog'. The new contract
tests assert the desired bare-document stream (exact madmin json tags,
empty body for an empty backlog) and fail against the current
render_mrf_backlog extraction, which preserves the envelope-only
behavior:

- mrf_stream_renders_bare_madmin_documents: envelope keys leak, no
  per-entry documents
- mrf_stream_renders_empty_body_for_no_entries: empty backlog still
  renders the envelope (phantom row)
- mrf_aggregate_envelope_retains_counters: PerObjectEntriesAvailable
  never advertises the enumerable stream

* fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf

The mrf endpoint returned a single aggregate envelope, which madmin's
json.Decoder loop decoded as one phantom row (empty object) in
'mc replicate backlog' (backlog#1675 P1-13, mrf half; the diff half was
fixed in #5799 and this mirrors its pattern).

- Default response is now a bare stream of ReplicationMRF documents
  (exact madmin json tags; Size/TargetARNs as ignored extension keys)
  built from the durable backlog ledger; an empty backlog renders an
  empty body, so mc shows zero rows instead of a phantom row.
- The aggregate counter envelope moves behind ?aggregate=true (RustFS
  extension) and now advertises PerObjectEntriesAvailable whenever the
  durable backlog is readable.
- An unreadable backlog is signalled out-of-band via
  x-rustfs-replication-mrf-backlog-unavailable (mirrors the diff
  truncation header) plus a warn event, since the bare stream cannot
  carry source health.
- The madmin node parameter is accepted but documented as a no-op: the
  durable ledger is cluster-shared with no per-node attribution.
- Delete-marker purge entries fall back to the marker version id so
  those rows keep a version identity.

* fix(admin): fail the mrf stream request when the durable ledger is unreadable

Review: madmin only decodes the body of a 200, so the out-of-band
unavailability header was invisible to it and an unreadable ledger read
as a clean zero-row backlog. Stream mode now returns 503; aggregate
mode keeps the availability fields.

* fix(admin): gate, bound, and null-map the mrf stream

Second review round:

- Authorization: the default stream enumerates object names and version
  ids, which a metrics-only principal must not see — it now requires
  admin:ReplicationDiff (MinIO parity, route policy updated);
  ?aggregate=true carries no object identities and keeps
  admin:GetReplicationMetrics.
- The nil UUID is RustFS's in-memory null-version sentinel and now
  leaves as the S3 wire token 'null' instead of a zero UUID (a
  pre-versioning object scanned after versioning + existing-object
  replication can persist it into the ledger).
- The durable ledger is not bounded by the in-memory pending cap and
  the body is buffered before send; the stream now stops at 10,000
  documents and signals truncation via
  x-rustfs-replication-mrf-truncated (mirroring the diff endpoint)
  plus a warn event, instead of staging an unbounded body.

* fix(admin): reject truncated MRF streams

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-08-16 05:55:15 +08:00
唐小鸭 c1f66969d7 fix(site-replication): merge incoming ILM expiry documents instead of overwriting (#6130)
* test(site-replication): pin ILM expiry merge contract for incoming lc-config

Red-light evidence for backlog#1675 P1-1: the lc-config receiver
overwrites the whole local lifecycle config with whatever the peer
sends (and deletes it wholesale on peer delete), so an expiry-only
document erases the receiver's local tier/transition rules, and peer
transition rules get installed across sites. The new tests pin the
MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening:

- incoming expiry documents merge with (never replace) local rules
- local transition sides are authoritative for same-id rules
- incoming transition fields are discarded at the trust boundary
- dropped expiry rules strip the expiry side but keep transitions;
  pure-expiry rules are removed
- delete merges with the empty set instead of dropping the config
- disabled rules survive; abort-mpu-only rules stay site-local
- deterministic order (idempotent re-delivery) and expiry_updated_at
  stamping for the staleness axis

All fail against the current overwrite implementation (identity
extraction of merge_incoming_lifecycle_config).

* fix(site-replication): merge incoming ILM expiry documents instead of overwriting

The lc-config receiver replaced the whole local lifecycle config with
the peer's document (and deleted it wholesale on peer delete), so an
expiry-only update erased the receiver's local tier/transition rules,
and a peer's transition rules were installed across sites
(backlog#1675 P1-1).

Receiver (apply_bucket_meta_item):
- lc-config now merges via merge_incoming_lifecycle_config, mirroring
  MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening:
  incoming transition fields are discarded outright; the local
  transition side of a same-id rule is authoritative. A peer delete
  merges with the empty set — pure-expiry rules go away, transition
  rules survive with their expiry side cleared, and only an empty
  result deletes the config file.
- Staleness moves to the expiry axis (config.expiry_updated_at):
  lifecycle_config_updated_at also moves on local transition-only
  edits, which shadowed newer peer expiry updates.
- Receiver-side replicateILMExpiry gate, symmetric with the sender
  hook (previously any peer could install expiry rules while the
  option was off).
- Rule order is deterministic (local order, incoming-new appended), so
  re-delivering the same document is byte-stable and does not rewrite
  bucket metadata per broadcast.

Sender:
- Both admin choke points — the bucket-meta hook and the SRInfo bucket
  entry feeding bootstrap/repair and consistency views — now emit only
  the expiry subset (transition fields stripped, non-expiry rules
  dropped). MinIO receivers install incoming rules verbatim, so
  transition rules must never leave the site. An unparseable local
  config is forwarded unfiltered rather than degraded to a delete.

Not covered here (follow-up): a two-site e2e with a real tier backend
to exercise transition-rule preservation end to end; receiver-side
validate_transition_tier for merged configs.

* fix(site-replication): close ILM merge review findings

Adversarial review of the lc-config merge surfaced four real defects,
all fixed here:

- Deletion tombstone regression: with the staleness axis moved to the
  in-config expiry_updated_at, a deleted lifecycle config fell back to
  UNIX_EPOCH and any delayed stale broadcast could resurrect deleted
  expiry rules. The axis now falls back to the whole-config write time
  (which survives deletion in bucket metadata as the deletion's lower
  bound), also covering legacy configs that predate the axis field.
- MinIO zero-rule documents: MinIO's delete tombstone / transition-only
  state marshals a lifecycle document with no <Rule>, which the strict
  s3s deserializer rejects — the receiver now recognizes it as the 'no
  expiry rules here' statement (delete semantics) instead of erroring
  on every MinIO heal pass.
- Inflated expiry axis at the sender: PutBucketLifecycle stamped
  expiry_updated_at unconditionally, so a transition-only edit advanced
  the axis and let this site's stale expiry subset shadow and roll back
  newer peer expiry edits fleet-wide. The stamp is now conditional
  (expiry subset present before or after the edit, MinIO parity), the
  hook item travels with the config's expiry axis (UNIX_EPOCH when the
  site has none), and the SRInfo bucket entry feeds bootstrap/repair
  the same axis instead of the whole-config write time.
- Del-marker parity: MinIO's CloneNonTransition never emits del-marker
  or abort-mpu fields, so treating del_marker_expiration as traveling
  expiry let a MinIO broadcast delete this site's del-marker-only
  rules. Both fields are now site-local on every edge: stripped from
  outbound subsets and inbound rules, restored from the local side on
  same-id merges, and never a deletion criterion.

Receiver-side validation of merged configs (object-lock / tier
constraints, MinIO runs finalLcCfg.Validate) remains a follow-up.

* fix(site-replication): close the second ILM review round

- Missed-delete repair: a deleted expiry state now travels through
  bootstrap/repair as an explicit timestamped lc-config delete item
  (lifecycle_expiry_statement distinguishes deletion — whole-config
  write time advanced past the created backfill — from never-configured
  buckets and from transition-only configs without an expiry axis,
  which say nothing). A peer that missed the live delete converges on
  repair; the receiver's staleness guard protects newer peer state.
- Strict tombstone recognition: only a well-delimited zero-rule
  <LifecycleConfiguration> document maps to delete semantics; truncated
  or foreign payloads that fail the strict deserializer are rejected
  instead of being treated as a delete that erases local expiry rules.
- Staleness fallback axis narrowed: the whole-config write time is used
  only for deleted or legacy-with-expiry state. A present
  transition-only config without an expiry axis compares at epoch — its
  whole-config time moves on transition edits and must not shadow or
  block independent peer expiry updates and same-timestamp repairs.

* fix(site-replication): validate tombstone children structurally

Second review round: a well-delimited root could still smuggle
malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt>
</LifecycleConfiguration> passed the no-<Rule check and was applied as
a delete. The tombstone body must now be a sequence of well-formed
simple children (matching open/close or self-closing, no nested markup,
no stray text, none named Rule); anything else surfaces InvalidRequest.
Malformed-child cases pinned in the recognition test.

* fix(site-replication): serialize lifecycle merges

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:34:17 +08:00
唐小鸭 cfa9276fad fix(admin): serialize replication metrics in minio-go wire shapes (#6127)
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics

Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2]
serializes the internal snake_case BucketStats family straight onto the
wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase
tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is
case-insensitive but does not ignore underscores, so 'mc replicate
status' shows all zeros without any error. The rewritten snapshot tests
assert the minio-go tags (plus a synthesized queueStats node — the
aggregation path leaves queue_stats.nodes empty today) and fail against
the current pass-through serialization.

* fix(admin): serialize replication metrics in minio-go wire shapes

?replication-metrics[=2] and the admin replicationmetrics endpoint
serialized the internal snake_case BucketStats family straight onto the
wire, so 'mc replicate status' decoded all zeros without any error
(backlog#1675 P1-11). The internal structs cannot be renamed: they are
the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in
node_service.rs), pinned by a new regression test.

- New admin/replication_metrics_wire.rs: Serialize-only projections onto
  minio-go replication.Metrics (v1 body, currStats) and MetricsV2
  (uptime/currStats/queueStats/downtimeInfo) with the exact json tags;
  per-target failed becomes the TimedErrStats envelope fed from the
  FailStats rolling window; the queue peak is dual-emitted as max
  (MinIO server tag) and peak (minio-go tag).
- queueStats synthesizes one node from the bucket queue snapshot — the
  aggregation path leaves queue_stats.nodes empty, and mc treats an
  empty node list as 'no data' — and carries transfer summaries
  (Large/Small/Total) derived from the per-target xfer rates.
- Both endpoints share the DTOs; source-health extension keys
  (provider_available/cluster_complete/...) ride along and are ignored
  by Go decoders.
- Widen the ecstore replication_stats_boundary re-exports
  (BucketReplicationStat/InQueueMetric/XferStats) so the admin facade
  chain can name the projected types.

* fix(replication): carry failure rolling windows through cluster aggregation

Review: both metrics endpoints aggregate first, and FailStats::merge
dropped the process-local samples (which also never cross the peer-RPC
wire — serde-skipped), so lastMinute/lastHour serialized as zero right
after a failure while totals was nonzero.

- FailStats gains serializable last_minute/last_hour window snapshots
  (serde default: old nodes read zeros, new fields are ignored by old
  decoders), recomputed on every add_size and re-stamped at the
  per-node collection point (get_latest_replication_stats), and summed
  by merge.
- The wire DTO takes the component-wise max of the live samples and the
  snapshot, so both the single-node and the aggregated path report the
  window.
- Regression test drives a stat through rmp round trip + merge before
  serialization, as requested.

Also restore the #[allow(dead_code)] attribute to route_policy — the
new module declaration had been inserted between the attribute and its
item, which broke the -D warnings CI lanes.

* fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path

Second review round:

- update_xfer_rate split at 1 MiB while the minio-go transferSummary
  labels (and RustFS's own worker-pool split) mean >= 128 MiB for
  Large, so a 2 MiB replication reported under Large with Small stuck
  at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2
  assertion covers 2 MiB / 127 MiB / exactly 128 MiB.
- add_size no longer recomputes the rolling windows: two full
  one-hour-deque scans per failure under the bucket-stats write lock
  made failure bursts quadratic (30k events ~2.1s). The windows are
  stamped only at the collection point (get_latest_replication_stats,
  which serves both the local leg and the peer RPC); the aggregation
  regression now drives that path explicitly before the RPC round trip
  and merge.

* fix(replication): average transfer summaries

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:27:23 +08:00
Henry Guo db8f55cb97 feat(table-catalog): finalize Iceberg REST behavior (#6072)
* feat(table-catalog): finalize Iceberg REST behavior

* fix(table-catalog): address REST finalization regressions

* test(table-catalog): expect REST commit conflicts

* test(table-catalog): avoid serialized view test deadlocks

* fix(table-catalog): adapt shared test backend

* fix(table-catalog): enforce Iceberg metadata invariants

* fix(table-catalog): preserve manifest length in test

* test(table-catalog): use valid metadata fixtures

* test(table-catalog): seed manifests before manifest lists

* fix(table-catalog): restore validation gates

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 03:05:09 +08:00
houseme 7f23a1ba91 feat(ecstore): report inline early-stop miss reasons (#6134)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-15 17:18:26 +00:00
Henry Guo 1619c4be60 fix(scanner): add context to corrupt metadata logs (#6099)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-15 21:32:05 +08:00
Zhengchao An 72fd7339c9 test(utils): allow ephemeral port reuse (#6122)
* test(utils): allow ephemeral port reuse

* test(kms): allow any ciphertext prefix
2026-08-15 08:32:10 +08:00
Zhengchao An 71e83aeec4 fix(ci): pin Docker images to release source (#6121) 2026-08-15 07:13:37 +08:00
唐小鸭 9138c24571 fix(site-replication): lift a rejoined site's restarted edit counter over stale marks (#6119)
fix(site-replication): lift a rejoined site's restarted edit counter over stale fence marks

A site removed while unreachable (unilateral removal: the receiver never
dropped it from its peer map, so parse_site_replication_state's load-time
mark pruning never fired) that later rejoins recreates its state object
and restarts edit_generation at zero. The receiver's surviving high-water
mark then silently fences out every stamped delivery from that origin —
peer edits and the add finalize fan-out alike are acked without applying
— until the restarted counter catches up.

Allocate the generation as a hybrid logical clock instead:
max(wall clock in unix nanoseconds, previous + 1), still inside the state
transaction under the distributed state-object lock. Every value a
lifetime hands out is capped by the wall clock at its own allocation, so
a recreated lifetime's first allocation exceeds them all and clears the
stale mark, while a pre-removal delivery still in flight stays below the
new floor and remains correctly fenced. previous+1 keeps allocations
strictly increasing across same-tick allocations and mid-lifetime clock
regressions.

Nothing changes on the wire or in the persisted schema: editGeneration
stays the single fence param and edit_generation the single counter
field, so pre-hybrid receivers get the fix as soon as the sender
upgrades, old binaries preserve the field across rolling up/downgrades,
and marks recorded by plain-counter receivers (small values) are cleared
by any wall-clock allocation. A clock that regresses across a
delete/recreate degrades to a fence that self-heals once real time
passes the previous lifetime's last allocation, and introduces no
rollback window beyond what the plain counter already had.

An epoch-based design (editEpoch wire param + per-origin epoch marks)
was built first and rejected under adversarial review: old binaries
rewriting the state object drop the unknown epoch fields, which both
disarms the fix mid-rolling-upgrade and — because epoch adoption lowers
the generation mark — reopens the pre-restart rollback the fence exists
to prevent; a backwards clock also fences an origin permanently instead
of self-healing. The hybrid clock has none of these modes.
2026-08-15 01:50:35 +08:00
Zhengchao An e9f5318027 chore(release): prepare 1.0.0-rc.2 2026-08-15 01:29:08 +08:00
Zhengchao An 69e8ef9af5 test(sse): align KMS context error assertion (#6118) 2026-08-15 00:44:36 +08:00
Henry Guo 0b2a46b36f fix(log-analyzer): remove stale heal manager anchor (#6100) 2026-08-14 23:49:31 +08:00
Zhengchao An 56509ead1f fix(ci): remove unused ecstore error conversion (#6117) 2026-08-14 23:15:08 +08:00
Zhengchao An ffe889ad59 fix(storage): restore multipart disk compression and make the legacy decompressor resumable (#6044)
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable

Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim.

Fixes #5957. Internal tracking: backlog#1848, backlog#1850.

* feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED

Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder.

* chore(compat): satisfy the cleanup-register guard for the multipart compression switch

The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition.

* chore(rio): drop a dead store in the poison guard and note the end-block branch

Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one.

* fix(s3): report empty compressed multipart part size

* fix(s3): report empty encrypted multipart part size
2026-08-14 22:14:26 +08:00
唐小鸭 e11ce2f132 fix(site-replication): route every state RMW through the locked transaction (#6097)
* fix(site-replication): route every state RMW through the locked transaction

P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the
follow-up promised by rustfs/rustfs#5882.

PR1 left ~26 read-modify-write call sites on
config/site-replication/state.json in the pre-transaction shape: a
process-local mutex around load / mutate / save, each IO taking its own
object lock. Nothing held a distributed lock across the whole sequence, so
two nodes of one site still lost each other's updates, and the transitional
mutex kept the old shape available to copy.

Every remaining RMW now runs inside update_site_replication_state;
read-only sites use load_site_replication_state, whose object read comes
with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner
helper are gone, together with their architecture-guard allowlist entry and
inventory row.

The multi-stage flows (add / edit / peer join / peer edit / remove / rotate)
keep their updated_at and pending-id CAS, but the CAS now runs inside the
transaction that writes, against the state that transaction loaded. Peer
probes, IAM work and fan-outs run between transactions and hold no lock at
all — the add no longer blocks every writer of the site across its peer join
round trips, and it re-checks the precondition right after the capability
probes so the common race is rejected before any IAM write or remote join.
When the add's commit CAS still fails, the error says the peers may already
be joined and that re-running the add reconverges. The add adopts only the
fields it computed (exhaustive destructure — adding a state field is a
compile error until classified); fields owned by writers that do not bump
updated_at keep their freshly loaded values.

Ordering of peer-edit deliveries now rests on the generation fence landed in
PR1 rather than on a guard that could never order two nodes: the add's
finalize fan-out carries the generation allocated in its commit. An accepted
peer join PRESERVES the applied-generation high-water marks — join fan-outs
are routine (adds and rotations both deliver SRPeerJoin to existing peers),
so wiping them would let stalled older edits land after any join; the
unilateral-removal rejoin misfence that a wipe would have patched is
pre-existing since the fence landed and needs an epoch in the fence instead.

The rotation handler now takes the lifecycle guard: the background
service-account reconciler runs its repair under a lifecycle try-acquire,
and its pending-rotation precheck is only sound if a rotation cannot start
mid-repair — an exclusion the removed process mutex used to provide as a
side effect.

update_site_replication_state_when_changed adds persist-or-skip so ack
markers and pending-clearing paths stop rewriting the object on a miss —
load-bearing, because the shared persist helper clears the whole object for
a ≤1-peer pending-free state — and save_site_replication_state is now
cfg(test): the pre-P1-15 shape can no longer be written in production code.

No on-disk format change.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (181 passed); site-replication
dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D
warnings; make pre-commit. Mutation checks: dropping the state-object lock
from the boundary reds the separate-node concurrency tests; flipping a
persist-or-skip miss to a persist reds
test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by
three independent adversarial passes (correctness/concurrency,
security/compatibility, simplicity/test-coverage); their confirmed findings
are folded in.

* fix(site-replication): serialize peer-join admission around its IAM write

Review follow-up (overtrue): two joins accepted by the same node could
interleave as "A checks a stale snapshot and pauses reading its body, B
applies secret B and commits, A resumes, overwrites IAM with secret A, and
A's commit is refused as superseded" — the persisted state advertised B's
contract while IAM only accepted A's secret, failing every peer
control-plane call. The pre-P1-15 process mutex serialized same-node joins
end to end; removing it dropped that exclusion.

admit_peer_join now runs the staleness check, the IAM upsert and the state
commit under the lifecycle guard, with the authoritative pre-check taken
against a load under that guard BEFORE IAM changes anything. The closing
transaction still re-checks staleness: the guard is process-local (exactly
as far as the old mutex reached) and the state-object lock arbitrates joins
accepted by different nodes. The body is fully read before the guard so a
stalling sender cannot block add/remove/rotate/reconciler.

The IAM step is injected, and the gated-body regression test reproduces the
review's ordering: join A is held mid-IAM while a newer join B arrives; B
must wait at the guard, and both IAM order and the final persisted state end
on B. Mutation-verified: removing the lifecycle guard from admit_peer_join
turns the test red.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (182 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.

* fix(site-replication): fence peer-join admission across nodes

Review follow-up (overtrue, round 2): the lifecycle guard only serializes
joins within one process. Node A could pass the staleness check for an
older T1, node B write secret B to IAM and commit a newer T2, and node A
then overwrite IAM with secret A while its own state commit is refused as
superseded — state advertising T2's contract while IAM only accepts A's
secret.

The admission (staleness check -> IAM upsert -> state commit) now also runs
under a distributed join-admission lock, a namespace-lock key with no
backing object, following the repair execution lock's pattern — including
its nesting of config-object locks (admission -> state), and delegating
crash safety to the lock subsystem's lease expiry instead of a hand-rolled
TTL. The staleness check runs against a load taken inside the lock, before
IAM changes anything, so a superseded join exits without touching IAM. The
closing transaction keeps its re-check for defence in depth and for
old-version nodes that do not take the admission lock during a rolling
upgrade (that mixed-version window keeps today's behavior and closes when
the upgrade completes).

admit_peer_join_across_nodes is the admission minus the process-local
lifecycle guard — exactly what a second node runs — and the new
separate-nodes regression test drives it directly with join A gated
mid-IAM: join B must wait at the distributed lock, and both the IAM write
order and the final persisted state end on B. Mutation-verified: removing
the admission lock turns the test red while the same-node test (which
drives the full admit_peer_join) stays green.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (183 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.
2026-08-14 22:13:37 +08:00
cxymds eca6bc1600 fix(ecstore): preserve CopyObject producer errors (#6090)
* fix(ecstore): preserve CopyObject producer errors

* fix(app): resume preserved relocation I/O errors

* fix(copy): preserve transformed source errors
2026-08-14 14:12:42 +00:00
cxymds 85be26b3c1 test(ecstore): cover cancelled PUT tmp cleanup (#6105) 2026-08-14 14:07:44 +00:00
Zhengchao An ebbcfa3ac2 fix(tier): decrypt transitioned objects instead of serving their ciphertext (#6107)
* fix(tier): decrypt transitioned objects instead of serving their ciphertext

A GET on a managed-SSE object that lifecycle had transitioned to a remote tier returned the ciphertext with the plaintext's Content-Length and no error: silent corruption on read-through, and worse than a failed request because nothing signals it. Restore of the same object failed server-side with IncompleteBody while POST ?restore still answered 200, so the object simply never came back and HEAD never showed an x-amz-restore marker.

Both symptoms are one cause. The transitioned read path built its fetch through new_getobjectreader, which decides nothing about encryption: it derived the range from the parts table — whose sizes are PLAINTEXT sizes — then used that range to fetch the object's STORED bytes from the tier, and handed the stream to the caller without any decrypt transform. The GET therefore served the first plaintext-length bytes of ciphertext; the restore copy-back, which validates against the stored size, came up short by exactly the encryption overhead.

The path now builds the same ReadPlan the local read path uses, so a single place decides how stored bytes map to requested bytes. ReadPlan gains a two-phase API — build_for_request to learn the storage coordinates before issuing the tier fetch, into_object_reader to wrap the returned stream — because the tier fetch has to be positioned before a stream exists. The encryption resolver reaches the path from InstanceContext, the same source the local read uses.

A restore read additionally stops synthesizing a range from the part number. A restore serves the stored representation (restore_request_active already forces the Plain branch), so a plaintext-coordinate range would be reinterpreted as a storage range and truncate the payload by its encoding overhead. An explicit caller range is already in storage coordinates on that path and is still honored, which two existing tests pin.

crates/e2e_test/src/kms/kms_ilm_sse_kms_test.rs drops its #[ignore]: the transition test now runs and asserts the plaintext round-trips byte-identically through transition, read-through and restore. The same file had its enforcement switch stuck at false from a control experiment; it is back to true, so the test again exercises what its name and module docs claim.

Fixes #6025. Refs rustfs/backlog#1582, rustfs/backlog#1637.

* test(tier): pass resolver to transitioned reader tests
2026-08-14 21:59:38 +08:00
Zhengchao An ebd0531124 chore(ecstore): drop the data_usage dead_code blanket (#6089)
Removing the blanket exposes eighteen items. Six are deleted; the rest belong to one feature that was never wired up.

crates/ecstore/src/data_usage/local_snapshot.rs and its aggregation entry point, aggregate_local_snapshots, form a complete per-disk usage-snapshot feature: read/write of snapshot files under the metadata bucket, cross-disk aggregation, and tests. Nothing calls it. It landed in #5307 on 2026-07-27 and git log -S over the whole history shows the entry point has never had a caller; the live data-usage path is load_data_usage_from_backend / store_data_usage_in_backend. Rather than delete a recent, tested feature on cleanup grounds, the module gets a header explaining the situation and its items carry individual allows, so the gap stays greppable without reintroducing a blanket. One commit flips this to deletion if that is preferred.

Deleted:

- DATA_USAGE_BLOOM_NAME together with DATA_USAGE_BLOOM_NAME_PATH. Both are a dead duplicate of the pair in crates/scanner/src/data_usage_define.rs, which has roughly fifty consumers across scanner.rs and remote_scanner.rs; the ecstore copies have none.
- increment_bucket_usage_memory and decrement_bucket_usage_memory, thin wrappers over the live record_bucket_object_write_memory and record_bucket_object_delete_memory.
- sync_memory_cache_with_backend, whose doc comment says it is "called by scanner" — nothing calls it.
- create_cache_entry_from_summary and cache_to_data_usage_info, neither with a consumer in any lane.

resolve_loaded_snapshot is kept with an allow: nine tests in the same file assert its primary/backup fallback.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. rustfs-scanner still compiles clean. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).
2026-08-14 21:57:29 +08:00
Zhengchao An 4421d4829f test(table-catalog): share the store test doubles and fold three commit-rejection cases (#6076)
Completes PR3 of the issue.

Store doubles: NoopTableCatalogStore (193 lines, a pure stub answering "nothing here") and TestTableCatalogStore (416 lines, a stateful fake with commit pauses and failure injection) move into test_support.rs, along with TestCatalogPublishPause which the latter needs. Per the issue's ruling both shapes are kept — they are different tools, not duplicates of each other. Being honest about the benefit: this does not reduce the number of TableCatalogStore implementations, it puts both in one file so a trait change is one file to edit instead of two.

row_level_conflict fold: rejects_stale_new_manifest_sequence, rejects_stale_added_entry_sequence, and rejects_historical_change_in_new_manifest were identical apart from four literals (manifest-list sequence, data-file name, manifest-entry snapshot id, failure message). They become one table-driven test with three rows, each row keeping its original values, and every assertion carries the case name.

Verification: cargo test -p rustfs --lib table_catalog 479 passed and --lib admin::handlers::table_catalog 165 passed (both down exactly 2 from the 3->1 fold; the store filter is a substring match that also covers the admin tests); clippy --lib --tests -D warnings clean; make pre-commit green.

Ref rustfs/backlog#1837 (PR3).
2026-08-14 21:56:20 +08:00
Zhengchao An d6c62b9601 chore(ecstore): drop the services dead_code blanket (#6103)
Removing the blanket exposes twenty-five items across tier, notification and rebalance. Only eight are deleted — the lowest ratio of this burn-down so far, and the reason is that these subsystems carry heavy test coverage, so the blanket was mostly hiding test-only seams rather than dead weight.

Deleted:

- crates/ecstore/src/services/tier/warm_backend_s3sdk.rs entirely (200 lines). Its WarmBackendS3 is never constructed; the type of the same name in warm_backend_s3.rs is the live one, wrapped by the Azure backend. Two implementations of one S3 warm backend, one of them never wired.
- TierConfigMgr::begin_publish_transition and publish_candidate_inner, thin wrappers whose _with_allowed_mutation_blocks siblings carry every real caller, plus retire_driver.
- The GCS backend's MAX_MULTIPART_PUT_OBJECT_SIZE, MAX_PARTS_COUNT and MIN_PART_SIZE, and its write-only storage_class field.
- mark_started_rebalance_pools_stopped and the RStats alias.

Two deletions were withdrawn after a per-name grep, both because of an inference rather than a check:

AsyncBatchProcessor::new was deleted on the strength of grepping only BATCH_PROCESSOR_OPERATION_CUSTOM, whose two hits are its definition and its use inside new. That looked like a self-contained dead pair; new in fact has seven test callers. The warning listed both items, and only one of them was actually checked.

Deleting the two dead publish wrappers then revealed a second layer — publish_candidate_owned, remove_and_save_with, clear_and_save_with, save_tiering_config_if_current. These are not dead: publish_candidate, their caller, is #[cfg(test)], so a callee that lives in the main body has no caller in the lib build and a live one in the test build. rustc reports the roots of a dead subgraph, and the next layer down can have a different character, so each layer needs its own grep.

Kept with allows: the tier mutation-intent record helpers (asserted by store::init tests), affected_targets, tier_object_blocks_target_rebind, the rebalance snapshot and retry-wait helpers, notification_sys's tier_config_reload_worker_active and call_peer_with_timeout, and active_operation_lease_count, whose only caller sits behind #[cfg(feature = "test-util")].

Also kept, with a module note rather than removal: the ecstore-side EventNotifier. All four of its methods are unreachable and init_bucket_targets logs that it is a no-op in this build; the working stack is rustfs-notify, whose own EventNotifier drives bucket configuration. Removing it means also retiring the InstanceContext slot that holds it (backlog#939 Phase 5), which belongs in its own PR.

Worth a separate issue: MAX_MULTIPART_PUT_OBJECT_SIZE, MAX_PARTS_COUNT and MIN_PART_SIZE are declared independently in eight warm-backend files plus client/constants.rs. Only the GCS copies were dead; the other seven backends each use their own.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).
2026-08-14 21:56:11 +08:00
cxymds d91086d094 test(scanner): refresh metadata after fixture mutation (#6113) 2026-08-14 13:37:51 +00:00
Zhengchao An 69719c257e chore(ecstore): remove the pool-level ListObjects pagination copy (#6078)
* chore(ecstore): remove the pool-level ListObjects pagination copy

The ListObjects pagination pipeline existed in three near-copies in one file; production listing never reaches the Sets copy, which ECStore bypasses by expanding straight to per-set disks. This removes it: impl ListOperations for Sets (61 lines of pure forwarding in core/sets.rs) and the impl Sets pagination block (826 lines of inner_list_objects_v2 / list_objects_generic / inner_list_object_versions / list_path / list_merged / walk_internal in store/list_objects.rs).

Two preconditions verified before deleting rather than taken on faith: the architecture guard pins only set_disks_implements_storage_list_operations_contract, so nothing requires the Sets trait impl; and the four Sets pagination methods had no cross-file caller besides that trait impl.

The single test consumer moves to the surviving pipeline instead of being deleted: writes still go through the pool, and the listing assertion now targets the set-level implementation. It is renamed accordingly so the name still describes what it covers.

The logging guardrail's TRACE-only requirement for Sets::list_objects_v2 retires in the same diff — the wrapper it pinned no longer exists. The ECStore and SetDisks entries are untouched.

The SetDisks copy stays for now: its trait impl is guard-pinned, so replacing the duplicate pipeline behind it needs the generic helper the issue schedules for post-1.0.

Verification: cargo nextest run -p rustfs-ecstore 4020 passed; check_architecture_migration_rules.sh and check_logging_guardrails.sh pass; clippy --lib --tests -D warnings clean; make pre-commit green.

Ref rustfs/backlog#1821 (PR1).

* chore(ecstore): fold the ListObjects forwarders into the ECStore impl

store/list.rs held two thin forwarders, handle_list_objects_v2 and handle_list_object_versions, that only re-entered the inner_* implementations. The ListOperations impl now calls those directly and the file goes away.

The logging guardrail's trace_hot_spans list pinned handle_list_objects_v2 as TRACE-only; that entry is retired in the same diff, adjacent to the sets.rs entry retired by the preceding commit.

Ref rustfs/backlog#1821.

* chore(ecstore): drop the type aliases orphaned by the pagination removal

core/sets.rs declared four local type aliases — ListObjectsV2Info, ListObjectVersionsInfo, ObjectInfoOrErr and WalkOptions — used only by the pool-level pagination pipeline removed earlier in this branch. store/list_objects.rs keeps its own live copies of the same aliases.

They only surface now that #6087 removed the core module's dead_code blanket: on that older base each PR was warning-free on its own, and the combination is what exposes them. Their storage_api_contracts imports go with them.

Ref rustfs/backlog#1823, rustfs/backlog#1821.

* fix(ecstore): preserve Sets listing compatibility
2026-08-14 13:19:06 +00:00
121 changed files with 16771 additions and 4233 deletions
+7 -1
View File
@@ -252,10 +252,16 @@ test-group = 'ecstore-serial-flaky'
# cluster, so it keeps the lane's parallel-safe / no-external-dependency
# properties. The RustFS warm backend has no loopback guard (that guard is
# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
#
# Disk compression (backlog#1848): the `compression` module joins the smoke
# lane so the multipart disk-compression roundtrips (restored after
# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
# Single-node servers on random ports with isolated temp dirs — meets the
# admission criteria unchanged.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
+25 -1
View File
@@ -94,6 +94,7 @@ jobs:
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -118,6 +119,7 @@ jobs:
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
@@ -137,6 +139,7 @@ jobs:
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
@@ -261,6 +264,23 @@ jobs:
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
@@ -271,6 +291,7 @@ jobs:
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
@@ -281,6 +302,7 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
@@ -308,6 +330,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -397,7 +420,8 @@ jobs:
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
Generated
+51 -48
View File
@@ -278,6 +278,7 @@ checksum = "312c1ea69e5fe9966e0029fb95aca8790100b85aff4f0d3b00a9337c74069a9c"
dependencies = [
"bigdecimal",
"bon",
"crc32fast",
"digest 0.11.3",
"log",
"miniz_oxide 0.9.1",
@@ -289,9 +290,11 @@ dependencies = [
"serde",
"serde_bytes",
"serde_json",
"snap",
"strum",
"thiserror 2.0.20",
"uuid",
"zstd",
]
[[package]]
@@ -3761,7 +3764,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_test"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -9090,7 +9093,7 @@ dependencies = [
[[package]]
name = "rustfs"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9200,6 +9203,7 @@ dependencies = [
"serial_test",
"sha2 0.11.0",
"shadow-rs",
"snap",
"socket2",
"subtle",
"sysinfo",
@@ -9227,7 +9231,7 @@ dependencies = [
[[package]]
name = "rustfs-audit"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"const-str",
@@ -9250,7 +9254,7 @@ dependencies = [
[[package]]
name = "rustfs-checksums"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"base64-simd",
"bytes",
@@ -9266,7 +9270,7 @@ dependencies = [
[[package]]
name = "rustfs-common"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"chrono",
"hotpath",
@@ -9284,7 +9288,7 @@ dependencies = [
[[package]]
name = "rustfs-concurrency"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"insta",
@@ -9297,7 +9301,7 @@ dependencies = [
[[package]]
name = "rustfs-config"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"const-str",
"hotpath",
@@ -9307,7 +9311,7 @@ dependencies = [
[[package]]
name = "rustfs-credentials"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"base64-simd",
"hmac 0.13.0",
@@ -9321,7 +9325,7 @@ dependencies = [
[[package]]
name = "rustfs-crypto"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"aes-gcm",
"argon2",
@@ -9342,7 +9346,7 @@ dependencies = [
[[package]]
name = "rustfs-data-usage"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"rmp-serde",
@@ -9352,7 +9356,7 @@ dependencies = [
[[package]]
name = "rustfs-ecstore"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"async-channel",
@@ -9459,7 +9463,6 @@ dependencies = [
"tokio-stream",
"tokio-util",
"tonic",
"tonic-prost",
"tower",
"tracing",
"tracing-core",
@@ -9492,7 +9495,7 @@ dependencies = [
[[package]]
name = "rustfs-extension-schema"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"serde",
@@ -9502,7 +9505,7 @@ dependencies = [
[[package]]
name = "rustfs-filemeta"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"byteorder",
@@ -9529,7 +9532,7 @@ dependencies = [
[[package]]
name = "rustfs-heal"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"base64 0.23.1",
@@ -9560,7 +9563,7 @@ dependencies = [
[[package]]
name = "rustfs-iam"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"async-trait",
@@ -9601,7 +9604,7 @@ dependencies = [
[[package]]
name = "rustfs-io-core"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"bytes",
"hotpath",
@@ -9614,7 +9617,7 @@ dependencies = [
[[package]]
name = "rustfs-io-metrics"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"criterion",
"hotpath",
@@ -9678,7 +9681,7 @@ dependencies = [
[[package]]
name = "rustfs-keystone"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"bytes",
"futures",
@@ -9705,7 +9708,7 @@ dependencies = [
[[package]]
name = "rustfs-kms"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9754,7 +9757,7 @@ dependencies = [
[[package]]
name = "rustfs-lifecycle"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"hotpath",
@@ -9777,7 +9780,7 @@ dependencies = [
[[package]]
name = "rustfs-lock"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"compact_str",
@@ -9800,7 +9803,7 @@ dependencies = [
[[package]]
name = "rustfs-log-analyzer"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"chrono",
"flate2",
@@ -9819,7 +9822,7 @@ dependencies = [
[[package]]
name = "rustfs-madmin"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"humantime",
@@ -9834,7 +9837,7 @@ dependencies = [
[[package]]
name = "rustfs-notify"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"async-trait",
@@ -9869,7 +9872,7 @@ dependencies = [
[[package]]
name = "rustfs-object-capacity"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"criterion",
"futures",
@@ -9889,7 +9892,7 @@ dependencies = [
[[package]]
name = "rustfs-object-data-cache"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"bytes",
"criterion",
@@ -9906,7 +9909,7 @@ dependencies = [
[[package]]
name = "rustfs-obs"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"chrono",
"crossbeam-channel",
@@ -9961,7 +9964,7 @@ dependencies = [
[[package]]
name = "rustfs-policy"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"base64-simd",
@@ -9992,7 +9995,7 @@ dependencies = [
[[package]]
name = "rustfs-protocols"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"astral-tokio-tar",
"async-compression",
@@ -10054,7 +10057,7 @@ dependencies = [
[[package]]
name = "rustfs-protos"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"flatbuffers",
"hotpath",
@@ -10078,7 +10081,7 @@ dependencies = [
[[package]]
name = "rustfs-replication"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"byteorder",
"bytes",
@@ -10096,7 +10099,7 @@ dependencies = [
[[package]]
name = "rustfs-rio"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"aes-gcm",
"arc-swap",
@@ -10134,7 +10137,7 @@ dependencies = [
[[package]]
name = "rustfs-rio-v2"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"aes-gcm",
"bytes",
@@ -10157,7 +10160,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-ops"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"rustfs-s3-types",
@@ -10165,7 +10168,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-types"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"serde",
@@ -10174,7 +10177,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-api"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"bytes",
@@ -10204,7 +10207,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-query"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-recursion",
"async-trait",
@@ -10223,7 +10226,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"bytes",
@@ -10263,7 +10266,7 @@ dependencies = [
[[package]]
name = "rustfs-security-governance"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"thiserror 2.0.20",
@@ -10271,7 +10274,7 @@ dependencies = [
[[package]]
name = "rustfs-signer"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"base64-simd",
"bytes",
@@ -10289,7 +10292,7 @@ dependencies = [
[[package]]
name = "rustfs-storage-api"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"hotpath",
@@ -10304,7 +10307,7 @@ dependencies = [
[[package]]
name = "rustfs-targets"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"async-nats",
@@ -10358,7 +10361,7 @@ dependencies = [
[[package]]
name = "rustfs-test-utils"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"rustfs-data-usage",
@@ -10374,7 +10377,7 @@ dependencies = [
[[package]]
name = "rustfs-tls-runtime"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"arc-swap",
"hotpath",
@@ -10395,7 +10398,7 @@ dependencies = [
[[package]]
name = "rustfs-trusted-proxies"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"axum",
@@ -10432,7 +10435,7 @@ dependencies = [
[[package]]
name = "rustfs-utils"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"base64-simd",
"blake2",
@@ -10474,7 +10477,7 @@ dependencies = [
[[package]]
name = "rustfs-zip"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
dependencies = [
"astral-tokio-tar",
"async-compression",
+48 -48
View File
@@ -69,7 +69,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.97.1"
version = "1.0.0-rc.1"
version = "1.0.0-rc.2"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,52 +86,52 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.0-rc.1" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.1" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.1" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.1" }
rustfs-common = { path = "crates/common", version = "1.0.0-rc.1" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.1" }
rustfs-config = { path = "./crates/config", version = "1.0.0-rc.1" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.1" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.1" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.1" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.1" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.1" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.1" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.1" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.1" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.1" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.1" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.1" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.1" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.1" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.1" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.1" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.1", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.1" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.1" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.1" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.1" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.1" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.1" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.1" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.1" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.1" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.1" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.1" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.1" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.1" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.1" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.1" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.1" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.1" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.1" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.1" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.1" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.1" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.1" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.1" }
rustfs = { path = "./rustfs", version = "1.0.0-rc.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.2" }
rustfs-common = { path = "crates/common", version = "1.0.0-rc.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.2" }
rustfs-config = { path = "./crates/config", version = "1.0.0-rc.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.2" }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
tower-http = { version = "0.7.0" }
# Serialization and Data Formats
apache-avro = "0.22.0"
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
bytesize = "2.7.0"
byteorder = "1.5.0"
+1 -1
View File
@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.2
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
+1 -1
View File
@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.2
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
+4 -1
View File
@@ -67,7 +67,10 @@ fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
}
fn capture_command_logs(command: &mut Command, log_path: Option<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
pub(crate) fn capture_command_logs(
command: &mut Command,
log_path: Option<&str>,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let Some(log_path) = log_path else {
return Ok(());
};
+663 -3
View File
@@ -2,6 +2,7 @@
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use serial_test::serial;
use std::fs;
use std::path::PathBuf;
@@ -25,6 +26,15 @@ fn generate_compressible_data(size: usize) -> Vec<u8> {
data
}
/// Deterministic 2048-byte-period binary pattern that compresses extremely well: every part
/// yields many compressed blocks, which is exactly the shape that reproduced the mid-payload
/// Pending truncation (rustfs/rustfs#5957).
fn generate_high_ratio_binary_data(size: usize, seed: u8) -> Vec<u8> {
(0..size)
.map(|i| ((i as u64).wrapping_mul(2_654_435_761).wrapping_add(seed as u64) >> 3) as u8)
.collect()
}
fn find_part_files(temp_dir: &str, bucket: &str, object_key: &str) -> Vec<PathBuf> {
let bucket_path = PathBuf::from(temp_dir).join(bucket);
let mut part_files = Vec::new();
@@ -55,9 +65,14 @@ async fn start_rustfs_with_compression(env: &mut RustFSTestEnvironment) -> Resul
env.cleanup_existing_processes().await?;
let binary_path = rustfs_binary_path();
let process = Command::new(&binary_path)
// Route the child's stdout/stderr through the shared RUSTFS_E2E_LOG_DIR
// capture (survives the temp-dir cleanup on Drop and is uploaded as a CI
// artifact); without the env var the child inherits stdio as before.
let mut command = Command::new(&binary_path);
command
.env("RUSTFS_CONSOLE_ENABLE", "false")
.env("RUSTFS_COMPRESSION_ENABLED", "true")
.env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true")
.args([
"--address",
&env.address,
@@ -66,8 +81,9 @@ async fn start_rustfs_with_compression(env: &mut RustFSTestEnvironment) -> Resul
"--secret-key",
&env.secret_key,
&env.temp_dir,
])
.spawn()?;
]);
crate::common::capture_command_logs(&mut command, env.capture_log_path.as_deref())?;
let process = command.spawn()?;
env.process = Some(process);
@@ -154,3 +170,647 @@ async fn test_compression_roundtrip() -> Result<(), Box<dyn std::error::Error +
env.stop_server();
Ok(())
}
const MULTIPART_COMPRESSION_BUCKET: &str = "compression-multipart-bucket";
const MPU_PART1_SIZE: usize = 5 * 1024 * 1024;
const MPU_PART2_SIZE: usize = 1024 * 1024;
async fn multipart_upload(
client: &aws_sdk_s3::Client,
bucket: &str,
key: &str,
parts: &[&[u8]],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let create = client.create_multipart_upload().bucket(bucket).key(key).send().await?;
let upload_id = create.upload_id().ok_or("missing upload id")?.to_string();
let mut completed_parts = Vec::with_capacity(parts.len());
for (i, part) in parts.iter().enumerate() {
let part_number = (i + 1) as i32;
let upload = client
.upload_part()
.bucket(bucket)
.key(key)
.upload_id(&upload_id)
.part_number(part_number)
.body(ByteStream::from(part.to_vec()))
.send()
.await?;
completed_parts.push(
CompletedPart::builder()
.part_number(part_number)
.e_tag(upload.e_tag().unwrap_or_default())
.build(),
);
}
client
.complete_multipart_upload()
.bucket(bucket)
.key(key)
.upload_id(&upload_id)
.multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
.send()
.await?;
Ok(())
}
async fn fetch_range(
client: &aws_sdk_s3::Client,
bucket: &str,
key: &str,
range: &str,
) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
let response = client.get_object().bucket(bucket).key(key).range(range).send().await?;
Ok(response.body.collect().await?.into_bytes().to_vec())
}
/// Multipart disk compression roundtrip: parts are written as independent
/// compressed streams and every GET shape must reassemble the original bytes
/// (rustfs/rustfs#5957: multipart uploads previously bypassed disk compression
/// entirely).
#[tokio::test]
#[serial]
async fn test_compression_multipart_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart compression roundtrip test");
let mut env = RustFSTestEnvironment::new().await?;
start_rustfs_with_compression(&mut env).await?;
let client = env.create_s3_client();
env.create_test_bucket(MULTIPART_COMPRESSION_BUCKET).await?;
let object_key = "multipart-compressible.txt";
let part1 = generate_compressible_data(MPU_PART1_SIZE);
let part2 = generate_compressible_data(MPU_PART2_SIZE);
let mut original_data = part1.clone();
original_data.extend_from_slice(&part2);
let total_size = original_data.len();
multipart_upload(&client, MULTIPART_COMPRESSION_BUCKET, object_key, &[&part1, &part2]).await?;
let head_response = client
.head_object()
.bucket(MULTIPART_COMPRESSION_BUCKET)
.key(object_key)
.send()
.await?;
assert_eq!(
head_response.content_length().unwrap_or(0) as usize,
total_size,
"Content-Length should be the logical object size"
);
let part_files = find_part_files(&env.temp_dir, MULTIPART_COMPRESSION_BUCKET, object_key);
assert!(!part_files.is_empty(), "expected on-disk part files for the multipart object");
let total_physical_size: u64 = part_files.iter().filter_map(|p| fs::metadata(p).ok()).map(|m| m.len()).sum();
assert!(
total_physical_size < (total_size / 2) as u64,
"Physical size {total_physical_size} should be well below original size {total_size} (multipart compression applied)"
);
info!("Multipart physical storage size: {total_physical_size} bytes (compressed from {total_size} bytes)");
// Full GET must reassemble both independently compressed parts.
let get_response = client
.get_object()
.bucket(MULTIPART_COMPRESSION_BUCKET)
.key(object_key)
.send()
.await?;
let downloaded = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded.len(), total_size);
assert_eq!(&downloaded[..], &original_data[..], "full GET data mismatch");
// Range fully inside part 1.
let range_inside_part1 = fetch_range(&client, MULTIPART_COMPRESSION_BUCKET, object_key, "bytes=1024-999423").await?;
assert_eq!(&range_inside_part1[..], &original_data[1024..999424], "part-1 range mismatch");
// Range crossing the part boundary.
let boundary_start = MPU_PART1_SIZE - 128 * 1024;
let boundary_end = MPU_PART1_SIZE + 128 * 1024 - 1;
let range_crossing = fetch_range(
&client,
MULTIPART_COMPRESSION_BUCKET,
object_key,
&format!("bytes={boundary_start}-{boundary_end}"),
)
.await?;
assert_eq!(
&range_crossing[..],
&original_data[boundary_start..boundary_end + 1],
"boundary-crossing range mismatch"
);
// Range fully inside part 2.
let part2_start = MPU_PART1_SIZE + 4096;
let part2_end = MPU_PART1_SIZE + 256 * 1024 - 1;
let range_inside_part2 = fetch_range(
&client,
MULTIPART_COMPRESSION_BUCKET,
object_key,
&format!("bytes={part2_start}-{part2_end}"),
)
.await?;
assert_eq!(
&range_inside_part2[..],
&original_data[part2_start..part2_end + 1],
"part-2 range mismatch"
);
// Suffix range (last 128 KiB, entirely in part 2).
let suffix_len = 128 * 1024;
let suffix = fetch_range(&client, MULTIPART_COMPRESSION_BUCKET, object_key, &format!("bytes=-{suffix_len}")).await?;
assert_eq!(&suffix[..], &original_data[total_size - suffix_len..], "suffix range mismatch");
// partNumber GETs must return each original part.
for (part_number, expected) in [(1, &part1), (2, &part2)] {
let response = client
.get_object()
.bucket(MULTIPART_COMPRESSION_BUCKET)
.key(object_key)
.part_number(part_number)
.send()
.await?;
let body = response.body.collect().await?.into_bytes();
assert_eq!(&body[..], &expected[..], "partNumber={part_number} GET mismatch");
}
info!("Multipart compression roundtrip test passed");
env.delete_test_bucket(MULTIPART_COMPRESSION_BUCKET).await?;
env.stop_server();
Ok(())
}
const MPU_HIGH_RATIO_BUCKET: &str = "compression-mpu-high-ratio-bucket";
/// High-ratio binary multipart payload: the object key is on the compression allow-list, so the
/// disk-compression path runs and each part is stored as many compressed blocks — the shape that
/// reproduced the mid-payload Pending truncation (rustfs/rustfs#5957). Every GET shape must return
/// the exact original bytes, and the stored size must show the data really was compressed.
#[tokio::test]
#[serial]
async fn test_compression_multipart_high_ratio_binary_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart high-ratio binary compression roundtrip test");
let mut env = RustFSTestEnvironment::new().await?;
start_rustfs_with_compression(&mut env).await?;
let client = env.create_s3_client();
env.create_test_bucket(MPU_HIGH_RATIO_BUCKET).await?;
let object_key = "multipart-high-ratio.txt";
let part1 = generate_high_ratio_binary_data(MPU_PART1_SIZE, 7);
let part2 = generate_high_ratio_binary_data(MPU_PART2_SIZE, 61);
let mut original_data = part1.clone();
original_data.extend_from_slice(&part2);
let total_size = original_data.len();
multipart_upload(&client, MPU_HIGH_RATIO_BUCKET, object_key, &[&part1, &part2]).await?;
let head_response = client
.head_object()
.bucket(MPU_HIGH_RATIO_BUCKET)
.key(object_key)
.send()
.await?;
assert_eq!(
head_response.content_length().unwrap_or(0) as usize,
total_size,
"Content-Length should be the logical object size"
);
// This pattern compresses to roughly 1/50 of its logical size, so a comfortably loose 2x
// margin still proves the parts were stored compressed rather than raw or double-encoded.
let part_files = find_part_files(&env.temp_dir, MPU_HIGH_RATIO_BUCKET, object_key);
assert!(!part_files.is_empty(), "expected on-disk part files for the multipart object");
let total_physical_size: u64 = part_files.iter().filter_map(|p| fs::metadata(p).ok()).map(|m| m.len()).sum();
assert!(
total_physical_size < (total_size as u64) / 2,
"Physical size {total_physical_size} should be far below the logical size {total_size} for high-ratio data"
);
info!("High-ratio multipart physical storage size: {total_physical_size} bytes (logical {total_size} bytes)");
info!("step: full GET");
let get_response = client
.get_object()
.bucket(MPU_HIGH_RATIO_BUCKET)
.key(object_key)
.send()
.await?;
let downloaded = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded.len(), total_size);
assert_eq!(&downloaded[..], &original_data[..], "full GET data mismatch");
// Range crossing the part boundary.
info!("step: boundary range GET");
let boundary_start = MPU_PART1_SIZE - 128 * 1024;
let boundary_end = MPU_PART1_SIZE + 128 * 1024 - 1;
let range_crossing = fetch_range(
&client,
MPU_HIGH_RATIO_BUCKET,
object_key,
&format!("bytes={boundary_start}-{boundary_end}"),
)
.await?;
assert_eq!(
&range_crossing[..],
&original_data[boundary_start..boundary_end + 1],
"boundary-crossing range mismatch"
);
// partNumber GET for the trailing part.
info!("step: partNumber GET");
let part2_response = client
.get_object()
.bucket(MPU_HIGH_RATIO_BUCKET)
.key(object_key)
.part_number(2)
.send()
.await?;
let part2_body = part2_response.body.collect().await?.into_bytes();
assert_eq!(&part2_body[..], &part2[..], "partNumber=2 GET mismatch");
info!("Multipart high-ratio binary compression roundtrip test passed");
env.delete_test_bucket(MPU_HIGH_RATIO_BUCKET).await?;
env.stop_server();
Ok(())
}
const MPU_COPY_COMPRESSION_BUCKET: &str = "compression-mpu-copy-bucket";
const MPU_COPY_SOURCE_SIZE: usize = 6 * 1024 * 1024;
const MPU_COPY_RANGE_LEN: usize = 5 * 1024 * 1024;
/// UploadPartCopy feeds a part from an already stored (and already compressed) object. The copied
/// range must be decompressed on read and re-compressed into the destination part, so the final
/// object has to match "source prefix + uploaded tail" byte for byte.
#[tokio::test]
#[serial]
async fn test_compression_multipart_upload_part_copy_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart upload-part-copy compression roundtrip test");
let mut env = RustFSTestEnvironment::new().await?;
start_rustfs_with_compression(&mut env).await?;
let client = env.create_s3_client();
env.create_test_bucket(MPU_COPY_COMPRESSION_BUCKET).await?;
// Source object: a plain PUT that goes through the single-stream compression path.
let source_key = "copy-source.txt";
let source_data = generate_compressible_data(MPU_COPY_SOURCE_SIZE);
client
.put_object()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(source_key)
.body(ByteStream::from(source_data.clone()))
.send()
.await?;
// Destination object: part 1 copied from the source, part 2 uploaded directly.
let target_key = "copy-target.txt";
let part2 = generate_compressible_data(MPU_PART2_SIZE);
let mut expected_data = source_data[..MPU_COPY_RANGE_LEN].to_vec();
expected_data.extend_from_slice(&part2);
let total_size = expected_data.len();
let create = client
.create_multipart_upload()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.send()
.await?;
let upload_id = create.upload_id().ok_or("missing upload id")?.to_string();
let copy_part = client
.upload_part_copy()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.upload_id(&upload_id)
.part_number(1)
.copy_source(format!("{MPU_COPY_COMPRESSION_BUCKET}/{source_key}"))
.copy_source_range(format!("bytes=0-{}", MPU_COPY_RANGE_LEN - 1))
.send()
.await?;
let copy_etag = copy_part
.copy_part_result()
.and_then(|r| r.e_tag())
.ok_or("missing copy part etag")?
.to_string();
let uploaded_part = client
.upload_part()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.upload_id(&upload_id)
.part_number(2)
.body(ByteStream::from(part2.clone()))
.send()
.await?;
client
.complete_multipart_upload()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.upload_id(&upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(CompletedPart::builder().part_number(1).e_tag(copy_etag).build())
.parts(
CompletedPart::builder()
.part_number(2)
.e_tag(uploaded_part.e_tag().unwrap_or_default())
.build(),
)
.build(),
)
.send()
.await?;
let head_response = client
.head_object()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.send()
.await?;
assert_eq!(
head_response.content_length().unwrap_or(0) as usize,
total_size,
"Content-Length should be the logical object size"
);
let part_files = find_part_files(&env.temp_dir, MPU_COPY_COMPRESSION_BUCKET, target_key);
assert!(!part_files.is_empty(), "expected on-disk part files for the copied object");
let total_physical_size: u64 = part_files.iter().filter_map(|p| fs::metadata(p).ok()).map(|m| m.len()).sum();
assert!(
total_physical_size < (total_size / 2) as u64,
"Physical size {total_physical_size} should be well below original size {total_size} (copied part compression applied)"
);
let get_response = client
.get_object()
.bucket(MPU_COPY_COMPRESSION_BUCKET)
.key(target_key)
.send()
.await?;
let downloaded = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded.len(), total_size);
assert_eq!(&downloaded[..], &expected_data[..], "copied multipart GET data mismatch");
info!("Multipart upload-part-copy compression roundtrip test passed");
env.delete_test_bucket(MPU_COPY_COMPRESSION_BUCKET).await?;
env.stop_server();
Ok(())
}
const MPU_THREE_PARTS_BUCKET: &str = "compression-mpu-three-parts-bucket";
const MPU_THREE_PARTS_TAIL_SIZE: usize = 512 * 1024;
/// Three-part upload with uneven part sizes: each partNumber GET must map back to exactly one
/// compressed part stream, and a suffix range must resolve inside the trailing part.
#[tokio::test]
#[serial]
async fn test_compression_multipart_three_parts_part_number_gets() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting three-part multipart compression partNumber test");
let mut env = RustFSTestEnvironment::new().await?;
start_rustfs_with_compression(&mut env).await?;
let client = env.create_s3_client();
env.create_test_bucket(MPU_THREE_PARTS_BUCKET).await?;
let object_key = "multipart-three-parts.txt";
let part1 = generate_compressible_data(MPU_PART1_SIZE);
let part2 = generate_compressible_data(MPU_PART1_SIZE);
let part3 = generate_compressible_data(MPU_THREE_PARTS_TAIL_SIZE);
let mut original_data = part1.clone();
original_data.extend_from_slice(&part2);
original_data.extend_from_slice(&part3);
let total_size = original_data.len();
multipart_upload(&client, MPU_THREE_PARTS_BUCKET, object_key, &[&part1, &part2, &part3]).await?;
let head_response = client
.head_object()
.bucket(MPU_THREE_PARTS_BUCKET)
.key(object_key)
.send()
.await?;
assert_eq!(
head_response.content_length().unwrap_or(0) as usize,
total_size,
"Content-Length should be the logical object size"
);
let part_files = find_part_files(&env.temp_dir, MPU_THREE_PARTS_BUCKET, object_key);
assert!(!part_files.is_empty(), "expected on-disk part files for the multipart object");
let total_physical_size: u64 = part_files.iter().filter_map(|p| fs::metadata(p).ok()).map(|m| m.len()).sum();
assert!(
total_physical_size < (total_size / 2) as u64,
"Physical size {total_physical_size} should be well below original size {total_size} (multipart compression applied)"
);
// Every partNumber GET must return exactly the bytes of the corresponding uploaded part.
for (part_number, expected) in [(1, &part1), (2, &part2), (3, &part3)] {
let response = client
.get_object()
.bucket(MPU_THREE_PARTS_BUCKET)
.key(object_key)
.part_number(part_number)
.send()
.await?;
let body = response.body.collect().await?.into_bytes();
assert_eq!(&body[..], &expected[..], "partNumber={part_number} GET mismatch");
}
// Suffix range (last 64 KiB) resolves inside the trailing part.
let suffix_len = 64 * 1024;
let suffix = fetch_range(&client, MPU_THREE_PARTS_BUCKET, object_key, &format!("bytes=-{suffix_len}")).await?;
assert_eq!(&suffix[..], &original_data[total_size - suffix_len..], "suffix range mismatch");
info!("Three-part multipart compression partNumber test passed");
env.delete_test_bucket(MPU_THREE_PARTS_BUCKET).await?;
env.stop_server();
Ok(())
}
const MPU_SSE_COMPRESSION_BUCKET: &str = "compression-mpu-sse-bucket";
async fn start_rustfs_with_compression_and_sse(
env: &mut RustFSTestEnvironment,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use base64::Engine;
env.cleanup_existing_processes().await?;
let binary_path = rustfs_binary_path();
let master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
// Server output goes to a file inside the per-test temp dir so a failing
// run can be diagnosed from the child's logs.
let server_log = std::fs::File::create(format!("{}/server.log", env.temp_dir))?;
let server_log_err = server_log.try_clone()?;
let process = Command::new(&binary_path)
.env("RUSTFS_CONSOLE_ENABLE", "false")
.env("RUSTFS_COMPRESSION_ENABLED", "true")
.env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true")
.env("RUSTFS_SSE_S3_MASTER_KEY", master_key)
.env("RUST_LOG", "rustfs=info,rustfs_ecstore=info")
.stdout(std::process::Stdio::from(server_log))
.stderr(std::process::Stdio::from(server_log_err))
.args([
"--address",
&env.address,
"--access-key",
&env.access_key,
"--secret-key",
&env.secret_key,
&env.temp_dir,
])
.spawn()?;
env.process = Some(process);
info!("Waiting for RustFS server with compression + SSE-S3 enabled on {}", env.address);
for i in 0..30 {
if TcpStream::connect(&env.address).await.is_ok() {
info!("RustFS server is ready after {} attempts", i + 1);
return Ok(());
}
if i == 29 {
return Err("RustFS server failed to become ready".into());
}
sleep(Duration::from_secs(1)).await;
}
Ok(())
}
/// SSE-S3 + disk compression multipart: each part is compressed and then encrypted, and every GET
/// shape must still return the original plaintext bytes. Physical size must shrink because the
/// compression runs before encryption.
#[tokio::test]
#[serial]
async fn test_compression_multipart_sse_s3_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use aws_sdk_s3::types::ServerSideEncryption;
init_logging();
info!("Starting SSE-S3 multipart compression roundtrip test");
let mut env = RustFSTestEnvironment::new().await?;
start_rustfs_with_compression_and_sse(&mut env).await?;
let client = env.create_s3_client();
env.create_test_bucket(MPU_SSE_COMPRESSION_BUCKET).await?;
let object_key = "multipart-sse-compressible.txt";
let part1 = generate_compressible_data(MPU_PART1_SIZE);
let part2 = generate_compressible_data(MPU_PART2_SIZE);
let mut original_data = part1.clone();
original_data.extend_from_slice(&part2);
let total_size = original_data.len();
let create = client
.create_multipart_upload()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.server_side_encryption(ServerSideEncryption::Aes256)
.send()
.await?;
let upload_id = create.upload_id().ok_or("missing upload id")?.to_string();
let mut completed_parts = Vec::new();
for (i, part) in [&part1, &part2].into_iter().enumerate() {
let part_number = (i + 1) as i32;
let upload = client
.upload_part()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.upload_id(&upload_id)
.part_number(part_number)
.body(ByteStream::from(part.clone()))
.send()
.await?;
completed_parts.push(
CompletedPart::builder()
.part_number(part_number)
.e_tag(upload.e_tag().unwrap_or_default())
.build(),
);
}
client
.complete_multipart_upload()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.upload_id(&upload_id)
.multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
.send()
.await?;
let head_response = client
.head_object()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.send()
.await?;
assert_eq!(
head_response.content_length().unwrap_or(0) as usize,
total_size,
"Content-Length should be the logical object size"
);
assert_eq!(
head_response.server_side_encryption(),
Some(&ServerSideEncryption::Aes256),
"HEAD must report SSE-S3"
);
let part_files = find_part_files(&env.temp_dir, MPU_SSE_COMPRESSION_BUCKET, object_key);
assert!(!part_files.is_empty(), "expected on-disk part files for the multipart object");
let total_physical_size: u64 = part_files.iter().filter_map(|p| fs::metadata(p).ok()).map(|m| m.len()).sum();
assert!(
total_physical_size < (total_size / 2) as u64,
"Physical size {total_physical_size} should be well below original size {total_size} (compress-then-encrypt applied)"
);
let get_response = client
.get_object()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.send()
.await?;
let downloaded = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded.len(), total_size);
assert_eq!(&downloaded[..], &original_data[..], "SSE-S3 multipart full GET data mismatch");
// Range crossing the part boundary must decrypt and decompress across parts.
let boundary_start = MPU_PART1_SIZE - 64 * 1024;
let boundary_end = MPU_PART1_SIZE + 64 * 1024 - 1;
let range_crossing = fetch_range(
&client,
MPU_SSE_COMPRESSION_BUCKET,
object_key,
&format!("bytes={boundary_start}-{boundary_end}"),
)
.await?;
assert_eq!(
&range_crossing[..],
&original_data[boundary_start..boundary_end + 1],
"SSE-S3 boundary-crossing range mismatch"
);
// partNumber GET for the trailing part.
let part2_response = client
.get_object()
.bucket(MPU_SSE_COMPRESSION_BUCKET)
.key(object_key)
.part_number(2)
.send()
.await?;
let part2_body = part2_response.body.collect().await?.into_bytes();
assert_eq!(&part2_body[..], &part2[..], "SSE-S3 partNumber=2 GET mismatch");
info!("SSE-S3 multipart compression roundtrip test passed");
env.delete_test_bucket(MPU_SSE_COMPRESSION_BUCKET).await?;
env.stop_server();
Ok(())
}
+91 -1
View File
@@ -76,6 +76,18 @@ const SOURCE_MTIME_HEADERS: [&str; 2] = ["x-rustfs-source-mtime", "x-minio-sourc
const SOURCE_REPLICATION_REQUEST_HEADERS: [&str; 2] =
["x-rustfs-source-replication-request", "x-minio-source-replication-request"];
const SOURCE_ETAG_HEADERS: [&str; 2] = ["x-rustfs-source-etag", "x-minio-source-etag"];
const SOURCE_TAGGING_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-tagging-timestamp",
"x-minio-source-replication-tagging-timestamp",
];
const SOURCE_RETENTION_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-retention-timestamp",
"x-minio-source-replication-retention-timestamp",
];
const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -118,6 +130,25 @@ pub enum FaultAction {
WrongEtag,
}
/// Replication LWW timestamp headers observed on a request, journaled so
/// sender-side tests can assert what a real target would receive.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ReplicationTimestampHeaders {
pub tagging: Option<String>,
pub retention: Option<String>,
pub legalhold: Option<String>,
}
impl ReplicationTimestampHeaders {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
tagging: header_value(headers, &SOURCE_TAGGING_TIMESTAMP_HEADERS).map(bounded_journal_value),
retention: header_value(headers, &SOURCE_RETENTION_TIMESTAMP_HEADERS).map(bounded_journal_value),
legalhold: header_value(headers, &SOURCE_LEGALHOLD_TIMESTAMP_HEADERS).map(bounded_journal_value),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -131,6 +162,7 @@ pub struct RequestRecord {
pub part_number: Option<i32>,
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub fault: Option<FaultAction>,
}
@@ -536,7 +568,15 @@ impl S3Access for FaultAccess {
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let fault = record_request(&self.control, operation, context.method().clone(), parsed, content_length);
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
context.method().clone(),
parsed,
content_length,
replication_timestamps,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
..
@@ -589,6 +629,7 @@ fn record_request(
method: Method,
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -613,6 +654,7 @@ fn record_request(
part_number: parsed.part_number,
content_length,
consumed_bytes: None,
replication_timestamps,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -1699,6 +1741,52 @@ mod tests {
.await?)
}
#[tokio::test]
async fn journals_replication_timestamp_headers() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
client
.put_object()
.bucket("target-bucket")
.key("plain")
.body(ByteStream::from_static(b"plain"))
.send()
.await?;
client
.put_object()
.bucket("target-bucket")
.key("stamped")
.body(ByteStream::from_static(b"stamped"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-source-replication-tagging-timestamp", "2026-01-02T03:04:05Z");
headers.insert("x-minio-source-replication-retention-timestamp", "2026-01-02T03:04:06Z");
headers.insert("x-rustfs-source-replication-legalhold-timestamp", "2026-01-02T03:04:07Z");
Ok::<_, std::convert::Infallible>(request)
})
.send()
.await?;
let requests = target.requests();
let plain = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some("plain"))
.expect("plain PUT must be journaled");
assert_eq!(plain.replication_timestamps, ReplicationTimestampHeaders::default());
let stamped = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some("stamped"))
.expect("stamped PUT must be journaled");
assert_eq!(stamped.replication_timestamps.tagging.as_deref(), Some("2026-01-02T03:04:05Z"));
assert_eq!(stamped.replication_timestamps.retention.as_deref(), Some("2026-01-02T03:04:06Z"));
assert_eq!(stamped.replication_timestamps.legalhold.as_deref(), Some("2026-01-02T03:04:07Z"));
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -2985,6 +3073,7 @@ mod tests {
part_number: None,
},
Some(0),
ReplicationTimestampHeaders::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3006,6 +3095,7 @@ mod tests {
part_number: None,
},
None,
ReplicationTimestampHeaders::default(),
);
{
let bounded_records = lock(&bounded_control);
@@ -1828,33 +1828,36 @@ async fn four_node_compressed_inline_fallback() -> TestResult {
Ok(())
}
/// Multipart disk compression is live again, so a compression-enabled cluster classifies multipart objects as compressed and the roundtrip (full GET plus partNumber GET) must still return the original bytes.
/// Reverting the multipart compression fix must fail this test.
#[tokio::test]
#[serial]
async fn four_node_multipart_ignores_disk_compression_fallback() -> TestResult {
async fn four_node_multipart_disk_compression_roundtrip() -> TestResult {
init_logging();
let collector = OtlpMetricCollector::start().await?;
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
configure_reader_metric_cluster(&mut cluster, &collector);
cluster.set_env("RUSTFS_COMPRESSION_ENABLED", "true");
cluster.set_env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true");
cluster.start().await?;
let bucket = "inline-multipart-compression-fallback";
let bucket = "inline-multipart-compression-roundtrip";
cluster.create_test_bucket(bucket).await?;
let client = cluster.create_s3_client(0)?;
let key = "multipart/compression-disabled.txt";
let key = "multipart/compressed.txt";
let (body, second_part, etag) = put_two_part_multipart(&client, bucket, key).await?;
assert_reader_path(
&collector,
&client,
ReaderPathExpectation::for_class(ReaderObject::new(bucket, key, &body, etag.as_deref(), None), LEGACY_DUPLEX, MULTIPART),
ReaderPathExpectation::for_class(ReaderObject::new(bucket, key, &body, etag.as_deref(), None), LEGACY_DUPLEX, COMPRESSED),
)
.await?;
assert_part_number_reader_path(
&collector,
&client,
PartNumberReaderPathExpectation::new(bucket, key, &second_part, body.len(), MULTIPART, LEGACY_DUPLEX),
PartNumberReaderPathExpectation::new(bucket, key, &second_part, body.len(), COMPRESSED, LEGACY_DUPLEX),
)
.await?;
@@ -1871,6 +1874,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> Te
let sse_master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
cluster.set_env("RUSTFS_SSE_S3_MASTER_KEY", sse_master_key);
cluster.set_env("RUSTFS_COMPRESSION_ENABLED", "true");
cluster.set_env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true");
configure_mixed_msgpack_cluster(&mut cluster, &collector)?;
cluster.start().await?;
@@ -1890,14 +1894,21 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> Te
ReaderPathExpectation::for_class(
ReaderObject::new(bucket, multipart_key, &multipart_body, multipart_etag.as_deref(), None),
LEGACY_DUPLEX,
MULTIPART,
COMPRESSED,
),
)
.await?;
assert_part_number_reader_path(
&collector,
&client,
PartNumberReaderPathExpectation::new(bucket, multipart_key, &second_part, multipart_body.len(), MULTIPART, LEGACY_DUPLEX),
PartNumberReaderPathExpectation::new(
bucket,
multipart_key,
&second_part,
multipart_body.len(),
COMPRESSED,
LEGACY_DUPLEX,
),
)
.await?;
assert_msgpack_decode_observed(&collector, &decode_before).await?;
@@ -2353,7 +2364,11 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
hot_client.create_bucket().bucket(bucket).send().await?;
put_lifecycle_with_transition_retry(&hot_client, bucket, &tier_name).await?;
let key = "transition/mixed-multipart.bin";
// `.zip` sits on the disk-compression exclusion list: this test pins
// msgpack compat controls across ILM transition, and a compressed object
// would classify as `compressed` instead of `remote` (and the warm-tier
// read path does not decode compression — tracked separately).
let key = "transition/mixed-multipart.zip";
let (body, second_part, etag) = put_two_part_multipart(&hot_client, bucket, key).await?;
wait_for_transition(&hot_client, bucket, key, &tier_name).await?;
assert!(
@@ -97,7 +97,7 @@ async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestRe
let envs = [
("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "false"),
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"),
("RUSTFS_SCANNER_CYCLE", "1"),
("RUSTFS_ILM_PROCESS_TIME", "1"),
("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
@@ -486,7 +486,6 @@ async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
/// depend on scanner scheduling; the 1s scanner cycle stays on as a backstop.
#[tokio::test]
#[serial]
#[ignore = "pins rustfs/rustfs#6025: GET on a transitioned managed-SSE object silently returns corrupt bytes (fails with enforcement on AND off, so it is not an authorization regression); un-ignore with the fix"]
async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> TestResult {
init_logging();
-1
View File
@@ -273,7 +273,6 @@ proptest = "1"
rcgen.workspace = true
insta = { workspace = true, features = ["yaml", "json"] }
rustfs-crypto = { workspace = true }
tonic-prost = { workspace = true }
[build-dependencies]
shadow-rs = { workspace = true, default-features = false, features = ["build", "metadata"] }
+17 -13
View File
@@ -135,7 +135,8 @@ pub mod bucket {
pub use crate::bucket::metadata_sys::ConfigWriteLockProbe;
pub use crate::bucket::metadata_sys::{
BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, get, get_accelerate_config, get_bucket_policy,
acquire_bucket_metadata_transaction_lock_for_incarnation, capture_bucket_metadata_incarnation, delete,
delete_if_incarnation, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
@@ -184,17 +185,18 @@ pub mod bucket {
mrf_backlog_observability_snapshot,
};
pub use crate::bucket::replication::{
BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
BucketReplicationResyncStatus, BucketReplicationStat, BucketReplicationStats, BucketStats,
DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, InQueueMetric,
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION,
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE,
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
@@ -278,7 +280,9 @@ pub mod cluster {
}
pub mod compression {
pub use crate::io_support::compress::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible, is_disk_compression_enabled};
pub use crate::io_support::compress::{
MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible, is_disk_compression_enabled, is_multipart_disk_compression_enabled,
};
}
pub mod config {
+70 -4
View File
@@ -58,7 +58,9 @@ use rustfs_utils::http::{
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
insert_header,
};
use rustls_pki_types::pem::PemObject;
use serde::{Deserialize, Serialize};
@@ -1476,9 +1478,12 @@ impl Default for AdvancedPutOptions {
replication_status: ReplicationStatusType::Pending,
source_mtime: OffsetDateTime::now_utc(),
replication_request: false,
retention_timestamp: OffsetDateTime::now_utc(),
tagging_timestamp: OffsetDateTime::now_utc(),
legalhold_timestamp: OffsetDateTime::now_utc(),
// UNIX_EPOCH means "never modified": header() must not emit a
// timestamp header for it, otherwise a receiver would treat an
// unset category as a modification made right now.
retention_timestamp: OffsetDateTime::UNIX_EPOCH,
tagging_timestamp: OffsetDateTime::UNIX_EPOCH,
legalhold_timestamp: OffsetDateTime::UNIX_EPOCH,
replication_validity_check: false,
}
}
@@ -1675,6 +1680,16 @@ impl PutObjectOptions {
);
}
for (suffix, timestamp) in [
(SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, self.internal.tagging_timestamp),
(SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, self.internal.retention_timestamp),
(SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, self.internal.legalhold_timestamp),
] {
if timestamp.unix_timestamp() != 0 {
insert_header(&mut header, suffix, timestamp.format(&Rfc3339).unwrap_or_default());
}
}
if self.internal.replication_request {
insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
}
@@ -2842,6 +2857,57 @@ mod tests {
);
}
#[test]
fn put_object_headers_carry_replication_timestamp_headers() {
// MinIO receivers resolve concurrent tag/retention/legal-hold edits by
// last-writer-wins on these headers (object-api-options.go parses them
// as RFC3339); a replica without them loses every conflict resolution.
let mut opts = PutObjectOptions::default();
opts.internal.replication_request = true;
let tagging = OffsetDateTime::from_unix_timestamp(1_700_000_001).expect("valid timestamp");
let retention = OffsetDateTime::from_unix_timestamp(1_700_000_002).expect("valid timestamp");
let legalhold = OffsetDateTime::from_unix_timestamp(1_700_000_003).expect("valid timestamp");
opts.internal.tagging_timestamp = tagging;
opts.internal.retention_timestamp = retention;
opts.internal.legalhold_timestamp = legalhold;
let header = opts.header();
for (suffix, expected) in [
("source-replication-tagging-timestamp", tagging),
("source-replication-retention-timestamp", retention),
("source-replication-legalhold-timestamp", legalhold),
] {
assert_eq!(
rustfs_utils::http::get_header(&header, suffix).as_deref(),
Some(expected.format(&Rfc3339).expect("RFC3339 timestamp").as_str()),
"replication put requests must carry the {suffix} header"
);
}
}
#[test]
fn put_object_headers_omit_unset_replication_timestamps() {
// UNIX_EPOCH means "never modified on the source"; sending it would
// make the receiver treat an unset category as a fresh modification.
let mut opts = PutObjectOptions::default();
opts.internal.replication_request = true;
opts.internal.tagging_timestamp = OffsetDateTime::UNIX_EPOCH;
opts.internal.retention_timestamp = OffsetDateTime::UNIX_EPOCH;
opts.internal.legalhold_timestamp = OffsetDateTime::UNIX_EPOCH;
let header = opts.header();
for suffix in [
"source-replication-tagging-timestamp",
"source-replication-retention-timestamp",
"source-replication-legalhold-timestamp",
] {
assert!(
rustfs_utils::http::get_header(&header, suffix).is_none(),
"unset {suffix} must not be sent to replication targets"
);
}
}
#[tokio::test]
async fn get_remote_target_client_internal_rejects_loopback_endpoint() {
let sys = BucketTargetSys::default();
@@ -46,15 +46,13 @@ use crate::bucket::lifecycle::transition_transaction::run_transition_transaction
use crate::bucket::object_lock::ObjectLockApi;
use crate::bucket::versioning::VersioningApi as _;
use crate::bucket::versioning_sys::BucketVersioningSys;
use crate::client::object_api_utils::new_getobjectreader;
use crate::disk::error::DiskError;
use crate::disk::{DeleteOptions, Disk, DiskAPI, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, STORAGE_FORMAT_FILE};
use crate::error::Error;
use crate::error::StorageError;
use crate::error::{
error_resp_to_object_err, is_err_object_not_found, is_err_read_quorum, is_err_version_not_found, is_network_or_host_down,
};
use crate::error::{is_err_object_not_found, is_err_read_quorum, is_err_version_not_found, is_network_or_host_down};
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
use crate::object_api::{ObjectEncryptionResolver, ReadPlan};
use crate::services::tier::{
tier::{TierConfigMgr, TierOperationLease, tier_destination_id_from_metadata},
warm_backend::WarmBackendGetOpts,
@@ -4400,9 +4398,10 @@ pub async fn get_transitioned_object_reader(
h: &HeaderMap,
oi: &ObjectInfo,
opts: &ObjectOptions,
resolver: Option<&dyn ObjectEncryptionResolver>,
) -> Result<GetObjectReader, std::io::Error> {
let tier_config_mgr = runtime_sources::tier_config_mgr_handle();
get_transitioned_object_reader_with_tier_manager(bucket, object, rs, h, oi, opts, &tier_config_mgr).await
get_transitioned_object_reader_with_tier_manager(bucket, object, rs, h, oi, opts, &tier_config_mgr, resolver).await
}
fn validate_transition_remote_version(oi: &ObjectInfo) -> Result<bool, std::io::Error> {
@@ -4422,6 +4421,10 @@ fn validate_transition_remote_version(oi: &ObjectInfo) -> Result<bool, std::io::
}
}
// The resolver joins the tier manager as the second injected port this read
// needs; grouping the request half into a struct would churn every call site of
// a bug fix.
#[allow(clippy::too_many_arguments)]
pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
bucket: &str,
object: &str,
@@ -4430,6 +4433,7 @@ pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
oi: &ObjectInfo,
opts: &ObjectOptions,
tier_config_mgr: &Arc<RwLock<TierConfigMgr>>,
resolver: Option<&dyn ObjectEncryptionResolver>,
) -> Result<GetObjectReader, std::io::Error> {
validate_transition_remote_version(oi)?;
let expected_identity = tier_destination_id_from_metadata(&oi.user_defined)?;
@@ -4447,11 +4451,16 @@ pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
tgt_client.validate_remote_version_id(&oi.transitioned_object.version_id)?;
let ret = new_getobjectreader(rs, oi, opts, h);
if let Err(err) = ret {
return Err(error_resp_to_object_err(err, vec![bucket, object]));
}
let (get_fn, off, length) = ret.expect("get_transitioned_object_reader should succeed after error check");
// The same read plan the local path uses, so the tier fetch is positioned in
// the object's *stored* coordinate system and the stream is handed the same
// decrypt/decompress transforms. Reading an encrypted object's ciphertext
// through a plaintext-coordinate range and skipping the transform is how a
// transitioned SSE object used to come back as silently corrupt bytes of the
// right length (rustfs/rustfs#6025).
let plan = ReadPlan::build_for_request(rs.clone(), oi, opts, h, resolver)
.await
.map_err(|err| std::io::Error::other(format!("building the read plan for {bucket}/{object} failed: {err}")))?;
let (off, length) = (plan.storage_offset() as i64, plan.storage_length());
let mut gopts = WarmBackendGetOpts::default();
if off >= 0 && length >= 0 {
@@ -4488,7 +4497,10 @@ pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
);
e
})?;
Ok(attach_tier_operation_lease(get_fn(reader, h.clone()), tgt_client))
let object_reader = plan
.into_object_reader(Box::new(reader), oi)
.map_err(|err| std::io::Error::other(format!("wrapping the tier stream for {bucket}/{object} failed: {err}")))?;
Ok(attach_tier_operation_lease(object_reader, tgt_client))
}
struct TierOperationLeaseReader {
@@ -5776,6 +5788,7 @@ mod tests {
&object_info,
&ObjectOptions::default(),
&manager,
None,
)
.await
.expect("transitioned reader should open");
@@ -5840,6 +5853,7 @@ mod tests {
&object_info,
&ObjectOptions::default(),
&manager,
None,
)
.await
{
@@ -5880,6 +5894,7 @@ mod tests {
&object_info,
&ObjectOptions::default(),
&manager,
None,
)
.await
{
@@ -6117,6 +6132,7 @@ mod tests {
&oi,
&ObjectOptions::default(),
&manager,
None,
)
.await
{
@@ -6140,6 +6156,7 @@ mod tests {
&oi,
&ObjectOptions::default(),
&manager,
None,
)
.await
{
+18
View File
@@ -656,6 +656,16 @@ pub async fn update_under_transaction_lock(
update_under_config_write_guard(get_bucket_metadata_sys()?, guard, config_file, data).await
}
/// Clear one config file while the caller holds this bucket's transaction lock.
pub async fn delete_under_transaction_lock(
guard: &BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
) -> Result<OffsetDateTime> {
guard.ensure_valid(bucket)?;
delete_under_config_write_guard(get_bucket_metadata_sys()?, guard, config_file).await
}
pub async fn update_quota_if_incarnation(
bucket: &str,
data: Vec<u8>,
@@ -795,6 +805,14 @@ pub async fn acquire_bucket_metadata_transaction_lock(bucket: &str) -> Result<Bu
acquire_config_write_guard(get_bucket_metadata_sys()?, bucket).await
}
/// Acquire the bucket transaction lock only if its incarnation still matches.
pub async fn acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket: &str,
expected_incarnation_id: Uuid,
) -> Result<BucketMetadataMutationGuard> {
acquire_config_write_guard_for_incarnation(get_bucket_metadata_sys()?, bucket, Some(expected_incarnation_id)).await
}
pub(crate) async fn acquire_bucket_metadata_transaction_lock_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
+1 -1
View File
@@ -81,6 +81,6 @@ pub use replication_queue_boundary::{
pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
pub use replication_scanner_bridge::ReplicationScannerBridge;
pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
pub use replication_stats_boundary::{BucketReplicationStats, BucketStats};
pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;
@@ -704,6 +704,12 @@ impl ReplicationStats {
} else {
BucketReplicationStats::new()
};
// Stamp the serializable failure windows from the live samples: the
// samples themselves do not cross the peer-RPC wire, so this snapshot
// is what cluster aggregation and the metrics endpoints see.
for stat in replication_stats.stats.values_mut() {
stat.fail_stats.refresh_windows();
}
let uptime = if cache.contains_key(bucket) {
SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
@@ -15,7 +15,9 @@
#[cfg(test)]
pub(crate) use rustfs_replication::FailStats;
pub(crate) use rustfs_replication::{
ActiveWorkerStat, BucketReplicationStat, InQueueMetric, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope,
SRMetricsSummary, XferStats,
ActiveWorkerStat, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope, SRMetricsSummary,
};
pub use rustfs_replication::{BucketReplicationStats, BucketStats};
// Public so the admin wire DTOs (rustfs/src/admin/replication_metrics_wire.rs)
// can project the internal stats onto the minio-go response shapes through
// the storage_api facade chain.
pub use rustfs_replication::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
@@ -27,8 +27,10 @@ use rustfs_utils::http::{
AMZ_OBJECT_TAGGING, AMZ_SERVER_SIDE_ENCRYPTION, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT, AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID,
AMZ_STORAGE_CLASS, AMZ_TAG_COUNT, CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE,
HeaderExt as _, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP,
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_TAGGING_TIMESTAMP, get_str, insert_header_map,
is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key, ssec_replication_transport_header,
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_TAGGING_TIMESTAMP,
get_str, insert_header_map, is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key,
ssec_replication_transport_header,
};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
@@ -119,6 +121,27 @@ fn classify_replication_source_encryption(metadata: &HashMap<String, String>) ->
}
}
fn is_legacy_source_replication_timestamp_key(key: &str) -> bool {
fn has_prefix_and_suffix(key: &str, prefix: &str, suffix: &str) -> bool {
let key = key.as_bytes();
key.len() == prefix.len() + suffix.len()
&& key[..prefix.len()].eq_ignore_ascii_case(prefix.as_bytes())
&& key[prefix.len()..].eq_ignore_ascii_case(suffix.as_bytes())
}
[
SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
]
.iter()
.any(|suffix| {
["x-rustfs-", "x-minio-"]
.iter()
.any(|prefix| has_prefix_and_suffix(key, prefix, suffix))
})
}
pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String, String>) -> bool {
rustfs_replication::is_ssec_encrypted(user_defined)
}
@@ -176,6 +199,11 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
continue;
}
if is_legacy_source_replication_timestamp_key(key) {
meta.insert(format!("x-amz-meta-{key}"), value.to_string());
continue;
}
if is_internal_key(key) || is_standard_header(key) {
continue;
}
@@ -259,15 +287,23 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
if !tags.is_empty() {
put_options.user_tags = tags;
put_options.internal.tagging_timestamp =
if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP) {
OffsetDateTime::parse(&timestamp, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse tagging timestamp: {err}")))?
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
}
}
// Load the stored tagging timestamp independently of whether any tags
// remain: DeleteObjectTagging leaves the object tagless but stamps this
// key, and the deletion's LWW timestamp must still reach the replica.
// With no stored key, fall back to mod_time only while tags exist
// (MinIO parity); a tagless object without the key was never tagged and
// keeps the epoch default (no header).
put_options.internal.tagging_timestamp = if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP)
{
OffsetDateTime::parse(&timestamp, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse tagging timestamp: {err}")))?
} else if !put_options.user_tags.is_empty() {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
OffsetDateTime::UNIX_EPOCH
};
let metadata = &*object_info.user_defined;
@@ -283,13 +319,15 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
put_options.cache_control = cache_control.to_string();
}
if let Some(mode) = metadata.lookup(AMZ_OBJECT_LOCK_MODE) {
if let Some(mode) = metadata.lookup(AMZ_OBJECT_LOCK_MODE).filter(|mode| !mode.is_empty()) {
put_options.mode = Some(ObjectLockRetentionMode::from(mode.to_uppercase().as_str()));
}
if let Some(retain_until_date) = metadata.lookup(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE) {
put_options.retain_until_date = OffsetDateTime::parse(retain_until_date, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse retain until date: {err}")))?;
if !retain_until_date.is_empty() {
put_options.retain_until_date = OffsetDateTime::parse(retain_until_date, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse retain until date: {err}")))?;
}
put_options.internal.retention_timestamp =
if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP) {
OffsetDateTime::parse(&timestamp, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
@@ -694,6 +732,110 @@ mod tests {
assert!(options.internal.replication_request);
}
/// DeleteObjectTagging leaves the object tagless but stamps the
/// tagging-timestamp internal key; the deletion's LWW timestamp must
/// still be loaded (and therefore sent) so the replica can order the
/// deletion against concurrent tag edits.
#[test]
fn replication_put_options_carry_tagging_timestamp_after_tag_deletion() {
let mut metadata = std::collections::HashMap::new();
rustfs_utils::http::insert_str(&mut metadata, SUFFIX_TAGGING_TIMESTAMP, "2026-01-02T03:04:05Z".to_string());
let object_info = ObjectInfo {
user_defined: Arc::new(metadata),
user_tags: Arc::new(String::new()),
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
version_id: Some(Uuid::nil()),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
assert!(options.user_tags.is_empty());
assert_eq!(
options.internal.tagging_timestamp,
OffsetDateTime::parse("2026-01-02T03:04:05Z", &Rfc3339).expect("valid timestamp"),
"the stored tagging timestamp must load independently of remaining tags"
);
// A tagless object without the stored key was never tagged: the epoch
// default keeps the header unsent.
let untagged = ObjectInfo {
user_tags: Arc::new(String::new()),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("timestamp")),
version_id: Some(Uuid::nil()),
..Default::default()
};
let (options, _) = replication_put_object_options("", &untagged).expect("build put options");
assert_eq!(options.internal.tagging_timestamp, OffsetDateTime::UNIX_EPOCH);
}
#[test]
fn replication_put_options_do_not_promote_legacy_user_timestamp_metadata() {
let legacy_keys = [
"x-rustfs-source-replication-tagging-timestamp",
"x-rustfs-source-replication-retention-timestamp",
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-tagging-timestamp",
"x-minio-source-replication-retention-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
let object_info = ObjectInfo {
user_defined: Arc::new(
legacy_keys
.iter()
.map(|key| (key.to_string(), "2099-01-02T03:04:05Z".to_string()))
.collect(),
),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
for legacy_key in legacy_keys {
assert!(!options.user_metadata.contains_key(legacy_key));
assert_eq!(
options
.user_metadata
.get(&format!("x-amz-meta-{legacy_key}"))
.map(String::as_str),
Some("2099-01-02T03:04:05Z")
);
}
assert_eq!(options.internal.tagging_timestamp, OffsetDateTime::UNIX_EPOCH);
assert_eq!(options.internal.retention_timestamp, OffsetDateTime::UNIX_EPOCH);
assert_eq!(options.internal.legalhold_timestamp, OffsetDateTime::UNIX_EPOCH);
}
#[test]
fn replication_put_options_carry_retention_timestamp_after_clear() {
let mut metadata = HashMap::from([
(AMZ_OBJECT_LOCK_MODE.to_string(), String::new()),
(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), String::new()),
]);
rustfs_utils::http::insert_str(&mut metadata, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, "2026-01-02T03:04:05Z".to_string());
let object_info = ObjectInfo {
user_defined: Arc::new(metadata),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("retention clear must replicate");
assert!(options.mode.is_none());
assert_eq!(options.retain_until_date, OffsetDateTime::UNIX_EPOCH);
assert_eq!(
options.internal.retention_timestamp,
OffsetDateTime::parse("2026-01-02T03:04:05Z", &Rfc3339).expect("valid timestamp")
);
let headers = options.header();
assert!(!headers.contains_key(AMZ_OBJECT_LOCK_MODE));
assert!(!headers.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE));
assert_eq!(
rustfs_utils::http::get_header(&headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP).as_deref(),
Some("2026-01-02T03:04:05Z")
);
}
#[test]
fn replication_put_options_strip_encryption_metadata_from_plaintext_objects() {
use rustfs_utils::http::object_encryption_keys::{INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER};
+52 -4
View File
@@ -40,7 +40,14 @@ impl ARN {
impl Display for ARN {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "arn:rustfs:{}:{}:{}:{}", self.arn_type, self.region, self.id, self.bucket)
// The `minio` partition is deliberate: madmin-go's ParseARN
// hard-rejects any other partition, so native mc/madmin tooling can
// only decode remote-target ARNs minted in this form (backlog#1675
// P1-7). Legacy `arn:rustfs:` ARNs persisted by older releases stay
// readable via the FromStr whitelist below; runtime matching between
// targets and replication rules is by full-string equality, so mixed
// partitions coexist safely.
write!(f, "arn:minio:{}:{}:{}:{}", self.arn_type, self.region, self.id, self.bucket)
}
}
@@ -48,7 +55,12 @@ impl FromStr for ARN {
type Err = std::io::Error;
fn from_str(s: &str) -> Result<Self, Self::Err> {
if !s.starts_with("arn:rustfs:") {
// Partition whitelist, not just an `arn:` check: `BucketTargetType::
// from_str(...).unwrap_or_default()` below never fails, so this is
// the only structural gate rejecting foreign ARNs. `arn:rustfs:` is
// the legacy partition and must stay accepted forever (persisted
// bucket-targets.json / replication configs from older releases).
if !s.starts_with("arn:minio:") && !s.starts_with("arn:rustfs:") {
return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "Invalid ARN format"));
}
@@ -101,14 +113,50 @@ mod tests {
}
/// RustFS commonly generates ARNs with an empty region:
/// `arn:rustfs:replication::<deployment_id>:<bucket>`.
/// `arn:minio:replication::<deployment_id>:<bucket>`.
#[test]
fn from_str_handles_empty_region_segment() {
let parsed = ARN::from_str("arn:rustfs:replication::depl-123:bucket-a").expect("valid ARN must parse");
let parsed = ARN::from_str("arn:minio:replication::depl-123:bucket-a").expect("valid ARN must parse");
assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
assert_eq!(parsed.region, "", "region segment is empty in this form");
assert_eq!(parsed.id, "depl-123");
assert_eq!(parsed.bucket, "bucket-a");
}
/// madmin-go's `ParseARN` hard-rejects anything that does not start with
/// `arn:minio:`, so generated ARNs must use the `minio` partition or the
/// native mc/madmin tooling cannot decode remote-target listings.
#[test]
fn display_emits_minio_partition() {
let arn = ARN::new(
BucketTargetType::ReplicationService,
"depl-123".to_string(),
String::new(),
"bucket-a".to_string(),
);
assert_eq!(arn.to_string(), "arn:minio:replication::depl-123:bucket-a");
}
/// Persisted bucket-targets.json files from older RustFS releases carry
/// `arn:rustfs:` ARNs; the legacy partition must stay parseable forever.
#[test]
fn from_str_accepts_legacy_rustfs_partition() {
let parsed = ARN::from_str("arn:rustfs:replication:us-east-1:depl-123:bucket-a").expect("legacy ARN must parse");
assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
assert_eq!(parsed.region, "us-east-1");
assert_eq!(parsed.id, "depl-123");
assert_eq!(parsed.bucket, "bucket-a");
}
/// The partition whitelist is the only structural gate: `BucketTargetType::
/// from_str(...).unwrap_or_default()` never fails, so any 6-segment string
/// would otherwise parse as `type=None`.
#[test]
fn from_str_rejects_unknown_partition() {
assert!(ARN::from_str("arn:aws:replication::depl-123:bucket-a").is_err());
assert!(ARN::from_str("not-an-arn").is_err());
}
}
@@ -15,6 +15,7 @@
use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
use crate::disk::error::DiskError;
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
use futures::future::join_all;
use metrics::counter;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
use std::{
@@ -655,6 +656,7 @@ async fn list_path_raw_inner(
errs.push(None);
}
let mut pending_entries: Vec<Option<MetaCacheEntry>> = vec![None; readers.len()];
let mut peek_outcomes: Vec<Option<PeekOutcome>> = std::iter::repeat_with(|| None).take(readers.len()).collect();
loop {
let mut current = MetaCacheEntry::default();
@@ -676,6 +678,21 @@ async fn list_path_raw_inner(
let mut has_err = 0;
let mut agree = 0;
// Start every missing head read in the same round so one stalled
// disk cannot multiply the wait budget by the erasure-set width.
// Outcomes are still consumed below in stable disk-index order.
let concurrent_peeks = readers.iter_mut().enumerate().filter_map(|(i, reader)| {
if errs[i].is_some() || pending_entries[i].is_some() {
return None;
}
let cancel = &revjob_rx;
Some(async move { (i, peek_with_timeout(cancel, reader, peek_timeout).await) })
});
for (i, outcome) in join_all(concurrent_peeks).await {
peek_outcomes[i] = Some(outcome);
}
for (i, r) in readers.iter_mut().enumerate() {
if errs[i].is_some() {
has_err += 1;
@@ -685,7 +702,10 @@ async fn list_path_raw_inner(
let entry = if let Some(entry) = pending_entries[i].take() {
entry
} else {
match peek_with_timeout(&revjob_rx, r, peek_timeout).await {
let Some(outcome) = peek_outcomes[i].take() else {
return Err(DiskError::Unexpected);
};
match outcome {
PeekOutcome::Ready(res) => {
if let Some(entry) = res {
// info!("read entry disk: {}, name: {}", i, entry.name);
@@ -1295,6 +1315,36 @@ mod tests {
assert_eq!(err, DiskError::Timeout);
}
#[tokio::test(start_paused = true)]
async fn list_path_raw_bounds_multiple_stalled_readers_by_one_peek_deadline() {
let peek_timeout = Duration::from_millis(20);
let started = tokio::time::Instant::now();
let err = list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![None, None, None, None],
min_disks: 1,
test_reader_behaviors: vec![
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
],
peek_timeout: Some(peek_timeout),
..Default::default()
},
)
.await
.expect_err("all stalled readers should fail the listing");
assert_eq!(err, DiskError::Timeout);
assert_eq!(
started.elapsed(),
peek_timeout,
"reader deadlines must overlap instead of accumulating once per disk"
);
}
#[tokio::test]
async fn list_path_raw_waits_past_producer_stall_for_slow_progressing_reader() {
let entry = MetaCacheEntry {
+22 -461
View File
@@ -59,7 +59,7 @@ use std::{
path::PathBuf,
sync::{
Arc,
atomic::{AtomicBool, AtomicU32, Ordering},
atomic::{AtomicU32, Ordering},
},
time::Duration,
};
@@ -216,8 +216,6 @@ where
#[derive(Debug)]
pub struct RemoteDisk {
/// Stable identity for this handle instance; replacement handles receive a new identity.
handle_id: Uuid,
pub id: Mutex<Option<Uuid>>,
pub addr: String,
endpoint: Endpoint,
@@ -228,22 +226,9 @@ pub struct RemoteDisk {
health: Arc<DiskHealthTracker>,
/// Cancellation token for monitoring tasks
cancel_token: CancellationToken,
recovery_monitor_active: Arc<AtomicBool>,
#[cfg(test)]
recovery_monitor_start_count: Arc<AtomicU32>,
data_transport: Arc<dyn InternodeDataTransport>,
}
struct RecoveryMonitorLease {
active: Arc<AtomicBool>,
}
impl Drop for RecoveryMonitorLease {
fn drop(&mut self) {
self.active.store(false, Ordering::Release);
}
}
// ── Connection lifecycle (grpc-optimization P3) ──
/// Whether to prewarm the internode control channel in the background at construction (default off).
@@ -383,15 +368,14 @@ impl RemoteDisk {
.await
}
fn recovery_monitor_span(addr: &str, endpoint: &Endpoint, handle_id: Uuid) -> tracing::Span {
fn recovery_monitor_span(addr: &str, endpoint: &Endpoint) -> tracing::Span {
tracing::info_span!(
"recovery-monitor",
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
kind = "remote_disk",
endpoint = %endpoint,
addr = %addr,
handle_id = %handle_id
addr = %addr
)
}
@@ -427,7 +411,6 @@ impl RemoteDisk {
rustfs_utils::get_env_bool(ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING);
let disk = Self {
handle_id: Uuid::new_v4(),
id: Mutex::new(None),
addr,
endpoint: ep.clone(),
@@ -435,9 +418,6 @@ impl RemoteDisk {
health_check: opt.health_check && env_health_check,
health: Arc::new(DiskHealthTracker::new()),
cancel_token: CancellationToken::new(),
recovery_monitor_active: Arc::new(AtomicBool::new(false)),
#[cfg(test)]
recovery_monitor_start_count: Arc::new(AtomicU32::new(0)),
data_transport,
};
record_drive_runtime_state(ep, RuntimeDriveHealthState::Online);
@@ -455,16 +435,6 @@ impl RemoteDisk {
self.health.runtime_state()
}
#[cfg(test)]
fn recovery_monitor_is_active(&self) -> bool {
self.recovery_monitor_active.load(Ordering::Acquire)
}
#[cfg(test)]
fn recovery_monitor_start_count(&self) -> u32 {
self.recovery_monitor_start_count.load(Ordering::Acquire)
}
pub fn offline_duration_secs(&self) -> Option<u64> {
self.health.offline_duration().map(|duration| duration.as_secs())
}
@@ -603,54 +573,13 @@ impl RemoteDisk {
return;
}
Self::schedule_recovery_monitor(
self.addr.clone(),
self.endpoint.clone(),
self.handle_id,
Arc::clone(&self.health),
self.cancel_token.clone(),
Arc::clone(&self.recovery_monitor_active),
#[cfg(test)]
Arc::clone(&self.recovery_monitor_start_count),
);
}
fn schedule_recovery_monitor(
addr: String,
endpoint: Endpoint,
handle_id: Uuid,
health: Arc<DiskHealthTracker>,
cancel_token: CancellationToken,
active: Arc<AtomicBool>,
#[cfg(test)] start_count: Arc<AtomicU32>,
) {
if active
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
.is_err()
{
return;
}
let span = Self::recovery_monitor_span(&addr, &endpoint, handle_id);
let addr = self.addr.clone();
let endpoint = self.endpoint.clone();
let health = Arc::clone(&self.health);
let cancel_token = self.cancel_token.clone();
let span = Self::recovery_monitor_span(&addr, &endpoint);
super::spawn_background_monitor(span, async move {
#[cfg(test)]
start_count.fetch_add(1, Ordering::AcqRel);
let lease = RecoveryMonitorLease {
active: Arc::clone(&active),
};
Self::monitor_remote_disk_recovery(addr.clone(), endpoint.clone(), Arc::clone(&health), cancel_token.clone()).await;
drop(lease);
if !cancel_token.is_cancelled() && health.runtime_state() != RuntimeDriveHealthState::Online {
Self::schedule_recovery_monitor(
addr,
endpoint,
handle_id,
health,
cancel_token,
active,
#[cfg(test)]
start_count,
);
}
Self::monitor_remote_disk_recovery(addr, endpoint, health, cancel_token).await;
});
}
@@ -659,7 +588,7 @@ impl RemoteDisk {
let (tx, rx) = tokio::sync::oneshot::channel();
let endpoint = self.endpoint.clone();
let addr = self.addr.clone();
let span = Self::recovery_monitor_span(&addr, &endpoint, self.handle_id);
let span = Self::recovery_monitor_span(&addr, &endpoint);
super::spawn_background_monitor(span, async move {
warn!(
event = EVENT_REMOTE_DISK_HEALTH,
@@ -690,11 +619,9 @@ impl RemoteDisk {
let cancel_token = self.cancel_token.clone();
let addr = self.addr.clone();
let endpoint = self.endpoint.clone();
let handle_id = self.handle_id;
let recovery_monitor_active = Arc::clone(&self.recovery_monitor_active);
tokio::spawn(async move {
Self::monitor_remote_disk_health(addr, endpoint, handle_id, health, cancel_token, recovery_monitor_active).await;
Self::monitor_remote_disk_health(addr, endpoint, health, cancel_token).await;
});
}
@@ -702,10 +629,8 @@ impl RemoteDisk {
async fn monitor_remote_disk_health(
addr: String,
endpoint: Endpoint,
handle_id: Uuid,
health: Arc<DiskHealthTracker>,
cancel_token: CancellationToken,
recovery_monitor_active: Arc<AtomicBool>,
) {
let mut interval = time::interval(get_drive_active_check_interval());
@@ -730,16 +655,11 @@ impl RemoteDisk {
let addr_clone = addr.clone();
let endpoint_clone = endpoint.clone();
let cancel_clone = cancel_token.clone();
Self::schedule_recovery_monitor(
addr_clone,
endpoint_clone,
handle_id,
health_clone,
cancel_clone,
Arc::clone(&recovery_monitor_active),
#[cfg(test)]
Arc::new(AtomicU32::new(0)),
);
let span = Self::recovery_monitor_span(&addr_clone, &endpoint_clone);
super::spawn_background_monitor(span, async move {
Self::monitor_remote_disk_recovery(addr_clone, endpoint_clone, health_clone, cancel_clone).await;
});
}
loop {
@@ -798,16 +718,11 @@ impl RemoteDisk {
let addr_clone = addr.clone();
let endpoint_clone = endpoint.clone();
let cancel_clone = cancel_token.clone();
Self::schedule_recovery_monitor(
addr_clone,
endpoint_clone,
handle_id,
health_clone,
cancel_clone,
Arc::clone(&recovery_monitor_active),
#[cfg(test)]
Arc::new(AtomicU32::new(0)),
);
let span = Self::recovery_monitor_span(&addr_clone, &endpoint_clone);
super::spawn_background_monitor(span, async move {
Self::monitor_remote_disk_recovery(addr_clone, endpoint_clone, health_clone, cancel_clone).await;
});
}
}
}
@@ -1058,7 +973,6 @@ impl RemoteDisk {
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
handle_id = %self.handle_id,
op,
state = "faulty_short_circuit",
"Remote disk operation short-circuited by faulty state"
@@ -3202,23 +3116,15 @@ mod tests {
use super::*;
use crate::cluster::rpc::internode_data_transport::{InternodeDataTransportCapabilities, TcpHttpInternodeDataTransport};
use crate::runtime::sources as runtime_sources;
use rustfs_protos::proto_gen::node_service::{DiskInfoResponse, ReadAllResponse};
use serde_json::Value;
use serial_test::serial;
use std::convert::Infallible;
use std::future::Future;
use std::io::{self as std_io, Write};
use std::pin::Pin;
use std::sync::{Arc, Mutex, Mutex as StdMutex, Once};
use std::task::{Context, Poll};
use tokio::io::{ReadBuf, duplex};
use tokio::net::TcpListener;
use tonic::transport::{Endpoint as TonicEndpoint, Server};
use tonic::{Response, Status};
use tonic::{
codegen::{Body as HttpBody, BoxFuture, StdError, http},
server::NamedService,
};
use tonic::transport::Endpoint as TonicEndpoint;
use tracing::Level;
use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
use uuid::Uuid;
@@ -3378,205 +3284,6 @@ mod tests {
ns_scanner_probe_status: Arc<StdMutex<Option<u16>>>,
}
#[derive(Clone, Debug)]
struct AuthenticatedReadPeer {
audience: String,
disk_info_calls: Arc<AtomicU32>,
read_all_calls: Arc<AtomicU32>,
read_all_disks: Arc<StdMutex<Vec<String>>>,
read_all_data: Bytes,
}
impl AuthenticatedReadPeer {
fn new(audience: String, read_all_data: Bytes) -> Self {
Self {
audience,
disk_info_calls: Arc::new(AtomicU32::new(0)),
read_all_calls: Arc::new(AtomicU32::new(0)),
read_all_disks: Arc::default(),
read_all_data,
}
}
fn disk_info_calls(&self) -> u32 {
self.disk_info_calls.load(Ordering::Acquire)
}
fn read_all_calls(&self) -> u32 {
self.read_all_calls.load(Ordering::Acquire)
}
fn read_all_disks(&self) -> Vec<String> {
self.read_all_disks.lock().expect("read_all disk list lock poisoned").clone()
}
fn verify_auth<T>(&self, request: &Request<T>, path: &str) -> std::result::Result<(), Status> {
let headers = request.metadata().clone().into_headers();
crate::cluster::rpc::verify_tonic_rpc_signature(&self.audience, path, &headers)
.map_err(|err| Status::unauthenticated(err.to_string()))
}
}
#[derive(Clone, Debug)]
struct AuthenticatedReadPeerService {
peer: AuthenticatedReadPeer,
}
impl NamedService for AuthenticatedReadPeerService {
const NAME: &'static str = "node_service.NodeService";
}
impl<B> tower::Service<http::Request<B>> for AuthenticatedReadPeerService
where
B: HttpBody + Send + 'static,
B::Error: Into<StdError> + Send + 'static,
{
type Response = http::Response<tonic::body::Body>;
type Error = Infallible;
type Future = BoxFuture<Self::Response, Self::Error>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<std::result::Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, request: http::Request<B>) -> Self::Future {
match request.uri().path() {
"/node_service.NodeService/DiskInfo" => {
#[derive(Clone)]
struct DiskInfoSvc(AuthenticatedReadPeer);
impl tonic::server::UnaryService<DiskInfoRequest> for DiskInfoSvc {
type Response = DiskInfoResponse;
type Future = Pin<Box<dyn Future<Output = std::result::Result<Response<Self::Response>, Status>> + Send>>;
fn call(&mut self, request: Request<DiskInfoRequest>) -> Self::Future {
let peer = self.0.clone();
Box::pin(async move {
peer.verify_auth(&request, "/node_service.NodeService/DiskInfo")?;
let request = request.into_inner();
let opts = serde_json::from_str::<DiskInfoOptions>(&request.opts)
.map_err(|err| Status::invalid_argument(err.to_string()))?;
if !opts.noop {
return Err(Status::invalid_argument("recovery probe must use noop disk_info"));
}
peer.disk_info_calls.fetch_add(1, Ordering::AcqRel);
let disk_info = serde_json::to_string(&DiskInfo {
total: 1,
free: 1,
endpoint: request.disk,
..Default::default()
})
.map_err(|err| Status::internal(err.to_string()))?;
Ok(Response::new(DiskInfoResponse {
success: true,
disk_info,
error: None,
}))
})
}
}
let peer = self.peer.clone();
Box::pin(async move {
let method = DiskInfoSvc(peer);
let codec = tonic_prost::ProstCodec::default();
let mut grpc = tonic::server::Grpc::new(codec);
Ok(grpc.unary(method, request).await)
})
}
"/node_service.NodeService/ReadAll" => {
#[derive(Clone)]
struct ReadAllSvc(AuthenticatedReadPeer);
impl tonic::server::UnaryService<ReadAllRequest> for ReadAllSvc {
type Response = ReadAllResponse;
type Future = Pin<Box<dyn Future<Output = std::result::Result<Response<Self::Response>, Status>> + Send>>;
fn call(&mut self, request: Request<ReadAllRequest>) -> Self::Future {
let peer = self.0.clone();
Box::pin(async move {
peer.verify_auth(&request, "/node_service.NodeService/ReadAll")?;
let request = request.into_inner();
peer.read_all_calls.fetch_add(1, Ordering::AcqRel);
peer.read_all_disks
.lock()
.expect("read_all disk list lock poisoned")
.push(request.disk);
Ok(Response::new(ReadAllResponse {
success: true,
data: peer.read_all_data.clone(),
error: None,
}))
})
}
}
let peer = self.peer.clone();
Box::pin(async move {
let method = ReadAllSvc(peer);
let codec = tonic_prost::ProstCodec::default();
let mut grpc = tonic::server::Grpc::new(codec);
Ok(grpc.unary(method, request).await)
})
}
_ => Box::pin(async move {
let mut response = http::Response::new(tonic::body::Body::default());
let headers = response.headers_mut();
headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into());
headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE);
Ok(response)
}),
}
}
}
struct TestGrpcPeer {
addr: String,
peer: AuthenticatedReadPeer,
shutdown: CancellationToken,
task: tokio::task::JoinHandle<()>,
}
impl TestGrpcPeer {
async fn spawn(read_all_data: Bytes) -> Option<Self> {
let listener = match TcpListener::bind("127.0.0.1:0").await {
Ok(listener) => listener,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
Err(err) => panic!("test gRPC listener should bind: {err}"),
};
let socket_addr = listener.local_addr().expect("listener local address should be available");
let addr = format!("http://{socket_addr}");
let audience = crate::cluster::rpc::normalize_tonic_rpc_audience(&socket_addr.to_string())
.expect("test audience should normalize");
let peer = AuthenticatedReadPeer::new(audience, read_all_data);
let service = AuthenticatedReadPeerService { peer: peer.clone() };
let shutdown = CancellationToken::new();
let shutdown_for_task = shutdown.clone();
let incoming = futures_util::stream::unfold(listener, |listener| async {
Some((listener.accept().await.map(|(stream, _)| stream), listener))
});
let task = tokio::spawn(async move {
Server::builder()
.add_service(service)
.serve_with_incoming_shutdown(incoming, shutdown_for_task.cancelled_owned())
.await
.expect("test gRPC peer should serve");
});
Some(Self {
addr,
peer,
shutdown,
task,
})
}
async fn stop(self) {
self.shutdown.cancel();
let _ = self.task.await;
}
}
impl RecordingInternodeDataTransport {
fn with_ns_scanner_probe_status(status: u16) -> Self {
Self {
@@ -4690,152 +4397,6 @@ mod tests {
accept_task.abort();
}
#[tokio::test]
async fn faulty_handle_runs_only_one_recovery_monitor() {
let endpoint = Endpoint {
url: url::Url::parse("http://remote-node:9000/data/rustfs0").expect("endpoint should parse"),
is_local: false,
pool_idx: 0,
set_idx: 0,
disk_idx: 0,
};
let disk = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: true,
},
Arc::new(TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk should construct");
if !disk.health_check {
return;
}
disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
disk.spawn_recovery_monitor_if_needed();
disk.spawn_recovery_monitor_if_needed();
tokio::time::timeout(Duration::from_secs(1), async {
while disk.recovery_monitor_start_count() == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("recovery monitor should start");
assert!(disk.recovery_monitor_is_active(), "only one recovery monitor should own the handle");
assert_eq!(
disk.recovery_monitor_start_count(),
1,
"the failed compare-exchange path must not start a second monitor"
);
disk.cancel_token.cancel();
tokio::time::timeout(Duration::from_secs(1), async {
while disk.recovery_monitor_is_active() {
tokio::task::yield_now().await;
}
})
.await
.expect("cancelled recovery monitor should release its single-flight state");
assert!(!disk.recovery_monitor_is_active());
}
#[tokio::test]
#[serial(remote_disk_recovery_probe)]
async fn recovery_monitor_restores_online_then_real_reads_use_replacement_handle() {
runtime_sources::ensure_test_rpc_secret();
let Some(peer) = TestGrpcPeer::spawn(Bytes::from_static(b"replacement-data")).await else {
return;
};
let url = url::Url::parse(&format!("{}/data/rustfs0", peer.addr)).expect("endpoint should parse");
let endpoint = Endpoint {
url,
is_local: false,
pool_idx: 0,
set_idx: 0,
disk_idx: 0,
};
let disk = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: true,
},
Arc::new(TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk should construct");
disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
temp_env::async_with_vars(
[
(rustfs_config::ENV_DRIVE_RETURNING_PROBE_INTERVAL_SECS, Some("1")),
(rustfs_config::ENV_DRIVE_RETURNING_SUCCESS_THRESHOLD, Some("3")),
(rustfs_config::ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS, Some("1")),
],
async {
let monitor = tokio::spawn(RemoteDisk::monitor_remote_disk_recovery(
disk.addr.clone(),
endpoint.clone(),
Arc::clone(&disk.health),
disk.cancel_token.clone(),
));
tokio::time::timeout(Duration::from_secs(5), async {
while disk.runtime_state() != RuntimeDriveHealthState::Online {
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.expect("three authenticated recovery probes should restore the disk online");
monitor.await.expect("recovery monitor should exit after restoring Online");
assert_eq!(
peer.peer.disk_info_calls(),
3,
"RemoteDisk recovery requires the configured three successful disk_info probes"
);
let recovered_read = disk.read_all("bucket", "object").await.expect("recovered handle should read");
assert_eq!(recovered_read, Bytes::from_static(b"replacement-data"));
let replacement = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
Arc::new(TcpHttpInternodeDataTransport),
)
.await
.expect("replacement remote disk should construct");
let replacement_id = Uuid::new_v4();
replacement
.set_disk_id(Some(replacement_id))
.await
.expect("replacement disk id should set");
let replacement_read = replacement
.read_all("bucket", "object")
.await
.expect("replacement handle should route real reads");
assert_eq!(replacement_read, Bytes::from_static(b"replacement-data"));
assert_eq!(peer.peer.read_all_calls(), 2);
assert_eq!(
peer.peer.read_all_disks(),
vec![endpoint.to_string(), replacement_id.to_string()],
"real reads must use the current handle's disk reference"
);
disk.cancel_token.cancel();
replacement.cancel_token.cancel();
},
)
.await;
peer.stop().await;
}
#[tokio::test]
async fn test_copy_stream_with_buffer_copies_full_payload() {
let payload = b"walk-dir-stream".repeat(1024);
@@ -12,6 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-disk usage snapshots persisted under the metadata bucket.
//!
//! **Nothing calls into this module.** It landed complete with tests in #5307
//! (2026-07-27) and its aggregation entry point,
//! [`crate::data_usage::aggregate_local_snapshots`], has never had a caller in
//! the tree's history. The live data-usage path is
//! `load_data_usage_from_backend` / `store_data_usage_in_backend`. The items
//! below therefore carry individual `dead_code` allows rather than a module
//! blanket, so the gap stays greppable until it is either wired up or removed.
use crate::data_usage::BucketUsageInfo;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result};
@@ -26,10 +36,12 @@ pub const DATA_USAGE_DIR: &str = "datausage";
/// Directory used to store incremental scan state files under the metadata bucket.
pub const DATA_USAGE_STATE_DIR: &str = "datausage/state";
/// Snapshot file format version, allows forward compatibility if the structure evolves.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub const LOCAL_USAGE_SNAPSHOT_VERSION: u32 = 1;
/// Additional metadata describing which disk produced the snapshot.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub struct LocalUsageSnapshotMeta {
/// Disk UUID stored as a string for simpler serialization.
pub disk_id: String,
@@ -43,6 +55,7 @@ pub struct LocalUsageSnapshotMeta {
/// Usage snapshot produced by a single disk.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub struct LocalUsageSnapshot {
/// Format version recorded in the snapshot.
pub format_version: u32,
@@ -64,6 +77,7 @@ pub struct LocalUsageSnapshot {
pub objects_total_size: u64,
}
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
impl LocalUsageSnapshot {
/// Create an empty snapshot with the default format version filled in.
pub fn new(meta: LocalUsageSnapshotMeta) -> Self {
@@ -99,11 +113,13 @@ impl LocalUsageSnapshot {
}
/// Build the snapshot file name `<disk-id>.json`.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_file_name(disk_id: &str) -> String {
format!("{disk_id}.json")
}
/// Build the object path relative to `RUSTFS_META_BUCKET`, e.g. `datausage/<disk-id>.json`.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_object_path(disk_id: &str) -> String {
format!("{}/{}", DATA_USAGE_DIR, snapshot_file_name(disk_id))
}
@@ -119,11 +135,13 @@ pub fn data_usage_state_dir(root: &Path) -> PathBuf {
}
/// Build the absolute path to the snapshot file for the provided disk ID.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_path(root: &Path, disk_id: &str) -> PathBuf {
data_usage_dir(root).join(snapshot_file_name(disk_id))
}
/// Read a snapshot from disk if it exists.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub async fn read_snapshot(root: &Path, disk_id: &str) -> Result<Option<LocalUsageSnapshot>> {
let path = snapshot_path(root, disk_id);
match fs::read(&path).await {
@@ -138,6 +156,7 @@ pub async fn read_snapshot(root: &Path, disk_id: &str) -> Result<Option<LocalUsa
}
/// Persist a snapshot to disk, creating directories as needed and overwriting any existing file.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub async fn write_snapshot(root: &Path, disk_id: &str, snapshot: &LocalUsageSnapshot) -> Result<()> {
let dir = data_usage_dir(root);
fs::create_dir_all(&dir).await.map_err(Error::other)?;
+14 -104
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: scanner/data-usage state is partially migrated and still owns staged cache helpers.
#![allow(dead_code)]
pub mod local_snapshot;
@@ -34,8 +33,8 @@ use crate::{
pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapshot, snapshot_path};
use rustfs_data_usage::{
BucketTargetUsageInfo, BucketUsageInfo, CompressionTotalInfo, DATA_USAGE_OBJECT_NAME, DATA_USAGE_OBSERVED_OBJECT_NAME,
DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, SizeSummary,
VersionsHistogram, observed_data_usage_is_newer,
DataUsageCache, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, VersionsHistogram,
observed_data_usage_is_newer,
};
use rustfs_io_metrics::record_system_path_failure;
use rustfs_utils::path::SLASH_SEPARATOR;
@@ -55,7 +54,6 @@ use tracing::{debug, error, info, instrument};
// Data usage storage constants
pub const DATA_USAGE_ROOT: &str = SLASH_SEPARATOR;
const DATA_COMPRESSION_TOTAL_NAME: &str = ".compression.json";
const DATA_USAGE_BLOOM_NAME: &str = ".bloomcycle.bin";
pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin";
const DATA_USAGE_CACHE_TTL_SECS: u64 = 30;
const LIVE_BUCKET_USAGE_MAX_ENTRIES: u64 = 1024;
@@ -313,11 +311,6 @@ lazy_static::lazy_static! {
LEGACY_DATA_USAGE_OBJECT_NAME
);
static ref LEGACY_DATA_USAGE_OBJ_BACKUP_PATH: String = format!("{}.bkp", LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str());
pub static ref DATA_USAGE_BLOOM_NAME_PATH: String = format!("{}{}{}",
crate::disk::BUCKET_META_PREFIX,
SLASH_SEPARATOR,
DATA_USAGE_BLOOM_NAME
);
pub static ref DATA_COMPRESSION_TOTAL_NAME_PATH: String = format!("{}{}{}",
crate::disk::BUCKET_META_PREFIX,
SLASH_SEPARATOR,
@@ -858,6 +851,10 @@ async fn resolve_loaded_snapshot_pair_with_source(
}
}
#[allow(
dead_code,
reason = "primary/backup snapshot fallback asserted by this file's tests (backlog#1823)"
)]
async fn resolve_loaded_snapshot(
primary: Result<Vec<u8>, Error>,
backup: impl Future<Output = Result<Vec<u8>, Error>>,
@@ -1187,6 +1184,10 @@ pub async fn invalidate_admin_data_usage_snapshot_cache() {
}
/// Aggregate usage information from local disk snapshots.
#[allow(
dead_code,
reason = "reached only through aggregate_local_snapshots, which has no caller (backlog#1823)"
)]
fn merge_snapshot(aggregated: &mut DataUsageInfo, mut snapshot: LocalUsageSnapshot, latest_update: &mut Option<SystemTime>) {
if let Some(update) = snapshot.last_update
&& latest_update.is_none_or(|current| update > current)
@@ -1220,6 +1221,10 @@ fn merge_snapshot(aggregated: &mut DataUsageInfo, mut snapshot: LocalUsageSnapsh
}
}
#[allow(
dead_code,
reason = "entry point of the local usage-snapshot feature, which has had no caller since it landed in #5307 (backlog#1823)"
)]
pub async fn aggregate_local_snapshots(store: Arc<ECStore>) -> Result<(Vec<DiskUsageStatus>, DataUsageInfo), Error> {
let mut aggregated = DataUsageInfo::default();
let mut latest_update: Option<SystemTime> = None;
@@ -1767,11 +1772,6 @@ pub async fn record_bucket_object_write_unknown_previous_memory(bucket: &str, ne
entry.pending_scanner_position = None;
}
/// Fast in-memory increment for immediate quota consistency.
pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
record_bucket_object_write_memory(bucket, None, size_increment).await;
}
/// Fast in-memory update for successful object deletes.
pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
ensure_bucket_usage_cached(bucket).await;
@@ -1814,11 +1814,6 @@ pub async fn record_bucket_delete_marker_memory(bucket: &str) {
entry.pending_scanner_position = None;
}
/// Fast in-memory decrement for immediate quota consistency
pub async fn decrement_bucket_usage_memory(bucket: &str, size_decrement: u64) {
record_bucket_object_delete_memory(bucket, size_decrement, size_decrement > 0).await;
}
/// Get bucket usage from the authoritative cache for this topology.
async fn get_persisted_bucket_usage(bucket: &str) -> Option<u64> {
let store = runtime_sources::object_store_handle()?;
@@ -2013,91 +2008,6 @@ pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageIn
apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info, authoritative).await;
}
/// Sync memory cache with backend data (called by scanner)
pub async fn sync_memory_cache_with_backend() -> Result<(), Error> {
if let Some(store) = runtime_sources::object_store_handle() {
match load_data_usage_from_backend(store.clone()).await {
Ok(data_usage_info) => {
replace_bucket_usage_memory_from_info(&data_usage_info).await;
}
Err(e) => {
debug!("Failed to sync memory cache with backend: {}", e);
}
}
}
Ok(())
}
/// Create a data usage cache entry from size summary
pub fn create_cache_entry_from_summary(summary: &SizeSummary) -> DataUsageEntry {
let mut entry = DataUsageEntry::default();
entry.add_sizes(summary);
entry
}
/// Convert data usage cache to DataUsageInfo
pub fn cache_to_data_usage_info(
cache: &DataUsageCache,
path: &str,
buckets: &[crate::storage_api_contracts::bucket::BucketInfo],
) -> DataUsageInfo {
let e = match cache.find(path) {
Some(e) => e,
None => return DataUsageInfo::default(),
};
let flat = cache.flatten(&e);
let mut buckets_usage = HashMap::new();
for bucket in buckets.iter() {
let e = match cache.find(&bucket.name) {
Some(e) => e,
None => continue,
};
let flat = cache.flatten(&e);
let mut bui = BucketUsageInfo {
size: flat.size as u64,
versions_count: flat.versions as u64,
objects_count: flat.objects as u64,
delete_markers_count: flat.delete_markers as u64,
object_size_histogram: flat.obj_sizes.to_map(),
object_versions_histogram: flat.obj_versions.to_map(),
..Default::default()
};
if let Some(rs) = &flat.replication_stats {
bui.replica_size = rs.replica_size;
bui.replica_count = rs.replica_count;
for (arn, stat) in rs.targets.iter() {
bui.replication_info.insert(
arn.clone(),
BucketTargetUsageInfo {
replication_pending_size: stat.pending_size,
replicated_size: stat.replicated_size,
replication_failed_size: stat.failed_size,
replication_pending_count: stat.pending_count,
replication_failed_count: stat.failed_count,
replicated_count: stat.replicated_count,
..Default::default()
},
);
}
}
buckets_usage.insert(bucket.name.clone(), bui);
}
DataUsageInfo {
last_update: cache.info.last_update,
objects_total_count: flat.objects as u64,
versions_total_count: flat.versions as u64,
delete_markers_total_count: flat.delete_markers as u64,
objects_total_size: flat.size as u64,
buckets_count: e.children.len() as u64,
buckets_usage,
..Default::default()
}
}
// Helper functions for DataUsageCache operations
pub async fn load_data_usage_cache(store: &crate::set_disk::SetDisks, name: &str) -> crate::error::Result<DataUsageCache> {
use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
+37
View File
@@ -190,6 +190,17 @@ pub(crate) const GET_METADATA_CACHE_REASON_VERSION_SUSPENDED: &str = "version_su
pub(crate) const GET_METADATA_CACHE_REASON_VERSIONED: &str = "versioned";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA: &str = "conflicting_metadata";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER: &str = "delete_marker";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY: &str = "data_read_inline_body_verify";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED: &str = "data_read_inline_deleted";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY: &str = "data_read_inline_geometry";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH: &str = "data_read_inline_identity_mismatch";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD: &str = "data_read_inline_missing_payload";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD: &str = "data_read_inline_missing_shard";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE: &str = "data_read_inline_not_inline";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE: &str = "data_read_inline_part_shape";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE: &str = "data_read_inline_remote";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE: &str = "data_read_inline_size";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED: &str = "data_read_inline_transformed";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_ERROR: &str = "error";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM: &str = "insufficient_quorum";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_NOT_FOUND: &str = "not_found";
@@ -551,6 +562,32 @@ mod tests {
assert_eq!(GET_METADATA_CACHE_REASON_VERSIONED, "versioned");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, "conflicting_metadata");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER, "delete_marker");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY,
"data_read_inline_body_verify"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED, "data_read_inline_deleted");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY, "data_read_inline_geometry");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
"data_read_inline_identity_mismatch"
);
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
"data_read_inline_missing_payload"
);
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD,
"data_read_inline_missing_shard"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE, "data_read_inline_not_inline");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE, "data_read_inline_part_shape");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE, "data_read_inline_remote");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE, "data_read_inline_size");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED,
"data_read_inline_transformed"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_ERROR, "error");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM, "insufficient_quorum");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, "not_found");
+39 -120
View File
@@ -418,17 +418,6 @@ pub struct DiskHealthTracker {
pub last_capacity_free: AtomicU64,
/// Last successful capacity probe timestamp
pub last_capacity_probe_unix_secs: AtomicI64,
/// Authoritative atomically published runtime/status pair.
state_snapshot: AtomicU64,
transition_lock: std::sync::Mutex<()>,
}
fn pack_health_state(runtime_state: RuntimeDriveHealthState, status: u32) -> u64 {
(u64::from(runtime_state as u32) << 32) | u64::from(status)
}
fn unpack_health_state(snapshot: u64) -> (RuntimeDriveHealthState, u32) {
(RuntimeDriveHealthState::from_u32((snapshot >> 32) as u32), snapshot as u32)
}
#[derive(Debug)]
@@ -648,14 +637,23 @@ impl Default for DiskOperationMetrics {
}
impl DiskOperationMetrics {
#[allow(
dead_code,
reason = "internal metrics recorder reached only from record() below (backlog#1823)"
)]
fn record_call(&mut self) {
self.lifetime_calls.fetch_add(1, Ordering::Relaxed);
}
#[allow(
dead_code,
reason = "internal metrics recorder reached only from record() below (backlog#1823)"
)]
fn record_latency(&mut self, now_sec: u64, elapsed: Duration) {
self.record_latency_atomic(now_sec, elapsed);
}
#[allow(dead_code, reason = "metrics roll-up with no caller in this port (backlog#1823)")]
fn record(&mut self, now_sec: u64, elapsed: Duration) {
self.record_call();
self.record_latency(now_sec, elapsed);
@@ -741,8 +739,6 @@ impl DiskHealthTracker {
last_capacity_used: AtomicU64::new(0),
last_capacity_free: AtomicU64::new(0),
last_capacity_probe_unix_secs: AtomicI64::new(0),
state_snapshot: AtomicU64::new(pack_health_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK)),
transition_lock: std::sync::Mutex::new(()),
}
}
@@ -779,56 +775,39 @@ impl DiskHealthTracker {
/// Check if disk is faulty
pub fn is_faulty(&self) -> bool {
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1 == DISK_HEALTH_FAULTY
}
pub fn health_state_snapshot(&self) -> (RuntimeDriveHealthState, bool) {
let (runtime_state, status) = unpack_health_state(self.state_snapshot.load(Ordering::Acquire));
(runtime_state, status == DISK_HEALTH_FAULTY)
}
fn publish_state(&self, runtime_state: RuntimeDriveHealthState, status: u32) {
self.state_snapshot
.store(pack_health_state(runtime_state, status), Ordering::Release);
self.runtime_state.store(runtime_state as u32, Ordering::Release);
self.status.store(status, Ordering::Release);
self.status.load(Ordering::Acquire) == DISK_HEALTH_FAULTY
}
/// Set disk as faulty
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn set_faulty(&self) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
}
/// Set disk as OK
pub fn set_ok(&self) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
}
#[cfg(test)]
pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let status = if state == RuntimeDriveHealthState::Offline {
DISK_HEALTH_FAULTY
} else {
DISK_HEALTH_OK
};
self.publish_state(state, status);
self.runtime_state.store(state as u32, Ordering::Release);
match state {
RuntimeDriveHealthState::Offline => self.set_faulty(),
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect | RuntimeDriveHealthState::Returning => {
self.set_ok();
}
}
}
pub fn swap_ok_to_faulty(&self) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let (_, status) = unpack_health_state(self.state_snapshot.load(Ordering::Acquire));
if status != DISK_HEALTH_OK {
return false;
}
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
true
self.status
.compare_exchange(DISK_HEALTH_OK, DISK_HEALTH_FAULTY, Ordering::AcqRel, Ordering::Relaxed)
.is_ok()
}
pub fn runtime_state(&self) -> RuntimeDriveHealthState {
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).0
RuntimeDriveHealthState::from_u32(self.runtime_state.load(Ordering::Acquire))
}
pub fn offline_duration(&self) -> Option<Duration> {
@@ -844,7 +823,6 @@ impl DiskHealthTracker {
}
pub fn mark_failure(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let now = current_unix_secs();
let next = match current {
@@ -873,18 +851,24 @@ impl DiskHealthTracker {
};
let became_offline = next == RuntimeDriveHealthState::Offline && current != RuntimeDriveHealthState::Offline;
if next == RuntimeDriveHealthState::Offline {
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
} else {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
}
self.transition_state(endpoint, current, next, reason);
became_offline
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn mark_offline(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
if current == RuntimeDriveHealthState::Offline {
return false;
}
self.consecutive_successes.store(0, Ordering::Release);
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
self.transition_state(endpoint, current, RuntimeDriveHealthState::Offline, reason);
true
}
@@ -898,10 +882,11 @@ impl DiskHealthTracker {
}
fn reset_for_store_init_retry_at(&self, endpoint: &Endpoint, now: Duration) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let now_nanos = unix_nanos(now);
let now_secs = unix_secs_i64(now);
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.runtime_state
.store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
self.offline_since_unix_secs.store(0, Ordering::Release);
@@ -913,7 +898,6 @@ impl DiskHealthTracker {
}
pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let next = match current {
RuntimeDriveHealthState::Online => RuntimeDriveHealthState::Online,
@@ -934,6 +918,7 @@ impl DiskHealthTracker {
let became_online = next == RuntimeDriveHealthState::Online;
if became_online {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
}
@@ -963,13 +948,7 @@ impl DiskHealthTracker {
return;
}
let current_status = unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1;
let status = match next {
RuntimeDriveHealthState::Offline => DISK_HEALTH_FAULTY,
RuntimeDriveHealthState::Returning => current_status,
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect => DISK_HEALTH_OK,
};
self.publish_state(next, status);
self.runtime_state.store(next as u32, Ordering::Release);
self.last_transition_unix_secs
.store(current_unix_secs() as i64, Ordering::Release);
@@ -1012,11 +991,13 @@ impl DiskHealthTracker {
}
/// Get waiting operations count
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn waiting_count(&self) -> u32 {
self.waiting.load(Ordering::Relaxed)
}
/// Get last success timestamp
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn last_success(&self) -> i64 {
self.last_success.load(Ordering::Acquire)
}
@@ -1058,21 +1039,6 @@ impl Default for DiskHealthTracker {
}
}
/// Health check context key for tracking disk operations
#[derive(Debug, Clone)]
struct HealthDiskCtxKey;
#[derive(Debug)]
struct HealthDiskCtxValue {
last_success: Arc<AtomicI64>,
}
impl HealthDiskCtxValue {
fn log_success(&self) {
self.last_success.store(current_unix_nanos(), Ordering::Relaxed);
}
}
/// LocalDiskWrapper wraps a DiskStore with health tracking capabilities.
/// This is similar to Go's xlStorageDiskIDCheck.
#[derive(Debug, Clone)]
@@ -1104,10 +1070,6 @@ impl LocalDiskWrapper {
)
}
pub(crate) fn new_with_health(disk: Arc<LocalDisk>, health_check: bool, health: Arc<DiskHealthTracker>) -> Self {
Self::new_with_health_and_metrics(disk, health_check, health, Arc::new(DiskHealthMetricEpoch::default()))
}
pub(crate) fn new_with_reconnect_state(
disk: Arc<LocalDisk>,
health_check: bool,
@@ -1255,7 +1217,7 @@ impl LocalDiskWrapper {
return;
}
if health.is_faulty() {
if health.status.load(Ordering::Relaxed) != DISK_HEALTH_OK {
continue;
}
@@ -1470,20 +1432,6 @@ impl LocalDiskWrapper {
}
}
async fn check_id(&self, want_id: Option<Uuid>) -> Result<()> {
if want_id.is_none() {
return Ok(());
}
let stored_disk_id = self.disk.get_disk_id().await?;
if stored_disk_id != want_id {
return Err(Error::other(format!("Disk ID mismatch wanted {want_id:?}, got {stored_disk_id:?}")));
}
Ok(())
}
/// Check if disk ID is stale
async fn check_disk_stale(&self) -> Result<()> {
let Some(current_disk_id) = *self.disk_id.read().await else {
@@ -2961,35 +2909,6 @@ mod tests {
});
}
#[test]
fn concurrent_failure_and_recovery_publish_one_health_snapshot() {
let endpoint = Endpoint::try_from("/tmp/concurrent-health-snapshot").expect("endpoint should parse");
let health = Arc::new(DiskHealthTracker::new());
let workers = (0..8)
.map(|_| {
let health = Arc::clone(&health);
let endpoint = endpoint.clone();
std::thread::spawn(move || {
for _ in 0..32 {
health.mark_failure(&endpoint, "concurrent_test");
health.mark_recovery_success(&endpoint, "concurrent_test");
let (runtime, faulty) = health.health_state_snapshot();
assert!(matches!(
(runtime, faulty),
(RuntimeDriveHealthState::Online, false)
| (RuntimeDriveHealthState::Suspect, false)
| (RuntimeDriveHealthState::Offline, true)
| (RuntimeDriveHealthState::Returning, true)
));
}
})
})
.collect::<Vec<_>>();
for worker in workers {
worker.join().expect("health transition worker should not panic");
}
}
#[test]
fn operation_success_recovers_suspect_drive_without_faulting() {
let endpoint = Endpoint::try_from("/tmp/runtime-state-suspect-success").expect("endpoint should parse");
+1
View File
@@ -48,6 +48,7 @@ pub fn to_volume_error(io_err: std::io::Error) -> std::io::Error {
}
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn to_disk_error(io_err: std::io::Error) -> std::io::Error {
match io_err.kind() {
std::io::ErrorKind::NotFound => DiskError::DiskNotFound.into(),
+1
View File
@@ -178,6 +178,7 @@ pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
}
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir_all if it's a directory
match fs::remove_file(path.as_ref()).await {
+63
View File
@@ -665,6 +665,7 @@ async fn remove_empty_directory_tree_under_mount_lease(
}
#[cfg(unix)]
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
async fn remove_empty_directory_tree_with(
root: &Path,
before_descend: impl FnMut(&Path) -> std::io::Result<()>,
@@ -1016,13 +1017,29 @@ fn record_direct_read_page_fault_delta(path: &'static str, stage: &'static str,
/// When enabled, shard reads bypass the page cache using O_DIRECT flag.
/// Requires aligned buffers (typically 512 bytes or 4096 bytes).
/// Default: false (uses page cache via mmap/pread).
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const ENV_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE: &str = "RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE";
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE: bool = false;
/// Minimum shard size threshold for O_DIRECT reads.
/// Only shards larger than this threshold will use O_DIRECT.
/// Default: 4MB.
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const ENV_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD: &str = "RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD";
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD: usize = 4 * 1024 * 1024;
/// Enable O_DIRECT for erasure shard / multipart part data writes (Linux only).
@@ -1036,7 +1053,15 @@ const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD: usize = 4 * 1024 * 1024;
/// EINVAL/EOPNOTSUPP (tmpfs, overlayfs, 9p, ...) latch the path off and fall
/// back to buffered writes for the whole disk. Non-Linux always falls back.
/// Default: false (buffered writes via the page cache, as before).
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const ENV_RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE: &str = "RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE";
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE: bool = false;
const ENV_RUSTFS_OBJECT_MMAP_POPULATE_ENABLE: &str = "RUSTFS_OBJECT_MMAP_POPULATE_ENABLE";
const DEFAULT_RUSTFS_OBJECT_MMAP_POPULATE_ENABLE: bool = false;
@@ -1095,12 +1120,14 @@ macro_rules! cached_read_env {
cached_read_env! {
/// Check if O_DIRECT reads are enabled.
#[allow(dead_code, reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)")]
fn is_direct_io_read_enabled() -> bool =
rustfs_utils::get_env_bool(ENV_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE, DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE);
}
cached_read_env! {
/// Check if O_DIRECT shard/part data writes are enabled.
#[allow(dead_code, reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)")]
fn is_direct_io_write_enabled() -> bool =
rustfs_utils::get_env_bool(ENV_RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE, DEFAULT_RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE);
}
@@ -1456,6 +1483,7 @@ pub(crate) fn effective_durability(volume: &str) -> DurabilityMode {
cached_read_env! {
/// Get the O_DIRECT read threshold size.
#[allow(dead_code, reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)")]
fn get_direct_io_read_threshold() -> usize =
rustfs_utils::get_env_usize(ENV_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD, DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD);
}
@@ -1673,12 +1701,20 @@ impl DirectIoWriteState {
/// Target staging size for O_DIRECT writes, rounded up to the DIO alignment.
/// Bounds the per-writer aligned bounce buffer and batches many shard blocks
/// into one positioned write to keep the syscall count low.
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
const DIRECT_WRITE_STAGING_BYTES: usize = 1024 * 1024;
/// Aligned bounce-buffer capacity for a given DIO alignment: the target staging
/// size rounded up to a whole multiple of `align` so the buffer address, every
/// flushed batch length, and every write offset stay alignment-correct.
/// Platform-independent (no O_DIRECT), so it is unit-tested on any host.
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
fn direct_write_staging_capacity(align: usize) -> usize {
debug_assert!(align.is_power_of_two() && align >= 512);
DIRECT_WRITE_STAGING_BYTES.div_ceil(align) * align
@@ -1687,6 +1723,10 @@ fn direct_write_staging_capacity(align: usize) -> usize {
/// Split `filled` staged bytes into the alignment-sized prefix written with
/// O_DIRECT and the sub-alignment tail written buffered. Platform-independent,
/// so the tail-boundary math is unit-tested on any host.
#[allow(
dead_code,
reason = "platform-conditional: production callers are inside #[cfg(target_os = \"linux\")] blocks, so this reads as dead on non-Linux hosts (backlog#1823)"
)]
fn direct_write_tail_split(filled: usize, align: usize) -> (usize, usize) {
let aligned = filled - (filled % align);
(aligned, filled - aligned)
@@ -2142,6 +2182,7 @@ fn set_delete_version_fail_after_data_staged(path: &str) {
}
#[cfg(test)]
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub(crate) fn set_delete_version_fail_after_commit(root: &Path, path: &str) {
DELETE_VERSION_FAIL_AFTER_COMMIT
.lock()
@@ -2447,6 +2488,10 @@ enum SyncMode {
FileOnly,
}
#[allow(
dead_code,
reason = "reclaim bookkeeping fields written by Drop but never read back (backlog#1823)"
)]
struct FileCacheReclaimWriter {
inner: File,
reclaim_len: usize,
@@ -2454,6 +2499,10 @@ struct FileCacheReclaimWriter {
reclaimed: bool,
}
#[allow(
dead_code,
reason = "reclaim bookkeeping fields written by Drop but never read back (backlog#1823)"
)]
struct FileCacheReclaimReader {
inner: File,
reclaim_offset: u64,
@@ -2519,6 +2568,10 @@ impl<R: AsyncRead + Unpin> AsyncRead for StallTimeoutReader<R> {
}
}
#[allow(
dead_code,
reason = "reclaim metrics emitter reached only from the Linux-gated reclaim paths (backlog#1823)"
)]
fn record_file_cache_reclaim_success(kind: &'static str, reclaim_len: usize, started: std::time::Instant) {
// Runs per read-stream page-cache reclaim window; skip the whole emission
// (three metric-key constructions) when general metrics are disabled.
@@ -3071,6 +3124,7 @@ impl LocalIoBackend for StdBackend {
use memmap2::MmapOptions;
use std::time::{Duration as StdDuration, Instant as StdInstant};
#[allow(dead_code, reason = "mmap copy result slot kept beside the mapping it owns (backlog#1823)")]
struct MmapCopyReadResult {
bytes: Bytes,
access_check_duration: StdDuration,
@@ -4704,6 +4758,10 @@ fn build_local_io_backend(root: PathBuf) -> Arc<dyn LocalIoBackend> {
Arc::new(StdBackend::new(root))
}
#[allow(
dead_code,
reason = "path cache and cwd slots retained beside the disk root they derive from (backlog#1823)"
)]
pub struct LocalDisk {
pub root: PathBuf,
publication_root: os::PublicationRoot,
@@ -5490,6 +5548,7 @@ impl LocalDisk {
Ok(Self::resolve_abs_path_from(&self.root, path.as_ref()))
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn io_resolve_abs_path(&self, path: impl AsRef<Path>) -> PathBuf {
let path_ref = path.as_ref();
let path_str = path_ref.to_string_lossy();
@@ -5567,15 +5626,18 @@ impl LocalDisk {
}
// Check if a path is valid
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn check_valid_path<P: AsRef<Path>>(&self, path: P) -> Result<()> {
check_local_disk_valid_path(self.io_root(), path)
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn reject_symlink_components(&self, path: &Path) -> Result<()> {
reject_local_disk_symlink_components(self.io_root(), path)
}
// Batch path generation with single lock acquisition
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn get_object_paths_batch(&self, requests: &[(String, String)]) -> Result<Vec<PathBuf>> {
let mut results = Vec::with_capacity(requests.len());
let mut cache_misses = Vec::new();
@@ -6488,6 +6550,7 @@ impl LocalDisk {
Ok(f)
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
async fn open_file_read_only(&self, path: impl AsRef<Path>) -> Result<File> {
let f = super::fs::open_file(path.as_ref(), O_RDONLY).await.map_err(to_file_error)?;
Ok(f)
+4 -1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: disk abstractions still carry staged health and direct-I/O migration paths.
#![allow(dead_code)]
pub mod disk_store;
pub mod endpoint;
@@ -1114,6 +1113,10 @@ pub struct DiskInfo {
}
#[derive(Clone, Debug, Default)]
#[allow(
dead_code,
reason = "MinIO-parity disk info shape with no constructor in this port (backlog#1823)"
)]
pub struct Info {
pub total: u64,
pub free: u64,
+5 -11
View File
@@ -571,6 +571,10 @@ fn regular_files(dir: &Path) -> io::Result<Vec<PathBuf>> {
/// Fdatasync every regular file directly inside `dir`, then fsync the directory
/// itself.
#[allow(
dead_code,
reason = "reached only through sync_dir_files, whose callers are tests (backlog#1823)"
)]
pub fn sync_dir_files_std(dir: impl AsRef<Path>) -> io::Result<()> {
for entry in std::fs::read_dir(dir.as_ref())? {
let entry = entry?;
@@ -583,6 +587,7 @@ pub fn sync_dir_files_std(dir: impl AsRef<Path>) -> io::Result<()> {
/// Async wrapper around [`sync_dir_files_std`]. Large directories flush files
/// concurrently, bounded both per directory and process-wide.
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub async fn sync_dir_files(dir: impl AsRef<Path>) -> io::Result<()> {
sync_dir_files_with_limiter(dir, Arc::new(Semaphore::new(MAX_PARALLEL_FILE_SYNCS))).await
}
@@ -1809,10 +1814,6 @@ impl RenameCommitGuard {
})
}
pub(crate) fn lock_destination_directory_for_path_access(&self, directory: &Path) -> io::Result<RenameDestinationPathGuard> {
self.destination_directory_guard(directory, false)
}
pub(crate) fn create_destination_directory_for_path_access(
&self,
directory: &Path,
@@ -2858,13 +2859,6 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
Ok(())
}
/// Check if a file exists.
/// Returns true if the file exists, false otherwise.
#[tracing::instrument(level = "debug", skip_all)]
pub fn file_exists(path: impl AsRef<Path>) -> bool {
std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
}
/// Whether an [`io::Error`] means "the directory is not empty".
///
/// POSIX lets `rmdir`/`rename` report a non-empty directory as either
-69
View File
@@ -1075,9 +1075,6 @@ pub struct GenericError {
#[derive(Debug, thiserror::Error, PartialEq, Eq)]
pub enum ObjectApiError {
#[error("BackendDown")]
BackendDown(String),
#[error("The operation is not valid for the current state of the object {}/{}({})", .0.bucket, .0.object, .0.version_id)]
InvalidObjectState(GenericError),
}
@@ -1094,72 +1091,6 @@ pub struct ErrorResponse {
pub host_id: String,
}
pub fn error_resp_to_object_err(err: ErrorResponse, params: Vec<&str>) -> std::io::Error {
let mut bucket = "";
let mut object = "";
let mut version_id = "";
if !params.is_empty() {
bucket = params[0];
}
if params.len() >= 2 {
object = params[1];
}
if params.len() >= 3 {
version_id = params[2];
}
if is_network_or_host_down(&err.to_string(), false) {
return std::io::Error::other(ObjectApiError::BackendDown(format!("{err}")));
}
let err_ = std::io::Error::other(err.to_string());
let r_err = err;
let err;
let bucket = bucket.to_string();
let object = object.to_string();
let version_id = version_id.to_string();
match r_err.code {
S3ErrorCode::BucketNotEmpty => {
err = std::io::Error::other(StorageError::BucketNotEmpty("".to_string()).to_string());
}
S3ErrorCode::InvalidBucketName => {
err = std::io::Error::other(StorageError::BucketNameInvalid(bucket));
}
S3ErrorCode::InvalidPart => {
err = std::io::Error::other(StorageError::InvalidPart(0, bucket, object /* , version_id */));
}
S3ErrorCode::NoSuchBucket => {
err = std::io::Error::other(StorageError::BucketNotFound(bucket));
}
S3ErrorCode::NoSuchKey => {
if !object.is_empty() {
err = std::io::Error::other(StorageError::ObjectNotFound(bucket, object));
} else {
err = std::io::Error::other(StorageError::BucketNotFound(bucket));
}
}
S3ErrorCode::NoSuchVersion => {
if !object.is_empty() {
err = std::io::Error::other(StorageError::ObjectNotFound(bucket, object)); //, version_id);
} else {
err = std::io::Error::other(StorageError::BucketNotFound(bucket));
}
}
S3ErrorCode::AccessDenied => {
err = std::io::Error::other(StorageError::PrefixAccessDenied(bucket, object));
}
S3ErrorCode::NoSuchUpload => {
err = std::io::Error::other(StorageError::InvalidUploadID(bucket, object, version_id));
}
_ => {
err = err_;
}
}
err
}
#[cfg(test)]
mod tests {
use super::*;
+4
View File
@@ -20,6 +20,10 @@ use std::sync::atomic::AtomicI64;
/// this type never grew past its counter. `total_events` is read by the
/// notifier's log line but nothing increments it, so that field reports zero.
#[derive(Default)]
#[allow(
dead_code,
reason = "held only by the dead ecstore EventNotifier; see services/event_notification.rs (backlog#1823)"
)]
pub struct TargetList {
pub total_events: AtomicI64,
}
+22
View File
@@ -31,6 +31,13 @@ pub const ENV_DISK_COMPRESSION_MIME_TYPES: &str = "RUSTFS_COMPRESSION_MIME_TYPES
// Environment variable for additional extensions to exclude from compression (comma-separated, e.g. ".foo,.bar")
pub const ENV_ADDED_EXCLUDE_COMPRESS_EXTENSIONS: &str = "RUSTFS_ADDED_EXCLUDE_COMPRESS_EXTENSIONS";
// Environment variable to additionally enable disk compression for multipart uploads.
// Default off: nodes from before the resumable decompressor fix fail transient reads of
// compressed objects, so multipart compression stays dark until the operator confirms the
// fleet has converged on a fixed build.
// RUSTFS_COMPAT_TODO(multipart-compression-default-off-window): staged rollout switch for restored multipart compression, flipping the default to enabled on retirement. Remove after the minimum supported direct-upgrade release ships the resumable DecompressReader.
pub const ENV_DISK_COMPRESSION_MULTIPART_ENABLED: &str = "RUSTFS_COMPRESSION_MULTIPART_ENABLED";
pub const DEFAULT_DISK_COMPRESS_EXTENSIONS: &str = ".txt,.log,.csv,.json,.tar,.xml,.bin";
pub const DEFAULT_DISK_COMPRESS_MIME_TYPES: &str = "text/*,application/json,application/xml,binary/octet-stream";
@@ -171,6 +178,21 @@ pub fn is_disk_compression_enabled() -> bool {
DISK_COMPRESSION_CONFIG.get_or_init(parse_disk_compression_config).enabled
}
// Parsed once at first use, mirroring DISK_COMPRESSION_CONFIG.
static MULTIPART_DISK_COMPRESSION_ENABLED: OnceLock<bool> = OnceLock::new();
/// Whether multipart uploads may advertise disk compression. Requires the
/// regular disk-compression gates to pass as well; this is the staged-rollout
/// switch that keeps multipart compression dark during rolling upgrades from
/// builds whose decompressor was not yet resumable.
pub fn is_multipart_disk_compression_enabled() -> bool {
*MULTIPART_DISK_COMPRESSION_ENABLED.get_or_init(|| {
env::var(ENV_DISK_COMPRESSION_MULTIPART_ENABLED)
.map(|s| matches!(s.to_ascii_lowercase().as_str(), "true" | "on" | "1"))
.unwrap_or(false)
})
}
fn is_disk_compressible_with_config(headers: &http::HeaderMap, object_name: &str, config: &DiskCompressionConfig) -> bool {
// Check if disk compression is enabled (read once at first use, then fixed for process lifetime)
if !config.enabled {
+1 -1
View File
@@ -22,7 +22,7 @@ use crate::bucket::replication::{
use crate::bucket::versioning::VersioningApi as _;
use crate::config::storageclass;
use crate::error::{Error, Result};
use crate::io_support::rio::{HashReader, LimitReader};
use crate::io_support::rio::{HardLimitReader, HashReader};
use crate::storage_api_contracts::{
lifecycle::{ExpirationOptions, TransitionedObject},
range::HTTPRangeSpec,
+475 -4
View File
@@ -479,7 +479,15 @@ enum ReadTransform {
},
}
struct ReadPlan {
/// How an object's stored bytes must be fetched and transformed to serve a
/// request.
///
/// Public so callers that fetch the stored bytes from somewhere other than the
/// local erasure set — the remote-tier read path — can position their own fetch
/// with [`ReadPlan::storage_offset`] / [`ReadPlan::storage_length`] and then
/// hand the resulting stream to [`ReadPlan::into_object_reader`], instead of
/// reimplementing the transform decisions (rustfs/rustfs#6025).
pub struct ReadPlan {
storage_offset: usize,
storage_length: i64,
object_size: i64,
@@ -487,6 +495,43 @@ struct ReadPlan {
}
impl ReadPlan {
/// Byte offset into the object's **stored** bytes where the fetch must
/// start. Encrypted and compressed objects address their storage in a
/// different coordinate system than the plaintext range the caller asked
/// for, which is exactly the distinction this plan resolves.
pub fn storage_offset(&self) -> usize {
self.storage_offset
}
/// Number of **stored** bytes the fetch must deliver, in the same
/// coordinate system as [`Self::storage_offset`].
pub fn storage_length(&self) -> i64 {
self.storage_length
}
/// Build the plan for a request without consuming a stream, so a caller
/// that has to issue its own positioned fetch can read the offsets first.
pub async fn build_for_request(
rs: Option<HTTPRangeSpec>,
oi: &ObjectInfo,
opts: &ObjectOptions,
h: &HeaderMap<HeaderValue>,
resolver: Option<&dyn ObjectEncryptionResolver>,
) -> Result<Self> {
Self::build_with_resolver(rs, oi, opts, h, resolver).await
}
/// Wrap `reader` — the stored bytes this plan asked for, already positioned
/// at [`Self::storage_offset`] — in the transforms that turn them into the
/// bytes the caller requested.
pub fn into_object_reader(
self,
reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
oi: &ObjectInfo,
) -> Result<GetObjectReader> {
self.into_reader(reader, oi).map(|(reader, _, _)| reader)
}
#[cfg(test)]
async fn build(rs: Option<HTTPRangeSpec>, oi: &ObjectInfo, opts: &ObjectOptions, h: &HeaderMap<HeaderValue>) -> Result<Self> {
Self::build_with_resolver(rs, oi, opts, h, Some(&tests::TEST_RESOLVER)).await
@@ -500,8 +545,17 @@ impl ReadPlan {
resolver: Option<&dyn ObjectEncryptionResolver>,
) -> Result<Self> {
let mut rs = rs;
// A part number addresses the object's PLAINTEXT bytes. A restore read
// serves the stored representation instead (see
// [`restore_request_active`]), where that synthesized range would be
// reinterpreted as a storage range and truncate an encrypted or
// compressed payload by exactly its encoding overhead — the copy-back
// then fails its length check partway through
// (rustfs/rustfs#6025). An explicit caller range is already in storage
// coordinates on that path and is still honored.
if let Some(part_number) = opts.part_number
&& rs.is_none()
&& !restore_request_active(opts)
{
rs = http_range_spec_from_object_info(oi, part_number);
}
@@ -754,7 +808,7 @@ impl ReadPlan {
}
}
} else {
Box::new(LimitReader::new(dec_reader, total_plaintext_size))
Box::new(HardLimitReader::new(dec_reader, decompressed_length))
};
let mut object_info = oi.clone();
@@ -846,7 +900,7 @@ impl ReadPlan {
)?;
Box::new(ranged_reader)
} else {
Box::new(LimitReader::new(decompressed_reader, total_plaintext_size))
Box::new(HardLimitReader::new(decompressed_reader, total_plaintext_size_i64))
}
} else if plaintext_offset > 0 || plaintext_length != total_plaintext_size_i64 {
Box::new(RangedDecompressReader::new(
@@ -856,7 +910,7 @@ impl ReadPlan {
total_plaintext_size,
)?)
} else {
Box::new(LimitReader::new(decrypted_reader, total_plaintext_size))
Box::new(HardLimitReader::new(decrypted_reader, total_plaintext_size_i64))
};
let mut object_info = oi.clone();
@@ -1727,6 +1781,423 @@ mod tests {
assert_eq!(actual, b"fghijkl");
}
/// Compresses one multipart part exactly like the write path does
/// (`WritePlan::with_compression` wraps each part in its own
/// `compression_reader`), returning the on-disk bytes and the storage-format
/// compression index.
async fn compressed_part_fixture(data: &[u8]) -> (Vec<u8>, Option<Bytes>) {
use crate::io_support::rio::TryGetIndex as _;
let mut compressor =
crate::io_support::rio::compression_reader(Cursor::new(data.to_vec()), CompressionAlgorithm::default(), false);
let mut compressed = Vec::new();
compressor.read_to_end(&mut compressed).await.expect("compress part stream");
let index = compressor
.try_get_index()
.map(crate::io_support::rio::compression_index_storage_bytes);
(compressed, index)
}
struct CompressedMultipartFixture {
object_info: ObjectInfo,
stored: Vec<u8>,
plaintext: Vec<u8>,
}
/// Builds the on-disk representation of a compressed multipart object: each
/// part is an independent compressed stream and the storage layer serves
/// their concatenation.
async fn compressed_multipart_fixture(part_sizes: &[usize]) -> CompressedMultipartFixture {
let pattern = b"compressed multipart read path fixture data ";
let mut plaintext = Vec::new();
let mut stored = Vec::new();
let mut parts = Vec::with_capacity(part_sizes.len());
for (i, part_size) in part_sizes.iter().enumerate() {
let mut part_plaintext = Vec::with_capacity(*part_size);
while part_plaintext.len() < *part_size {
part_plaintext.extend_from_slice(pattern);
part_plaintext.push(i as u8);
}
part_plaintext.truncate(*part_size);
let (compressed, index) = compressed_part_fixture(&part_plaintext).await;
parts.push(ObjectPartInfo {
number: i + 1,
size: compressed.len(),
actual_size: *part_size as i64,
index,
..Default::default()
});
stored.extend_from_slice(&compressed);
plaintext.extend_from_slice(&part_plaintext);
}
let mut user_defined = HashMap::new();
rustfs_utils::http::insert_str(
&mut user_defined,
rustfs_utils::http::SUFFIX_COMPRESSION,
crate::io_support::rio::compression_metadata_value(CompressionAlgorithm::default()),
);
rustfs_utils::http::insert_str(&mut user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, plaintext.len().to_string());
let object_info = ObjectInfo {
bucket: "test-bucket".to_string(),
name: "compressed-multipart".to_string(),
size: stored.len() as i64,
etag: Some(format!("6bcf86bed8807b8e78f0fc6e0a53079d-{}", part_sizes.len())),
parts: Arc::new(parts),
user_defined: Arc::new(user_defined),
..Default::default()
};
CompressedMultipartFixture {
object_info,
stored,
plaintext,
}
}
/// Plans the read once to learn the storage window, then serves exactly that
/// window — mirroring how `set_disk` feeds the erasure read into the
/// returned reader.
async fn read_compressed_multipart(
fixture: &CompressedMultipartFixture,
rs: Option<HTTPRangeSpec>,
opts: &ObjectOptions,
) -> Vec<u8> {
let headers = HeaderMap::new();
let (_, offset, length) =
GetObjectReader::new(Box::new(Cursor::new(Vec::new())), rs.clone(), &fixture.object_info, opts, &headers)
.await
.expect("plan compressed multipart read");
let end = offset + usize::try_from(length).expect("storage window length must be non-negative");
assert!(
end <= fixture.stored.len(),
"planned storage window {offset}..{end} exceeds stored stream of {} bytes",
fixture.stored.len()
);
let window = fixture.stored[offset..end].to_vec();
let (mut reader, replay_offset, replay_length) =
GetObjectReader::new(Box::new(Cursor::new(window)), rs, &fixture.object_info, opts, &headers)
.await
.expect("build compressed multipart reader");
assert_eq!((replay_offset, replay_length), (offset, length), "read plan must be deterministic");
reader.read_all().await.expect("read compressed multipart stream")
}
/// Byte pattern with a 2 KiB period: it compresses extremely well while
/// looking nothing like ASCII fixtures. Mirrors the e2e generator that
/// exposed a truncated full GET on high-ratio multipart payloads.
fn high_ratio_binary_payload(size: usize, seed: u8) -> Vec<u8> {
(0..size)
.map(|i| ((i as u64).wrapping_mul(2_654_435_761).wrapping_add(seed as u64) >> 3) as u8)
.collect()
}
#[tokio::test]
async fn compressed_multipart_full_get_handles_high_ratio_binary_payload() {
let part_sizes = [5 * 1024 * 1024_usize, 1024 * 1024];
let mut plaintext = Vec::new();
let mut stored = Vec::new();
let mut parts = Vec::with_capacity(part_sizes.len());
for (i, part_size) in part_sizes.iter().enumerate() {
let part_plaintext = high_ratio_binary_payload(*part_size, if i == 0 { 7 } else { 61 });
let (compressed, index) = compressed_part_fixture(&part_plaintext).await;
parts.push(ObjectPartInfo {
number: i + 1,
size: compressed.len(),
actual_size: *part_size as i64,
index,
..Default::default()
});
stored.extend_from_slice(&compressed);
plaintext.extend_from_slice(&part_plaintext);
}
let mut user_defined = HashMap::new();
rustfs_utils::http::insert_str(
&mut user_defined,
rustfs_utils::http::SUFFIX_COMPRESSION,
crate::io_support::rio::compression_metadata_value(CompressionAlgorithm::default()),
);
rustfs_utils::http::insert_str(&mut user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, plaintext.len().to_string());
let fixture = CompressedMultipartFixture {
object_info: ObjectInfo {
bucket: "test-bucket".to_string(),
name: "high-ratio-multipart".to_string(),
size: stored.len() as i64,
etag: Some("6bcf86bed8807b8e78f0fc6e0a53079d-2".to_string()),
parts: Arc::new(parts),
user_defined: Arc::new(user_defined),
..Default::default()
},
stored,
plaintext,
};
let read = read_compressed_multipart(&fixture, None, &ObjectOptions::default()).await;
assert_eq!(read.len(), fixture.plaintext.len(), "full GET must return the logical size");
assert_eq!(read, fixture.plaintext, "high-ratio multipart payload must survive the roundtrip");
}
/// Full GET over a compressed multipart object must decode across part
/// boundaries: every part is an independent compressed stream (this is also
/// the on-disk shape written by builds before rustfs/rustfs#5169 disabled
/// multipart compression, so this pins legacy-object readability).
#[tokio::test]
async fn compressed_multipart_full_get_decodes_across_part_boundaries() {
let fixture = compressed_multipart_fixture(&[3 * 1024 * 1024, 2 * 1024 * 1024, 512 * 1024]).await;
let read = read_compressed_multipart(&fixture, None, &ObjectOptions::default()).await;
assert_eq!(read.len(), fixture.plaintext.len(), "full GET must return the logical size");
assert_eq!(read, fixture.plaintext, "full GET must reassemble all parts");
}
#[tokio::test]
async fn compressed_multipart_range_get_crosses_part_boundary() {
let fixture = compressed_multipart_fixture(&[3 * 1024 * 1024, 2 * 1024 * 1024]).await;
let boundary = 3 * 1024 * 1024_i64;
let rs = HTTPRangeSpec {
is_suffix_length: false,
start: boundary - 100_000,
end: boundary + 100_000 - 1,
};
let read = read_compressed_multipart(&fixture, Some(rs), &ObjectOptions::default()).await;
let expected = &fixture.plaintext[(boundary - 100_000) as usize..(boundary + 100_000) as usize];
assert_eq!(read, expected, "boundary-crossing range must splice both parts");
}
#[tokio::test]
async fn compressed_multipart_range_get_seeks_into_later_part() {
let fixture = compressed_multipart_fixture(&[3 * 1024 * 1024, 4 * 1024 * 1024]).await;
// Deep inside part 2 so the plan skips part 1 entirely and (when the
// part carries an index) seeks within part 2.
let start = 3 * 1024 * 1024_i64 + 2 * 1024 * 1024_i64 + 137;
let rs = HTTPRangeSpec {
is_suffix_length: false,
start,
end: start + 64 * 1024 - 1,
};
let read = read_compressed_multipart(&fixture, Some(rs), &ObjectOptions::default()).await;
let expected = &fixture.plaintext[start as usize..(start + 64 * 1024) as usize];
assert_eq!(read, expected, "range inside a later part must decode from that part");
}
/// Parts written without a compression index (small parts skip the index in
/// the rio-v2 backend) must still be rangeable: the plan starts at the part
/// boundary and skips decompressed bytes.
#[tokio::test]
async fn compressed_multipart_range_get_works_without_part_indexes() {
let mut fixture = compressed_multipart_fixture(&[1024 * 1024, 1024 * 1024]).await;
let parts = fixture
.object_info
.parts
.iter()
.map(|part| ObjectPartInfo {
index: None,
..part.clone()
})
.collect::<Vec<_>>();
fixture.object_info.parts = Arc::new(parts);
let start = 1024 * 1024_i64 + 4096;
let rs = HTTPRangeSpec {
is_suffix_length: false,
start,
end: start + 32 * 1024 - 1,
};
let read = read_compressed_multipart(&fixture, Some(rs), &ObjectOptions::default()).await;
let expected = &fixture.plaintext[start as usize..(start + 32 * 1024) as usize];
assert_eq!(read, expected, "index-less parts must fall back to part-boundary skip");
}
#[tokio::test]
async fn compressed_multipart_part_number_get_returns_single_part() {
let part_sizes = [3 * 1024 * 1024, 2 * 1024 * 1024, 512 * 1024];
let fixture = compressed_multipart_fixture(&part_sizes).await;
let mut logical_offset = 0_usize;
for (i, part_size) in part_sizes.iter().enumerate() {
let opts = ObjectOptions {
part_number: Some(i + 1),
..Default::default()
};
let read = read_compressed_multipart(&fixture, None, &opts).await;
let expected = &fixture.plaintext[logical_offset..logical_offset + part_size];
assert_eq!(read.len(), *part_size, "partNumber={} GET must return the part's logical size", i + 1);
assert_eq!(read, expected, "partNumber={} GET must return the original part bytes", i + 1);
logical_offset += part_size;
}
}
#[tokio::test]
async fn compressed_multipart_suffix_range_reads_tail() {
let fixture = compressed_multipart_fixture(&[3 * 1024 * 1024, 1024 * 1024]).await;
let suffix_len = 128 * 1024_i64;
let rs = HTTPRangeSpec {
is_suffix_length: true,
start: suffix_len,
end: -1,
};
let read = read_compressed_multipart(&fixture, Some(rs), &ObjectOptions::default()).await;
let expected = &fixture.plaintext[fixture.plaintext.len() - suffix_len as usize..];
assert_eq!(read, expected, "suffix range must return the tail of the last part");
}
/// Builds an SSE-C + disk-compression multipart object exactly like the
/// write path: each part is compressed into its own stream and then
/// encrypted with the per-part key schedule. The fixture is
/// legacy-encryption-specific (`rustfs_rio::EncryptReader`), matching the
/// pre-existing `build_legacy_ssec_multipart_fixture` shape, while the
/// compression layer follows the active backend feature.
async fn compressed_encrypted_multipart_fixture(key_bytes: [u8; 32], part_sizes: &[usize]) -> CompressedMultipartFixture {
let pattern = b"compressed encrypted multipart fixture data ";
let mut plaintext = Vec::new();
let mut stored = Vec::new();
let mut parts = Vec::with_capacity(part_sizes.len());
for (i, part_size) in part_sizes.iter().enumerate() {
let part_number = i + 1;
let mut part_plaintext = Vec::with_capacity(*part_size);
while part_plaintext.len() < *part_size {
part_plaintext.extend_from_slice(pattern);
part_plaintext.push(part_number as u8);
}
part_plaintext.truncate(*part_size);
let (compressed, index) = compressed_part_fixture(&part_plaintext).await;
let mut part_cipher = Vec::new();
rustfs_rio::EncryptReader::new_multipart(Cursor::new(compressed), key_bytes, LEGACY_FIXTURE_BASE_NONCE, part_number)
.read_to_end(&mut part_cipher)
.await
.expect("encrypt compressed fixture part");
parts.push(ObjectPartInfo {
number: part_number,
size: part_cipher.len(),
actual_size: *part_size as i64,
index,
..Default::default()
});
stored.extend_from_slice(&part_cipher);
plaintext.extend_from_slice(&part_plaintext);
}
let mut user_defined = legacy_ssec_multipart_metadata(key_bytes, plaintext.len());
rustfs_utils::http::insert_str(
&mut user_defined,
rustfs_utils::http::SUFFIX_COMPRESSION,
crate::io_support::rio::compression_metadata_value(CompressionAlgorithm::default()),
);
rustfs_utils::http::insert_str(&mut user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, plaintext.len().to_string());
let object_info = ObjectInfo {
bucket: "test-bucket".to_string(),
name: "compressed-encrypted-multipart".to_string(),
size: stored.len() as i64,
etag: Some(format!("6bcf86bed8807b8e78f0fc6e0a53079d-{}", part_sizes.len())),
parts: Arc::new(parts),
user_defined: Arc::new(user_defined),
..Default::default()
};
CompressedMultipartFixture {
object_info,
stored,
plaintext,
}
}
async fn read_compressed_encrypted_multipart(
fixture: &CompressedMultipartFixture,
key_bytes: [u8; 32],
rs: Option<HTTPRangeSpec>,
opts: &ObjectOptions,
) -> Vec<u8> {
let headers = ssec_headers_from_key(key_bytes);
let (_, offset, length) =
GetObjectReader::new(Box::new(Cursor::new(Vec::new())), rs.clone(), &fixture.object_info, opts, &headers)
.await
.expect("plan compressed encrypted multipart read");
let end = offset + usize::try_from(length).expect("storage window length must be non-negative");
assert!(
end <= fixture.stored.len(),
"planned storage window {offset}..{end} exceeds stored stream of {} bytes",
fixture.stored.len()
);
let window = fixture.stored[offset..end].to_vec();
let (mut reader, replay_offset, replay_length) =
GetObjectReader::new(Box::new(Cursor::new(window)), rs, &fixture.object_info, opts, &headers)
.await
.expect("build compressed encrypted multipart reader");
assert_eq!((replay_offset, replay_length), (offset, length), "read plan must be deterministic");
reader.read_all().await.expect("read compressed encrypted multipart stream")
}
#[tokio::test]
async fn compressed_encrypted_multipart_full_get_roundtrip() {
let key_bytes = [0x6Eu8; 32];
let fixture = compressed_encrypted_multipart_fixture(key_bytes, &[3 * 1024 * 1024, 1024 * 1024]).await;
let read = read_compressed_encrypted_multipart(&fixture, key_bytes, None, &ObjectOptions::default()).await;
assert_eq!(read.len(), fixture.plaintext.len(), "full GET must return the logical size");
assert_eq!(read, fixture.plaintext, "SSE-C + compression full GET must reassemble all parts");
}
#[tokio::test]
async fn compressed_encrypted_multipart_range_crosses_part_boundary() {
let key_bytes = [0x6Eu8; 32];
let fixture = compressed_encrypted_multipart_fixture(key_bytes, &[3 * 1024 * 1024, 1024 * 1024]).await;
let boundary = 3 * 1024 * 1024_i64;
let rs = HTTPRangeSpec {
is_suffix_length: false,
start: boundary - 65_536,
end: boundary + 65_536 - 1,
};
let read = read_compressed_encrypted_multipart(&fixture, key_bytes, Some(rs), &ObjectOptions::default()).await;
let expected = &fixture.plaintext[(boundary - 65_536) as usize..(boundary + 65_536) as usize];
assert_eq!(read, expected, "SSE-C + compression boundary-crossing range must splice both parts");
}
#[tokio::test]
async fn compressed_encrypted_multipart_part_number_get_returns_single_part() {
let key_bytes = [0x6Eu8; 32];
let part_sizes = [3 * 1024 * 1024, 1024 * 1024];
let fixture = compressed_encrypted_multipart_fixture(key_bytes, &part_sizes).await;
let opts = ObjectOptions {
part_number: Some(2),
..Default::default()
};
let read = read_compressed_encrypted_multipart(&fixture, key_bytes, None, &opts).await;
let expected = &fixture.plaintext[part_sizes[0]..];
assert_eq!(read.len(), part_sizes[1], "partNumber=2 GET must return the part's logical size");
assert_eq!(read, expected, "partNumber=2 GET must return the original part bytes");
}
#[tokio::test]
async fn test_get_object_reader_rejects_ssec_read_without_headers() {
let object_info = ObjectInfo {
+6
View File
@@ -277,6 +277,12 @@ pub struct ObjectOptions {
/// fence avoids recursively acquiring the read lock behind a queued writer.
pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
pub replication_request: bool,
/// Source-cluster LWW timestamps carried by an authorized replication
/// request; None when the source never modified the category. Only the
/// replication-authorized options builders may set these.
pub replication_tagging_timestamp: Option<OffsetDateTime>,
pub replication_retention_timestamp: Option<OffsetDateTime>,
pub replication_legalhold_timestamp: Option<OffsetDateTime>,
/// Authorized SSE-C replication passthrough: the body is already
/// ciphertext, so the write path must not encrypt or compress it and
/// stores the restored encryption metadata verbatim. Only the
@@ -23,6 +23,10 @@ use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::task::JoinSet;
#[allow(
dead_code,
reason = "default operation label for the test-only AsyncBatchProcessor::new (backlog#1823)"
)]
const BATCH_PROCESSOR_OPERATION_CUSTOM: &str = "custom";
const BATCH_PROCESSOR_OPERATION_READ: &str = "read";
const BATCH_PROCESSOR_OPERATION_WRITE: &str = "write";
@@ -211,6 +215,7 @@ pub struct AsyncBatchProcessor {
}
impl AsyncBatchProcessor {
#[allow(dead_code, reason = "constructor used only by this file's tests (backlog#1823)")]
pub fn new(max_concurrent: usize) -> Self {
Self::new_with_operation(max_concurrent, BATCH_PROCESSOR_OPERATION_CUSTOM)
}
@@ -26,11 +26,26 @@ use std::sync::atomic::Ordering;
use tokio::sync::RwLock;
use tracing::warn;
/// Dead ecstore-side notification skeleton.
///
/// The working notification stack is `rustfs-notify`, whose own `EventNotifier`
/// is the one bucket configuration actually drives. Nothing calls the methods
/// below; `init_bucket_targets` even logs that it is a no-op in this build.
/// Removing it means also retiring the `InstanceContext` slot that holds it
/// (backlog#939 Phase 5), so it is left explicit here rather than half-removed.
#[allow(
dead_code,
reason = "ecstore-side notification skeleton superseded by rustfs-notify; see module note (backlog#1823)"
)]
pub struct EventNotifier {
target_list: TargetList,
//bucket_rules_map: HashMap<String , HashMap<EventName, Rules>>,
}
#[allow(
dead_code,
reason = "ecstore-side notification skeleton superseded by rustfs-notify; see module note (backlog#1823)"
)]
impl EventNotifier {
pub fn new() -> Arc<RwLock<Self>> {
Arc::new(RwLock::new(Self {
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: background service owners still contain staged notification/rebalance/tier paths.
#![allow(dead_code)]
pub(crate) mod batch_processor;
pub(crate) mod event_notification;
@@ -1623,6 +1623,7 @@ impl NotificationSys {
workers.peers.remove(host);
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn tier_config_reload_worker_active(&self, host: &str) -> bool {
self.tier_config_reload_workers
.lock()
@@ -1796,6 +1797,7 @@ where
.map_err(|_| Error::other(format!("scanner activity peer {host} timed out after {timeout_duration:?}")))?
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
async fn call_peer_with_timeout<F, Fut>(
timeout_dur: Duration,
host_label: &str,
@@ -864,6 +864,10 @@ pub(super) fn merge_rebalance_meta(remote: &mut RebalanceMeta, local: &Rebalance
RebalanceMetaMergeOutcome::Merged
}
#[allow(
dead_code,
reason = "stop-transition helper retained beside stop_rebalance_meta_snapshot; no caller yet (backlog#1823)"
)]
pub(super) fn mark_started_rebalance_pools_stopped(meta: &mut RebalanceMeta, stop_time: OffsetDateTime) {
for pool_stat in meta.pool_stats.iter_mut() {
if pool_stat.info.status == RebalStatus::Started {
@@ -964,6 +968,7 @@ pub(super) fn rollback_rebalance_start_meta_snapshot_for_id(
})
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub(super) fn stop_rebalance_meta_snapshot(meta: Option<&mut RebalanceMeta>, now: OffsetDateTime) -> Option<RebalanceMeta> {
let meta = meta?;
stop_rebalance_state(meta, now);
@@ -171,6 +171,7 @@ where
}
#[allow(clippy::too_many_arguments)]
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub(super) async fn migrate_entry_version_with_retry_wait<Backend, F, Fut, D, DFut, W, WFut>(
set: &Backend,
bucket: String,
@@ -1,5 +1,4 @@
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use time::OffsetDateTime;
use tokio_util::sync::CancellationToken;
@@ -32,8 +31,6 @@ pub struct RebalanceStats {
pub cleanup_warnings: RebalanceCleanupWarnings,
}
pub type RStats = Vec<Arc<RebalanceStats>>;
#[derive(Debug, Default)]
pub(super) struct RebalanceBucketConfigs {
pub(super) bucket_incarnation_id: Option<uuid::Uuid>,
-1
View File
@@ -30,6 +30,5 @@ pub mod warm_backend_minio;
pub mod warm_backend_r2;
pub mod warm_backend_rustfs;
pub mod warm_backend_s3;
pub mod warm_backend_s3sdk;
pub mod warm_backend_tencent;
pub mod warm_backend_wasabi;
+10 -20
View File
@@ -488,6 +488,7 @@ impl TierCandidateMutation {
targets
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn affected_targets(
&self,
manager: &TierConfigMgr,
@@ -802,6 +803,7 @@ fn tier_persisted_reference_blocks_any_target(
.any(|target| tier_persisted_reference_blocks_target(tier_name, backend_identity, target))
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
fn tier_object_blocks_target_rebind(object: &ObjectInfo, target: &TierMutationIntentTarget) -> io::Result<bool> {
tier_object_blocks_any_target_rebind(object, std::slice::from_ref(target))
}
@@ -2726,14 +2728,6 @@ impl TierConfigMgr {
Self::publish_candidate_owned(handle, candidate, driver_tier.map(str::to_string), update).await
}
fn begin_publish_transition(
handle: &Arc<RwLock<Self>>,
manager: &mut Self,
candidate: &Self,
) -> std::result::Result<TierPublishTransition, AdminError> {
Self::begin_publish_transition_with_allowed_mutation_blocks(handle, manager, candidate, None)
}
fn begin_publish_transition_with_allowed_mutation_blocks(
handle: &Arc<RwLock<Self>>,
manager: &mut Self,
@@ -2819,14 +2813,6 @@ impl TierConfigMgr {
})
}
async fn publish_candidate_inner(
handle: &Arc<RwLock<Self>>,
candidate: Self,
driver_tier: Option<&str>,
) -> std::result::Result<(), AdminError> {
Self::publish_candidate_inner_with_allowed_mutation_blocks(handle, candidate, driver_tier, None).await
}
async fn publish_candidate_inner_with_allowed_mutation_blocks(
handle: &Arc<RwLock<Self>>,
candidate: Self,
@@ -2939,6 +2925,7 @@ impl TierConfigMgr {
admin_err
}
#[allow(dead_code, reason = "reached only through #[cfg(test)] helpers in this file (backlog#1823)")]
async fn publish_candidate_owned(
handle: &Arc<RwLock<Self>>,
candidate: Self,
@@ -3541,6 +3528,7 @@ impl TierConfigMgr {
Self::update_candidate_with_config_lock(handle, api, TierCandidateMutation::Remove(tier_name.to_string(), force)).await
}
#[allow(dead_code, reason = "reached only through #[cfg(test)] helpers in this file (backlog#1823)")]
async fn remove_and_save_with<S>(
handle: &Arc<RwLock<Self>>,
api: Arc<S>,
@@ -3574,6 +3562,7 @@ impl TierConfigMgr {
Self::update_candidate_with_config_lock(handle, api, TierCandidateMutation::Clear(force)).await
}
#[allow(dead_code, reason = "reached only through #[cfg(test)] helpers in this file (backlog#1823)")]
async fn clear_and_save_with<S>(
handle: &Arc<RwLock<Self>>,
api: Arc<S>,
@@ -3612,6 +3601,10 @@ impl TierConfigMgr {
}
#[cfg(test)]
#[allow(
dead_code,
reason = "lease accounting asserted by a bucket_lifecycle_ops test behind `--features test-util` (backlog#1823)"
)]
pub(crate) async fn active_operation_lease_count(handle: &Arc<RwLock<Self>>, tier_name: &str) -> usize {
let manager = handle.read().await;
let Some(runtime) = registered_tier_driver_runtime(&manager) else {
@@ -3717,10 +3710,6 @@ impl TierConfigMgr {
Ok(())
}
fn retire_driver(&mut self, tier_name: &str) {
self.revoke_driver(tier_name);
}
fn revoke_all_drivers(&mut self) {
if let Some(runtime) = registered_tier_driver_runtime(self) {
let mut runtime = lock_unpoisoned(&runtime);
@@ -3884,6 +3873,7 @@ impl TierConfigMgr {
self.save_config(api, &config_file, data).await
}
#[allow(dead_code, reason = "reached only through #[cfg(test)] helpers in this file (backlog#1823)")]
async fn save_tiering_config_if_current<S>(
&self,
api: Arc<S>,
@@ -305,6 +305,10 @@ impl TierMutationIntent {
}
}
#[allow(
dead_code,
reason = "intent-record persistence asserted by store::init tests (backlog#1823)"
)]
pub(crate) fn tier_mutation_intent_record_object_name(mutation_id: Uuid) -> Result<String> {
tier_mutation_intent_record_object_name_with_prefix(TIER_MUTATION_INTENT_RECORD_PREFIX, mutation_id)
}
@@ -317,6 +321,10 @@ fn tier_mutation_intent_record_object_name_with_prefix(prefix: &str, mutation_id
Ok(format!("{}/{}/{}/{}.json", prefix, &mutation_key[..2], &mutation_key[2..4], mutation_key))
}
#[allow(
dead_code,
reason = "intent-record persistence asserted by store::init tests (backlog#1823)"
)]
pub(crate) fn tier_mutation_intent_id_from_record_object_name(object: &str) -> Result<Uuid> {
tier_mutation_intent_id_from_record_object_name_with_prefix(TIER_MUTATION_INTENT_RECORD_PREFIX, object)
}
@@ -355,6 +363,10 @@ fn tier_mutation_intent_id_from_record_object_name_with_prefix(prefix: &str, obj
Uuid::parse_str(mutation_key).map_err(|_| TierMutationIntentError::Corrupt("intent record path has invalid uuid"))
}
#[allow(
dead_code,
reason = "intent-record persistence asserted by store::init tests (backlog#1823)"
)]
pub(crate) async fn save_tier_mutation_intent_record<S>(api: Arc<S>, intent: &TierMutationIntent) -> EcstoreResult<()>
where
S: EcstoreObjectIO,
@@ -446,6 +458,10 @@ where
Ok((intent, etag))
}
#[allow(
dead_code,
reason = "intent-record persistence asserted by store::init tests (backlog#1823)"
)]
pub(crate) async fn save_tier_mutation_intent_record_if_current<S>(
api: Arc<S>,
intent: &TierMutationIntent,
@@ -41,10 +41,7 @@ use crate::services::tier::{
};
use tracing::warn;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
fn parse_generation(remote_version: &str) -> Result<Option<i64>, Error> {
if remote_version.is_empty() {
@@ -64,7 +61,6 @@ pub struct WarmBackendGCS {
pub control: Arc<StorageControl>,
pub bucket: String,
pub prefix: String,
pub storage_class: String,
}
impl WarmBackendGCS {
@@ -104,7 +100,6 @@ impl WarmBackendGCS {
control,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
})
}
@@ -33,8 +33,6 @@ use crate::client::{
transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore},
transition_api::{ReadCloser, ReaderImpl},
};
use crate::error::ErrorResponse;
use crate::error::error_resp_to_object_err;
use crate::services::tier::{
tier_config::TierS3,
warm_backend::{
@@ -1,200 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(unused_imports)]
#![allow(unused_variables)]
#![allow(unused_mut)]
#![allow(unused_assignments)]
#![allow(unused_must_use)]
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use url::Url;
use aws_config::meta::region::RegionProviderChain;
use aws_sdk_s3::Client;
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use crate::client::{
api_get_options::GetObjectOptions,
api_put_object::PutObjectOptions,
api_remove::RemoveObjectOptions,
transition_api::{ReadCloser, ReaderImpl},
};
use crate::error::ErrorResponse;
use crate::error::error_resp_to_object_err;
use crate::services::tier::{
tier_config::TierS3,
warm_backend::{WarmBackend, WarmBackendGetOpts},
};
pub struct WarmBackendS3 {
pub client: Arc<Client>,
pub bucket: String,
pub prefix: String,
pub storage_class: String,
}
impl WarmBackendS3 {
pub async fn new(conf: &TierS3, tier: &str) -> Result<Self, std::io::Error> {
let u = match Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(err) => {
return Err(std::io::Error::other(err.to_string()));
}
};
if conf.aws_role_web_identity_token_file == "" && conf.aws_role_arn != ""
|| conf.aws_role_web_identity_token_file != "" && conf.aws_role_arn == ""
{
return Err(std::io::Error::other("both the token file and the role ARN are required"));
} else if conf.access_key == "" && conf.secret_key != "" || conf.access_key != "" && conf.secret_key == "" {
return Err(std::io::Error::other("both the access and secret keys are required"));
} else if conf.aws_role
&& (conf.aws_role_web_identity_token_file != ""
|| conf.aws_role_arn != ""
|| conf.access_key != ""
|| conf.secret_key != "")
{
return Err(std::io::Error::other(
"AWS Role cannot be activated with static credentials or the web identity token file",
));
} else if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let creds;
if conf.access_key != "" && conf.secret_key != "" {
creds = Credentials::new(
conf.access_key.clone(), // access_key_id
conf.secret_key.clone(), // secret_access_key
None, // session_token (optional)
None,
"Static",
);
} else {
return Err(std::io::Error::other("insufficient parameters for S3 backend authentication"));
}
let region_provider = RegionProviderChain::default_provider().or_else(Region::new(conf.region.clone()));
#[allow(deprecated)]
let config = aws_config::from_env()
.endpoint_url(conf.endpoint.clone())
.region(region_provider)
.credentials_provider(creds)
.load()
.await;
let client = Client::new(&config);
let client = Arc::new(client);
Ok(Self {
client,
bucket: conf.bucket.clone(),
prefix: conf.prefix.clone().trim_matches('/').to_string(),
storage_class: conf.storage_class.clone(),
})
}
pub fn get_dest(&self, object: &str) -> String {
let mut dest_obj = object.to_string();
if self.prefix != "" {
dest_obj = format!("{}/{}", &self.prefix, object);
}
return dest_obj;
}
}
#[async_trait::async_trait]
impl WarmBackend for WarmBackendS3 {
async fn put_with_meta(
&self,
object: &str,
r: ReaderImpl,
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let client = self.client.clone();
let Ok(res) = client
.put_object()
.bucket(&self.bucket)
.key(&self.get_dest(object))
.body(match r {
ReaderImpl::Body(content_body) => ByteStream::from(content_body.to_vec()),
ReaderImpl::ObjectBody(mut content_body) => ByteStream::from(content_body.read_all().await?),
})
.send()
.await
else {
return Err(std::io::Error::other("put_object error"));
};
Ok(res.version_id().unwrap_or("").to_string())
}
async fn put(&self, object: &str, r: ReaderImpl, length: i64) -> Result<String, std::io::Error> {
self.put_with_meta(object, r, length, HashMap::new()).await
}
async fn get(&self, object: &str, rv: &str, opts: WarmBackendGetOpts) -> Result<ReadCloser, std::io::Error> {
let client = self.client.clone();
let mut req = client.get_object().bucket(&self.bucket).key(&self.get_dest(object));
if !rv.is_empty() {
req = req.version_id(rv);
}
if opts.start_offset >= 0 && opts.length > 0 {
let end = opts
.start_offset
.checked_add(opts.length)
.and_then(|v| v.checked_sub(1))
.ok_or_else(|| std::io::Error::other("invalid range: overflow"))?;
req = req.range(format!("bytes={}-{}", opts.start_offset, end));
}
let res = req.send().await.map_err(|e| std::io::Error::other(e.to_string()))?;
Ok(ReadCloser::new(std::io::Cursor::new(
res.body.collect().await.map(|data| data.into_bytes().to_vec())?,
)))
}
async fn remove(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
let client = self.client.clone();
let mut req = client.delete_object().bucket(&self.bucket).key(&self.get_dest(object));
if !rv.is_empty() {
req = req.version_id(rv);
}
req.send().await.map_err(|e| std::io::Error::other(e.to_string()))?;
Ok(())
}
async fn in_use(&self) -> Result<bool, std::io::Error> {
let client = self.client.clone();
let Ok(res) = client
.list_objects_v2()
.bucket(&self.bucket)
//.max_keys(10)
//.into_paginator()
.send()
.await
else {
return Err(std::io::Error::other("list_objects_v2 error"));
};
Ok(res.common_prefixes.unwrap_or_default().len() > 0 || res.contents.unwrap_or_default().len() > 0)
}
}
+421 -48
View File
@@ -32,15 +32,22 @@ use crate::diagnostics::get::{
GET_METADATA_CACHE_REASON_NOT_READ_DATA, GET_METADATA_CACHE_REASON_PART_NUMBER,
GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, GET_METADATA_CACHE_REASON_USABLE, GET_METADATA_CACHE_REASON_VERSION_ID,
GET_METADATA_CACHE_REASON_VERSION_SUSPENDED, GET_METADATA_CACHE_REASON_VERSIONED,
GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER,
GET_METADATA_EARLY_STOP_REASON_ERROR, GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM,
GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST,
GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM,
GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND, GET_METADATA_RESPONSE_CORRUPT, GET_METADATA_RESPONSE_DISK_NOT_FOUND,
GET_METADATA_RESPONSE_ERROR, GET_METADATA_RESPONSE_IGNORED, GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT,
GET_METADATA_RESPONSE_VALID, GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING,
GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_SET_DISK,
GET_STAGE_DECODE, GET_STAGE_METADATA_CACHE_LOOKUP, GET_STAGE_METADATA_RESOLVE, GET_STAGE_RANGE, GET_STAGE_READER_SETUP,
GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED,
GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER, GET_METADATA_EARLY_STOP_REASON_ERROR,
GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM, GET_METADATA_EARLY_STOP_REASON_NOT_FOUND,
GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST, GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM,
GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND,
GET_METADATA_RESPONSE_CORRUPT, GET_METADATA_RESPONSE_DISK_NOT_FOUND, GET_METADATA_RESPONSE_ERROR,
GET_METADATA_RESPONSE_IGNORED, GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT, GET_METADATA_RESPONSE_VALID,
GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING, GET_OBJECT_PATH_DIRECT_MEMORY,
GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE,
GET_STAGE_METADATA_CACHE_LOOKUP, GET_STAGE_METADATA_RESOLVE, GET_STAGE_RANGE, GET_STAGE_READER_SETUP,
GET_STAGE_READER_SETUP_DROP_PENDING, GET_STAGE_READER_SETUP_SCHEDULE, GET_STAGE_READER_SETUP_WAIT_QUORUM,
GET_STAGE_READER_TASK_BITROT_READER_INIT, GET_STAGE_READER_TASK_FILE_OPEN, GET_STAGE_READER_TASK_READER_CONSTRUCTION,
GetObjectFailureReason, classify_disk_error, get_stage_timer_if_enabled, record_get_object_pipeline_failure,
@@ -652,36 +659,49 @@ pub(in crate::set_disk) fn metadata_early_stop_candidate_matches(left: &FileInfo
&& left.erasure.distribution == right.erasure.distribution
}
pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
pub(in crate::set_disk) async fn data_read_early_stop_inline_body_miss_reason(
bucket: &str,
object: &str,
candidate: &FileInfo,
parts_metadata: &[FileInfo],
disks: &[Option<DiskStore>],
) -> bool {
if !candidate.inline_data()
|| candidate.is_compressed()
) -> Option<&'static str> {
// `inline_data` excludes remote objects; this diagnostic reports them separately.
if !rustfs_utils::http::contains_key_str(&candidate.metadata, rustfs_utils::http::SUFFIX_INLINE_DATA) {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE);
}
if candidate.is_compressed()
|| candidate
.metadata
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
|| candidate.is_remote()
|| candidate.deleted
|| candidate.size <= 0
|| candidate.parts.len() != 1
|| !candidate.has_valid_erasure_geometry()
{
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED);
}
if candidate.is_remote() {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE);
}
if candidate.deleted {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED);
}
if candidate.size <= 0 {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
}
if candidate.parts.len() != 1 {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
}
if !candidate.has_valid_erasure_geometry() {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
}
let Ok(object_size) = usize::try_from(candidate.size) else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
};
if candidate.parts.first().is_none_or(|part| part.size != object_size) {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
}
if !can_try_inline_data_shards_direct(object_size, candidate.erasure.block_size) {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
}
let Ok(erasure) = coding::Erasure::try_new_with_options(
@@ -690,18 +710,18 @@ pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
candidate.erasure.block_size,
candidate.uses_legacy_checksum,
) else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
};
let Some(data_files) =
collect_inline_data_shard_fileinfos_by_index(parts_metadata, candidate, erasure.data_shards, |index| {
let data_files =
match collect_inline_data_shard_fileinfos_by_index_or_reason(parts_metadata, candidate, erasure.data_shards, |index| {
disks.get(index).is_some_and(Option::is_some)
})
else {
return false;
};
}) {
Ok(data_files) => data_files,
Err(reason) => return Some(reason),
};
let Some(part) = candidate.parts.first() else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
};
let checksum_info = candidate.erasure.get_checksum_info(part.number);
let checksum_algo = if candidate.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
@@ -721,12 +741,70 @@ pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
let Ok(mut readers) =
build_inline_bitrot_readers_from_refs(&data_files, bucket, object, read_length, shard_size, &checksum_algo, false).await
else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY);
};
try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size)
.await
.is_some_and(|body| body.len() == object_size)
match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await {
Some(body) if body.len() == object_size => None,
_ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
}
}
fn data_read_inline_missing_shards_are_pending(
candidate: &FileInfo,
parts_metadata: &[FileInfo],
errors: &[Option<DiskError>],
disks: &[Option<DiskStore>],
fanout_order: &[usize],
scheduled_fanout_len: usize,
) -> bool {
let Ok(erasure) = coding::Erasure::try_new_with_options(
candidate.erasure.data_blocks,
candidate.erasure.parity_blocks,
candidate.erasure.block_size,
candidate.uses_legacy_checksum,
) else {
return false;
};
let distribution = &candidate.erasure.distribution;
let mut data_shards_seen_or_pending = vec![false; erasure.data_shards];
let mut missing_pending_data_shards = 0usize;
for (disk_index, file_info) in parts_metadata.iter().enumerate() {
let Some(&block_index) = distribution.get(disk_index) else {
return false;
};
if block_index == 0 || block_index > erasure.data_shards {
continue;
}
if !disks.get(disk_index).is_some_and(Option::is_some) {
return false;
}
let data_slot = block_index - 1;
if file_info.name.is_empty() {
let scheduled_and_not_failed = fanout_order
.get(..scheduled_fanout_len)
.is_some_and(|scheduled_disks| scheduled_disks.contains(&disk_index))
&& errors.get(disk_index).is_some_and(Option::is_none);
if scheduled_and_not_failed {
data_shards_seen_or_pending[data_slot] = true;
missing_pending_data_shards = missing_pending_data_shards.saturating_add(1);
continue;
}
return false;
}
if file_info.erasure.index != block_index
|| !file_info.has_valid_erasure_geometry()
|| !metadata_early_stop_candidate_matches(file_info, candidate)
|| file_info.data.as_ref().is_none_or(|data| data.is_empty())
{
return false;
}
data_shards_seen_or_pending[data_slot] = true;
}
missing_pending_data_shards > 0 && data_shards_seen_or_pending.into_iter().all(|seen_or_pending| seen_or_pending)
}
pub(in crate::set_disk) fn classify_metadata_response_error(err: &DiskError) -> &'static str {
@@ -2469,6 +2547,7 @@ impl SetDisks {
let mut next_fanout_index = 0usize;
let mut scheduled_count = 0usize;
let mut force_full_wait = false;
let mut final_miss_reason_override = None;
let spawn_read_version =
|join_set: &mut JoinSet<(usize, disk::error::Result<FileInfo>, Duration)>, index: usize, disk: Option<DiskStore>| {
let task_opts = opts;
@@ -2511,6 +2590,7 @@ impl SetDisks {
}
while let Some(result) = join_set.join_next().await {
let mut defer_pending_inline_data_shard = false;
match result {
Ok((index, res, elapsed)) => match res {
Ok(file_info) => {
@@ -2541,17 +2621,43 @@ impl SetDisks {
.or_else(|| accumulator.version_early_stop_decision())
{
let should_return_early = if read_data {
let allow_data_read_early_stop = match accumulator.candidate.as_ref() {
Some(candidate) => {
data_read_early_stop_inline_body_verified(bucket.as_ref(), object.as_ref(), candidate, &ress, disks)
.await
match accumulator.candidate.as_ref() {
Some(candidate) => match data_read_early_stop_inline_body_miss_reason(
bucket.as_ref(),
object.as_ref(),
candidate,
&ress,
disks,
)
.await
{
None => true,
Some(reason) => {
final_miss_reason_override = Some(reason);
if bounded_fanout
&& reason == GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD
&& data_read_inline_missing_shards_are_pending(
candidate,
&ress,
&errors,
disks,
&fanout_order,
next_fanout_index,
)
{
defer_pending_inline_data_shard = true;
} else {
force_full_wait = true;
}
false
}
},
None => {
force_full_wait = true;
final_miss_reason_override = Some(GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM);
false
}
None => false,
};
if !allow_data_read_early_stop {
force_full_wait = true;
}
allow_data_read_early_stop
} else {
true
};
@@ -2588,6 +2694,7 @@ impl SetDisks {
let pending_responses = join_set.len();
let should_hedge_single_pending_data_read = read_data
&& !force_full_wait
&& !defer_pending_inline_data_shard
&& pending_responses == 1
&& accumulator.can_still_reach_early_stop_with_pending(pending_responses);
if bounded_fanout && force_full_wait {
@@ -2600,6 +2707,7 @@ impl SetDisks {
next_fanout_index = next_fanout_index.saturating_add(1);
}
} else if bounded_fanout
&& !defer_pending_inline_data_shard
&& next_fanout_index < disks.len()
&& (!accumulator.can_still_reach_early_stop_with_pending(pending_responses)
|| should_hedge_single_pending_data_read)
@@ -2613,7 +2721,12 @@ impl SetDisks {
}
}
rustfs_io_metrics::record_get_object_metadata_early_stop_miss(metrics_path, accumulator.final_miss_reason());
let accumulator_miss_reason = accumulator.final_miss_reason();
let final_miss_reason = match (final_miss_reason_override, accumulator_miss_reason) {
(Some(reason), GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM) => reason,
_ => accumulator_miss_reason,
};
rustfs_io_metrics::record_get_object_metadata_early_stop_miss(metrics_path, final_miss_reason);
rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0);
rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0);
let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
@@ -5752,9 +5865,20 @@ mod tests {
object: &str,
payload: &[u8],
uses_legacy_checksum: bool,
) -> Vec<FileInfo> {
inline_metadata_fanout_fileinfos_with_geometry(bucket, object, payload, uses_legacy_checksum, 2, 2).await
}
async fn inline_metadata_fanout_fileinfos_with_geometry(
bucket: &str,
object: &str,
payload: &[u8],
uses_legacy_checksum: bool,
data_shards: usize,
parity_shards: usize,
) -> Vec<FileInfo> {
let distribution_key = metadata_distribution_key(bucket, object);
let mut base = FileInfo::new(&distribution_key, 2, 2);
let mut base = FileInfo::new(&distribution_key, data_shards, parity_shards);
base.volume = bucket.to_string();
base.name = object.to_string();
base.size = i64::try_from(payload.len()).expect("test payload should fit i64");
@@ -5817,6 +5941,21 @@ mod tests {
install_inline_metadata_fanout_files(disks, bucket, object, files).await;
}
async fn install_inline_metadata_fanout_fileinfo_with_geometry(
disks: &[Option<DiskStore>],
bucket: &str,
object: &str,
payload: &[u8],
data_shards: usize,
parity_shards: usize,
mutate: impl FnOnce(&mut [FileInfo]),
) {
let mut files =
inline_metadata_fanout_fileinfos_with_geometry(bucket, object, payload, false, data_shards, parity_shards).await;
mutate(&mut files);
install_inline_metadata_fanout_files(disks, bucket, object, files).await;
}
async fn install_inline_metadata_fanout_files(disks: &[Option<DiskStore>], bucket: &str, object: &str, files: Vec<FileInfo>) {
let distribution = files
.first()
@@ -6037,6 +6176,118 @@ mod tests {
drop(dirs);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn bounded_metadata_early_stop_waits_for_pending_inline_data_shard() {
const DISKS: usize = 6;
const DATA_SHARDS: usize = 4;
const PARITY_SHARDS: usize = 2;
let bucket = "bounded-inline-data-get-pending-shard-bucket";
let object =
object_with_initial_data_shards(bucket, "bounded-inline-data-get-pending-shard-object", DATA_SHARDS, DATA_SHARDS);
let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
install_inline_metadata_fanout_fileinfo_with_geometry(
&disks,
bucket,
&object,
b"verified inline payload",
DATA_SHARDS,
PARITY_SHARDS,
|_| {},
)
.await;
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
],
async {
let fanout_order = bounded_metadata_fanout_order(bucket, &object, DISKS, PARITY_SHARDS);
let distribution_key = metadata_distribution_key(bucket, &object);
let distribution = FileInfo::new(&distribution_key, DATA_SHARDS, PARITY_SHARDS)
.erasure
.distribution;
let paused_data_disk = *fanout_order
.iter()
.take(DATA_SHARDS)
.find(|disk_index| {
distribution
.get(**disk_index)
.is_some_and(|block_index| (1..=DATA_SHARDS).contains(block_index))
})
.expect("initial fanout should include a data shard to pause");
let hedged_parity_disk = fanout_order[DATA_SHARDS];
let unscheduled_parity_disk = fanout_order[DATA_SHARDS + 1];
let barrier = rename_fanout_barrier::arm(&object, paused_data_disk, rename_fanout_barrier::PHASE_READ_VERSION);
let tracker = rename_fanout_barrier::observe_tasks(&object);
let calls = disk_call_counters::observe(&object);
let disks_for_read = disks.clone();
let object_for_read = object.clone();
let mut read = tokio::spawn(async move {
SetDisks::read_all_fileinfo_observed(
&disks_for_read,
bucket,
bucket,
&object_for_read,
"",
true,
false,
false,
true,
PARITY_SHARDS,
)
.await
});
tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
.await
.expect("initial data shard should pause before returning");
tokio::time::timeout(BARRIER_PAUSE_GUARD, async {
while calls.for_disk(disk_call_counters::KIND_READ_VERSION, hedged_parity_disk) == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("bounded fanout should hedge one parity disk while the data shard is pending");
assert!(
tokio::time::timeout(BARRIER_PAUSE_GUARD, &mut read).await.is_err(),
"inline data-read early-stop must wait for a scheduled missing data shard instead of forcing full wait"
);
barrier.release();
let (parts_metadata, errs, diagnostics) = read
.await
.expect("metadata read task should not panic")
.expect("pending data shard should let the inline verifier finish");
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
5,
"pending data-shard defer should not schedule the final parity disk"
);
assert_eq!(
calls.for_disk(disk_call_counters::KIND_READ_VERSION, unscheduled_parity_disk),
0,
"the remaining parity disk must stay unissued when pending data verification succeeds"
);
assert_eq!(
tracker.running(),
0,
"early-stop should drain spawned read_version tasks before returning"
);
assert_eq!(diagnostics.total_responses(), 5);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == object).count(), 5);
assert!(errs.iter().all(Option::is_none));
},
)
.await;
drop(dirs);
}
#[tokio::test]
async fn data_read_early_stop_verifies_legacy_inline_checksum_payload() {
let bucket = "legacy-inline-data-get-fanout-bucket";
@@ -6067,11 +6318,133 @@ mod tests {
.clone();
assert!(
data_read_early_stop_inline_body_verified(bucket, object, &candidate, &parts_metadata, &disks).await,
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &parts_metadata, &disks)
.await
.is_none(),
"legacy inline metadata must use the legacy bitrot shard sizing and checksum algorithm"
);
}
#[tokio::test]
async fn data_read_early_stop_reports_inline_miss_reasons() {
let bucket = "inline-data-get-miss-reason-bucket";
let object = "inline-data-get-miss-reason-object";
let payload = b"verified inline payload";
let (_dirs, disks) = call_counter_local_disks(bucket, 4).await;
let files = inline_metadata_fanout_fileinfos_with_mode(bucket, object, payload, false).await;
let distribution = files
.first()
.map(|file| file.erasure.distribution.clone())
.expect("fixture should include metadata");
let order = bounded_metadata_fanout_order(bucket, object, 4, 2);
let mut parts_metadata = vec![FileInfo::default(); 4];
for disk_index in order.into_iter().take(3) {
let block_index = distribution
.get(disk_index)
.copied()
.expect("fixture distribution should cover every disk");
parts_metadata[disk_index] = files
.get(block_index.checked_sub(1).expect("erasure block indexes are one-based"))
.expect("fixture should include every distributed shard")
.clone();
}
let candidate = parts_metadata
.iter()
.find(|file| file.name == object)
.expect("fixture should include observed metadata")
.clone();
let data_disk = distribution
.iter()
.position(|block_index| *block_index == 1)
.expect("fixture distribution should include first data shard");
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &parts_metadata, &disks).await,
None
);
let mut not_inline = candidate.clone();
rustfs_utils::http::remove_str(&mut not_inline.metadata, rustfs_utils::http::SUFFIX_INLINE_DATA);
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &not_inline, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE)
);
let mut remote = candidate.clone();
remote.transition_status = TRANSITION_COMPLETE.to_string();
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &remote, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE)
);
let mut transformed = candidate.clone();
rustfs_utils::http::insert_str(&mut transformed.metadata, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &transformed, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED)
);
let mut deleted = candidate.clone();
deleted.deleted = true;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &deleted, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED)
);
let mut zero_size = candidate.clone();
zero_size.size = 0;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &zero_size, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE)
);
let mut multipart = candidate.clone();
multipart.parts.push(multipart.parts[0].clone());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &multipart, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE)
);
let mut invalid_geometry = candidate.clone();
invalid_geometry.erasure.data_blocks = 0;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &invalid_geometry, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY)
);
let mut missing_shard = parts_metadata.clone();
missing_shard[data_disk] = FileInfo::default();
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &missing_shard, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD)
);
let mut missing_payload = parts_metadata.clone();
missing_payload[data_disk].data = None;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &missing_payload, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD)
);
let mut identity_mismatch = parts_metadata.clone();
identity_mismatch[data_disk].version_id = Some(Uuid::new_v4());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &identity_mismatch, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH)
);
let mut corrupt = parts_metadata.clone();
if let Some(data) = corrupt[data_disk].data.as_mut() {
let mut corrupt_data = data.to_vec();
corrupt_data[0] ^= 0x01;
*data = Bytes::from(corrupt_data);
}
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &corrupt, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY)
);
}
#[test]
#[serial_test::serial]
fn metadata_fanout_lifecycle_records_real_early_stop_abort() {
@@ -6161,7 +6534,7 @@ mod tests {
&[
("path", GET_OBJECT_PATH_INTERNAL_META),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM),
("reason", GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
],
),
1,
@@ -6173,7 +6546,7 @@ mod tests {
&[
("path", GET_OBJECT_PATH_LEGACY_DUPLEX),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM),
("reason", GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
],
),
0,
+28 -8
View File
@@ -59,7 +59,10 @@ use crate::client::{object_api_utils::get_raw_etag, transition_api::ReaderImpl};
use crate::cluster::rpc::heal_bucket_local_on_disks;
use crate::data_usage::record_compression_total_memory;
use crate::diagnostics::get::{
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING,
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING,
GET_OBJECT_PATH_CODEC_STREAMING_LEGACY_ENGINE, GET_OBJECT_PATH_CODEC_STREAMING_RUSTFS_ENGINE, GET_OBJECT_PATH_DIRECT_MEMORY,
GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX,
GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT, GET_STAGE_INLINE_PREPARE,
@@ -3866,8 +3869,17 @@ fn collect_inline_data_shard_fileinfos_by_index<'a>(
parts_metadata: &'a [FileInfo],
fi: &FileInfo,
data_shards: usize,
mut disk_is_online: impl FnMut(usize) -> bool,
disk_is_online: impl FnMut(usize) -> bool,
) -> Option<Vec<&'a FileInfo>> {
collect_inline_data_shard_fileinfos_by_index_or_reason(parts_metadata, fi, data_shards, disk_is_online).ok()
}
fn collect_inline_data_shard_fileinfos_by_index_or_reason<'a>(
parts_metadata: &'a [FileInfo],
fi: &FileInfo,
data_shards: usize,
mut disk_is_online: impl FnMut(usize) -> bool,
) -> std::result::Result<Vec<&'a FileInfo>, &'static str> {
let distribution = &fi.erasure.distribution;
let mut data_files = vec![None; data_shards];
@@ -3875,27 +3887,35 @@ fn collect_inline_data_shard_fileinfos_by_index<'a>(
if !disk_is_online(disk_index) {
continue;
}
let block_index = *distribution.get(disk_index)?;
let Some(&block_index) = distribution.get(disk_index) else {
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
};
if block_index == 0 || block_index > data_shards {
continue;
}
if file_info.name.is_empty() {
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD);
}
if file_info.erasure.index != block_index {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH);
}
if !file_info.has_valid_erasure_geometry() {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
}
if !core::io_primitives::metadata_early_stop_candidate_matches(file_info, fi) {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH);
}
if file_info.data.as_ref().is_none_or(|data| data.is_empty()) {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD);
}
data_files[block_index - 1] = Some(file_info);
}
data_files.into_iter().collect()
data_files
.into_iter()
.collect::<Option<Vec<_>>>()
.ok_or(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD)
}
impl SetDisks {
+503 -20
View File
@@ -315,6 +315,41 @@ async fn get_object_reader_with_context(
GetObjectReader::new_with_resolver(reader, range, object_info, opts, headers, ctx.object_encryption_resolver()).await
}
async fn get_legacy_object_reader_with_context<R>(
ctx: &InstanceContext,
reader: R,
terminal: tokio::sync::oneshot::Receiver<Result<()>>,
range: Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
opts: &ObjectOptions,
headers: &HeaderMap<HeaderValue>,
) -> Result<(GetObjectReader, usize, i64)>
where
R: AsyncRead + Unpin + Send + Sync + 'static,
{
// ReadPlan validates this size below; failure here only keeps the terminal
// guard inside the transform until that validation returns its typed error.
let full_plaintext_size = object_info.get_actual_size().ok();
let whole_object = opts.part_number.is_none()
&& match (&range, full_plaintext_size) {
(None, _) => true,
(Some(range), Some(size)) => range
.get_offset_length(size)
.is_ok_and(|(offset, length)| offset == 0 && length == size),
(Some(_), None) => false,
};
let (source, terminal): (Box<dyn AsyncRead + Unpin + Send + Sync>, _) = if whole_object {
(Box::new(reader), Some(terminal))
} else {
(Box::new(LegacyDuplexProducerReader::new(reader, terminal)), None)
};
let (mut reader, offset, length) = get_object_reader_with_context(ctx, source, range, object_info, opts, headers).await?;
if let Some(terminal) = terminal {
reader.stream = Box::new(LegacyDuplexProducerReader::new(reader.stream, terminal));
}
Ok((reader, offset, length))
}
fn data_read_metadata_early_stop_request_shape_allowed(range: &Option<HTTPRangeSpec>, opts: &ObjectOptions) -> bool {
range.is_none()
&& opts.part_number.is_none()
@@ -899,6 +934,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
&object_info,
&opts,
&self.ctx.tier_config_mgr(),
self.ctx.object_encryption_resolver(),
)
.await?;
return Ok(finish_set_disk_read_lock(gr, read_lock_guard.take(), bucket, object));
@@ -1089,8 +1125,9 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
let (producer_terminal_tx, producer_terminal_rx) = tokio::sync::oneshot::channel();
let (mut reader, offset, length) =
get_object_reader_with_context(&self.ctx, Box::new(rd), range, &object_info, opts, &h).await?;
get_legacy_object_reader_with_context(&self.ctx, rd, producer_terminal_rx, range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its now-redundant
// lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -1110,7 +1147,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
// `get_object_with_fileinfo` also waits on `writer`, so an outer timeout
// would incorrectly treat downstream backpressure as disk-read latency.
// Disk read timeouts must be enforced at the actual disk I/O operations.
if let Err(e) = Self::get_object_with_fileinfo(
let producer_result = Self::get_object_with_fileinfo(
&bucket,
&object,
erasure_cache,
@@ -1128,9 +1165,9 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
object_class.as_str(),
size_bucket,
)
.await
{
let reason = classify_storage_error(&e);
.await;
if let Err(e) = &producer_result {
let reason = classify_storage_error(e);
if reason == GetObjectFailureReason::DownstreamClosed {
debug!(
event = EVENT_SET_DISK_WRITE,
@@ -1169,6 +1206,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
);
}
};
let _ = producer_terminal_tx.send(producer_result.map(|_| ()));
});
Ok(reader)
@@ -2557,6 +2595,420 @@ impl<R: AsyncRead + Unpin> AsyncRead for TransitionUploadReader<R> {
}
}
struct LegacyDuplexProducerReader<R> {
inner: Option<R>,
terminal: Option<tokio::sync::oneshot::Receiver<Result<()>>>,
inner_eof: bool,
}
impl<R> LegacyDuplexProducerReader<R> {
fn new(inner: R, terminal: tokio::sync::oneshot::Receiver<Result<()>>) -> Self {
Self {
inner: Some(inner),
terminal: Some(terminal),
inner_eof: false,
}
}
}
impl<R: AsyncRead + Unpin> AsyncRead for LegacyDuplexProducerReader<R> {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
if buf.remaining() == 0 {
return Poll::Ready(Ok(()));
}
if !self.inner_eof {
let before = buf.filled().len();
if let Some(inner) = self.inner.as_mut() {
match Pin::new(inner).poll_read(cx, buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),
Poll::Ready(Ok(())) if buf.filled().len() > before => return Poll::Ready(Ok(())),
Poll::Ready(Ok(())) => {
self.inner_eof = true;
self.inner = None;
}
}
} else {
self.inner_eof = true;
}
}
let Some(terminal) = self.terminal.as_mut() else {
return Poll::Ready(Ok(()));
};
match Pin::new(terminal).poll(cx) {
Poll::Pending => Poll::Pending,
Poll::Ready(Ok(Ok(()))) => {
self.terminal = None;
Poll::Ready(Ok(()))
}
Poll::Ready(Ok(Err(err))) => {
self.terminal = None;
Poll::Ready(Err(std::io::Error::other(err)))
}
Poll::Ready(Err(_)) => {
self.terminal = None;
Poll::Ready(Err(std::io::Error::other(StorageError::Unexpected)))
}
}
}
}
#[cfg(test)]
mod legacy_duplex_producer_reader_tests {
use super::*;
use crate::object_api::{EncryptionResolutionError, ObjectEncryptionResolver, ReadEncryptionMaterial, ReadEncryptionMode};
use rustfs_utils::CompressionAlgorithm;
use std::collections::HashMap;
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
const TEST_DUPLEX_CAPACITY: usize = 64 * 1024;
fn storage_error_source(error: &std::io::Error) -> &StorageError {
error
.get_ref()
.and_then(|source| source.downcast_ref::<StorageError>())
.expect("legacy duplex terminal error should retain StorageError source")
}
async fn compressed_fixture(plaintext: Vec<u8>, recorded_size: usize) -> (Vec<u8>, ObjectInfo) {
let mut compressor = rustfs_rio::CompressReader::new(std::io::Cursor::new(plaintext), CompressionAlgorithm::default());
let mut compressed = Vec::new();
compressor
.read_to_end(&mut compressed)
.await
.expect("compress test plaintext");
let mut metadata = HashMap::new();
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_COMPRESSION,
CompressionAlgorithm::default().to_string(),
);
rustfs_utils::http::insert_str(&mut metadata, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, recorded_size.to_string());
let object_info = ObjectInfo {
size: i64::try_from(compressed.len()).expect("compressed fixture length should fit in i64"),
user_defined: Arc::new(metadata),
..Default::default()
};
(compressed, object_info)
}
#[tokio::test]
async fn legacy_duplex_reader_allows_clean_completion() {
let (mut writer, reader) = tokio::io::duplex(64);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
writer
.write_all(b"complete")
.await
.expect("duplex write should fit in buffer");
drop(writer);
terminal_tx.send(Ok(())).expect("terminal receiver should remain installed");
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut out = Vec::new();
reader
.read_to_end(&mut out)
.await
.expect("clean producer completion should surface clean EOF");
assert_eq!(out, b"complete");
}
#[tokio::test]
async fn legacy_duplex_reader_ignores_zero_capacity_read_buf() {
let (mut writer, reader) = tokio::io::duplex(64);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
writer.write_all(b"body").await.expect("duplex write should fit in buffer");
drop(writer);
terminal_tx
.send(Err(StorageError::FileCorrupt))
.expect("terminal receiver should remain installed");
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut empty = [];
std::future::poll_fn(|cx| {
let mut read_buf = ReadBuf::new(&mut empty);
Pin::new(&mut reader).poll_read(cx, &mut read_buf)
})
.await
.expect("zero-capacity reads should complete without observing EOF or terminal state");
assert!(!reader.inner_eof);
assert!(reader.terminal.is_some());
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("subsequent reads must still receive data and the terminal error");
assert_eq!(out, b"body");
assert!(matches!(storage_error_source(&err), StorageError::FileCorrupt));
}
#[tokio::test]
async fn legacy_duplex_reader_surfaces_terminal_error_after_partial_data() {
let (mut writer, reader) = tokio::io::duplex(64);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
writer.write_all(b"partial").await.expect("duplex write should fit in buffer");
drop(writer);
terminal_tx
.send(Err(StorageError::FileCorrupt))
.expect("terminal receiver should remain installed");
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("terminal producer error must not become clean EOF");
assert_eq!(out, b"partial");
assert!(matches!(storage_error_source(&err), StorageError::FileCorrupt));
}
#[tokio::test]
async fn legacy_duplex_reader_surfaces_terminal_error_after_declared_length() {
let (mut writer, reader) = tokio::io::duplex(64);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
writer.write_all(b"exact").await.expect("duplex write should fit in buffer");
drop(writer);
terminal_tx
.send(Err(StorageError::Io(std::io::Error::new(
std::io::ErrorKind::ConnectionReset,
"remote body reset after final byte",
))))
.expect("terminal receiver should remain installed");
let reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut reader =
HashReader::from_stream(reader, 5, 5, None, None, false).expect("hash reader should accept exact declared length");
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("producer terminal error after the declared length must still fail");
assert_eq!(out, b"exact");
assert!(
matches!(storage_error_source(&err), StorageError::Io(io_error) if io_error.kind() == std::io::ErrorKind::ConnectionReset)
);
}
#[tokio::test]
async fn legacy_compressed_reader_surfaces_terminal_error_after_complete_plaintext() {
let plaintext = b"compressed terminal result must survive the plaintext limit".repeat(16);
let (compressed, object_info) = compressed_fixture(plaintext.clone(), plaintext.len()).await;
let full_range = HTTPRangeSpec {
is_suffix_length: false,
start: 0,
end: i64::try_from(plaintext.len()).expect("plaintext fixture length should fit in i64") - 1,
};
for range in [None, Some(full_range)] {
let (mut writer, reader) = tokio::io::duplex(compressed.len().max(1));
writer
.write_all(&compressed)
.await
.expect("compressed body should fit in duplex buffer");
drop(writer);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
terminal_tx
.send(Err(StorageError::FileCorrupt))
.expect("terminal receiver should remain installed");
let (mut reader, _, _) = get_legacy_object_reader_with_context(
&InstanceContext::new(),
reader,
terminal_rx,
range,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await
.expect("compressed read plan should build");
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("terminal error after complete decompression must not become clean EOF");
assert_eq!(out, plaintext);
assert!(matches!(storage_error_source(&err), StorageError::FileCorrupt));
}
}
#[tokio::test]
async fn legacy_exact_reader_rejects_extra_data_without_backpressure_deadlock() {
let payload = vec![0x5a; TEST_DUPLEX_CAPACITY * 2];
let (mut writer, reader) = tokio::io::duplex(TEST_DUPLEX_CAPACITY);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
let producer = tokio::spawn(async move {
let result = writer.write_all(&payload).await;
drop(writer);
let terminal_result = result
.as_ref()
.map(|_| ())
.map_err(|err| StorageError::Io(std::io::Error::new(err.kind(), err.to_string())));
let _ = terminal_tx.send(terminal_result);
result
});
let reader = crate::io_support::rio::HardLimitReader::new(reader, 1);
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut out = Vec::new();
tokio::time::timeout(std::time::Duration::from_secs(1), reader.read_to_end(&mut out))
.await
.expect("extra data beyond the declared size must not deadlock")
.expect_err("extra data beyond the declared size must fail closed");
assert_eq!(out, [0x5a]);
drop(reader);
let _ = tokio::time::timeout(std::time::Duration::from_secs(1), producer)
.await
.expect("producer must unblock after the read fails")
.expect("producer task should not panic");
}
#[tokio::test]
async fn legacy_terminal_reader_releases_unconsumed_source_before_waiting() {
let payload = vec![0x5a; TEST_DUPLEX_CAPACITY * 2];
let (mut writer, reader) = tokio::io::duplex(TEST_DUPLEX_CAPACITY);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
let producer = tokio::spawn(async move {
let result = writer.write_all(&payload).await;
drop(writer);
let terminal_result = result
.as_ref()
.map(|_| ())
.map_err(|err| StorageError::Io(std::io::Error::new(err.kind(), err.to_string())));
let _ = terminal_tx.send(terminal_result);
result
});
let reader = rustfs_rio::LimitReader::new(reader, 1);
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut out = Vec::new();
let err = tokio::time::timeout(std::time::Duration::from_secs(1), reader.read_to_end(&mut out))
.await
.expect("terminal wait must not deadlock behind unconsumed source data")
.expect_err("unconsumed source data must fail the producer terminal result");
assert_eq!(out, [0x5a]);
assert!(
matches!(storage_error_source(&err), StorageError::Io(io_error) if io_error.kind() == std::io::ErrorKind::BrokenPipe)
);
producer
.await
.expect("producer task should not panic")
.expect_err("source should close early");
}
struct FixedEncryptionResolver {
key_bytes: [u8; 32],
base_nonce: [u8; 12],
}
#[async_trait::async_trait]
impl ObjectEncryptionResolver for FixedEncryptionResolver {
async fn resolve_read_material(
&self,
_request: crate::object_api::ReadEncryptionRequest<'_>,
) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
Ok(Some(ReadEncryptionMaterial {
key_bytes: self.key_bytes,
mode: ReadEncryptionMode::Direct {
base_nonce: self.base_nonce,
},
}))
}
}
#[tokio::test]
async fn legacy_encrypted_reader_surfaces_terminal_error_after_complete_plaintext() {
let plaintext = b"encrypted terminal result must survive the plaintext limit".repeat(16);
let key_bytes = [0x31; 32];
let base_nonce = [0x42; 12];
let mut encryptor = rustfs_rio::EncryptReader::new(std::io::Cursor::new(plaintext.clone()), key_bytes, base_nonce);
let mut encrypted = Vec::new();
encryptor.read_to_end(&mut encrypted).await.expect("encrypt test plaintext");
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "encrypted-object".to_string(),
size: i64::try_from(encrypted.len()).expect("encrypted fixture length should fit in i64"),
user_defined: Arc::new(HashMap::from([
("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
(
"x-amz-server-side-encryption-customer-original-size".to_string(),
plaintext.len().to_string(),
),
])),
..Default::default()
};
let ctx = InstanceContext::new();
assert!(
ctx.set_object_encryption_resolver(Arc::new(FixedEncryptionResolver { key_bytes, base_nonce }))
.is_ok(),
"fresh context should accept resolver"
);
let full_range = HTTPRangeSpec {
is_suffix_length: false,
start: 0,
end: i64::try_from(plaintext.len()).expect("plaintext fixture length should fit in i64") - 1,
};
for range in [None, Some(full_range)] {
let (mut writer, reader) = tokio::io::duplex(encrypted.len().max(1));
writer
.write_all(&encrypted)
.await
.expect("encrypted body should fit in duplex buffer");
drop(writer);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
terminal_tx
.send(Err(StorageError::FileCorrupt))
.expect("terminal receiver should remain installed");
let (mut reader, _, _) = get_legacy_object_reader_with_context(
&ctx,
reader,
terminal_rx,
range,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await
.expect("encrypted read plan should build");
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("terminal error after complete decryption must not become clean EOF");
assert_eq!(out, plaintext);
assert!(matches!(storage_error_source(&err), StorageError::FileCorrupt));
}
}
#[tokio::test]
async fn legacy_duplex_reader_fails_closed_when_terminal_channel_closes() {
let (mut writer, reader) = tokio::io::duplex(64);
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel::<Result<()>>();
writer.write_all(b"body").await.expect("duplex write should fit in buffer");
drop(writer);
drop(terminal_tx);
let mut reader = LegacyDuplexProducerReader::new(reader, terminal_rx);
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("producer disappearance must fail closed");
assert_eq!(out, b"body");
assert!(matches!(storage_error_source(&err), StorageError::Unexpected));
}
}
struct TransitionUploadWriter<W> {
inner: W,
produced: u64,
@@ -6065,6 +6517,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
&oi,
&opts,
&self_.ctx.tier_config_mgr(),
self_.ctx.object_encryption_resolver(),
)
.await;
if let Err(err) = gr {
@@ -6134,6 +6587,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
&oi,
&part_opts,
&self_.ctx.tier_config_mgr(),
self_.ctx.object_encryption_resolver(),
)
.await
.map_err(StorageError::Io)?;
@@ -11180,7 +11634,7 @@ mod put_object_tmp_cleanup_tests {
use tokio::io::AsyncReadExt;
/// Large enough that the erasure shards are written as real tmp files
/// (never inlined into xl.meta), so both tests exercise actual cleanup.
/// (never inlined into xl.meta), so the cleanup tests exercise actual cleanup.
const TEST_OBJECT_SIZE: usize = 1 << 20;
/// Entries under `.rustfs.sys/tmp` on every disk, excluding the `.trash`
@@ -11204,6 +11658,18 @@ mod put_object_tmp_cleanup_tests {
leftovers
}
async fn wait_for_tmp_workspace_to_drain(temp_dirs: &[TempDir], failure_context: &str) {
let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
let leftovers = non_trash_tmp_entries(temp_dirs).await;
if leftovers.is_empty() {
break;
}
assert!(tokio::time::Instant::now() < deadline, "{failure_context}, leftovers: {leftovers:?}");
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
#[tokio::test]
async fn put_object_success_eventually_cleans_tmp_workspace() {
let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -11219,22 +11685,39 @@ mod put_object_tmp_cleanup_tests {
.await
.expect("put_object should succeed");
// The speculative cleanup runs on a spawned task off the PUT response
// path, so poll for the tmp workspace to drain instead of asserting
// immediately.
let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
let leftovers = non_trash_tmp_entries(&temp_dirs).await;
if leftovers.is_empty() {
break;
}
assert!(
tokio::time::Instant::now() < deadline,
"tmp workspace should drain after a successful PUT, leftovers: {leftovers:?}"
);
tokio::time::sleep(Duration::from_millis(25)).await;
wait_for_tmp_workspace_to_drain(&temp_dirs, "tmp workspace should drain after a successful PUT").await;
drop(temp_dirs);
}
#[tokio::test]
async fn cancelled_put_before_rename_cleans_tmp_workspace() {
let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "tmp-clean-cancelled-bucket";
let object = "cancelled-object";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
let barrier = PutObjectCommitBarrier::install(bucket, object, PutObjectCommitPause::AfterQuotaReservation);
let cancelled_set = set_disks.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(vec![8u8; TEST_OBJECT_SIZE]);
cancelled_set
.put_object(bucket, object, &mut reader, &ObjectOptions::default())
.await
});
barrier.wait_until_paused().await;
put.abort();
let join_error = put.await.expect_err("the paused PUT task must be cancelled");
assert!(join_error.is_cancelled(), "the paused PUT task must not panic");
// Keep the barrier armed so a detached child cannot proceed and hide
// missing cancellation cleanup.
wait_for_tmp_workspace_to_drain(&temp_dirs, "cancelling before rename should drain the tmp workspace").await;
drop(barrier);
drop(temp_dirs);
}
-81
View File
@@ -1,81 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
impl ECStore {
#[instrument(level = "trace", skip(self))]
#[allow(clippy::too_many_arguments)]
pub(super) async fn handle_list_objects_v2(
self: Arc<Self>,
bucket: &str,
prefix: &str,
continuation_token: Option<String>,
delimiter: Option<String>,
max_keys: i32,
fetch_owner: bool,
start_after: Option<String>,
incl_deleted: bool,
) -> Result<ListObjectsV2Info> {
self.inner_list_objects_v2(
bucket,
prefix,
continuation_token,
delimiter,
max_keys,
fetch_owner,
start_after,
incl_deleted,
)
.await
}
#[instrument(skip(self))]
pub(super) async fn handle_list_object_versions(
self: Arc<Self>,
bucket: &str,
prefix: &str,
marker: Option<String>,
version_marker: Option<String>,
delimiter: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
self.inner_list_object_versions(bucket, prefix, marker, version_marker, delimiter, max_keys)
.await
}
pub(crate) async fn list_object_versions_for_lifecycle(
self: Arc<Self>,
bucket: &str,
prefix: &str,
marker: Option<String>,
version_marker: Option<String>,
delimiter: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
self.inner_list_object_versions_for_lifecycle(bucket, prefix, marker, version_marker, delimiter, max_keys)
.await
}
pub(super) async fn handle_walk(
self: Arc<Self>,
rx: CancellationToken,
bucket: &str,
prefix: &str,
result: tokio::sync::mpsc::Sender<ObjectInfoOrErr>,
opts: WalkOptions,
) -> Result<()> {
self.walk_internal(rx, bucket, prefix, result, opts).await
}
}
+1 -1
View File
@@ -3845,7 +3845,7 @@ impl ECStore {
.await
}
pub(crate) async fn inner_list_object_versions_for_lifecycle(
pub(crate) async fn list_object_versions_for_lifecycle(
self: Arc<Self>,
bucket: &str,
prefix: &str,
+3 -4
View File
@@ -148,7 +148,6 @@ mod heal_walk;
pub use heal_walk::HealWalkVersion;
mod init;
pub(crate) mod init_format;
mod list;
pub(crate) mod list_objects;
mod multipart;
mod object;
@@ -601,7 +600,7 @@ impl crate::storage_api_contracts::list::ListOperations for ECStore {
start_after: Option<String>,
incl_deleted: bool,
) -> Result<ListObjectsV2Info> {
self.handle_list_objects_v2(
self.inner_list_objects_v2(
bucket,
prefix,
continuation_token,
@@ -624,7 +623,7 @@ impl crate::storage_api_contracts::list::ListOperations for ECStore {
delimiter: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
self.handle_list_object_versions(bucket, prefix, marker, version_marker, delimiter, max_keys)
self.inner_list_object_versions(bucket, prefix, marker, version_marker, delimiter, max_keys)
.await
}
@@ -636,7 +635,7 @@ impl crate::storage_api_contracts::list::ListOperations for ECStore {
result: tokio::sync::mpsc::Sender<ObjectInfoOrErr>,
opts: WalkOptions,
) -> Result<()> {
self.handle_walk(rx, bucket, prefix, result, opts).await
self.walk_internal(rx, bucket, prefix, result, opts).await
}
}
@@ -157,6 +157,18 @@ fn ecstore_implements_storage_list_operations_contract() {
assert!(storage_list_operations_type_name::<ECStore>().ends_with("::ECStore"));
}
#[test]
fn ecstore_pools_expose_storage_list_operations_contract() {
fn assert_contract(store: &ECStore) {
let future = store.pools[0]
.clone()
.list_objects_v2("bucket", "", None, None, 1, false, None, false);
drop(future);
}
let _ = assert_contract;
}
#[test]
fn ecstore_implements_storage_multipart_operations_contract() {
assert!(storage_multipart_operations_type_name::<ECStore>().ends_with("::ECStore"));
+1 -4
View File
@@ -135,10 +135,7 @@ impl FileMeta {
let i = buf.len() as u64;
// check version, buf = buf[8..]
let (buf, _, _) = Self::check_xl2_v1(buf).map_err(|e| {
error!("failed to check XL2 v1 format: {}", e);
e
})?;
let (buf, _, _) = Self::check_xl2_v1(buf)?;
if buf.len() < 5 {
error!(
+7 -2
View File
@@ -82,8 +82,8 @@ impl Error {
/// Whether a heal operation can be retried without changing its inputs.
pub(crate) fn is_recoverable_heal(&self) -> bool {
match self {
Error::TaskCancelled => false,
Error::TaskTimeout | Error::TransientSkip { .. } => true,
Error::TaskCancelled | Error::TaskTimeout => false,
Error::TransientSkip { .. } => true,
Error::Storage(err) => {
err.is_quorum_error()
|| matches!(
@@ -165,4 +165,9 @@ mod tests {
assert!(Error::Storage(EcstoreError::DiskNotFound).is_recoverable_heal());
assert!(Error::Storage(EcstoreError::VolumeNotFound).is_recoverable_heal());
}
#[test]
fn task_timeout_is_terminal() {
assert!(!Error::TaskTimeout.is_recoverable_heal());
}
}
+30 -20
View File
@@ -673,6 +673,12 @@ fn retry_request_for_result(task: &HealTask, result: &Result<()>) -> Option<(Hea
Some((request, delay, error))
}
async fn retry_request_for_result_with_budget(task: &HealTask, result: &Result<()>) -> Option<(HealRequest, Duration, String)> {
let (_, delay, error) = retry_request_for_result(task, result)?;
let request = task.retry_request_with_remaining_timeout().await.ok()?;
Some((request, delay, error))
}
fn recoverable_heal_retry_delay(retry_attempt: u32) -> Duration {
let retry_attempt = retry_attempt.clamp(1, 5);
let delay = Duration::from_secs(2_u64.saturating_pow(retry_attempt));
@@ -690,7 +696,7 @@ pub struct HealConfig {
pub max_concurrent_heals: usize,
/// Maximum concurrent heal tasks allowed for a single erasure set
pub max_concurrent_per_set: usize,
/// Task timeout
/// Aggregate task execution timeout across recoverable retries
pub task_timeout: Duration,
/// Queue size
pub queue_size: usize,
@@ -3106,7 +3112,7 @@ impl HealManager {
"Heal scheduler task started"
);
let result = task.execute().await;
let retry_request = retry_request_for_result(task.as_ref(), &result);
let retry_request = retry_request_for_result_with_budget(task.as_ref(), &result).await;
match &result {
Ok(_) => {
debug!(
@@ -4539,6 +4545,25 @@ mod tests {
assert!(retry_error.contains("Lock acquisition timeout"));
}
#[tokio::test]
async fn retry_request_for_result_preserves_remaining_timeout_budget() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
let mut request = HealRequest::object("retry-transition".to_string(), "object".to_string(), None);
request.options.timeout = Some(Duration::from_secs(60));
let task = HealTask::from_request(request, storage);
let result = task.execute().await;
let (retry_request, _, _) = retry_request_for_result_with_budget(&task, &result)
.await
.expect("read quorum failure should retain the unused timeout budget");
let remaining = retry_request
.options
.timeout
.expect("configured timeout should remain present");
assert!(remaining < Duration::from_secs(60));
assert!(remaining > Duration::from_secs(59));
}
#[test]
fn test_retry_request_for_incomplete_heal_rename() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -6054,7 +6079,7 @@ mod tests {
process_manager_queue_once(&manager).await;
let defaulted_status = tokio::time::timeout(Duration::from_secs(1), async {
loop {
if let Ok(status @ HealTaskStatus::Retrying { .. }) = manager.get_task_status(&defaulted_id).await {
if let Ok(status @ HealTaskStatus::Timeout) = manager.get_task_status(&defaulted_id).await {
break status;
}
tokio::task::yield_now().await;
@@ -6062,23 +6087,8 @@ mod tests {
})
.await
.expect("configured timeout should finish the task");
assert!(matches!(defaulted_status, HealTaskStatus::Retrying { .. }));
assert_eq!(
manager
.retrying_heals
.lock()
.await
.get(&defaulted_id)
.expect("timed out task should retain its retry request")
.request
.options
.timeout,
Some(Duration::ZERO)
);
manager
.cancel_task(&defaulted_id)
.await
.expect("retrying timeout task should be cancelled");
assert_eq!(defaulted_status, HealTaskStatus::Timeout);
assert!(manager.retrying_heals.lock().await.get(&defaulted_id).is_none());
let mut explicit = bucket_request("explicit-timeout", HealPriority::Normal, HealRequestSource::Admin);
explicit.options.timeout = Some(Duration::from_secs(60));
+39 -1
View File
@@ -196,7 +196,7 @@ pub struct HealOptions {
/// Whether to skip namespace locking
#[serde(default)]
pub no_lock: bool,
/// Timeout
/// Aggregate execution timeout across recoverable manager retries
pub timeout: Option<Duration>,
/// pool index
pub pool_index: Option<usize>,
@@ -442,6 +442,14 @@ impl HealTask {
}
}
pub(crate) async fn retry_request_with_remaining_timeout(&self) -> Result<HealRequest> {
let mut request = self.retry_request();
if self.options.timeout.is_some() {
request.options.timeout = self.remaining_timeout().await?;
}
Ok(request)
}
pub(crate) fn from_replacement_recovery_request(
request: HealRequest,
storage: Arc<dyn HealStorageAPI>,
@@ -2657,6 +2665,36 @@ mod tests {
use super::super::storage_api::status::BucketInfo;
#[tokio::test]
async fn retry_request_carries_remaining_timeout_budget() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage::default());
let mut request = HealRequest::bucket("bucket".to_string());
request.options.timeout = Some(Duration::from_secs(100));
let task = HealTask::from_request(request, storage.clone());
*task.task_start_instant.write().await = Some(Instant::now() - Duration::from_secs(40));
let retry = task
.retry_request_with_remaining_timeout()
.await
.expect("first retry should retain the unused timeout budget");
let first_remaining = retry.options.timeout.expect("configured timeout should remain present");
assert!(first_remaining <= Duration::from_secs(60));
assert!(first_remaining > Duration::from_secs(59));
let retry_task = HealTask::from_request(retry, storage);
*retry_task.task_start_instant.write().await = Some(Instant::now() - Duration::from_secs(20));
let second_retry = retry_task
.retry_request_with_remaining_timeout()
.await
.expect("second retry should retain only the unused aggregate budget");
let second_remaining = second_retry
.options
.timeout
.expect("configured timeout should remain present");
assert!(second_remaining <= Duration::from_secs(40));
assert!(second_remaining > Duration::from_secs(39));
}
#[test]
fn format_result_requires_every_requested_target_to_be_ok() {
let result = HealResultItem {
-2
View File
@@ -225,8 +225,6 @@ async fn nothing_readable_leaves_the_bundle_unwrapped() {
"artifact {} carries the raw on-disk record",
artifact.path
);
// A cheap structural check too: an encrypted payload is not JSON.
assert_ne!(payload.first(), Some(&b'{'), "artifact {} looks like plaintext JSON", artifact.path);
}
// The manifest itself is not encrypted, so assert directly that it carries
+1 -5
View File
@@ -103,11 +103,7 @@ pub(super) fn rules() -> Vec<Rule> {
P2Degraded,
"heal",
"heal 任务调度/执行失败",
any([
prefix("Heal task timeout"),
prefix("Heal task execution failed"),
contains("Heal manager is not running"),
]),
any([prefix("Heal task timeout"), prefix("Heal task execution failed")]),
"heal 任务调度/执行层故障。",
"检查 heal 后台服务状态与资源压力。",
)
+32 -1
View File
@@ -520,6 +520,14 @@ struct FailureSample {
pub struct FailStats {
pub count: i64,
pub size: i64,
/// Rolling-window snapshots refreshed at collection time
/// ([`Self::refresh_windows`]). The raw samples (`recent`) are process
/// local (serde-skipped), so these fields are what survives the peer-RPC
/// wire and [`Self::merge`]-based cluster aggregation.
#[serde(default)]
pub last_minute: FailedMetric,
#[serde(default)]
pub last_hour: FailedMetric,
#[serde(skip)]
recent: VecDeque<FailureSample>,
}
@@ -537,6 +545,17 @@ impl FailStats {
self.prune(observed_at);
}
/// Recompute the serializable rolling-window snapshots from the local
/// samples. Called at the collection point (per-node stats snapshot),
/// never on the failure hot path — the two deque scans are O(window) and
/// `add_size` runs under the bucket-stats write lock. Only meaningful on
/// the live per-node struct: a deserialized or merged struct has no
/// samples, and refreshing it would wipe the aggregated windows.
pub fn refresh_windows(&mut self) {
self.last_minute = self.recent_since(Duration::from_secs(60));
self.last_hour = self.recent_since(Duration::from_secs(3600));
}
fn prune(&mut self, observed_at: Instant) {
while self
.recent
@@ -565,6 +584,16 @@ impl FailStats {
Self {
count: self.count.saturating_add(other.count),
size: self.size.saturating_add(other.size),
// The window snapshots sum across nodes; the raw samples do not
// travel and stay empty on aggregated structs.
last_minute: FailedMetric {
count: self.last_minute.count.saturating_add(other.last_minute.count),
size: self.last_minute.size.saturating_add(other.last_minute.size),
},
last_hour: FailedMetric {
count: self.last_hour.count.saturating_add(other.last_hour.count),
size: self.last_hour.size.saturating_add(other.last_hour.size),
},
recent: VecDeque::new(),
}
}
@@ -636,7 +665,9 @@ impl BucketReplicationStat {
}
pub fn update_xfer_rate(&mut self, size: i64, duration: Duration) {
if size > 1024 * 1024 {
// Same boundary as the worker-pool split and minio-go's
// Large/Small transfer-summary labels: >= 128 MiB is "large".
if size >= crate::runtime::MIN_LARGE_OBJ_SIZE {
self.xfer_rate_lrg.add_size(size, duration);
} else {
self.xfer_rate_sml.add_size(size, duration);
+361 -51
View File
@@ -71,6 +71,7 @@ where
/// Optional: allow users to customize block_size
pub fn with_block_size(inner: R, block_size: usize, compression_algorithm: CompressionAlgorithm) -> Self {
debug_assert!(block_size > 0, "CompressReader block_size must be non-zero");
Self {
inner,
buffer: Vec::new(),
@@ -183,11 +184,21 @@ pin_project! {
buffer: Vec<u8>,
buffer_pos: usize,
finished: bool,
// A previously surfaced stream error is sticky: without this, a caller
// that polls again after an error would restart at the header phase and
// read a truncated tail as a clean EOF, converting the error into a
// silently short body.
poisoned: bool,
// Fields for saving header read progress across polls
header_buf: [u8; 8],
header_read: usize,
header_done: bool,
// Fields for saving compressed block read progress across polls
// Fields for saving compressed block read progress across polls.
// `compressed_len > 0` means a block payload is in flight: the header has
// been fully parsed and `compressed_read` bytes of the payload are already
// consumed from the inner stream. The header phase must not run again (and
// must not reset `compressed_read`) until this block completes, or a
// `Poll::Pending` in the middle of a payload would silently drop the bytes
// read so far and desynchronize the block framing.
compressed_buf: Vec<u8>,
compressed_read: usize,
compressed_len: usize,
@@ -205,9 +216,9 @@ where
buffer: Vec::new(),
buffer_pos: 0,
finished: false,
poisoned: false,
header_buf: [0u8; 8],
header_read: 0,
header_done: false,
compressed_buf: Vec::new(),
compressed_read: 0,
compressed_len: 0,
@@ -236,54 +247,74 @@ where
if *this.finished {
return Poll::Ready(Ok(()));
}
// Read header
while !*this.header_done && *this.header_read < HEADER_LEN {
let mut temp = [0u8; HEADER_LEN];
let mut temp_buf = ReadBuf::new(&mut temp[0..HEADER_LEN - *this.header_read]);
match this.inner.as_mut().poll_read(cx, &mut temp_buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Ok(())) => {
let n = temp_buf.filled().len();
if n == 0 {
break;
if *this.poisoned {
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "decompress reader previously failed")));
}
if *this.compressed_len == 0 {
// Read the 8-byte block header, resuming across polls via `header_read`.
while *this.header_read < HEADER_LEN {
let mut temp = [0u8; HEADER_LEN];
let mut temp_buf = ReadBuf::new(&mut temp[0..HEADER_LEN - *this.header_read]);
match this.inner.as_mut().poll_read(cx, &mut temp_buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Ok(())) => {
let n = temp_buf.filled().len();
if n == 0 {
if *this.header_read == 0 {
// Clean EOF on a block boundary.
*this.finished = true;
return Poll::Ready(Ok(()));
}
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"unexpected EOF while reading compressed block header",
)));
}
this.header_buf[*this.header_read..*this.header_read + n].copy_from_slice(&temp_buf.filled()[..n]);
*this.header_read += n;
}
Poll::Ready(Err(e)) => {
// error!("DecompressReader poll_read: read header error: {e}");
*this.poisoned = true;
return Poll::Ready(Err(e));
}
this.header_buf[*this.header_read..*this.header_read + n].copy_from_slice(&temp_buf.filled()[..n]);
*this.header_read += n;
}
Poll::Ready(Err(e)) => {
// error!("DecompressReader poll_read: read header error: {e}");
return Poll::Ready(Err(e));
}
}
if *this.header_read < HEADER_LEN {
return Poll::Pending;
}
}
if !*this.header_done && *this.header_read == 0 {
return Poll::Ready(Ok(()));
}
let typ = this.header_buf[0];
let len = (this.header_buf[1] as usize) | ((this.header_buf[2] as usize) << 8) | ((this.header_buf[3] as usize) << 16);
let crc = (this.header_buf[4] as u32)
| ((this.header_buf[5] as u32) << 8)
| ((this.header_buf[6] as u32) << 16)
| ((this.header_buf[7] as u32) << 24);
*this.header_read = 0;
*this.header_done = true;
if typ == COMPRESS_TYPE_END {
let typ = this.header_buf[0];
let len =
(this.header_buf[1] as usize) | ((this.header_buf[2] as usize) << 8) | ((this.header_buf[3] as usize) << 16);
*this.header_read = 0;
// `CompressReader` never emits an end block — a stream terminates on
// inner EOF, which is what lets concatenated per-part streams decode as
// one. This branch is kept for streams that do carry the marker.
if typ == COMPRESS_TYPE_END {
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.finished = true;
return Poll::Ready(Ok(()));
}
if typ != COMPRESS_TYPE_COMPRESSED && typ != COMPRESS_TYPE_UNCOMPRESSED {
// error!("DecompressReader unknown compression type: {typ}");
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Unknown compression type")));
}
if len == 0 {
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Invalid compressed block length")));
}
if this.compressed_buf.len() < len {
this.compressed_buf.resize(len, 0);
}
*this.compressed_len = len;
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.finished = true;
return Poll::Ready(Ok(()));
}
if this.compressed_buf.len() < len {
this.compressed_buf.resize(len, 0);
}
*this.compressed_len = len;
*this.compressed_read = 0;
// Fill the in-flight block payload, resuming across polls via `compressed_read`.
while *this.compressed_read < *this.compressed_len {
let mut temp_buf = ReadBuf::new(&mut this.compressed_buf[*this.compressed_read..*this.compressed_len]);
match this.inner.as_mut().poll_read(cx, &mut temp_buf) {
@@ -291,7 +322,13 @@ where
Poll::Ready(Ok(())) => {
let n = temp_buf.filled().len();
if n == 0 {
break;
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"unexpected EOF while reading compressed block payload",
)));
}
*this.compressed_read += n;
}
@@ -299,10 +336,17 @@ where
// error!("DecompressReader poll_read: read compressed block error: {e}");
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(e));
}
}
}
let typ = this.header_buf[0];
let crc = (this.header_buf[4] as u32)
| ((this.header_buf[5] as u32) << 8)
| ((this.header_buf[6] as u32) << 16)
| ((this.header_buf[7] as u32) << 24);
let compressed_buf = &this.compressed_buf[..*this.compressed_len];
// `compressed_buf`'s length comes from the untrusted 24-bit header length field, so it
// can be shorter than 16 bytes. `uvarint` is safe on any slice length (reads at most 10
@@ -316,6 +360,7 @@ where
if uvarint <= 0 || uvarint as usize > compressed_buf.len() {
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Invalid compressed block length prefix")));
}
let compressed_data = &compressed_buf[uvarint as usize..];
@@ -326,21 +371,29 @@ where
// error!("DecompressReader decompress_block error: {e}");
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(e));
}
}
} else if typ == COMPRESS_TYPE_UNCOMPRESSED {
compressed_data.to_vec()
} else {
// error!("DecompressReader unknown compression type: {typ}");
// The header phase already rejected every type other than
// COMPRESS_TYPE_COMPRESSED / COMPRESS_TYPE_UNCOMPRESSED.
compressed_data.to_vec()
};
if decompressed.is_empty() {
// The writer never emits zero-length plaintext blocks; an empty
// decode surfacing as Ready(Ok) with no bytes would read as EOF and
// silently truncate the stream.
*this.poisoned = true;
*this.compressed_read = 0;
*this.compressed_len = 0;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Unknown compression type")));
};
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Empty compressed block")));
}
if decompressed.len() != uncompress_len as usize {
// error!("DecompressReader decompressed length mismatch: {} != {}", decompressed.len(), uncompress_len);
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "Decompressed length mismatch")));
}
let actual_crc = {
@@ -352,13 +405,13 @@ where
// error!("DecompressReader CRC32 mismatch: actual {actual_crc} != expected {crc}");
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.poisoned = true;
return Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, "CRC32 mismatch")));
}
*this.buffer = decompressed;
*this.buffer_pos = 0;
*this.compressed_read = 0;
*this.compressed_len = 0;
*this.header_done = false;
let to_copy = min(buf.remaining(), this.buffer.len());
buf.put_slice(&this.buffer[..to_copy]);
*this.buffer_pos += to_copy;
@@ -493,6 +546,184 @@ mod tests {
assert_eq!(&decompressed, &data);
}
/// Wraps a reader so every other poll returns `Poll::Pending` and every
/// `Ready` poll serves at most `chunk` bytes. This is the shape a duplex
/// pipe produces when the erasure writer is slower than the decoder, which
/// is exactly what desynchronized the block framing before the resumable
/// payload state was added (rustfs/rustfs#5957 multipart GET truncation).
struct PendingChunkReader<R> {
inner: R,
chunk: usize,
pending_next: bool,
}
impl<R> PendingChunkReader<R> {
fn new(inner: R, chunk: usize) -> Self {
Self {
inner,
chunk,
pending_next: true,
}
}
}
impl<R: AsyncRead + Unpin> AsyncRead for PendingChunkReader<R> {
fn poll_read(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
if self.pending_next {
self.pending_next = false;
cx.waker().wake_by_ref();
return std::task::Poll::Pending;
}
self.pending_next = true;
let cap = self.chunk.min(buf.remaining());
let mut scratch = vec![0u8; cap];
let mut inner_buf = tokio::io::ReadBuf::new(&mut scratch);
match std::pin::Pin::new(&mut self.inner).poll_read(cx, &mut inner_buf) {
std::task::Poll::Ready(Ok(())) => {
buf.put_slice(inner_buf.filled());
std::task::Poll::Ready(Ok(()))
}
other => other,
}
}
}
fn patterned_payload(size: usize, seed: u8) -> Vec<u8> {
(0..size)
.map(|i| ((i as u64).wrapping_mul(2_654_435_761).wrapping_add(seed as u64) >> 3) as u8)
.collect()
}
/// Root-cause regression for the multipart compressed GET truncation: a
/// `Poll::Pending` in the middle of a block payload must not drop the bytes
/// already consumed. Before the resumable payload state, the decoder reset
/// `compressed_read` on every re-poll and surfaced
/// `LZ4 error: ERROR_frameType_unknown` mid-stream.
#[tokio::test]
async fn test_decompress_reader_survives_pending_mid_payload() {
let data = patterned_payload(100 * 1024, 7);
let mut compress_reader =
CompressReader::with_block_size(Cursor::new(data.clone()), 8192, CompressionAlgorithm::default());
let mut compressed = Vec::new();
compress_reader.read_to_end(&mut compressed).await.unwrap();
for chunk in [1usize, 3, 7, 8, 17, 1000, 8192] {
let inner = PendingChunkReader::new(Cursor::new(compressed.clone()), chunk);
let mut decompress_reader = DecompressReader::new(inner, CompressionAlgorithm::default());
let mut decompressed = Vec::new();
decompress_reader.read_to_end(&mut decompressed).await.unwrap();
assert_eq!(decompressed, data, "pending-chunked decode must be byte-exact for chunk={chunk}");
}
}
/// Two independently compressed streams concatenated back to back — the
/// on-disk shape of a compressed multipart object — must decode across the
/// stream boundary even when every poll can suspend mid-block.
#[tokio::test]
async fn test_decompress_reader_survives_pending_across_concatenated_streams() {
let part1 = patterned_payload(64 * 1024, 7);
let part2 = patterned_payload(24 * 1024, 61);
let mut stored = Vec::new();
for part in [&part1, &part2] {
let mut compress_reader =
CompressReader::with_block_size(Cursor::new(part.clone()), 8192, CompressionAlgorithm::default());
let mut compressed = Vec::new();
compress_reader.read_to_end(&mut compressed).await.unwrap();
stored.extend_from_slice(&compressed);
}
let mut expected = part1;
expected.extend_from_slice(&part2);
for chunk in [1usize, 5, 8, 13, 4096] {
let inner = PendingChunkReader::new(Cursor::new(stored.clone()), chunk);
let mut decompress_reader = DecompressReader::new(inner, CompressionAlgorithm::default());
let mut decompressed = Vec::new();
decompress_reader.read_to_end(&mut decompressed).await.unwrap();
assert_eq!(
decompressed, expected,
"concatenated part streams must decode byte-exact for chunk={chunk}"
);
}
}
/// After the first stream error, every further poll must keep failing.
/// Without the sticky poison a retrying caller would restart at the header
/// phase and read the truncated tail as a clean EOF — converting a hard
/// error into a silently short body.
#[tokio::test]
async fn test_decompress_reader_error_is_sticky() {
let data = patterned_payload(32 * 1024, 7);
let mut compress_reader = CompressReader::with_block_size(Cursor::new(data), 8192, CompressionAlgorithm::default());
let mut compressed = Vec::new();
compress_reader.read_to_end(&mut compressed).await.unwrap();
compressed.truncate(compressed.len() - 3);
let mut decompress_reader = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
let mut out = Vec::new();
let first = decompress_reader
.read_to_end(&mut out)
.await
.expect_err("truncated payload must error");
assert_eq!(first.kind(), std::io::ErrorKind::UnexpectedEof);
let mut retry = Vec::new();
let second = decompress_reader
.read_to_end(&mut retry)
.await
.expect_err("a poll after the first error must not turn into a clean EOF");
assert_eq!(second.kind(), std::io::ErrorKind::InvalidData);
assert!(retry.is_empty(), "no bytes may be produced after the stream failed");
}
/// A stream cut off in the middle of a block payload must fail with a clean
/// UnexpectedEof instead of decoding a short buffer.
#[tokio::test]
async fn test_decompress_reader_truncated_payload_is_unexpected_eof() {
let data = patterned_payload(32 * 1024, 7);
let mut compress_reader = CompressReader::with_block_size(Cursor::new(data), 8192, CompressionAlgorithm::default());
let mut compressed = Vec::new();
compress_reader.read_to_end(&mut compressed).await.unwrap();
compressed.truncate(compressed.len() - 3);
let mut decompress_reader = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
let mut out = Vec::new();
let err = decompress_reader
.read_to_end(&mut out)
.await
.expect_err("truncated payload must error");
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
}
/// A stream cut off in the middle of a block header must fail with a clean
/// UnexpectedEof instead of parsing a garbage header.
#[tokio::test]
async fn test_decompress_reader_truncated_header_is_unexpected_eof() {
let data = patterned_payload(12 * 1024, 7);
let mut compress_reader = CompressReader::with_block_size(Cursor::new(data), 8192, CompressionAlgorithm::default());
let mut compressed = Vec::new();
compress_reader.read_to_end(&mut compressed).await.unwrap();
// Keep the first full block plus 3 bytes of the next header.
let ln = (compressed[1] as usize) | ((compressed[2] as usize) << 8) | ((compressed[3] as usize) << 16);
let first_block_end = 8 + ln;
assert!(compressed.len() > first_block_end, "fixture must contain more than one block");
compressed.truncate(first_block_end + 3);
let mut decompress_reader = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
let mut out = Vec::new();
let err = decompress_reader
.read_to_end(&mut out)
.await
.expect_err("truncated header must error");
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
}
// Regression: a corrupted block whose 24-bit length field is < 16 must not panic.
// Header layout (HEADER_LEN = 8): [type, len_lo, len_mid, len_hi, crc0..crc3], then `len`
// bytes of block body. Pre-fix, poll_read sliced `compressed_buf[0..16]` unconditionally,
@@ -518,6 +749,85 @@ mod tests {
assert_eq!(res.unwrap_err().kind(), std::io::ErrorKind::InvalidData);
}
// Header-level fail-closed matrix, built by hand so the decoder is exercised against bytes no
// encoder in this crate can produce. Header layout (HEADER_LEN = 8):
// [type, len_lo, len_mid, len_hi, crc0..crc3], then `len` body bytes = uvarint(plain_len) + data.
#[tokio::test]
async fn test_decompress_reader_header_validation_matrix() {
// Build a block whose body is `uvarint(plain.len()) + plain` (i.e. the
// COMPRESS_TYPE_UNCOMPRESSED shape), with the header CRC taken over the plaintext exactly
// like the production writer does.
fn build_raw_block(typ: u8, plain: &[u8], len_override: Option<usize>) -> Vec<u8> {
let crc = {
let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
hasher.update(plain);
hasher.finalize() as u32
};
let mut uvarint_buf = [0u8; 10];
let int_len = put_uvarint(&mut uvarint_buf[..], plain.len() as u64);
let body_len = int_len + plain.len();
let len = len_override.unwrap_or(body_len);
let mut out = Vec::with_capacity(HEADER_LEN + body_len);
out.push(typ);
out.push((len & 0xFF) as u8);
out.push(((len >> 8) & 0xFF) as u8);
out.push(((len >> 16) & 0xFF) as u8);
out.extend_from_slice(&crc.to_le_bytes());
out.extend_from_slice(&uvarint_buf[..int_len]);
out.extend_from_slice(plain);
out
}
let plain = b"uncompressed passthrough payload";
// (a) A well-formed uncompressed block decodes to the plaintext verbatim.
let mut out = Vec::new();
DecompressReader::new(
Cursor::new(build_raw_block(COMPRESS_TYPE_UNCOMPRESSED, plain, None)),
CompressionAlgorithm::default(),
)
.read_to_end(&mut out)
.await
.expect("a well-formed uncompressed block must decode");
assert_eq!(out.as_slice(), plain.as_slice());
// (b) An unknown block type must be rejected instead of being treated as passthrough.
let mut out = Vec::new();
let err = DecompressReader::new(Cursor::new(build_raw_block(0x7E, plain, None)), CompressionAlgorithm::default())
.read_to_end(&mut out)
.await
.expect_err("unknown compression type must error");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Unknown compression type"), "got: {err}");
// (c) A zero-length block would stall the decoder, so it must be rejected up front.
let mut out = Vec::new();
let err = DecompressReader::new(
Cursor::new(build_raw_block(COMPRESS_TYPE_UNCOMPRESSED, plain, Some(0))),
CompressionAlgorithm::default(),
)
.read_to_end(&mut out)
.await
.expect_err("zero-length block must error");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Invalid compressed block length"), "got: {err}");
// (d) A block that decodes to zero plaintext bytes must be rejected: the
// writer never emits empty blocks, and an empty decode surfacing as
// Ready(Ok) with no bytes would read as EOF and silently truncate.
let mut out = Vec::new();
let err = DecompressReader::new(
Cursor::new(build_raw_block(COMPRESS_TYPE_UNCOMPRESSED, b"", None)),
CompressionAlgorithm::default(),
)
.read_to_end(&mut out)
.await
.expect_err("empty block must error");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Empty compressed block"), "got: {err}");
}
// Directly exercises the length-prefix guard: an unterminated varint (all continuation bytes)
// makes `uvarint` return 0, which must be rejected as an invalid length prefix.
#[tokio::test]
+81 -28
View File
@@ -24,12 +24,17 @@ pin_project! {
#[pin]
pub inner: R,
remaining: i64,
scratch: Vec<u8>,
}
}
impl<R> HardLimitReader<R> {
pub fn new(inner: R, limit: i64) -> Self {
HardLimitReader { inner, remaining: limit }
HardLimitReader {
inner,
remaining: limit,
scratch: Vec::new(),
}
}
}
@@ -37,19 +42,21 @@ impl<R> AsyncRead for HardLimitReader<R>
where
R: AsyncRead,
{
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<Result<()>> {
if self.remaining < 0 {
fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<Result<()>> {
let mut this = self.project();
if *this.remaining < 0 {
return Poll::Ready(Err(Error::other("input provided more bytes than specified")));
}
let original_filled = buf.filled().len();
if self.remaining == 0 {
if buf.remaining() == 0 {
return Poll::Ready(Ok(()));
}
if *this.remaining == 0 {
let mut discard = [0u8; 8192];
let mut discard_buf = ReadBuf::new(&mut discard);
return match self.as_mut().project().inner.poll_read(cx, &mut discard_buf) {
return match this.inner.as_mut().poll_read(cx, &mut discard_buf) {
Poll::Pending => Poll::Pending,
Poll::Ready(Ok(())) => {
if discard_buf.filled().is_empty() {
debug_assert_eq!(buf.filled().len(), original_filled);
Poll::Ready(Ok(()))
} else {
Poll::Ready(Err(Error::other("input provided more bytes than specified")))
@@ -58,30 +65,46 @@ where
Poll::Ready(Err(err)) => Poll::Ready(Err(err)),
};
}
// Save the initial length
let before = original_filled;
// Poll the inner reader
let this = self.as_mut().project();
let poll = this.inner.poll_read(cx, buf);
if let Poll::Ready(Ok(())) = &poll {
let after = buf.filled().len();
let read = (after - before) as i64;
if read == 0 && *this.remaining > 0 {
return Poll::Ready(Err(Error::new(
std::io::ErrorKind::UnexpectedEof,
IncompleteBody {
remaining: *this.remaining,
},
)));
let remaining = match usize::try_from(*this.remaining) {
Ok(remaining) => remaining,
Err(_) => usize::MAX,
};
let allowed = remaining.min(buf.remaining());
let read = if allowed == buf.remaining() {
let before = buf.filled().len();
match this.inner.as_mut().poll_read(cx, buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),
Poll::Ready(Ok(())) => buf.filled().len() - before,
}
*this.remaining -= read;
if *this.remaining < 0 {
return Poll::Ready(Err(Error::other("input provided more bytes than specified")));
} else {
this.scratch.resize(allowed, 0);
let mut scratch_buf = ReadBuf::new(&mut this.scratch[..allowed]);
match this.inner.as_mut().poll_read(cx, &mut scratch_buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),
Poll::Ready(Ok(())) => {
let read = scratch_buf.filled().len();
buf.put_slice(scratch_buf.filled());
read
}
}
};
if read == 0 {
return Poll::Ready(Err(Error::new(
std::io::ErrorKind::UnexpectedEof,
IncompleteBody {
remaining: *this.remaining,
},
)));
}
poll
let read = match i64::try_from(read) {
Ok(read) => read,
Err(_) => return Poll::Ready(Err(Error::other("read count exceeds i64::MAX"))),
};
*this.remaining -= read;
Poll::Ready(Ok(()))
}
}
@@ -140,7 +163,12 @@ mod tests {
assert!(err.is_some());
let err = err.unwrap();
assert_eq!(err.kind(), std::io::ErrorKind::Other);
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
assert!(
err.get_ref()
.and_then(|source| source.downcast_ref::<std::io::Error>())
.is_some_and(|source| source.to_string().contains("more bytes than specified"))
);
}
#[tokio::test]
@@ -155,6 +183,17 @@ mod tests {
assert_eq!(&buf, data);
}
#[tokio::test]
async fn test_hardlimit_reader_zero_capacity_read_does_not_consume_input() {
let mut reader = HardLimitReader::new(BufReader::new(&b"abc"[..]), 3);
let mut empty = [];
assert_eq!(reader.read(&mut empty).await.expect("zero-capacity read should succeed"), 0);
let mut out = Vec::new();
reader.read_to_end(&mut out).await.expect("input should remain readable");
assert_eq!(out, b"abc");
}
#[tokio::test]
async fn test_hardlimit_reader_short_input_returns_unexpected_eof() {
let data = b"abc";
@@ -195,4 +234,18 @@ mod tests {
assert_eq!(err.kind(), std::io::ErrorKind::Other);
assert!(err.to_string().contains("more bytes than specified"));
}
#[tokio::test]
async fn test_hardlimit_reader_caps_each_read_before_reporting_extra_bytes() {
let mut reader = HardLimitReader::new(BufReader::new(&b"abcdef"[..]), 3);
let mut out = Vec::new();
let err = reader
.read_to_end(&mut out)
.await
.expect_err("bytes beyond the declared limit must be rejected");
assert_eq!(out, b"abc");
assert!(err.to_string().contains("more bytes than specified"));
}
}
+52 -8
View File
@@ -138,6 +138,12 @@ impl std::fmt::Display for InternodeHttpErrorKind {
}
}
#[derive(thiserror::Error, Debug, Clone, Copy, Eq, PartialEq)]
#[error("internode body stalled for {timeout:?}")]
pub struct BodyStalled {
pub timeout: Duration,
}
#[derive(Debug, Clone, Eq, PartialEq)]
pub struct InternodeHttpRequestContext {
method: String,
@@ -271,6 +277,10 @@ pub fn internode_http_timeout_error(method: &Method, url: &str) -> io::Error {
internode_kind_error(method, url, internode_rpc_operation(url), InternodeHttpErrorKind::ConnectTimeout)
}
fn body_stalled_error(stall_timeout: Duration) -> io::Error {
Error::new(io::ErrorKind::TimedOut, BodyStalled { timeout: stall_timeout })
}
/// Clone an internode HTTP I/O error while retaining its structured classification.
///
/// The underlying transport source is intentionally omitted because it is not
@@ -1085,10 +1095,7 @@ impl AsyncRead for HttpReader {
);
record_internode_stall_timeout(*this.track_internode_metrics, *this.internode_operation);
record_internode_error(*this.track_internode_metrics, *this.internode_operation);
Poll::Ready(Err(Error::new(
io::ErrorKind::TimedOut,
"HttpReader stall timeout: no data received before deadline",
)))
Poll::Ready(Err(body_stalled_error(stall_timeout)))
} else {
Poll::Pending
}
@@ -1217,10 +1224,7 @@ impl ChunkReader for HttpChunkReader {
);
record_internode_stall_timeout(*this.track_internode_metrics, *this.internode_operation);
record_internode_error(*this.track_internode_metrics, *this.internode_operation);
return Poll::Ready(Err(Error::new(
io::ErrorKind::TimedOut,
"HttpReader stall timeout: no data received before deadline",
)));
return Poll::Ready(Err(body_stalled_error(stall_timeout)));
}
return Poll::Pending;
}
@@ -2379,6 +2383,46 @@ mod tests {
Err(err) => err,
};
assert_eq!(err.kind(), io::ErrorKind::TimedOut);
let stalled = err
.get_ref()
.and_then(|source| source.downcast_ref::<BodyStalled>())
.expect("stall timeout should retain typed body-stalled source");
assert_eq!(stalled.timeout, Duration::from_millis(20));
handle.abort();
}
#[tokio::test]
async fn http_chunk_reader_stall_timeout_retains_typed_source() {
let state = TestState::default();
let Some((base_url, handle)) = start_test_server(state).await else {
return;
};
let url = base_url.replace("/stream", "/stall");
let mut reader =
HttpChunkReader::new_with_stall_timeout(url, Method::GET, HeaderMap::new(), None, Some(Duration::from_millis(20)))
.await
.expect("chunk reader should open");
let first = std::future::poll_fn(|cx| Pin::new(&mut reader).poll_read_chunk(cx, 64))
.await
.expect("initial body chunk should arrive")
.expect("initial body chunk should not be EOF");
assert_eq!(first, b"hello"[..]);
let err = tokio::time::timeout(
Duration::from_secs(1),
std::future::poll_fn(|cx| Pin::new(&mut reader).poll_read_chunk(cx, 64)),
)
.await
.expect("stall timeout should wake chunk reader")
.expect_err("chunk reader should return a timeout error");
assert_eq!(err.kind(), io::ErrorKind::TimedOut);
let stalled = err
.get_ref()
.and_then(|source| source.downcast_ref::<BodyStalled>())
.expect("chunk stall timeout should retain typed body-stalled source");
assert_eq!(stalled.timeout, Duration::from_millis(20));
handle.abort();
}
+1 -1
View File
@@ -102,7 +102,7 @@ bytes.workspace = true
hex-simd.workspace = true
[dev-dependencies]
tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
serial_test = { workspace = true }
temp-env = { workspace = true }
tempfile = { workspace = true }
+111 -12
View File
@@ -65,6 +65,7 @@ const LOG_SUBSYSTEM_FOLDER: &str = "folder";
const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
const LOG_SUBSYSTEM_HEAL: &str = "heal";
const EVENT_SCANNER_FOLDER_STATE: &str = "scanner_folder_state";
const EVENT_SCANNER_METADATA_CORRUPT: &str = "scanner_metadata_corrupt";
const EVENT_SCANNER_LIFECYCLE_ACTION: &str = "scanner_lifecycle_action";
const EVENT_SCANNER_HEAL_ADMISSION: &str = "scanner_heal_admission";
const EVENT_SCANNER_ALERT_STATE: &str = "scanner_alert_state";
@@ -2154,17 +2155,34 @@ impl FolderScanner {
self.record_failed(&item.path);
if should_log_failed_object(into.failed_objects) {
warn!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
path = %item.path,
failed_objects = into.failed_objects,
state = "get_size_failed",
error = %e,
"Scanner folder failed to get object size"
);
if let GetSizeFailureAction::HealMetadata { object } = &failure_action {
error!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_METADATA_CORRUPT,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
drive = %self.local_disk.path().display(),
bucket = %item.bucket,
object = %object,
metadata_path = %item.path,
failed_objects = into.failed_objects,
state = "metadata_corrupt",
error = %e,
"Scanner detected corrupt object metadata"
);
} else {
warn!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
path = %item.path,
failed_objects = into.failed_objects,
state = "get_size_failed",
error = %e,
"Scanner folder failed to get object size"
);
}
}
}
@@ -3054,12 +3072,59 @@ mod tests {
use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass};
use rustfs_filemeta::{FileInfo, FileMeta};
use serial_test::serial;
use std::io::Write;
#[cfg(unix)]
use std::os::unix::fs::{PermissionsExt, symlink};
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use temp_env::{with_var, with_var_unset};
use tracing_subscriber::fmt::MakeWriter;
use uuid::Uuid;
#[derive(Clone, Default)]
struct CapturedLogs {
buffer: Arc<Mutex<Vec<u8>>>,
}
struct CapturedLogWriter {
buffer: Arc<Mutex<Vec<u8>>>,
}
impl CapturedLogs {
fn contents(&self) -> String {
let buffer = self
.buffer
.lock()
.expect("captured logs mutex should not be poisoned")
.clone();
String::from_utf8(buffer).expect("captured logs should be valid UTF-8")
}
}
impl Write for CapturedLogWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.buffer
.lock()
.expect("captured logs mutex should not be poisoned")
.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
impl<'a> MakeWriter<'a> for CapturedLogs {
type Writer = CapturedLogWriter;
fn make_writer(&'a self) -> Self::Writer {
CapturedLogWriter {
buffer: Arc::clone(&self.buffer),
}
}
}
#[test]
fn scanner_size_summary_application_saturates_usage_counters() {
let target = "arn:minio:replication::target".to_string();
@@ -4542,9 +4607,19 @@ mod tests {
assert!(budget.entries_visited() >= 1);
}
#[tokio::test]
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
.json()
.with_max_level(tracing::Level::ERROR)
.with_writer(logs.clone())
.with_ansi(false)
.without_time()
.finish();
let _subscriber_guard = tracing::subscriber::set_default(subscriber);
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -4596,6 +4671,30 @@ mod tests {
assert!(!budget.budget_elapsed());
assert_eq!(budget.reason(), None);
let captured = logs.contents();
assert!(
!captured.contains("failed to check XL2 v1 format"),
"the context-free filemeta parser error must not be emitted"
);
let events = captured
.lines()
.map(|line| serde_json::from_str::<serde_json::Value>(line).expect("captured scanner log should be valid JSON"))
.filter(|line| line["fields"]["event"] == EVENT_SCANNER_METADATA_CORRUPT)
.collect::<Vec<_>>();
assert_eq!(
events.len(),
1,
"one corrupt metadata observation must emit one scanner-owned diagnostic event"
);
let fields = &events[0]["fields"];
assert_eq!(fields["component"], LOG_COMPONENT_SCANNER);
assert_eq!(fields["subsystem"], LOG_SUBSYSTEM_FOLDER);
assert_eq!(fields["drive"], temp_dir.to_string_lossy().as_ref());
assert_eq!(fields["bucket"], "bucket");
assert_eq!(fields["object"], "object");
assert_eq!(fields["metadata_path"], metadata_path.to_string_lossy().as_ref());
assert_eq!(fields["state"], "metadata_corrupt");
let retry_budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
crate::scanner_budget::ScannerCycleBudgetConfig {
+1 -11
View File
@@ -3849,17 +3849,6 @@ impl ScannerIODisk for Disk {
let fivs = match meta.get_file_info_versions(item.bucket.as_str(), item.object_path().as_str(), false) {
Ok(versions) => versions,
Err(e) => {
error!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_DISK_BUCKET_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
bucket = %item.bucket,
object = %item.object_path(),
state = "file_info_versions_failed",
error = %e,
"Scanner disk bucket failed to resolve file info versions"
);
return Err(scanner_metadata_corrupt_error(
format!("failed to resolve file info versions: {e}"),
&item.bucket,
@@ -4262,6 +4251,7 @@ mod tests {
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("bucket should be removed from the first pool only");
init_bucket_metadata_sys_for_scanner_tests(store.clone()).await;
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
+29
View File
@@ -50,6 +50,14 @@ pub const SUFFIX_SOURCE_DELETEMARKER: &str = "source-deletemarker";
pub const SUFFIX_SOURCE_PROXY_REQUEST: &str = "source-proxy-request";
pub const SUFFIX_SOURCE_REPLICATION_REQUEST: &str = "source-replication-request";
pub const SUFFIX_SOURCE_REPLICATION_CHECK: &str = "source-replication-check";
// LWW timestamps for replicated tag/retention/legal-hold modifications. MinIO
// declares these with mixed case (internal/http/headers.go:
// X-Minio-Source-Replication-Tagging-Timestamp / -Retention-Timestamp /
// -LegalHold-Timestamp); HTTP header names compare case-insensitively, so the
// lowercase suffix forms interoperate. Values are RFC3339 on the wire.
pub const SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP: &str = "source-replication-tagging-timestamp";
pub const SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP: &str = "source-replication-retention-timestamp";
pub const SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP: &str = "source-replication-legalhold-timestamp";
pub const SUFFIX_REPLICATION_SSEC_CRC: &str = "replication-ssec-crc";
/// Returns true if the key is object-encryption metadata understood by RustFS or MinIO.
@@ -196,6 +204,27 @@ mod tests {
assert_eq!(get_object_encryption_original_size(&metadata).expect("valid size"), Some(42));
}
#[test]
fn replication_timestamp_headers_match_minio_wire_names() {
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, "2026-01-02T03:04:05Z");
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, "2026-01-02T03:04:06Z");
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, "2026-01-02T03:04:07Z");
// The exact names MinIO's object-api-options.go reads (its Get()
// canonicalizes case, so a case-insensitive match is wire-equivalent).
for name in [
"X-Minio-Source-Replication-Tagging-Timestamp",
"X-Minio-Source-Replication-Retention-Timestamp",
"X-Minio-Source-Replication-LegalHold-Timestamp",
"x-rustfs-source-replication-tagging-timestamp",
"x-rustfs-source-replication-retention-timestamp",
"x-rustfs-source-replication-legalhold-timestamp",
] {
assert!(headers.contains_key(name), "replication timestamp header {name} must be written");
}
}
#[test]
fn test_get_header() {
let mut headers = HeaderMap::new();
-3
View File
@@ -659,9 +659,6 @@ mod test {
// Port should be in valid range (u16 max is always <= 65535)
assert!(port1 > 0);
assert!(port2 > 0);
// Different calls should typically return different ports
assert_ne!(port1, port2);
}
#[test]
@@ -33,6 +33,7 @@ for later deletion.
- `tonic-013-status-render` peer RPC failure classification: internode failures that reach a node only as text (a peer's error_info payload, a status flattened through format!) are classified by matching the rendering of an Unavailable gRPC status. Releases up to 1.0.0-alpha.38 shipped tonic 0.13, which rendered that status as "status: Unavailable, message: ..."; tonic 0.14 renders it as "code: 'The service is currently unavailable', message: ...". Both forms are matched so an older peer's relayed text still marks an unreachable peer offline. Remove the tonic 0.13 form after the minimum supported RustFS peer version ships tonic 0.14 or later.
- `rustfs-5063` pre-beta.9 Local KMS recovery: persisted Local KMS configs from beta.8 and earlier predate the explicit insecure-development flag, and encrypted key files use the legacy SHA-256 KDF. Remove the config fallback after supported upgrades have rewritten or explicitly resaved all pre-beta.9 configs with the development-default field, and remove the legacy KDF after supported upgrades have rewritten all pre-beta.9 Local KMS key files with explicit at-rest protection.
- `sse-local-dek-json-v1` legacy local SSE DEK decoding: releases before the JSON envelope wrote wrapped DEKs as `base64(nonce):base64(ciphertext)`, so readers retain that decoder while all new writes use the versioned JSON envelope. Remove the colon decoder after the minimum supported direct-upgrade release writes JSON envelopes and migration tooling has rewritten every retained legacy object.
- `multipart-compression-default-off-window` staged multipart disk-compression rollout: releases before the resumable legacy decompressor fail transient reads of compressed objects under mid-payload suspension, so multipart uploads advertise the compression marker only when RUSTFS_COMPRESSION_MULTIPART_ENABLED is set in addition to RUSTFS_COMPRESSION_ENABLED, keeping rolling upgrades from creating new compressed multipart objects while pre-fix nodes may still serve reads. Flip the default to enabled (and retire the extra switch) after the minimum supported direct-upgrade release ships the resumable decompressor.
## Review Checklist
+1 -1
View File
@@ -111,7 +111,7 @@ inventory. Generic function-local names such as `CACHE`, `LOCK`, `INIT`, and
| `GET_OBJECT_BUFFER_THRESHOLD_WARNED`, `GET_READER_STREAM_BUFFER_SIZE_OVERRIDE`, function-local `ENABLED`, `OBJECT_SEEK_SUPPORT_THRESHOLD`, `OBJECT_SEEK_SUPPORT_CONCURRENCY_THRESHOLDS` | `rustfs/src/app/object_usecase.rs` | Cache or constant / owner-local cache | Object GET/seek tuning caches and warning guards stay private to object usecase helpers. |
| `SUPPORTED_HEADERS` | `rustfs/src/storage/options.rs` | Cache or constant / owner-local constant | Supported-header lookup state stays private to storage option parsing. |
| `AUDIT_TARGET_SPECS`, `NOTIFICATION_TARGET_SPECS` | `rustfs/src/admin/handlers/audit.rs`, `rustfs/src/admin/handlers/event.rs`, `rustfs/src/admin/handlers/plugins_instances.rs` | Cache or constant / owner-local constant | Admin target descriptor tables stay private to their handler owners. |
| `SITE_REPLICATION_PEER_CLIENT`, `SITE_REPLICATION_STATE_LOCK` | `rustfs/src/admin/handlers/site_replication.rs` | Process-global owner-local cache / guard | Site-replication peer client cache and state lock stay private to site-replication handlers. |
| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/admin/handlers/site_replication.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to site-replication handlers. The state RMW transaction holds no process-local mutex — see `rustfs/src/admin/site_replication_state.rs`. |
| `AUDIT_MODULE_ENABLED`, `NOTIFY_MODULE_ENABLED`, `PERSISTED_NOTIFY_MODULE_ENABLED`, `PERSISTED_AUDIT_MODULE_ENABLED`, `PERSISTED_MODULE_SWITCH_CONFIGURED` | `rustfs/src/server/audit.rs`, `rustfs/src/server/event.rs`, `rustfs/src/server/module_switch.rs` | Process-global owner-local toggles | Audit/notify module snapshots stay private to the server module switch owners. |
| `DELETE_TAIL_TOTAL`, `DELETE_CLEANUP_TOTAL`, `DELETE_REPLICATION_TOTAL`, `DELETE_NOTIFY_TOTAL` | `rustfs/src/delete_tail_activity.rs` | Process-global owner-local counters | Delete-tail activity counters stay private behind delete-tail activity helpers. |
| `EMBEDDED_SERVER_STARTED` | `rustfs/src/startup_lifecycle.rs` | Process-global owner-local guard | Embedded startup single-start protection stays private to startup lifecycle. |
@@ -61,17 +61,17 @@ catalog extension.
| Area | Status | Covered behavior |
|---|---|---|
| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and route capabilities. |
| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and only the supported OpenAPI REST paths in `endpoints`. RustFS administration, maintenance, migration, diagnostics, refs, and metadata-location extensions remain available but are not presented as standard Iceberg REST endpoints. |
| Table bucket discovery | Supported | `PUT` and `GET /v1/buckets/{warehouse}` enable and inspect table bucket state. |
| Namespaces | Supported | Create, list, load, existence check, and drop namespace routes are registered on both catalog prefixes. List responses support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Namespace identifiers are limited to 512 ASCII characters so persisted paths and stateless continuation tokens remain bounded. |
| Tables | Supported | Create, register, list, load, existence check, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. |
| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. |
| Tables | Supported | Create, register, list, load, existence check, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Commit identifiers must match the URL resource; unknown requirements, updates, and snapshot operations fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. Standard statistics, partition statistics, and schema/spec cleanup updates are accepted. |
| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Externally supplied metadata transitions preserve monotonic column, partition, and sequence assignment watermarks and immutable definitions for retained schemas, partition specs, sort orders, and snapshots. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. The catalog does not advertise `idempotency-key-lifetime`; clients must treat standard mutation-wide `Idempotency-Key` semantics as unsupported. |
| Commit recovery | Supported | Commit log, idempotency lookup, diagnostics, and recovery routes expose staged/finalization gaps and repair safe idempotency gaps without moving the table pointer. |
| Snapshot refs | Supported | Refs can be listed, created or replaced, and deleted through catalog commits. `main` is protected and refs with explicit retention require forced delete. |
| Iceberg views | Supported | Basic create, list, load, replace, existence check, and drop routes persist view metadata with view-scoped authorization. |
| Table credentials endpoint | Supported | Returns an empty `storage-credentials` list by default. Returns table-scoped temporary credentials only when credential vending is enabled. |
| Iceberg views | Supported | Basic create, list, load, replace, existence check, and drop routes persist view metadata with view-scoped authorization. Replace identifiers must match the URL resource, `schema-id: -1` resolves to the last added schema, one commit timestamp is used consistently, and only Iceberg view format version 1 is accepted. |
| Table credentials endpoint | Supported | Returns an empty `storage-credentials` list by default. Returns table-scoped temporary credentials only when credential vending is enabled. Credential responses set `Cache-Control: no-store, private`, `Pragma: no-cache`, and `Expires: 0`. |
| Catalog diagnostics and export | Supported | Exposes recovery state, consistency state, backing manifest, recoverable commit-log WAL state, strong backing migration target, single-active-writer policy, and scale validation matrix. |
| Catalog import and rollback | Supported | Import/register and rollback use catalog validation and commit paths rather than direct pointer mutation. |
| Catalog import and rollback | Supported | Import/register and online rollback use catalog validation and commit paths rather than direct pointer mutation. Online rollback accepts only a forward-safe metadata target that preserves assignment watermarks and retained definitions. Restoring an older target that lowers those watermarks is an offline disaster-recovery operation and requires every writer to be stopped. |
| External catalog bridge | Supported operator path | Operator-supplied metadata pointer sync/import is supported for external catalog identity boundaries. Online vendor SDK polling and policy mirroring are not claimed. |
| Multi-table transactions | Not claimed | RustFS currently claims single-table commit atomicity only. |
+1 -1
View File
@@ -59,7 +59,7 @@
{
default = rustPlatform.buildRustPackage {
pname = "rustfs";
version = "1.0.0-rc.1";
version = "1.0.0-rc.2";
src = ./.;
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: rustfs
description: RustFS helm chart to deploy RustFS on kubernetes cluster.
type: application
version: "1.0.0-rc.1"
appVersion: "1.0.0-rc.1"
version: "1.0.0-rc.2"
appVersion: "1.0.0-rc.2"
home: https://rustfs.com
icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
maintainers:
+5 -2
View File
@@ -1,9 +1,9 @@
%global _enable_debug_packages 0
%global _empty_manifest_terminate_build 0
%global prerelease rc.1
%global prerelease rc.2
Name: rustfs
Version: 1.0.0
Release: rc.1
Release: rc.2
Summary: High-performance distributed object storage for MinIO alternative
License: Apache-2.0
@@ -58,6 +58,9 @@ install %_builddir/%{name}-%{version}-%{prerelease}/target/%_arch/%_arch-unknown
%_bindir/rustfs
%changelog
* Fri Aug 14 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.0-rc.2
* Sat Aug 08 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.0-rc.1
+2 -1
View File
@@ -278,6 +278,8 @@ rustfs-signer.workspace = true
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["raw_value"] }
serde_urlencoded = { workspace = true }
snap.workspace = true
zstd.workspace = true
# Cryptography and Security
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
@@ -355,7 +357,6 @@ rcgen = { workspace = true }
rustfs-test-utils.workspace = true
# diagnose_e2e fixtures (archives are generated in-test, never checked in)
zip = { workspace = true }
zstd = { workspace = true }
# Enables the shared MockWarmBackend / xl.meta assertion helpers exposed via
# the ecstore `api::tier::test_util` facade module (rustfs/backlog#1148 ilm-6).
rustfs-ecstore = { workspace = true, features = ["test-util"] }
+8 -7
View File
@@ -40,7 +40,7 @@ use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::LazyLock;
use tracing::{Span, error, info, warn};
use tracing::{error, info, warn};
const LOG_COMPONENT_ADMIN_API: &str = "admin_api";
const LOG_SUBSYSTEM_AUDIT_TARGET: &str = "audit_target";
@@ -278,8 +278,6 @@ pub struct AuditTargetConfig {}
#[async_trait::async_trait]
impl Operation for AuditTargetConfig {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;
@@ -345,8 +343,6 @@ pub struct ListAuditTargets {}
#[async_trait::async_trait]
impl Operation for ListAuditTargets {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_audit_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
let mut runtime_statuses = HashMap::new();
@@ -370,8 +366,6 @@ pub struct RemoveAuditTarget {}
#[async_trait::async_trait]
impl Operation for RemoveAuditTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;
@@ -838,6 +832,13 @@ mod tests {
extract_block_between_markers(src, "impl Operation for ListAuditTargets", "pub struct RemoveAuditTarget");
let delete_block = extract_block_between_markers(src, "impl Operation for RemoveAuditTarget", "#[cfg(test)]");
for block in [put_block, list_block, delete_block] {
assert!(
!block.contains(".enter()"),
"async audit handlers must rely on request-future instrumentation instead of holding span guards across awaits"
);
}
assert!(
put_block.contains("authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;"),
"audit target writes should require SetBucketTargetAction"
+8 -9
View File
@@ -43,7 +43,7 @@ use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::LazyLock;
use tracing::{Span, error, info, warn};
use tracing::{error, info, warn};
const LOG_COMPONENT_ADMIN_API: &str = "admin_api";
@@ -333,8 +333,6 @@ pub struct NotificationTarget {}
#[async_trait::async_trait]
impl Operation for NotificationTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
let context = app_context_from_req(&req);
@@ -401,8 +399,6 @@ pub struct ListNotificationTargets {}
#[async_trait::async_trait]
impl Operation for ListNotificationTargets {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_notification_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
refresh_persisted_module_switches_from_store().await.map_err(|err| {
warn!(
@@ -439,8 +435,6 @@ pub struct ListTargetsArns {}
#[async_trait::async_trait]
impl Operation for ListTargetsArns {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_notification_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
if let Some(reason) = notification_target_operation_block_reason(
"querying notification target ARNs for bucket associations from the console",
@@ -485,8 +479,6 @@ pub struct RemoveNotificationTarget {}
#[async_trait::async_trait]
impl Operation for RemoveNotificationTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
let context = app_context_from_req(&req);
@@ -1006,6 +998,13 @@ mod tests {
extract_block_between_markers(src, "impl Operation for ListTargetsArns", "pub struct RemoveNotificationTarget");
let delete_block = extract_block_between_markers(src, "impl Operation for RemoveNotificationTarget", "fn extract_param");
for block in [put_block, list_block, arns_block, delete_block] {
assert!(
!block.contains(".enter()"),
"async notification handlers must rely on request-future instrumentation instead of holding span guards across awaits"
);
}
assert!(
put_block.contains("authorize_notification_admin_request(&req, AdminAction::SetBucketTargetAction).await?;"),
"notification target writes should require SetBucketTargetAction"
+353 -19
View File
@@ -535,8 +535,13 @@ impl Operation for GetReplicationMetricsHandler {
let bucket_stats = cluster_replication_stats(bucket, app_context_from_req(&req)).await;
let data = serde_json::to_vec(&bucket_stats.replication_stats)
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "serialize failed"))?;
// Same minio-go `replication.Metrics` wire shape as
// `?replication-metrics` — the internal snake_case stats are the peer
// RPC wire format and must not leak here.
let data = serde_json::to_vec(&crate::admin::replication_metrics_wire::MetricsWire::from(
&bucket_stats.replication_stats,
))
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "serialize failed"))?;
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
Ok(S3Response::with_headers((StatusCode::OK, Body::from(data)), headers))
@@ -1153,7 +1158,7 @@ struct MrfResponse {
fn build_mrf_response(
bucket: String,
bucket_stats: &BucketStats,
durable: crate::admin::storage_api::replication::DurableMrfBacklog,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
) -> MrfResponse {
let observation_scope = if bucket_stats.replication_stats.cluster_complete {
"cluster_aggregated"
@@ -1223,7 +1228,10 @@ fn build_mrf_response(
total_failed_size,
queued_count: queued.count,
queued_size: queued.bytes,
per_object_entries_available: false,
// The default (non-aggregate) response mode streams the durable
// backlog per object, so the enumerable API exists whenever the
// backlog is readable.
per_object_entries_available: durable.available,
runtime_stats_available: bucket_stats.replication_stats.provider_available,
cluster_complete: bucket_stats.replication_stats.cluster_complete,
observed_node_count: bucket_stats.replication_stats.observed_node_count,
@@ -1235,23 +1243,165 @@ fn build_mrf_response(
}
}
/// One durable MRF backlog entry rendered for the default (madmin-compatible)
/// stream. Field names are the exact json tags of madmin-go `ReplicationMRF`
/// (replication-api.go), which `mc replicate backlog` decodes one JSON
/// document at a time. `Size` and `TargetARNs` are RustFS extension keys with
/// no madmin counterpart; Go decoders ignore unknown keys.
#[derive(Debug, Serialize)]
struct MrfEntryDocument {
/// The durable backlog is a cluster-shared ledger with no per-node
/// attribution, so the madmin `nodeName` tag is always empty.
#[serde(rename = "nodeName")]
node_name: String,
#[serde(rename = "bucket")]
bucket: String,
#[serde(rename = "object")]
object: String,
#[serde(rename = "versionId")]
version_id: String,
#[serde(rename = "retryCount")]
retry_count: i32,
#[serde(rename = "Size")]
size: i64,
#[serde(rename = "TargetARNs", skip_serializing_if = "Vec::is_empty")]
target_arns: Vec<String>,
}
/// Upper bound on the number of documents one stream response emits. The
/// durable ledger is not bounded by the in-memory pending cap (recovery can
/// persist far larger generations), and the body is buffered before send, so
/// an unbounded read could stage hundreds of MB per request. The handler
/// rejects a response beyond this bound instead of returning a partial 200.
const REPLICATION_MRF_MAX_STREAM_ENTRIES: usize = 10_000;
/// Project the durable backlog into madmin `ReplicationMRF` documents,
/// scoped to `bucket` when it is non-empty (madmin allows an empty bucket to
/// mean "across all buckets"), bounded by
/// [`REPLICATION_MRF_MAX_STREAM_ENTRIES`]. Returns the documents and whether
/// the backlog was truncated.
fn mrf_entry_documents(
bucket: &str,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
) -> (Vec<MrfEntryDocument>, bool) {
let mut documents = Vec::new();
let mut truncated = false;
for entry in durable
.entries
.iter()
.filter(|entry| bucket.is_empty() || entry.bucket == bucket)
{
if documents.len() >= REPLICATION_MRF_MAX_STREAM_ENTRIES {
truncated = true;
break;
}
documents.push(MrfEntryDocument {
node_name: String::new(),
bucket: entry.bucket.clone(),
object: entry.object.clone(),
// Delete-marker purge entries track the marker version separately;
// fall back to it so those rows still carry a version identity.
// The nil UUID is RustFS's in-memory null-version sentinel and
// must leave as the S3 wire token, not a zero UUID.
version_id: entry
.version_id
.or(entry.delete_marker_version_id)
.map(|v| {
if v.is_nil() {
rustfs_filemeta::NULL_VERSION_ID.to_string()
} else {
v.to_string()
}
})
.unwrap_or_default(),
retry_count: entry.retry_count,
size: entry.size,
target_arns: entry.target_arns.clone(),
});
}
(documents, truncated)
}
/// Render the MRF backlog as a response body.
///
/// Default (madmin-compatible) mode emits one `ReplicationMRF` JSON document
/// per line with no envelope — madmin's `BucketReplicationMRF` reads the body
/// with a `json.Decoder` loop, so an envelope object would decode as a single
/// entry whose `"Bucket"` key case-insensitively matches
/// `ReplicationMRF.Bucket` (a phantom row in `mc replicate backlog`), and an
/// empty backlog must render an empty body so the loop ends on io.EOF with
/// zero rows.
///
/// `aggregate=true` (RustFS extension) keeps the enveloped counter shape;
/// backlog-source health (`RuntimeStatsAvailable`/`DurableBacklogAvailable`)
/// is only representable there — an unreadable ledger fails the stream
/// request outright in the handler (madmin only decodes the body of a 200,
/// so an empty stream would read as a healthy zero-row backlog).
fn render_mrf_backlog(
response: &MrfResponse,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
aggregate: bool,
) -> Result<(Vec<u8>, bool), serde_json::Error> {
if aggregate {
return Ok((serde_json::to_vec(response)?, false));
}
let (documents, truncated) = mrf_entry_documents(&response.bucket, durable);
let mut data = Vec::new();
for entry in documents {
serde_json::to_writer(&mut data, &entry)?;
data.push(b'\n');
}
Ok((data, truncated))
}
fn ensure_complete_mrf_stream(truncated: bool) -> S3Result<()> {
if truncated {
return Err(S3Error::with_message(
S3ErrorCode::ServiceUnavailable,
"durable MRF backlog exceeds the stream limit; narrow the bucket scope or drain the backlog".to_string(),
));
}
Ok(())
}
/// `GET /v3/replication/mrf`
///
/// Reports the failed-replication backlog (MinIO's MRF concept) for a bucket.
///
/// Compatibility note: MinIO returns a stream of individual MRF entries. RustFS
/// deliberately returns aggregate runtime and durable counters instead.
/// `PerObjectEntriesAvailable` remains false until an enumerable API exists.
/// `PerTargetDurableEntriesAvailable` is false when the durable backlog includes
/// older entries that cannot be attributed to a target.
/// The default response is a madmin-compatible stream of `ReplicationMRF`
/// documents built from the durable backlog ledger (in-memory failures that
/// have not been flushed yet — the persister runs every few seconds — are not
/// visible). `?aggregate=true` (RustFS extension) returns the enveloped
/// runtime + durable counter shape instead; `PerTargetDurableEntriesAvailable`
/// is false there when the durable backlog includes older entries that cannot
/// be attributed to a target.
///
/// The madmin `node` parameter is accepted but has no filtering effect: the
/// durable ledger is cluster-shared with no per-node attribution, so every
/// node serves the same (complete) backlog.
///
/// Authorization: the stream requires `admin:ReplicationDiff` (it enumerates
/// object names and version ids, MinIO parity); `?aggregate=true` carries no
/// object identities and requires only `admin:GetReplicationMetrics`.
pub struct ReplicationMrfHandler {}
#[async_trait::async_trait]
impl Operation for ReplicationMrfHandler {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
validate_replication_admin_request(&req, AdminAction::GetReplicationMetricsAction).await?;
let queries = extract_query_params(&req.uri);
let aggregate = queries.get("aggregate").map(String::as_str) == Some("true");
// The default stream enumerates object names and version ids, which
// a metrics-only principal must not see; gate it on the same action
// MinIO uses for this endpoint. The aggregate counters carry no
// object identities and keep the metrics action.
let action = if aggregate {
AdminAction::GetReplicationMetricsAction
} else {
AdminAction::ReplicationDiff
};
validate_replication_admin_request(&req, action).await?;
let Some(bucket) = queries.get("bucket").filter(|b| !b.is_empty()).cloned() else {
return Err(s3_error!(InvalidRequest, "bucket is required"));
};
@@ -1275,14 +1425,48 @@ impl Operation for ReplicationMrfHandler {
return Err(ApiError::from(err).into());
}
if let Some(node) = queries.get("node").filter(|node| !node.is_empty() && node.as_str() != "all") {
// The durable backlog ledger is cluster-shared with no per-node
// attribution, so a node-scoped request still sees the complete
// (superset) backlog.
debug!(node = %node, "replication mrf node filter has no effect on the cluster-shared backlog");
}
let durable = crate::admin::storage_api::replication::read_durable_mrf_backlog(store).await;
let bucket_stats = cluster_replication_stats(&bucket, app_context_from_req(&req)).await;
let response = build_mrf_response(bucket, &bucket_stats, durable);
let response = build_mrf_response(bucket, &bucket_stats, &durable);
let data = serde_json::to_vec(&response)
if !durable.available && !aggregate {
// The madmin stream has no envelope to carry source health, and
// madmin only decodes the body of a 200 — an empty stream would
// read as a clean, healthy zero-row backlog. Fail loudly instead;
// aggregate mode still reports the availability fields.
tracing::warn!(
bucket = %response.bucket,
"durable MRF backlog is unreadable; failing the stream request — use aggregate=true to see source health"
);
return Err(S3Error::with_message(
S3ErrorCode::ServiceUnavailable,
"durable MRF backlog is unreadable; retry, or use aggregate=true for source health".to_string(),
));
}
let (data, truncated) = render_mrf_backlog(&response, &durable, aggregate)
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize failed: {e}")))?;
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
if truncated {
tracing::warn!(
event = "replication_mrf_stream_rejected",
component = "admin",
subsystem = "replication",
result = "rejected",
bucket = %response.bucket,
max_entries = REPLICATION_MRF_MAX_STREAM_ENTRIES,
"replication mrf stream exceeds the response limit"
);
}
ensure_complete_mrf_stream(truncated)?;
Ok(S3Response::with_headers((StatusCode::OK, Body::from(data)), headers))
}
}
@@ -1292,7 +1476,8 @@ mod tests {
use super::{
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, RemoteTargetCredentialsRequest, RemoteTargetRequest,
ReplicationDiffEntry, SUPPORTED_REMOTE_TARGET_API, TargetUpdateOp, build_mrf_response, extract_query_params,
parse_remote_target_update_ops, render_replication_diff, unique_replication_peers, validate_remote_target_tls_settings,
parse_remote_target_update_ops, render_mrf_backlog, render_replication_diff, unique_replication_peers,
validate_remote_target_tls_settings,
};
use crate::admin::storage_api::bucket::target::{BucketTarget, LatencyStat};
use crate::admin::storage_api::replication::{BucketStats, DurableMrfBacklog, MrfOpKind, MrfReplicateEntry};
@@ -1510,7 +1695,7 @@ mod tests {
],
};
let response = build_mrf_response("bucket-a".to_string(), &stats, durable);
let response = build_mrf_response("bucket-a".to_string(), &stats, &durable);
let json = serde_json::to_value(response).expect("MRF response should serialize");
assert_eq!(json["TotalFailedCount"], 3);
@@ -1522,7 +1707,9 @@ mod tests {
assert_eq!(json["RuntimeStatsAvailable"], true);
assert_eq!(json["ClusterComplete"], false);
assert_eq!(json["Targets"][0]["ObservationScope"], "partial_cluster");
assert_eq!(json["PerObjectEntriesAvailable"], false);
// The bare stream enumerates the durable backlog per object, so a
// readable backlog advertises the enumerable API.
assert_eq!(json["PerObjectEntriesAvailable"], true);
assert_eq!(json["PerTargetDurableEntriesAvailable"], true);
let targets = json["Targets"].as_array().expect("targets should serialize as an array");
@@ -1569,7 +1756,7 @@ mod tests {
}],
};
let response = build_mrf_response("bucket-a".to_string(), &stats, durable);
let response = build_mrf_response("bucket-a".to_string(), &stats, &durable);
let json = serde_json::to_value(response).expect("MRF response should serialize");
assert_eq!(json["DurableBacklogAvailable"], true);
@@ -1587,7 +1774,7 @@ mod tests {
#[test]
fn mrf_response_distinguishes_unavailable_sources_from_valid_zero() {
let unavailable = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), DurableMrfBacklog::default());
let unavailable = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &DurableMrfBacklog::default());
let unavailable_json = serde_json::to_value(unavailable).expect("unavailable response should serialize");
assert_eq!(unavailable_json["RuntimeStatsAvailable"], false);
assert_eq!(unavailable_json["DurableBacklogAvailable"], false);
@@ -1600,7 +1787,7 @@ mod tests {
let valid_empty = build_mrf_response(
"bucket-a".to_string(),
&valid_empty_stats,
DurableMrfBacklog {
&DurableMrfBacklog {
available: true,
entries: Vec::new(),
},
@@ -1613,6 +1800,153 @@ mod tests {
assert_eq!(valid_empty_json["PerTargetDurableEntriesAvailable"], true);
}
fn sample_durable_backlog() -> DurableMrfBacklog {
DurableMrfBacklog {
available: true,
entries: vec![
MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: "object-a".to_string(),
version_id: Some(uuid::Uuid::from_u128(7)),
retry_count: 2,
size: 250,
op: MrfOpKind::Object,
target_arns: vec!["arn-a".to_string()],
..Default::default()
},
MrfReplicateEntry {
bucket: "other-bucket".to_string(),
object: "object-b".to_string(),
version_id: None,
retry_count: 0,
size: 999,
op: MrfOpKind::Object,
target_arns: Vec::new(),
..Default::default()
},
],
}
}
/// madmin's `BucketReplicationMRF` decodes the body one `ReplicationMRF`
/// JSON document at a time; the default response must therefore be a bare
/// document stream with madmin's exact json tags, not an envelope.
#[test]
fn mrf_stream_renders_bare_madmin_documents() {
let durable = sample_durable_backlog();
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
let text = String::from_utf8(body).expect("body should be utf-8");
let lines: Vec<&str> = text.lines().filter(|line| !line.trim().is_empty()).collect();
// Only the entry matching the requested bucket is streamed.
assert_eq!(lines.len(), 1, "expected one MRF document, got: {text}");
let doc: serde_json::Value = serde_json::from_str(lines[0]).expect("each line should be a JSON document");
assert_eq!(doc["bucket"], "bucket-a");
assert_eq!(doc["object"], "object-a");
assert_eq!(doc["versionId"], uuid::Uuid::from_u128(7).to_string());
assert_eq!(doc["retryCount"], 2);
// madmin `ReplicationMRF` has a `nodeName` tag; the durable backlog is
// cluster-shared, so RustFS reports an empty node name.
assert_eq!(doc["nodeName"], "");
// The envelope keys must not leak into the stream: a `"Bucket"` key
// would case-insensitively populate `ReplicationMRF.Bucket` and render
// a phantom row in `mc replicate backlog`.
assert!(doc.get("Bucket").is_none());
assert!(doc.get("Targets").is_none());
}
/// An empty backlog must produce an empty body: madmin's decoder loop then
/// terminates on io.EOF with zero rows instead of one phantom row.
#[test]
fn mrf_stream_renders_empty_body_for_no_entries() {
let durable = DurableMrfBacklog {
available: true,
entries: Vec::new(),
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(
body.is_empty(),
"empty backlog must serialize to an empty body, got: {}",
String::from_utf8_lossy(&body)
);
}
/// `?aggregate=true` (RustFS extension) keeps the enveloped counter shape.
#[test]
fn mrf_aggregate_envelope_retains_counters() {
let durable = sample_durable_backlog();
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, true).expect("aggregate body should serialize");
let json: serde_json::Value = serde_json::from_slice(&body).expect("aggregate body should be one JSON object");
assert_eq!(json["Bucket"], "bucket-a");
assert_eq!(json["DurableCount"], 1);
assert_eq!(json["DurableBacklogAvailable"], true);
// The bare stream is an enumerable per-object API, so the aggregate
// shell now truthfully advertises it whenever the backlog is readable.
assert_eq!(json["PerObjectEntriesAvailable"], true);
}
/// The nil UUID is RustFS's in-memory null-version sentinel; the wire
/// token is `null`, never the zero UUID (second review round).
#[test]
fn mrf_stream_maps_nil_version_to_null_token() {
let durable = DurableMrfBacklog {
available: true,
entries: vec![MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: "null-version-object".to_string(),
version_id: Some(uuid::Uuid::nil()),
retry_count: 1,
size: 10,
op: MrfOpKind::Object,
..Default::default()
}],
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, truncated) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(!truncated);
let doc: serde_json::Value =
serde_json::from_str(String::from_utf8(body).expect("utf-8").lines().next().expect("one line"))
.expect("line should be a JSON document");
assert_eq!(doc["versionId"], "null");
}
/// The durable ledger is not bounded by the in-memory pending cap; the
/// stream must stop at the documented bound and signal truncation
/// (second review round).
#[test]
fn mrf_stream_truncates_at_the_documented_bound() {
let entries = (0..super::REPLICATION_MRF_MAX_STREAM_ENTRIES + 1)
.map(|index| MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: format!("object-{index}"),
retry_count: 1,
op: MrfOpKind::Object,
..Default::default()
})
.collect();
let durable = DurableMrfBacklog {
available: true,
entries,
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, truncated) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(truncated, "one entry past the bound must signal truncation");
assert_eq!(
String::from_utf8(body).expect("utf-8").lines().count(),
super::REPLICATION_MRF_MAX_STREAM_ENTRIES
);
let error = super::ensure_complete_mrf_stream(truncated).expect_err("partial streams must not return 200");
assert_eq!(error.code(), &s3s::S3ErrorCode::ServiceUnavailable);
}
#[test]
fn test_extract_query_params_decodes_percent_encoded_values() {
let uri: Uri = "/rustfs/admin/v3/list-remote-targets?bucket=foo%2Fbar&flag=a+b"
File diff suppressed because it is too large Load Diff
@@ -29,6 +29,6 @@ impl Operation for RestLoadCredentialsHandler {
let issuer = IamTableCredentialIssuer::from_request(&req)?;
let response =
load_credentials_response(&store, &warehouse, &namespace, &table, &issuer, Some(&principal.credentials)).await?;
build_json_response(StatusCode::OK, &response)
build_sensitive_json_response(StatusCode::OK, &response)
}
}
File diff suppressed because it is too large Load Diff
@@ -125,7 +125,9 @@ impl Operation for RestLoadTableHandler {
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let response = load_table_response(&store, &metadata_backend, &warehouse, &namespace, &table).await?;
let snapshot_selection = rest_table_snapshot_selection_from_query(&req.uri)?;
let mut response = load_table_response(&store, &metadata_backend, &warehouse, &namespace, &table).await?;
apply_rest_table_snapshot_selection(&mut response.metadata, snapshot_selection);
build_json_response(StatusCode::OK, &response)
}
}
@@ -158,7 +160,7 @@ impl Operation for RestCommitTableHandler {
let principal = authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
install_table_catalog_s3_request_info(&mut req, &principal)?;
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let request = read_json_body::<RestCommitTableRequest>(std::mem::take(&mut req.input)).await?;
let request = read_rest_commit_table_request(std::mem::take(&mut req.input)).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let commit_backend = TableCommitObjectBackend::for_request(metadata_backend, req);
@@ -178,6 +180,14 @@ impl Operation for RestDropTableHandler {
let table = table_name_from_params(&params)?;
let resource = TableCatalogResource::table(&warehouse, &namespace, &table);
authorize_table_catalog_resource_request(&req, &resource, AdminAction::DeleteTableAction).await?;
let purge_requested = rest_purge_requested_from_query(&req.uri)?;
if purge_requested {
return Err(iceberg_rest_error(
ICEBERG_ERROR_UNSUPPORTED_OPERATION,
StatusCode::NOT_ACCEPTABLE,
"purgeRequested=true is not supported",
));
}
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let store = table_catalog_store_from_extensions(&req.extensions)?;
drop_table_in_store(&store, &warehouse, &namespace, &table).await?;
File diff suppressed because it is too large Load Diff
@@ -43,8 +43,9 @@ impl Operation for RestCreateViewHandler {
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let table_bucket_enabled = table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let publication_backend = TableCommitObjectBackend::preauthorized(metadata_backend);
let response =
create_view_response(&store, &metadata_backend, &warehouse, &namespace, request, table_bucket_enabled).await?;
create_view_response(&store, &publication_backend, &warehouse, &namespace, request, table_bucket_enabled).await?;
build_json_response(StatusCode::OK, &response)
}
}
@@ -87,17 +88,20 @@ pub struct RestReplaceViewHandler {}
#[async_trait::async_trait]
impl Operation for RestReplaceViewHandler {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
async fn call(&self, mut req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let warehouse = warehouse_from_params(&params)?;
let namespace = namespace_from_params(&params)?;
let view = view_name_from_params(&params)?;
let resource = TableCatalogResource::view(&warehouse, &namespace, &view);
authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
let principal = authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
install_table_catalog_s3_request_info(&mut req, &principal)?;
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let request = read_json_body::<RestCommitViewRequest>(req.input).await?;
let request = read_rest_commit_view_request(std::mem::take(&mut req.input)).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let response = replace_view_response(&store, &metadata_backend, &warehouse, &namespace, &view, request).await?;
let commit_backend = TableCommitObjectBackend::for_request(metadata_backend, req);
let result = replace_view_response(&store, &commit_backend, &warehouse, &namespace, &view, request).await;
let response = commit_backend.finish(result).await?;
build_json_response(StatusCode::OK, &response)
}
}
+1
View File
@@ -17,6 +17,7 @@ mod auth;
pub mod console;
pub mod handlers;
mod plugin_contract;
pub(crate) mod replication_metrics_wire;
// Contract inventory is validated by tests before later runtime integration.
#[allow(dead_code)]
pub(crate) mod route_policy;
@@ -0,0 +1,673 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Serialize-only wire projections of the internal replication statistics
//! onto the minio-go `replication.Metrics` / `replication.MetricsV2` json
//! shapes consumed by `mc replicate status` (`?replication-metrics[=2]` and
//! the admin `replicationmetrics` endpoint).
//!
//! Red line: the internal `BucketStats` family in
//! `crates/replication/src/stats.rs` is ALSO the intra-cluster peer-RPC wire
//! format — `node_service.rs` encodes it with `rmp_serde::to_vec_named`, so
//! its Rust field names travel between nodes as msgpack map keys. Renaming
//! those serde names would break mixed-version clusters mid rolling upgrade.
//! All madmin/minio-go interop therefore happens in these DTOs; never add
//! `#[serde(rename)]` to the internal structs instead.
//!
//! Field names below are the exact json tags of minio-go
//! `pkg/replication/replication.go` (v7.0.91). Keys minio-go does not know
//! are RustFS extensions; Go decoders ignore unknown keys. `max`/`peak` are
//! both emitted for the queue peak because the MinIO server writes `max`
//! while minio-go reads `peak` (an upstream drift); emitting both keeps every
//! decoder working.
use serde::Serialize;
use std::collections::HashMap;
use std::time::Duration;
use crate::admin::storage_api::replication::{
BucketReplicationStat as InternalReplicationStat, BucketReplicationStats as InternalReplicationStats, BucketStats,
InQueueMetric as InternalInQueueMetric, XferStats as InternalXferStats,
};
/// minio-go `replication.RStat`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct RStatWire {
#[serde(rename = "count")]
pub count: f64,
#[serde(rename = "bytes")]
pub bytes: i64,
}
/// minio-go `replication.TimedErrStats`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct TimedErrStatsWire {
#[serde(rename = "lastMinute")]
pub last_minute: RStatWire,
#[serde(rename = "lastHour")]
pub last_hour: RStatWire,
#[serde(rename = "totals")]
pub totals: RStatWire,
}
impl TimedErrStatsWire {
fn add(self, other: TimedErrStatsWire) -> TimedErrStatsWire {
fn add(a: RStatWire, b: RStatWire) -> RStatWire {
RStatWire {
count: a.count + b.count,
bytes: a.bytes.saturating_add(b.bytes),
}
}
TimedErrStatsWire {
last_minute: add(self.last_minute, other.last_minute),
last_hour: add(self.last_hour, other.last_hour),
totals: add(self.totals, other.totals),
}
}
}
/// minio-go `replication.QStat`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct QStatWire {
#[serde(rename = "count")]
pub count: f64,
#[serde(rename = "bytes")]
pub bytes: f64,
}
/// minio-go `replication.InQueueMetric`, with the queue peak emitted under
/// both `peak` (minio-go tag) and `max` (MinIO server tag).
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct InQueueMetricWire {
#[serde(rename = "curr")]
pub curr: QStatWire,
#[serde(rename = "avg")]
pub avg: QStatWire,
#[serde(rename = "max")]
pub max: QStatWire,
#[serde(rename = "peak")]
pub peak: QStatWire,
}
impl From<&InternalInQueueMetric> for InQueueMetricWire {
fn from(metric: &InternalInQueueMetric) -> Self {
fn qstat(bytes: i64, count: i64) -> QStatWire {
QStatWire {
count: count as f64,
bytes: bytes as f64,
}
}
let peak = qstat(metric.max.bytes, metric.max.count);
InQueueMetricWire {
curr: qstat(metric.curr.bytes, metric.curr.count),
avg: qstat(metric.avg.bytes, metric.avg.count),
max: peak,
peak,
}
}
}
/// minio-go `replication.XferStats`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct XferStatsWire {
#[serde(rename = "avgRate")]
pub avg_rate: f64,
#[serde(rename = "peakRate")]
pub peak_rate: f64,
#[serde(rename = "currRate")]
pub curr_rate: f64,
}
#[derive(Default)]
struct XferStatsAverage {
sum: XferStatsWire,
active: u32,
}
impl XferStatsAverage {
fn add_active(&mut self, stats: XferStatsWire) {
if stats.peak_rate <= 0.0 {
return;
}
self.add_raw(stats);
self.active += 1;
}
fn add_raw(&mut self, stats: XferStatsWire) {
self.sum.avg_rate += stats.avg_rate;
self.sum.curr_rate += stats.curr_rate;
self.sum.peak_rate = self.sum.peak_rate.max(stats.peak_rate);
}
fn finish(self) -> XferStatsWire {
let active = self.active;
self.finish_with_divisor(active)
}
fn finish_with_divisor(self, divisor: u32) -> XferStatsWire {
if divisor == 0 {
return self.sum;
}
XferStatsWire {
avg_rate: self.sum.avg_rate / f64::from(divisor),
peak_rate: self.sum.peak_rate,
curr_rate: self.sum.curr_rate / f64::from(divisor),
}
}
}
impl From<&InternalXferStats> for XferStatsWire {
fn from(stats: &InternalXferStats) -> Self {
XferStatsWire {
avg_rate: stats.avg,
peak_rate: stats.peak,
curr_rate: stats.curr,
}
}
}
/// minio-go `replication.WorkerStat`. RustFS does not track per-bucket worker
/// occupancy yet, so this always reports zeros.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct WorkerStatWire {
#[serde(rename = "curr")]
pub curr: i32,
#[serde(rename = "avg")]
pub avg: f32,
#[serde(rename = "max")]
pub max: i32,
}
/// minio-go `replication.ReplMRFStats`. RustFS does not track the 5-minute /
/// dropped MRF windows, so this always reports zeros; the durable backlog is
/// enumerable via `/v3/replication/mrf` instead.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct ReplMrfStatsWire {
#[serde(rename = "failedCount_last5min")]
pub last_failed_count: u64,
#[serde(rename = "droppedCount_since_uptime")]
pub total_dropped_count: u64,
#[serde(rename = "droppedBytes_since_uptime")]
pub total_dropped_bytes: u64,
}
/// minio-go `replication.CounterSummary`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct CounterSummaryWire {
#[serde(rename = "last1hr")]
pub last1hr: u64,
#[serde(rename = "last1m")]
pub last1m: u64,
#[serde(rename = "total")]
pub total: u64,
}
/// minio-go `replication.TargetMetrics` (one remote target / ARN).
#[derive(Debug, Default, Serialize)]
pub(crate) struct TargetMetricsWire {
#[serde(rename = "replicationCount")]
pub replicated_count: i64,
#[serde(rename = "completedReplicationSize")]
pub replicated_size: i64,
/// Bandwidth limit for this target. The tag says "bits" but both MinIO
/// and minio-go treat the value as bytes/sec; keep bytes/sec.
#[serde(rename = "limitInBits")]
pub bandwidth_limit_bytes_per_sec: i64,
#[serde(rename = "currentBandwidth")]
pub current_bandwidth_bytes_per_sec: f64,
#[serde(rename = "failed")]
pub failed: TimedErrStatsWire,
#[serde(rename = "failedReplicationSize")]
pub failed_size: i64,
#[serde(rename = "failedReplicationCount")]
pub failed_count: i64,
}
fn target_timed_err_stats(stat: &InternalReplicationStat) -> TimedErrStatsWire {
// Cluster aggregation merges FailStats without the process-local samples,
// so the serializable window snapshots (refreshed at each node's
// collection point, summed by merge) are authoritative here; the live
// samples only ever agree with or lag them, so take the larger.
let sampled_minute = stat.fail_stats.recent_since(Duration::from_secs(60));
let sampled_hour = stat.fail_stats.recent_since(Duration::from_secs(3600));
let window = |sampled_count: i64, sampled_size: i64, snapshot_count: i64, snapshot_size: i64| RStatWire {
count: sampled_count.max(snapshot_count) as f64,
bytes: sampled_size.max(snapshot_size),
};
TimedErrStatsWire {
last_minute: window(
sampled_minute.count,
sampled_minute.size,
stat.fail_stats.last_minute.count,
stat.fail_stats.last_minute.size,
),
last_hour: window(
sampled_hour.count,
sampled_hour.size,
stat.fail_stats.last_hour.count,
stat.fail_stats.last_hour.size,
),
totals: RStatWire {
count: stat.failed.count as f64,
bytes: stat.failed.size,
},
}
}
impl From<&InternalReplicationStat> for TargetMetricsWire {
fn from(stat: &InternalReplicationStat) -> Self {
TargetMetricsWire {
replicated_count: stat.replicated_count,
replicated_size: stat.replicated_size,
bandwidth_limit_bytes_per_sec: stat.bandwidth_limit_bytes_per_sec,
current_bandwidth_bytes_per_sec: stat.current_bandwidth_bytes_per_sec,
failed: target_timed_err_stats(stat),
failed_size: stat.failed.size,
failed_count: stat.failed.count,
}
}
}
/// minio-go `replication.Metrics` — the `currStats` member of `MetricsV2` and
/// the whole v1 response body. The trailing snake_case fields are RustFS
/// source-health extension keys (ignored by Go decoders) carried over from
/// the previous response shape.
#[derive(Debug, Default, Serialize)]
pub(crate) struct MetricsWire {
#[serde(rename = "Stats")]
pub stats: HashMap<String, TargetMetricsWire>,
#[serde(rename = "completedReplicationSize")]
pub replicated_size: i64,
#[serde(rename = "replicaSize")]
pub replica_size: i64,
#[serde(rename = "replicaCount")]
pub replica_count: i64,
#[serde(rename = "replicationCount")]
pub replicated_count: i64,
#[serde(rename = "failed")]
pub failed: TimedErrStatsWire,
#[serde(rename = "queued")]
pub queued: InQueueMetricWire,
// RustFS extension keys (source health of the aggregation).
pub provider_available: bool,
pub cluster_complete: bool,
pub observed_node_count: u32,
pub expected_node_count: u32,
}
impl From<&InternalReplicationStats> for MetricsWire {
fn from(stats: &InternalReplicationStats) -> Self {
let mut failed = TimedErrStatsWire::default();
let mut targets = HashMap::with_capacity(stats.stats.len());
for (arn, stat) in &stats.stats {
let target = TargetMetricsWire::from(stat);
failed = failed.add(target.failed);
targets.insert(arn.clone(), target);
}
MetricsWire {
stats: targets,
replicated_size: stats.replicated_size,
replica_size: stats.replica_size,
replica_count: stats.replica_count,
replicated_count: stats.replicated_count,
failed,
queued: InQueueMetricWire::from(&stats.q_stat),
provider_available: stats.provider_available,
cluster_complete: stats.cluster_complete,
observed_node_count: stats.observed_node_count,
expected_node_count: stats.expected_node_count,
}
}
}
/// minio-go `replication.ReplQNodeStats`.
#[derive(Debug, Default, Serialize)]
pub(crate) struct ReplQNodeStatsWire {
#[serde(rename = "nodeName")]
pub node_name: String,
#[serde(rename = "uptime")]
pub uptime: i64,
#[serde(rename = "activeWorkers")]
pub workers: WorkerStatWire,
#[serde(rename = "transferSummary")]
pub xfer_stats: XferSummaryWire,
#[serde(rename = "tgtTransferStats")]
pub tgt_xfer_stats: TargetXferSummaryWire,
#[serde(rename = "queueStats")]
pub q_stats: InQueueMetricWire,
#[serde(rename = "mrfStats")]
pub mrf_stats: ReplMrfStatsWire,
#[serde(rename = "retries")]
pub retries: CounterSummaryWire,
#[serde(rename = "errors")]
pub errors: CounterSummaryWire,
}
/// minio-go `replication.ReplQueueStats`.
#[derive(Debug, Default, Serialize)]
pub(crate) struct ReplQueueStatsWire {
#[serde(rename = "nodes")]
pub nodes: Vec<ReplQNodeStatsWire>,
}
/// minio-go `replication.MetricsV2` — the `?replication-metrics=2` body.
#[derive(Debug, Default, Serialize)]
pub(crate) struct MetricsV2Wire {
#[serde(rename = "uptime")]
pub uptime: i64,
#[serde(rename = "currStats")]
pub current_stats: MetricsWire,
#[serde(rename = "queueStats")]
pub queue_stats: ReplQueueStatsWire,
#[serde(rename = "downtimeInfo")]
pub downtime_info: HashMap<String, serde_json::Value>,
}
/// `transferSummary` map keyed by minio-go `MetricName` (Large/Small/Total).
type XferSummaryWire = HashMap<&'static str, XferStatsWire>;
/// `tgtTransferStats` map keyed by target ARN.
type TargetXferSummaryWire = HashMap<String, XferSummaryWire>;
fn transfer_summaries(stats: &InternalReplicationStats) -> (XferSummaryWire, TargetXferSummaryWire) {
let mut per_target: TargetXferSummaryWire = HashMap::new();
let mut large_summary = XferStatsAverage::default();
let mut small_summary = XferStatsAverage::default();
let mut total_summary = XferStatsAverage::default();
let mut active_targets = 0;
for (arn, stat) in &stats.stats {
let large = XferStatsWire::from(&stat.xfer_rate_lrg);
let small = XferStatsWire::from(&stat.xfer_rate_sml);
let mut target_total = XferStatsAverage::default();
target_total.add_active(large);
target_total.add_active(small);
let total = target_total.finish();
per_target.insert(arn.clone(), HashMap::from([("Large", large), ("Small", small), ("Total", total)]));
if large.peak_rate > 0.0 || small.peak_rate > 0.0 {
active_targets += 1;
large_summary.add_raw(large);
small_summary.add_raw(small);
total_summary.add_raw(large);
total_summary.add_raw(small);
}
}
let summary = HashMap::from([
("Large", large_summary.finish_with_divisor(active_targets)),
("Small", small_summary.finish_with_divisor(active_targets)),
("Total", total_summary.finish_with_divisor(active_targets)),
]);
(summary, per_target)
}
impl MetricsV2Wire {
/// Project the aggregated internal stats onto the `MetricsV2` shape.
///
/// The aggregation path leaves `queue_stats.nodes` empty today, so a
/// single node entry is synthesized from the bucket queue snapshot —
/// `mc replicate status` derives its queue/worker panels from
/// `queueStats.nodes` and treats an empty list as "no data".
pub(crate) fn from_stats(bucket_stats: &BucketStats, node_name: &str) -> Self {
let (xfer_stats, tgt_xfer_stats) = transfer_summaries(&bucket_stats.replication_stats);
let mut nodes: Vec<ReplQNodeStatsWire> = bucket_stats
.queue_stats
.nodes
.iter()
.map(|node| ReplQNodeStatsWire {
node_name: node_name.to_string(),
uptime: bucket_stats.uptime,
q_stats: InQueueMetricWire::from(&node.q_stats),
..Default::default()
})
.collect();
if nodes.is_empty() {
nodes.push(ReplQNodeStatsWire {
node_name: node_name.to_string(),
uptime: bucket_stats.uptime,
q_stats: InQueueMetricWire::from(&bucket_stats.replication_stats.q_stat),
xfer_stats: xfer_stats.clone(),
tgt_xfer_stats: tgt_xfer_stats.clone(),
..Default::default()
});
} else {
// Attach the transfer summaries to the first node; the internal
// snapshot does not attribute transfer rates per node.
if let Some(first) = nodes.first_mut() {
first.xfer_stats = xfer_stats.clone();
first.tgt_xfer_stats = tgt_xfer_stats.clone();
}
}
MetricsV2Wire {
uptime: bucket_stats.uptime,
current_stats: MetricsWire::from(&bucket_stats.replication_stats),
queue_stats: ReplQueueStatsWire { nodes },
downtime_info: HashMap::new(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn sample_bucket_stats() -> BucketStats {
let mut stats = BucketStats {
uptime: 42,
..Default::default()
};
stats.replication_stats.replica_count = 2;
stats.replication_stats.replica_size = 128;
stats.replication_stats.replicated_count = 9;
stats.replication_stats.replicated_size = 4096;
let target = stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default();
target.replicated_count = 9;
target.replicated_size = 4096;
target.failed.count = 3;
target.failed.size = 900;
target.bandwidth_limit_bytes_per_sec = 1024;
target.current_bandwidth_bytes_per_sec = 512.5;
stats
.replication_stats
.q_stat
.curr
.now_count
.store(4, std::sync::atomic::Ordering::Relaxed);
stats
.replication_stats
.q_stat
.curr
.now_bytes
.store(1200, std::sync::atomic::Ordering::Relaxed);
stats.replication_stats.q_stat = stats.replication_stats.q_stat.snapshot();
stats
}
#[test]
fn metrics_wire_matches_minio_go_tags() {
let stats = sample_bucket_stats();
let json = serde_json::to_value(MetricsWire::from(&stats.replication_stats)).expect("v1 wire should serialize");
assert_eq!(json["replicaCount"], 2);
assert_eq!(json["replicaSize"], 128);
assert_eq!(json["replicationCount"], 9);
assert_eq!(json["completedReplicationSize"], 4096);
assert_eq!(json["queued"]["curr"]["count"], 4.0);
assert_eq!(json["queued"]["curr"]["bytes"], 1200.0);
let target = &json["Stats"]["arn:minio:replication::t:b"];
assert_eq!(target["replicationCount"], 9);
assert_eq!(target["completedReplicationSize"], 4096);
assert_eq!(target["limitInBits"], 1024);
assert_eq!(target["currentBandwidth"], 512.5);
// failed is the madmin TimedErrStats envelope, not the internal
// {count,size} pair.
assert_eq!(target["failed"]["totals"]["count"], 3.0);
assert_eq!(target["failed"]["totals"]["bytes"], 900);
assert!(target["failed"].get("count").is_none());
// Aggregate failed mirrors the per-target totals.
assert_eq!(json["failed"]["totals"]["count"], 3.0);
}
#[test]
fn metrics_v2_wire_synthesizes_queue_node() {
let stats = sample_bucket_stats();
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1:9000")).expect("v2 wire should serialize");
assert_eq!(json["uptime"], 42);
assert_eq!(json["currStats"]["replicaCount"], 2);
let node = &json["queueStats"]["nodes"][0];
assert_eq!(node["nodeName"], "node-1:9000");
assert_eq!(node["uptime"], 42);
assert_eq!(node["queueStats"]["curr"]["count"], 4.0);
// The queue peak is emitted under both the minio-go tag (`peak`) and
// the MinIO server tag (`max`).
assert_eq!(node["queueStats"]["peak"], node["queueStats"]["max"]);
assert!(node["activeWorkers"].get("curr").is_some());
assert!(node["transferSummary"].get("Total").is_some());
assert_eq!(json["downtimeInfo"], serde_json::json!({}));
}
/// minio-go's transferSummary labels mean >= 128 MiB for Large; the
/// producer must bin on the same boundary (MIN_LARGE_OBJ_SIZE, shared
/// with the worker-pool split), or a 2 MiB replication shows under Large
/// while Small stays zero.
#[test]
fn transfer_summary_bins_on_the_128_mib_boundary() {
const MIB: i64 = 1024 * 1024;
let mut stats = BucketStats::default();
let stat = stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default();
stat.update_xfer_rate(2 * MIB, std::time::Duration::from_secs(1));
stat.update_xfer_rate(127 * MIB, std::time::Duration::from_secs(1));
stat.update_xfer_rate(128 * MIB, std::time::Duration::from_secs(1));
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1")).expect("v2 wire should serialize");
let summary = &json["queueStats"]["nodes"][0]["tgtTransferStats"]["arn:minio:replication::t:b"];
let small_peak = summary["Small"]["peakRate"].as_f64().expect("Small peakRate");
let large_peak = summary["Large"]["peakRate"].as_f64().expect("Large peakRate");
assert!(
(small_peak - (127 * MIB) as f64).abs() < 1.0,
"2 MiB and 127 MiB transfers must bin as Small (peak {small_peak})"
);
assert!(
(large_peak - (128 * MIB) as f64).abs() < 1.0,
"exactly 128 MiB must bin as Large (peak {large_peak})"
);
}
#[test]
fn transfer_summaries_average_active_bins_and_targets() {
let mut stats = BucketStats::default();
let first = stats.replication_stats.stats.entry("target-a".to_string()).or_default();
first.xfer_rate_sml.avg = 50.0;
first.xfer_rate_sml.curr = 40.0;
first.xfer_rate_sml.peak = 60.0;
first.xfer_rate_lrg.avg = 100.0;
first.xfer_rate_lrg.curr = 80.0;
first.xfer_rate_lrg.peak = 120.0;
let second = stats.replication_stats.stats.entry("target-b".to_string()).or_default();
second.xfer_rate_sml.avg = 30.0;
second.xfer_rate_sml.curr = 20.0;
second.xfer_rate_sml.peak = 40.0;
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1")).expect("v2 wire should serialize");
let node = &json["queueStats"]["nodes"][0];
let target_a = &node["tgtTransferStats"]["target-a"]["Total"];
assert_eq!(target_a["avgRate"], 75.0);
assert_eq!(target_a["currRate"], 60.0);
assert_eq!(target_a["peakRate"], 120.0);
let summary = &node["transferSummary"];
assert_eq!(summary["Small"]["avgRate"], 40.0);
assert_eq!(summary["Small"]["currRate"], 30.0);
assert_eq!(summary["Large"]["avgRate"], 50.0);
assert_eq!(summary["Total"]["avgRate"], 90.0);
assert_eq!(summary["Total"]["currRate"], 70.0);
assert_eq!(summary["Total"]["peakRate"], 120.0);
}
/// Review regression: both metrics endpoints aggregate first, and the
/// FailStats merge drops the process-local samples — the rolling windows
/// must survive a peer-RPC round trip plus aggregation and still reach
/// the wire body.
#[test]
fn failure_windows_survive_aggregation_before_serialization() {
// Node A: live failure; the windows are stamped at the collection
// point (get_latest_replication_stats calls refresh_windows before
// the stats cross the wire), never on the failure hot path.
let mut node_a = crate::admin::storage_api::replication::BucketReplicationStat::default();
node_a.fail_stats.add_size(512, None::<&std::io::Error>);
node_a.fail_stats.refresh_windows();
node_a.failed = node_a.fail_stats.to_metric();
// Node A's stats cross the peer RPC wire: the samples are dropped,
// the window snapshots travel.
let encoded = rmp_serde::to_vec_named(&node_a).expect("stat should encode");
let remote: crate::admin::storage_api::replication::BucketReplicationStat =
rmp_serde::from_slice(&encoded).expect("stat should decode");
// Aggregation merges the remote stat with an empty local one.
let merged_fail = remote.fail_stats.merge(&Default::default());
let aggregated = crate::admin::storage_api::replication::BucketReplicationStat {
failed: merged_fail.to_metric(),
fail_stats: merged_fail,
..Default::default()
};
let mut stats = BucketStats::default();
stats
.replication_stats
.stats
.insert("arn:minio:replication::t:b".to_string(), aggregated);
let json = serde_json::to_value(MetricsWire::from(&stats.replication_stats)).expect("wire should serialize");
let failed = &json["Stats"]["arn:minio:replication::t:b"]["failed"];
assert_eq!(failed["totals"]["count"], 1.0);
assert_eq!(
failed["lastMinute"]["count"], 1.0,
"the rolling minute window must survive RPC + aggregation"
);
assert_eq!(failed["lastMinute"]["bytes"], 512);
assert_eq!(failed["lastHour"]["count"], 1.0);
}
/// Pin the intra-cluster peer-RPC wire format of the internal stats: it
/// is msgpack with the Rust field names as map keys
/// (`rmp_serde::to_vec_named` in node_service.rs). If someone "fixes"
/// the interop bug by renaming the internal serde fields instead of using
/// these DTOs, this test fails and points them here.
#[test]
fn internal_bucket_stats_rpc_wire_stays_snake_case() {
let stats = sample_bucket_stats();
let encoded = rmp_serde::to_vec_named(&stats).expect("internal stats should encode");
let value: serde_json::Value = rmp_serde::from_slice(&encoded).expect("named msgpack should decode generically");
assert!(
value.get("replication_stats").is_some(),
"peer RPC key replication_stats must not be renamed"
);
assert!(value["replication_stats"].get("q_stat").is_some());
assert!(value.get("queue_stats").is_some());
assert!(value.get("proxy_stats").is_some());
let decoded: BucketStats = rmp_serde::from_slice(&encoded).expect("round-trip through the peer RPC wire");
assert_eq!(decoded.replication_stats.replica_count, 2);
}
}
+4 -1
View File
@@ -1459,10 +1459,13 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
REPLICATION_DIFF,
RouteRiskLevel::Sensitive,
),
// The default stream enumerates object names/version ids and requires
// ReplicationDiff (MinIO parity); only ?aggregate=true relaxes to
// GetReplicationMetrics in the handler.
admin(
HttpMethod::Get,
"/rustfs/admin/v3/replication/mrf",
GET_REPLICATION_METRICS,
REPLICATION_DIFF,
RouteRiskLevel::Sensitive,
),
];
+67 -13
View File
@@ -1548,7 +1548,8 @@ async fn build_replication_metrics_response(
let bucket_stats = apply_replication_metrics_bandwidth_report(bucket_stats, collect_replication_metrics_bandwidth(bucket));
let bucket_stats = apply_replication_metrics_runtime_fields(bucket_stats, route, replication_metrics_uptime_seconds());
let body = serialize_replication_metrics_body(&bucket_stats, route)?;
let node_name = crate::runtime_sources::current_local_node_name().await.unwrap_or_default();
let body = serialize_replication_metrics_body(&bucket_stats, route, &node_name)?;
let mut resp = S3Response::with_status(Body::from(body), StatusCode::OK);
resp.headers
@@ -1608,12 +1609,24 @@ fn apply_replication_metrics_runtime_fields(
bucket_stats
}
fn serialize_replication_metrics_body(bucket_stats: &BucketStats, route: ReplicationExtRoute) -> S3Result<Vec<u8>> {
/// Serialize the metrics body in the minio-go wire shapes
/// (`replication.Metrics` for v1, `replication.MetricsV2` for v2). The
/// internal `BucketStats` serde names are the intra-cluster peer-RPC wire
/// format and must never appear here — see
/// `crate::admin::replication_metrics_wire`.
fn serialize_replication_metrics_body(
bucket_stats: &BucketStats,
route: ReplicationExtRoute,
node_name: &str,
) -> S3Result<Vec<u8>> {
use crate::admin::replication_metrics_wire::{MetricsV2Wire, MetricsWire};
match route {
ReplicationExtRoute::MetricsV1 => {
serde_json::to_vec(&bucket_stats.replication_stats).map_err(|e| s3_error!(InternalError, "{e}"))
serde_json::to_vec(&MetricsWire::from(&bucket_stats.replication_stats)).map_err(|e| s3_error!(InternalError, "{e}"))
}
ReplicationExtRoute::MetricsV2 => {
serde_json::to_vec(&MetricsV2Wire::from_stats(bucket_stats, node_name)).map_err(|e| s3_error!(InternalError, "{e}"))
}
ReplicationExtRoute::MetricsV2 => serde_json::to_vec(bucket_stats).map_err(|e| s3_error!(InternalError, "{e}")),
ReplicationExtRoute::Check | ReplicationExtRoute::ResetStart | ReplicationExtRoute::ResetStatus => {
Err(s3_error!(InternalError, "invalid route for metrics response"))
}
@@ -4147,22 +4160,37 @@ mod tests {
assert!(err.message().unwrap_or_default().contains("rule-stale"));
}
/// The v1 body must decode into minio-go `replication.Metrics` (exact
/// json tags); Go's decoder matches case-insensitively but does not
/// ignore underscores, so the internal snake_case names read as all-zero.
#[test]
fn serialize_replication_metrics_body_v1_returns_replication_stats_only() {
fn serialize_replication_metrics_body_v1_returns_minio_go_metrics_shape() {
let mut stats = BucketStats {
uptime: 99,
..Default::default()
};
stats.replication_stats.replica_count = 7;
stats.replication_stats.replicated_size = 2048;
stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default()
.replicated_count = 5;
stats.proxy_stats.put_total = 3;
let body =
serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV1).expect("metrics v1 body should serialize");
let body = serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV1, "node-1:9000")
.expect("metrics v1 body should serialize");
let payload: serde_json::Value = serde_json::from_slice(&body).expect("body should be json");
assert_eq!(payload["replica_count"], 7);
assert_eq!(payload["replicaCount"], 7);
assert_eq!(payload["completedReplicationSize"], 2048);
assert_eq!(payload["Stats"]["arn:minio:replication::t:b"]["replicationCount"], 5);
assert!(payload.get("uptime").is_none());
assert!(payload.get("proxy_stats").is_none());
// The internal snake_case names must not leak into the wire body.
assert!(payload.get("replica_count").is_none());
assert!(payload.get("q_stat").is_none());
}
#[test]
@@ -4248,22 +4276,48 @@ mod tests {
assert_eq!(target.current_bandwidth_bytes_per_sec, 3000.0);
}
/// The v2 body must decode into minio-go `replication.MetricsV2`
/// (`uptime`/`currStats`/`queueStats`); `mc replicate status` reads
/// `currStats` and `queueStats.nodes` and silently shows zeros when the
/// keys do not match.
#[test]
fn serialize_replication_metrics_body_v2_returns_full_bucket_stats() {
fn serialize_replication_metrics_body_v2_returns_minio_go_metrics_v2_shape() {
let mut stats = BucketStats {
uptime: 99,
..Default::default()
};
stats.replication_stats.replica_count = 7;
stats
.replication_stats
.q_stat
.curr
.now_count
.store(4, std::sync::atomic::Ordering::Relaxed);
stats
.replication_stats
.q_stat
.curr
.now_bytes
.store(1200, std::sync::atomic::Ordering::Relaxed);
stats.replication_stats.q_stat = stats.replication_stats.q_stat.snapshot();
stats.proxy_stats.put_total = 3;
let body =
serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV2).expect("metrics v2 body should serialize");
let body = serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV2, "node-1:9000")
.expect("metrics v2 body should serialize");
let payload: serde_json::Value = serde_json::from_slice(&body).expect("body should be json");
assert_eq!(payload["uptime"], 99);
assert_eq!(payload["replication_stats"]["replica_count"], 7);
assert_eq!(payload["proxy_stats"]["put_total"], 3);
assert_eq!(payload["currStats"]["replicaCount"], 7);
assert_eq!(payload["currStats"]["queued"]["curr"]["count"], 4.0);
// The queue snapshot must surface at least one node: mc derives the
// worker/queue panels from queueStats.nodes and treats an empty list
// as "no data".
assert_eq!(payload["queueStats"]["nodes"][0]["queueStats"]["curr"]["count"], 4.0);
assert_eq!(payload["queueStats"]["nodes"][0]["uptime"], 99);
// The internal snake_case names must not leak into the wire body.
assert!(payload.get("replication_stats").is_none());
assert!(payload.get("queue_stats").is_none());
assert!(payload.get("proxy_stats").is_none());
}
#[test]
+16 -49
View File
@@ -18,28 +18,21 @@
//! `config/site-replication/state.json` is mutated by read-modify-write
//! sequences spread over many call sites: admin handlers, the retry-event
//! writers on every hook broadcast path, and the service-side reload driven
//! over node RPC. Historically only some of them held the process-local
//! mutex and none held a distributed lock across the whole RMW, so
//! concurrent writers overwrote each other (single-process for the unlocked
//! writers, cross-node for everyone).
//! over node RPC.
//!
//! `with_site_replication_state_lock` is the single transaction boundary:
//! it holds the process-local mutex AND the distributed config-object write
//! lock (the pattern proven by the repair state,
//! `update_site_replication_repair_state`) for the duration of the caller's
//! closure. All IO inside the closure must use the `*_no_lock` config
//! helpers — the locked variants would self-deadlock on the same object
//! lock. Do not perform peer network calls or take other config locks
//! it holds the distributed config-object write lock (the pattern proven by
//! the repair state, `update_site_replication_repair_state`) for the
//! duration of the caller's closure. The object lock is the sole mechanism —
//! it is the only thing that can serialize two nodes of the same site, so a
//! process-local lock must never be reintroduced in front of it as if it
//! added protection. All IO inside the closure must use the `*_no_lock`
//! config helpers — the locked variants would self-deadlock on the same
//! object lock. Do not perform peer network calls or take other config locks
//! inside the closure.
//!
//! The process-local mutex is transitional: call sites still outside this
//! primitive serialize against migrated ones through it. Once every RMW
//! call site goes through here (P1-15 PR2) it will be removed, leaving the
//! object lock as the only mechanism.
//!
//! Lock order (unchanged from the historical comment next to the mutex):
//! lifecycle -> bucket operation -> repair admission -> state (process
//! mutex, then state object lock) -> per-bucket metadata.
//! Lock order: lifecycle -> bucket operation -> repair admission
//! -> state object lock -> per-bucket metadata.
use crate::admin::storage_api::runtime::ECStore;
use crate::admin::storage_api::s3::{S3Error, S3ErrorCode, S3Result};
@@ -53,24 +46,8 @@ use super::runtime_sources::current_object_store_handle;
/// byte-level tolerant reload on the service side.
pub(crate) const SITE_REPLICATION_STATE_PATH: &str = "config/site-replication/state.json";
/// Transitional process-local mutex — see the module docs. Stays private to
/// this module (owner-local static, enforced by
/// `scripts/check_architecture_migration_rules.sh`); callers go through
/// [`site_replication_state_process_guard`].
static SITE_REPLICATION_STATE_LOCK: std::sync::LazyLock<tokio::sync::Mutex<()>> =
std::sync::LazyLock::new(|| tokio::sync::Mutex::new(()));
/// Owner helper for the transitional process mutex: the RMW call sites in
/// `handlers::site_replication` that PR2 has not migrated to
/// [`with_site_replication_state_lock`] yet hold this guard so they stay
/// mutually exclusive with the migrated ones. Removed together with the
/// mutex once every call site runs inside the transaction boundary.
pub(crate) async fn site_replication_state_process_guard() -> tokio::sync::MutexGuard<'static, ()> {
SITE_REPLICATION_STATE_LOCK.lock().await
}
/// Run `operation` under the site-replication state transaction boundary:
/// process mutex first, then the distributed state-object write lock.
/// the distributed state-object write lock.
pub(crate) async fn with_site_replication_state_lock<T, F, Fut>(operation: F) -> S3Result<T>
where
T: Send + 'static,
@@ -83,21 +60,11 @@ where
/// Context-store variant for callers that resolve their store from an
/// explicit [`AppContext`] (the service-side reload driven over node RPC).
///
/// This is the whole boundary: a state-object write lock serializes writers
/// in *different* processes, which is what two nodes of one site are and
/// what a process mutex could never cover.
pub(crate) async fn with_site_replication_state_lock_on<T, F, Fut>(store: Arc<ECStore>, operation: F) -> S3Result<T>
where
T: Send + 'static,
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = S3Result<T>> + Send + 'static,
{
let _process_guard = SITE_REPLICATION_STATE_LOCK.lock().await;
with_site_replication_state_object_lock(store, operation).await
}
/// The distributed half of the boundary on its own: the state-object write
/// lock, without the process mutex. This is the only thing that serializes
/// writers in *different* processes (the mutex cannot), so it is also what
/// the separate-nodes regression test drives.
pub(crate) async fn with_site_replication_state_object_lock<T, F, Fut>(store: Arc<ECStore>, operation: F) -> S3Result<T>
where
T: Send + 'static,
F: FnOnce() -> Fut + Send + 'static,
+32
View File
@@ -321,6 +321,34 @@ pub(crate) mod metadata_sys {
crate::storage::storage_api::acquire_bucket_metadata_transaction_lock(bucket).await
}
pub(crate) async fn acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket: &str,
expected_incarnation_id: uuid::Uuid,
) -> Result<super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard> {
super::ecstore_bucket::metadata_sys::acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket,
expected_incarnation_id,
)
.await
}
pub(crate) async fn update_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
data: Vec<u8>,
) -> Result<OffsetDateTime> {
super::ecstore_bucket::metadata_sys::update_under_transaction_lock(guard, bucket, config_file, data).await
}
pub(crate) async fn delete_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
) -> Result<OffsetDateTime> {
super::ecstore_bucket::metadata_sys::delete_under_transaction_lock(guard, bucket, config_file).await
}
pub(crate) async fn update_bucket_targets_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
@@ -417,6 +445,10 @@ pub(crate) mod replication {
};
pub(crate) type BucketReplicationResyncStatus = super::ecstore_bucket::replication::BucketReplicationResyncStatus;
pub(crate) type BucketStats = super::ecstore_bucket::replication::BucketStats;
pub(crate) type BucketReplicationStats = super::ecstore_bucket::replication::BucketReplicationStats;
pub(crate) type BucketReplicationStat = super::ecstore_bucket::replication::BucketReplicationStat;
pub(crate) type InQueueMetric = super::ecstore_bucket::replication::InQueueMetric;
pub(crate) type XferStats = super::ecstore_bucket::replication::XferStats;
pub(crate) type ReplicationStatusType = super::ecstore_bucket::replication::ReplicationStatusType;
pub(crate) type ResyncOpts = super::ecstore_bucket::replication::ResyncOpts;
pub(crate) type ResyncStatusType = super::ecstore_bucket::replication::ResyncStatusType;
+39 -2
View File
@@ -1158,6 +1158,19 @@ fn lifecycle_has_expiry_rules(config: &BucketLifecycleConfiguration) -> bool {
})
}
/// Status-independent presence of the expiry subset that site replication
/// propagates (`replicateILMExpiry`): expiration / noncurrent-version
/// expiration only. Distinct from [`lifecycle_has_expiry_rules`], which
/// filters on ENABLED for scanner scheduling — editing a Disabled expiry rule
/// must still advance the replication axis. Del-marker expiration and
/// abort-multipart are site-local and never travel.
fn lifecycle_rules_have_expiry(config: &BucketLifecycleConfiguration) -> bool {
config
.rules
.iter()
.any(|rule| rule.expiration.is_some() || rule.noncurrent_version_expiration.is_some())
}
fn lifecycle_has_abort_multipart_rules(config: &BucketLifecycleConfiguration) -> bool {
config.rules.iter().any(|rule| {
rule.status == ExpirationStatus::from_static(ExpirationStatus::ENABLED)
@@ -2186,7 +2199,24 @@ impl DefaultBucketUsecase {
return Err(s3_error!(InvalidArgument, "{err}"));
}
input_cfg.expiry_updated_at = Some(Timestamp::from(time::OffsetDateTime::now_utc()));
// Stamp the expiry axis only when the expiry subset can have changed
// (MinIO: HasExpiry() || expiryRuleRemoved). Site-replication peers
// judge lc-config staleness on this axis; a transition-only edit that
// advanced it would let this site's stale expiry subset shadow — and
// roll back — a newer peer expiry edit fleet-wide.
let previous_expiry_updated_at = match metadata_sys::get_lifecycle_config(&bucket).await {
Ok((previous, _)) => {
if lifecycle_rules_have_expiry(&input_cfg) || lifecycle_rules_have_expiry(&previous) {
Some(Timestamp::from(time::OffsetDateTime::now_utc()))
} else {
previous.expiry_updated_at
}
}
// No previous config (or unreadable): stamping is the
// conservative pre-existing behavior.
Err(_) => lifecycle_rules_have_expiry(&input_cfg).then(|| Timestamp::from(time::OffsetDateTime::now_utc())),
};
input_cfg.expiry_updated_at = previous_expiry_updated_at;
let data = serialize_config(&input_cfg)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_LIFECYCLE_CONFIG, data, expected_incarnation_id)
.await
@@ -2197,7 +2227,14 @@ impl DefaultBucketUsecase {
let mut item = sr_bucket_meta_item(bucket.clone(), "lc-config");
item.expiry_lc_config =
Some(serialize_config(&input_cfg).and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?);
item.expiry_updated_at = item.updated_at;
// The item travels with the expiry axis, not the wall clock: a site
// whose expiry knowledge is old (or absent — UNIX_EPOCH) must not
// out-rank newer peer expiry state at the receivers.
item.expiry_updated_at = input_cfg
.expiry_updated_at
.clone()
.map(time::OffsetDateTime::from)
.or(Some(time::OffsetDateTime::UNIX_EPOCH));
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket lifecycle hook failed");
}
+65 -6
View File
@@ -27,6 +27,7 @@ use super::storage_api::multipart_usecase::bucket::{
replication::{must_replicate_object, schedule_object_replication},
versioning_sys::BucketVersioningSys,
};
use super::storage_api::multipart_usecase::compression::{is_disk_compressible, is_multipart_disk_compression_enabled};
#[cfg(test)]
use super::storage_api::multipart_usecase::contract::http::HTTPPreconditions;
use super::storage_api::multipart_usecase::contract::multipart::{CompletePart, MultipartOperations as _, MultipartUploadResult};
@@ -39,12 +40,12 @@ use super::storage_api::multipart_usecase::error::{StorageError, is_err_object_n
use super::storage_api::multipart_usecase::helper::OperationHelper;
#[cfg(test)]
use super::storage_api::multipart_usecase::io::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader};
use super::storage_api::multipart_usecase::io::{HashReader, WriteEncryption, WritePlan};
use super::storage_api::multipart_usecase::io::{HashReader, WriteEncryption, WritePlan, compression_metadata_value};
use super::storage_api::multipart_usecase::object_utils::to_s3s_etag;
use super::storage_api::multipart_usecase::options::{
copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts_with_replication_authorization,
get_content_sha256_with_query, get_opts, namespace_reserved_user_metadata, parse_copy_source_range,
put_opts_with_replication_authorization, validate_archive_content_encoding,
get_content_sha256_with_query, get_opts, has_replication_retention_update, namespace_reserved_user_metadata,
parse_copy_source_range, put_opts_with_replication_authorization, validate_archive_content_encoding,
};
use super::storage_api::multipart_usecase::request_context::spawn_traced_join;
use super::storage_api::multipart_usecase::s3_api::multipart::{
@@ -210,6 +211,28 @@ fn create_multipart_upload_metadata(
metadata
}
/// A multipart session advertises disk compression only when the staged-rollout
/// switch (`RUSTFS_COMPRESSION_MULTIPART_ENABLED`) is on, the object key/headers
/// qualify, AND the session is not an SSE-C ciphertext-passthrough replication
/// session, which must preserve source bytes verbatim.
///
/// The rollout switch defaults to off so a rolling upgrade never creates new
/// compressed multipart objects while pre-fix nodes (whose decompressor is not
/// resumable) may still serve reads. Enable it once the fleet has converged on a
/// fixed build; the default flips per the `multipart-compression-default-off-window`
/// entry in docs/architecture/compat-cleanup-register.md.
///
/// Each part is compressed as an independent stream; the GET path decodes across part
/// boundaries (see `ReadTransform::Compressed`), so the session may advertise
/// object-level compression again.
///
/// Unlike single PUT there is no `MIN_DISK_COMPRESSIBLE_SIZE` floor here: the total
/// object size is unknown at CreateMultipartUpload time, so tiny multipart objects pay
/// the (harmless) framing overhead. This is a deliberate trade-off, not a bug.
fn should_advertise_session_compression(multipart_enabled: bool, ciphertext_passthrough: bool, disk_compressible: bool) -> bool {
multipart_enabled && !ciphertext_passthrough && disk_compressible
}
async fn validate_table_catalog_object_mutation(bucket: &str, key: &str) -> S3Result<()> {
table_catalog::validate_bucket_object_mutation(bucket, key)
.await
@@ -754,7 +777,9 @@ impl DefaultMultipartUsecase {
let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
|| has_replication_retention_update(&req.headers, replication_authorized);
let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata(
&bucket,
@@ -837,8 +862,17 @@ impl DefaultMultipartUsecase {
None => (None, None),
};
// Multipart parts are independent physical streams. Advertising object-level
// compression here would make GET decode the completed object as one stream.
if should_advertise_session_compression(
is_multipart_disk_compression_enabled(),
ciphertext_passthrough,
is_disk_compressible(&req.headers, &key),
) {
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_COMPRESSION,
compression_metadata_value(CompressionAlgorithm::default()),
);
}
let mt2 = metadata.clone();
let mut opts: ObjectOptions =
@@ -1632,6 +1666,31 @@ mod tests {
DefaultMultipartUsecase::without_context()
}
#[test]
fn session_compression_is_advertised_only_for_non_passthrough_compressible_uploads() {
// (multipart_enabled, ciphertext_passthrough, disk_compressible, expected)
let cases = [
(true, false, false, false),
(true, false, true, true),
(true, true, false, false),
(true, true, true, false),
// The staged-rollout switch keeps multipart compression dark by
// default regardless of the other gates.
(false, false, true, false),
(false, false, false, false),
(false, true, true, false),
(false, true, false, false),
];
for (multipart_enabled, ciphertext_passthrough, disk_compressible, expected) in cases {
assert_eq!(
should_advertise_session_compression(multipart_enabled, ciphertext_passthrough, disk_compressible),
expected,
"multipart_enabled={multipart_enabled} ciphertext_passthrough={ciphertext_passthrough} disk_compressible={disk_compressible}"
);
}
}
#[test]
fn quota_accounting_uses_logical_size_when_available() {
let mut metadata = HashMap::new();
+24 -7
View File
@@ -86,8 +86,8 @@ use super::storage_api::object_usecase::object_utils::to_s3s_etag;
use super::storage_api::object_usecase::options::{
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_content_sha256_with_query, get_opts,
namespace_reserved_user_metadata, normalize_content_encoding_for_storage, preserve_unclassified_user_metadata,
put_opts_with_replication_authorization, validate_archive_content_encoding,
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
};
use super::storage_api::object_usecase::request_context::{self, spawn_traced, spawn_traced_join};
use super::storage_api::object_usecase::s3_api::multipart::parse_list_parts_params;
@@ -2241,10 +2241,11 @@ fn get_object_resume_control(ctx: GetObjectResumeContext) -> GetObjectResumeCont
/// disks" failures keep the existing fail-loud behavior.
fn is_object_relocation_error(err: &std::io::Error) -> bool {
let Some(inner) = err.get_ref() else { return false };
matches!(
inner.downcast_ref::<StorageError>(),
Some(StorageError::FileNotFound | StorageError::ObjectNotFound(..) | StorageError::InsufficientReadQuorum(..))
)
match inner.downcast_ref::<StorageError>() {
Some(StorageError::FileNotFound | StorageError::ObjectNotFound(..) | StorageError::InsufficientReadQuorum(..)) => true,
Some(StorageError::Io(source)) => source.kind() == std::io::ErrorKind::NotFound,
_ => false,
}
}
/// Resolve the S3 request-body inter-chunk read timeout from the environment.
@@ -5798,7 +5799,9 @@ impl DefaultObjectUsecase {
)?;
let mut metadata = metadata.unwrap_or_default();
let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
|| has_replication_retention_update(&req.headers, inbound_replication_put);
let object_lock_config_stage_start = put_stage_metrics_enabled.then(Instant::now);
let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
rustfs_io_metrics::record_put_object_stage_duration_from("app_object_lock_config_lookup", object_lock_config_stage_start);
@@ -10080,6 +10083,19 @@ mod tests {
assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some("COMPLIANCE"));
assert!(metadata.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER));
assert_eq!(metadata.get("x-amz-meta-x-amz-object-lock-mode").map(String::as_str), Some("GOVERNANCE"));
let mut replication_headers = HeaderMap::new();
insert_header(&mut replication_headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
insert_header(
&mut replication_headers,
rustfs_utils::http::SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
"2026-01-01T00:00:00Z",
);
let mut replica_metadata = HashMap::new();
let explicit_clear = has_replication_retention_update(&replication_headers, true);
apply_bucket_default_lock_retention("bucket", &state, &mut replica_metadata, explicit_clear).unwrap();
assert!(!replica_metadata.contains_key(AMZ_OBJECT_LOCK_MODE_LOWER));
assert!(!replica_metadata.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER));
}
fn pax_record(key: &str, value: &[u8]) -> Vec<u8> {
@@ -13163,6 +13179,7 @@ mod tests {
StorageError::FileNotFound,
StorageError::ObjectNotFound("test-bucket".to_string(), "relocated-object".to_string()),
StorageError::InsufficientReadQuorum("test-bucket".to_string(), "relocated-object".to_string()),
StorageError::Io(std::io::Error::new(std::io::ErrorKind::NotFound, "relocated shard disappeared")),
] {
let reopen_count = Arc::new(AtomicUsize::new(0));
let control = counting_resume_control(Arc::clone(&reopen_count), |emitted| {
+7 -4
View File
@@ -942,7 +942,9 @@ pub(crate) mod concurrency {
}
pub(crate) mod compression {
pub(crate) use crate::storage::storage_api::ecstore_compression::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
pub(crate) use crate::storage::storage_api::ecstore_compression::{
MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible, is_multipart_disk_compression_enabled,
};
}
pub(crate) mod deadlock_detector {
@@ -994,8 +996,9 @@ pub(crate) mod options {
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
extract_metadata_from_mime, extract_metadata_from_mime_with_object_name, filter_object_metadata,
get_complete_multipart_upload_opts_with_replication_authorization, get_content_sha256_with_query, get_opts,
namespace_reserved_user_metadata, normalize_content_encoding_for_storage, parse_copy_source_range,
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
parse_copy_source_range, preserve_unclassified_user_metadata, put_opts_with_replication_authorization,
validate_archive_content_encoding,
};
}
@@ -1153,7 +1156,7 @@ pub(crate) mod multipart_usecase {
}
pub(crate) use super::{
access, bucket, data_usage, error, helper, io, object_utils, options, request_context, s3_api, set_disk, sse,
access, bucket, compression, data_usage, error, helper, io, object_utils, options, request_context, s3_api, set_disk, sse,
};
pub(crate) use crate::storage::storage_api::{ECStore, StorageObjectInfo, StorageObjectOptions, StoragePutObjReader};
}
+49 -4
View File
@@ -326,7 +326,11 @@ impl From<StorageError> for ApiError {
_ => S3ErrorCode::InternalError,
};
let message = if matches!(&err, StorageError::QuotaExceeded { .. }) || code == S3ErrorCode::InternalError {
let message = if matches!(&err, StorageError::QuotaExceeded { .. }) {
err.to_string()
} else if code == S3ErrorCode::InternalError && matches!(&err, StorageError::Io(_)) {
ApiError::error_code_to_message(&code)
} else if code == S3ErrorCode::InternalError {
err.to_string()
} else if let StorageError::InvalidArgument(_, _, reason) = &err
&& !reason.is_empty()
@@ -525,6 +529,25 @@ mod tests {
assert!(api_error.source.is_some());
}
#[test]
fn storage_io_internal_error_redacts_public_message_and_retains_source() {
let sensitive_path = "/sensitive/storage/path";
let api_error = ApiError::from(StorageError::Io(IoError::new(
ErrorKind::PermissionDenied,
format!("permission denied: {sensitive_path}"),
)));
assert_eq!(api_error.code, S3ErrorCode::InternalError);
assert_eq!(api_error.message, ApiError::error_code_to_message(&S3ErrorCode::InternalError));
assert!(!api_error.message.contains(sensitive_path));
let source = api_error
.source
.as_deref()
.and_then(|source| source.downcast_ref::<StorageError>())
.expect("API error should retain the storage error source");
assert!(matches!(source, StorageError::Io(io_error) if io_error.to_string().contains(sensitive_path)));
}
#[test]
fn test_kms_service_unavailable_maps_to_retryable_error() {
let api_error = ApiError::from(StorageError::other(KmsUnavailableError));
@@ -669,14 +692,36 @@ mod tests {
assert!(api_error.source.is_some());
}
#[test]
fn test_api_error_from_storage_io_copy_object_terminal_error_stays_internal() {
let io_error = IoError::other(StorageError::FileCorrupt);
let storage_error: StorageError = io_error.into();
assert!(matches!(storage_error, StorageError::FileCorrupt));
let api_error: ApiError = storage_error.into();
assert_eq!(api_error.code, S3ErrorCode::InternalError);
let source = api_error
.source
.as_deref()
.and_then(|source| source.downcast_ref::<StorageError>())
.expect("API error should retain the storage error source");
assert!(matches!(source, StorageError::FileCorrupt));
}
#[test]
fn test_api_error_from_iam_error() {
let iam_error = rustfs_iam::error::Error::other("IAM test error");
let api_error: ApiError = iam_error.into();
// IAM error is first converted to StorageError, then to ApiError
assert!(api_error.source.is_some());
assert!(api_error.message.contains("test error"));
assert_eq!(api_error.code, S3ErrorCode::InternalError);
assert_eq!(api_error.message, ApiError::error_code_to_message(&S3ErrorCode::InternalError));
let source = api_error
.source
.as_deref()
.and_then(|source| source.downcast_ref::<StorageError>())
.expect("API error should retain the storage error source");
assert!(matches!(source, StorageError::Io(io_error) if io_error.to_string().contains("IAM test error")));
}
#[test]

Some files were not shown because too many files have changed in this diff Show More