Compare commits

...

17 Commits

Author SHA1 Message Date
houseme a118d7e4fd perf(ecstore): enable inline data read early-stop by default (#6140)
* perf(ecstore): enable inline data read early-stop by default

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(scanner): box large ILM transition flow future

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align internal meta early-stop miss

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-16 06:38:51 +00:00
Zhengchao An ed1bedf1fb fix(storage): skip table guards for multipart parts (#6143) 2026-08-16 14:34:46 +08:00
houseme 4392f94e1a chore(deps): update flake.lock (#6142)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/70ce234' (2026-08-06)
  → 'github:NixOS/nixpkgs/8be7bd0' (2026-08-14)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/6df7076' (2026-08-09)
  → 'github:oxalica/rust-overlay/b211ead' (2026-08-16)
2026-08-16 13:35:41 +08:00
Zhengchao An 81d7b7d07a chore(ecstore): drop the client dead_code blanket (#6138) 2026-08-16 03:20:52 +00:00
唐小鸭 e26668e62c fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition (#6128)
* test(ecstore): pin madmin-compatible ARN partition contract

Red-light evidence for backlog#1675 P1-7: madmin-go's ParseARN
hard-rejects any ARN that does not start with 'arn:minio:', while RustFS
generates and only accepts 'arn:rustfs:'. mc/madmin tooling therefore
cannot decode RustFS remote-target listings, and MinIO-era replication
configs are rejected as StaleTarget when re-registered. The new tests
pin the target contract (generate arn:minio:, parse both partitions,
reject unknown partitions) and fail against the current single-partition
gate.

* fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition

madmin-go's ParseARN hard-rejects any partition other than 'arn:minio:',
so native mc/madmin tooling could not decode RustFS remote-target
listings, and re-registering a MinIO-era replication config failed its
StaleTarget check against freshly minted arn:rustfs: targets
(backlog#1675 P1-7, route A).

- ARN Display now emits 'arn:minio:'; FromStr accepts a {minio, rustfs}
  partition whitelist (the legacy partition stays readable forever for
  persisted bucket-targets.json / replication configs). The whitelist is
  the only structural gate — BucketTargetType::from_str never fails —
  so it deliberately rejects foreign partitions such as arn:aws:.
- No data migration: every runtime match between targets, rules and
  stats keys is full-string equality, so existing arn:rustfs: targets
  keep matching their persisted rules; site replication already
  preserves MinIO-era ARNs on reconcile (pinned by existing tests).
- Rolling upgrade note: upgrade all cluster nodes before creating new
  remote targets — a not-yet-upgraded node rejects remove-remote-target
  for a freshly minted arn:minio: ARN with BucketRemoteArnInvalid.
- Out of scope: notification/SQS ARNs (crates/targets) keep the
  arn:rustfs:sqs: partition; they have their own compatibility story.
2026-08-16 10:28:45 +08:00
Henry Guo 8d3511c1b3 fix(heal): preserve timeout budget across retries (#6101)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-16 10:28:11 +08:00
Henry Guo d172d05e86 fix(ecstore): overlap metacache reader deadlines (#6098)
* fix(ecstore): overlap metacache reader deadlines

* fix(admin): avoid span guards across awaits

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-16 10:27:59 +08:00
Zhengchao An 0d86c50760 fix(ecstore): classify remote inline early-stop misses (#6136) 2026-08-16 09:50:16 +08:00
houseme 526d6f667e perf(ecstore): defer pending inline data shards (#6137) 2026-08-16 09:44:00 +08:00
唐小鸭 dcf3e4b9e8 fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold (#6129)
* test(replication): pin missing LWW timestamp header transport

Red-light tests for the replication timestamp three-header contract:

- put_object_headers_carry_replication_timestamp_headers pins that
  PutObjectOptions::header() must emit the
  x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
  headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
  and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
  pin that an authorized replication PUT / multipart complete must persist
  the inbound timestamps into the internal metadata keys while unauthorized
  requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
  (ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
  assertions can observe what a real target receives; self-test included.

* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold

Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.

- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
  timestamp from PutObjectOptions::header(); names and RFC3339 values
  interoperate with MinIO (minio-go constants.go, object-api-options.go),
  pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
  epoch values in header(), so "never modified" is not sent as a
  modification made now.
- Parse the headers only on authorized replication PUTs and multipart
  completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
  persist them into the dual-prefix internal metadata keys so the
  outbound pass (replication_target_boundary) reads the source's
  timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
  DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
  without it the sender only ever had the mod_time fallback to offer.

Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.

* fix(replication): load the stored tagging timestamp independently of remaining tags

Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.

* fix(storage): reserve replication transport names at metadata ingest

Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.

The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.

* fix(replication): harden timestamp replay

* fix(app): route retention helper through facade

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:56:04 +08:00
唐小鸭 04b9c8fd36 fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf (#6126)
* test(admin): pin madmin ReplicationMRF stream contract for /v3/replication/mrf

Red-light evidence for backlog#1675 P1-13 (mrf half): madmin's
BucketReplicationMRF decodes the response one ReplicationMRF document at
a time, so the current aggregate envelope decodes as a single phantom
row with an empty object in 'mc replicate backlog'. The new contract
tests assert the desired bare-document stream (exact madmin json tags,
empty body for an empty backlog) and fail against the current
render_mrf_backlog extraction, which preserves the envelope-only
behavior:

- mrf_stream_renders_bare_madmin_documents: envelope keys leak, no
  per-entry documents
- mrf_stream_renders_empty_body_for_no_entries: empty backlog still
  renders the envelope (phantom row)
- mrf_aggregate_envelope_retains_counters: PerObjectEntriesAvailable
  never advertises the enumerable stream

* fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf

The mrf endpoint returned a single aggregate envelope, which madmin's
json.Decoder loop decoded as one phantom row (empty object) in
'mc replicate backlog' (backlog#1675 P1-13, mrf half; the diff half was
fixed in #5799 and this mirrors its pattern).

- Default response is now a bare stream of ReplicationMRF documents
  (exact madmin json tags; Size/TargetARNs as ignored extension keys)
  built from the durable backlog ledger; an empty backlog renders an
  empty body, so mc shows zero rows instead of a phantom row.
- The aggregate counter envelope moves behind ?aggregate=true (RustFS
  extension) and now advertises PerObjectEntriesAvailable whenever the
  durable backlog is readable.
- An unreadable backlog is signalled out-of-band via
  x-rustfs-replication-mrf-backlog-unavailable (mirrors the diff
  truncation header) plus a warn event, since the bare stream cannot
  carry source health.
- The madmin node parameter is accepted but documented as a no-op: the
  durable ledger is cluster-shared with no per-node attribution.
- Delete-marker purge entries fall back to the marker version id so
  those rows keep a version identity.

* fix(admin): fail the mrf stream request when the durable ledger is unreadable

Review: madmin only decodes the body of a 200, so the out-of-band
unavailability header was invisible to it and an unreadable ledger read
as a clean zero-row backlog. Stream mode now returns 503; aggregate
mode keeps the availability fields.

* fix(admin): gate, bound, and null-map the mrf stream

Second review round:

- Authorization: the default stream enumerates object names and version
  ids, which a metrics-only principal must not see — it now requires
  admin:ReplicationDiff (MinIO parity, route policy updated);
  ?aggregate=true carries no object identities and keeps
  admin:GetReplicationMetrics.
- The nil UUID is RustFS's in-memory null-version sentinel and now
  leaves as the S3 wire token 'null' instead of a zero UUID (a
  pre-versioning object scanned after versioning + existing-object
  replication can persist it into the ledger).
- The durable ledger is not bounded by the in-memory pending cap and
  the body is buffered before send; the stream now stops at 10,000
  documents and signals truncation via
  x-rustfs-replication-mrf-truncated (mirroring the diff endpoint)
  plus a warn event, instead of staging an unbounded body.

* fix(admin): reject truncated MRF streams

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-08-16 05:55:15 +08:00
唐小鸭 c1f66969d7 fix(site-replication): merge incoming ILM expiry documents instead of overwriting (#6130)
* test(site-replication): pin ILM expiry merge contract for incoming lc-config

Red-light evidence for backlog#1675 P1-1: the lc-config receiver
overwrites the whole local lifecycle config with whatever the peer
sends (and deletes it wholesale on peer delete), so an expiry-only
document erases the receiver's local tier/transition rules, and peer
transition rules get installed across sites. The new tests pin the
MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening:

- incoming expiry documents merge with (never replace) local rules
- local transition sides are authoritative for same-id rules
- incoming transition fields are discarded at the trust boundary
- dropped expiry rules strip the expiry side but keep transitions;
  pure-expiry rules are removed
- delete merges with the empty set instead of dropping the config
- disabled rules survive; abort-mpu-only rules stay site-local
- deterministic order (idempotent re-delivery) and expiry_updated_at
  stamping for the staleness axis

All fail against the current overwrite implementation (identity
extraction of merge_incoming_lifecycle_config).

* fix(site-replication): merge incoming ILM expiry documents instead of overwriting

The lc-config receiver replaced the whole local lifecycle config with
the peer's document (and deleted it wholesale on peer delete), so an
expiry-only update erased the receiver's local tier/transition rules,
and a peer's transition rules were installed across sites
(backlog#1675 P1-1).

Receiver (apply_bucket_meta_item):
- lc-config now merges via merge_incoming_lifecycle_config, mirroring
  MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening:
  incoming transition fields are discarded outright; the local
  transition side of a same-id rule is authoritative. A peer delete
  merges with the empty set — pure-expiry rules go away, transition
  rules survive with their expiry side cleared, and only an empty
  result deletes the config file.
- Staleness moves to the expiry axis (config.expiry_updated_at):
  lifecycle_config_updated_at also moves on local transition-only
  edits, which shadowed newer peer expiry updates.
- Receiver-side replicateILMExpiry gate, symmetric with the sender
  hook (previously any peer could install expiry rules while the
  option was off).
- Rule order is deterministic (local order, incoming-new appended), so
  re-delivering the same document is byte-stable and does not rewrite
  bucket metadata per broadcast.

Sender:
- Both admin choke points — the bucket-meta hook and the SRInfo bucket
  entry feeding bootstrap/repair and consistency views — now emit only
  the expiry subset (transition fields stripped, non-expiry rules
  dropped). MinIO receivers install incoming rules verbatim, so
  transition rules must never leave the site. An unparseable local
  config is forwarded unfiltered rather than degraded to a delete.

Not covered here (follow-up): a two-site e2e with a real tier backend
to exercise transition-rule preservation end to end; receiver-side
validate_transition_tier for merged configs.

* fix(site-replication): close ILM merge review findings

Adversarial review of the lc-config merge surfaced four real defects,
all fixed here:

- Deletion tombstone regression: with the staleness axis moved to the
  in-config expiry_updated_at, a deleted lifecycle config fell back to
  UNIX_EPOCH and any delayed stale broadcast could resurrect deleted
  expiry rules. The axis now falls back to the whole-config write time
  (which survives deletion in bucket metadata as the deletion's lower
  bound), also covering legacy configs that predate the axis field.
- MinIO zero-rule documents: MinIO's delete tombstone / transition-only
  state marshals a lifecycle document with no <Rule>, which the strict
  s3s deserializer rejects — the receiver now recognizes it as the 'no
  expiry rules here' statement (delete semantics) instead of erroring
  on every MinIO heal pass.
- Inflated expiry axis at the sender: PutBucketLifecycle stamped
  expiry_updated_at unconditionally, so a transition-only edit advanced
  the axis and let this site's stale expiry subset shadow and roll back
  newer peer expiry edits fleet-wide. The stamp is now conditional
  (expiry subset present before or after the edit, MinIO parity), the
  hook item travels with the config's expiry axis (UNIX_EPOCH when the
  site has none), and the SRInfo bucket entry feeds bootstrap/repair
  the same axis instead of the whole-config write time.
- Del-marker parity: MinIO's CloneNonTransition never emits del-marker
  or abort-mpu fields, so treating del_marker_expiration as traveling
  expiry let a MinIO broadcast delete this site's del-marker-only
  rules. Both fields are now site-local on every edge: stripped from
  outbound subsets and inbound rules, restored from the local side on
  same-id merges, and never a deletion criterion.

Receiver-side validation of merged configs (object-lock / tier
constraints, MinIO runs finalLcCfg.Validate) remains a follow-up.

* fix(site-replication): close the second ILM review round

- Missed-delete repair: a deleted expiry state now travels through
  bootstrap/repair as an explicit timestamped lc-config delete item
  (lifecycle_expiry_statement distinguishes deletion — whole-config
  write time advanced past the created backfill — from never-configured
  buckets and from transition-only configs without an expiry axis,
  which say nothing). A peer that missed the live delete converges on
  repair; the receiver's staleness guard protects newer peer state.
- Strict tombstone recognition: only a well-delimited zero-rule
  <LifecycleConfiguration> document maps to delete semantics; truncated
  or foreign payloads that fail the strict deserializer are rejected
  instead of being treated as a delete that erases local expiry rules.
- Staleness fallback axis narrowed: the whole-config write time is used
  only for deleted or legacy-with-expiry state. A present
  transition-only config without an expiry axis compares at epoch — its
  whole-config time moves on transition edits and must not shadow or
  block independent peer expiry updates and same-timestamp repairs.

* fix(site-replication): validate tombstone children structurally

Second review round: a well-delimited root could still smuggle
malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt>
</LifecycleConfiguration> passed the no-<Rule check and was applied as
a delete. The tombstone body must now be a sequence of well-formed
simple children (matching open/close or self-closing, no nested markup,
no stray text, none named Rule); anything else surfaces InvalidRequest.
Malformed-child cases pinned in the recognition test.

* fix(site-replication): serialize lifecycle merges

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:34:17 +08:00
唐小鸭 cfa9276fad fix(admin): serialize replication metrics in minio-go wire shapes (#6127)
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics

Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2]
serializes the internal snake_case BucketStats family straight onto the
wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase
tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is
case-insensitive but does not ignore underscores, so 'mc replicate
status' shows all zeros without any error. The rewritten snapshot tests
assert the minio-go tags (plus a synthesized queueStats node — the
aggregation path leaves queue_stats.nodes empty today) and fail against
the current pass-through serialization.

* fix(admin): serialize replication metrics in minio-go wire shapes

?replication-metrics[=2] and the admin replicationmetrics endpoint
serialized the internal snake_case BucketStats family straight onto the
wire, so 'mc replicate status' decoded all zeros without any error
(backlog#1675 P1-11). The internal structs cannot be renamed: they are
the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in
node_service.rs), pinned by a new regression test.

- New admin/replication_metrics_wire.rs: Serialize-only projections onto
  minio-go replication.Metrics (v1 body, currStats) and MetricsV2
  (uptime/currStats/queueStats/downtimeInfo) with the exact json tags;
  per-target failed becomes the TimedErrStats envelope fed from the
  FailStats rolling window; the queue peak is dual-emitted as max
  (MinIO server tag) and peak (minio-go tag).
- queueStats synthesizes one node from the bucket queue snapshot — the
  aggregation path leaves queue_stats.nodes empty, and mc treats an
  empty node list as 'no data' — and carries transfer summaries
  (Large/Small/Total) derived from the per-target xfer rates.
- Both endpoints share the DTOs; source-health extension keys
  (provider_available/cluster_complete/...) ride along and are ignored
  by Go decoders.
- Widen the ecstore replication_stats_boundary re-exports
  (BucketReplicationStat/InQueueMetric/XferStats) so the admin facade
  chain can name the projected types.

* fix(replication): carry failure rolling windows through cluster aggregation

Review: both metrics endpoints aggregate first, and FailStats::merge
dropped the process-local samples (which also never cross the peer-RPC
wire — serde-skipped), so lastMinute/lastHour serialized as zero right
after a failure while totals was nonzero.

- FailStats gains serializable last_minute/last_hour window snapshots
  (serde default: old nodes read zeros, new fields are ignored by old
  decoders), recomputed on every add_size and re-stamped at the
  per-node collection point (get_latest_replication_stats), and summed
  by merge.
- The wire DTO takes the component-wise max of the live samples and the
  snapshot, so both the single-node and the aggregated path report the
  window.
- Regression test drives a stat through rmp round trip + merge before
  serialization, as requested.

Also restore the #[allow(dead_code)] attribute to route_policy — the
new module declaration had been inserted between the attribute and its
item, which broke the -D warnings CI lanes.

* fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path

Second review round:

- update_xfer_rate split at 1 MiB while the minio-go transferSummary
  labels (and RustFS's own worker-pool split) mean >= 128 MiB for
  Large, so a 2 MiB replication reported under Large with Small stuck
  at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2
  assertion covers 2 MiB / 127 MiB / exactly 128 MiB.
- add_size no longer recomputes the rolling windows: two full
  one-hour-deque scans per failure under the bucket-stats write lock
  made failure bursts quadratic (30k events ~2.1s). The windows are
  stamped only at the collection point (get_latest_replication_stats,
  which serves both the local leg and the peer RPC); the aggregation
  regression now drives that path explicitly before the RPC round trip
  and merge.

* fix(replication): average transfer summaries

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 05:27:23 +08:00
Henry Guo db8f55cb97 feat(table-catalog): finalize Iceberg REST behavior (#6072)
* feat(table-catalog): finalize Iceberg REST behavior

* fix(table-catalog): address REST finalization regressions

* test(table-catalog): expect REST commit conflicts

* test(table-catalog): avoid serialized view test deadlocks

* fix(table-catalog): adapt shared test backend

* fix(table-catalog): enforce Iceberg metadata invariants

* fix(table-catalog): preserve manifest length in test

* test(table-catalog): use valid metadata fixtures

* test(table-catalog): seed manifests before manifest lists

* fix(table-catalog): restore validation gates

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-08-16 03:05:09 +08:00
houseme 7f23a1ba91 feat(ecstore): report inline early-stop miss reasons (#6134)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-15 17:18:26 +00:00
Henry Guo 1619c4be60 fix(scanner): add context to corrupt metadata logs (#6099)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-15 21:32:05 +08:00
Zhengchao An 72fd7339c9 test(utils): allow ephemeral port reuse (#6122)
* test(utils): allow ephemeral port reuse

* test(kms): allow any ciphertext prefix
2026-08-15 08:32:10 +08:00
80 changed files with 12303 additions and 1880 deletions
Generated
+4
View File
@@ -278,6 +278,7 @@ checksum = "312c1ea69e5fe9966e0029fb95aca8790100b85aff4f0d3b00a9337c74069a9c"
dependencies = [
"bigdecimal",
"bon",
"crc32fast",
"digest 0.11.3",
"log",
"miniz_oxide 0.9.1",
@@ -289,9 +290,11 @@ dependencies = [
"serde",
"serde_bytes",
"serde_json",
"snap",
"strum",
"thiserror 2.0.20",
"uuid",
"zstd",
]
[[package]]
@@ -9200,6 +9203,7 @@ dependencies = [
"serial_test",
"sha2 0.11.0",
"shadow-rs",
"snap",
"socket2",
"subtle",
"sysinfo",
+1 -1
View File
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
tower-http = { version = "0.7.0" }
# Serialization and Data Formats
apache-avro = "0.22.0"
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
bytesize = "2.7.0"
byteorder = "1.5.0"
+91 -1
View File
@@ -76,6 +76,18 @@ const SOURCE_MTIME_HEADERS: [&str; 2] = ["x-rustfs-source-mtime", "x-minio-sourc
const SOURCE_REPLICATION_REQUEST_HEADERS: [&str; 2] =
["x-rustfs-source-replication-request", "x-minio-source-replication-request"];
const SOURCE_ETAG_HEADERS: [&str; 2] = ["x-rustfs-source-etag", "x-minio-source-etag"];
const SOURCE_TAGGING_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-tagging-timestamp",
"x-minio-source-replication-tagging-timestamp",
];
const SOURCE_RETENTION_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-retention-timestamp",
"x-minio-source-replication-retention-timestamp",
];
const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -118,6 +130,25 @@ pub enum FaultAction {
WrongEtag,
}
/// Replication LWW timestamp headers observed on a request, journaled so
/// sender-side tests can assert what a real target would receive.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ReplicationTimestampHeaders {
pub tagging: Option<String>,
pub retention: Option<String>,
pub legalhold: Option<String>,
}
impl ReplicationTimestampHeaders {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
tagging: header_value(headers, &SOURCE_TAGGING_TIMESTAMP_HEADERS).map(bounded_journal_value),
retention: header_value(headers, &SOURCE_RETENTION_TIMESTAMP_HEADERS).map(bounded_journal_value),
legalhold: header_value(headers, &SOURCE_LEGALHOLD_TIMESTAMP_HEADERS).map(bounded_journal_value),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -131,6 +162,7 @@ pub struct RequestRecord {
pub part_number: Option<i32>,
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub fault: Option<FaultAction>,
}
@@ -536,7 +568,15 @@ impl S3Access for FaultAccess {
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let fault = record_request(&self.control, operation, context.method().clone(), parsed, content_length);
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
context.method().clone(),
parsed,
content_length,
replication_timestamps,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
..
@@ -589,6 +629,7 @@ fn record_request(
method: Method,
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -613,6 +654,7 @@ fn record_request(
part_number: parsed.part_number,
content_length,
consumed_bytes: None,
replication_timestamps,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -1699,6 +1741,52 @@ mod tests {
.await?)
}
#[tokio::test]
async fn journals_replication_timestamp_headers() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
client
.put_object()
.bucket("target-bucket")
.key("plain")
.body(ByteStream::from_static(b"plain"))
.send()
.await?;
client
.put_object()
.bucket("target-bucket")
.key("stamped")
.body(ByteStream::from_static(b"stamped"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-source-replication-tagging-timestamp", "2026-01-02T03:04:05Z");
headers.insert("x-minio-source-replication-retention-timestamp", "2026-01-02T03:04:06Z");
headers.insert("x-rustfs-source-replication-legalhold-timestamp", "2026-01-02T03:04:07Z");
Ok::<_, std::convert::Infallible>(request)
})
.send()
.await?;
let requests = target.requests();
let plain = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some("plain"))
.expect("plain PUT must be journaled");
assert_eq!(plain.replication_timestamps, ReplicationTimestampHeaders::default());
let stamped = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some("stamped"))
.expect("stamped PUT must be journaled");
assert_eq!(stamped.replication_timestamps.tagging.as_deref(), Some("2026-01-02T03:04:05Z"));
assert_eq!(stamped.replication_timestamps.retention.as_deref(), Some("2026-01-02T03:04:06Z"));
assert_eq!(stamped.replication_timestamps.legalhold.as_deref(), Some("2026-01-02T03:04:07Z"));
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -2985,6 +3073,7 @@ mod tests {
part_number: None,
},
Some(0),
ReplicationTimestampHeaders::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3006,6 +3095,7 @@ mod tests {
part_number: None,
},
None,
ReplicationTimestampHeaders::default(),
);
{
let bounded_records = lock(&bounded_control);
+14 -12
View File
@@ -135,7 +135,8 @@ pub mod bucket {
pub use crate::bucket::metadata_sys::ConfigWriteLockProbe;
pub use crate::bucket::metadata_sys::{
BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, get, get_accelerate_config, get_bucket_policy,
acquire_bucket_metadata_transaction_lock_for_incarnation, capture_bucket_metadata_incarnation, delete,
delete_if_incarnation, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
@@ -184,17 +185,18 @@ pub mod bucket {
mrf_backlog_observability_snapshot,
};
pub use crate::bucket::replication::{
BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
BucketReplicationResyncStatus, BucketReplicationStat, BucketReplicationStats, BucketStats,
DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, InQueueMetric,
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION,
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE,
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
+70 -4
View File
@@ -58,7 +58,9 @@ use rustfs_utils::http::{
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
insert_header,
};
use rustls_pki_types::pem::PemObject;
use serde::{Deserialize, Serialize};
@@ -1476,9 +1478,12 @@ impl Default for AdvancedPutOptions {
replication_status: ReplicationStatusType::Pending,
source_mtime: OffsetDateTime::now_utc(),
replication_request: false,
retention_timestamp: OffsetDateTime::now_utc(),
tagging_timestamp: OffsetDateTime::now_utc(),
legalhold_timestamp: OffsetDateTime::now_utc(),
// UNIX_EPOCH means "never modified": header() must not emit a
// timestamp header for it, otherwise a receiver would treat an
// unset category as a modification made right now.
retention_timestamp: OffsetDateTime::UNIX_EPOCH,
tagging_timestamp: OffsetDateTime::UNIX_EPOCH,
legalhold_timestamp: OffsetDateTime::UNIX_EPOCH,
replication_validity_check: false,
}
}
@@ -1675,6 +1680,16 @@ impl PutObjectOptions {
);
}
for (suffix, timestamp) in [
(SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, self.internal.tagging_timestamp),
(SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, self.internal.retention_timestamp),
(SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, self.internal.legalhold_timestamp),
] {
if timestamp.unix_timestamp() != 0 {
insert_header(&mut header, suffix, timestamp.format(&Rfc3339).unwrap_or_default());
}
}
if self.internal.replication_request {
insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
}
@@ -2842,6 +2857,57 @@ mod tests {
);
}
#[test]
fn put_object_headers_carry_replication_timestamp_headers() {
// MinIO receivers resolve concurrent tag/retention/legal-hold edits by
// last-writer-wins on these headers (object-api-options.go parses them
// as RFC3339); a replica without them loses every conflict resolution.
let mut opts = PutObjectOptions::default();
opts.internal.replication_request = true;
let tagging = OffsetDateTime::from_unix_timestamp(1_700_000_001).expect("valid timestamp");
let retention = OffsetDateTime::from_unix_timestamp(1_700_000_002).expect("valid timestamp");
let legalhold = OffsetDateTime::from_unix_timestamp(1_700_000_003).expect("valid timestamp");
opts.internal.tagging_timestamp = tagging;
opts.internal.retention_timestamp = retention;
opts.internal.legalhold_timestamp = legalhold;
let header = opts.header();
for (suffix, expected) in [
("source-replication-tagging-timestamp", tagging),
("source-replication-retention-timestamp", retention),
("source-replication-legalhold-timestamp", legalhold),
] {
assert_eq!(
rustfs_utils::http::get_header(&header, suffix).as_deref(),
Some(expected.format(&Rfc3339).expect("RFC3339 timestamp").as_str()),
"replication put requests must carry the {suffix} header"
);
}
}
#[test]
fn put_object_headers_omit_unset_replication_timestamps() {
// UNIX_EPOCH means "never modified on the source"; sending it would
// make the receiver treat an unset category as a fresh modification.
let mut opts = PutObjectOptions::default();
opts.internal.replication_request = true;
opts.internal.tagging_timestamp = OffsetDateTime::UNIX_EPOCH;
opts.internal.retention_timestamp = OffsetDateTime::UNIX_EPOCH;
opts.internal.legalhold_timestamp = OffsetDateTime::UNIX_EPOCH;
let header = opts.header();
for suffix in [
"source-replication-tagging-timestamp",
"source-replication-retention-timestamp",
"source-replication-legalhold-timestamp",
] {
assert!(
rustfs_utils::http::get_header(&header, suffix).is_none(),
"unset {suffix} must not be sent to replication targets"
);
}
}
#[tokio::test]
async fn get_remote_target_client_internal_rejects_loopback_endpoint() {
let sys = BucketTargetSys::default();
+18
View File
@@ -656,6 +656,16 @@ pub async fn update_under_transaction_lock(
update_under_config_write_guard(get_bucket_metadata_sys()?, guard, config_file, data).await
}
/// Clear one config file while the caller holds this bucket's transaction lock.
pub async fn delete_under_transaction_lock(
guard: &BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
) -> Result<OffsetDateTime> {
guard.ensure_valid(bucket)?;
delete_under_config_write_guard(get_bucket_metadata_sys()?, guard, config_file).await
}
pub async fn update_quota_if_incarnation(
bucket: &str,
data: Vec<u8>,
@@ -795,6 +805,14 @@ pub async fn acquire_bucket_metadata_transaction_lock(bucket: &str) -> Result<Bu
acquire_config_write_guard(get_bucket_metadata_sys()?, bucket).await
}
/// Acquire the bucket transaction lock only if its incarnation still matches.
pub async fn acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket: &str,
expected_incarnation_id: Uuid,
) -> Result<BucketMetadataMutationGuard> {
acquire_config_write_guard_for_incarnation(get_bucket_metadata_sys()?, bucket, Some(expected_incarnation_id)).await
}
pub(crate) async fn acquire_bucket_metadata_transaction_lock_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
+1 -1
View File
@@ -81,6 +81,6 @@ pub use replication_queue_boundary::{
pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
pub use replication_scanner_bridge::ReplicationScannerBridge;
pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
pub use replication_stats_boundary::{BucketReplicationStats, BucketStats};
pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;
@@ -704,6 +704,12 @@ impl ReplicationStats {
} else {
BucketReplicationStats::new()
};
// Stamp the serializable failure windows from the live samples: the
// samples themselves do not cross the peer-RPC wire, so this snapshot
// is what cluster aggregation and the metrics endpoints see.
for stat in replication_stats.stats.values_mut() {
stat.fail_stats.refresh_windows();
}
let uptime = if cache.contains_key(bucket) {
SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
@@ -15,7 +15,9 @@
#[cfg(test)]
pub(crate) use rustfs_replication::FailStats;
pub(crate) use rustfs_replication::{
ActiveWorkerStat, BucketReplicationStat, InQueueMetric, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope,
SRMetricsSummary, XferStats,
ActiveWorkerStat, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope, SRMetricsSummary,
};
pub use rustfs_replication::{BucketReplicationStats, BucketStats};
// Public so the admin wire DTOs (rustfs/src/admin/replication_metrics_wire.rs)
// can project the internal stats onto the minio-go response shapes through
// the storage_api facade chain.
pub use rustfs_replication::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
@@ -27,8 +27,10 @@ use rustfs_utils::http::{
AMZ_OBJECT_TAGGING, AMZ_SERVER_SIDE_ENCRYPTION, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT, AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID,
AMZ_STORAGE_CLASS, AMZ_TAG_COUNT, CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE,
HeaderExt as _, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP,
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_TAGGING_TIMESTAMP, get_str, insert_header_map,
is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key, ssec_replication_transport_header,
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_TAGGING_TIMESTAMP,
get_str, insert_header_map, is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key,
ssec_replication_transport_header,
};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
@@ -119,6 +121,27 @@ fn classify_replication_source_encryption(metadata: &HashMap<String, String>) ->
}
}
fn is_legacy_source_replication_timestamp_key(key: &str) -> bool {
fn has_prefix_and_suffix(key: &str, prefix: &str, suffix: &str) -> bool {
let key = key.as_bytes();
key.len() == prefix.len() + suffix.len()
&& key[..prefix.len()].eq_ignore_ascii_case(prefix.as_bytes())
&& key[prefix.len()..].eq_ignore_ascii_case(suffix.as_bytes())
}
[
SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
]
.iter()
.any(|suffix| {
["x-rustfs-", "x-minio-"]
.iter()
.any(|prefix| has_prefix_and_suffix(key, prefix, suffix))
})
}
pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String, String>) -> bool {
rustfs_replication::is_ssec_encrypted(user_defined)
}
@@ -176,6 +199,11 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
continue;
}
if is_legacy_source_replication_timestamp_key(key) {
meta.insert(format!("x-amz-meta-{key}"), value.to_string());
continue;
}
if is_internal_key(key) || is_standard_header(key) {
continue;
}
@@ -259,15 +287,23 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
if !tags.is_empty() {
put_options.user_tags = tags;
put_options.internal.tagging_timestamp =
if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP) {
OffsetDateTime::parse(&timestamp, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse tagging timestamp: {err}")))?
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
}
}
// Load the stored tagging timestamp independently of whether any tags
// remain: DeleteObjectTagging leaves the object tagless but stamps this
// key, and the deletion's LWW timestamp must still reach the replica.
// With no stored key, fall back to mod_time only while tags exist
// (MinIO parity); a tagless object without the key was never tagged and
// keeps the epoch default (no header).
put_options.internal.tagging_timestamp = if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP)
{
OffsetDateTime::parse(&timestamp, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse tagging timestamp: {err}")))?
} else if !put_options.user_tags.is_empty() {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
OffsetDateTime::UNIX_EPOCH
};
let metadata = &*object_info.user_defined;
@@ -283,13 +319,15 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
put_options.cache_control = cache_control.to_string();
}
if let Some(mode) = metadata.lookup(AMZ_OBJECT_LOCK_MODE) {
if let Some(mode) = metadata.lookup(AMZ_OBJECT_LOCK_MODE).filter(|mode| !mode.is_empty()) {
put_options.mode = Some(ObjectLockRetentionMode::from(mode.to_uppercase().as_str()));
}
if let Some(retain_until_date) = metadata.lookup(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE) {
put_options.retain_until_date = OffsetDateTime::parse(retain_until_date, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse retain until date: {err}")))?;
if !retain_until_date.is_empty() {
put_options.retain_until_date = OffsetDateTime::parse(retain_until_date, &Rfc3339)
.map_err(|err| Error::other(format!("Failed to parse retain until date: {err}")))?;
}
put_options.internal.retention_timestamp =
if let Some(timestamp) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP) {
OffsetDateTime::parse(&timestamp, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
@@ -694,6 +732,110 @@ mod tests {
assert!(options.internal.replication_request);
}
/// DeleteObjectTagging leaves the object tagless but stamps the
/// tagging-timestamp internal key; the deletion's LWW timestamp must
/// still be loaded (and therefore sent) so the replica can order the
/// deletion against concurrent tag edits.
#[test]
fn replication_put_options_carry_tagging_timestamp_after_tag_deletion() {
let mut metadata = std::collections::HashMap::new();
rustfs_utils::http::insert_str(&mut metadata, SUFFIX_TAGGING_TIMESTAMP, "2026-01-02T03:04:05Z".to_string());
let object_info = ObjectInfo {
user_defined: Arc::new(metadata),
user_tags: Arc::new(String::new()),
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
version_id: Some(Uuid::nil()),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
assert!(options.user_tags.is_empty());
assert_eq!(
options.internal.tagging_timestamp,
OffsetDateTime::parse("2026-01-02T03:04:05Z", &Rfc3339).expect("valid timestamp"),
"the stored tagging timestamp must load independently of remaining tags"
);
// A tagless object without the stored key was never tagged: the epoch
// default keeps the header unsent.
let untagged = ObjectInfo {
user_tags: Arc::new(String::new()),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("timestamp")),
version_id: Some(Uuid::nil()),
..Default::default()
};
let (options, _) = replication_put_object_options("", &untagged).expect("build put options");
assert_eq!(options.internal.tagging_timestamp, OffsetDateTime::UNIX_EPOCH);
}
#[test]
fn replication_put_options_do_not_promote_legacy_user_timestamp_metadata() {
let legacy_keys = [
"x-rustfs-source-replication-tagging-timestamp",
"x-rustfs-source-replication-retention-timestamp",
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-tagging-timestamp",
"x-minio-source-replication-retention-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
let object_info = ObjectInfo {
user_defined: Arc::new(
legacy_keys
.iter()
.map(|key| (key.to_string(), "2099-01-02T03:04:05Z".to_string()))
.collect(),
),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
for legacy_key in legacy_keys {
assert!(!options.user_metadata.contains_key(legacy_key));
assert_eq!(
options
.user_metadata
.get(&format!("x-amz-meta-{legacy_key}"))
.map(String::as_str),
Some("2099-01-02T03:04:05Z")
);
}
assert_eq!(options.internal.tagging_timestamp, OffsetDateTime::UNIX_EPOCH);
assert_eq!(options.internal.retention_timestamp, OffsetDateTime::UNIX_EPOCH);
assert_eq!(options.internal.legalhold_timestamp, OffsetDateTime::UNIX_EPOCH);
}
#[test]
fn replication_put_options_carry_retention_timestamp_after_clear() {
let mut metadata = HashMap::from([
(AMZ_OBJECT_LOCK_MODE.to_string(), String::new()),
(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), String::new()),
]);
rustfs_utils::http::insert_str(&mut metadata, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, "2026-01-02T03:04:05Z".to_string());
let object_info = ObjectInfo {
user_defined: Arc::new(metadata),
..Default::default()
};
let (options, _) = replication_put_object_options("", &object_info).expect("retention clear must replicate");
assert!(options.mode.is_none());
assert_eq!(options.retain_until_date, OffsetDateTime::UNIX_EPOCH);
assert_eq!(
options.internal.retention_timestamp,
OffsetDateTime::parse("2026-01-02T03:04:05Z", &Rfc3339).expect("valid timestamp")
);
let headers = options.header();
assert!(!headers.contains_key(AMZ_OBJECT_LOCK_MODE));
assert!(!headers.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE));
assert_eq!(
rustfs_utils::http::get_header(&headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP).as_deref(),
Some("2026-01-02T03:04:05Z")
);
}
#[test]
fn replication_put_options_strip_encryption_metadata_from_plaintext_objects() {
use rustfs_utils::http::object_encryption_keys::{INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER};
+52 -4
View File
@@ -40,7 +40,14 @@ impl ARN {
impl Display for ARN {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "arn:rustfs:{}:{}:{}:{}", self.arn_type, self.region, self.id, self.bucket)
// The `minio` partition is deliberate: madmin-go's ParseARN
// hard-rejects any other partition, so native mc/madmin tooling can
// only decode remote-target ARNs minted in this form (backlog#1675
// P1-7). Legacy `arn:rustfs:` ARNs persisted by older releases stay
// readable via the FromStr whitelist below; runtime matching between
// targets and replication rules is by full-string equality, so mixed
// partitions coexist safely.
write!(f, "arn:minio:{}:{}:{}:{}", self.arn_type, self.region, self.id, self.bucket)
}
}
@@ -48,7 +55,12 @@ impl FromStr for ARN {
type Err = std::io::Error;
fn from_str(s: &str) -> Result<Self, Self::Err> {
if !s.starts_with("arn:rustfs:") {
// Partition whitelist, not just an `arn:` check: `BucketTargetType::
// from_str(...).unwrap_or_default()` below never fails, so this is
// the only structural gate rejecting foreign ARNs. `arn:rustfs:` is
// the legacy partition and must stay accepted forever (persisted
// bucket-targets.json / replication configs from older releases).
if !s.starts_with("arn:minio:") && !s.starts_with("arn:rustfs:") {
return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "Invalid ARN format"));
}
@@ -101,14 +113,50 @@ mod tests {
}
/// RustFS commonly generates ARNs with an empty region:
/// `arn:rustfs:replication::<deployment_id>:<bucket>`.
/// `arn:minio:replication::<deployment_id>:<bucket>`.
#[test]
fn from_str_handles_empty_region_segment() {
let parsed = ARN::from_str("arn:rustfs:replication::depl-123:bucket-a").expect("valid ARN must parse");
let parsed = ARN::from_str("arn:minio:replication::depl-123:bucket-a").expect("valid ARN must parse");
assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
assert_eq!(parsed.region, "", "region segment is empty in this form");
assert_eq!(parsed.id, "depl-123");
assert_eq!(parsed.bucket, "bucket-a");
}
/// madmin-go's `ParseARN` hard-rejects anything that does not start with
/// `arn:minio:`, so generated ARNs must use the `minio` partition or the
/// native mc/madmin tooling cannot decode remote-target listings.
#[test]
fn display_emits_minio_partition() {
let arn = ARN::new(
BucketTargetType::ReplicationService,
"depl-123".to_string(),
String::new(),
"bucket-a".to_string(),
);
assert_eq!(arn.to_string(), "arn:minio:replication::depl-123:bucket-a");
}
/// Persisted bucket-targets.json files from older RustFS releases carry
/// `arn:rustfs:` ARNs; the legacy partition must stay parseable forever.
#[test]
fn from_str_accepts_legacy_rustfs_partition() {
let parsed = ARN::from_str("arn:rustfs:replication:us-east-1:depl-123:bucket-a").expect("legacy ARN must parse");
assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
assert_eq!(parsed.region, "us-east-1");
assert_eq!(parsed.id, "depl-123");
assert_eq!(parsed.bucket, "bucket-a");
}
/// The partition whitelist is the only structural gate: `BucketTargetType::
/// from_str(...).unwrap_or_default()` never fails, so any 6-segment string
/// would otherwise parse as `type=None`.
#[test]
fn from_str_rejects_unknown_partition() {
assert!(ARN::from_str("arn:aws:replication::depl-123:bucket-a").is_err());
assert!(ARN::from_str("not-an-arn").is_err());
}
}
@@ -15,6 +15,7 @@
use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
use crate::disk::error::DiskError;
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
use futures::future::join_all;
use metrics::counter;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
use std::{
@@ -655,6 +656,7 @@ async fn list_path_raw_inner(
errs.push(None);
}
let mut pending_entries: Vec<Option<MetaCacheEntry>> = vec![None; readers.len()];
let mut peek_outcomes: Vec<Option<PeekOutcome>> = std::iter::repeat_with(|| None).take(readers.len()).collect();
loop {
let mut current = MetaCacheEntry::default();
@@ -676,6 +678,21 @@ async fn list_path_raw_inner(
let mut has_err = 0;
let mut agree = 0;
// Start every missing head read in the same round so one stalled
// disk cannot multiply the wait budget by the erasure-set width.
// Outcomes are still consumed below in stable disk-index order.
let concurrent_peeks = readers.iter_mut().enumerate().filter_map(|(i, reader)| {
if errs[i].is_some() || pending_entries[i].is_some() {
return None;
}
let cancel = &revjob_rx;
Some(async move { (i, peek_with_timeout(cancel, reader, peek_timeout).await) })
});
for (i, outcome) in join_all(concurrent_peeks).await {
peek_outcomes[i] = Some(outcome);
}
for (i, r) in readers.iter_mut().enumerate() {
if errs[i].is_some() {
has_err += 1;
@@ -685,7 +702,10 @@ async fn list_path_raw_inner(
let entry = if let Some(entry) = pending_entries[i].take() {
entry
} else {
match peek_with_timeout(&revjob_rx, r, peek_timeout).await {
let Some(outcome) = peek_outcomes[i].take() else {
return Err(DiskError::Unexpected);
};
match outcome {
PeekOutcome::Ready(res) => {
if let Some(entry) = res {
// info!("read entry disk: {}, name: {}", i, entry.name);
@@ -1295,6 +1315,36 @@ mod tests {
assert_eq!(err, DiskError::Timeout);
}
#[tokio::test(start_paused = true)]
async fn list_path_raw_bounds_multiple_stalled_readers_by_one_peek_deadline() {
let peek_timeout = Duration::from_millis(20);
let started = tokio::time::Instant::now();
let err = list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![None, None, None, None],
min_disks: 1,
test_reader_behaviors: vec![
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
TestReaderBehavior::Stall,
],
peek_timeout: Some(peek_timeout),
..Default::default()
},
)
.await
.expect_err("all stalled readers should fail the listing");
assert_eq!(err, DiskError::Timeout);
assert_eq!(
started.elapsed(),
peek_timeout,
"reader deadlines must overlap instead of accumulating once per disk"
);
}
#[tokio::test]
async fn list_path_raw_waits_past_producer_stall_for_slow_progressing_reader() {
let entry = MetaCacheEntry {
@@ -229,17 +229,6 @@ pub fn http_resp_to_error_response(
err_resp
}
pub fn err_transfer_acceleration_bucket(bucket_name: &str) -> ErrorResponse {
ErrorResponse {
status_code: StatusCode::BAD_REQUEST,
code: S3ErrorCode::InvalidArgument,
message: "The name of the bucket used for Transfer Acceleration must be DNS-compliant and must not contain periods .."
.to_string(),
bucket_name: bucket_name.to_string(),
..Default::default()
}
}
pub fn err_entity_too_large(total_size: i64, max_object_size: i64, bucket_name: &str, object_name: &str) -> ErrorResponse {
let msg = format!(
"Your proposed upload size {} exceeds the maximum allowed object size {} for single PUT operation.",
@@ -295,16 +284,6 @@ pub fn err_invalid_argument(message: &str) -> ErrorResponse {
}
}
pub fn err_api_not_supported(message: &str) -> ErrorResponse {
ErrorResponse {
status_code: StatusCode::NOT_IMPLEMENTED,
code: S3ErrorCode::Custom("APINotSupported".into()),
message: message.to_string(),
request_id: "rustfs".to_string(),
..Default::default()
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -135,6 +135,10 @@ impl Object {
Self { ..Default::default() }
}
#[allow(
dead_code,
reason = "MinIO-parity reader surface with no caller in this port (backlog#1823)"
)]
fn do_get_request(&self, request: &GetRequest) -> Result<GetResponse, std::io::Error> {
let _ = request.did_offset_change;
let _ = request.offset;
@@ -150,12 +154,20 @@ impl Object {
))
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn set_offset(&mut self, bytes_read: i64) -> Result<(), std::io::Error> {
self.curr_offset += bytes_read;
Ok(())
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn read(&mut self, b: &[u8]) -> Result<i64, std::io::Error> {
let mut read_req = GetRequest {
is_read_op: true,
@@ -180,6 +192,10 @@ impl Object {
Ok(response.size)
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn stat(&self) -> Result<ObjectInfo, std::io::Error> {
if !self.is_started || !self.object_info_set {
let _ = self.do_get_request(&GetRequest {
@@ -192,6 +208,10 @@ impl Object {
Ok(self.object_info.clone())
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn read_at(&mut self, b: &[u8], offset: i64) -> Result<i64, std::io::Error> {
self.curr_offset = offset;
@@ -219,6 +239,10 @@ impl Object {
Ok(response.size)
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn seek(&mut self, offset: i64, whence: i64) -> Result<i64, std::io::Error> {
if !self.is_started || !self.object_info_set {
let seek_req = GetRequest {
@@ -253,6 +277,10 @@ impl Object {
Ok(self.curr_offset)
}
#[allow(
dead_code,
reason = "MinIO-parity Object reader method with no caller in this port (backlog#1823)"
)]
fn close(&mut self) -> Result<(), std::io::Error> {
self.is_closed = true;
Ok(())
+1 -1
View File
@@ -37,7 +37,7 @@ use crate::client::{
api_put_object_common::optimal_part_info,
api_put_object_multipart::UploadPartParams,
api_s3_datatypes::{CompleteMultipartUpload, CompletePart, ObjectPart},
constants::{ISO8601_DATEFORMAT, MAX_MULTIPART_PUT_OBJECT_SIZE, MIN_PART_SIZE, TOTAL_WORKERS},
constants::{ISO8601_DATEFORMAT, MAX_MULTIPART_PUT_OBJECT_SIZE, MIN_PART_SIZE},
credentials::SignatureType,
transition_api::{ReaderImpl, TransitionClient, UploadInfo},
utils::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header},
@@ -30,10 +30,6 @@ pub fn is_object(reader: &ReaderImpl) -> bool {
matches!(reader, ReaderImpl::ObjectBody(_))
}
pub fn is_read_at(reader: ReaderImpl) -> bool {
matches!(reader, ReaderImpl::ObjectBody(_))
}
pub fn optimal_part_info(object_size: i64, configured_part_size: u64) -> Result<(i64, i64, i64), std::io::Error> {
let unknown_size;
let mut object_size = object_size;
@@ -81,18 +81,6 @@ async fn read_multipart_part(reader: &mut ReaderImpl, want: usize) -> Result<Vec
}
}
pub struct UploadedPartRes {
pub error: std::io::Error,
pub part_num: i64,
pub size: i64,
pub part: ObjectPart,
}
pub struct UploadPartReq {
pub part_num: i64,
pub part: ObjectPart,
}
impl TransitionClient {
pub async fn put_object_multipart_stream(
self: Arc<Self>,
+18 -38
View File
@@ -29,10 +29,6 @@ use crate::client::utils::base64_decode;
use super::transition_api;
pub struct ListAllMyBucketsResult {
pub owner: Owner,
}
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct CommonPrefix {
pub prefix: String,
@@ -89,6 +85,10 @@ pub struct ListVersionsResult {
pub next_version_id_marker: String,
}
#[allow(
dead_code,
reason = "fields of a MinIO-parity list result that this port builds but never reads back (backlog#1823)"
)]
pub struct ListBucketResult {
common_prefixes: Vec<CommonPrefix>,
contents: Vec<transition_api::ObjectInfo>,
@@ -102,6 +102,10 @@ pub struct ListBucketResult {
prefix: String,
}
#[allow(
dead_code,
reason = "fields of a MinIO-parity list result that this port builds but never reads back (backlog#1823)"
)]
pub struct ListMultipartUploadsResult {
bucket: String,
key_marker: String,
@@ -117,16 +121,15 @@ pub struct ListMultipartUploadsResult {
common_prefixes: Vec<CommonPrefix>,
}
#[allow(
dead_code,
reason = "fields of a MinIO-parity list result that this port builds but never reads back (backlog#1823)"
)]
pub struct Initiator {
id: String,
display_name: String,
}
pub struct CopyObjectResult {
pub etag: String,
pub last_modified: OffsetDateTime,
}
#[derive(Debug, Clone)]
pub struct ObjectPart {
pub etag: String,
@@ -260,6 +263,7 @@ pub struct CompletePart {
}
impl CompletePart {
#[allow(dead_code, reason = "MinIO-parity accessor with no caller in this port (backlog#1823)")]
fn checksum(&self, t: &ChecksumMode) -> String {
match t {
ChecksumMode::ChecksumCRC32C => {
@@ -284,11 +288,6 @@ impl CompletePart {
}
}
pub struct CopyObjectPartResult {
pub etag: String,
pub last_modified: OffsetDateTime,
}
#[derive(Debug, Default, serde::Serialize)]
#[serde(rename = "CompleteMultipartUpload")]
pub struct CompleteMultipartUpload {
@@ -357,10 +356,10 @@ impl CompleteMultipartUpload {
}
}
pub struct CreateBucketConfiguration {
pub location: String,
}
#[allow(
dead_code,
reason = "live via quick_xml::de::from_str in bucket_cache.rs; serde deserialization is not a construction (backlog#1823)"
)]
#[derive(serde::Serialize)]
pub struct DeleteObject {
//api has
@@ -368,21 +367,6 @@ pub struct DeleteObject {
pub version_id: String,
}
pub struct DeletedObject {
//s3s has
pub key: String,
pub version_id: String,
pub deletemarker: bool,
pub deletemarker_version_id: String,
}
pub struct NonDeletedObject {
pub key: String,
pub code: String,
pub message: String,
pub version_id: String,
}
#[derive(serde::Serialize)]
pub struct DeleteMultiObjects {
pub quiet: bool,
@@ -402,6 +386,7 @@ impl DeleteMultiObjects {
Ok(buf)
}
#[allow(dead_code, reason = "MinIO-parity XML helper with no caller in this port (backlog#1823)")]
pub fn unmarshal(buf: &[u8]) -> Result<Self, std::io::Error> {
#[derive(Debug, Deserialize)]
struct WireDeleteObject {
@@ -436,8 +421,3 @@ impl DeleteMultiObjects {
})
}
}
pub struct DeleteMultiObjectsResult {
pub deleted_objects: Vec<DeletedObject>,
pub undeleted_objects: Vec<NonDeletedObject>,
}
+4
View File
@@ -365,6 +365,10 @@ mod tests {
pub struct Checksum {
checksum_type: ChecksumMode,
r: Vec<u8>,
#[allow(
dead_code,
reason = "checksum bookkeeping field kept beside the value it guards (backlog#1823)"
)]
computed: bool,
}
-3
View File
@@ -32,8 +32,5 @@ pub const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
pub const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
pub const UNSIGNED_PAYLOAD_TRAILER: &str = "STREAMING-UNSIGNED-PAYLOAD-TRAILER";
pub const TOTAL_WORKERS: i64 = 4;
pub const SIGN_V4_ALGORITHM: &str = "AWS4-HMAC-SHA256";
pub const ISO8601_DATEFORMAT: &[FormatItem<'_>] =
format_description!("[year]-[month]-[day]T[hour]:[minute]:[second].[subsecond]Z");
+12 -19
View File
@@ -67,6 +67,10 @@ impl<P: Provider + Default> Credentials<P> {
Ok(self.creds.clone())
}
#[allow(
dead_code,
reason = "MinIO-parity credential surface with no caller in this port (backlog#1823)"
)]
fn expire(&mut self) {
self.force_refresh = true;
}
@@ -133,6 +137,10 @@ impl Provider for Static {
#[derive(Debug, Clone, Default)]
pub struct STSError {
#[allow(
dead_code,
reason = "MinIO-parity STS error detail that this port never reads back (backlog#1823)"
)]
pub r#type: String,
pub code: String,
pub message: String,
@@ -141,6 +149,10 @@ pub struct STSError {
#[derive(Debug, Clone, thiserror::Error)]
pub struct ErrorResponse {
pub sts_error: STSError,
#[allow(
dead_code,
reason = "MinIO-parity STS error detail that this port never reads back (backlog#1823)"
)]
pub request_id: String,
}
@@ -158,22 +170,3 @@ impl ErrorResponse {
return self.sts_error.message.clone();
}
}
pub fn xml_decoder<T>(body: &[u8]) -> Result<T, Error>
where
for<'de> T: Deserialize<'de>,
{
match std::str::from_utf8(body) {
Ok(xml_body) => quick_xml::de::from_str::<T>(xml_body).map_err(|err| Error::new(ErrorKind::InvalidData, err.to_string())),
Err(err) => Err(Error::new(ErrorKind::InvalidData, err.to_string())),
}
}
pub fn xml_decode_and_body<T>(body_reader: &[u8]) -> Result<(Vec<u8>, T), std::io::Error>
where
for<'de> T: Deserialize<'de>,
{
let body = body_reader.to_vec();
let parsed = xml_decoder(&body)?;
Ok((body, parsed))
}
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: S3 client compatibility models are kept while ECStore callers move to narrower facades.
#![allow(dead_code)]
pub mod admin_handler_utils;
pub mod api_error_response;
@@ -77,39 +77,6 @@ fn part_number_to_rangespec(oi: ObjectInfo, part_number: usize) -> Option<HTTPRa
})
}
fn get_compressed_offsets(oi: ObjectInfo, offset: i64) -> (i64, i64, i64, i64, u64) {
let mut skip_length: i64 = 0;
let mut cumulative_actual_size: i64 = 0;
let mut first_part_idx: i64 = 0;
let mut compressed_offset: i64 = 0;
let mut part_skip: i64 = 0;
let mut decrypt_skip: i64 = 0;
let mut seq_num: u64 = 0;
for (i, part) in oi.parts.iter().enumerate() {
cumulative_actual_size += part.actual_size as i64;
if cumulative_actual_size <= offset {
compressed_offset += part.size as i64;
} else {
first_part_idx = i as i64;
skip_length = cumulative_actual_size - part.actual_size as i64;
break;
}
}
skip_length = offset - skip_length;
let parts: &[ObjectPartInfo] = &oi.parts;
if skip_length > 0
&& parts.len() > first_part_idx as usize
&& parts[first_part_idx as usize].index.as_ref().is_some_and(|idx| idx.len() > 0)
{
let _ = part_skip;
let _ = decrypt_skip;
let _ = seq_num;
}
(compressed_offset, part_skip, first_part_idx, decrypt_skip, seq_num)
}
pub fn new_getobjectreader<'a>(
rs: &Option<HTTPRangeSpec>,
oi: &'a ObjectInfo,
@@ -23,6 +23,7 @@ const X_OBS_VERSION_ID: &str = "x-obs-version-id";
const MAX_REMOTE_VERSION_ID_LEN: usize = 1024;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
#[allow(dead_code, reason = "bucket versioning states kept as a complete vocabulary (backlog#1823)")]
pub(crate) enum BucketVersioningState {
Unknown,
Disabled,
@@ -47,6 +48,7 @@ impl RemoteVersion {
}
}
#[allow(dead_code, reason = "MinIO-parity accessor with no caller in this port (backlog#1823)")]
pub(crate) fn exact_request_id(&self) -> Result<Option<&str>, Error> {
match self {
Self::Unknown => Err(Error::new(
@@ -101,6 +101,10 @@ where
const C_UNKNOWN: i32 = -1;
const C_OFFLINE: i32 = 0;
#[allow(
dead_code,
reason = "reachable only from the unused transition client methods below (backlog#1823)"
)]
const C_ONLINE: i32 = 1;
fn invalid_utf8_header_error(scope: &str, header_name: &str) -> std::io::Error {
@@ -320,6 +324,10 @@ impl TransitionClient {
Ok(client)
}
#[allow(
dead_code,
reason = "MinIO-parity transition client surface with no caller in this port (backlog#1823)"
)]
fn endpoint_url(&self) -> Url {
self.endpoint_url.clone()
}
@@ -348,12 +356,20 @@ impl TransitionClient {
.to_string())
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn trace_errors_only_off(&self) {
if let Ok(mut trace_errors_only) = self.trace_errors_only.lock() {
*trace_errors_only = false;
}
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn trace_off(&self) {
if let Ok(mut is_trace_enabled) = self.is_trace_enabled.lock() {
*is_trace_enabled = false;
@@ -363,12 +379,20 @@ impl TransitionClient {
}
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn set_s3_transfer_accelerate(&self, accelerate_endpoint: &str) {
if let Ok(mut endpoint) = self.s3_accelerate_endpoint.lock() {
*endpoint = accelerate_endpoint.to_string();
}
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn set_s3_enable_dual_stack(&self, enabled: bool) {
if let Ok(mut dual_stack) = self.s3_dual_stack_enabled.lock() {
*dual_stack = enabled;
@@ -398,10 +422,18 @@ impl TransitionClient {
(hash_algos, hash_sums)
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn is_online(&self) -> bool {
!self.is_offline()
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn mark_offline(&self) {
self.health_status
.compare_exchange(C_ONLINE, C_OFFLINE, Ordering::SeqCst, Ordering::SeqCst);
@@ -411,10 +443,18 @@ impl TransitionClient {
self.health_status.load(Ordering::SeqCst) == C_OFFLINE
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn health_check(hc_duration: Duration) {
let _ = hc_duration;
}
#[allow(
dead_code,
reason = "MinIO-parity transition client method with no caller in this port (backlog#1823)"
)]
fn dump_http(&self, req: &Request<s3s::Body>, resp: &Response<Incoming>) -> Result<(), std::io::Error> {
let mut resp_trace: Vec<u8>;
@@ -1102,6 +1142,7 @@ impl Default for ObjectInfo {
}
impl ObjectInfo {
#[allow(dead_code, reason = "MinIO-parity accessor with no caller in this port (backlog#1823)")]
pub(crate) fn remote_version(
&self,
capabilities: ProviderVersionCapabilities,
-4
View File
@@ -48,10 +48,6 @@ lazy_static! {
};
}
pub fn is_standard_query_value(qs_key: &str) -> bool {
SUPPORTED_QUERY_VALUES[qs_key]
}
pub fn is_storageclass_header(header_key: &str) -> bool {
header_key.to_lowercase() == X_AMZ_STORAGE_CLASS.as_str().to_lowercase()
}
+37
View File
@@ -190,6 +190,17 @@ pub(crate) const GET_METADATA_CACHE_REASON_VERSION_SUSPENDED: &str = "version_su
pub(crate) const GET_METADATA_CACHE_REASON_VERSIONED: &str = "versioned";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA: &str = "conflicting_metadata";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER: &str = "delete_marker";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY: &str = "data_read_inline_body_verify";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED: &str = "data_read_inline_deleted";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY: &str = "data_read_inline_geometry";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH: &str = "data_read_inline_identity_mismatch";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD: &str = "data_read_inline_missing_payload";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD: &str = "data_read_inline_missing_shard";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE: &str = "data_read_inline_not_inline";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE: &str = "data_read_inline_part_shape";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE: &str = "data_read_inline_remote";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE: &str = "data_read_inline_size";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED: &str = "data_read_inline_transformed";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_ERROR: &str = "error";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM: &str = "insufficient_quorum";
pub(crate) const GET_METADATA_EARLY_STOP_REASON_NOT_FOUND: &str = "not_found";
@@ -551,6 +562,32 @@ mod tests {
assert_eq!(GET_METADATA_CACHE_REASON_VERSIONED, "versioned");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, "conflicting_metadata");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER, "delete_marker");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY,
"data_read_inline_body_verify"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED, "data_read_inline_deleted");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY, "data_read_inline_geometry");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
"data_read_inline_identity_mismatch"
);
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
"data_read_inline_missing_payload"
);
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD,
"data_read_inline_missing_shard"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE, "data_read_inline_not_inline");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE, "data_read_inline_part_shape");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE, "data_read_inline_remote");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE, "data_read_inline_size");
assert_eq!(
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED,
"data_read_inline_transformed"
);
assert_eq!(GET_METADATA_EARLY_STOP_REASON_ERROR, "error");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM, "insufficient_quorum");
assert_eq!(GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, "not_found");
+6
View File
@@ -277,6 +277,12 @@ pub struct ObjectOptions {
/// fence avoids recursively acquiring the read lock behind a queued writer.
pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
pub replication_request: bool,
/// Source-cluster LWW timestamps carried by an authorized replication
/// request; None when the source never modified the category. Only the
/// replication-authorized options builders may set these.
pub replication_tagging_timestamp: Option<OffsetDateTime>,
pub replication_retention_timestamp: Option<OffsetDateTime>,
pub replication_legalhold_timestamp: Option<OffsetDateTime>,
/// Authorized SSE-C replication passthrough: the body is already
/// ciphertext, so the write path must not encrypt or compress it and
/// stores the restored encryption metadata verbatim. Only the
+423 -50
View File
@@ -32,15 +32,22 @@ use crate::diagnostics::get::{
GET_METADATA_CACHE_REASON_NOT_READ_DATA, GET_METADATA_CACHE_REASON_PART_NUMBER,
GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, GET_METADATA_CACHE_REASON_USABLE, GET_METADATA_CACHE_REASON_VERSION_ID,
GET_METADATA_CACHE_REASON_VERSION_SUSPENDED, GET_METADATA_CACHE_REASON_VERSIONED,
GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER,
GET_METADATA_EARLY_STOP_REASON_ERROR, GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM,
GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST,
GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM,
GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND, GET_METADATA_RESPONSE_CORRUPT, GET_METADATA_RESPONSE_DISK_NOT_FOUND,
GET_METADATA_RESPONSE_ERROR, GET_METADATA_RESPONSE_IGNORED, GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT,
GET_METADATA_RESPONSE_VALID, GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING,
GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_SET_DISK,
GET_STAGE_DECODE, GET_STAGE_METADATA_CACHE_LOOKUP, GET_STAGE_METADATA_RESOLVE, GET_STAGE_RANGE, GET_STAGE_READER_SETUP,
GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED,
GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER, GET_METADATA_EARLY_STOP_REASON_ERROR,
GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM, GET_METADATA_EARLY_STOP_REASON_NOT_FOUND,
GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST, GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM,
GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND,
GET_METADATA_RESPONSE_CORRUPT, GET_METADATA_RESPONSE_DISK_NOT_FOUND, GET_METADATA_RESPONSE_ERROR,
GET_METADATA_RESPONSE_IGNORED, GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT, GET_METADATA_RESPONSE_VALID,
GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING, GET_OBJECT_PATH_DIRECT_MEMORY,
GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE,
GET_STAGE_METADATA_CACHE_LOOKUP, GET_STAGE_METADATA_RESOLVE, GET_STAGE_RANGE, GET_STAGE_READER_SETUP,
GET_STAGE_READER_SETUP_DROP_PENDING, GET_STAGE_READER_SETUP_SCHEDULE, GET_STAGE_READER_SETUP_WAIT_QUORUM,
GET_STAGE_READER_TASK_BITROT_READER_INIT, GET_STAGE_READER_TASK_FILE_OPEN, GET_STAGE_READER_TASK_READER_CONSTRUCTION,
GetObjectFailureReason, classify_disk_error, get_stage_timer_if_enabled, record_get_object_pipeline_failure,
@@ -652,36 +659,49 @@ pub(in crate::set_disk) fn metadata_early_stop_candidate_matches(left: &FileInfo
&& left.erasure.distribution == right.erasure.distribution
}
pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
pub(in crate::set_disk) async fn data_read_early_stop_inline_body_miss_reason(
bucket: &str,
object: &str,
candidate: &FileInfo,
parts_metadata: &[FileInfo],
disks: &[Option<DiskStore>],
) -> bool {
if !candidate.inline_data()
|| candidate.is_compressed()
) -> Option<&'static str> {
// `inline_data` excludes remote objects; this diagnostic reports them separately.
if !rustfs_utils::http::contains_key_str(&candidate.metadata, rustfs_utils::http::SUFFIX_INLINE_DATA) {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE);
}
if candidate.is_compressed()
|| candidate
.metadata
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
|| candidate.is_remote()
|| candidate.deleted
|| candidate.size <= 0
|| candidate.parts.len() != 1
|| !candidate.has_valid_erasure_geometry()
{
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED);
}
if candidate.is_remote() {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE);
}
if candidate.deleted {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED);
}
if candidate.size <= 0 {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
}
if candidate.parts.len() != 1 {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
}
if !candidate.has_valid_erasure_geometry() {
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
}
let Ok(object_size) = usize::try_from(candidate.size) else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
};
if candidate.parts.first().is_none_or(|part| part.size != object_size) {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
}
if !can_try_inline_data_shards_direct(object_size, candidate.erasure.block_size) {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE);
}
let Ok(erasure) = coding::Erasure::try_new_with_options(
@@ -690,18 +710,18 @@ pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
candidate.erasure.block_size,
candidate.uses_legacy_checksum,
) else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
};
let Some(data_files) =
collect_inline_data_shard_fileinfos_by_index(parts_metadata, candidate, erasure.data_shards, |index| {
let data_files =
match collect_inline_data_shard_fileinfos_by_index_or_reason(parts_metadata, candidate, erasure.data_shards, |index| {
disks.get(index).is_some_and(Option::is_some)
})
else {
return false;
};
}) {
Ok(data_files) => data_files,
Err(reason) => return Some(reason),
};
let Some(part) = candidate.parts.first() else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE);
};
let checksum_info = candidate.erasure.get_checksum_info(part.number);
let checksum_algo = if candidate.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
@@ -721,12 +741,70 @@ pub(in crate::set_disk) async fn data_read_early_stop_inline_body_verified(
let Ok(mut readers) =
build_inline_bitrot_readers_from_refs(&data_files, bucket, object, read_length, shard_size, &checksum_algo, false).await
else {
return false;
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY);
};
try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size)
.await
.is_some_and(|body| body.len() == object_size)
match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await {
Some(body) if body.len() == object_size => None,
_ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
}
}
fn data_read_inline_missing_shards_are_pending(
candidate: &FileInfo,
parts_metadata: &[FileInfo],
errors: &[Option<DiskError>],
disks: &[Option<DiskStore>],
fanout_order: &[usize],
scheduled_fanout_len: usize,
) -> bool {
let Ok(erasure) = coding::Erasure::try_new_with_options(
candidate.erasure.data_blocks,
candidate.erasure.parity_blocks,
candidate.erasure.block_size,
candidate.uses_legacy_checksum,
) else {
return false;
};
let distribution = &candidate.erasure.distribution;
let mut data_shards_seen_or_pending = vec![false; erasure.data_shards];
let mut missing_pending_data_shards = 0usize;
for (disk_index, file_info) in parts_metadata.iter().enumerate() {
let Some(&block_index) = distribution.get(disk_index) else {
return false;
};
if block_index == 0 || block_index > erasure.data_shards {
continue;
}
if !disks.get(disk_index).is_some_and(Option::is_some) {
return false;
}
let data_slot = block_index - 1;
if file_info.name.is_empty() {
let scheduled_and_not_failed = fanout_order
.get(..scheduled_fanout_len)
.is_some_and(|scheduled_disks| scheduled_disks.contains(&disk_index))
&& errors.get(disk_index).is_some_and(Option::is_none);
if scheduled_and_not_failed {
data_shards_seen_or_pending[data_slot] = true;
missing_pending_data_shards = missing_pending_data_shards.saturating_add(1);
continue;
}
return false;
}
if file_info.erasure.index != block_index
|| !file_info.has_valid_erasure_geometry()
|| !metadata_early_stop_candidate_matches(file_info, candidate)
|| file_info.data.as_ref().is_none_or(|data| data.is_empty())
{
return false;
}
data_shards_seen_or_pending[data_slot] = true;
}
missing_pending_data_shards > 0 && data_shards_seen_or_pending.into_iter().all(|seen_or_pending| seen_or_pending)
}
pub(in crate::set_disk) fn classify_metadata_response_error(err: &DiskError) -> &'static str {
@@ -2469,6 +2547,7 @@ impl SetDisks {
let mut next_fanout_index = 0usize;
let mut scheduled_count = 0usize;
let mut force_full_wait = false;
let mut final_miss_reason_override = None;
let spawn_read_version =
|join_set: &mut JoinSet<(usize, disk::error::Result<FileInfo>, Duration)>, index: usize, disk: Option<DiskStore>| {
let task_opts = opts;
@@ -2511,6 +2590,7 @@ impl SetDisks {
}
while let Some(result) = join_set.join_next().await {
let mut defer_pending_inline_data_shard = false;
match result {
Ok((index, res, elapsed)) => match res {
Ok(file_info) => {
@@ -2541,17 +2621,43 @@ impl SetDisks {
.or_else(|| accumulator.version_early_stop_decision())
{
let should_return_early = if read_data {
let allow_data_read_early_stop = match accumulator.candidate.as_ref() {
Some(candidate) => {
data_read_early_stop_inline_body_verified(bucket.as_ref(), object.as_ref(), candidate, &ress, disks)
.await
match accumulator.candidate.as_ref() {
Some(candidate) => match data_read_early_stop_inline_body_miss_reason(
bucket.as_ref(),
object.as_ref(),
candidate,
&ress,
disks,
)
.await
{
None => true,
Some(reason) => {
final_miss_reason_override = Some(reason);
if bounded_fanout
&& reason == GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD
&& data_read_inline_missing_shards_are_pending(
candidate,
&ress,
&errors,
disks,
&fanout_order,
next_fanout_index,
)
{
defer_pending_inline_data_shard = true;
} else {
force_full_wait = true;
}
false
}
},
None => {
force_full_wait = true;
final_miss_reason_override = Some(GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM);
false
}
None => false,
};
if !allow_data_read_early_stop {
force_full_wait = true;
}
allow_data_read_early_stop
} else {
true
};
@@ -2588,6 +2694,7 @@ impl SetDisks {
let pending_responses = join_set.len();
let should_hedge_single_pending_data_read = read_data
&& !force_full_wait
&& !defer_pending_inline_data_shard
&& pending_responses == 1
&& accumulator.can_still_reach_early_stop_with_pending(pending_responses);
if bounded_fanout && force_full_wait {
@@ -2600,6 +2707,7 @@ impl SetDisks {
next_fanout_index = next_fanout_index.saturating_add(1);
}
} else if bounded_fanout
&& !defer_pending_inline_data_shard
&& next_fanout_index < disks.len()
&& (!accumulator.can_still_reach_early_stop_with_pending(pending_responses)
|| should_hedge_single_pending_data_read)
@@ -2613,7 +2721,12 @@ impl SetDisks {
}
}
rustfs_io_metrics::record_get_object_metadata_early_stop_miss(metrics_path, accumulator.final_miss_reason());
let accumulator_miss_reason = accumulator.final_miss_reason();
let final_miss_reason = match (final_miss_reason_override, accumulator_miss_reason) {
(Some(reason), GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM) => reason,
_ => accumulator_miss_reason,
};
rustfs_io_metrics::record_get_object_metadata_early_stop_miss(metrics_path, final_miss_reason);
rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0);
rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0);
let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
@@ -5752,9 +5865,20 @@ mod tests {
object: &str,
payload: &[u8],
uses_legacy_checksum: bool,
) -> Vec<FileInfo> {
inline_metadata_fanout_fileinfos_with_geometry(bucket, object, payload, uses_legacy_checksum, 2, 2).await
}
async fn inline_metadata_fanout_fileinfos_with_geometry(
bucket: &str,
object: &str,
payload: &[u8],
uses_legacy_checksum: bool,
data_shards: usize,
parity_shards: usize,
) -> Vec<FileInfo> {
let distribution_key = metadata_distribution_key(bucket, object);
let mut base = FileInfo::new(&distribution_key, 2, 2);
let mut base = FileInfo::new(&distribution_key, data_shards, parity_shards);
base.volume = bucket.to_string();
base.name = object.to_string();
base.size = i64::try_from(payload.len()).expect("test payload should fit i64");
@@ -5817,6 +5941,21 @@ mod tests {
install_inline_metadata_fanout_files(disks, bucket, object, files).await;
}
async fn install_inline_metadata_fanout_fileinfo_with_geometry(
disks: &[Option<DiskStore>],
bucket: &str,
object: &str,
payload: &[u8],
data_shards: usize,
parity_shards: usize,
mutate: impl FnOnce(&mut [FileInfo]),
) {
let mut files =
inline_metadata_fanout_fileinfos_with_geometry(bucket, object, payload, false, data_shards, parity_shards).await;
mutate(&mut files);
install_inline_metadata_fanout_files(disks, bucket, object, files).await;
}
async fn install_inline_metadata_fanout_files(disks: &[Option<DiskStore>], bucket: &str, object: &str, files: Vec<FileInfo>) {
let distribution = files
.first()
@@ -6037,6 +6176,118 @@ mod tests {
drop(dirs);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn bounded_metadata_early_stop_waits_for_pending_inline_data_shard() {
const DISKS: usize = 6;
const DATA_SHARDS: usize = 4;
const PARITY_SHARDS: usize = 2;
let bucket = "bounded-inline-data-get-pending-shard-bucket";
let object =
object_with_initial_data_shards(bucket, "bounded-inline-data-get-pending-shard-object", DATA_SHARDS, DATA_SHARDS);
let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
install_inline_metadata_fanout_fileinfo_with_geometry(
&disks,
bucket,
&object,
b"verified inline payload",
DATA_SHARDS,
PARITY_SHARDS,
|_| {},
)
.await;
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
],
async {
let fanout_order = bounded_metadata_fanout_order(bucket, &object, DISKS, PARITY_SHARDS);
let distribution_key = metadata_distribution_key(bucket, &object);
let distribution = FileInfo::new(&distribution_key, DATA_SHARDS, PARITY_SHARDS)
.erasure
.distribution;
let paused_data_disk = *fanout_order
.iter()
.take(DATA_SHARDS)
.find(|disk_index| {
distribution
.get(**disk_index)
.is_some_and(|block_index| (1..=DATA_SHARDS).contains(block_index))
})
.expect("initial fanout should include a data shard to pause");
let hedged_parity_disk = fanout_order[DATA_SHARDS];
let unscheduled_parity_disk = fanout_order[DATA_SHARDS + 1];
let barrier = rename_fanout_barrier::arm(&object, paused_data_disk, rename_fanout_barrier::PHASE_READ_VERSION);
let tracker = rename_fanout_barrier::observe_tasks(&object);
let calls = disk_call_counters::observe(&object);
let disks_for_read = disks.clone();
let object_for_read = object.clone();
let mut read = tokio::spawn(async move {
SetDisks::read_all_fileinfo_observed(
&disks_for_read,
bucket,
bucket,
&object_for_read,
"",
true,
false,
false,
true,
PARITY_SHARDS,
)
.await
});
tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
.await
.expect("initial data shard should pause before returning");
tokio::time::timeout(BARRIER_PAUSE_GUARD, async {
while calls.for_disk(disk_call_counters::KIND_READ_VERSION, hedged_parity_disk) == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("bounded fanout should hedge one parity disk while the data shard is pending");
assert!(
tokio::time::timeout(BARRIER_PAUSE_GUARD, &mut read).await.is_err(),
"inline data-read early-stop must wait for a scheduled missing data shard instead of forcing full wait"
);
barrier.release();
let (parts_metadata, errs, diagnostics) = read
.await
.expect("metadata read task should not panic")
.expect("pending data shard should let the inline verifier finish");
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
5,
"pending data-shard defer should not schedule the final parity disk"
);
assert_eq!(
calls.for_disk(disk_call_counters::KIND_READ_VERSION, unscheduled_parity_disk),
0,
"the remaining parity disk must stay unissued when pending data verification succeeds"
);
assert_eq!(
tracker.running(),
0,
"early-stop should drain spawned read_version tasks before returning"
);
assert_eq!(diagnostics.total_responses(), 5);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == object).count(), 5);
assert!(errs.iter().all(Option::is_none));
},
)
.await;
drop(dirs);
}
#[tokio::test]
async fn data_read_early_stop_verifies_legacy_inline_checksum_payload() {
let bucket = "legacy-inline-data-get-fanout-bucket";
@@ -6067,11 +6318,133 @@ mod tests {
.clone();
assert!(
data_read_early_stop_inline_body_verified(bucket, object, &candidate, &parts_metadata, &disks).await,
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &parts_metadata, &disks)
.await
.is_none(),
"legacy inline metadata must use the legacy bitrot shard sizing and checksum algorithm"
);
}
#[tokio::test]
async fn data_read_early_stop_reports_inline_miss_reasons() {
let bucket = "inline-data-get-miss-reason-bucket";
let object = "inline-data-get-miss-reason-object";
let payload = b"verified inline payload";
let (_dirs, disks) = call_counter_local_disks(bucket, 4).await;
let files = inline_metadata_fanout_fileinfos_with_mode(bucket, object, payload, false).await;
let distribution = files
.first()
.map(|file| file.erasure.distribution.clone())
.expect("fixture should include metadata");
let order = bounded_metadata_fanout_order(bucket, object, 4, 2);
let mut parts_metadata = vec![FileInfo::default(); 4];
for disk_index in order.into_iter().take(3) {
let block_index = distribution
.get(disk_index)
.copied()
.expect("fixture distribution should cover every disk");
parts_metadata[disk_index] = files
.get(block_index.checked_sub(1).expect("erasure block indexes are one-based"))
.expect("fixture should include every distributed shard")
.clone();
}
let candidate = parts_metadata
.iter()
.find(|file| file.name == object)
.expect("fixture should include observed metadata")
.clone();
let data_disk = distribution
.iter()
.position(|block_index| *block_index == 1)
.expect("fixture distribution should include first data shard");
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &parts_metadata, &disks).await,
None
);
let mut not_inline = candidate.clone();
rustfs_utils::http::remove_str(&mut not_inline.metadata, rustfs_utils::http::SUFFIX_INLINE_DATA);
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &not_inline, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_NOT_INLINE)
);
let mut remote = candidate.clone();
remote.transition_status = TRANSITION_COMPLETE.to_string();
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &remote, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_REMOTE)
);
let mut transformed = candidate.clone();
rustfs_utils::http::insert_str(&mut transformed.metadata, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &transformed, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_TRANSFORMED)
);
let mut deleted = candidate.clone();
deleted.deleted = true;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &deleted, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_DELETED)
);
let mut zero_size = candidate.clone();
zero_size.size = 0;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &zero_size, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_SIZE)
);
let mut multipart = candidate.clone();
multipart.parts.push(multipart.parts[0].clone());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &multipart, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_PART_SHAPE)
);
let mut invalid_geometry = candidate.clone();
invalid_geometry.erasure.data_blocks = 0;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &invalid_geometry, &parts_metadata, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY)
);
let mut missing_shard = parts_metadata.clone();
missing_shard[data_disk] = FileInfo::default();
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &missing_shard, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD)
);
let mut missing_payload = parts_metadata.clone();
missing_payload[data_disk].data = None;
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &missing_payload, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD)
);
let mut identity_mismatch = parts_metadata.clone();
identity_mismatch[data_disk].version_id = Some(Uuid::new_v4());
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &identity_mismatch, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH)
);
let mut corrupt = parts_metadata.clone();
if let Some(data) = corrupt[data_disk].data.as_mut() {
let mut corrupt_data = data.to_vec();
corrupt_data[0] ^= 0x01;
*data = Bytes::from(corrupt_data);
}
assert_eq!(
data_read_early_stop_inline_body_miss_reason(bucket, object, &candidate, &corrupt, &disks).await,
Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY)
);
}
#[test]
#[serial_test::serial]
fn metadata_fanout_lifecycle_records_real_early_stop_abort() {
@@ -6161,7 +6534,7 @@ mod tests {
&[
("path", GET_OBJECT_PATH_INTERNAL_META),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM),
("reason", GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
],
),
1,
@@ -6173,7 +6546,7 @@ mod tests {
&[
("path", GET_OBJECT_PATH_LEGACY_DUPLEX),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM),
("reason", GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
],
),
0,
@@ -6768,7 +7141,7 @@ mod tests {
}
#[tokio::test]
async fn bounded_metadata_early_stop_defaults_keep_data_get_full_fanout() {
async fn bounded_metadata_early_stop_defaults_keep_non_inline_data_get_full_fanout() {
const DISKS: usize = 4;
let bucket = "bounded-data-get-default-bucket";
let object = "bounded-data-get-default-object";
@@ -6791,7 +7164,7 @@ mod tests {
assert_eq!(
calls.total(disk_call_counters::KIND_READ_VERSION),
DISKS as u64,
"default GET data-read metadata must keep full fanout for read-failure tolerance"
"default non-inline GET data-read metadata must keep full fanout for read-failure tolerance"
);
assert_eq!(diagnostics.total_responses(), DISKS);
assert_eq!(parts_metadata.iter().filter(|fi| fi.name == object).count(), DISKS);
+47 -32
View File
@@ -59,7 +59,10 @@ use crate::client::{object_api_utils::get_raw_etag, transition_api::ReaderImpl};
use crate::cluster::rpc::heal_bucket_local_on_disks;
use crate::data_usage::record_compression_total_memory;
use crate::diagnostics::get::{
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING,
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD,
GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING,
GET_OBJECT_PATH_CODEC_STREAMING_LEGACY_ENGINE, GET_OBJECT_PATH_CODEC_STREAMING_RUSTFS_ENGINE, GET_OBJECT_PATH_DIRECT_MEMORY,
GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX,
GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT, GET_STAGE_INLINE_PREPARE,
@@ -689,8 +692,9 @@ const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: usize = 128 * 102
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ENABLE";
// Enabled by default (backlog#872): the early-stop path only engages for
// requests `should_allow_metadata_early_stop` classifies as safe (latest-version
// metadata-only reads by default, without version_id / healing / free-version
// needs) and still requires a full read-quorum agreement before stopping. Set
// reads by default, without version_id / healing / free-version needs) and still
// requires a full read-quorum agreement before stopping. Data-read requests add
// a separate inline-shard verifier before cancelling the remaining fanout. Set
// the env var to `false` to fall back to full-wait metadata fanout.
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: bool = true;
@@ -701,7 +705,7 @@ const ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_META
const DEFAULT_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: bool = false;
const ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE";
const DEFAULT_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE: bool = false;
const DEFAULT_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE: bool = true;
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT";
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = false;
@@ -912,12 +916,6 @@ mod prepared_get_object_metadata_tests {
.expect("4-disk test geometry should leave one bounded spare disk")
}
fn bounded_slow_initial_disk_index(bucket: &str, object: &str) -> usize {
*bounded_metadata_fanout_order(bucket, object, 4, 2)
.get(2)
.expect("4-disk test geometry should include a third initial metadata disk")
}
#[tokio::test]
async fn prepared_metadata_is_consumed_exactly_once() {
let snapshot = GetObjectFileInfo::owned(FileInfo::default(), Vec::new(), Vec::new());
@@ -1036,7 +1034,7 @@ mod prepared_get_object_metadata_tests {
#[test]
#[serial_test::serial(body_cache_hook)]
fn inline_data_read_early_stop_reader_returns_exact_body() {
fn inline_data_read_early_stop_defaults_return_exact_body() {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
@@ -1068,14 +1066,14 @@ mod prepared_get_object_metadata_tests {
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", None::<&str>),
],
async {
let slow_initial_disk = bounded_slow_initial_disk_index(bucket, &object);
let slow_parity_disk = bounded_spare_disk_index(bucket, &object);
let barrier =
rename_fanout_barrier::arm(&object, slow_initial_disk, rename_fanout_barrier::PHASE_READ_VERSION);
rename_fanout_barrier::arm(&object, slow_parity_disk, rename_fanout_barrier::PHASE_READ_VERSION);
let calls = disk_call_counters::observe(&object);
let set_disks_for_read = Arc::clone(&set_disks);
let opts_for_read = opts.clone();
@@ -1088,10 +1086,10 @@ mod prepared_get_object_metadata_tests {
tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
.await
.expect("bounded inline GET should pause a slow initial metadata read");
.expect("default inline GET should pause a slow parity metadata read");
let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut open_reader)
.await
.expect("production inline GET should return before the paused metadata response")
.expect("default production inline GET should return before the paused parity metadata response")
.expect("inline GET reader task should not panic")
.expect("inline GET reader should open");
let object_size = reader.object_info.size;
@@ -1112,14 +1110,14 @@ mod prepared_get_object_metadata_tests {
assert_eq!(object_size, payload.len() as i64);
assert_eq!(restored, payload);
assert_eq!(calls_total, 4, "bounded production GET should schedule the initial quorum plus one spare");
assert_eq!(calls_total, 4, "default production GET should eagerly schedule the full metadata fanout");
assert_eq!(
recorder.histogram_values(
"rustfs_io_get_object_metadata_fanout_scheduled",
&[("path", GET_OBJECT_PATH_LEGACY_DUPLEX)]
),
vec![4.0],
"bounded production GET should record all scheduled metadata tasks"
"default production GET should record all scheduled metadata tasks"
);
assert_eq!(
recorder.histogram_values(
@@ -1127,7 +1125,7 @@ mod prepared_get_object_metadata_tests {
&[("path", GET_OBJECT_PATH_LEGACY_DUPLEX)]
),
vec![3.0],
"bounded production GET should record only observed metadata responses as completed"
"default production GET should record only observed metadata responses as completed"
);
assert_eq!(
recorder.histogram_values(
@@ -1135,7 +1133,7 @@ mod prepared_get_object_metadata_tests {
&[("path", GET_OBJECT_PATH_LEGACY_DUPLEX)]
),
vec![1.0],
"bounded production GET should record the aborted slow metadata task"
"default production GET should record the aborted slow parity metadata task"
);
}
@@ -1282,9 +1280,9 @@ mod prepared_get_object_metadata_tests {
temp_env::async_with_vars(
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", None::<&str>),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", None::<&str>),
],
async {
let calls = disk_call_counters::observe(&object);
@@ -3866,8 +3864,17 @@ fn collect_inline_data_shard_fileinfos_by_index<'a>(
parts_metadata: &'a [FileInfo],
fi: &FileInfo,
data_shards: usize,
mut disk_is_online: impl FnMut(usize) -> bool,
disk_is_online: impl FnMut(usize) -> bool,
) -> Option<Vec<&'a FileInfo>> {
collect_inline_data_shard_fileinfos_by_index_or_reason(parts_metadata, fi, data_shards, disk_is_online).ok()
}
fn collect_inline_data_shard_fileinfos_by_index_or_reason<'a>(
parts_metadata: &'a [FileInfo],
fi: &FileInfo,
data_shards: usize,
mut disk_is_online: impl FnMut(usize) -> bool,
) -> std::result::Result<Vec<&'a FileInfo>, &'static str> {
let distribution = &fi.erasure.distribution;
let mut data_files = vec![None; data_shards];
@@ -3875,27 +3882,35 @@ fn collect_inline_data_shard_fileinfos_by_index<'a>(
if !disk_is_online(disk_index) {
continue;
}
let block_index = *distribution.get(disk_index)?;
let Some(&block_index) = distribution.get(disk_index) else {
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
};
if block_index == 0 || block_index > data_shards {
continue;
}
if file_info.name.is_empty() {
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD);
}
if file_info.erasure.index != block_index {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH);
}
if !file_info.has_valid_erasure_geometry() {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_GEOMETRY);
}
if !core::io_primitives::metadata_early_stop_candidate_matches(file_info, fi) {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH);
}
if file_info.data.as_ref().is_none_or(|data| data.is_empty()) {
continue;
return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_PAYLOAD);
}
data_files[block_index - 1] = Some(file_info);
}
data_files.into_iter().collect()
data_files
.into_iter()
.collect::<Option<Vec<_>>>()
.ok_or(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD)
}
impl SetDisks {
+11 -11
View File
@@ -7505,7 +7505,7 @@ mod get_object_downstream_close_accounting_tests {
use super::hermetic_set_disks_support::hermetic_set_disks;
use super::*;
use crate::diagnostics::get::{
GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST, GET_OBJECT_PATH_INTERNAL_META, GET_STAGE_DECODE, GET_STAGE_EMIT,
GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, GET_OBJECT_PATH_INTERNAL_META, GET_STAGE_DECODE, GET_STAGE_EMIT,
GetObjectFailureReason,
};
use crate::disk::RUSTFS_META_BUCKET;
@@ -7637,8 +7637,8 @@ mod get_object_downstream_close_accounting_tests {
legacy_completed,
internal_cancelled,
legacy_cancelled,
internal_unsafe_miss,
legacy_unsafe_miss,
internal_not_found_miss,
legacy_not_found_miss,
internal_saved,
legacy_saved,
) = metrics::with_local_recorder(&recorder, || {
@@ -7714,7 +7714,7 @@ mod get_object_downstream_close_accounting_tests {
&[
("path", GET_OBJECT_PATH_INTERNAL_META),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST),
("reason", GET_METADATA_EARLY_STOP_REASON_NOT_FOUND),
],
),
recorder.counter_value(
@@ -7722,7 +7722,7 @@ mod get_object_downstream_close_accounting_tests {
&[
("path", GET_OBJECT_PATH_LEGACY_DUPLEX),
("decision", "miss"),
("reason", GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST),
("reason", GET_METADATA_EARLY_STOP_REASON_NOT_FOUND),
],
),
recorder.histogram_values(
@@ -7773,21 +7773,21 @@ mod get_object_downstream_close_accounting_tests {
"internal metadata lifecycle cancelled count must not leak into legacy_duplex"
);
assert_eq!(
internal_unsafe_miss, 1,
"internal metadata unsafe early-stop miss must retain its path label"
internal_not_found_miss, 1,
"internal metadata not-found early-stop miss must retain its path label"
);
assert_eq!(
legacy_unsafe_miss, 0,
"internal metadata unsafe early-stop miss must not leak into legacy_duplex"
legacy_not_found_miss, 0,
"internal metadata not-found early-stop miss must not leak into legacy_duplex"
);
assert_eq!(
internal_saved,
vec![0.0],
"internal metadata unsafe miss must record zero saved responses on internal_meta"
"internal metadata not-found miss must record zero saved responses on internal_meta"
);
assert!(
legacy_saved.is_empty(),
"internal metadata unsafe miss saved responses must not leak into legacy_duplex"
"internal metadata not-found miss saved responses must not leak into legacy_duplex"
);
}
}
+33 -3
View File
@@ -3886,13 +3886,15 @@ mod tests {
assert!(metadata_early_stop_permitted(true, true, false, "", false, false));
// observe=false (non-observed fanout) also disables early-stop.
assert!(!metadata_early_stop_permitted(true, false, false, "", false, false));
assert!(!metadata_early_stop_permitted(true, true, true, "", false, false));
// Whole/latest data-read metadata is now allowed by default;
// the inline verifier still decides whether it can stop early.
assert!(metadata_early_stop_permitted(true, true, true, "", false, false));
},
);
}
#[test]
fn metadata_early_stop_keeps_data_reads_opt_in_by_default() {
fn metadata_early_stop_allows_safe_data_reads_by_default() {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
@@ -3900,7 +3902,7 @@ mod tests {
(ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, None),
],
|| {
assert!(!should_allow_metadata_early_stop(true, "", false, false));
assert!(should_allow_metadata_early_stop(true, "", false, false));
assert!(!should_allow_metadata_early_stop(true, "version-id", false, false));
assert!(should_allow_metadata_early_stop(false, "", false, false));
assert!(!should_allow_metadata_early_stop(false, "version-id", false, false));
@@ -3932,6 +3934,34 @@ mod tests {
);
}
#[test]
fn metadata_early_stop_bounded_fanout_defaults_to_disabled() {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
(ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, None),
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT, None),
],
|| {
assert!(is_get_metadata_data_read_early_stop_enabled());
assert!(!is_get_metadata_early_stop_bounded_fanout_enabled());
},
);
temp_env::with_vars([(ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT, Some("true"))], || {
assert!(is_get_metadata_early_stop_bounded_fanout_enabled());
});
temp_env::with_vars(
[
(ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, Some("false")),
(ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT, Some("false")),
],
|| {
assert!(!is_get_metadata_data_read_early_stop_enabled());
assert!(!is_get_metadata_early_stop_bounded_fanout_enabled());
},
);
}
#[test]
fn metadata_early_stop_rejects_healing_and_free_version_requests() {
temp_env::with_vars(
+1 -4
View File
@@ -135,10 +135,7 @@ impl FileMeta {
let i = buf.len() as u64;
// check version, buf = buf[8..]
let (buf, _, _) = Self::check_xl2_v1(buf).map_err(|e| {
error!("failed to check XL2 v1 format: {}", e);
e
})?;
let (buf, _, _) = Self::check_xl2_v1(buf)?;
if buf.len() < 5 {
error!(
+7 -2
View File
@@ -82,8 +82,8 @@ impl Error {
/// Whether a heal operation can be retried without changing its inputs.
pub(crate) fn is_recoverable_heal(&self) -> bool {
match self {
Error::TaskCancelled => false,
Error::TaskTimeout | Error::TransientSkip { .. } => true,
Error::TaskCancelled | Error::TaskTimeout => false,
Error::TransientSkip { .. } => true,
Error::Storage(err) => {
err.is_quorum_error()
|| matches!(
@@ -165,4 +165,9 @@ mod tests {
assert!(Error::Storage(EcstoreError::DiskNotFound).is_recoverable_heal());
assert!(Error::Storage(EcstoreError::VolumeNotFound).is_recoverable_heal());
}
#[test]
fn task_timeout_is_terminal() {
assert!(!Error::TaskTimeout.is_recoverable_heal());
}
}
+30 -20
View File
@@ -673,6 +673,12 @@ fn retry_request_for_result(task: &HealTask, result: &Result<()>) -> Option<(Hea
Some((request, delay, error))
}
async fn retry_request_for_result_with_budget(task: &HealTask, result: &Result<()>) -> Option<(HealRequest, Duration, String)> {
let (_, delay, error) = retry_request_for_result(task, result)?;
let request = task.retry_request_with_remaining_timeout().await.ok()?;
Some((request, delay, error))
}
fn recoverable_heal_retry_delay(retry_attempt: u32) -> Duration {
let retry_attempt = retry_attempt.clamp(1, 5);
let delay = Duration::from_secs(2_u64.saturating_pow(retry_attempt));
@@ -690,7 +696,7 @@ pub struct HealConfig {
pub max_concurrent_heals: usize,
/// Maximum concurrent heal tasks allowed for a single erasure set
pub max_concurrent_per_set: usize,
/// Task timeout
/// Aggregate task execution timeout across recoverable retries
pub task_timeout: Duration,
/// Queue size
pub queue_size: usize,
@@ -3106,7 +3112,7 @@ impl HealManager {
"Heal scheduler task started"
);
let result = task.execute().await;
let retry_request = retry_request_for_result(task.as_ref(), &result);
let retry_request = retry_request_for_result_with_budget(task.as_ref(), &result).await;
match &result {
Ok(_) => {
debug!(
@@ -4539,6 +4545,25 @@ mod tests {
assert!(retry_error.contains("Lock acquisition timeout"));
}
#[tokio::test]
async fn retry_request_for_result_preserves_remaining_timeout_budget() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
let mut request = HealRequest::object("retry-transition".to_string(), "object".to_string(), None);
request.options.timeout = Some(Duration::from_secs(60));
let task = HealTask::from_request(request, storage);
let result = task.execute().await;
let (retry_request, _, _) = retry_request_for_result_with_budget(&task, &result)
.await
.expect("read quorum failure should retain the unused timeout budget");
let remaining = retry_request
.options
.timeout
.expect("configured timeout should remain present");
assert!(remaining < Duration::from_secs(60));
assert!(remaining > Duration::from_secs(59));
}
#[test]
fn test_retry_request_for_incomplete_heal_rename() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -6054,7 +6079,7 @@ mod tests {
process_manager_queue_once(&manager).await;
let defaulted_status = tokio::time::timeout(Duration::from_secs(1), async {
loop {
if let Ok(status @ HealTaskStatus::Retrying { .. }) = manager.get_task_status(&defaulted_id).await {
if let Ok(status @ HealTaskStatus::Timeout) = manager.get_task_status(&defaulted_id).await {
break status;
}
tokio::task::yield_now().await;
@@ -6062,23 +6087,8 @@ mod tests {
})
.await
.expect("configured timeout should finish the task");
assert!(matches!(defaulted_status, HealTaskStatus::Retrying { .. }));
assert_eq!(
manager
.retrying_heals
.lock()
.await
.get(&defaulted_id)
.expect("timed out task should retain its retry request")
.request
.options
.timeout,
Some(Duration::ZERO)
);
manager
.cancel_task(&defaulted_id)
.await
.expect("retrying timeout task should be cancelled");
assert_eq!(defaulted_status, HealTaskStatus::Timeout);
assert!(manager.retrying_heals.lock().await.get(&defaulted_id).is_none());
let mut explicit = bucket_request("explicit-timeout", HealPriority::Normal, HealRequestSource::Admin);
explicit.options.timeout = Some(Duration::from_secs(60));
+39 -1
View File
@@ -196,7 +196,7 @@ pub struct HealOptions {
/// Whether to skip namespace locking
#[serde(default)]
pub no_lock: bool,
/// Timeout
/// Aggregate execution timeout across recoverable manager retries
pub timeout: Option<Duration>,
/// pool index
pub pool_index: Option<usize>,
@@ -442,6 +442,14 @@ impl HealTask {
}
}
pub(crate) async fn retry_request_with_remaining_timeout(&self) -> Result<HealRequest> {
let mut request = self.retry_request();
if self.options.timeout.is_some() {
request.options.timeout = self.remaining_timeout().await?;
}
Ok(request)
}
pub(crate) fn from_replacement_recovery_request(
request: HealRequest,
storage: Arc<dyn HealStorageAPI>,
@@ -2657,6 +2665,36 @@ mod tests {
use super::super::storage_api::status::BucketInfo;
#[tokio::test]
async fn retry_request_carries_remaining_timeout_budget() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage::default());
let mut request = HealRequest::bucket("bucket".to_string());
request.options.timeout = Some(Duration::from_secs(100));
let task = HealTask::from_request(request, storage.clone());
*task.task_start_instant.write().await = Some(Instant::now() - Duration::from_secs(40));
let retry = task
.retry_request_with_remaining_timeout()
.await
.expect("first retry should retain the unused timeout budget");
let first_remaining = retry.options.timeout.expect("configured timeout should remain present");
assert!(first_remaining <= Duration::from_secs(60));
assert!(first_remaining > Duration::from_secs(59));
let retry_task = HealTask::from_request(retry, storage);
*retry_task.task_start_instant.write().await = Some(Instant::now() - Duration::from_secs(20));
let second_retry = retry_task
.retry_request_with_remaining_timeout()
.await
.expect("second retry should retain only the unused aggregate budget");
let second_remaining = second_retry
.options
.timeout
.expect("configured timeout should remain present");
assert!(second_remaining <= Duration::from_secs(40));
assert!(second_remaining > Duration::from_secs(39));
}
#[test]
fn format_result_requires_every_requested_target_to_be_ok() {
let result = HealResultItem {
-2
View File
@@ -225,8 +225,6 @@ async fn nothing_readable_leaves_the_bundle_unwrapped() {
"artifact {} carries the raw on-disk record",
artifact.path
);
// A cheap structural check too: an encrypted payload is not JSON.
assert_ne!(payload.first(), Some(&b'{'), "artifact {} looks like plaintext JSON", artifact.path);
}
// The manifest itself is not encrypted, so assert directly that it carries
+32 -1
View File
@@ -520,6 +520,14 @@ struct FailureSample {
pub struct FailStats {
pub count: i64,
pub size: i64,
/// Rolling-window snapshots refreshed at collection time
/// ([`Self::refresh_windows`]). The raw samples (`recent`) are process
/// local (serde-skipped), so these fields are what survives the peer-RPC
/// wire and [`Self::merge`]-based cluster aggregation.
#[serde(default)]
pub last_minute: FailedMetric,
#[serde(default)]
pub last_hour: FailedMetric,
#[serde(skip)]
recent: VecDeque<FailureSample>,
}
@@ -537,6 +545,17 @@ impl FailStats {
self.prune(observed_at);
}
/// Recompute the serializable rolling-window snapshots from the local
/// samples. Called at the collection point (per-node stats snapshot),
/// never on the failure hot path — the two deque scans are O(window) and
/// `add_size` runs under the bucket-stats write lock. Only meaningful on
/// the live per-node struct: a deserialized or merged struct has no
/// samples, and refreshing it would wipe the aggregated windows.
pub fn refresh_windows(&mut self) {
self.last_minute = self.recent_since(Duration::from_secs(60));
self.last_hour = self.recent_since(Duration::from_secs(3600));
}
fn prune(&mut self, observed_at: Instant) {
while self
.recent
@@ -565,6 +584,16 @@ impl FailStats {
Self {
count: self.count.saturating_add(other.count),
size: self.size.saturating_add(other.size),
// The window snapshots sum across nodes; the raw samples do not
// travel and stay empty on aggregated structs.
last_minute: FailedMetric {
count: self.last_minute.count.saturating_add(other.last_minute.count),
size: self.last_minute.size.saturating_add(other.last_minute.size),
},
last_hour: FailedMetric {
count: self.last_hour.count.saturating_add(other.last_hour.count),
size: self.last_hour.size.saturating_add(other.last_hour.size),
},
recent: VecDeque::new(),
}
}
@@ -636,7 +665,9 @@ impl BucketReplicationStat {
}
pub fn update_xfer_rate(&mut self, size: i64, duration: Duration) {
if size > 1024 * 1024 {
// Same boundary as the worker-pool split and minio-go's
// Large/Small transfer-summary labels: >= 128 MiB is "large".
if size >= crate::runtime::MIN_LARGE_OBJ_SIZE {
self.xfer_rate_lrg.add_size(size, duration);
} else {
self.xfer_rate_sml.add_size(size, duration);
+1 -1
View File
@@ -102,7 +102,7 @@ bytes.workspace = true
hex-simd.workspace = true
[dev-dependencies]
tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
serial_test = { workspace = true }
temp-env = { workspace = true }
tempfile = { workspace = true }
+111 -12
View File
@@ -65,6 +65,7 @@ const LOG_SUBSYSTEM_FOLDER: &str = "folder";
const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
const LOG_SUBSYSTEM_HEAL: &str = "heal";
const EVENT_SCANNER_FOLDER_STATE: &str = "scanner_folder_state";
const EVENT_SCANNER_METADATA_CORRUPT: &str = "scanner_metadata_corrupt";
const EVENT_SCANNER_LIFECYCLE_ACTION: &str = "scanner_lifecycle_action";
const EVENT_SCANNER_HEAL_ADMISSION: &str = "scanner_heal_admission";
const EVENT_SCANNER_ALERT_STATE: &str = "scanner_alert_state";
@@ -2154,17 +2155,34 @@ impl FolderScanner {
self.record_failed(&item.path);
if should_log_failed_object(into.failed_objects) {
warn!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
path = %item.path,
failed_objects = into.failed_objects,
state = "get_size_failed",
error = %e,
"Scanner folder failed to get object size"
);
if let GetSizeFailureAction::HealMetadata { object } = &failure_action {
error!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_METADATA_CORRUPT,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
drive = %self.local_disk.path().display(),
bucket = %item.bucket,
object = %object,
metadata_path = %item.path,
failed_objects = into.failed_objects,
state = "metadata_corrupt",
error = %e,
"Scanner detected corrupt object metadata"
);
} else {
warn!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
path = %item.path,
failed_objects = into.failed_objects,
state = "get_size_failed",
error = %e,
"Scanner folder failed to get object size"
);
}
}
}
@@ -3054,12 +3072,59 @@ mod tests {
use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass};
use rustfs_filemeta::{FileInfo, FileMeta};
use serial_test::serial;
use std::io::Write;
#[cfg(unix)]
use std::os::unix::fs::{PermissionsExt, symlink};
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use temp_env::{with_var, with_var_unset};
use tracing_subscriber::fmt::MakeWriter;
use uuid::Uuid;
#[derive(Clone, Default)]
struct CapturedLogs {
buffer: Arc<Mutex<Vec<u8>>>,
}
struct CapturedLogWriter {
buffer: Arc<Mutex<Vec<u8>>>,
}
impl CapturedLogs {
fn contents(&self) -> String {
let buffer = self
.buffer
.lock()
.expect("captured logs mutex should not be poisoned")
.clone();
String::from_utf8(buffer).expect("captured logs should be valid UTF-8")
}
}
impl Write for CapturedLogWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.buffer
.lock()
.expect("captured logs mutex should not be poisoned")
.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
impl<'a> MakeWriter<'a> for CapturedLogs {
type Writer = CapturedLogWriter;
fn make_writer(&'a self) -> Self::Writer {
CapturedLogWriter {
buffer: Arc::clone(&self.buffer),
}
}
}
#[test]
fn scanner_size_summary_application_saturates_usage_counters() {
let target = "arn:minio:replication::target".to_string();
@@ -4542,9 +4607,19 @@ mod tests {
assert!(budget.entries_visited() >= 1);
}
#[tokio::test]
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
.json()
.with_max_level(tracing::Level::ERROR)
.with_writer(logs.clone())
.with_ansi(false)
.without_time()
.finish();
let _subscriber_guard = tracing::subscriber::set_default(subscriber);
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -4596,6 +4671,30 @@ mod tests {
assert!(!budget.budget_elapsed());
assert_eq!(budget.reason(), None);
let captured = logs.contents();
assert!(
!captured.contains("failed to check XL2 v1 format"),
"the context-free filemeta parser error must not be emitted"
);
let events = captured
.lines()
.map(|line| serde_json::from_str::<serde_json::Value>(line).expect("captured scanner log should be valid JSON"))
.filter(|line| line["fields"]["event"] == EVENT_SCANNER_METADATA_CORRUPT)
.collect::<Vec<_>>();
assert_eq!(
events.len(),
1,
"one corrupt metadata observation must emit one scanner-owned diagnostic event"
);
let fields = &events[0]["fields"];
assert_eq!(fields["component"], LOG_COMPONENT_SCANNER);
assert_eq!(fields["subsystem"], LOG_SUBSYSTEM_FOLDER);
assert_eq!(fields["drive"], temp_dir.to_string_lossy().as_ref());
assert_eq!(fields["bucket"], "bucket");
assert_eq!(fields["object"], "object");
assert_eq!(fields["metadata_path"], metadata_path.to_string_lossy().as_ref());
assert_eq!(fields["state"], "metadata_corrupt");
let retry_budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
crate::scanner_budget::ScannerCycleBudgetConfig {
-11
View File
@@ -3849,17 +3849,6 @@ impl ScannerIODisk for Disk {
let fivs = match meta.get_file_info_versions(item.bucket.as_str(), item.object_path().as_str(), false) {
Ok(versions) => versions,
Err(e) => {
error!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_DISK_BUCKET_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
bucket = %item.bucket,
object = %item.object_path(),
state = "file_info_versions_failed",
error = %e,
"Scanner disk bucket failed to resolve file info versions"
);
return Err(scanner_metadata_corrupt_error(
format!("failed to resolve file info versions: {e}"),
&item.bucket,
+284 -279
View File
@@ -1069,309 +1069,314 @@ mod serial_tests {
#[serial]
#[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
async fn test_transition_and_restore_flows() {
let (disk_paths, ecstore) = setup_test_env().await;
async move {
let (disk_paths, ecstore) = setup_test_env().await;
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&tier_name).await;
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&tier_name).await;
let put_bucket = format!("test-immediate-put-{}", &Uuid::new_v4().simple().to_string()[..8]);
let put_object = "test/object.txt";
let put_payload = b"Hello, immediate transition!";
let put_bucket = format!("test-immediate-put-{}", &Uuid::new_v4().simple().to_string()[..8]);
let put_object = "test/object.txt";
let put_payload = b"Hello, immediate transition!";
create_test_bucket(&ecstore, put_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(put_bucket.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
let mut reader = PutObjReader::from_vec(put_payload.to_vec());
let mut metadata = HashMap::new();
metadata.insert("content-type".to_string(), "text/plain".to_string());
ecstore
.put_object(
put_bucket.as_str(),
put_object,
&mut reader,
&ObjectOptions {
user_defined: metadata,
..Default::default()
},
)
.await
.expect("Failed to upload transition metadata test object");
enqueue_transition_for_existing_objects(ecstore.clone(), put_bucket.as_str())
.await
.expect("Failed to enqueue transitioned put object");
let put_info = wait_for_transition(&ecstore, put_bucket.as_str(), put_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("object should transition after enqueueing existing objects");
assert_eq!(put_info.transitioned_object.status, "complete");
assert_eq!(put_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&put_info.transitioned_object.name).await);
{
let transitioned = backend
.stored(&put_info.transitioned_object.name)
create_test_bucket(&ecstore, put_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(put_bucket.as_str(), &tier_name)
.await
.expect("transitioned object should be present in mock backend");
assert_eq!(transitioned.metadata.get("content-type"), Some(&"text/plain".to_string()));
assert!(
!transitioned.metadata.contains_key("x-amz-replication-status"),
"transitioned objects must not inherit replication status defaults"
);
assert!(
!transitioned.metadata.contains_key("x-amz-object-lock-legal-hold"),
"transitioned objects must not invent object lock headers"
);
}
.expect("Failed to set lifecycle configuration");
// Cross-shard xl.meta transition assertion helper (rustfs/backlog#1148 ilm-6):
// every disk must agree on the transition tuple for the object.
let put_meta = assert_transition_meta_consistent(&disk_paths, put_bucket.as_str(), put_object).await;
assert_eq!(put_meta.status, "complete");
assert_eq!(put_meta.tier, tier_name);
let mut reader = PutObjReader::from_vec(put_payload.to_vec());
let mut metadata = HashMap::new();
metadata.insert("content-type".to_string(), "text/plain".to_string());
ecstore
.put_object(
put_bucket.as_str(),
put_object,
&mut reader,
&ObjectOptions {
user_defined: metadata,
..Default::default()
},
)
.await
.expect("Failed to upload transition metadata test object");
let multipart_bucket = format!("test-immediate-mpu-{}", &Uuid::new_v4().simple().to_string()[..8]);
let multipart_object = "test/multipart.txt";
enqueue_transition_for_existing_objects(ecstore.clone(), put_bucket.as_str())
.await
.expect("Failed to enqueue transitioned put object");
create_test_bucket(&ecstore, multipart_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(multipart_bucket.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
let put_info = wait_for_transition(&ecstore, put_bucket.as_str(), put_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("object should transition after enqueueing existing objects");
let upload = ecstore
.new_multipart_upload(multipart_bucket.as_str(), multipart_object, &ObjectOptions::default())
.await
.expect("Failed to create multipart upload");
assert_eq!(put_info.transitioned_object.status, "complete");
assert_eq!(put_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&put_info.transitioned_object.name).await);
{
let transitioned = backend
.stored(&put_info.transitioned_object.name)
.await
.expect("transitioned object should be present in mock backend");
assert_eq!(transitioned.metadata.get("content-type"), Some(&"text/plain".to_string()));
assert!(
!transitioned.metadata.contains_key("x-amz-replication-status"),
"transitioned objects must not inherit replication status defaults"
);
assert!(
!transitioned.metadata.contains_key("x-amz-object-lock-legal-hold"),
"transitioned objects must not invent object lock headers"
);
}
let part_data = b"multipart immediate transition";
let mut reader = PutObjReader::from_vec(part_data.to_vec());
let part = ecstore
.put_object_part(
multipart_bucket.as_str(),
multipart_object,
&upload.upload_id,
1,
&mut reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload multipart part");
// Cross-shard xl.meta transition assertion helper (rustfs/backlog#1148 ilm-6):
// every disk must agree on the transition tuple for the object.
let put_meta = assert_transition_meta_consistent(&disk_paths, put_bucket.as_str(), put_object).await;
assert_eq!(put_meta.status, "complete");
assert_eq!(put_meta.tier, tier_name);
ecstore
.clone()
.complete_multipart_upload(
multipart_bucket.as_str(),
multipart_object,
&upload.upload_id,
vec![CompletePart {
part_num: 1,
etag: part.etag.clone(),
..Default::default()
}],
&ObjectOptions::default(),
)
.await
.expect("Failed to complete multipart upload");
let multipart_bucket = format!("test-immediate-mpu-{}", &Uuid::new_v4().simple().to_string()[..8]);
let multipart_object = "test/multipart.txt";
enqueue_transition_for_existing_objects(ecstore.clone(), multipart_bucket.as_str())
.await
.expect("Failed to enqueue transitioned multipart object");
create_test_bucket(&ecstore, multipart_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(multipart_bucket.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
let multipart_info = wait_for_transition(&ecstore, multipart_bucket.as_str(), multipart_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("object should transition after enqueueing existing objects");
let upload = ecstore
.new_multipart_upload(multipart_bucket.as_str(), multipart_object, &ObjectOptions::default())
.await
.expect("Failed to create multipart upload");
assert_eq!(multipart_info.transitioned_object.status, "complete");
assert_eq!(multipart_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&multipart_info.transitioned_object.name).await);
let part_data = b"multipart immediate transition";
let mut reader = PutObjReader::from_vec(part_data.to_vec());
let part = ecstore
.put_object_part(
multipart_bucket.as_str(),
multipart_object,
&upload.upload_id,
1,
&mut reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload multipart part");
let src_bucket = format!("test-immediate-copy-src-{}", &Uuid::new_v4().simple().to_string()[..8]);
let dst_bucket = format!("test-immediate-copy-dst-{}", &Uuid::new_v4().simple().to_string()[..8]);
let src_object = "test/source.txt";
let dst_object = "test/copied.txt";
let payload = b"copy object immediate transition";
create_test_bucket(&ecstore, src_bucket.as_str()).await;
create_test_bucket(&ecstore, dst_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(dst_bucket.as_str(), &tier_name)
.await
.expect("Failed to set destination lifecycle configuration");
upload_test_object(&ecstore, src_bucket.as_str(), src_object, payload).await;
let mut src_info = ecstore
.get_object_info(src_bucket.as_str(), src_object, &ObjectOptions::default())
.await
.expect("Failed to load source object info");
src_info.put_object_reader = Some(PutObjReader::from_vec(payload.to_vec()));
ecstore
.copy_object(
src_bucket.as_str(),
src_object,
dst_bucket.as_str(),
dst_object,
&mut src_info,
&ObjectOptions::default(),
&ObjectOptions::default(),
)
.await
.expect("Failed to copy object");
enqueue_transition_for_existing_objects(ecstore.clone(), dst_bucket.as_str())
.await
.expect("Failed to enqueue transitioned copied object");
let copy_info = wait_for_transition(&ecstore, dst_bucket.as_str(), dst_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("copied object should transition after enqueueing existing objects");
assert_eq!(copy_info.transitioned_object.status, "complete");
assert_eq!(copy_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&copy_info.transitioned_object.name).await);
let bucket_name = format!("test-lifecycle-update-{}", &Uuid::new_v4().simple().to_string()[..8]);
let object_name = "test/existing.txt";
let payload = b"existing object before lifecycle";
create_test_bucket(&ecstore, bucket_name.as_str()).await;
upload_test_object(&ecstore, bucket_name.as_str(), object_name, payload).await;
set_bucket_lifecycle_transition_with_tier(bucket_name.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
enqueue_transition_for_existing_objects(ecstore.clone(), bucket_name.as_str())
.await
.expect("Failed to enqueue transition for existing objects");
let info = wait_for_transition(&ecstore, bucket_name.as_str(), object_name, TRANSITION_WAIT_TIMEOUT)
.await
.expect("existing object should transition after lifecycle update");
assert_eq!(info.transitioned_object.status, "complete");
assert_eq!(info.transitioned_object.tier, tier_name);
assert!(backend.contains(&info.transitioned_object.name).await);
let bucket_name = format!("test-restore-mpu-{}", &Uuid::new_v4().simple().to_string()[..8]);
let object_name = "test/restore.txt";
let part1 = vec![b'a'; 5 * 1024 * 1024];
let part2 = b"restored-tail".to_vec();
let expected = [part1.clone(), part2.clone()].concat();
create_test_bucket(&ecstore, bucket_name.as_str()).await;
set_bucket_lifecycle_transition_with_tier(bucket_name.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
let upload = ecstore
.new_multipart_upload(bucket_name.as_str(), object_name, &ObjectOptions::default())
.await
.expect("Failed to create multipart upload");
let mut part1_reader = PutObjReader::from_vec(part1);
let uploaded_part1 = ecstore
.put_object_part(
bucket_name.as_str(),
object_name,
&upload.upload_id,
1,
&mut part1_reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload first multipart part");
let mut part2_reader = PutObjReader::from_vec(part2);
let uploaded_part2 = ecstore
.put_object_part(
bucket_name.as_str(),
object_name,
&upload.upload_id,
2,
&mut part2_reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload second multipart part");
ecstore
.clone()
.complete_multipart_upload(
bucket_name.as_str(),
object_name,
&upload.upload_id,
vec![
CompletePart {
ecstore
.clone()
.complete_multipart_upload(
multipart_bucket.as_str(),
multipart_object,
&upload.upload_id,
vec![CompletePart {
part_num: 1,
etag: uploaded_part1.etag.clone(),
etag: part.etag.clone(),
..Default::default()
},
CompletePart {
part_num: 2,
etag: uploaded_part2.etag.clone(),
..Default::default()
},
],
&ObjectOptions::default(),
)
.await
.expect("Failed to complete multipart upload");
}],
&ObjectOptions::default(),
)
.await
.expect("Failed to complete multipart upload");
enqueue_transition_for_existing_objects(ecstore.clone(), bucket_name.as_str())
.await
.expect("Failed to enqueue transitioned restore object");
enqueue_transition_for_existing_objects(ecstore.clone(), multipart_bucket.as_str())
.await
.expect("Failed to enqueue transitioned multipart object");
let transitioned = wait_for_transition(&ecstore, bucket_name.as_str(), object_name, TRANSITION_WAIT_TIMEOUT)
.await
.expect("multipart object should transition after enqueueing existing objects");
assert_eq!(transitioned.parts.len(), 2);
let multipart_info =
wait_for_transition(&ecstore, multipart_bucket.as_str(), multipart_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("object should transition after enqueueing existing objects");
ecstore
.clone()
.restore_transitioned_object(
bucket_name.as_str(),
object_name,
&ObjectOptions {
transition: TransitionOptions {
restore_request: RestoreRequest {
days: Some(1),
description: None,
glacier_job_parameters: None,
output_location: None,
select_parameters: None,
tier: None,
type_: None,
assert_eq!(multipart_info.transitioned_object.status, "complete");
assert_eq!(multipart_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&multipart_info.transitioned_object.name).await);
let src_bucket = format!("test-immediate-copy-src-{}", &Uuid::new_v4().simple().to_string()[..8]);
let dst_bucket = format!("test-immediate-copy-dst-{}", &Uuid::new_v4().simple().to_string()[..8]);
let src_object = "test/source.txt";
let dst_object = "test/copied.txt";
let payload = b"copy object immediate transition";
create_test_bucket(&ecstore, src_bucket.as_str()).await;
create_test_bucket(&ecstore, dst_bucket.as_str()).await;
set_bucket_lifecycle_transition_with_tier(dst_bucket.as_str(), &tier_name)
.await
.expect("Failed to set destination lifecycle configuration");
upload_test_object(&ecstore, src_bucket.as_str(), src_object, payload).await;
let mut src_info = ecstore
.get_object_info(src_bucket.as_str(), src_object, &ObjectOptions::default())
.await
.expect("Failed to load source object info");
src_info.put_object_reader = Some(PutObjReader::from_vec(payload.to_vec()));
ecstore
.copy_object(
src_bucket.as_str(),
src_object,
dst_bucket.as_str(),
dst_object,
&mut src_info,
&ObjectOptions::default(),
&ObjectOptions::default(),
)
.await
.expect("Failed to copy object");
enqueue_transition_for_existing_objects(ecstore.clone(), dst_bucket.as_str())
.await
.expect("Failed to enqueue transitioned copied object");
let copy_info = wait_for_transition(&ecstore, dst_bucket.as_str(), dst_object, TRANSITION_WAIT_TIMEOUT)
.await
.expect("copied object should transition after enqueueing existing objects");
assert_eq!(copy_info.transitioned_object.status, "complete");
assert_eq!(copy_info.transitioned_object.tier, tier_name);
assert!(backend.contains(&copy_info.transitioned_object.name).await);
let bucket_name = format!("test-lifecycle-update-{}", &Uuid::new_v4().simple().to_string()[..8]);
let object_name = "test/existing.txt";
let payload = b"existing object before lifecycle";
create_test_bucket(&ecstore, bucket_name.as_str()).await;
upload_test_object(&ecstore, bucket_name.as_str(), object_name, payload).await;
set_bucket_lifecycle_transition_with_tier(bucket_name.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
enqueue_transition_for_existing_objects(ecstore.clone(), bucket_name.as_str())
.await
.expect("Failed to enqueue transition for existing objects");
let info = wait_for_transition(&ecstore, bucket_name.as_str(), object_name, TRANSITION_WAIT_TIMEOUT)
.await
.expect("existing object should transition after lifecycle update");
assert_eq!(info.transitioned_object.status, "complete");
assert_eq!(info.transitioned_object.tier, tier_name);
assert!(backend.contains(&info.transitioned_object.name).await);
let bucket_name = format!("test-restore-mpu-{}", &Uuid::new_v4().simple().to_string()[..8]);
let object_name = "test/restore.txt";
let part1 = vec![b'a'; 5 * 1024 * 1024];
let part2 = b"restored-tail".to_vec();
let expected = [part1.clone(), part2.clone()].concat();
create_test_bucket(&ecstore, bucket_name.as_str()).await;
set_bucket_lifecycle_transition_with_tier(bucket_name.as_str(), &tier_name)
.await
.expect("Failed to set lifecycle configuration");
let upload = ecstore
.new_multipart_upload(bucket_name.as_str(), object_name, &ObjectOptions::default())
.await
.expect("Failed to create multipart upload");
let mut part1_reader = PutObjReader::from_vec(part1);
let uploaded_part1 = ecstore
.put_object_part(
bucket_name.as_str(),
object_name,
&upload.upload_id,
1,
&mut part1_reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload first multipart part");
let mut part2_reader = PutObjReader::from_vec(part2);
let uploaded_part2 = ecstore
.put_object_part(
bucket_name.as_str(),
object_name,
&upload.upload_id,
2,
&mut part2_reader,
&ObjectOptions::default(),
)
.await
.expect("Failed to upload second multipart part");
ecstore
.clone()
.complete_multipart_upload(
bucket_name.as_str(),
object_name,
&upload.upload_id,
vec![
CompletePart {
part_num: 1,
etag: uploaded_part1.etag.clone(),
..Default::default()
},
CompletePart {
part_num: 2,
etag: uploaded_part2.etag.clone(),
..Default::default()
},
],
&ObjectOptions::default(),
)
.await
.expect("Failed to complete multipart upload");
enqueue_transition_for_existing_objects(ecstore.clone(), bucket_name.as_str())
.await
.expect("Failed to enqueue transitioned restore object");
let transitioned = wait_for_transition(&ecstore, bucket_name.as_str(), object_name, TRANSITION_WAIT_TIMEOUT)
.await
.expect("multipart object should transition after enqueueing existing objects");
assert_eq!(transitioned.parts.len(), 2);
ecstore
.clone()
.restore_transitioned_object(
bucket_name.as_str(),
object_name,
&ObjectOptions {
transition: TransitionOptions {
restore_request: RestoreRequest {
days: Some(1),
description: None,
glacier_job_parameters: None,
output_location: None,
select_parameters: None,
tier: None,
type_: None,
},
..Default::default()
},
..Default::default()
},
..Default::default()
},
)
.await
.expect("Failed to restore transitioned multipart object");
)
.await
.expect("Failed to restore transitioned multipart object");
let restored = ecstore
.get_object_info(bucket_name.as_str(), object_name, &ObjectOptions::default())
.await
.expect("Failed to load restored object info");
assert_eq!(restored.parts.len(), 2);
assert!(restored.restore_expires.is_some());
assert!(!restored.restore_ongoing);
let restored = ecstore
.get_object_info(bucket_name.as_str(), object_name, &ObjectOptions::default())
.await
.expect("Failed to load restored object info");
assert_eq!(restored.parts.len(), 2);
assert!(restored.restore_expires.is_some());
assert!(!restored.restore_ongoing);
let mut reader = ecstore
.get_object_reader(bucket_name.as_str(), object_name, None, http::HeaderMap::new(), &ObjectOptions::default())
.await
.expect("Failed to read restored object");
let mut data = Vec::new();
reader
.stream
.read_to_end(&mut data)
.await
.expect("Failed to consume restored object stream");
assert_eq!(data, expected);
let mut reader = ecstore
.get_object_reader(bucket_name.as_str(), object_name, None, http::HeaderMap::new(), &ObjectOptions::default())
.await
.expect("Failed to read restored object");
let mut data = Vec::new();
reader
.stream
.read_to_end(&mut data)
.await
.expect("Failed to consume restored object stream");
assert_eq!(data, expected);
}
.boxed_local()
.await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+29
View File
@@ -50,6 +50,14 @@ pub const SUFFIX_SOURCE_DELETEMARKER: &str = "source-deletemarker";
pub const SUFFIX_SOURCE_PROXY_REQUEST: &str = "source-proxy-request";
pub const SUFFIX_SOURCE_REPLICATION_REQUEST: &str = "source-replication-request";
pub const SUFFIX_SOURCE_REPLICATION_CHECK: &str = "source-replication-check";
// LWW timestamps for replicated tag/retention/legal-hold modifications. MinIO
// declares these with mixed case (internal/http/headers.go:
// X-Minio-Source-Replication-Tagging-Timestamp / -Retention-Timestamp /
// -LegalHold-Timestamp); HTTP header names compare case-insensitively, so the
// lowercase suffix forms interoperate. Values are RFC3339 on the wire.
pub const SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP: &str = "source-replication-tagging-timestamp";
pub const SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP: &str = "source-replication-retention-timestamp";
pub const SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP: &str = "source-replication-legalhold-timestamp";
pub const SUFFIX_REPLICATION_SSEC_CRC: &str = "replication-ssec-crc";
/// Returns true if the key is object-encryption metadata understood by RustFS or MinIO.
@@ -196,6 +204,27 @@ mod tests {
assert_eq!(get_object_encryption_original_size(&metadata).expect("valid size"), Some(42));
}
#[test]
fn replication_timestamp_headers_match_minio_wire_names() {
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, "2026-01-02T03:04:05Z");
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, "2026-01-02T03:04:06Z");
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, "2026-01-02T03:04:07Z");
// The exact names MinIO's object-api-options.go reads (its Get()
// canonicalizes case, so a case-insensitive match is wire-equivalent).
for name in [
"X-Minio-Source-Replication-Tagging-Timestamp",
"X-Minio-Source-Replication-Retention-Timestamp",
"X-Minio-Source-Replication-LegalHold-Timestamp",
"x-rustfs-source-replication-tagging-timestamp",
"x-rustfs-source-replication-retention-timestamp",
"x-rustfs-source-replication-legalhold-timestamp",
] {
assert!(headers.contains_key(name), "replication timestamp header {name} must be written");
}
}
#[test]
fn test_get_header() {
let mut headers = HeaderMap::new();
-3
View File
@@ -659,9 +659,6 @@ mod test {
// Port should be in valid range (u16 max is always <= 65535)
assert!(port1 > 0);
assert!(port2 > 0);
// Different calls should typically return different ports
assert_ne!(port1, port2);
}
#[test]
@@ -61,17 +61,17 @@ catalog extension.
| Area | Status | Covered behavior |
|---|---|---|
| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and route capabilities. |
| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and only the supported OpenAPI REST paths in `endpoints`. RustFS administration, maintenance, migration, diagnostics, refs, and metadata-location extensions remain available but are not presented as standard Iceberg REST endpoints. |
| Table bucket discovery | Supported | `PUT` and `GET /v1/buckets/{warehouse}` enable and inspect table bucket state. |
| Namespaces | Supported | Create, list, load, existence check, and drop namespace routes are registered on both catalog prefixes. List responses support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Namespace identifiers are limited to 512 ASCII characters so persisted paths and stateless continuation tokens remain bounded. |
| Tables | Supported | Create, register, list, load, existence check, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. |
| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. |
| Tables | Supported | Create, register, list, load, existence check, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Commit identifiers must match the URL resource; unknown requirements, updates, and snapshot operations fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. Standard statistics, partition statistics, and schema/spec cleanup updates are accepted. |
| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Externally supplied metadata transitions preserve monotonic column, partition, and sequence assignment watermarks and immutable definitions for retained schemas, partition specs, sort orders, and snapshots. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. The catalog does not advertise `idempotency-key-lifetime`; clients must treat standard mutation-wide `Idempotency-Key` semantics as unsupported. |
| Commit recovery | Supported | Commit log, idempotency lookup, diagnostics, and recovery routes expose staged/finalization gaps and repair safe idempotency gaps without moving the table pointer. |
| Snapshot refs | Supported | Refs can be listed, created or replaced, and deleted through catalog commits. `main` is protected and refs with explicit retention require forced delete. |
| Iceberg views | Supported | Basic create, list, load, replace, existence check, and drop routes persist view metadata with view-scoped authorization. |
| Table credentials endpoint | Supported | Returns an empty `storage-credentials` list by default. Returns table-scoped temporary credentials only when credential vending is enabled. |
| Iceberg views | Supported | Basic create, list, load, replace, existence check, and drop routes persist view metadata with view-scoped authorization. Replace identifiers must match the URL resource, `schema-id: -1` resolves to the last added schema, one commit timestamp is used consistently, and only Iceberg view format version 1 is accepted. |
| Table credentials endpoint | Supported | Returns an empty `storage-credentials` list by default. Returns table-scoped temporary credentials only when credential vending is enabled. Credential responses set `Cache-Control: no-store, private`, `Pragma: no-cache`, and `Expires: 0`. |
| Catalog diagnostics and export | Supported | Exposes recovery state, consistency state, backing manifest, recoverable commit-log WAL state, strong backing migration target, single-active-writer policy, and scale validation matrix. |
| Catalog import and rollback | Supported | Import/register and rollback use catalog validation and commit paths rather than direct pointer mutation. |
| Catalog import and rollback | Supported | Import/register and online rollback use catalog validation and commit paths rather than direct pointer mutation. Online rollback accepts only a forward-safe metadata target that preserves assignment watermarks and retained definitions. Restoring an older target that lowers those watermarks is an offline disaster-recovery operation and requires every writer to be stopped. |
| External catalog bridge | Supported operator path | Operator-supplied metadata pointer sync/import is supported for external catalog identity boundaries. Online vendor SDK polling and policy mirroring are not claimed. |
| Multi-table transactions | Not claimed | RustFS currently claims single-table commit atomicity only. |
Generated
+6 -6
View File
@@ -2,11 +2,11 @@
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1785975029,
"narHash": "sha256-X44cn5rzytELc3NNoQsh0aLkjWA/QzPfc6HPQmsG3sU=",
"lastModified": 1786719841,
"narHash": "sha256-QcpQOT0NQEFkI77t+YXPZqDJc35iIodG7zinieOwFUg=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "70ce234312134a463ba7728e94da2486a1d237ac",
"rev": "8be7bd0c83f12e2e3bbba07c9044d6fed9e66f7f",
"type": "github"
},
"original": {
@@ -29,11 +29,11 @@
]
},
"locked": {
"lastModified": 1786247265,
"narHash": "sha256-cVTcTqAhzSNMOVddtBhFpuv+Vx6/SgrhgZWJiI61H24=",
"lastModified": 1786849542,
"narHash": "sha256-MS8cOa/ii1+QA8R3JWVzqEIoXimu9n50GwQDiBVTFOM=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "6df7076ea0f5697e3242719a4c71211f00411cf5",
"rev": "b211eadeba8b180da9453ec3413a8a3535c85b3f",
"type": "github"
},
"original": {
+2 -1
View File
@@ -278,6 +278,8 @@ rustfs-signer.workspace = true
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["raw_value"] }
serde_urlencoded = { workspace = true }
snap.workspace = true
zstd.workspace = true
# Cryptography and Security
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
@@ -355,7 +357,6 @@ rcgen = { workspace = true }
rustfs-test-utils.workspace = true
# diagnose_e2e fixtures (archives are generated in-test, never checked in)
zip = { workspace = true }
zstd = { workspace = true }
# Enables the shared MockWarmBackend / xl.meta assertion helpers exposed via
# the ecstore `api::tier::test_util` facade module (rustfs/backlog#1148 ilm-6).
rustfs-ecstore = { workspace = true, features = ["test-util"] }
+8 -7
View File
@@ -40,7 +40,7 @@ use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::LazyLock;
use tracing::{Span, error, info, warn};
use tracing::{error, info, warn};
const LOG_COMPONENT_ADMIN_API: &str = "admin_api";
const LOG_SUBSYSTEM_AUDIT_TARGET: &str = "audit_target";
@@ -278,8 +278,6 @@ pub struct AuditTargetConfig {}
#[async_trait::async_trait]
impl Operation for AuditTargetConfig {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;
@@ -345,8 +343,6 @@ pub struct ListAuditTargets {}
#[async_trait::async_trait]
impl Operation for ListAuditTargets {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_audit_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
let mut runtime_statuses = HashMap::new();
@@ -370,8 +366,6 @@ pub struct RemoveAuditTarget {}
#[async_trait::async_trait]
impl Operation for RemoveAuditTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;
@@ -838,6 +832,13 @@ mod tests {
extract_block_between_markers(src, "impl Operation for ListAuditTargets", "pub struct RemoveAuditTarget");
let delete_block = extract_block_between_markers(src, "impl Operation for RemoveAuditTarget", "#[cfg(test)]");
for block in [put_block, list_block, delete_block] {
assert!(
!block.contains(".enter()"),
"async audit handlers must rely on request-future instrumentation instead of holding span guards across awaits"
);
}
assert!(
put_block.contains("authorize_audit_admin_request(&req, AdminAction::SetBucketTargetAction).await?;"),
"audit target writes should require SetBucketTargetAction"
+8 -9
View File
@@ -43,7 +43,7 @@ use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::LazyLock;
use tracing::{Span, error, info, warn};
use tracing::{error, info, warn};
const LOG_COMPONENT_ADMIN_API: &str = "admin_api";
@@ -333,8 +333,6 @@ pub struct NotificationTarget {}
#[async_trait::async_trait]
impl Operation for NotificationTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
let context = app_context_from_req(&req);
@@ -401,8 +399,6 @@ pub struct ListNotificationTargets {}
#[async_trait::async_trait]
impl Operation for ListNotificationTargets {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_notification_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
refresh_persisted_module_switches_from_store().await.map_err(|err| {
warn!(
@@ -439,8 +435,6 @@ pub struct ListTargetsArns {}
#[async_trait::async_trait]
impl Operation for ListTargetsArns {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
authorize_notification_admin_request(&req, AdminAction::GetBucketTargetAction).await?;
if let Some(reason) = notification_target_operation_block_reason(
"querying notification target ARNs for bucket associations from the console",
@@ -485,8 +479,6 @@ pub struct RemoveNotificationTarget {}
#[async_trait::async_trait]
impl Operation for RemoveNotificationTarget {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let span = Span::current();
let _enter = span.enter();
let (target_type, target_name) = extract_target_params(&params)?;
let context = app_context_from_req(&req);
@@ -1006,6 +998,13 @@ mod tests {
extract_block_between_markers(src, "impl Operation for ListTargetsArns", "pub struct RemoveNotificationTarget");
let delete_block = extract_block_between_markers(src, "impl Operation for RemoveNotificationTarget", "fn extract_param");
for block in [put_block, list_block, arns_block, delete_block] {
assert!(
!block.contains(".enter()"),
"async notification handlers must rely on request-future instrumentation instead of holding span guards across awaits"
);
}
assert!(
put_block.contains("authorize_notification_admin_request(&req, AdminAction::SetBucketTargetAction).await?;"),
"notification target writes should require SetBucketTargetAction"
+353 -19
View File
@@ -535,8 +535,13 @@ impl Operation for GetReplicationMetricsHandler {
let bucket_stats = cluster_replication_stats(bucket, app_context_from_req(&req)).await;
let data = serde_json::to_vec(&bucket_stats.replication_stats)
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "serialize failed"))?;
// Same minio-go `replication.Metrics` wire shape as
// `?replication-metrics` — the internal snake_case stats are the peer
// RPC wire format and must not leak here.
let data = serde_json::to_vec(&crate::admin::replication_metrics_wire::MetricsWire::from(
&bucket_stats.replication_stats,
))
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "serialize failed"))?;
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
Ok(S3Response::with_headers((StatusCode::OK, Body::from(data)), headers))
@@ -1153,7 +1158,7 @@ struct MrfResponse {
fn build_mrf_response(
bucket: String,
bucket_stats: &BucketStats,
durable: crate::admin::storage_api::replication::DurableMrfBacklog,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
) -> MrfResponse {
let observation_scope = if bucket_stats.replication_stats.cluster_complete {
"cluster_aggregated"
@@ -1223,7 +1228,10 @@ fn build_mrf_response(
total_failed_size,
queued_count: queued.count,
queued_size: queued.bytes,
per_object_entries_available: false,
// The default (non-aggregate) response mode streams the durable
// backlog per object, so the enumerable API exists whenever the
// backlog is readable.
per_object_entries_available: durable.available,
runtime_stats_available: bucket_stats.replication_stats.provider_available,
cluster_complete: bucket_stats.replication_stats.cluster_complete,
observed_node_count: bucket_stats.replication_stats.observed_node_count,
@@ -1235,23 +1243,165 @@ fn build_mrf_response(
}
}
/// One durable MRF backlog entry rendered for the default (madmin-compatible)
/// stream. Field names are the exact json tags of madmin-go `ReplicationMRF`
/// (replication-api.go), which `mc replicate backlog` decodes one JSON
/// document at a time. `Size` and `TargetARNs` are RustFS extension keys with
/// no madmin counterpart; Go decoders ignore unknown keys.
#[derive(Debug, Serialize)]
struct MrfEntryDocument {
/// The durable backlog is a cluster-shared ledger with no per-node
/// attribution, so the madmin `nodeName` tag is always empty.
#[serde(rename = "nodeName")]
node_name: String,
#[serde(rename = "bucket")]
bucket: String,
#[serde(rename = "object")]
object: String,
#[serde(rename = "versionId")]
version_id: String,
#[serde(rename = "retryCount")]
retry_count: i32,
#[serde(rename = "Size")]
size: i64,
#[serde(rename = "TargetARNs", skip_serializing_if = "Vec::is_empty")]
target_arns: Vec<String>,
}
/// Upper bound on the number of documents one stream response emits. The
/// durable ledger is not bounded by the in-memory pending cap (recovery can
/// persist far larger generations), and the body is buffered before send, so
/// an unbounded read could stage hundreds of MB per request. The handler
/// rejects a response beyond this bound instead of returning a partial 200.
const REPLICATION_MRF_MAX_STREAM_ENTRIES: usize = 10_000;
/// Project the durable backlog into madmin `ReplicationMRF` documents,
/// scoped to `bucket` when it is non-empty (madmin allows an empty bucket to
/// mean "across all buckets"), bounded by
/// [`REPLICATION_MRF_MAX_STREAM_ENTRIES`]. Returns the documents and whether
/// the backlog was truncated.
fn mrf_entry_documents(
bucket: &str,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
) -> (Vec<MrfEntryDocument>, bool) {
let mut documents = Vec::new();
let mut truncated = false;
for entry in durable
.entries
.iter()
.filter(|entry| bucket.is_empty() || entry.bucket == bucket)
{
if documents.len() >= REPLICATION_MRF_MAX_STREAM_ENTRIES {
truncated = true;
break;
}
documents.push(MrfEntryDocument {
node_name: String::new(),
bucket: entry.bucket.clone(),
object: entry.object.clone(),
// Delete-marker purge entries track the marker version separately;
// fall back to it so those rows still carry a version identity.
// The nil UUID is RustFS's in-memory null-version sentinel and
// must leave as the S3 wire token, not a zero UUID.
version_id: entry
.version_id
.or(entry.delete_marker_version_id)
.map(|v| {
if v.is_nil() {
rustfs_filemeta::NULL_VERSION_ID.to_string()
} else {
v.to_string()
}
})
.unwrap_or_default(),
retry_count: entry.retry_count,
size: entry.size,
target_arns: entry.target_arns.clone(),
});
}
(documents, truncated)
}
/// Render the MRF backlog as a response body.
///
/// Default (madmin-compatible) mode emits one `ReplicationMRF` JSON document
/// per line with no envelope — madmin's `BucketReplicationMRF` reads the body
/// with a `json.Decoder` loop, so an envelope object would decode as a single
/// entry whose `"Bucket"` key case-insensitively matches
/// `ReplicationMRF.Bucket` (a phantom row in `mc replicate backlog`), and an
/// empty backlog must render an empty body so the loop ends on io.EOF with
/// zero rows.
///
/// `aggregate=true` (RustFS extension) keeps the enveloped counter shape;
/// backlog-source health (`RuntimeStatsAvailable`/`DurableBacklogAvailable`)
/// is only representable there — an unreadable ledger fails the stream
/// request outright in the handler (madmin only decodes the body of a 200,
/// so an empty stream would read as a healthy zero-row backlog).
fn render_mrf_backlog(
response: &MrfResponse,
durable: &crate::admin::storage_api::replication::DurableMrfBacklog,
aggregate: bool,
) -> Result<(Vec<u8>, bool), serde_json::Error> {
if aggregate {
return Ok((serde_json::to_vec(response)?, false));
}
let (documents, truncated) = mrf_entry_documents(&response.bucket, durable);
let mut data = Vec::new();
for entry in documents {
serde_json::to_writer(&mut data, &entry)?;
data.push(b'\n');
}
Ok((data, truncated))
}
fn ensure_complete_mrf_stream(truncated: bool) -> S3Result<()> {
if truncated {
return Err(S3Error::with_message(
S3ErrorCode::ServiceUnavailable,
"durable MRF backlog exceeds the stream limit; narrow the bucket scope or drain the backlog".to_string(),
));
}
Ok(())
}
/// `GET /v3/replication/mrf`
///
/// Reports the failed-replication backlog (MinIO's MRF concept) for a bucket.
///
/// Compatibility note: MinIO returns a stream of individual MRF entries. RustFS
/// deliberately returns aggregate runtime and durable counters instead.
/// `PerObjectEntriesAvailable` remains false until an enumerable API exists.
/// `PerTargetDurableEntriesAvailable` is false when the durable backlog includes
/// older entries that cannot be attributed to a target.
/// The default response is a madmin-compatible stream of `ReplicationMRF`
/// documents built from the durable backlog ledger (in-memory failures that
/// have not been flushed yet — the persister runs every few seconds — are not
/// visible). `?aggregate=true` (RustFS extension) returns the enveloped
/// runtime + durable counter shape instead; `PerTargetDurableEntriesAvailable`
/// is false there when the durable backlog includes older entries that cannot
/// be attributed to a target.
///
/// The madmin `node` parameter is accepted but has no filtering effect: the
/// durable ledger is cluster-shared with no per-node attribution, so every
/// node serves the same (complete) backlog.
///
/// Authorization: the stream requires `admin:ReplicationDiff` (it enumerates
/// object names and version ids, MinIO parity); `?aggregate=true` carries no
/// object identities and requires only `admin:GetReplicationMetrics`.
pub struct ReplicationMrfHandler {}
#[async_trait::async_trait]
impl Operation for ReplicationMrfHandler {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
validate_replication_admin_request(&req, AdminAction::GetReplicationMetricsAction).await?;
let queries = extract_query_params(&req.uri);
let aggregate = queries.get("aggregate").map(String::as_str) == Some("true");
// The default stream enumerates object names and version ids, which
// a metrics-only principal must not see; gate it on the same action
// MinIO uses for this endpoint. The aggregate counters carry no
// object identities and keep the metrics action.
let action = if aggregate {
AdminAction::GetReplicationMetricsAction
} else {
AdminAction::ReplicationDiff
};
validate_replication_admin_request(&req, action).await?;
let Some(bucket) = queries.get("bucket").filter(|b| !b.is_empty()).cloned() else {
return Err(s3_error!(InvalidRequest, "bucket is required"));
};
@@ -1275,14 +1425,48 @@ impl Operation for ReplicationMrfHandler {
return Err(ApiError::from(err).into());
}
if let Some(node) = queries.get("node").filter(|node| !node.is_empty() && node.as_str() != "all") {
// The durable backlog ledger is cluster-shared with no per-node
// attribution, so a node-scoped request still sees the complete
// (superset) backlog.
debug!(node = %node, "replication mrf node filter has no effect on the cluster-shared backlog");
}
let durable = crate::admin::storage_api::replication::read_durable_mrf_backlog(store).await;
let bucket_stats = cluster_replication_stats(&bucket, app_context_from_req(&req)).await;
let response = build_mrf_response(bucket, &bucket_stats, durable);
let response = build_mrf_response(bucket, &bucket_stats, &durable);
let data = serde_json::to_vec(&response)
if !durable.available && !aggregate {
// The madmin stream has no envelope to carry source health, and
// madmin only decodes the body of a 200 — an empty stream would
// read as a clean, healthy zero-row backlog. Fail loudly instead;
// aggregate mode still reports the availability fields.
tracing::warn!(
bucket = %response.bucket,
"durable MRF backlog is unreadable; failing the stream request — use aggregate=true to see source health"
);
return Err(S3Error::with_message(
S3ErrorCode::ServiceUnavailable,
"durable MRF backlog is unreadable; retry, or use aggregate=true for source health".to_string(),
));
}
let (data, truncated) = render_mrf_backlog(&response, &durable, aggregate)
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize failed: {e}")))?;
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
if truncated {
tracing::warn!(
event = "replication_mrf_stream_rejected",
component = "admin",
subsystem = "replication",
result = "rejected",
bucket = %response.bucket,
max_entries = REPLICATION_MRF_MAX_STREAM_ENTRIES,
"replication mrf stream exceeds the response limit"
);
}
ensure_complete_mrf_stream(truncated)?;
Ok(S3Response::with_headers((StatusCode::OK, Body::from(data)), headers))
}
}
@@ -1292,7 +1476,8 @@ mod tests {
use super::{
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, RemoteTargetCredentialsRequest, RemoteTargetRequest,
ReplicationDiffEntry, SUPPORTED_REMOTE_TARGET_API, TargetUpdateOp, build_mrf_response, extract_query_params,
parse_remote_target_update_ops, render_replication_diff, unique_replication_peers, validate_remote_target_tls_settings,
parse_remote_target_update_ops, render_mrf_backlog, render_replication_diff, unique_replication_peers,
validate_remote_target_tls_settings,
};
use crate::admin::storage_api::bucket::target::{BucketTarget, LatencyStat};
use crate::admin::storage_api::replication::{BucketStats, DurableMrfBacklog, MrfOpKind, MrfReplicateEntry};
@@ -1510,7 +1695,7 @@ mod tests {
],
};
let response = build_mrf_response("bucket-a".to_string(), &stats, durable);
let response = build_mrf_response("bucket-a".to_string(), &stats, &durable);
let json = serde_json::to_value(response).expect("MRF response should serialize");
assert_eq!(json["TotalFailedCount"], 3);
@@ -1522,7 +1707,9 @@ mod tests {
assert_eq!(json["RuntimeStatsAvailable"], true);
assert_eq!(json["ClusterComplete"], false);
assert_eq!(json["Targets"][0]["ObservationScope"], "partial_cluster");
assert_eq!(json["PerObjectEntriesAvailable"], false);
// The bare stream enumerates the durable backlog per object, so a
// readable backlog advertises the enumerable API.
assert_eq!(json["PerObjectEntriesAvailable"], true);
assert_eq!(json["PerTargetDurableEntriesAvailable"], true);
let targets = json["Targets"].as_array().expect("targets should serialize as an array");
@@ -1569,7 +1756,7 @@ mod tests {
}],
};
let response = build_mrf_response("bucket-a".to_string(), &stats, durable);
let response = build_mrf_response("bucket-a".to_string(), &stats, &durable);
let json = serde_json::to_value(response).expect("MRF response should serialize");
assert_eq!(json["DurableBacklogAvailable"], true);
@@ -1587,7 +1774,7 @@ mod tests {
#[test]
fn mrf_response_distinguishes_unavailable_sources_from_valid_zero() {
let unavailable = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), DurableMrfBacklog::default());
let unavailable = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &DurableMrfBacklog::default());
let unavailable_json = serde_json::to_value(unavailable).expect("unavailable response should serialize");
assert_eq!(unavailable_json["RuntimeStatsAvailable"], false);
assert_eq!(unavailable_json["DurableBacklogAvailable"], false);
@@ -1600,7 +1787,7 @@ mod tests {
let valid_empty = build_mrf_response(
"bucket-a".to_string(),
&valid_empty_stats,
DurableMrfBacklog {
&DurableMrfBacklog {
available: true,
entries: Vec::new(),
},
@@ -1613,6 +1800,153 @@ mod tests {
assert_eq!(valid_empty_json["PerTargetDurableEntriesAvailable"], true);
}
fn sample_durable_backlog() -> DurableMrfBacklog {
DurableMrfBacklog {
available: true,
entries: vec![
MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: "object-a".to_string(),
version_id: Some(uuid::Uuid::from_u128(7)),
retry_count: 2,
size: 250,
op: MrfOpKind::Object,
target_arns: vec!["arn-a".to_string()],
..Default::default()
},
MrfReplicateEntry {
bucket: "other-bucket".to_string(),
object: "object-b".to_string(),
version_id: None,
retry_count: 0,
size: 999,
op: MrfOpKind::Object,
target_arns: Vec::new(),
..Default::default()
},
],
}
}
/// madmin's `BucketReplicationMRF` decodes the body one `ReplicationMRF`
/// JSON document at a time; the default response must therefore be a bare
/// document stream with madmin's exact json tags, not an envelope.
#[test]
fn mrf_stream_renders_bare_madmin_documents() {
let durable = sample_durable_backlog();
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
let text = String::from_utf8(body).expect("body should be utf-8");
let lines: Vec<&str> = text.lines().filter(|line| !line.trim().is_empty()).collect();
// Only the entry matching the requested bucket is streamed.
assert_eq!(lines.len(), 1, "expected one MRF document, got: {text}");
let doc: serde_json::Value = serde_json::from_str(lines[0]).expect("each line should be a JSON document");
assert_eq!(doc["bucket"], "bucket-a");
assert_eq!(doc["object"], "object-a");
assert_eq!(doc["versionId"], uuid::Uuid::from_u128(7).to_string());
assert_eq!(doc["retryCount"], 2);
// madmin `ReplicationMRF` has a `nodeName` tag; the durable backlog is
// cluster-shared, so RustFS reports an empty node name.
assert_eq!(doc["nodeName"], "");
// The envelope keys must not leak into the stream: a `"Bucket"` key
// would case-insensitively populate `ReplicationMRF.Bucket` and render
// a phantom row in `mc replicate backlog`.
assert!(doc.get("Bucket").is_none());
assert!(doc.get("Targets").is_none());
}
/// An empty backlog must produce an empty body: madmin's decoder loop then
/// terminates on io.EOF with zero rows instead of one phantom row.
#[test]
fn mrf_stream_renders_empty_body_for_no_entries() {
let durable = DurableMrfBacklog {
available: true,
entries: Vec::new(),
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(
body.is_empty(),
"empty backlog must serialize to an empty body, got: {}",
String::from_utf8_lossy(&body)
);
}
/// `?aggregate=true` (RustFS extension) keeps the enveloped counter shape.
#[test]
fn mrf_aggregate_envelope_retains_counters() {
let durable = sample_durable_backlog();
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, _) = render_mrf_backlog(&response, &durable, true).expect("aggregate body should serialize");
let json: serde_json::Value = serde_json::from_slice(&body).expect("aggregate body should be one JSON object");
assert_eq!(json["Bucket"], "bucket-a");
assert_eq!(json["DurableCount"], 1);
assert_eq!(json["DurableBacklogAvailable"], true);
// The bare stream is an enumerable per-object API, so the aggregate
// shell now truthfully advertises it whenever the backlog is readable.
assert_eq!(json["PerObjectEntriesAvailable"], true);
}
/// The nil UUID is RustFS's in-memory null-version sentinel; the wire
/// token is `null`, never the zero UUID (second review round).
#[test]
fn mrf_stream_maps_nil_version_to_null_token() {
let durable = DurableMrfBacklog {
available: true,
entries: vec![MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: "null-version-object".to_string(),
version_id: Some(uuid::Uuid::nil()),
retry_count: 1,
size: 10,
op: MrfOpKind::Object,
..Default::default()
}],
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, truncated) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(!truncated);
let doc: serde_json::Value =
serde_json::from_str(String::from_utf8(body).expect("utf-8").lines().next().expect("one line"))
.expect("line should be a JSON document");
assert_eq!(doc["versionId"], "null");
}
/// The durable ledger is not bounded by the in-memory pending cap; the
/// stream must stop at the documented bound and signal truncation
/// (second review round).
#[test]
fn mrf_stream_truncates_at_the_documented_bound() {
let entries = (0..super::REPLICATION_MRF_MAX_STREAM_ENTRIES + 1)
.map(|index| MrfReplicateEntry {
bucket: "bucket-a".to_string(),
object: format!("object-{index}"),
retry_count: 1,
op: MrfOpKind::Object,
..Default::default()
})
.collect();
let durable = DurableMrfBacklog {
available: true,
entries,
};
let response = build_mrf_response("bucket-a".to_string(), &BucketStats::default(), &durable);
let (body, truncated) = render_mrf_backlog(&response, &durable, false).expect("stream body should serialize");
assert!(truncated, "one entry past the bound must signal truncation");
assert_eq!(
String::from_utf8(body).expect("utf-8").lines().count(),
super::REPLICATION_MRF_MAX_STREAM_ENTRIES
);
let error = super::ensure_complete_mrf_stream(truncated).expect_err("partial streams must not return 200");
assert_eq!(error.code(), &s3s::S3ErrorCode::ServiceUnavailable);
}
#[test]
fn test_extract_query_params_decodes_percent_encoded_values() {
let uri: Uri = "/rustfs/admin/v3/list-remote-targets?bucket=foo%2Fbar&flag=a+b"
File diff suppressed because it is too large Load Diff
@@ -29,6 +29,6 @@ impl Operation for RestLoadCredentialsHandler {
let issuer = IamTableCredentialIssuer::from_request(&req)?;
let response =
load_credentials_response(&store, &warehouse, &namespace, &table, &issuer, Some(&principal.credentials)).await?;
build_json_response(StatusCode::OK, &response)
build_sensitive_json_response(StatusCode::OK, &response)
}
}
File diff suppressed because it is too large Load Diff
@@ -125,7 +125,9 @@ impl Operation for RestLoadTableHandler {
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let response = load_table_response(&store, &metadata_backend, &warehouse, &namespace, &table).await?;
let snapshot_selection = rest_table_snapshot_selection_from_query(&req.uri)?;
let mut response = load_table_response(&store, &metadata_backend, &warehouse, &namespace, &table).await?;
apply_rest_table_snapshot_selection(&mut response.metadata, snapshot_selection);
build_json_response(StatusCode::OK, &response)
}
}
@@ -158,7 +160,7 @@ impl Operation for RestCommitTableHandler {
let principal = authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
install_table_catalog_s3_request_info(&mut req, &principal)?;
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let request = read_json_body::<RestCommitTableRequest>(std::mem::take(&mut req.input)).await?;
let request = read_rest_commit_table_request(std::mem::take(&mut req.input)).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let commit_backend = TableCommitObjectBackend::for_request(metadata_backend, req);
@@ -178,6 +180,14 @@ impl Operation for RestDropTableHandler {
let table = table_name_from_params(&params)?;
let resource = TableCatalogResource::table(&warehouse, &namespace, &table);
authorize_table_catalog_resource_request(&req, &resource, AdminAction::DeleteTableAction).await?;
let purge_requested = rest_purge_requested_from_query(&req.uri)?;
if purge_requested {
return Err(iceberg_rest_error(
ICEBERG_ERROR_UNSUPPORTED_OPERATION,
StatusCode::NOT_ACCEPTABLE,
"purgeRequested=true is not supported",
));
}
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let store = table_catalog_store_from_extensions(&req.extensions)?;
drop_table_in_store(&store, &warehouse, &namespace, &table).await?;
File diff suppressed because it is too large Load Diff
@@ -43,8 +43,9 @@ impl Operation for RestCreateViewHandler {
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let table_bucket_enabled = table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let publication_backend = TableCommitObjectBackend::preauthorized(metadata_backend);
let response =
create_view_response(&store, &metadata_backend, &warehouse, &namespace, request, table_bucket_enabled).await?;
create_view_response(&store, &publication_backend, &warehouse, &namespace, request, table_bucket_enabled).await?;
build_json_response(StatusCode::OK, &response)
}
}
@@ -87,17 +88,20 @@ pub struct RestReplaceViewHandler {}
#[async_trait::async_trait]
impl Operation for RestReplaceViewHandler {
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
async fn call(&self, mut req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
let warehouse = warehouse_from_params(&params)?;
let namespace = namespace_from_params(&params)?;
let view = view_name_from_params(&params)?;
let resource = TableCatalogResource::view(&warehouse, &namespace, &view);
authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
let principal = authorize_table_catalog_resource_request(&req, &resource, AdminAction::CommitTableAction).await?;
install_table_catalog_s3_request_info(&mut req, &principal)?;
ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?;
let request = read_json_body::<RestCommitViewRequest>(req.input).await?;
let request = read_rest_commit_view_request(std::mem::take(&mut req.input)).await?;
let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?;
let store = table_catalog_store_from_backend(metadata_backend.clone())?;
let response = replace_view_response(&store, &metadata_backend, &warehouse, &namespace, &view, request).await?;
let commit_backend = TableCommitObjectBackend::for_request(metadata_backend, req);
let result = replace_view_response(&store, &commit_backend, &warehouse, &namespace, &view, request).await;
let response = commit_backend.finish(result).await?;
build_json_response(StatusCode::OK, &response)
}
}
+1
View File
@@ -17,6 +17,7 @@ mod auth;
pub mod console;
pub mod handlers;
mod plugin_contract;
pub(crate) mod replication_metrics_wire;
// Contract inventory is validated by tests before later runtime integration.
#[allow(dead_code)]
pub(crate) mod route_policy;
@@ -0,0 +1,673 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Serialize-only wire projections of the internal replication statistics
//! onto the minio-go `replication.Metrics` / `replication.MetricsV2` json
//! shapes consumed by `mc replicate status` (`?replication-metrics[=2]` and
//! the admin `replicationmetrics` endpoint).
//!
//! Red line: the internal `BucketStats` family in
//! `crates/replication/src/stats.rs` is ALSO the intra-cluster peer-RPC wire
//! format — `node_service.rs` encodes it with `rmp_serde::to_vec_named`, so
//! its Rust field names travel between nodes as msgpack map keys. Renaming
//! those serde names would break mixed-version clusters mid rolling upgrade.
//! All madmin/minio-go interop therefore happens in these DTOs; never add
//! `#[serde(rename)]` to the internal structs instead.
//!
//! Field names below are the exact json tags of minio-go
//! `pkg/replication/replication.go` (v7.0.91). Keys minio-go does not know
//! are RustFS extensions; Go decoders ignore unknown keys. `max`/`peak` are
//! both emitted for the queue peak because the MinIO server writes `max`
//! while minio-go reads `peak` (an upstream drift); emitting both keeps every
//! decoder working.
use serde::Serialize;
use std::collections::HashMap;
use std::time::Duration;
use crate::admin::storage_api::replication::{
BucketReplicationStat as InternalReplicationStat, BucketReplicationStats as InternalReplicationStats, BucketStats,
InQueueMetric as InternalInQueueMetric, XferStats as InternalXferStats,
};
/// minio-go `replication.RStat`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct RStatWire {
#[serde(rename = "count")]
pub count: f64,
#[serde(rename = "bytes")]
pub bytes: i64,
}
/// minio-go `replication.TimedErrStats`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct TimedErrStatsWire {
#[serde(rename = "lastMinute")]
pub last_minute: RStatWire,
#[serde(rename = "lastHour")]
pub last_hour: RStatWire,
#[serde(rename = "totals")]
pub totals: RStatWire,
}
impl TimedErrStatsWire {
fn add(self, other: TimedErrStatsWire) -> TimedErrStatsWire {
fn add(a: RStatWire, b: RStatWire) -> RStatWire {
RStatWire {
count: a.count + b.count,
bytes: a.bytes.saturating_add(b.bytes),
}
}
TimedErrStatsWire {
last_minute: add(self.last_minute, other.last_minute),
last_hour: add(self.last_hour, other.last_hour),
totals: add(self.totals, other.totals),
}
}
}
/// minio-go `replication.QStat`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct QStatWire {
#[serde(rename = "count")]
pub count: f64,
#[serde(rename = "bytes")]
pub bytes: f64,
}
/// minio-go `replication.InQueueMetric`, with the queue peak emitted under
/// both `peak` (minio-go tag) and `max` (MinIO server tag).
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct InQueueMetricWire {
#[serde(rename = "curr")]
pub curr: QStatWire,
#[serde(rename = "avg")]
pub avg: QStatWire,
#[serde(rename = "max")]
pub max: QStatWire,
#[serde(rename = "peak")]
pub peak: QStatWire,
}
impl From<&InternalInQueueMetric> for InQueueMetricWire {
fn from(metric: &InternalInQueueMetric) -> Self {
fn qstat(bytes: i64, count: i64) -> QStatWire {
QStatWire {
count: count as f64,
bytes: bytes as f64,
}
}
let peak = qstat(metric.max.bytes, metric.max.count);
InQueueMetricWire {
curr: qstat(metric.curr.bytes, metric.curr.count),
avg: qstat(metric.avg.bytes, metric.avg.count),
max: peak,
peak,
}
}
}
/// minio-go `replication.XferStats`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct XferStatsWire {
#[serde(rename = "avgRate")]
pub avg_rate: f64,
#[serde(rename = "peakRate")]
pub peak_rate: f64,
#[serde(rename = "currRate")]
pub curr_rate: f64,
}
#[derive(Default)]
struct XferStatsAverage {
sum: XferStatsWire,
active: u32,
}
impl XferStatsAverage {
fn add_active(&mut self, stats: XferStatsWire) {
if stats.peak_rate <= 0.0 {
return;
}
self.add_raw(stats);
self.active += 1;
}
fn add_raw(&mut self, stats: XferStatsWire) {
self.sum.avg_rate += stats.avg_rate;
self.sum.curr_rate += stats.curr_rate;
self.sum.peak_rate = self.sum.peak_rate.max(stats.peak_rate);
}
fn finish(self) -> XferStatsWire {
let active = self.active;
self.finish_with_divisor(active)
}
fn finish_with_divisor(self, divisor: u32) -> XferStatsWire {
if divisor == 0 {
return self.sum;
}
XferStatsWire {
avg_rate: self.sum.avg_rate / f64::from(divisor),
peak_rate: self.sum.peak_rate,
curr_rate: self.sum.curr_rate / f64::from(divisor),
}
}
}
impl From<&InternalXferStats> for XferStatsWire {
fn from(stats: &InternalXferStats) -> Self {
XferStatsWire {
avg_rate: stats.avg,
peak_rate: stats.peak,
curr_rate: stats.curr,
}
}
}
/// minio-go `replication.WorkerStat`. RustFS does not track per-bucket worker
/// occupancy yet, so this always reports zeros.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct WorkerStatWire {
#[serde(rename = "curr")]
pub curr: i32,
#[serde(rename = "avg")]
pub avg: f32,
#[serde(rename = "max")]
pub max: i32,
}
/// minio-go `replication.ReplMRFStats`. RustFS does not track the 5-minute /
/// dropped MRF windows, so this always reports zeros; the durable backlog is
/// enumerable via `/v3/replication/mrf` instead.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct ReplMrfStatsWire {
#[serde(rename = "failedCount_last5min")]
pub last_failed_count: u64,
#[serde(rename = "droppedCount_since_uptime")]
pub total_dropped_count: u64,
#[serde(rename = "droppedBytes_since_uptime")]
pub total_dropped_bytes: u64,
}
/// minio-go `replication.CounterSummary`.
#[derive(Debug, Default, Clone, Copy, Serialize)]
pub(crate) struct CounterSummaryWire {
#[serde(rename = "last1hr")]
pub last1hr: u64,
#[serde(rename = "last1m")]
pub last1m: u64,
#[serde(rename = "total")]
pub total: u64,
}
/// minio-go `replication.TargetMetrics` (one remote target / ARN).
#[derive(Debug, Default, Serialize)]
pub(crate) struct TargetMetricsWire {
#[serde(rename = "replicationCount")]
pub replicated_count: i64,
#[serde(rename = "completedReplicationSize")]
pub replicated_size: i64,
/// Bandwidth limit for this target. The tag says "bits" but both MinIO
/// and minio-go treat the value as bytes/sec; keep bytes/sec.
#[serde(rename = "limitInBits")]
pub bandwidth_limit_bytes_per_sec: i64,
#[serde(rename = "currentBandwidth")]
pub current_bandwidth_bytes_per_sec: f64,
#[serde(rename = "failed")]
pub failed: TimedErrStatsWire,
#[serde(rename = "failedReplicationSize")]
pub failed_size: i64,
#[serde(rename = "failedReplicationCount")]
pub failed_count: i64,
}
fn target_timed_err_stats(stat: &InternalReplicationStat) -> TimedErrStatsWire {
// Cluster aggregation merges FailStats without the process-local samples,
// so the serializable window snapshots (refreshed at each node's
// collection point, summed by merge) are authoritative here; the live
// samples only ever agree with or lag them, so take the larger.
let sampled_minute = stat.fail_stats.recent_since(Duration::from_secs(60));
let sampled_hour = stat.fail_stats.recent_since(Duration::from_secs(3600));
let window = |sampled_count: i64, sampled_size: i64, snapshot_count: i64, snapshot_size: i64| RStatWire {
count: sampled_count.max(snapshot_count) as f64,
bytes: sampled_size.max(snapshot_size),
};
TimedErrStatsWire {
last_minute: window(
sampled_minute.count,
sampled_minute.size,
stat.fail_stats.last_minute.count,
stat.fail_stats.last_minute.size,
),
last_hour: window(
sampled_hour.count,
sampled_hour.size,
stat.fail_stats.last_hour.count,
stat.fail_stats.last_hour.size,
),
totals: RStatWire {
count: stat.failed.count as f64,
bytes: stat.failed.size,
},
}
}
impl From<&InternalReplicationStat> for TargetMetricsWire {
fn from(stat: &InternalReplicationStat) -> Self {
TargetMetricsWire {
replicated_count: stat.replicated_count,
replicated_size: stat.replicated_size,
bandwidth_limit_bytes_per_sec: stat.bandwidth_limit_bytes_per_sec,
current_bandwidth_bytes_per_sec: stat.current_bandwidth_bytes_per_sec,
failed: target_timed_err_stats(stat),
failed_size: stat.failed.size,
failed_count: stat.failed.count,
}
}
}
/// minio-go `replication.Metrics` — the `currStats` member of `MetricsV2` and
/// the whole v1 response body. The trailing snake_case fields are RustFS
/// source-health extension keys (ignored by Go decoders) carried over from
/// the previous response shape.
#[derive(Debug, Default, Serialize)]
pub(crate) struct MetricsWire {
#[serde(rename = "Stats")]
pub stats: HashMap<String, TargetMetricsWire>,
#[serde(rename = "completedReplicationSize")]
pub replicated_size: i64,
#[serde(rename = "replicaSize")]
pub replica_size: i64,
#[serde(rename = "replicaCount")]
pub replica_count: i64,
#[serde(rename = "replicationCount")]
pub replicated_count: i64,
#[serde(rename = "failed")]
pub failed: TimedErrStatsWire,
#[serde(rename = "queued")]
pub queued: InQueueMetricWire,
// RustFS extension keys (source health of the aggregation).
pub provider_available: bool,
pub cluster_complete: bool,
pub observed_node_count: u32,
pub expected_node_count: u32,
}
impl From<&InternalReplicationStats> for MetricsWire {
fn from(stats: &InternalReplicationStats) -> Self {
let mut failed = TimedErrStatsWire::default();
let mut targets = HashMap::with_capacity(stats.stats.len());
for (arn, stat) in &stats.stats {
let target = TargetMetricsWire::from(stat);
failed = failed.add(target.failed);
targets.insert(arn.clone(), target);
}
MetricsWire {
stats: targets,
replicated_size: stats.replicated_size,
replica_size: stats.replica_size,
replica_count: stats.replica_count,
replicated_count: stats.replicated_count,
failed,
queued: InQueueMetricWire::from(&stats.q_stat),
provider_available: stats.provider_available,
cluster_complete: stats.cluster_complete,
observed_node_count: stats.observed_node_count,
expected_node_count: stats.expected_node_count,
}
}
}
/// minio-go `replication.ReplQNodeStats`.
#[derive(Debug, Default, Serialize)]
pub(crate) struct ReplQNodeStatsWire {
#[serde(rename = "nodeName")]
pub node_name: String,
#[serde(rename = "uptime")]
pub uptime: i64,
#[serde(rename = "activeWorkers")]
pub workers: WorkerStatWire,
#[serde(rename = "transferSummary")]
pub xfer_stats: XferSummaryWire,
#[serde(rename = "tgtTransferStats")]
pub tgt_xfer_stats: TargetXferSummaryWire,
#[serde(rename = "queueStats")]
pub q_stats: InQueueMetricWire,
#[serde(rename = "mrfStats")]
pub mrf_stats: ReplMrfStatsWire,
#[serde(rename = "retries")]
pub retries: CounterSummaryWire,
#[serde(rename = "errors")]
pub errors: CounterSummaryWire,
}
/// minio-go `replication.ReplQueueStats`.
#[derive(Debug, Default, Serialize)]
pub(crate) struct ReplQueueStatsWire {
#[serde(rename = "nodes")]
pub nodes: Vec<ReplQNodeStatsWire>,
}
/// minio-go `replication.MetricsV2` — the `?replication-metrics=2` body.
#[derive(Debug, Default, Serialize)]
pub(crate) struct MetricsV2Wire {
#[serde(rename = "uptime")]
pub uptime: i64,
#[serde(rename = "currStats")]
pub current_stats: MetricsWire,
#[serde(rename = "queueStats")]
pub queue_stats: ReplQueueStatsWire,
#[serde(rename = "downtimeInfo")]
pub downtime_info: HashMap<String, serde_json::Value>,
}
/// `transferSummary` map keyed by minio-go `MetricName` (Large/Small/Total).
type XferSummaryWire = HashMap<&'static str, XferStatsWire>;
/// `tgtTransferStats` map keyed by target ARN.
type TargetXferSummaryWire = HashMap<String, XferSummaryWire>;
fn transfer_summaries(stats: &InternalReplicationStats) -> (XferSummaryWire, TargetXferSummaryWire) {
let mut per_target: TargetXferSummaryWire = HashMap::new();
let mut large_summary = XferStatsAverage::default();
let mut small_summary = XferStatsAverage::default();
let mut total_summary = XferStatsAverage::default();
let mut active_targets = 0;
for (arn, stat) in &stats.stats {
let large = XferStatsWire::from(&stat.xfer_rate_lrg);
let small = XferStatsWire::from(&stat.xfer_rate_sml);
let mut target_total = XferStatsAverage::default();
target_total.add_active(large);
target_total.add_active(small);
let total = target_total.finish();
per_target.insert(arn.clone(), HashMap::from([("Large", large), ("Small", small), ("Total", total)]));
if large.peak_rate > 0.0 || small.peak_rate > 0.0 {
active_targets += 1;
large_summary.add_raw(large);
small_summary.add_raw(small);
total_summary.add_raw(large);
total_summary.add_raw(small);
}
}
let summary = HashMap::from([
("Large", large_summary.finish_with_divisor(active_targets)),
("Small", small_summary.finish_with_divisor(active_targets)),
("Total", total_summary.finish_with_divisor(active_targets)),
]);
(summary, per_target)
}
impl MetricsV2Wire {
/// Project the aggregated internal stats onto the `MetricsV2` shape.
///
/// The aggregation path leaves `queue_stats.nodes` empty today, so a
/// single node entry is synthesized from the bucket queue snapshot —
/// `mc replicate status` derives its queue/worker panels from
/// `queueStats.nodes` and treats an empty list as "no data".
pub(crate) fn from_stats(bucket_stats: &BucketStats, node_name: &str) -> Self {
let (xfer_stats, tgt_xfer_stats) = transfer_summaries(&bucket_stats.replication_stats);
let mut nodes: Vec<ReplQNodeStatsWire> = bucket_stats
.queue_stats
.nodes
.iter()
.map(|node| ReplQNodeStatsWire {
node_name: node_name.to_string(),
uptime: bucket_stats.uptime,
q_stats: InQueueMetricWire::from(&node.q_stats),
..Default::default()
})
.collect();
if nodes.is_empty() {
nodes.push(ReplQNodeStatsWire {
node_name: node_name.to_string(),
uptime: bucket_stats.uptime,
q_stats: InQueueMetricWire::from(&bucket_stats.replication_stats.q_stat),
xfer_stats: xfer_stats.clone(),
tgt_xfer_stats: tgt_xfer_stats.clone(),
..Default::default()
});
} else {
// Attach the transfer summaries to the first node; the internal
// snapshot does not attribute transfer rates per node.
if let Some(first) = nodes.first_mut() {
first.xfer_stats = xfer_stats.clone();
first.tgt_xfer_stats = tgt_xfer_stats.clone();
}
}
MetricsV2Wire {
uptime: bucket_stats.uptime,
current_stats: MetricsWire::from(&bucket_stats.replication_stats),
queue_stats: ReplQueueStatsWire { nodes },
downtime_info: HashMap::new(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn sample_bucket_stats() -> BucketStats {
let mut stats = BucketStats {
uptime: 42,
..Default::default()
};
stats.replication_stats.replica_count = 2;
stats.replication_stats.replica_size = 128;
stats.replication_stats.replicated_count = 9;
stats.replication_stats.replicated_size = 4096;
let target = stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default();
target.replicated_count = 9;
target.replicated_size = 4096;
target.failed.count = 3;
target.failed.size = 900;
target.bandwidth_limit_bytes_per_sec = 1024;
target.current_bandwidth_bytes_per_sec = 512.5;
stats
.replication_stats
.q_stat
.curr
.now_count
.store(4, std::sync::atomic::Ordering::Relaxed);
stats
.replication_stats
.q_stat
.curr
.now_bytes
.store(1200, std::sync::atomic::Ordering::Relaxed);
stats.replication_stats.q_stat = stats.replication_stats.q_stat.snapshot();
stats
}
#[test]
fn metrics_wire_matches_minio_go_tags() {
let stats = sample_bucket_stats();
let json = serde_json::to_value(MetricsWire::from(&stats.replication_stats)).expect("v1 wire should serialize");
assert_eq!(json["replicaCount"], 2);
assert_eq!(json["replicaSize"], 128);
assert_eq!(json["replicationCount"], 9);
assert_eq!(json["completedReplicationSize"], 4096);
assert_eq!(json["queued"]["curr"]["count"], 4.0);
assert_eq!(json["queued"]["curr"]["bytes"], 1200.0);
let target = &json["Stats"]["arn:minio:replication::t:b"];
assert_eq!(target["replicationCount"], 9);
assert_eq!(target["completedReplicationSize"], 4096);
assert_eq!(target["limitInBits"], 1024);
assert_eq!(target["currentBandwidth"], 512.5);
// failed is the madmin TimedErrStats envelope, not the internal
// {count,size} pair.
assert_eq!(target["failed"]["totals"]["count"], 3.0);
assert_eq!(target["failed"]["totals"]["bytes"], 900);
assert!(target["failed"].get("count").is_none());
// Aggregate failed mirrors the per-target totals.
assert_eq!(json["failed"]["totals"]["count"], 3.0);
}
#[test]
fn metrics_v2_wire_synthesizes_queue_node() {
let stats = sample_bucket_stats();
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1:9000")).expect("v2 wire should serialize");
assert_eq!(json["uptime"], 42);
assert_eq!(json["currStats"]["replicaCount"], 2);
let node = &json["queueStats"]["nodes"][0];
assert_eq!(node["nodeName"], "node-1:9000");
assert_eq!(node["uptime"], 42);
assert_eq!(node["queueStats"]["curr"]["count"], 4.0);
// The queue peak is emitted under both the minio-go tag (`peak`) and
// the MinIO server tag (`max`).
assert_eq!(node["queueStats"]["peak"], node["queueStats"]["max"]);
assert!(node["activeWorkers"].get("curr").is_some());
assert!(node["transferSummary"].get("Total").is_some());
assert_eq!(json["downtimeInfo"], serde_json::json!({}));
}
/// minio-go's transferSummary labels mean >= 128 MiB for Large; the
/// producer must bin on the same boundary (MIN_LARGE_OBJ_SIZE, shared
/// with the worker-pool split), or a 2 MiB replication shows under Large
/// while Small stays zero.
#[test]
fn transfer_summary_bins_on_the_128_mib_boundary() {
const MIB: i64 = 1024 * 1024;
let mut stats = BucketStats::default();
let stat = stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default();
stat.update_xfer_rate(2 * MIB, std::time::Duration::from_secs(1));
stat.update_xfer_rate(127 * MIB, std::time::Duration::from_secs(1));
stat.update_xfer_rate(128 * MIB, std::time::Duration::from_secs(1));
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1")).expect("v2 wire should serialize");
let summary = &json["queueStats"]["nodes"][0]["tgtTransferStats"]["arn:minio:replication::t:b"];
let small_peak = summary["Small"]["peakRate"].as_f64().expect("Small peakRate");
let large_peak = summary["Large"]["peakRate"].as_f64().expect("Large peakRate");
assert!(
(small_peak - (127 * MIB) as f64).abs() < 1.0,
"2 MiB and 127 MiB transfers must bin as Small (peak {small_peak})"
);
assert!(
(large_peak - (128 * MIB) as f64).abs() < 1.0,
"exactly 128 MiB must bin as Large (peak {large_peak})"
);
}
#[test]
fn transfer_summaries_average_active_bins_and_targets() {
let mut stats = BucketStats::default();
let first = stats.replication_stats.stats.entry("target-a".to_string()).or_default();
first.xfer_rate_sml.avg = 50.0;
first.xfer_rate_sml.curr = 40.0;
first.xfer_rate_sml.peak = 60.0;
first.xfer_rate_lrg.avg = 100.0;
first.xfer_rate_lrg.curr = 80.0;
first.xfer_rate_lrg.peak = 120.0;
let second = stats.replication_stats.stats.entry("target-b".to_string()).or_default();
second.xfer_rate_sml.avg = 30.0;
second.xfer_rate_sml.curr = 20.0;
second.xfer_rate_sml.peak = 40.0;
let json = serde_json::to_value(MetricsV2Wire::from_stats(&stats, "node-1")).expect("v2 wire should serialize");
let node = &json["queueStats"]["nodes"][0];
let target_a = &node["tgtTransferStats"]["target-a"]["Total"];
assert_eq!(target_a["avgRate"], 75.0);
assert_eq!(target_a["currRate"], 60.0);
assert_eq!(target_a["peakRate"], 120.0);
let summary = &node["transferSummary"];
assert_eq!(summary["Small"]["avgRate"], 40.0);
assert_eq!(summary["Small"]["currRate"], 30.0);
assert_eq!(summary["Large"]["avgRate"], 50.0);
assert_eq!(summary["Total"]["avgRate"], 90.0);
assert_eq!(summary["Total"]["currRate"], 70.0);
assert_eq!(summary["Total"]["peakRate"], 120.0);
}
/// Review regression: both metrics endpoints aggregate first, and the
/// FailStats merge drops the process-local samples — the rolling windows
/// must survive a peer-RPC round trip plus aggregation and still reach
/// the wire body.
#[test]
fn failure_windows_survive_aggregation_before_serialization() {
// Node A: live failure; the windows are stamped at the collection
// point (get_latest_replication_stats calls refresh_windows before
// the stats cross the wire), never on the failure hot path.
let mut node_a = crate::admin::storage_api::replication::BucketReplicationStat::default();
node_a.fail_stats.add_size(512, None::<&std::io::Error>);
node_a.fail_stats.refresh_windows();
node_a.failed = node_a.fail_stats.to_metric();
// Node A's stats cross the peer RPC wire: the samples are dropped,
// the window snapshots travel.
let encoded = rmp_serde::to_vec_named(&node_a).expect("stat should encode");
let remote: crate::admin::storage_api::replication::BucketReplicationStat =
rmp_serde::from_slice(&encoded).expect("stat should decode");
// Aggregation merges the remote stat with an empty local one.
let merged_fail = remote.fail_stats.merge(&Default::default());
let aggregated = crate::admin::storage_api::replication::BucketReplicationStat {
failed: merged_fail.to_metric(),
fail_stats: merged_fail,
..Default::default()
};
let mut stats = BucketStats::default();
stats
.replication_stats
.stats
.insert("arn:minio:replication::t:b".to_string(), aggregated);
let json = serde_json::to_value(MetricsWire::from(&stats.replication_stats)).expect("wire should serialize");
let failed = &json["Stats"]["arn:minio:replication::t:b"]["failed"];
assert_eq!(failed["totals"]["count"], 1.0);
assert_eq!(
failed["lastMinute"]["count"], 1.0,
"the rolling minute window must survive RPC + aggregation"
);
assert_eq!(failed["lastMinute"]["bytes"], 512);
assert_eq!(failed["lastHour"]["count"], 1.0);
}
/// Pin the intra-cluster peer-RPC wire format of the internal stats: it
/// is msgpack with the Rust field names as map keys
/// (`rmp_serde::to_vec_named` in node_service.rs). If someone "fixes"
/// the interop bug by renaming the internal serde fields instead of using
/// these DTOs, this test fails and points them here.
#[test]
fn internal_bucket_stats_rpc_wire_stays_snake_case() {
let stats = sample_bucket_stats();
let encoded = rmp_serde::to_vec_named(&stats).expect("internal stats should encode");
let value: serde_json::Value = rmp_serde::from_slice(&encoded).expect("named msgpack should decode generically");
assert!(
value.get("replication_stats").is_some(),
"peer RPC key replication_stats must not be renamed"
);
assert!(value["replication_stats"].get("q_stat").is_some());
assert!(value.get("queue_stats").is_some());
assert!(value.get("proxy_stats").is_some());
let decoded: BucketStats = rmp_serde::from_slice(&encoded).expect("round-trip through the peer RPC wire");
assert_eq!(decoded.replication_stats.replica_count, 2);
}
}
+4 -1
View File
@@ -1459,10 +1459,13 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
REPLICATION_DIFF,
RouteRiskLevel::Sensitive,
),
// The default stream enumerates object names/version ids and requires
// ReplicationDiff (MinIO parity); only ?aggregate=true relaxes to
// GetReplicationMetrics in the handler.
admin(
HttpMethod::Get,
"/rustfs/admin/v3/replication/mrf",
GET_REPLICATION_METRICS,
REPLICATION_DIFF,
RouteRiskLevel::Sensitive,
),
];
+67 -13
View File
@@ -1548,7 +1548,8 @@ async fn build_replication_metrics_response(
let bucket_stats = apply_replication_metrics_bandwidth_report(bucket_stats, collect_replication_metrics_bandwidth(bucket));
let bucket_stats = apply_replication_metrics_runtime_fields(bucket_stats, route, replication_metrics_uptime_seconds());
let body = serialize_replication_metrics_body(&bucket_stats, route)?;
let node_name = crate::runtime_sources::current_local_node_name().await.unwrap_or_default();
let body = serialize_replication_metrics_body(&bucket_stats, route, &node_name)?;
let mut resp = S3Response::with_status(Body::from(body), StatusCode::OK);
resp.headers
@@ -1608,12 +1609,24 @@ fn apply_replication_metrics_runtime_fields(
bucket_stats
}
fn serialize_replication_metrics_body(bucket_stats: &BucketStats, route: ReplicationExtRoute) -> S3Result<Vec<u8>> {
/// Serialize the metrics body in the minio-go wire shapes
/// (`replication.Metrics` for v1, `replication.MetricsV2` for v2). The
/// internal `BucketStats` serde names are the intra-cluster peer-RPC wire
/// format and must never appear here — see
/// `crate::admin::replication_metrics_wire`.
fn serialize_replication_metrics_body(
bucket_stats: &BucketStats,
route: ReplicationExtRoute,
node_name: &str,
) -> S3Result<Vec<u8>> {
use crate::admin::replication_metrics_wire::{MetricsV2Wire, MetricsWire};
match route {
ReplicationExtRoute::MetricsV1 => {
serde_json::to_vec(&bucket_stats.replication_stats).map_err(|e| s3_error!(InternalError, "{e}"))
serde_json::to_vec(&MetricsWire::from(&bucket_stats.replication_stats)).map_err(|e| s3_error!(InternalError, "{e}"))
}
ReplicationExtRoute::MetricsV2 => {
serde_json::to_vec(&MetricsV2Wire::from_stats(bucket_stats, node_name)).map_err(|e| s3_error!(InternalError, "{e}"))
}
ReplicationExtRoute::MetricsV2 => serde_json::to_vec(bucket_stats).map_err(|e| s3_error!(InternalError, "{e}")),
ReplicationExtRoute::Check | ReplicationExtRoute::ResetStart | ReplicationExtRoute::ResetStatus => {
Err(s3_error!(InternalError, "invalid route for metrics response"))
}
@@ -4147,22 +4160,37 @@ mod tests {
assert!(err.message().unwrap_or_default().contains("rule-stale"));
}
/// The v1 body must decode into minio-go `replication.Metrics` (exact
/// json tags); Go's decoder matches case-insensitively but does not
/// ignore underscores, so the internal snake_case names read as all-zero.
#[test]
fn serialize_replication_metrics_body_v1_returns_replication_stats_only() {
fn serialize_replication_metrics_body_v1_returns_minio_go_metrics_shape() {
let mut stats = BucketStats {
uptime: 99,
..Default::default()
};
stats.replication_stats.replica_count = 7;
stats.replication_stats.replicated_size = 2048;
stats
.replication_stats
.stats
.entry("arn:minio:replication::t:b".to_string())
.or_default()
.replicated_count = 5;
stats.proxy_stats.put_total = 3;
let body =
serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV1).expect("metrics v1 body should serialize");
let body = serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV1, "node-1:9000")
.expect("metrics v1 body should serialize");
let payload: serde_json::Value = serde_json::from_slice(&body).expect("body should be json");
assert_eq!(payload["replica_count"], 7);
assert_eq!(payload["replicaCount"], 7);
assert_eq!(payload["completedReplicationSize"], 2048);
assert_eq!(payload["Stats"]["arn:minio:replication::t:b"]["replicationCount"], 5);
assert!(payload.get("uptime").is_none());
assert!(payload.get("proxy_stats").is_none());
// The internal snake_case names must not leak into the wire body.
assert!(payload.get("replica_count").is_none());
assert!(payload.get("q_stat").is_none());
}
#[test]
@@ -4248,22 +4276,48 @@ mod tests {
assert_eq!(target.current_bandwidth_bytes_per_sec, 3000.0);
}
/// The v2 body must decode into minio-go `replication.MetricsV2`
/// (`uptime`/`currStats`/`queueStats`); `mc replicate status` reads
/// `currStats` and `queueStats.nodes` and silently shows zeros when the
/// keys do not match.
#[test]
fn serialize_replication_metrics_body_v2_returns_full_bucket_stats() {
fn serialize_replication_metrics_body_v2_returns_minio_go_metrics_v2_shape() {
let mut stats = BucketStats {
uptime: 99,
..Default::default()
};
stats.replication_stats.replica_count = 7;
stats
.replication_stats
.q_stat
.curr
.now_count
.store(4, std::sync::atomic::Ordering::Relaxed);
stats
.replication_stats
.q_stat
.curr
.now_bytes
.store(1200, std::sync::atomic::Ordering::Relaxed);
stats.replication_stats.q_stat = stats.replication_stats.q_stat.snapshot();
stats.proxy_stats.put_total = 3;
let body =
serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV2).expect("metrics v2 body should serialize");
let body = serialize_replication_metrics_body(&stats, ReplicationExtRoute::MetricsV2, "node-1:9000")
.expect("metrics v2 body should serialize");
let payload: serde_json::Value = serde_json::from_slice(&body).expect("body should be json");
assert_eq!(payload["uptime"], 99);
assert_eq!(payload["replication_stats"]["replica_count"], 7);
assert_eq!(payload["proxy_stats"]["put_total"], 3);
assert_eq!(payload["currStats"]["replicaCount"], 7);
assert_eq!(payload["currStats"]["queued"]["curr"]["count"], 4.0);
// The queue snapshot must surface at least one node: mc derives the
// worker/queue panels from queueStats.nodes and treats an empty list
// as "no data".
assert_eq!(payload["queueStats"]["nodes"][0]["queueStats"]["curr"]["count"], 4.0);
assert_eq!(payload["queueStats"]["nodes"][0]["uptime"], 99);
// The internal snake_case names must not leak into the wire body.
assert!(payload.get("replication_stats").is_none());
assert!(payload.get("queue_stats").is_none());
assert!(payload.get("proxy_stats").is_none());
}
#[test]
+32
View File
@@ -321,6 +321,34 @@ pub(crate) mod metadata_sys {
crate::storage::storage_api::acquire_bucket_metadata_transaction_lock(bucket).await
}
pub(crate) async fn acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket: &str,
expected_incarnation_id: uuid::Uuid,
) -> Result<super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard> {
super::ecstore_bucket::metadata_sys::acquire_bucket_metadata_transaction_lock_for_incarnation(
bucket,
expected_incarnation_id,
)
.await
}
pub(crate) async fn update_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
data: Vec<u8>,
) -> Result<OffsetDateTime> {
super::ecstore_bucket::metadata_sys::update_under_transaction_lock(guard, bucket, config_file, data).await
}
pub(crate) async fn delete_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
config_file: &str,
) -> Result<OffsetDateTime> {
super::ecstore_bucket::metadata_sys::delete_under_transaction_lock(guard, bucket, config_file).await
}
pub(crate) async fn update_bucket_targets_under_transaction_lock(
guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
bucket: &str,
@@ -417,6 +445,10 @@ pub(crate) mod replication {
};
pub(crate) type BucketReplicationResyncStatus = super::ecstore_bucket::replication::BucketReplicationResyncStatus;
pub(crate) type BucketStats = super::ecstore_bucket::replication::BucketStats;
pub(crate) type BucketReplicationStats = super::ecstore_bucket::replication::BucketReplicationStats;
pub(crate) type BucketReplicationStat = super::ecstore_bucket::replication::BucketReplicationStat;
pub(crate) type InQueueMetric = super::ecstore_bucket::replication::InQueueMetric;
pub(crate) type XferStats = super::ecstore_bucket::replication::XferStats;
pub(crate) type ReplicationStatusType = super::ecstore_bucket::replication::ReplicationStatusType;
pub(crate) type ResyncOpts = super::ecstore_bucket::replication::ResyncOpts;
pub(crate) type ResyncStatusType = super::ecstore_bucket::replication::ResyncStatusType;
+39 -2
View File
@@ -1158,6 +1158,19 @@ fn lifecycle_has_expiry_rules(config: &BucketLifecycleConfiguration) -> bool {
})
}
/// Status-independent presence of the expiry subset that site replication
/// propagates (`replicateILMExpiry`): expiration / noncurrent-version
/// expiration only. Distinct from [`lifecycle_has_expiry_rules`], which
/// filters on ENABLED for scanner scheduling — editing a Disabled expiry rule
/// must still advance the replication axis. Del-marker expiration and
/// abort-multipart are site-local and never travel.
fn lifecycle_rules_have_expiry(config: &BucketLifecycleConfiguration) -> bool {
config
.rules
.iter()
.any(|rule| rule.expiration.is_some() || rule.noncurrent_version_expiration.is_some())
}
fn lifecycle_has_abort_multipart_rules(config: &BucketLifecycleConfiguration) -> bool {
config.rules.iter().any(|rule| {
rule.status == ExpirationStatus::from_static(ExpirationStatus::ENABLED)
@@ -2186,7 +2199,24 @@ impl DefaultBucketUsecase {
return Err(s3_error!(InvalidArgument, "{err}"));
}
input_cfg.expiry_updated_at = Some(Timestamp::from(time::OffsetDateTime::now_utc()));
// Stamp the expiry axis only when the expiry subset can have changed
// (MinIO: HasExpiry() || expiryRuleRemoved). Site-replication peers
// judge lc-config staleness on this axis; a transition-only edit that
// advanced it would let this site's stale expiry subset shadow — and
// roll back — a newer peer expiry edit fleet-wide.
let previous_expiry_updated_at = match metadata_sys::get_lifecycle_config(&bucket).await {
Ok((previous, _)) => {
if lifecycle_rules_have_expiry(&input_cfg) || lifecycle_rules_have_expiry(&previous) {
Some(Timestamp::from(time::OffsetDateTime::now_utc()))
} else {
previous.expiry_updated_at
}
}
// No previous config (or unreadable): stamping is the
// conservative pre-existing behavior.
Err(_) => lifecycle_rules_have_expiry(&input_cfg).then(|| Timestamp::from(time::OffsetDateTime::now_utc())),
};
input_cfg.expiry_updated_at = previous_expiry_updated_at;
let data = serialize_config(&input_cfg)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_LIFECYCLE_CONFIG, data, expected_incarnation_id)
.await
@@ -2197,7 +2227,14 @@ impl DefaultBucketUsecase {
let mut item = sr_bucket_meta_item(bucket.clone(), "lc-config");
item.expiry_lc_config =
Some(serialize_config(&input_cfg).and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?);
item.expiry_updated_at = item.updated_at;
// The item travels with the expiry axis, not the wall clock: a site
// whose expiry knowledge is old (or absent — UNIX_EPOCH) must not
// out-rank newer peer expiry state at the receivers.
item.expiry_updated_at = input_cfg
.expiry_updated_at
.clone()
.map(time::OffsetDateTime::from)
.or(Some(time::OffsetDateTime::UNIX_EPOCH));
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket lifecycle hook failed");
}
+5 -3
View File
@@ -44,8 +44,8 @@ use super::storage_api::multipart_usecase::io::{HashReader, WriteEncryption, Wri
use super::storage_api::multipart_usecase::object_utils::to_s3s_etag;
use super::storage_api::multipart_usecase::options::{
copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts_with_replication_authorization,
get_content_sha256_with_query, get_opts, namespace_reserved_user_metadata, parse_copy_source_range,
put_opts_with_replication_authorization, validate_archive_content_encoding,
get_content_sha256_with_query, get_opts, has_replication_retention_update, namespace_reserved_user_metadata,
parse_copy_source_range, put_opts_with_replication_authorization, validate_archive_content_encoding,
};
use super::storage_api::multipart_usecase::request_context::spawn_traced_join;
use super::storage_api::multipart_usecase::s3_api::multipart::{
@@ -777,7 +777,9 @@ impl DefaultMultipartUsecase {
let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
|| has_replication_retention_update(&req.headers, replication_authorized);
let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata(
&bucket,
+18 -3
View File
@@ -86,8 +86,8 @@ use super::storage_api::object_usecase::object_utils::to_s3s_etag;
use super::storage_api::object_usecase::options::{
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_content_sha256_with_query, get_opts,
namespace_reserved_user_metadata, normalize_content_encoding_for_storage, preserve_unclassified_user_metadata,
put_opts_with_replication_authorization, validate_archive_content_encoding,
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
};
use super::storage_api::object_usecase::request_context::{self, spawn_traced, spawn_traced_join};
use super::storage_api::object_usecase::s3_api::multipart::parse_list_parts_params;
@@ -5799,7 +5799,9 @@ impl DefaultObjectUsecase {
)?;
let mut metadata = metadata.unwrap_or_default();
let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
|| has_replication_retention_update(&req.headers, inbound_replication_put);
let object_lock_config_stage_start = put_stage_metrics_enabled.then(Instant::now);
let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
rustfs_io_metrics::record_put_object_stage_duration_from("app_object_lock_config_lookup", object_lock_config_stage_start);
@@ -10081,6 +10083,19 @@ mod tests {
assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some("COMPLIANCE"));
assert!(metadata.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER));
assert_eq!(metadata.get("x-amz-meta-x-amz-object-lock-mode").map(String::as_str), Some("GOVERNANCE"));
let mut replication_headers = HeaderMap::new();
insert_header(&mut replication_headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
insert_header(
&mut replication_headers,
rustfs_utils::http::SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
"2026-01-01T00:00:00Z",
);
let mut replica_metadata = HashMap::new();
let explicit_clear = has_replication_retention_update(&replication_headers, true);
apply_bucket_default_lock_retention("bucket", &state, &mut replica_metadata, explicit_clear).unwrap();
assert!(!replica_metadata.contains_key(AMZ_OBJECT_LOCK_MODE_LOWER));
assert!(!replica_metadata.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER));
}
fn pax_record(key: &str, value: &[u8]) -> Vec<u8> {
+3 -2
View File
@@ -996,8 +996,9 @@ pub(crate) mod options {
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
extract_metadata_from_mime, extract_metadata_from_mime_with_object_name, filter_object_metadata,
get_complete_multipart_upload_opts_with_replication_authorization, get_content_sha256_with_query, get_opts,
namespace_reserved_user_metadata, normalize_content_encoding_for_storage, parse_copy_source_range,
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
parse_copy_source_range, preserve_unclassified_user_metadata, put_opts_with_replication_authorization,
validate_archive_content_encoding,
};
}
+115 -6
View File
@@ -111,6 +111,9 @@ pub(crate) struct PostObjectRequestMarker;
#[derive(Clone, Debug)]
struct InternalObjectAuthorization;
#[derive(Clone, Debug)]
struct StagedMultipartPartAuthorization;
#[derive(Clone, Default)]
struct TableDataPlanePublicationGuards {
state: Arc<parking_lot::Mutex<TableDataPlanePublicationState>>,
@@ -1333,6 +1336,15 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
Err(denial.deny("no_policy_allows_action", action))
}
// Multipart parts are staged and become visible only after CompleteMultipartUpload,
// which uses the normal publication-fenced authorization path.
async fn authorize_staged_multipart_part<T>(req: &mut S3Request<T>) -> S3Result<()> {
req.extensions.insert(StagedMultipartPartAuthorization);
let result = authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await;
req.extensions.remove::<StagedMultipartPartAuthorization>();
result
}
/// Check if the request has the x-amz-bypass-governance-retention header set to true
pub fn has_bypass_governance_header(headers: &http::HeaderMap) -> bool {
headers
@@ -1435,6 +1447,10 @@ fn table_data_plane_content_mutation(action: Action) -> bool {
)
}
fn table_data_plane_publication_fence_required<T>(req: &S3Request<T>, action: Action) -> bool {
req.extensions.get::<StagedMultipartPartAuthorization>().is_none() && table_data_plane_content_mutation(action)
}
fn table_catalog_backend_for_data_plane<T>(
req: &S3Request<T>,
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
@@ -1478,7 +1494,7 @@ async fn retain_table_data_plane_publication_guard<T>(
.map_err(|err| s3_error!(InternalError, "failed to acquire table publication guard: {}", err))?;
let mut state = retained.state.lock();
state.keys.insert(key);
state.guards.push(guard);
state.guards.push(Box::new(guard));
drop(state);
req.extensions.insert(retained);
Ok(())
@@ -1613,7 +1629,7 @@ async fn authorize_table_data_plane_if_needed<T>(
let Some(admin_action) = table_data_plane_admin_action(action) else {
return Ok(());
};
if table_data_plane_content_mutation(action) {
if table_data_plane_publication_fence_required(req, action) {
retain_table_bucket_publication_guard(req, bucket).await?;
}
let table_bucket_enabled = table_bucket_enabled_for_data_plane(req, bucket).await?;
@@ -1639,7 +1655,7 @@ async fn authorize_table_data_plane_if_needed<T>(
})
.await;
if allowed {
if table_data_plane_content_mutation(action) {
if table_data_plane_publication_fence_required(req, action) {
retain_table_publication_guard(req, &resource).await?;
}
return Ok(());
@@ -1675,7 +1691,7 @@ async fn deny_anonymous_table_data_plane_if_needed<T>(
if table_data_plane_admin_action(action).is_none() {
return Ok(());
}
if table_data_plane_content_mutation(action) {
if table_data_plane_publication_fence_required(req, action) {
retain_table_bucket_publication_guard(req, bucket).await?;
}
let table_bucket_enabled = table_bucket_enabled_for_data_plane(req, bucket).await?;
@@ -2906,7 +2922,7 @@ impl S3Access for FS {
req_info.bucket = Some(req.input.bucket.clone());
req_info.object = Some(req.input.key.clone());
authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?;
authorize_staged_multipart_part(req).await?;
req.extensions.insert(bucket_generation?);
Ok(())
}
@@ -2943,7 +2959,7 @@ impl S3Access for FS {
req_info.object = Some(req.input.key.clone());
req_info.version_id = None;
authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?;
authorize_staged_multipart_part(req).await?;
req.extensions.insert(bucket_generation?);
Ok(())
}
@@ -3159,6 +3175,99 @@ mod tests {
}
}
#[tokio::test]
#[serial]
async fn multipart_parts_skip_table_publication_fence_but_completion_retains_it() {
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
let server_ctx = ServerContextSlot::new();
assert!(server_ctx.install(Arc::new(AppContext::new(Arc::clone(&store), Arc::new(UnreadyIam), Arc::new(TestKms),))));
let fs = FS::with_server_ctx(server_ctx);
let bucket = format!("multipart-table-fence-{}", uuid::Uuid::new_v4());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("test bucket should be created");
let policy_json = format!(
r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"#
);
let mut metadata = (*crate::storage::get_bucket_metadata(&bucket)
.await
.expect("new bucket metadata should be cached"))
.clone();
metadata.policy_config = Some(serde_json::from_str(&policy_json).expect("test bucket policy should parse"));
metadata.policy_config_json = policy_json.into_bytes();
crate::storage::storage_api::set_bucket_metadata(bucket.clone(), metadata)
.await
.expect("test bucket policy should be published");
let mut part_req = build_request(
UploadPartInput::builder()
.bucket(bucket.clone())
.key("object".to_string())
.upload_id("upload-id".to_string())
.part_number(1)
.build()
.expect("upload part input should build"),
Method::PUT,
);
ensure_req_info(&mut part_req);
part_req.extensions.insert(fs.server_ctx().clone());
fs.upload_part(&mut part_req)
.await
.expect("anonymous UploadPart should be authorized by the test policy");
assert!(
part_req.extensions.get::<TableDataPlanePublicationGuards>().is_none(),
"staged part data must not retain a table publication fence"
);
let mut copy_part_req = build_request(
UploadPartCopyInput::builder()
.bucket(bucket.clone())
.key("object".to_string())
.upload_id("upload-id".to_string())
.part_number(2)
.copy_source(CopySource::Bucket {
bucket: bucket.clone().into(),
key: "source".into(),
version_id: None,
})
.build()
.expect("upload part copy input should build"),
Method::PUT,
);
ensure_req_info(&mut copy_part_req);
copy_part_req.extensions.insert(fs.server_ctx().clone());
fs.upload_part_copy(&mut copy_part_req)
.await
.expect("anonymous UploadPartCopy should be authorized by the test policy");
assert!(
copy_part_req.extensions.get::<TableDataPlanePublicationGuards>().is_none(),
"staged copied part data must not retain a table publication fence"
);
let mut complete_req = build_request(
CompleteMultipartUploadInput::builder()
.bucket(bucket.clone())
.key("object".to_string())
.upload_id("upload-id".to_string())
.multipart_upload(Some(CompletedMultipartUpload::default()))
.build()
.expect("complete multipart input should build"),
Method::POST,
);
ensure_req_info(&mut complete_req);
complete_req.extensions.insert(fs.server_ctx().clone());
fs.complete_multipart_upload(&mut complete_req)
.await
.expect("anonymous CompleteMultipartUpload should be authorized by the test policy");
let bucket_fence = (bucket, crate::table_catalog::default_table_bucket_publication_lock_path());
let guards = complete_req
.extensions
.get::<TableDataPlanePublicationGuards>()
.expect("completion should retain a table publication fence");
assert!(guards.state.lock().keys.contains(&bucket_fence));
}
#[tokio::test]
async fn table_data_plane_request_reuses_table_resource_for_distinct_object_while_commit_waits() {
let resource = crate::table_catalog::TableDataPlaneResource {
+11 -1
View File
@@ -45,7 +45,7 @@ use rustfs_targets::EventName;
use rustfs_utils::http::headers::{
AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
};
use rustfs_utils::http::{SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, insert_str};
use rustfs_utils::http::{SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_TAGGING_TIMESTAMP, insert_str};
use s3s::{S3, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, dto::*, s3_error};
use std::collections::HashMap;
use std::fmt::Debug;
@@ -461,6 +461,11 @@ impl S3 for FS {
let mut eval_metadata = HashMap::new();
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, dsc.pending_status().unwrap_or_default());
insert_str(
&mut eval_metadata,
SUFFIX_TAGGING_TIMESTAMP,
OffsetDateTime::now_utc().format(&Rfc3339).unwrap_or_default(),
);
opts.eval_metadata = Some(eval_metadata);
}
@@ -1645,6 +1650,11 @@ impl S3 for FS {
let mut eval_metadata = HashMap::new();
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, dsc.pending_status().unwrap_or_default());
insert_str(
&mut eval_metadata,
SUFFIX_TAGGING_TIMESTAMP,
OffsetDateTime::now_utc().format(&Rfc3339).unwrap_or_default(),
);
opts.eval_metadata = Some(eval_metadata);
}
+216 -7
View File
@@ -17,11 +17,13 @@ use crate::storage::storage_api::options_consumer::contract::{object::HTTPPrecon
use http::header::{IF_MATCH, IF_NONE_MATCH};
use http::{HeaderMap, HeaderValue};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_FORCE_DELETE, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_VERSION_ID, get_header,
AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_FORCE_DELETE, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID, SUFFIX_TAGGING_TIMESTAMP, get_header,
header_compat::{MINIO_ENCRYPTION_PREFIX, RUSTFS_ENCRYPTION_PREFIX},
insert_header_map,
insert_header_map, insert_str,
metadata_compat::{MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX},
};
use rustfs_utils::http::{
@@ -422,6 +424,9 @@ pub fn get_complete_multipart_upload_opts_with_replication_authorization(
preserve_etag,
..Default::default()
};
if replication_request {
apply_replication_timestamps_from_headers(headers, &mut opts);
}
apply_replica_status_from_headers(headers, &mut opts, replication_request_authorized);
fill_conditional_writes_opts_from_header(headers, &mut opts)?;
@@ -458,6 +463,12 @@ pub fn put_opts_from_headers(headers: &HeaderMap<HeaderValue>, metadata: HashMap
put_opts_from_headers_with_replication_authorization(headers, metadata, false)
}
pub(crate) fn has_replication_retention_update(headers: &HeaderMap<HeaderValue>, replication_request_authorized: bool) -> bool {
replication_request_authorized
&& get_header(headers, SUFFIX_SOURCE_REPLICATION_REQUEST).as_deref() == Some("true")
&& replication_timestamp_header(headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP).is_some()
}
pub fn put_opts_from_headers_with_replication_authorization(
headers: &HeaderMap<HeaderValue>,
metadata: HashMap<String, String>,
@@ -479,6 +490,7 @@ pub fn put_opts_from_headers_with_replication_authorization(
if let Some(crc) = get_header(headers, SUFFIX_REPLICATION_SSEC_CRC) {
insert_header_map(&mut opts.user_defined, SUFFIX_REPLICATION_SSEC_CRC, crc.into_owned());
}
apply_replication_timestamps_from_headers(headers, &mut opts);
}
Ok(opts)
}
@@ -504,6 +516,47 @@ fn replication_source_mtime(headers: &HeaderMap<HeaderValue>) -> Option<time::Of
}
}
/// Parses one replication LWW timestamp header. Invalid values are dropped
/// with a warning (same tolerance as [`replication_source_mtime`]) so a
/// malformed source header cannot wedge the replication queue.
fn replication_timestamp_header(headers: &HeaderMap<HeaderValue>, suffix: &str) -> Option<time::OffsetDateTime> {
let value = get_header(headers, suffix)?;
let value = value.trim();
match time::OffsetDateTime::parse(value, &time::format_description::well_known::Rfc3339) {
Ok(timestamp) => Some(timestamp),
Err(err) => {
tracing::warn!("Invalid {} value '{}' (replication request=true): {}", suffix, value, err);
None
}
}
}
/// Callers must gate on an authorized replication request: these headers are
/// trusted source-cluster state, not client input.
fn apply_replication_timestamps_from_headers(headers: &HeaderMap<HeaderValue>, opts: &mut ObjectOptions) {
opts.replication_tagging_timestamp = replication_timestamp_header(headers, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP);
opts.replication_retention_timestamp = replication_timestamp_header(headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP);
opts.replication_legalhold_timestamp = replication_timestamp_header(headers, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP);
// Persist into the internal metadata keys so a later outbound replication
// pass (replication_target_boundary) reads the source's modification
// times instead of falling back to mod_time.
// TODO(P1-6): receiver-side LWW is still missing — when the stored
// per-category timestamp is newer than the inbound one, the existing
// tags/retention/legal-hold should win instead of being overwritten.
for (timestamp, suffix) in [
(opts.replication_tagging_timestamp, SUFFIX_TAGGING_TIMESTAMP),
(opts.replication_retention_timestamp, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP),
(opts.replication_legalhold_timestamp, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP),
] {
if let Some(timestamp) = timestamp
&& let Ok(value) = timestamp.format(&time::format_description::well_known::Rfc3339)
{
insert_str(&mut opts.user_defined, suffix, value);
}
}
}
fn apply_replica_status_from_headers(headers: &HeaderMap<HeaderValue>, opts: &mut ObjectOptions, authorized: bool) {
if !authorized {
return;
@@ -663,6 +716,13 @@ fn is_reserved_user_metadata_key(key: &str) -> bool {
|| starts_with_ignore_ascii_case(key, MINIO_INTERNAL_PREFIX)
|| starts_with_ignore_ascii_case(key, RUSTFS_ENCRYPTION_PREFIX)
|| starts_with_ignore_ascii_case(key, MINIO_ENCRYPTION_PREFIX)
// Replication transport names (source-replication timestamps,
// source-mtime/-etag/-version-id, ...). A bare stored key with one of
// these names is forwarded verbatim by the outbound replication
// header builder on a server-authorized request, so the receiver
// would persist attacker-chosen values as trusted internal LWW state.
|| starts_with_ignore_ascii_case(key, "x-rustfs-source-")
|| starts_with_ignore_ascii_case(key, "x-minio-source-")
}
fn stored_user_metadata_key(key: &str) -> String {
@@ -1082,15 +1142,16 @@ mod tests {
del_opts_with_versioning, detect_content_type_from_object_name, extract_metadata, extract_metadata_from_mime,
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_complete_multipart_upload_opts,
get_complete_multipart_upload_opts_with_replication_authorization, get_default_opts, get_opts,
namespace_reserved_user_metadata, parse_copy_source_range, put_opts, put_opts_from_headers,
put_opts_from_headers_with_replication_authorization, put_opts_with_replication_authorization,
has_replication_retention_update, namespace_reserved_user_metadata, parse_copy_source_range, put_opts,
put_opts_from_headers, put_opts_from_headers_with_replication_authorization, put_opts_with_replication_authorization,
validate_archive_content_encoding,
};
use http::{HeaderMap, HeaderValue};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER,
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG,
SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
SUFFIX_SOURCE_VERSION_ID, insert_header,
};
use s3s::S3ErrorCode;
use s3s::dto::{BucketVersioningStatus, ExcludedPrefix, VersioningConfiguration};
@@ -1520,6 +1581,24 @@ mod tests {
assert!(opts.preserve_etag.is_none());
}
#[test]
fn test_replication_retention_update_requires_authorization() {
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, "2026-01-01T00:00:00Z");
assert!(!has_replication_retention_update(&headers, false));
assert!(has_replication_retention_update(&headers, true));
let mut missing_request = HeaderMap::new();
insert_header(
&mut missing_request,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP,
"2026-01-01T00:00:00Z",
);
assert!(!has_replication_retention_update(&missing_request, true));
}
#[test]
fn test_put_opts_from_headers_gates_ssec_passthrough_on_authorization() {
use rustfs_utils::http::object_encryption_keys::{
@@ -1599,6 +1678,136 @@ mod tests {
assert!(opts_invalid.mod_time.is_none());
}
/// A client PUT must not materialize the replication transport names as
/// bare stored user-metadata keys: the outbound replication header
/// builder forwards user metadata verbatim on a server-authorized
/// request, so a bare `x-rustfs-source-replication-*-timestamp` key would
/// deliver an attacker-chosen value into the replica's trusted internal
/// LWW state (for a tagless object nothing later overwrites it).
#[test]
fn test_replication_transport_names_cannot_be_forged_via_user_metadata() {
let mut headers = HeaderMap::new();
for name in [
"x-amz-meta-x-rustfs-source-replication-tagging-timestamp",
"x-amz-meta-x-minio-source-replication-legalhold-timestamp",
"x-rustfs-meta-x-rustfs-source-replication-retention-timestamp",
"x-amz-meta-x-rustfs-source-mtime",
] {
headers.insert(
http::header::HeaderName::from_static(name),
HeaderValue::from_static("2026-01-02T03:04:05Z"),
);
}
let metadata = extract_metadata(&headers);
for forged in [
"x-rustfs-source-replication-tagging-timestamp",
"x-minio-source-replication-legalhold-timestamp",
"x-rustfs-source-replication-retention-timestamp",
"x-rustfs-source-mtime",
] {
assert!(
!metadata.contains_key(forged),
"{forged} must not be storable as a bare user-metadata key"
);
}
// The values survive, namespaced back under the user-metadata prefix.
assert_eq!(
metadata
.get("x-amz-meta-x-rustfs-source-replication-tagging-timestamp")
.map(String::as_str),
Some("2026-01-02T03:04:05Z")
);
}
#[test]
fn test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization() {
// Sender-side LWW state (replication_target_boundary.rs) is read back
// from these internal metadata keys, so an authorized replication PUT
// must persist the inbound timestamp headers; an unauthorized client
// must not be able to forge them.
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
insert_header(&mut headers, "source-replication-tagging-timestamp", "2026-01-02T03:04:05Z");
insert_header(&mut headers, "source-replication-retention-timestamp", "2026-01-02T03:04:06Z");
insert_header(&mut headers, "source-replication-legalhold-timestamp", "2026-01-02T03:04:07Z");
let untrusted = put_opts_from_headers(&headers, HashMap::new()).expect("ordinary PUT options should be created");
for suffix in [
"tagging-timestamp",
"objectlock-retention-timestamp",
"objectlock-legalhold-timestamp",
] {
assert!(
rustfs_utils::http::get_str(&untrusted.user_defined, suffix).is_none(),
"unauthorized clients must not persist the {suffix} internal key"
);
}
assert!(untrusted.replication_tagging_timestamp.is_none());
assert!(untrusted.replication_retention_timestamp.is_none());
assert!(untrusted.replication_legalhold_timestamp.is_none());
let trusted = put_opts_from_headers_with_replication_authorization(&headers, HashMap::new(), true)
.expect("authorized replication request should parse");
let parse = |value: &str| {
time::OffsetDateTime::parse(value, &time::format_description::well_known::Rfc3339).expect("valid RFC3339")
};
assert_eq!(trusted.replication_tagging_timestamp, Some(parse("2026-01-02T03:04:05Z")));
assert_eq!(trusted.replication_retention_timestamp, Some(parse("2026-01-02T03:04:06Z")));
assert_eq!(trusted.replication_legalhold_timestamp, Some(parse("2026-01-02T03:04:07Z")));
for (suffix, expected) in [
("tagging-timestamp", "2026-01-02T03:04:05Z"),
("objectlock-retention-timestamp", "2026-01-02T03:04:06Z"),
("objectlock-legalhold-timestamp", "2026-01-02T03:04:07Z"),
] {
assert_eq!(
rustfs_utils::http::get_str(&trusted.user_defined, suffix).as_deref(),
Some(expected),
"authorized replication must persist the {suffix} internal key"
);
}
}
#[test]
fn test_complete_multipart_opts_persist_replication_timestamps_when_authorized() {
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
insert_header(&mut headers, "replication-actual-object-size", "1");
insert_header(&mut headers, "source-replication-tagging-timestamp", "2026-01-02T03:04:05Z");
insert_header(&mut headers, "source-replication-retention-timestamp", "2026-01-02T03:04:06Z");
insert_header(&mut headers, "source-replication-legalhold-timestamp", "2026-01-02T03:04:07Z");
let untrusted = get_complete_multipart_upload_opts(&headers).expect("ordinary multipart options should be created");
for suffix in [
"tagging-timestamp",
"objectlock-retention-timestamp",
"objectlock-legalhold-timestamp",
] {
assert!(
rustfs_utils::http::get_str(&untrusted.user_defined, suffix).is_none(),
"unauthorized multipart completes must not persist the {suffix} internal key"
);
}
let trusted = get_complete_multipart_upload_opts_with_replication_authorization(&headers, true)
.expect("authorized multipart complete should parse");
for (suffix, expected) in [
("tagging-timestamp", "2026-01-02T03:04:05Z"),
("objectlock-retention-timestamp", "2026-01-02T03:04:06Z"),
("objectlock-legalhold-timestamp", "2026-01-02T03:04:07Z"),
] {
assert_eq!(
rustfs_utils::http::get_str(&trusted.user_defined, suffix).as_deref(),
Some(expected),
"authorized multipart completes must persist the {suffix} internal key"
);
}
assert!(trusted.replication_tagging_timestamp.is_some());
assert!(trusted.replication_retention_timestamp.is_some());
assert!(trusted.replication_legalhold_timestamp.is_some());
}
#[test]
fn test_put_opts_from_headers_with_replica_status() {
let mut headers = HeaderMap::new();
+3 -2
View File
@@ -185,8 +185,9 @@ pub(crate) mod options_consumer {
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
extract_metadata_from_mime, extract_metadata_from_mime_with_object_name, filter_object_metadata,
get_complete_multipart_upload_opts_with_replication_authorization, get_content_sha256_with_query, get_opts,
namespace_reserved_user_metadata, normalize_content_encoding_for_storage, parse_copy_source_range,
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
parse_copy_source_range, preserve_unclassified_user_metadata, put_opts_with_replication_authorization,
validate_archive_content_encoding,
};
pub(crate) mod contract {
+234 -21
View File
@@ -16,6 +16,8 @@ use std::io::Read;
use super::super::*;
const AVRO_ZSTANDARD_MAX_WINDOW_LOG: u32 = 27;
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct ManifestDataFileReference {
pub location: String,
@@ -66,6 +68,7 @@ pub(crate) struct DecodedManifestList {
pub(crate) struct DecodedManifest {
pub references: Vec<ManifestDataFileReference>,
pub decoded_size: usize,
pub partition_spec_id: Option<i32>,
}
pub(crate) fn manifest_paths_from_manifest_list_avro(data: &[u8]) -> TableCatalogStoreResult<Vec<String>> {
@@ -92,6 +95,25 @@ pub(crate) fn decode_manifest_list_avro(data: &[u8]) -> TableCatalogStoreResult<
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to read manifest list Avro: {err}")))?;
let format_version =
avro_record_format_version(reader.writer_schema(), &["sequence_number", "min_sequence_number"], "manifest list")?;
if format_version == 2 {
let apache_avro::Schema::Record(record) = reader.writer_schema() else {
return Err(TableCatalogStoreError::Invalid("manifest list Avro schema must be a record".to_string()));
};
for field in [
"added_files_count",
"existing_files_count",
"deleted_files_count",
"added_rows_count",
"existing_rows_count",
"deleted_rows_count",
] {
if !record.lookup.contains_key(field) {
return Err(TableCatalogStoreError::Invalid(format!(
"Iceberg v2 manifest list Avro schema is missing {field}"
)));
}
}
}
let mut manifest_paths = Vec::new();
for value in reader {
if manifest_paths.len() >= TABLE_MANIFEST_AVRO_MAX_RECORDS {
@@ -115,24 +137,12 @@ pub(crate) fn decode_manifest_list_avro(data: &[u8]) -> TableCatalogStoreResult<
sequence_number: avro_record_field(&value, "sequence_number").and_then(avro_i64_value),
min_sequence_number: avro_record_field(&value, "min_sequence_number").and_then(avro_i64_value),
added_snapshot_id: avro_record_field(&value, "added_snapshot_id").and_then(avro_i64_value),
added_files_count: avro_record_field(&value, "added_files_count")
.and_then(avro_i32_value)
.and_then(|value| u64::try_from(value).ok()),
existing_files_count: avro_record_field(&value, "existing_files_count")
.and_then(avro_i32_value)
.and_then(|value| u64::try_from(value).ok()),
deleted_files_count: avro_record_field(&value, "deleted_files_count")
.and_then(avro_i32_value)
.and_then(|value| u64::try_from(value).ok()),
added_rows_count: avro_record_field(&value, "added_rows_count")
.and_then(avro_i64_value)
.and_then(|value| u64::try_from(value).ok()),
existing_rows_count: avro_record_field(&value, "existing_rows_count")
.and_then(avro_i64_value)
.and_then(|value| u64::try_from(value).ok()),
deleted_rows_count: avro_record_field(&value, "deleted_rows_count")
.and_then(avro_i64_value)
.and_then(|value| u64::try_from(value).ok()),
added_files_count: avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list")?,
existing_files_count: avro_nullable_non_negative_i32(&value, "existing_files_count", "manifest list")?,
deleted_files_count: avro_nullable_non_negative_i32(&value, "deleted_files_count", "manifest list")?,
added_rows_count: avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list")?,
existing_rows_count: avro_nullable_non_negative_i64(&value, "existing_rows_count", "manifest list")?,
deleted_rows_count: avro_nullable_non_negative_i64(&value, "deleted_rows_count", "manifest list")?,
});
}
Ok(DecodedManifestList {
@@ -171,6 +181,17 @@ pub(crate) fn decode_manifest_avro(data: &[u8]) -> TableCatalogStoreResult<Decod
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to read manifest Avro: {err}")))?;
let format_version =
avro_record_format_version(reader.writer_schema(), &["sequence_number", "file_sequence_number"], "manifest")?;
let partition_spec_id = reader
.user_metadata()
.get("partition-spec-id")
.map(|value| {
std::str::from_utf8(value)
.ok()
.and_then(|value| value.parse::<i32>().ok())
.filter(|value| *value >= 0)
.ok_or_else(|| TableCatalogStoreError::Invalid("manifest partition-spec-id metadata is invalid".to_string()))
})
.transpose()?;
let mut files = Vec::new();
for value in reader {
if files.len() >= TABLE_MANIFEST_AVRO_MAX_RECORDS {
@@ -202,6 +223,9 @@ pub(crate) fn decode_manifest_avro(data: &[u8]) -> TableCatalogStoreResult<Decod
)));
}
};
let partition = avro_record_field(data_file, "partition")
.and_then(avro_record_value_fields)
.ok_or_else(|| TableCatalogStoreError::Invalid("manifest data file partition must be a record".to_string()))?;
files.push(ManifestDataFileReference {
location: file_path.to_string(),
format_version,
@@ -218,15 +242,14 @@ pub(crate) fn decode_manifest_avro(data: &[u8]) -> TableCatalogStoreResult<Decod
file_size_bytes: avro_record_field(data_file, "file_size_in_bytes")
.and_then(avro_i64_value)
.and_then(|value| u64::try_from(value).ok()),
partition: avro_record_field(data_file, "partition")
.and_then(avro_record_value_fields)
.unwrap_or_default(),
partition,
sort_order_id: avro_record_field(data_file, "sort_order_id").and_then(avro_i32_value),
});
}
Ok(DecodedManifest {
references: files,
decoded_size,
partition_spec_id,
})
}
@@ -240,6 +263,8 @@ pub(crate) async fn decode_manifest_avro_async(data: Vec<u8>) -> TableCatalogSto
enum AvroContainerCodec {
Null,
Deflate,
Snappy,
Zstandard,
}
fn validate_avro_container(data: &[u8]) -> TableCatalogStoreResult<usize> {
@@ -300,6 +325,8 @@ fn validate_avro_container(data: &[u8]) -> TableCatalogStoreResult<usize> {
let codec = match codec.unwrap_or(b"null") {
b"null" => AvroContainerCodec::Null,
b"deflate" => AvroContainerCodec::Deflate,
b"snappy" => AvroContainerCodec::Snappy,
b"zstandard" => AvroContainerCodec::Zstandard,
codec => {
return Err(TableCatalogStoreError::Unsupported(format!(
"Avro codec {} is not supported for table commit validation",
@@ -369,6 +396,34 @@ fn avro_block_decoded_size(codec: AvroContainerCodec, block: &[u8], remaining_si
usize::try_from(decoded_size)
.map_err(|_| TableCatalogStoreError::Invalid("Avro decoded data size is invalid".to_string()))
}
AvroContainerCodec::Snappy => {
let data_end = block
.len()
.checked_sub(4)
.ok_or_else(|| TableCatalogStoreError::Invalid("Avro snappy block is missing its checksum".to_string()))?;
let decoded_size = snap::raw::decompress_len(&block[..data_end])
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to inspect Avro snappy block: {err}")))?;
if decoded_size > remaining_size {
return Err(TableCatalogStoreError::Invalid("Avro decoded data exceeds the commit limit".to_string()));
}
Ok(decoded_size)
}
AvroContainerCodec::Zstandard => {
let limit = remaining_size
.checked_add(1)
.ok_or_else(|| TableCatalogStoreError::Invalid("Avro decoded data size limit overflowed".to_string()))?;
let limit = u64::try_from(limit)
.map_err(|_| TableCatalogStoreError::Invalid("Avro decoded data size limit is invalid".to_string()))?;
let mut decoder = zstd::stream::read::Decoder::new(block)
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to decompress Avro zstandard block: {err}")))?;
decoder
.window_log_max(AVRO_ZSTANDARD_MAX_WINDOW_LOG)
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to bound Avro zstandard window: {err}")))?;
let decoded_size = std::io::copy(&mut decoder.take(limit), &mut std::io::sink())
.map_err(|err| TableCatalogStoreError::Invalid(format!("failed to decompress Avro zstandard block: {err}")))?;
usize::try_from(decoded_size)
.map_err(|_| TableCatalogStoreError::Invalid("Avro decoded data size is invalid".to_string()))
}
}
}
@@ -445,6 +500,40 @@ fn avro_record_value_fields(value: &apache_avro::types::Value) -> Option<Vec<(St
)
}
fn avro_nullable_non_negative_i32(
value: &apache_avro::types::Value,
field: &str,
label: &str,
) -> TableCatalogStoreResult<Option<u64>> {
let Some(value) = avro_record_field(value, field) else {
return Ok(None);
};
match avro_non_union_value(value) {
apache_avro::types::Value::Null => Ok(None),
apache_avro::types::Value::Int(value) => u64::try_from(*value)
.map(Some)
.map_err(|_| TableCatalogStoreError::Invalid(format!("{label} field {field} must be a non-negative int"))),
_ => Err(TableCatalogStoreError::Invalid(format!("{label} field {field} must be a nullable int"))),
}
}
fn avro_nullable_non_negative_i64(
value: &apache_avro::types::Value,
field: &str,
label: &str,
) -> TableCatalogStoreResult<Option<u64>> {
let Some(value) = avro_record_field(value, field) else {
return Ok(None);
};
match avro_non_union_value(value) {
apache_avro::types::Value::Null => Ok(None),
apache_avro::types::Value::Long(value) => u64::try_from(*value)
.map(Some)
.map_err(|_| TableCatalogStoreError::Invalid(format!("{label} field {field} must be a non-negative long"))),
_ => Err(TableCatalogStoreError::Invalid(format!("{label} field {field} must be a nullable long"))),
}
}
pub(crate) fn avro_non_union_value(value: &apache_avro::types::Value) -> &apache_avro::types::Value {
match value {
apache_avro::types::Value::Union(_, inner) => avro_non_union_value(inner),
@@ -472,3 +561,127 @@ fn avro_i64_value(value: &apache_avro::types::Value) -> Option<i64> {
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn rejects_v2_manifest_lists_without_required_count_fields() {
let schema = apache_avro::Schema::parse_str(
r#"{
"type": "record",
"name": "manifest_file",
"fields": [
{"name": "manifest_path", "type": "string"},
{"name": "manifest_length", "type": "long"},
{"name": "partition_spec_id", "type": "int"},
{"name": "content", "type": "int"},
{"name": "sequence_number", "type": "long"},
{"name": "min_sequence_number", "type": "long"},
{"name": "added_snapshot_id", "type": "long"}
]
}"#,
)
.expect("incomplete manifest-list schema should parse");
let data = apache_avro::Writer::new(&schema, Vec::new())
.expect("manifest-list writer should initialize")
.into_inner()
.expect("manifest-list bytes should flush");
let error = match decode_manifest_list_avro(&data) {
Ok(_) => panic!("v2 count fields must be declared in the writer schema"),
Err(error) => error,
};
assert_eq!(
error,
TableCatalogStoreError::Invalid("Iceberg v2 manifest list Avro schema is missing added_files_count".to_string())
);
}
#[test]
fn rejects_negative_nullable_manifest_list_counts() {
let value = apache_avro::types::Value::Record(vec![
("added_files_count".to_string(), apache_avro::types::Value::Int(-1)),
("added_rows_count".to_string(), apache_avro::types::Value::Long(-1)),
]);
assert_eq!(
avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list")
.expect_err("negative file counts must be rejected"),
TableCatalogStoreError::Invalid("manifest list field added_files_count must be a non-negative int".to_string())
);
assert_eq!(
avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list")
.expect_err("negative row counts must be rejected"),
TableCatalogStoreError::Invalid("manifest list field added_rows_count must be a non-negative long".to_string())
);
}
#[test]
fn rejects_manifest_partition_with_non_record_schema() {
let schema = apache_avro::Schema::parse_str(
r#"{
"type": "record",
"name": "manifest_entry",
"fields": [
{"name": "status", "type": "int"},
{"name": "snapshot_id", "type": "long"},
{
"name": "data_file",
"type": {
"type": "record",
"name": "data_file",
"fields": [
{"name": "file_path", "type": "string"},
{"name": "record_count", "type": "long"},
{"name": "file_size_in_bytes", "type": "long"},
{"name": "partition", "type": "string"}
]
}
}
]
}"#,
)
.expect("manifest schema should parse");
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
writer
.append_value(apache_avro::types::Value::Record(vec![
("status".to_string(), apache_avro::types::Value::Int(1)),
("snapshot_id".to_string(), apache_avro::types::Value::Long(1)),
(
"data_file".to_string(),
apache_avro::types::Value::Record(vec![
(
"file_path".to_string(),
apache_avro::types::Value::String("s3://warehouse/tables/table-id/data/file.parquet".to_string()),
),
("record_count".to_string(), apache_avro::types::Value::Long(1)),
("file_size_in_bytes".to_string(), apache_avro::types::Value::Long(1)),
("partition".to_string(), apache_avro::types::Value::String("not-a-record".to_string())),
]),
),
]))
.expect("manifest record should append");
let data = writer.into_inner().expect("manifest bytes should flush");
let error = match decode_manifest_avro(&data) {
Ok(_) => panic!("manifest partitions must preserve their record shape"),
Err(error) => error,
};
assert_eq!(
error,
TableCatalogStoreError::Invalid("manifest data file partition must be a record".to_string())
);
}
#[test]
fn rejects_oversized_zstandard_windows() {
// Non-single-segment frame with a 2^28-byte window and one empty final block.
let compressed = [0x28, 0xb5, 0x2f, 0xfd, 0x00, 0x90, 0x01, 0x00, 0x00];
let error = avro_block_decoded_size(AvroContainerCodec::Zstandard, &compressed, TABLE_MANIFEST_AVRO_MAX_DECODED_SIZE)
.expect_err("zstandard windows larger than the manifest decode budget must be rejected");
assert!(matches!(error, TableCatalogStoreError::Invalid(_)));
}
}
File diff suppressed because it is too large Load Diff
+4
View File
@@ -111,8 +111,12 @@ const TABLE_MANIFEST_AVRO_MAX_DECODED_SIZE: usize = 128 * 1024 * 1024;
const TABLE_MANIFEST_AVRO_MAX_RECORDS: usize = 1_000_000;
const TABLE_MANIFEST_AVRO_MAX_HEADER_ENTRIES: usize = 1_024;
const TABLE_COMMIT_MAX_MANIFESTS: usize = 10_000;
const TABLE_COMMIT_MAX_MANIFEST_TRAVERSALS: usize = 20_000;
const TABLE_COMMIT_MAX_AVRO_BYTES: usize = 512 * 1024 * 1024;
const TABLE_COMMIT_MAX_FILE_REFERENCES: usize = 1_000_000;
const TABLE_COMMIT_MAX_STATISTICS_OBJECTS: usize = 1_024;
const TABLE_COMMIT_MAX_STATISTICS_BYTES: usize = 512 * 1024 * 1024;
const TABLE_STATISTICS_FILE_MAX_SIZE: usize = 128 * 1024 * 1024;
pub(crate) const TABLE_COMMIT_OBJECT_VALIDATION_CONCURRENCY: usize = 16;
pub const TABLE_RESERVED_PREFIX: &str = BUCKET_TABLE_RESERVED_PREFIX;
const WAREHOUSE_ROOT: &str = "warehouses";
+3 -3
View File
@@ -216,7 +216,7 @@ where
Ok(fence)
}
pub(super) async fn acquire_table_bucket_registry_write_permit(&self) -> TableCatalogStoreResult<Box<dyn Send>> {
pub(super) async fn acquire_table_bucket_registry_write_permit(&self) -> TableCatalogStoreResult<TableCatalogLockGuard> {
let fence_path = self.paths.backing_migration_global_fence_path();
let lock_path = self.paths.backing_migration_global_fence_lock_path();
let guard = self.backend.acquire_read_lock(self.catalog_bucket(), &lock_path).await?;
@@ -235,7 +235,7 @@ where
pub(super) async fn acquire_object_backed_catalog_write_permit(
&self,
table_bucket: &str,
) -> TableCatalogStoreResult<Box<dyn Send>> {
) -> TableCatalogStoreResult<TableCatalogLockGuard> {
let lock_path = self.paths.backing_migration_fence_lock_path(table_bucket);
let guard = self.backend.acquire_read_lock(self.catalog_bucket(), &lock_path).await?;
if self.read_backing_migration_fence(table_bucket).await?.is_some() {
@@ -290,7 +290,7 @@ where
async fn collect_bucket_snapshot_with_locks(
&self,
table_bucket: &str,
guards: &mut Vec<Box<dyn Send>>,
guards: &mut Vec<TableCatalogLockGuard>,
) -> TableCatalogStoreResult<StrongTableCatalogBucketSnapshot> {
let bucket_path = self.paths.table_bucket_entry_path(table_bucket);
guards.push(self.backend.acquire_write_lock(self.catalog_bucket(), &bucket_path).await?);
+118 -10
View File
@@ -262,6 +262,29 @@ pub(crate) trait TableCatalogStore: Send + Sync {
async fn create_view(&self, entry: ViewEntry) -> TableCatalogStoreResult<()>;
async fn create_view_with_publication(
&self,
entry: ViewEntry,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<()> {
publication.begin_table_bucket(&entry.table_bucket).await?;
if !publication.holds_table_bucket(&entry.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view creation requires a table-bucket publication fence".to_string(),
));
}
publication
.prepare(&entry.table_bucket, &entry.namespace, &entry.view)
.await?;
if !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.view) {
return Err(TableCatalogStoreError::Internal(
"view creation requires a view publication fence".to_string(),
));
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
self.create_view(entry).await
}
async fn list_views(&self, table_bucket: &str, namespace: &str) -> TableCatalogStoreResult<Vec<ViewEntry>>;
async fn list_views_page(
@@ -283,6 +306,32 @@ pub(crate) trait TableCatalogStore: Send + Sync {
async fn replace_view(&self, request: ViewCommitRequest) -> TableCatalogStoreResult<ViewCommitResult>;
async fn replace_view_with_publication(
&self,
request: ViewCommitRequest,
table_bucket_fence_required: bool,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<ViewCommitResult> {
if table_bucket_fence_required {
publication.begin_table_bucket(&request.table_bucket).await?;
if !publication.holds_table_bucket(&request.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a table-bucket publication fence".to_string(),
));
}
}
publication
.prepare(&request.table_bucket, &request.namespace, &request.view)
.await?;
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.view) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a view publication fence".to_string(),
));
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
self.replace_view(request).await
}
async fn drop_view(&self, table_bucket: &str, namespace: &str, view: &str) -> TableCatalogStoreResult<()>;
async fn get_commit_by_id(
@@ -338,7 +387,7 @@ struct TableCommitLockPublication<'a, B> {
struct TableCommitLockPublicationState {
table_bucket: Option<String>,
table: Option<(String, String, String)>,
guards: Vec<Box<dyn Send>>,
guards: Vec<TableCatalogLockGuard>,
}
impl<'a, B> TableCommitLockPublication<'a, B> {
@@ -409,15 +458,17 @@ where
}
fn holds_table_bucket(&self, table_bucket: &str) -> bool {
self.state.lock().table_bucket.as_deref() == Some(table_bucket)
let state = self.state.lock();
state.table_bucket.as_deref() == Some(table_bucket) && state.guards.iter().all(|guard| !guard.is_lock_lost())
}
fn holds_table(&self, table_bucket: &str, namespace: &str, table: &str) -> bool {
self.state
.lock()
let state = self.state.lock();
state
.table
.as_ref()
.is_some_and(|held| held.0 == table_bucket && held.1 == namespace && held.2 == table)
&& state.guards.iter().all(|guard| !guard.is_lock_lost())
}
fn complete(&self) {
@@ -438,6 +489,32 @@ pub(crate) struct TableCatalogObjectMetadata {
pub mod_time: Option<OffsetDateTime>,
}
pub(crate) struct TableCatalogLockGuard {
_guard: Box<dyn Send>,
lock_lost: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
}
impl TableCatalogLockGuard {
pub(crate) fn stable(guard: impl Send + 'static) -> Self {
Self {
_guard: Box::new(guard),
lock_lost: None,
}
}
fn namespace(guard: rustfs_lock::NamespaceLockGuard) -> Self {
let lock_lost = guard.lock_lost_signal();
Self {
_guard: Box::new(guard),
lock_lost,
}
}
pub(crate) fn is_lock_lost(&self) -> bool {
self.lock_lost.as_ref().is_some_and(|signal| signal.is_lost())
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct TableCatalogObjectListPage {
pub objects: Vec<String>,
@@ -588,11 +665,11 @@ pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static {
Ok(TableCatalogObjectListPage { objects, is_truncated })
}
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard> {
self.acquire_write_lock(bucket, object).await
}
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>>;
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard>;
async fn begin_table_bucket_commit_publication(&self, _table_bucket: &str) -> TableCatalogStoreResult<()> {
Ok(())
@@ -1169,6 +1246,17 @@ where
}
}
async fn create_view_with_publication(
&self,
entry: ViewEntry,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<()> {
match self {
Self::ObjectBacked(store) => store.create_view_with_publication(entry, publication).await,
Self::DurableStrong(store) => store.create_view_with_publication(entry, publication).await,
}
}
async fn list_views(&self, table_bucket: &str, namespace: &str) -> TableCatalogStoreResult<Vec<ViewEntry>> {
match self {
Self::ObjectBacked(store) => store.list_views(table_bucket, namespace).await,
@@ -1203,6 +1291,26 @@ where
}
}
async fn replace_view_with_publication(
&self,
request: ViewCommitRequest,
table_bucket_fence_required: bool,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<ViewCommitResult> {
match self {
Self::ObjectBacked(store) => {
store
.replace_view_with_publication(request, table_bucket_fence_required, publication)
.await
}
Self::DurableStrong(store) => {
store
.replace_view_with_publication(request, table_bucket_fence_required, publication)
.await
}
}
}
async fn drop_view(&self, table_bucket: &str, namespace: &str, view: &str) -> TableCatalogStoreResult<()> {
match self {
Self::ObjectBacked(store) => store.drop_view(table_bucket, namespace, view).await,
@@ -1686,7 +1794,7 @@ where
})
}
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard> {
let lock = self
.store
.new_ns_lock(bucket, object)
@@ -1696,10 +1804,10 @@ where
.get_write_lock(get_lock_acquire_timeout())
.await
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to acquire catalog table lock: {err}")))?;
Ok(Box::new(guard))
Ok(TableCatalogLockGuard::namespace(guard))
}
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard> {
let lock = self
.store
.new_ns_lock(bucket, object)
@@ -1709,7 +1817,7 @@ where
.get_read_lock(get_lock_acquire_timeout())
.await
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to acquire catalog migration lock: {err}")))?;
Ok(Box::new(guard))
Ok(TableCatalogLockGuard::namespace(guard))
}
}
+170 -24
View File
@@ -1036,6 +1036,20 @@ where
.await
}
async fn restore_table_warehouse_index_after_failed_drop(&self, entry: &TableEntry, reason: &'static str) {
if let Err(err) = self.reserve_table_warehouse_index(entry).await {
tracing::warn!(
table_bucket = %entry.table_bucket,
namespace = %entry.namespace,
table = %entry.table,
table_id = %entry.table_id,
reason,
error = %err,
"failed to restore table warehouse index after table drop stopped"
);
}
}
async fn delete_table_warehouse_index_if_changed(&self, current: &TableEntry, next: &TableEntry) {
let Ok(current_index) = table_warehouse_index_entry(current) else {
return;
@@ -1316,6 +1330,15 @@ where
}
self.ensure_table_warehouse_prefix_available(&entry).await?;
let reservation = self.reserve_table_warehouse_index(&entry).await?;
if !publication.holds_table_bucket(&entry.table_bucket)
|| !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table)
{
self.delete_created_table_warehouse_index(&entry, reservation, "table publication fence lost")
.await;
return Err(TableCatalogStoreError::Internal(
"table registration publication fence was lost before catalog update".to_string(),
));
}
let result = self
.write_entry_unlocked(self.catalog_bucket(), &table_path, &entry, precondition)
.await;
@@ -1327,7 +1350,25 @@ where
}
async fn write_view_entry(&self, entry: ViewEntry, precondition: TableCatalogPutPrecondition) -> TableCatalogStoreResult<()> {
let publication = TableCommitLockPublication::new(&self.backend);
self.write_view_entry_with_publication(entry, precondition, &publication)
.await
}
async fn write_view_entry_with_publication(
&self,
entry: ViewEntry,
precondition: TableCatalogPutPrecondition,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<()> {
validate_view_entry_version_and_id(&entry)?;
publication.begin_table_bucket(&entry.table_bucket).await?;
if !publication.holds_table_bucket(&entry.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view creation requires a table-bucket publication fence".to_string(),
));
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
self.require_table_bucket(&entry.table_bucket).await?;
let namespace = parse_namespace_for_store(&entry.namespace)?;
let view = parse_table_for_store(&entry.view)?;
@@ -1353,6 +1394,17 @@ where
entry.table_bucket, entry.namespace, entry.view
)));
}
// Preserve catalog -> publication -> object lock order across rolling upgrades.
publication
.prepare(&entry.table_bucket, &entry.namespace, &entry.view)
.await?;
if !publication.holds_table_bucket(&entry.table_bucket)
|| !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.view)
{
return Err(TableCatalogStoreError::Internal(
"view creation publication fence was lost before catalog update".to_string(),
));
}
self.write_entry_unlocked(self.catalog_bucket(), &view_path, &entry, precondition)
.await
}
@@ -4493,16 +4545,16 @@ where
validate_commit_metadata_digest(&request, &new_metadata_object)?;
let table_bucket = request.table_bucket.clone();
let metadata_location = request.new_metadata_location.clone();
let next_warehouse_location = tokio::task::spawn_blocking(move || {
table_metadata_warehouse_location(&table_bucket, &metadata_location, &new_metadata_object)
let next_metadata_state = tokio::task::spawn_blocking(move || {
table_metadata_commit_state(&table_bucket, &metadata_location, &new_metadata_object)
})
.await
.map_err(|err| TableCatalogStoreError::Internal(format!("table metadata parser task failed: {err}")))??;
if next_warehouse_location
let warehouse_relocation = next_metadata_state
.warehouse_location
.as_ref()
.is_some_and(|warehouse_location| warehouse_location != &current.warehouse_location)
&& !publication.holds_table_bucket(&request.table_bucket)
{
.is_some_and(|warehouse_location| warehouse_location != &current.warehouse_location);
if warehouse_relocation && !publication.holds_table_bucket(&request.table_bucket) {
return table_commit_result(
&request.table_bucket,
&request.namespace,
@@ -4537,9 +4589,12 @@ where
let mut next = current.clone();
next.metadata_location = staged_commit_log.new_metadata_location.clone();
if let Some(warehouse_location) = next_warehouse_location {
if let Some(warehouse_location) = next_metadata_state.warehouse_location {
next.warehouse_location = warehouse_location;
}
if let Some(format_version) = next_metadata_state.format_version {
next.format_version = format_version;
}
next.version_token = staged_commit_log.new_version_token.clone();
next.generation = current.generation.saturating_add(1);
if next.warehouse_location != current.warehouse_location {
@@ -4585,6 +4640,24 @@ where
);
}
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.table)
|| (warehouse_relocation && !publication.holds_table_bucket(&request.table_bucket))
{
self.delete_created_table_warehouse_index(&next, reservation, "table publication fence lost")
.await;
return table_commit_result(
&request.table_bucket,
&request.namespace,
&request.table,
&request.commit_id,
&request.operation,
commit_started,
Err(TableCatalogStoreError::Internal(
"table commit publication fence was lost before pointer update".to_string(),
)),
);
}
let cas_started = Instant::now();
let cas_result = self
.write_entry_unlocked(
@@ -4662,20 +4735,21 @@ where
)));
};
self.delete_owned_table_warehouse_index_for_drop(&entry).await?;
if !publication.holds_table_bucket(table_bucket)
|| !publication.holds_table(table_bucket, &namespace.public_name(), table.as_str())
{
self.restore_table_warehouse_index_after_failed_drop(&entry, "table publication fence lost")
.await;
return Err(TableCatalogStoreError::Internal(
"table drop publication fence was lost before catalog update".to_string(),
));
}
if let Err(err) = self.backend.delete_object_unlocked(self.catalog_bucket(), &object).await {
match self.read_table_with_etag_unlocked(table_bucket, &namespace, &table).await {
Ok(None) => return Ok(()),
Ok(Some((current, _))) if current == entry => {
if let Err(restore_err) = self.reserve_table_warehouse_index(&entry).await {
tracing::warn!(
table_bucket = %entry.table_bucket,
namespace = %entry.namespace,
table = %entry.table,
table_id = %entry.table_id,
error = %restore_err,
"failed to restore table warehouse index after table entry delete failure"
);
}
self.restore_table_warehouse_index_after_failed_drop(&entry, "table entry delete failed")
.await;
}
Ok(Some(_)) => {
return Err(TableCatalogStoreError::Internal(format!(
@@ -4703,6 +4777,15 @@ where
self.write_view_entry(entry, TableCatalogPutPrecondition::IfAbsent).await
}
async fn create_view_with_publication(
&self,
entry: ViewEntry,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<()> {
self.write_view_entry_with_publication(entry, TableCatalogPutPrecondition::IfAbsent, publication)
.await
}
async fn list_views(&self, table_bucket: &str, namespace: &str) -> TableCatalogStoreResult<Vec<ViewEntry>> {
let namespace = parse_namespace_for_store(namespace)?;
let mut entries = Vec::new();
@@ -4757,8 +4840,26 @@ where
}
async fn replace_view(&self, request: ViewCommitRequest) -> TableCatalogStoreResult<ViewCommitResult> {
let publication = TableCommitLockPublication::new(&self.backend);
self.replace_view_with_publication(request, true, &publication).await
}
async fn replace_view_with_publication(
&self,
request: ViewCommitRequest,
table_bucket_fence_required: bool,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<ViewCommitResult> {
let namespace = parse_namespace_for_store(&request.namespace)?;
let view = parse_table_for_store(&request.view)?;
if table_bucket_fence_required {
publication.begin_table_bucket(&request.table_bucket).await?;
if !publication.holds_table_bucket(&request.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a table-bucket publication fence".to_string(),
));
}
}
let _migration_guard = self.acquire_object_backed_catalog_write_permit(&request.table_bucket).await?;
let namespace_path = self.paths.namespace_entry_path(&request.table_bucket, &namespace);
let _namespace_guard = self
@@ -4767,6 +4868,16 @@ where
.await?;
let view_path = self.paths.view_entry_path(&request.table_bucket, &namespace, &view);
let _guard = self.backend.acquire_write_lock(self.catalog_bucket(), &view_path).await?;
// Preserve catalog -> publication -> object lock order across rolling upgrades.
publication
.prepare(&request.table_bucket, &request.namespace, &request.view)
.await?;
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.view) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a table publication fence".to_string(),
));
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
let Some((current, current_etag)) = self
.read_view_with_etag_unlocked(&request.table_bucket, &namespace, &view)
.await?
@@ -4814,6 +4925,14 @@ where
})
.await
.map_err(|err| TableCatalogStoreError::Internal(format!("view metadata parser task failed: {err}")))??;
let warehouse_relocation = next_warehouse_location
.as_deref()
.is_some_and(|location| location != current.warehouse_location);
if warehouse_relocation && !publication.holds_table_bucket(&request.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view warehouse relocation requires a table-bucket publication fence".to_string(),
));
}
let mut next = current;
next.metadata_location = request.new_metadata_location;
@@ -4822,13 +4941,40 @@ where
}
next.version_token = format!("token-{}", Uuid::new_v4());
next.generation = next.generation.saturating_add(1);
self.write_entry_unlocked(
self.catalog_bucket(),
&view_path,
&next,
TableCatalogPutPrecondition::IfMatch(current_etag),
)
.await?;
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.view)
|| ((table_bucket_fence_required || warehouse_relocation) && !publication.holds_table_bucket(&request.table_bucket))
{
return Err(TableCatalogStoreError::Internal(
"view replacement publication fence was lost before catalog update".to_string(),
));
}
let write_result = self
.write_entry_unlocked(
self.catalog_bucket(),
&view_path,
&next,
TableCatalogPutPrecondition::IfMatch(current_etag),
)
.await;
if let Err(err) = write_result {
match self
.read_view_with_etag_unlocked(&request.table_bucket, &namespace, &view)
.await
{
Ok(Some((persisted, _))) if persisted == next => {}
Ok(_) => return Err(err),
Err(read_err) => {
tracing::warn!(
table_bucket = %request.table_bucket,
namespace = %request.namespace,
view = %request.view,
error = %read_err,
"failed to verify view state after an ambiguous catalog update"
);
return Err(err);
}
}
}
Ok(ViewCommitResult { view: next })
}
+128 -20
View File
@@ -554,7 +554,7 @@ where
// Ordinary mutations hold the global migration read lock before the local write lock; migration takes the
// write side before invoking its dedicated snapshot mutation methods.
async fn acquire_snapshot_write_permit(&self) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_snapshot_write_permit(&self) -> TableCatalogStoreResult<TableCatalogLockGuard> {
let lock_path = TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path();
self.object_backend.acquire_read_lock(RUSTFS_META_BUCKET, &lock_path).await
}
@@ -1775,7 +1775,7 @@ where
request: &TableCommitRequest,
namespace: &Namespace,
table: &IdentifierSegment,
next_warehouse_location: Option<String>,
next_metadata_state: TableMetadataCommitState,
) -> TableCatalogStoreResult<TableCommitResult> {
let key = Self::table_key(&request.table_bucket, namespace, table);
let current = Self::validate_new_table_commit_locked(state, &key, request)?;
@@ -1802,9 +1802,12 @@ where
let mut next = current;
next.metadata_location = commit_log.new_metadata_location.clone();
if let Some(warehouse_location) = next_warehouse_location {
if let Some(warehouse_location) = next_metadata_state.warehouse_location {
next.warehouse_location = warehouse_location;
}
if let Some(format_version) = next_metadata_state.format_version {
next.format_version = format_version;
}
Self::ensure_table_warehouse_prefix_available_locked(state, &next, &key)?;
next.version_token = commit_log.new_version_token.clone();
next.generation = next.generation.saturating_add(1);
@@ -2170,6 +2173,7 @@ where
let _write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let key = Self::table_key(&entry.table_bucket, &namespace, &table);
let publication_identity = (entry.table_bucket.clone(), entry.namespace.clone(), entry.table.clone());
let (snapshot, precondition, postcondition) = {
let state = self.state.lock().await;
Self::require_table_bucket_in_state(&state, &entry.table_bucket)?;
@@ -2198,6 +2202,13 @@ where
StrongSnapshotWritePostcondition::TablePresent(entry),
)
};
if !publication.holds_table_bucket(&publication_identity.0)
|| !publication.holds_table(&publication_identity.0, &publication_identity.1, &publication_identity.2)
{
return Err(TableCatalogStoreError::Internal(
"table registration publication fence was lost before snapshot update".to_string(),
));
}
self.finalize_snapshot_write(snapshot, precondition, postcondition).await
}
@@ -2479,9 +2490,15 @@ where
let result = match prepared_result {
Ok((result, Some((snapshot, precondition)))) => {
let postcondition = Self::commit_write_postcondition(&request.table_bucket, &result.commit_log);
self.finalize_snapshot_write(snapshot, precondition, postcondition)
.await
.map(|_| result)
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.table) {
Err(TableCatalogStoreError::Internal(
"table commit publication fence was lost before snapshot update".to_string(),
))
} else {
self.finalize_snapshot_write(snapshot, precondition, postcondition)
.await
.map(|_| result)
}
}
Ok((result, None)) => Ok(result),
Err(err) => Err(err),
@@ -2519,8 +2536,8 @@ where
validate_commit_metadata_digest(&request, &new_metadata_object)?;
let table_bucket = request.table_bucket.clone();
let metadata_location = request.new_metadata_location.clone();
let next_warehouse_location = tokio::task::spawn_blocking(move || {
table_metadata_warehouse_location(&table_bucket, &metadata_location, &new_metadata_object)
let next_metadata_state = tokio::task::spawn_blocking(move || {
table_metadata_commit_state(&table_bucket, &metadata_location, &new_metadata_object)
})
.await
.map_err(|err| TableCatalogStoreError::Internal(format!("table metadata parser task failed: {err}")))??;
@@ -2539,11 +2556,11 @@ where
))
})?
};
if next_warehouse_location
let warehouse_relocation = next_metadata_state
.warehouse_location
.as_ref()
.is_some_and(|warehouse_location| warehouse_location != &current_warehouse_location)
&& !publication.holds_table_bucket(&request.table_bucket)
{
.is_some_and(|warehouse_location| warehouse_location != &current_warehouse_location);
if warehouse_relocation && !publication.holds_table_bucket(&request.table_bucket) {
return table_commit_result(
&request.table_bucket,
&request.namespace,
@@ -2561,7 +2578,7 @@ where
let prepared_result = {
let state = self.state.lock().await;
let (precondition, mut draft_state) = Self::snapshot_draft_context_locked(&state);
match Self::apply_commit_locked(&mut draft_state, &request, &namespace, &table, next_warehouse_location) {
match Self::apply_commit_locked(&mut draft_state, &request, &namespace, &table, next_metadata_state) {
Ok(result) => Self::snapshot_from_mutated_state_locked(&mut draft_state, self.snapshot_write_version)
.map(|snapshot| (result, snapshot, precondition)),
Err(err) => Err(err),
@@ -2570,7 +2587,16 @@ where
let result = match prepared_result {
Ok((result, snapshot, precondition)) => {
let postcondition = Self::commit_write_postcondition(&request.table_bucket, &result.commit_log);
match self.finalize_snapshot_write(snapshot, precondition, postcondition).await {
let snapshot_result = if publication.holds_table(&request.table_bucket, &request.namespace, &request.table)
&& (!warehouse_relocation || publication.holds_table_bucket(&request.table_bucket))
{
self.finalize_snapshot_write(snapshot, precondition, postcondition).await
} else {
Err(TableCatalogStoreError::Internal(
"table commit publication fence was lost before snapshot update".to_string(),
))
};
match snapshot_result {
Ok(()) => Ok(result),
Err(err) => {
let replay = {
@@ -2647,13 +2673,26 @@ where
},
)
};
if !publication.holds_table_bucket(table_bucket)
|| !publication.holds_table(table_bucket, &namespace.public_name(), table.as_str())
{
return Err(TableCatalogStoreError::Internal(
"table drop publication fence was lost before snapshot update".to_string(),
));
}
self.finalize_snapshot_write(snapshot, precondition, postcondition).await
}
async fn create_view(&self, entry: ViewEntry) -> TableCatalogStoreResult<()> {
let _migration_guard = self.acquire_snapshot_write_permit().await?;
let _write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let publication = TableCommitLockPublication::new(&self.object_backend);
self.create_view_with_publication(entry, &publication).await
}
async fn create_view_with_publication(
&self,
entry: ViewEntry,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<()> {
validate_view_entry_version_and_id(&entry)?;
validate_view_warehouse_location(&entry.table_bucket, &entry.warehouse_location)?;
let namespace = parse_namespace_for_store(&entry.namespace)?;
@@ -2663,7 +2702,26 @@ where
"view metadata location must be inside the view metadata directory".to_string(),
));
}
publication.begin_table_bucket(&entry.table_bucket).await?;
if !publication.holds_table_bucket(&entry.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view creation requires a table-bucket publication fence".to_string(),
));
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
let _migration_guard = self.acquire_snapshot_write_permit().await?;
publication
.prepare(&entry.table_bucket, &entry.namespace, &entry.view)
.await?;
if !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.view) {
return Err(TableCatalogStoreError::Internal(
"view creation requires a table publication fence".to_string(),
));
}
let _write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let key = Self::table_key(&entry.table_bucket, &namespace, &view);
let publication_identity = (entry.table_bucket.clone(), entry.namespace.clone(), entry.view.clone());
let (snapshot, precondition, postcondition) = {
let state = self.state.lock().await;
Self::require_table_bucket_in_state(&state, &entry.table_bucket)?;
@@ -2682,6 +2740,13 @@ where
StrongSnapshotWritePostcondition::ViewPresent(entry),
)
};
if !publication.holds_table_bucket(&publication_identity.0)
|| !publication.holds_table(&publication_identity.0, &publication_identity.1, &publication_identity.2)
{
return Err(TableCatalogStoreError::Internal(
"view creation publication fence was lost before snapshot update".to_string(),
));
}
self.finalize_snapshot_write(snapshot, precondition, postcondition).await
}
@@ -2751,11 +2816,38 @@ where
}
async fn replace_view(&self, request: ViewCommitRequest) -> TableCatalogStoreResult<ViewCommitResult> {
let publication = TableCommitLockPublication::new(&self.object_backend);
self.replace_view_with_publication(request, true, &publication).await
}
async fn replace_view_with_publication(
&self,
request: ViewCommitRequest,
table_bucket_fence_required: bool,
publication: &(dyn TableCommitPublication + Sync),
) -> TableCatalogStoreResult<ViewCommitResult> {
if table_bucket_fence_required {
publication.begin_table_bucket(&request.table_bucket).await?;
if !publication.holds_table_bucket(&request.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a table-bucket publication fence".to_string(),
));
}
}
let _publication_completion = TableCommitPublicationCompletion::new(publication);
let _migration_guard = self.acquire_snapshot_write_permit().await?;
let write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let namespace = parse_namespace_for_store(&request.namespace)?;
let view = parse_table_for_store(&request.view)?;
publication
.prepare(&request.table_bucket, &request.namespace, &request.view)
.await?;
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.view) {
return Err(TableCatalogStoreError::Internal(
"view replacement requires a table publication fence".to_string(),
));
}
let write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let key = Self::table_key(&request.table_bucket, &namespace, &view);
let expected_view_id = {
let state = self.state.lock().await;
@@ -2798,7 +2890,7 @@ where
let _write_guard = self.write_lock.lock().await;
self.hydrate_state().await?;
let (snapshot, precondition, next, postcondition) = {
let (snapshot, precondition, next, postcondition, warehouse_relocation) = {
let state = self.state.lock().await;
Self::ensure_identifier_is_unambiguous_locked(&state, &key)?;
let Some(current) = state.views.get(&key).cloned() else {
@@ -2828,6 +2920,14 @@ where
"current view metadata location does not match expected location".to_string(),
));
}
let warehouse_relocation = next_warehouse_location
.as_deref()
.is_some_and(|location| location != current.warehouse_location);
if warehouse_relocation && !publication.holds_table_bucket(&request.table_bucket) {
return Err(TableCatalogStoreError::Internal(
"view warehouse relocation requires a table-bucket publication fence".to_string(),
));
}
let mut next = current;
next.metadata_location = request.new_metadata_location;
@@ -2843,8 +2943,16 @@ where
precondition,
next.clone(),
StrongSnapshotWritePostcondition::ViewPresent(next),
warehouse_relocation,
)
};
if !publication.holds_table(&request.table_bucket, &request.namespace, &request.view)
|| ((table_bucket_fence_required || warehouse_relocation) && !publication.holds_table_bucket(&request.table_bucket))
{
return Err(TableCatalogStoreError::Internal(
"view replacement publication fence was lost before snapshot update".to_string(),
));
}
self.finalize_snapshot_write(snapshot, precondition, postcondition).await?;
Ok(ViewCommitResult { view: next })
}
+132 -19
View File
@@ -55,23 +55,35 @@ pub(crate) fn table_metadata_json(table_uuid: &str, location: &str) -> serde_jso
})
}
pub(crate) fn manifest_list_avro_bytes(manifest_paths: &[&str], sequence_number: i64, snapshot_id: i64) -> Vec<u8> {
let manifests = manifest_paths
.iter()
.map(|manifest_path| (*manifest_path, 0, sequence_number, snapshot_id))
.collect::<Vec<_>>();
manifest_list_avro_entries_with_partition_specs(&manifests)
}
pub(crate) fn manifest_list_avro_entries(manifests: &[(&str, i64, i64)]) -> Vec<u8> {
pub(crate) fn manifest_list_avro_bytes(manifests: &[(&str, usize)], sequence_number: i64, snapshot_id: i64) -> Vec<u8> {
let manifests = manifests
.iter()
.map(|(manifest_path, sequence_number, snapshot_id)| (*manifest_path, 0, *sequence_number, *snapshot_id))
.map(|(manifest_path, manifest_length)| (*manifest_path, *manifest_length, 0, sequence_number, snapshot_id))
.collect::<Vec<_>>();
manifest_list_avro_entries_with_partition_specs(&manifests)
}
pub(crate) fn manifest_list_avro_entries_with_partition_specs(manifests: &[(&str, i32, i64, i64)]) -> Vec<u8> {
pub(crate) fn manifest_list_avro_entries(manifests: &[(&str, usize, i64, i64)]) -> Vec<u8> {
let manifests = manifests
.iter()
.map(|(manifest_path, manifest_length, sequence_number, snapshot_id)| {
(*manifest_path, *manifest_length, 0, *sequence_number, *snapshot_id)
})
.collect::<Vec<_>>();
manifest_list_avro_entries_with_partition_specs(&manifests)
}
pub(crate) fn manifest_list_avro_entries_with_partition_specs(manifests: &[(&str, usize, i32, i64, i64)]) -> Vec<u8> {
let manifests = manifests
.iter()
.map(|(path, length, spec_id, sequence_number, snapshot_id)| {
(*path, *length, *spec_id, 0, *sequence_number, *snapshot_id)
})
.collect::<Vec<_>>();
manifest_list_avro_entries_with_content(&manifests)
}
pub(crate) fn manifest_list_avro_entries_with_content(manifests: &[(&str, usize, i32, i32, i64, i64)]) -> Vec<u8> {
let schema = apache_avro::Schema::parse_str(
r#"
{
@@ -97,16 +109,19 @@ pub(crate) fn manifest_list_avro_entries_with_partition_specs(manifests: &[(&str
)
.expect("manifest list avro schema should parse");
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest list writer should initialize");
for (manifest_path, partition_spec_id, sequence_number, snapshot_id) in manifests {
for (manifest_path, manifest_length, partition_spec_id, content, sequence_number, snapshot_id) in manifests {
writer
.append_value(apache_avro::types::Value::Record(vec![
(
"manifest_path".to_string(),
apache_avro::types::Value::String((*manifest_path).to_string()),
),
("manifest_length".to_string(), apache_avro::types::Value::Long(1)),
(
"manifest_length".to_string(),
apache_avro::types::Value::Long(i64::try_from(*manifest_length).expect("test manifest length should fit")),
),
("partition_spec_id".to_string(), apache_avro::types::Value::Int(*partition_spec_id)),
("content".to_string(), apache_avro::types::Value::Int(0)),
("content".to_string(), apache_avro::types::Value::Int(*content)),
("sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
("min_sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
("added_snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
@@ -122,7 +137,86 @@ pub(crate) fn manifest_list_avro_entries_with_partition_specs(manifests: &[(&str
writer.into_inner().expect("manifest list avro bytes should flush")
}
pub(crate) fn manifest_list_avro_entries_with_nullable_counts(manifests: &[(&str, usize, i32, i64, i64)]) -> Vec<u8> {
let schema = apache_avro::Schema::parse_str(
r#"
{
"type": "record",
"name": "manifest_file",
"fields": [
{"name": "manifest_path", "type": "string"},
{"name": "manifest_length", "type": "long"},
{"name": "partition_spec_id", "type": "int"},
{"name": "content", "type": "int"},
{"name": "sequence_number", "type": "long"},
{"name": "min_sequence_number", "type": "long"},
{"name": "added_snapshot_id", "type": "long"},
{"name": "added_files_count", "type": ["null", "int"], "default": null},
{"name": "existing_files_count", "type": ["null", "int"], "default": null},
{"name": "deleted_files_count", "type": ["null", "int"], "default": null},
{"name": "added_rows_count", "type": ["null", "long"], "default": null},
{"name": "existing_rows_count", "type": ["null", "long"], "default": null},
{"name": "deleted_rows_count", "type": ["null", "long"], "default": null}
]
}
"#,
)
.expect("manifest list avro schema should parse");
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest list writer should initialize");
for (manifest_path, manifest_length, partition_spec_id, sequence_number, snapshot_id) in manifests {
writer
.append_value(apache_avro::types::Value::Record(vec![
(
"manifest_path".to_string(),
apache_avro::types::Value::String((*manifest_path).to_string()),
),
(
"manifest_length".to_string(),
apache_avro::types::Value::Long(i64::try_from(*manifest_length).expect("test manifest length should fit")),
),
("partition_spec_id".to_string(), apache_avro::types::Value::Int(*partition_spec_id)),
("content".to_string(), apache_avro::types::Value::Int(0)),
("sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
("min_sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
("added_snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
(
"added_files_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
(
"existing_files_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
(
"deleted_files_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
(
"added_rows_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
(
"existing_rows_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
(
"deleted_rows_count".to_string(),
apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
),
]))
.expect("manifest list record should append");
}
writer.into_inner().expect("manifest list avro bytes should flush")
}
pub(crate) fn manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u8> {
manifest_avro_bytes_with_partition_spec(files, None)
}
pub(crate) fn manifest_avro_bytes_with_partition_spec(
files: &[(&str, i32, i32, i64, i64)],
partition_spec_id: Option<i32>,
) -> Vec<u8> {
let schema = apache_avro::Schema::parse_str(
r#"
{
@@ -141,6 +235,7 @@ pub(crate) fn manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u
"fields": [
{"name": "content", "type": "int"},
{"name": "file_path", "type": "string"},
{"name": "partition", "type": {"type": "record", "name": "partition", "fields": []}},
{"name": "record_count", "type": "long"},
{"name": "file_size_in_bytes", "type": "long"}
]
@@ -152,6 +247,11 @@ pub(crate) fn manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u
)
.expect("manifest avro schema should parse");
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
if let Some(partition_spec_id) = partition_spec_id {
writer
.add_user_metadata("partition-spec-id".to_string(), partition_spec_id.to_string())
.expect("manifest partition spec metadata should write");
}
for (file_path, content, status, snapshot_id, sequence_number) in files {
writer
.append_value(apache_avro::types::Value::Record(vec![
@@ -164,6 +264,7 @@ pub(crate) fn manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u
apache_avro::types::Value::Record(vec![
("content".to_string(), apache_avro::types::Value::Int(*content)),
("file_path".to_string(), apache_avro::types::Value::String((*file_path).to_string())),
("partition".to_string(), apache_avro::types::Value::Record(Vec::new())),
("record_count".to_string(), apache_avro::types::Value::Long(1)),
("file_size_in_bytes".to_string(), apache_avro::types::Value::Long(1)),
]),
@@ -200,6 +301,7 @@ pub(crate) fn manifest_avro_bytes_with_nullable_sequences(files: &[(&str, i32, i
"fields": [
{"name": "content", "type": "int"},
{"name": "file_path", "type": "string"},
{"name": "partition", "type": {"type": "record", "name": "partition", "fields": []}},
{"name": "record_count", "type": "long"},
{"name": "file_size_in_bytes", "type": "long"}
]
@@ -223,6 +325,7 @@ pub(crate) fn manifest_avro_bytes_with_nullable_sequences(files: &[(&str, i32, i
apache_avro::types::Value::Record(vec![
("content".to_string(), apache_avro::types::Value::Int(*content)),
("file_path".to_string(), apache_avro::types::Value::String((*file_path).to_string())),
("partition".to_string(), apache_avro::types::Value::Record(Vec::new())),
("record_count".to_string(), apache_avro::types::Value::Long(1)),
("file_size_in_bytes".to_string(), apache_avro::types::Value::Long(1)),
]),
@@ -284,7 +387,7 @@ pub(crate) struct BlockingObjectPublication {
backend: TestCatalogObjectBackend,
object: String,
started: Arc<tokio::sync::Notify>,
guard: Arc<parking_lot::Mutex<Option<Box<dyn Send>>>>,
guard: Arc<parking_lot::Mutex<Option<TableCatalogLockGuard>>>,
}
impl BlockingObjectPublication {
@@ -812,7 +915,7 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend {
.collect())
}
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_write_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard> {
self.lock_attempts.lock().await.push((bucket.to_string(), object.to_string()));
{
let mut state = self.state.lock().await;
@@ -828,10 +931,10 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend {
.or_insert_with(|| std::sync::Arc::new(tokio::sync::RwLock::new(())))
.clone()
};
Ok(Box::new(lock.write_owned().await))
Ok(TableCatalogLockGuard::stable(lock.write_owned().await))
}
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Box<dyn Send>> {
async fn acquire_read_lock(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<TableCatalogLockGuard> {
// The admin fake implemented only acquire_write_lock, so the trait's
// default read->write delegation made read acquisitions observable in
// lock_attempts as well; keep that (backlog#1837 PR2).
@@ -850,7 +953,7 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend {
.or_insert_with(|| std::sync::Arc::new(tokio::sync::RwLock::new(())))
.clone()
};
Ok(Box::new(lock.read_owned().await))
Ok(TableCatalogLockGuard::stable(lock.read_owned().await))
}
}
@@ -1165,6 +1268,8 @@ pub(crate) struct TestTableCatalogStore {
pub(crate) fail_put_table_bucket: tokio::sync::Mutex<bool>,
pub(crate) register_table_pause: Option<TestCatalogPublishPause>,
pub(crate) commit_table_pause: Option<TestCatalogPublishPause>,
pub(crate) create_view_pause: Option<TestCatalogPublishPause>,
pub(crate) replace_view_pause: Option<TestCatalogPublishPause>,
}
#[async_trait::async_trait]
@@ -1473,6 +1578,10 @@ impl crate::table_catalog::TableCatalogStore for TestTableCatalogStore {
entry.table_bucket, entry.namespace
)));
}
if let Some(pause) = &self.create_view_pause {
pause.started.notify_one();
pause.release.notified().await;
}
self.views.lock().await.push(entry);
Ok(())
}
@@ -1531,6 +1640,10 @@ impl crate::table_catalog::TableCatalogStore for TestTableCatalogStore {
"current view metadata location does not match expected location".to_string(),
));
}
if let Some(pause) = &self.replace_view_pause {
pause.started.notify_one();
pause.release.notified().await;
}
let mut next = current;
next.metadata_location = request.new_metadata_location;
next.version_token = "token-view-committed".to_string();
File diff suppressed because it is too large Load Diff
+4 -12
View File
@@ -132,18 +132,8 @@ def failure_probe_plan(warehouse: str, namespace: str, table: str, rest_path: st
"expected-version-token": "stale-token-from-previous-load",
"expected-metadata-location": "current-metadata-location-from-load-table",
"new-metadata-location": f"s3://{warehouse}/tables/table-id/metadata/conflict_probe.metadata.json",
"requirements": [
{
"type": "assert-current-snapshot-id",
"snapshot-id": 0,
}
],
"updates": [
{
"action": "set-current-schema",
"schema-id": 0,
}
],
"requirements": [],
"updates": [],
},
),
probe_step(
@@ -157,6 +147,8 @@ def failure_probe_plan(warehouse: str, namespace: str, table: str, rest_path: st
"expected-version-token": "current-version-token-from-load-table",
"expected-metadata-location": "current-metadata-location-from-load-table",
"new-metadata-location": f"s3://{warehouse}/tables/table-id/metadata/does_not_exist.metadata.json",
"requirements": [],
"updates": [],
},
),
probe_step(
@@ -48,6 +48,8 @@ class FailureCoverageTest(unittest.TestCase):
self.assertIn("expected-version-token", by_name["stale-token-commit-conflict"]["body"])
self.assertIn("expected-metadata-location", by_name["stale-token-commit-conflict"]["body"])
self.assertIn("new-metadata-location", by_name["stale-token-commit-conflict"]["body"])
self.assertEqual(by_name["stale-token-commit-conflict"]["body"]["requirements"], [])
self.assertEqual(by_name["stale-token-commit-conflict"]["body"]["updates"], [])
self.assertNotIn("base", by_name["stale-token-commit-conflict"]["body"])
self.assertEqual(
by_name["diagnostics-after-finalization-gap"]["path"],
@@ -56,6 +58,8 @@ class FailureCoverageTest(unittest.TestCase):
self.assertEqual(by_name["diagnostics-after-finalization-gap"]["method"], "GET")
self.assertEqual(by_name["recovery-repairs-idempotency-index"]["method"], "POST")
self.assertIn("does_not_exist.metadata.json", json.dumps(by_name["missing-metadata-object-rejected"]))
self.assertEqual(by_name["missing-metadata-object-rejected"]["body"]["requirements"], [])
self.assertEqual(by_name["missing-metadata-object-rejected"]["body"]["updates"], [])
self.assertNotIn("base", by_name["missing-metadata-object-rejected"]["body"])
def test_cli_prints_failure_matrix_and_probe_plan(self) -> None: