* test(odm): add the fault, concurrency, interaction and real-source e2e
Twenty-one scenarios for on-demand migration: source failures and the
circuit breaker, single-flight and the pull-concurrency limits, how a
pulled object composes with encryption, Object Lock, quota, events,
replication, delete markers and the disable switch, and two cases against
a real second RustFS as the source.
Every assertion carries a source-request-count dimension so a case cannot
pass by serving the client while quietly re-reading the source.
* ci(odm): route the slow on-demand-migration e2e to the nightly lane
The fault, concurrency and real-source modules wait out the 30 s breaker
window, drive 100-deep bursts, or start extra RustFS processes, so they
join e2e-nightly and are subtracted from the e2e-full merge budget. The
e2e-smoke selection is unchanged.
feat(rustfs): serve GET misses from the migration source
Wire the on-demand migration read-through into the GET path, after the
local read and the replication proxy have both missed (rustfs/backlog#2156).
A source HEAD supplies size, validators and metadata. Conditional headers
are evaluated locally against it and never forwarded, so a source 304/412
cannot be mistaken for a source failure. An object within inline_max_bytes
is teed: the primary streams to the client while the secondary commits the
local copy in a background task, so a client disconnect still stores the
whole object and a failed write-back never touches the client stream.
Range reads and larger objects stream straight through and queue a
background pull per policy. Concurrent misses of one key share the
singleflight slot: the leader tees, followers re-read local after it
commits or degrade to passthrough after first_byte_ms.
Version reads, partNumber reads, anti-loop marked requests and a respected
local delete marker keep their original 404. Source answers carry
x-rustfs-on-demand-migration: source; local hits are untouched, and the
local hit path gains no await or lock.
The guard added in #7021 fails a >5 GiB single-PutObject replication up
front instead of streaming the body to a target that must reject it. Its
message asserted a conclusion: "was not written as multipart on the
source ... re-upload it with multipart". That text is only as right as
the transport decision feeding it, and until #7047 that decision was
wrong for multipart objects carrying a full-object checksum. On 1.0.0-rc.5
a 768-part object was misrouted to the single-PUT path, and the new
default-level error line told the operator to re-upload as multipart an
object whose own ETag ended in -768.
State the evidence instead of the conclusion. The message now quotes the
ETag the decision was read from and says what was read from it (no
part-count suffix), so an operator can check the line against the
object's listing. A misroute then reads as a visible contradiction --
a suffixed ETag on a single-PUT line -- and the message says that case is
a transport-selection defect to report, not something to fix by
re-uploading. A missing or empty ETag is printed as <none> rather than
hidden.
The routing itself is already fixed by #7047; this changes only what the
guard says when it fires.
* fix(ecstore): persist merged checksum type for full-object multipart
complete_multipart_upload built the object-level checksum record from a
ChecksumType copied before the MULTIPART / INCLUDES_MULTIPART flags were
merged in. ChecksumType::merge takes &mut self, so the merge updated the
local variable while the copy already inside the Checksum struct stayed
behind. The composite branch rebuilt the Checksum from the merged type
and was unaffected; the full-object branch never rebuilt it, so those
flags never reached disk.
rustfs_rio::read_checksums only sets its multipart flag and only emits
the "x-amz-checksum-type" = "FULL_OBJECT" entry inside its MULTIPART
branch, so a full-object multipart object read back as non-multipart with
no type entry, and GetObject and HeadObject answered with no
x-amz-checksum-type header at all where AWS returns FULL_OBJECT.
Hand the full-object branch the merged type instead of rebuilding the
Checksum: the value must stay the running merge produced by add_part,
because hashing the concatenated part digests would yield the COMPOSITE
value, a different number than the one the client sent. The serialization
now lives in multipart_object_checksum_record so both shapes are covered
by unit tests.
Records written by earlier builds carry the bare algorithm type with no
MULTIPART flags and no trailing part block; they keep reading back to the
same checksum value, and the FULL_OBJECT reader arm predates this change
so older peers parse the new record shape correctly too.
Found while root-causing rustfs#6825.
* fix(s3): reject contradicting multipart checksum type as client error
A CompleteMultipartUpload declaring an x-amz-checksum-type that
contradicts the type recorded at CreateMultipartUpload answered 500
InternalError, telling the caller to retry a request that can only ever
fail. The storage layer does refuse the combination, but through a
generic error that maps to InternalError.
Validate the header against the recorded type in the usecase, where the
upload metadata returned by get_multipart_info is already in hand, and
answer InvalidRequest naming both types, matching AWS. The storage-layer
check stays as a backstop for non-HTTP callers.
Uploads created without a checksum algorithm record no type, so there is
nothing to contradict and the header is left alone rather than newly
rejected. Replication is unaffected: replication_put_object_options
already excludes x-amz-checksum-type from the metadata it forwards.
* test(e2e): cover full-object multipart checksum type round-trip
Adds an end-to-end test that a CRC32 FULL_OBJECT multipart upload reports
x-amz-checksum-type: FULL_OBJECT and the unsuffixed full-object value on
both GetObject and HeadObject, and one that a CompleteMultipartUpload
contradicting the recorded type is rejected as InvalidRequest while
leaving the upload intact. Extends the existing CRC64NVME multipart test
with the same checksum-type assertion.
* fix(s3): keep checksum-type validation off the s3s error macro
The s3s footprint ratchet (scripts/check_s3s_footprint.sh) counts
s3_error! invocation lines and is lower-only: new code must route
through the gateway abstractions rather than widen the direct s3s
surface the s3gate migration is shrinking.
Raise the contradiction through ApiError::invalid_request instead. The
response is byte-for-byte identical -- From<ApiError> for S3Error carries
the InvalidRequest code and the message through unchanged -- and the
usecase already returns ApiError elsewhere, so this is the idiomatic
path rather than a way around the counter.
The explanatory comment deliberately says "the s3s error macro" instead
of naming the macro: the ratchet counts raw matches, so spelling it out
in a comment tripped the same check.
* feat(obs): export on-demand migration bucket metrics
Add the on_demand_migration metric subsystem: per-bucket request,
pull, failure, inflight, queue depth, source latency distribution and
breaker state series fed from the ODM runtime snapshot through the
storage boundary, collected alongside bucket replication metrics, and
retired once a bucket's config disappears.
* feat(admin): report the full on-demand migration status snapshot
Extend GET /v3/on-demand-migration/{bucket}/status with provider,
endpoint host, breaker state, runtime counters, last source error,
inflight and queue gauges and the config timestamp. served_by_source_ratio
stays null: no per-bucket GET total exists to divide by. Update the
madmin status type and golden fixture together.
* feat(ecstore): add on-demand migration pull queue and write-back pipeline
Background pull queue per bucket (bounded by pull_queue_capacity, concurrency via the state's pull slot), OdmWriteBack/PullSource traits, single-part and multipart write-back with a pumped body that enforces idle timeout, cancel and content length, retry policy for retryable source errors, inline commit helper, and stats accounting (rustfs/backlog#2153).
* feat(object): implement on-demand migration write-back over internal put
OdmWriteBack impl mapping source heads onto InternalPutContext (content-header allowlist, x-amz-meta copy, tags, dual-prefix odm-* provenance, ETag policy), injected into OnDemandMigrationSys at startup; removes the dead-code gates left by ODM-06a (rustfs/backlog#2153).
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(admin): add on-demand migration bucket admin API
Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.
- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
ODM-01 config against local endpoints and replication targets, probes the
source with SourceClient::probe(), then persists through the incarnation
gate and asks peers to reload. Responses carry the redacted config and a
probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
admin:GetBucketOnDemandMigration, route policy matrix rows, registration
and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
asserted byte-for-byte by both the handler and the client tests.
Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.
* refactor(admin): route on-demand migration handler errors through the s3 facade
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(ecstore): add on-demand migration runtime OnDemandMigrationSys
Per-node runtime for On-Demand Migration (rustfs/backlog#2152): turns each
bucket's persisted config into a live SourceClient guarded by a three-state
circuit breaker, a TTL negative cache, per-key singleflight, a pull
concurrency semaphore and lock-free counters with a serializable snapshot.
- sys.rs: OnceLock singleton; `apply` installs/rebuilds/removes bucket state
(config compared by value, counters preserved across rebuilds, old
cancellation token fired); `publish` is the metadata publish-hook entry
(sync removal, spawned install, generation-ordered so a slow older install
cannot overwrite a newer one); `resolve(bucket, key)` judges module switch,
bucket state, prefix filter, client availability, negative cache, breaker.
- breaker.rs: Closed/Open/HalfOpen with fixed constants (5 failures / 30 s
window / 30 s open / 1 probe); NotFound resets, AccessDenied is neutral.
- negative_cache.rs: moka sync cache keyed by local key, ttl=0 disables.
- stats.rs: requests_total{op,outcome}, pulled_bytes/objects, pull_failures,
inflight/queue gauges, log-bucket latency histogram, last_source_error;
snake_case snapshot pinned by a golden JSON test.
- Anonymous sources surface as a typed `OdmStateError::AnonymousUnsupported`
until the shared client builder gains an anonymous mode.
- rustfs: `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (default false)
published to module_switches and injected into ecstore before bucket
metadata loads; hook registered at the same point.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* test(e2e): rename stall timing variable flagged by typos
* test(storage): heap-pin the RestoreObject usecase future in the generation guard test
Add `tee_reader` / `tee_reader_with_options` in `rustfs-rio`: a
`TeePrimary` that drives the source and a `TeeSecondary` that observes
an identical copy of every chunk through a byte-bounded queue. The
primary returns `Pending` when the queue is full, so both sides advance
at the pace of the slowest consumer; it is meant for small objects only.
Termination: source EOF and errors propagate to the secondary with the
same `io::ErrorKind`; dropping the secondary turns the primary into a
pass-through; dropping the primary early fails the secondary with
`BrokenPipe` by default, or hands the remaining source to a background
drain task bounded by `max_drain_bytes` when
`TeeOptions::drain_on_primary_drop` is set. `TeeSecondary::into_stream`
exposes the queued `Bytes` chunks without an extra copy.
Includes a proptest equivalence test, backpressure, error, drop,
drain-limit and cancel-safety tests, and a criterion bench comparing
tee throughput against a direct read (64 MiB in 1 MiB chunks).
* test(e2e): extend fake S3 target as an on-demand migration source
Add ListObjectsV2 paging, Range GET/HEAD, unversioned buckets, standard
and user metadata replay, ResponseStatus/TruncateBodyAt/Stall fault
actions, Range/User-Agent/prefix/continuation-token journal fields,
count_requests, direct seeding, and a configurable object cap to the
programmable fake S3 target, and add the on_demand_migration e2e
harness (OdmTestEnv, admin wrappers, source seeding, local-state
assertions, second RustFS source) with its self-test.
* test(ci): refresh darwin e2e-full selection for ODM harness
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
A 6 GiB object uploaded to the source as a 768-part multipart upload was
replicated to a generic S3 target with a single PutObject, and the target
rejected the body with EntityTooLarge. No CreateMultipartUpload was ever
issued, so the multipart replication transport never ran for the object
it exists for.
`replication_put_object_options` seeded the transport from
`object_info.is_multipart()` and then overwrote it with the second
return value of `decrypt_checksums`. Those two booleans do not mean the
same thing: the first is the object's storage shape, read from the ETag,
while the second reports whether the stored *checksum record* carries
per-part data. A full-object checksum -- what `aws s3 cp` writes by
default for a CRC algorithm -- is serialized with no MULTIPART flag even
on a multipart upload, so the record reports false and the object was
routed as a single PUT. `decrypt_checksums` documents this in
object_api/types.rs: callers that need routing must consult
`is_multipart()`. Replication did the opposite.
Route on the object's own shape, and let the checksum record only add
multipart-ness, never take it away. Objects already stored with such a
record are fixed too: the ETag was always right.
This also repairs the diagnosis of rustfs#6825, where the single-PUT
5 GiB guard fired against an object that was multipart all along and
told the operator to re-upload it as multipart.
Tests cover the three shapes the router has to separate: a multipart
object with a full-object checksum record (the regression, which fails
without this change), a multipart object with a composite record, and a
single-part object that must not be promoted onto multipart.
DeleteBucket answers from a raw per-disk residue scan rather than from a
listing, so it can refuse for a reason no S3 request can observe: the
client drains every version the API will show, DeleteBucket still returns
BucketNotEmpty, and the client-visible message is the generic "The bucket
you tried to delete is not empty" for every blocker kind.
The server does know which residue blocked it, and where — that is what
`bucket_delete_blocked` carries. But it was emitted at `debug`, below
both the `error` DEFAULT_LOG_LEVEL and the `info` the CI s3-tests lane
runs at, so it was never actually written down. An intermittent
BucketNotEmpty in that lane leaves a server log with no trace of the
refusal at all, which is not a diagnosable state: confirmed against the
artifact log of a failing run, where the rejected bucket appears only in
span-close lines and the blocker event is absent entirely.
Split the blocker kinds by whether the client can still reach the
residue. A visible version or a tier free-version is an ordinary 409 —
the bucket really is not empty and the caller can list and delete what is
left — so that stays at `warn`. UnknownXlMeta, OrphanDirectory, and
DiagnosticBudgetExceeded are on-disk state no S3 request can remove; that
is a server-side integrity problem and is now reported at `error`, with
the blocker kind, the residue counts, and the sample path.
This does not change what DeleteBucket accepts or rejects, and does not
retry or suppress anything — it makes the existing diagnosis reachable.
Refs #7005, #7010
Update the workspace s3s pin and refresh the lockfile with cargo update/upgrade.
Remove the unused lifecycle url dependency reported by cargo shear.
Tighten the s3s footprint ratchet to the current observed baseline.
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
`build_metrics_summary` emitted a single metric entry for the local
deployment with `online` hardcoded to `true` and `last_online` stamped
with the current time, so `mc admin replicate status` reported "I am
online" rather than whether the remote site was reachable. A peer could
be down for minutes with replication failing while the status page
stayed green, leaving operators with no signal that the link had
dropped.
Emit an entry for every peer instead, deriving `online` from the
`reachable_peers` set the handler already computes by probing each peer,
and take `total_downtime`/`last_online` from the replication heartbeat's
existing `EpHealth` tracking. Node-local replication counters stay on
the local entry so a two-site cluster does not double-count its own
traffic.
The new `BucketTargetSys::endpoint_health` accessor deliberately does not
call `init_hc`: unlike `is_offline` it must not create health entries as
a side effect, or merely rendering the status page would mark an unknown
peer online.
Failure counters (`Errors`) are unchanged and still read zero; that is a
separate defect in the bucket-level statistics path and is not addressed
here.
Six set_disk::ops tests failed non-deterministically only under
concurrent full-suite load, rotating between runs while each passed in
isolation. All six share one root cause: a lock-owning put_object
quorum-acks once the rename fanout reaches write quorum and lets a
detached tail task finish the lagging disks, so a fixture that inspects
per-disk state immediately after PUT can observe a disk the tail has not
reached yet.
The two heal report fixtures, the inline-commit fixture, and the
transaction-fencing fixture read or delete physical shards right after
PUT, and hit FileNotFound on a lagging disk. The two metadata-cache
fixtures prime the cache after PUT, and the read fanout refuses to publish
a cache entry while any disk still reports an error, so the priming read
observably published nothing.
Keep every affected setup PUT on the full-fanout commit path with
no_lock: true, following the existing precedent in this module, so PUT
returns only after every disk has committed. The option only governs lock
acquisition, so it does not weaken what any of these fixtures assert; the
transaction-fencing gate in particular is driven by the fleet proof and
env vars, never by the lock option. Where a fixture also depends on cache
publication, re-prime until the current generation is observably cached
instead of asserting on a single read that a loaded host can stall past
the cache TTL. The heal race fixture's shard damage injection is
best-effort by construction, so it now skips injection when the previous
round's tail still lags rather than unwrapping a read that may
legitimately race.
No production code changes, and no retries or sleeps added.
* test(ecstore): retain final decommission capacity snapshot override
take_decommission_capacity_info_override_for_test used to pop the queue
to exhaustion, after which get_decommission_all_pool_capacity_infos
silently fell back to the host's real statfs numbers. Any new sampling
point added to the decommission start paths re-introduced that host
dependency and broke tests on some dev machines (#6989 patched one
instance by topping up snapshot counts, but the coupling remained).
Keep the final queued snapshot and replay it for every subsequent
sample so tests always observe injected capacity once an override is
installed. All existing injection patterns (single snapshot, repeated
identical snapshots, decreasing sequences ending at the post-operation
state) keep their semantics.
* test(ci): serialize load-sensitive heal and cache-generation tests
Under a heavily parallel nextest run (~792 ecstore tests), two tests of
set_disk::ops::heal::heal_result_report_tests failed nondeterministically
per round (different members each time; all 29 pass standalone). Every
test in the module builds a TempDir-backed 4-disk hermetic erasure set
and drives MiB-scale writes plus deep-scan heal: under load a single
disk's IO can fail while write quorum still holds, flipping per-disk
readback and aggregate-outcome assertions. The module's #[serial]
markers do not serialize across nextest's process-per-test boundary.
Verification also caught complete_multipart_generation_retires_cached_snapshot
failing once under the same load; it and its object.rs sibling carry
#[serial(metadata_cache_invalidation_probe)] and assert
get_object_metadata_cache generation semantics - the same shape that
forced the transition matrix tests into the serial group.
Add both families to the ecstore-serial-flaky test-group in the default
and ci profiles. Preventive serialization only, no retries. Three full
parallel rounds after the change: 792/792 passed each round.
A completed multipart upload's staging cleanup prunes empty parent
directories up to the volume root, which removes shared prefixes such as
`data-movement/` and the per-object `<sha>/` while a concurrent
new_multipart_upload builds its destination chain below them. The writer
holds a descriptor to the pruned component, so its next handle-relative
mkdirat fails NotFound. Because rename never retried NotFound, the cleanup
fan-out failed several disks in the same window and broke write quorum.
Give rename preparation its own retry rule: a NotFound is retried once per
component below the base directory, so a rebuilt walk outlasts a pruning
walk, which removes ancestors monotonically upward and stops at the base.
A destination whose parent is the base keeps NotFound terminal, so
speculative cleanup renames still fail fast, and the base is only ever
opened, never created, so a genuinely missing base still fails. The rename
itself keeps its own budget and its unchanged NotFound-is-terminal rule.
The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
Zhengchao An