* feat(obs): export on-demand migration bucket metrics
Add the on_demand_migration metric subsystem: per-bucket request,
pull, failure, inflight, queue depth, source latency distribution and
breaker state series fed from the ODM runtime snapshot through the
storage boundary, collected alongside bucket replication metrics, and
retired once a bucket's config disappears.
* feat(admin): report the full on-demand migration status snapshot
Extend GET /v3/on-demand-migration/{bucket}/status with provider,
endpoint host, breaker state, runtime counters, last source error,
inflight and queue gauges and the config timestamp. served_by_source_ratio
stays null: no per-bucket GET total exists to divide by. Update the
madmin status type and golden fixture together.
* feat(ecstore): add on-demand migration pull queue and write-back pipeline
Background pull queue per bucket (bounded by pull_queue_capacity, concurrency via the state's pull slot), OdmWriteBack/PullSource traits, single-part and multipart write-back with a pumped body that enforces idle timeout, cancel and content length, retry policy for retryable source errors, inline commit helper, and stats accounting (rustfs/backlog#2153).
* feat(object): implement on-demand migration write-back over internal put
OdmWriteBack impl mapping source heads onto InternalPutContext (content-header allowlist, x-amz-meta copy, tags, dual-prefix odm-* provenance, ETag policy), injected into OnDemandMigrationSys at startup; removes the dead-code gates left by ODM-06a (rustfs/backlog#2153).
feat(object): proxy HEAD misses to the on-demand migration source
Add the ODM HEAD passthrough (rustfs/backlog#2155): after the local lookup
and the replication proxy both miss, resolve the bucket through
OnDemandMigrationSys and answer from the source's HEAD without writing back
or queueing a pull. Versioned reads, requests carrying the source-proxy
anti-loop marker, and a respected latest delete marker never consult the
source; policy.head=local_only, the negative cache and an open breaker
answer 404 locally. Source 404 feeds the negative cache; other source
failures map to 424 SourceUnavailable (class only) or 404 per
policy.source_error, and unsupported source objects always map to 424.
Source answers carry x-rustfs-on-demand-migration: source and omit version,
SSE and replication headers.
The delete-marker probe, request gate, 424 constructor and response marker
live in shared.rs for the GET passthrough to reuse.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(admin): add on-demand migration bucket admin API
Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.
- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
ODM-01 config against local endpoints and replication targets, probes the
source with SourceClient::probe(), then persists through the incarnation
gate and asks peers to reload. Responses carry the redacted config and a
probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
admin:GetBucketOnDemandMigration, route policy matrix rows, registration
and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
asserted byte-for-byte by both the handler and the client tests.
Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.
* refactor(admin): route on-demand migration handler errors through the s3 facade
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(ecstore): add on-demand migration runtime OnDemandMigrationSys
Per-node runtime for On-Demand Migration (rustfs/backlog#2152): turns each
bucket's persisted config into a live SourceClient guarded by a three-state
circuit breaker, a TTL negative cache, per-key singleflight, a pull
concurrency semaphore and lock-free counters with a serializable snapshot.
- sys.rs: OnceLock singleton; `apply` installs/rebuilds/removes bucket state
(config compared by value, counters preserved across rebuilds, old
cancellation token fired); `publish` is the metadata publish-hook entry
(sync removal, spawned install, generation-ordered so a slow older install
cannot overwrite a newer one); `resolve(bucket, key)` judges module switch,
bucket state, prefix filter, client availability, negative cache, breaker.
- breaker.rs: Closed/Open/HalfOpen with fixed constants (5 failures / 30 s
window / 30 s open / 1 probe); NotFound resets, AccessDenied is neutral.
- negative_cache.rs: moka sync cache keyed by local key, ttl=0 disables.
- stats.rs: requests_total{op,outcome}, pulled_bytes/objects, pull_failures,
inflight/queue gauges, log-bucket latency histogram, last_source_error;
snake_case snapshot pinned by a golden JSON test.
- Anonymous sources surface as a typed `OdmStateError::AnonymousUnsupported`
until the shared client builder gains an anonymous mode.
- rustfs: `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (default false)
published to module_switches and injected into ecstore before bucket
metadata loads; hook registered at the same point.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* test(e2e): rename stall timing variable flagged by typos
* test(storage): heap-pin the RestoreObject usecase future in the generation guard test
Add `tee_reader` / `tee_reader_with_options` in `rustfs-rio`: a
`TeePrimary` that drives the source and a `TeeSecondary` that observes
an identical copy of every chunk through a byte-bounded queue. The
primary returns `Pending` when the queue is full, so both sides advance
at the pace of the slowest consumer; it is meant for small objects only.
Termination: source EOF and errors propagate to the secondary with the
same `io::ErrorKind`; dropping the secondary turns the primary into a
pass-through; dropping the primary early fails the secondary with
`BrokenPipe` by default, or hands the remaining source to a background
drain task bounded by `max_drain_bytes` when
`TeeOptions::drain_on_primary_drop` is set. `TeeSecondary::into_stream`
exposes the queued `Bytes` chunks without an extra copy.
Includes a proptest equivalence test, backpressure, error, drop,
drain-limit and cancel-safety tests, and a criterion bench comparing
tee throughput against a direct read (64 MiB in 1 MiB chunks).
* test(e2e): extend fake S3 target as an on-demand migration source
Add ListObjectsV2 paging, Range GET/HEAD, unversioned buckets, standard
and user metadata replay, ResponseStatus/TruncateBodyAt/Stall fault
actions, Range/User-Agent/prefix/continuation-token journal fields,
count_requests, direct seeding, and a configurable object cap to the
programmable fake S3 target, and add the on_demand_migration e2e
harness (OdmTestEnv, admin wrappers, source seeding, local-state
assertions, second RustFS source) with its self-test.
* test(ci): refresh darwin e2e-full selection for ODM harness
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
A 6 GiB object uploaded to the source as a 768-part multipart upload was
replicated to a generic S3 target with a single PutObject, and the target
rejected the body with EntityTooLarge. No CreateMultipartUpload was ever
issued, so the multipart replication transport never ran for the object
it exists for.
`replication_put_object_options` seeded the transport from
`object_info.is_multipart()` and then overwrote it with the second
return value of `decrypt_checksums`. Those two booleans do not mean the
same thing: the first is the object's storage shape, read from the ETag,
while the second reports whether the stored *checksum record* carries
per-part data. A full-object checksum -- what `aws s3 cp` writes by
default for a CRC algorithm -- is serialized with no MULTIPART flag even
on a multipart upload, so the record reports false and the object was
routed as a single PUT. `decrypt_checksums` documents this in
object_api/types.rs: callers that need routing must consult
`is_multipart()`. Replication did the opposite.
Route on the object's own shape, and let the checksum record only add
multipart-ness, never take it away. Objects already stored with such a
record are fixed too: the ETag was always right.
This also repairs the diagnosis of rustfs#6825, where the single-PUT
5 GiB guard fired against an object that was multipart all along and
told the operator to re-upload it as multipart.
Tests cover the three shapes the router has to separate: a multipart
object with a full-object checksum record (the regression, which fails
without this change), a multipart object with a composite record, and a
single-part object that must not be promoted onto multipart.
DeleteBucket answers from a raw per-disk residue scan rather than from a
listing, so it can refuse for a reason no S3 request can observe: the
client drains every version the API will show, DeleteBucket still returns
BucketNotEmpty, and the client-visible message is the generic "The bucket
you tried to delete is not empty" for every blocker kind.
The server does know which residue blocked it, and where — that is what
`bucket_delete_blocked` carries. But it was emitted at `debug`, below
both the `error` DEFAULT_LOG_LEVEL and the `info` the CI s3-tests lane
runs at, so it was never actually written down. An intermittent
BucketNotEmpty in that lane leaves a server log with no trace of the
refusal at all, which is not a diagnosable state: confirmed against the
artifact log of a failing run, where the rejected bucket appears only in
span-close lines and the blocker event is absent entirely.
Split the blocker kinds by whether the client can still reach the
residue. A visible version or a tier free-version is an ordinary 409 —
the bucket really is not empty and the caller can list and delete what is
left — so that stays at `warn`. UnknownXlMeta, OrphanDirectory, and
DiagnosticBudgetExceeded are on-disk state no S3 request can remove; that
is a server-side integrity problem and is now reported at `error`, with
the blocker kind, the residue counts, and the sample path.
This does not change what DeleteBucket accepts or rejects, and does not
retry or suppress anything — it makes the existing diagnosis reachable.
Refs #7005, #7010
fix(docs): update stale docs/README.md references to docs/architecture/README.md
The file docs/README.md was removed in a previous commit but references
in AGENTS.md, ARCHITECTURE.md, and CLAUDE.md were not updated. The
expanded check_doc_paths.sh now catches these stale references.
Update the workspace s3s pin and refresh the lockfile with cargo update/upgrade.
Remove the unused lifecycle url dependency reported by cargo shear.
Tighten the s3s footprint ratchet to the current observed baseline.
* fix(http): drain request bodies after early responses
* fix(http): log early response body drain failures
---------
Co-authored-by: houseme <housemecn@gmail.com>
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
Relocate issue-triage and pr-review from the tool-specific .mimocode
directory into the shared .agents/skills tree that every agent already
reads (AGENTS.md, .claude/skills symlink), and ignore .mimocode/ so a
local copy never gets recommitted.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <housemecn@gmail.com>
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
`build_metrics_summary` emitted a single metric entry for the local
deployment with `online` hardcoded to `true` and `last_online` stamped
with the current time, so `mc admin replicate status` reported "I am
online" rather than whether the remote site was reachable. A peer could
be down for minutes with replication failing while the status page
stayed green, leaving operators with no signal that the link had
dropped.
Emit an entry for every peer instead, deriving `online` from the
`reachable_peers` set the handler already computes by probing each peer,
and take `total_downtime`/`last_online` from the replication heartbeat's
existing `EpHealth` tracking. Node-local replication counters stay on
the local entry so a two-site cluster does not double-count its own
traffic.
The new `BucketTargetSys::endpoint_health` accessor deliberately does not
call `init_hc`: unlike `is_offline` it must not create health entries as
a side effect, or merely rendering the status page would mark an unknown
peer online.
Failure counters (`Errors`) are unchanged and still read zero; that is a
separate defect in the bucket-level statistics path and is not addressed
here.
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
`local_idp_settings` stamped the site region into the reported OpenID
settings whenever the federated identity service was published, which it
is even with OpenID disabled and no provider configured. The add
preflight compares those settings verbatim, so two sites in different
regions could never be paired: `replicate add` failed with `IDP settings
mismatch` while both sites reported an identical, empty `identity_openid`
config.
Report the empty OpenID settings when no provider is configured, so the
region only qualifies real provider identities, and name the diverging
field in the rejection instead of emitting a bare mismatch. Scalar values
are echoed; nested objects and credential-derived leaves are reported by
presence only.
Fixes#7003
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
Zhengchao An