* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
(cherry picked from commit 8f150d1d8e)
* fix(heal): preserve retryable batch failures during recovery (#7642)
* fix(heal): preserve retryable batch failures during recovery
* test(heal): pin prebuilt hooks binaries in ci
(cherry picked from commit 5cd58319ed)
* fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635)
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
(cherry picked from commit 50b31bc75b)
* fix(ecstore): make directory mtime fixture portable (#7623)
* fix(ecstore): make directory mtime fixture portable
* style(ecstore): format mtime fixture assertion
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
(cherry picked from commit b1cc286cac)
* fix(storage): prevent readiness after native migration failures (#7652)
* fix(storage): prevent readiness after native migration failures
* fix(storage): skip unsupported IAM records before reading
* fix(storage): use stable typed migration metadata errors
* fix(storage): include migration record in startup errors
* test(storage): cover native migration startup failures
* test(storage): use array chunks in migration fixture
---------
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 0cbc3ffe61)
* fix(admin): expose OIDC account display fields (#7654)
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
(cherry picked from commit f02bc947cd)
* fix(replication): correct peer joins and remote-state reporting (#7650)
* fix(replication): propagate verified peer deployment identities
* fix(replication): report actual remote peer state
* fix(replication): defer initial sync until all peers join
* test(replication): shut down TLS fixtures cleanly
---------
Co-authored-by: houseme <housemecn@gmail.com>
(cherry picked from commit 853ae63b6a)
* fix(s3): bound stalled UploadPart request bodies (#7659)
* fix(s3): bound stalled UploadPart request bodies
* fix(ci): preserve the S3S footprint ratchet
(cherry picked from commit 666dfd9f9f)
* fix(tables): reject reserved warehouse locations (#7671)
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 414176c47f)
* fix(ci): bind nightly lanes to one resolved source (#7688)
(cherry picked from commit 01d8e4347f)
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
(cherry picked from commit aeaba86d73)
* fix(e2e): require a verified server binary for every e2e run (#7687)
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
(cherry picked from commit 2909b1bfe1)
* fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697)
* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
(cherry picked from commit e2e6a2535a)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over
Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).
Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:
- Wiring updates a target in place only under the same ARN, or when it is
recognisably the site's own under an older ARN shape (same peer,
same-name target bucket, site replication service account). Anything
else gets the site target added next to it.
- The site resync manifest takes the target the derived
`site-repl-<deployment id>` rule names (same-name shape as fallback), so
an operator target to the peer neither aborts the bucket as "multiple
remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
same-name target bucket; operator targets stamped with the peer's
deployment id survive together with their rules.
Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.
* test(site-replication): drop a redundant clone flagged by clippy
The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.
(cherry picked from commit ecdc55fa4b)
* fix: enforce S3 permissions for recursive force deletion (#7661)
* fix: enforce S3 authorization for recursive deletion
* fix: satisfy the s3s footprint guard
* fix: restore list versions policy compatibility (#7686)
(cherry picked from commit 3fd1ce414d)
* fix(ci): repair functional defaults and chain regression checks (#7664)
* fix(ci): default functional suites to nightly packages
* test(ci): follow the fault-tolerance chain handoff
(cherry picked from commit 509a0fa90c)
* fix(ci): align security workflow tests with chain (#7679)
(cherry picked from commit d9e47d2813)
* test(ecstore): keep tier cleanup tests stable after immediate receipt queueing
Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics.
* Revert "fix(ci): align security workflow tests with chain (#7679)"
This reverts commit 4544359f6d.
* Revert "fix(ci): repair functional defaults and chain regression checks (#7664)"
This reverts commit 5ba7ec0291.
* fix(ecstore): pass shard integrity to backported ingest-mode test
The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup.
---------
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
The overnight run after #7708 proved the security verdict greps still
counted zero: real verdict lines are '\e[1;31m[FAIL]\e[0m STS-105 ...'
— the reset escape sits between the tag and the case id, and the
pattern only tolerated escapes before the tag. Allow escapes on both
sides; the fixture now emits the reset too, mirroring the real suite.
The tier case table is produced by rustfs_tier_report.py and its rows
lead with the topology column, so the case-ID-first table grep matched
nothing ('Product result: 0 passed, 0 failed'). Parse the PASS/FAIL
counts from the '## Case Summary' bullets the report always emits,
falling back to a topology-aware table grep.
First full serial pass with the green-on-case-failure semantics
(run 34693745171 / 34695021651) exposed three report-layer defects:
security: the report step referenced LOG_FILE, which is undefined in
this workflow (set -u killed the step before writing report.md), and
the verdict greps could not match the ANSI-escaped [PASS]/[FAIL] tags
in the real suite log. Point it at the artifacts suite.log, allow any
number of color escapes before the verdict tag, and count [SKIP]
lines separately (45 passed, 6 failed, 3 skipped was reported as an
unbound-variable crash).
pool: warp is stopped early (SIGINT) at the storage threshold and
only writes its final report on a clean exit, so an empty warp.log is
the expected shape of a healthy run - require its presence, not its
size. A mid-script die() abort or a FAIL step verdict must also turn
the validator red now that the run step is continue-on-error.
tier: add the standard 'Product result: N passed, M failed' summary
line computed from the case table, matching the other suites.
The contract test fixture previously injected LOG_FILE into the
environment and wrote verdict lines without ANSI escapes, which hid
both real-world defects; the fixture now mirrors the real suite
(stdout+tee with color tags) and asserts the pass/fail/skip counters.
A failing product case used to turn the whole workflow red, so the run
conclusion carried no signal beyond 'something failed' and the report
was suppressed. New semantics across the functional suites:
- Suite steps run with continue-on-error: the outcome is still recorded
for the report and the backlog issue manager (security/tier already
carried the flag).
- Generate report always publishes the full per-case table plus a
'Product result: N passed, M failed' summary, and its exit gate is
harness health: red only when the suite never reached case level (no
case verdicts), failed wholesale (zero passes, >=3 failures), or was
cancelled/skipped. performance is unchanged (parked).
- tier's structured gate no longer fails on case failures; it keeps red
for evidence-init and missing-gate-result breakdowns.
- pool/performance keep their existing red sources (install/benchmark).
Workflow contract tests updated to the new exit semantics: the security
report matrix keys green off the suite outcome, the evidence matrix
expects green for failure outcomes with recorded case rows (except
performance), the heal staged-rerun block expects the per-step table to
always publish, and run steps are now required to carry
continue-on-error.
Verified locally: actionlint clean; test_security_workflow.py 21/21.
RUSTFS_POOL_NODE_ENDPOINTS has no secret/var configured, so the suite
ran with the workflow's inline 3-endpoint fallback and the explicit
--node-endpoints flag overrode the script default fixed in
rustfs/auto-testing#61 — every dispatch died at startup with 'must
provide at least 4 direct node endpoints' (run 34677538650). Add
rustfs-node4 to the fallback; explicit secret/var still wins.
* ci: manage backlog issues by signal instead of per-run filing
The suite workflows used to file one backlog issue per failed run
(dedup was by run ID, which never matched), so issues accumulated
without bound. Replace the inline filing step in every suite workflow
(s3, kms, tier, storage, heal, pool, security, replication, upgrade,
performance) with a single call to
auto-testing/scripts/issue_manager.py, which:
- dedups by signal: failing cases are searched among open issues by
label (suite category + case ID); covered cases become a coalesced
comment on the existing issue, only uncovered cases file a new one
- labels new issues with functional-test, the suite category, one
label per failing case ID (lazily created), and env for
bootstrap-class failures (no cases ran, wholesale failure, or
404/ssh/clone/dpkg signatures in the log)
- closes open issues of the suite after a fully green run, citing the
run as evidence; cancelled runs never file or close anything
The step is skipped cleanly when auto-testing (private checkout) does
not contain the manager, or when PF_TESTING_GH_TOKEN is unset.
* fix(ci): satisfy actionlint and workflow contract tests for the manager step
- heal and performance workflows have no rustfs_version dispatch input;
referencing `${{ inputs.rustfs_version }}` in the manager step failed
actionlint's expression type check. Their package source now resolves
from package_url with the nightly fallback.
- scripts/test_security_workflow.py pinned the removed inline filing
step. The wiring assertions now pin the manager step (manager path +
per-suite report argument), and the evidence/stale-file tests assert
the skip contract instead: without the private auto-testing checkout
present, the step exits 0, publishes nothing, and leaves stale
evidence untouched.
Verified locally: actionlint clean, shellcheck clean,
test_security_workflow.py 21/21.
The rustfs_version dispatch input defaulted to 1.0.0-rc.4-preview.1,
which shadowed the nightly fallback and started 404ing once that
release was deleted. Manual dispatches with no inputs now fall through
to the nightly package (same contract the pool suite already has);
passing rustfs_version or package_url still pins the build exactly as
before. Chain (repository_dispatch) runs are unaffected: the inputs
context is empty there, so they always used the nightly fallback.
Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios are
derived from product source (default_parity_count, erasure set sizing):
- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
inside the reported divergence window (read quorum met while the
lock majority is broken)
By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.
Covers the findings of the 2026-09 external degradation report
(reads at read quorum, health-endpoint readiness truthfulness,
degradedReasons capture) as automated regression probes.
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
fix(nightly): push assets with plain git — the build fleet has no gh CLI
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Pin the star history workflow to the freshly released v1.4.0 of
overtrue/repo-visuals-action, which adds the chart-layout input and a
refreshed editorial chart/contributor-wall style. The workflow keeps its
existing inputs, so chart-layout stays at the editorial default.
Since rc.1 the latest tag has been frozen at 1.0.0-beta.12 because the
docker workflow only allowed alpha/beta prereleases to update latest
(#2732 dropped the rc case). Before 1.0.0 GA, latest is expected to
track the newest test build.
- Prereleases (alpha/beta/rc) now update latest as long as no stable
vX.Y.Z tag exists on origin, so the rule retires itself at GA.
- Channel tags (alpha/beta/rc) are now always added for prereleases;
the previous if/elif skipped the channel tag whenever latest was
created, which is why no :beta tag was ever published.
fix(heal): fall back to set-wide format for directory-backed replacement targets
Since #7018 `renew_disk` routes an unformatted local endpoint through the automatic replacement heal, which requires the target to be an independently mounted disk. Directory-backed deployments (the operator set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK`, which the startup disk-independence check already requires for endpoints sharing a device) can never pass that admission, so a runtime-wiped or replaced directory disk stayed unformatted forever: the heal task failed with "replacement target is not a stable mounted disk" and the auto-scan kept deferring the endpoint. This broke the Issue #1533 contract and the `heal_erasure_disk_rebuild_test` lane on main since 2026-09-02.
When the disk-check bypass is set and the replacement target fails admission, the erasure-set heal now logs a warning and downgrades to the ordinary set-wide `heal_format` path that predated replacement admission, and the auto-scan no longer defers such endpoints. The mount admission itself is unchanged and still cannot be bypassed by any environment variable; deployments without the bypass keep failing closed.
The endpoint-blackhole heal scenario now probes whether `iptables` can read the OUTPUT chain and logs an explicit skip when the host lacks `CAP_NET_ADMIN` (containerised runners report "Permission denied" from the nf_tables backend even under sudo); `RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1` turns that into a failure for lanes that provision the capability. The CI full-gate job surfaces the missing capability as a workflow warning, and the runtime-wipe fixture retries `remove_dir_all` on the listing race macOS surfaces as `DirectoryNotEmpty`.
Refs rustfs/backlog#2357.
* test(e2e): add distributed 4x4 validation
* ci(e2e): use tmpfs for distributed pool isolation
sm-standard-4 is an ARC pod without usable loop devices, so
mount -o loop fails with ENOENT before any pool filesystem is
attached. Sized tmpfs still gives each pool a distinct st_dev
and independent 1G statfs capacity.
Co-authored-by: RustFS <hello@rustfs.com>
* test(e2e): prove operations overlap data movement
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* fix(ci): preserve reported functional suite failures
* fix(ci): isolate functional evidence and preserve every result
* fix(ci): exclude sensitive scratch files from suite artifacts
Two fixes for the functional test chain:
1. Report upload fails with 'jq: Argument list too long' when the base64
report is passed through '--arg content' (pool reports exceed the OS
argv limit; last night's pool run lost its Step Results report this
way). Write the base64 payload to a temp file and load it in jq via
--rawfile instead. Applied uniformly to all nine suite workflows
that share this upload step.
2. The security workflow cloned rustfs/auto-testing into the workspace
and then ran actions/checkout at the workspace root for the OIDC
live gate script, which wiped the auto-testing clone and killed the
suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
Check out the repository into the rustfs-repo/ subdirectory instead
and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.
Co-authored-by: rustfs-ci <ci@rustfs.com>
The workflow_dispatch inputs already accept arbitrary release tags, but
the run failed late and unclearly when a tag had no .deb asset, and the
from_version default pointed at 1.0.0-rc.4-preview.1, whose release
ships no .deb at all - so scheduled runs died on a 404 while installing
the old package.
- Add a fail-fast preflight that resolves each requested tag via the
GitHub release API and verifies the rustfs_<tag>_amd64.deb asset
exists before the suite starts, with an actionable error message
otherwise (e.g. 1.0.0-rc.4 ships only zip/sbom assets).
- Change the from_version default to 1.0.0-rc.3, the newest release
that actually ships a .deb asset.
- Reword the from_version/to_version descriptions so manual triggers
state the .deb-asset requirement and the nightly fallback.
- Pass PF_TESTING_GH_TOKEN as GH_TOKEN to the suite step for the gh api
release lookups, matching the other functional workflows.
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
* ci(upgrade): render an upgrade matrix in the report; fix from default
The upgrade report only ever showed the requested deb URLs and a case
table. The nightly chain runs died installing the OLD package (default
from_version 1.0.0-rc.4-preview.1 has no .deb asset on its release, and
release 1.0.0-rc.4 ships none either), leaving an empty Total: 0 report
with no indication of what was upgraded.
- Default from_version is now 1.0.0-rc.3 (ships rustfs_1.0.0.rc.3_amd64.deb).
Matches the auto-testing default from PR #32.
- The report generator also parses the [UPG-TOPO] lines the suite now
emits and renders an 'Upgrade Matrix' section: per topology and KMS
backend, the versions actually in place before/after (captured via
'rustfs --version' on the node) and the aggregated result. When the
suite dies before any topology completes, the matrix says so instead
of silently showing nothing.
* fix(ci): use English headers in the upgrade matrix table
* ci(heal,pool): render step results and version in the reports
The heal and pool-expansion reports were only a raw log tail: no
structured indication of which steps passed, no overall verdict, and no
version information for the cluster under test.
The suites now emit machine-readable lines (auto-testing PR):
[HEAL-STEP] <n> <desc> PASS|FAIL [POOL-STEP] <n> <desc> PASS|FAIL
[HEAL-VERSION] <ver> (node <n>) [POOL-VERSION] <ver> (node <n>)
[HEAL-RESULT] PASS|FAIL <detail> [POOL-RESULT] PASS|FAIL <detail>
Both report generators parse them and emit a '## Step Results' section
before the log tail: the version captured in place via 'rustfs --version'
on a node, the overall verdict, and a per-step table. When a run dies
before any step reports (old script or early crash), the table shows a
NOT RUN placeholder row instead of silently showing nothing.
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
cxymds